Distributing data on cluster (using torrents?)

user_6036156 0
Viewed 722

I hope this is a good place to ask this, otherwise please redirect me to the correct forum. I have a large amount of data (~400GB) I need to distribute to all nodes in a cluster (~100 nodes). Any help into how to do this will be appreciated, following here is what Ive tried. I was thinking of doing this using torrents but I'm running into a bunch of issues. These are the steps I tried:

  1. I downloaded ctorrent to create the torrent and seed and download it. I had a problem because I didn't have a tracker.
  2. I found that qbittorrent-nox has an embedded tracker so I downloaded that on one of my nodes and set the tracker up.
  3. I now created the torrent using the tracker I created and copied it to my nodes.

  4. When I run the torrent with ctorrent on the node with the actual data on it to seed the data I get:

     Seed for others 72 hours
 - 0/0/1 [1/1/1] 0MB,0MB | 0,0K/s | 0,0K E:0,1 Connecting

  5. When I run on one of the nodes to download the data I get: 

     - 0/0/1 [0/1/0] 0MB,0MB | 0,0K/s | 0,0K E:0,1

  6. So it seems they aren't connecting to the tracker ok, but I don't know why

I am probably doing something very wrong, but I can't figure it out. If anyone can help me with what I am doing, or has any way of distributing the data efficiently, even not with torrents, I would be very happy to hear.

Thanks in advance for any help available.

bittorrent file-transfer cluster-computing
2 Answers

but the node thats supposed to be seeding thinks it has 0% of the file, and so it doesn't seed.

If you create a metadata file (.torrent) with tool A and then want to seed it with tool B then you need to point B to both the metadata and the data (the content files) itself.

I know it is a different issue now, and might require a different topic, but Im hoping you might have ideas.

You should create a new question which will have more room for you to provide details.

user_13627550

So this is embarrassing, I might have had it working for a while now, but I did change my implementation since I started. I just re-checked and the files I was transferring were corrupted on one of my earlier tries and I have been using them since.

So to sum up this is what worked for me if anybody else ends up needing the same setup: I create torrents using "transmission-create /path/to/file/or/directory/to/be/torrented -o /path/to/output/directory/output_file_name.torrent" (this is because qbittorrent-nox doesn't provide a tool that I could find to create torrents) I run the torrent on the computer with the actual files so it will seed using "qbittorrent-nox ~/path/to/torrent/file/name_of_file.torrent" I copy the .torrent file to all nodes and run "qbittorrent-nox ~/path/to/torrent/file/name_of_file.torrent" to start downloading

qbittorrent settings I needed to configure: In "Downloads" change "Save files to location" to the location of the data in the node that is going to be seeding #otherwise that node wont know it has the files specified in the torrent and wont seed them.

To avoid issues with the torrents sometimes starting as queued and requiring a "force resume". This doesn't appear to have fixed the problem 100% though

In "Speed" tab uncheck "Enable bandwidth management (uTP)"
uncheck "Apply rate limit to uTP connections"
In "BitTorrent" tab uncheck "Torrent Queueing"

Thanks for all the help and Im sorry I hassled people for no reason from some point..

user_60361560
Related