I am looking to implement a solution where I use a remote bittorrent client on the cloud server and then download the files to local system over http. This is expected to improve my bandwidth and overcome blacklisting my ip(suppose if I am behind proxy). My solution would be exactly like seedr.cc . One another improvement I want to provide is that the files/directories to be downloaded in parts locally(client application) simultaneously. This can also be done very easily if my solution is a client server application. Now, the real question, suppose if I want to not download the files/directories on the server at all, but on clients directly without mounting remote disk(client's disk). Here, remember the server is used for just the network, no storage is used at all. Do you know of any thing that can help?
3 Answers
Let me first describe your topology to be clear.
You build a server and some clients, the clients can download files using p2p. And you want a client can download file both from sever and other clients to accelerate the speed.
Firstly, the bt download can improve your speed but the limitation comes from your ISP. If you only buy a 100Mb bandwidth, you can not get a higher speed than it.
Secondly, if this topology can help you, then you can build such a systems.
NAT traversal is needed, it's the basic a p2p communication. You may need natpmp, upnp or ICE. Since you have a server, maybe ICE is better.
LSD and DHT will help you find other clients.
peer wire communiction protocol is needed to connect to other peers and download files.
There is a open source project named libtorrent may help you. It builds the whole system, you can use it or build your own by learning it.
As for downloading both form server and clients, there need to have a peer manager design, that is relatively hard, and I cannot make it short, you need to do many examples and optimization by yourself.
well , "rsync" in the title is a bit misleading and you're running into several problems here: the remote has to be reachable .. so having an "active tunnel" back to your clients will be a heavy task
for the incoming part: you could use lftp for torrents https://lftp.yar.ru/torrent.html and separate this per user , also helping you with quotas etc. , suggestion: use it in a screen or byobu session
for the download part:
- maybe use a ramdisk (
mkdir /tmp/ramtamtam;mount -t tmpfs /dev/null /tmp/ramtamtam/ -o size=64M) and let the client access this - use archivemount https://github.com/bramp/archivemount and send this to the client
- use external providers like mega with rclone and let the clients access this
- a neat optoin would is letting the clients access a torrent on your server through torrent itself with a custom magnet link , and add this to the running lftp session , since the download is at 100% for this you could use https://webtorrent.io/
- downloading over http can also be achieved with lftp's
mirrorcommand
Bittorrent clients generally require random access to the data they are transferring since they download and upload at the same time and have to serve random requests from remote peers. This means if you use a network filesystem then the performance of a bittorrent client will be constrained by both the upload and download bandwidth and latency of the host providing the storage. This is likely going to cripple the performance that a cloud instance could otherwise achieve.
Instead you could use a cloud instance (server henceforth) with ephemeral storage as accelerator. This can be achieved by letting both the server and your local client implement bittorrent but give preferential bandwidth treatment from the server to the client. This way the server will be downloading the data from other peers and at the same time transfer it to the client. The server can keep seeding a torrent from its temporary storage until it's rebooted.
Related