Big file transferring problem

Help Kubo
1

Hi all,

I am working on ipfs with big file sharing. The file size is ~10GB. I have done 2 tests.

Question first

  1. What is the expected throughput when transferring big files with IPFS?
  2. For my tests, any suggestions to optimize the throughput?
  3. I have read issue#3786 of bitswap session, but it seems not work in my test cases.

######################################

IPFS version

0.4.17-release

Test 1

environment

40 vm node, cpu 4 core, mem 8GB, disk (local disk, which means using the files on the host machine)
1 node is seeder, hosting a 9.26GB video file.
all 40 nodes are in the same local network.

Scenario

run ‘ipfs get xxxxx -o yyyy’ simultaneously on the rest 39 ipfs node.

Observation

  1. Each node could download within ~30MB/s throughput.
  2. CPU: ~200%, Mem: 1GB, Net: 600Mbps
  3. I have tried to change the node amount, but the throughput is always ~30MB/s.

Test 2

environment

2 server node, cpu 48 core, mem 192GB, disk 2TB
1 node is seeder, hosting a 9.26GB video file.

Scenario

run ‘ipfs get xxxxx -o yyyy’ on the other node.

Observation

  1. Throughput: 60MB/s
  2. CPU: ~200%, Mem: 1GB
1 Like
2

I’m pretty sure bitswap sessions are not implemented yet, so the issues it’s expected to address are likely still present in your test environment.

1 Like
3

Also note: 600Mbps = 75MB/s. Bitswap sessions and better read-ahead should improve this.

4

Making ipfs get use bitswap sessions (Which are definitely implemented) should help this. In addition, the work being done here: https://github.com/ipfs/go-bitswap/pull/8 should also help noticeably.

2 Likes
5

My mistake. Are bitswap sessions something that has to be enabled for every command that requests data from other nodes? If so, it sounds like there might be other commands besides ipfs get that aren’t using them (assuming based on your post that’s currently the case for ipfs get).

6

Yeah, it’s something that needs to be enabled in the code for each command. For example, the resolution of the path and the fetching of the data needs to use the same session for maximum effectiveness. We haven’t been pushing this forward lately, but we should get back on that soon.

3 Likes
7

Sorry for the mistake.

Net: ~500Mb/s, observed by iftop

Throughput: ~60MB/s , caculated by 9.26GB / transferring time

There should be some overhead on the protocol.

8

Thank you, @whyrusleeping !
So my test results are with bitswap sessions, and https://github.com/ipfs/go-bitswap/pull/8 could help more, right?

Hoping it could be merged soon~