Fastest way to retrieve a folder's content from IPFS

Help IPFS
1

I’m trying to find a way to get all the contents of a folder within the least amount of time possible. I’ve tried several things but they none of them were satisfying. I want to mention that the data I’m trying to retrieve is very small (usually the size of the whole folder is less than 400 KBs) and all the tests were run on a server with 1 Gbit download speed so I don’t think bandwidth is the issue here.

  1. Most of the times, the content I’m trying to retrieve is indexed sequentially so I tried making a nodejs script to download each content of the folder by making GET requests to public IPFS gateways like this https://<IPFS_GATEWAY>/ipfs/cid/index and then incrementing the index. I also implemented multi threading to make it faster but because of public gateways being slow or rate limiting, I can’t go faster than 100-200 requests per second which makes it very slow if the folder has a lot of contents in it
  2. Given the low speed of public gateways, I tried accessing the IPFS data through my local go-ipfs node’s HTTP api, but it looks like my node couldn’t handle that many requests as I was getting timeouts from my own node.
  3. I tried changing approach by trying to retrieve the full folder rather than its contents one by one. I used go-ipfs’ api endpoints such as http://localhost:8080/api/v0/get?arg=/ipfs/CID/&compress=false&archive=true but it takes ages (over 3-4 minutes)

The only solution that comes to my mind would by trying to scale up option number 1 by switching to a more powerful server but it would be too much just to fetch 400 KB worth of data. I’m hoping there are smarter alternatives.

2

ipfs get the folder at once is the best way (your third solution).

Note the DAG traversal of get is pretty poor.
You can first ipfs pin add the folder, then ipfs get it, it’s usualy faster if you are downloading from shitty nodes. (even if you don’t care about pinning the file and ipfs pin rm it after)