I am brand new to the world of ipfs, so please forgive me if I misuse any terms, or use terms that aren’t applicable to ipfs.
I want to build a tool that makes API requests and stores their responses, kind of like a metadata scraper for an API.
Specifically, I want to make a tool that retrieves .json responses from known addresses in the form of ipfs://exampleCID/index
The simple solution would just be to fire a bunch of requests from startIndex to endIndex, but usually the simplest solution is not the most optimal…
So I was wondering… what is the best way to accomplish this? Is there some black magic batching where I can download the entire directory or parallel request feature with ipfs?
Also, what is the best way to actually send the equivalent to a GET request? Without the use of a specialized client, the only way that I can find right now is to send a regular GET request to a gateway of some sort, like ipfs.io/ipfs/
I am planning to use Django for a backend, although if anyone knows a framework that works better with ipfs, or is better suited for this kind of task, please do not hesitate to recommend it.
Any help is greatly appreciated.