Loading a batch of data from IPFS (Dag)

Hi :slight_smile:

We created a social media app on top of IPFS that is similar to Medium. We store posts as JSON objects to DAG, and we save images as files (not DAG) to IPFS. We load images through IPFS gateway and it works fast and images get cached by the browser. But posts work not so good. We cannot load them via IPFS gateway (because a post is stored as DAG?). And looks like the posts are not cached by the browser. Can you help us with the next questions, please?

  1. Does the content caches when it’s loaded from IPFS via the “GET” request? AFAIK, but still not sure, only images that we fetch directly from IPFS Gateway are cached.
  2. If only content from IPFS Gateway is cached, then how to access it using Dag? I’m asking because we cannot fetch content from the gateway using CID returned by Dag.
  3. Is there a way to load a batch of data from IPFS providing, for example, an array of CIDs?

You can track the issue in the network tab (DevTools) on the main page, or this one if you set pagination to 100 elements/page.

Here’s how it looks like:

1 Like

Hello IPFS team! We need your help :pleading_face:

1 Like

It’s best to use ipfs.dag.get or ipfs.dag.tree to pull the contents of a dag.

Check the Documentation

We’re developing a comprehensive social network framework on IPFS @ questnetwork.github.io/docs - It’s still going to take us a few weeks until we have all the features from Discord, Facebook, Twitter, etc but in 0.9.4 we’re already using dags for the timeline. In the future, you can always check the example app and library for how to do most operations on IPFS from the browser. You might even want to use our engine in your app once we’re production ready, it’s open source and we publish every release to NPM.
Check out if it could work for you, maybe you want to join us, work together on the library and use our library for your app too =)

Check the headers attached to the request. What do they say? I would expect they are the same as for images, and they indicate that content is immutable and therefore cacheable. By IPFS Gateway do you mean fetching content via :8080 (gateway port) vs :5001 (API port) ? I don’t understand why you would not be able to load a post (or any CID-addressed content) from the gateway.

Maybe there is difference in headers as exposed by the gateway and by the API, and maybe that’s the issue with caching.

Why not? What is the problem?

No, but these days connections have keep-alives and are re-used so that’s not a huge source of delays. What it seems from the screenshot is that your application triggers requests sequentially, and perhaps it could trigger them in parallel or in batches?

Why not? Some gateways expose a read-only sub-set of the API on their gateway port. I don’t know if the dag APIs are included in that subset or not. Nor do I know if/where might be a definitive list of which APIs are exposed. But it’s easy enough to test with curl if you have a dag CID handy!

Just tested and these all work (note especially the last one):

curl -X POST http://ipfs.io/api/v0/block/get?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

curl -X POST http://ipfs.io/api/v0/block/stat?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

curl -X POST http://ipfs.io/api/v0/cat?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

curl -X POST http://ipfs.io/api/v0/dag/get?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

By Gateway I mean exactly :8080/ipfs.
For example [working link] and [link with Dag provided].

That’s actually not what I meant:

  • You used the default IPFS CID in these examples.
  • You use IPFS API, not IPFS Gateway.

Check this message

We make requests in parallel using Promise.all(), but it looks like IPFS node still responds sequentially to our parallel requests.

1 Like

Hello friends, any suggestions?

Does your app not have a ‘backend’? The normal way this would be done is that your backend piece would collect all the data it needs to send to the browser all at once, in one request to load the entire page.