IPFS gateway propagation lag

I am running an IPFS instance using the latest Docker (0.4.13) and it’s connected to 826 peers. When I add a file to my IPFS, it usually takes some time to become available through the gateway https://ipfs.io/ipfs/ - usually about an hour.

Is there anything that I could be doing to make certain the file is available sooner?

You definitely want version 0.4.14 - and when you start your ipfs node (if it is 0.4.14) add this flag:

–enable-namesys-pubsub

DHT lookups shouldn’t take that long; they’ve been nearly instantaneous for me through the gateway when I’ve tested in the past. Even if it’s not directly related to your issue, I’d second updating to 0.4.14.

Do you get any errors from the daemon in docker when trying and failing to retrieve the file through the gateway?

You could also consider setting up a pinning service via ipfs-cluster. What I did for a 40GB dataset: I used rsync to get the files to the cluster bootstrapping node, which is in a datacenter with a 1GB NIC. Once there, I ipfs added them…

There is a number of factors that play into this. First and most important is your connection to the gateways themselves. Make sure you’re connected to peers Qmbut9Ywz9YEDrz8ySBSgWyJk41Uvm2QJPhwDJzJyGFsD6 and QmZMxNdpMkewiVZLMRxaNxUeZpDUb34pWjZ1kZvsd16Zic which are the current gateways (chappy and scrappy). A direct connection to those peers will hopefully help. But sometimes it doesn’t, and it could be because the gateways are already sending a lot of traffic, or they have a hard time downloading from you.

This will do nothing for general performance, it’s only a flag for activating publish/subscribe with pubsub for IPNS.

Is enable-namesys-pubsub an argument to the ipfs daemon, or something I would set via ipfs config? I don’t see that switch listed in the official docs

It’s a daemon option. So you’d enable it by using ipfs daemon --enable-namesys-pubsub.

That option worked! For reference, here’s my docker-compose.yml:

  ipfs:
    container_name: sampleproject_ipfs
    command: "daemon --migrate=true --enable-namesys-pubsub"
    environment:
      IPFS_FD_MAX: 32768
    image: "jbenet/go-ipfs:master"
    networks:
      - sampleprojectnetwork
    ports:
      - "8080:8080"
      - "4001:4001"
      - "127.0.0.1:5001:5001"
    volumes:
      - "./data/docker-ipfs-data:/data/ipfs"
      - "./data/share_content:/share_content"
      - "./tmp/ipfs-docker-staging:/export"

1 Like

That’s true, they are very outdated right now. I’ve made a PR to update them, in the meantime you can access them here: https://ipfs.io/ipfs/QmYBwwsCD15qBKTnipiTJBdvKyfJHLKN82EXMiL8Y5F7eP/docs/commands/

Glad that worked for you @leerspace