Can't fetch files from other nodes

I deployed a private network, where there are multiple nodes connected to a single bootstrap node. For this example, let’s assume there are 3 nodes(A, B, C). A is the bootstrap node. B and C added A to their bootstrap nodes. When B adds a file, C can’t fetch that file until A fetches the file. Is it how it’s implemented, is there a way around or did I make any mistake?

1 Like

Interesting, I don’t observe this behaviour! Here’s what I tried, maybe you’ll see something I’m doing differently (nodes are not behind NATs):

First I generate a key, copying it to Node A, B, and C:

ipfs-swarm-key-gen > ~/.ipfs/swarm.key

Then I create Node A, the bootstrap node:

ipfs config profile apply server
ipfs bootstrap rm --all
systemctl --user start ipfs

I grab an address I can bootstrap with too:

user@localhost:~$ ipfs id
{
        "ID": "12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
        "PublicKey": "CAESIIKtysHcZnikKrEY/OdafraHo1StpVSKbMQd6I06I++b",
        "Addresses": [
                "/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
                "/ip4/139.177.196.92/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
                "/ip6/2600:3c04::f03c:92ff:fe49:b939/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
                "/ip6/::1/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC"
        ],
        "AgentVersion": "go-ipfs/0.8.0/",
        "ProtocolVersion": "ipfs/0.1.0",
        "Protocols": [
                "/floodsub/1.0.0",
                "/ipfs/bitswap",
                "/ipfs/bitswap/1.0.0",
                "/ipfs/bitswap/1.1.0",
                "/ipfs/bitswap/1.2.0",
                "/ipfs/id/1.0.0",
                "/ipfs/id/push/1.0.0",
                "/ipfs/kad/1.0.0",
                "/ipfs/lan/kad/1.0.0",
                "/ipfs/ping/1.0.0",
                "/libp2p/autonat/1.0.0",
                "/libp2p/circuit/relay/0.1.0",
                "/libp2p/fetch/0.0.1",
                "/meshsub/1.0.0",
                "/meshsub/1.1.0",
                "/p2p/id/delta/1.0.0",
                "/x/"
        ]
}

I’m using /ip4/139.177.196.92/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC in this example.

I create Nodes B and C the same way:

ipfs init
ipfs config profile apply server
ipfs bootstrap rm --all
ipfs bootstrap add /ip4/139.177.196.92/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC
systemctl --user start ipfs

Then I test:

First I verified each node could see each-other with ipfs swarm peers, I had 2 lines of output on each node, signifying each node is aware of the other 2.

I created a file named testfile on Node B:

Hello world :)

Node B

I add the file to Node B.

user@localhost:~$ ipfs add testfile 
added QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF testfile
 15 B / 15 B [===========================================] 100.00%

Node C

I can retrieve it on C instantaneously :slight_smile:.

user@localhost:~$ ipfs get QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF
Saving file(s) to QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF
 15 B / 15 B [======================] 100.00% 0s
user@localhost:~$ cat QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF 
Hello world :)

I didn’t change the profile to server. Does that effect the expected behaviour? :thinking: I’ll try changing the profile to server and see.

1 Like

Nope. Some times it works but most of the time it doesn’t. I waited for around 20-30 min but still no use.

What’s your output of ipfs swarm peers? Do the nodes just not see eachother?

If you see some nodes but not others, is there a pattern there? (Behind NAT)

Are you able to try with IPv6?

Edit: BTW is this you? distributed system - can't fetch files from other nodes - ipfs - Stack Overflow

In bootstrap node I’m getting all the other nodes but in other nodes I can only see Bootstrap nodes. I have 2 EC2 instances, where I changed Security Policy to allow all the traffic. Except for these 2 I think all other nodes are behind NAT.

If this is a NAT issue, have you considered using Relaying to defeat it? You could make the NATless bootstrap nodes into relays for your private network. Using AutoRelay I believe nodes should be able to discover the advertised relays.

Also FWIW you can try manually connecting to nodes using ipfs swarm connect, this could help with diagnosing.

Thanks for the help. Relay doesn’t seem to work for me. Though ipfs swarm connect is working I don’t think it’s a solution for the problem as it is impossible to add every node in other nodes.

Sorry for the delay!

If you’re able to manually connect two nodes behind the NAT, then the bootstrap node should be advertising those peers to eachother, allowing them to all connect to each-other and send data. The reason I suggested ipfs swarm connect is to assist in diagnostics.

What should be happening is you connect to a bootstrap node, that bootstrap node advertises it has us as a peer when other nodes connect. When we add a file, assuming you’re using a DHT, that file should get added to the DHT, and then the bootstrap node will be able to let other nodes know they can get the file from your node. Assuming they’re able to connect (ipfs swarm connect), which is supposed to happen transparently, automatically, they’ll connect to the node, and request the file.

If they’re unable to connect though, that’s where relaying comes in. Either using AutoRelay, or another mechanism to advertise the relay pathway. For a demo we’ll be publishing in the near future, I use a python script on a relay node, and advertise the relay pathway of connected clients over PubSub, which allows clients to discover each-other over the relay if they’re stuck behind NAT (I use that because the demo is browser-based, there should be better solutions for go-ipfs…).

I hope this helps, if you need more specific info on relaying, please don’t hesitate to ask. I know figuring some of this out can be daunting.

Thanks a lot. It’s a side project, which I started to explore new technologies. I work on it, in my free time, which is a bit scarce this days :sweat_smile:. I’ll look into relaying. I’ll post a comment if I get struck.

1 Like