Minimal code to get pubsub consistently working using go-ipfs coreapi?

Hello everyone,

I’m trying to come up with the most minimal amount of code to get just pubsub working consistently.

Did this example: ipfs coreapi minimal pubsub · GitHub

The commented code does not seem to change anything regarding the consistency. Basically this sometimes works almost immediately or after a few seconds (subscriber receiving successfully the published messages) but most of the time the messages are just never received.

So I wanted to ask if there are important bits missing, or that I’m doing it wrong or what should be done to get a bit more reliability or to get an idea if it is having trouble subscribing and such.

Since you don’t bootstrap and rely on mDNS discovery for nodes to connect (all nodes must be on the local network) it is normal that things are very fragile.

For pubsub you want a well connected network where everyone is subscribed to the topic.

imho you are better off using a libp2p node + pubsub, rather than a crippled IPFS node and core API.

See this example: go-libp2p/examples/pubsub/chat at master · libp2p/go-libp2p · GitHub instead to start with.

If you are going to use the public network, you should go out of your way to locate peers that are subscribed to your pubsub topic and connect to them, and protect those connections (re-dial if they’re dropped etc).

This other example shows how to do rendezvous/peer discovery using the DHT: go-libp2p/examples/chat-with-rendezvous at master · libp2p/go-libp2p · GitHub

Thanks for the reply!

Since you don’t bootstrap and rely on mDNS discovery for nodes to connect (all nodes must be on the local network) it is normal that things are very fragile.

By following the code I thought that bootstrapping with the default values and peer addresses was already done when creating the node ?
I’ve been testing with two computers on different networks, the only way for them to connect is over the internet and pubsub worked so I was of the impression that DHT was doing its thing.

If you are going to use the public network, you should go out of your way to locate peers that are subscribed to your pubsub topic and connect to them, and protect those connections (re-dial if they’re dropped etc).

Ok, how could I achieve this with the ipfs api ?

I’ll check out the rendezvous example as it seems to be what I want to achieve, for 2 peers on the internet to communicate and send some short data segments. My focus is on pubsub but I was also interested in leveraging ipfs for the file storage while using pubsub.

Perhaps you are right. If they are on different networks and they are getting connected sometimes then it is pure luck, probably caused by both bootstrapping to the same peers and discovering other peers from there.

I assume there is a way you can access the underlying libp2p host/dht etc. or that the IPFS core api gives similar methods. See how ipfs dht commands work for example (in core/commands/ in go-ipfs). They use the core api to do all the stuff, like providing values on the dht etc.

I did more research and tinkering with the official ipfs binary. Wasn’t able to actually receive anything using it in pubsub mode even after many tries :frowning:

I tried

ipfs add <file>
ipfs dht provide <key>

on one computer

ipfs dht findprovs <key>

on the other and it was successful in finding the peerid of the other computer.

ipfs pubsub peers topic

on the second computer, listed the other computer. But doing it on the first one would not return anything.

ipfs swarm peers

on the first computer I can actually see the address of the second computer.

No message is coming through though.

So, I tried using go-libp2p directly to do pubsub and also using streams:

libp2p pubsub:

libp2p streams:

I noticed that most of the peers received after Advertising the Rendezvous string are just empty for some reason ?

Other than that it’s not doing any better than using pubsub via ipfs coreapi. The network of my second computer seems to be problematic, fetching files via ipfs works though.
Connecting to a wireguard VPN on that computer makes pubsub work more reliably.