Data can only be found in IPFS after the server has been restarted

Hello, I have the problem that my data is no longer available after a while.
I don’t appear among the providers either, after a restart it usually works again.

My server uses the Routing: dhtclient is that the right mode?
I only want to distribute and deliver files, because of the laws in my country hosting foreign data is risky.
So I only want to deliver data that I added myself.
At the moment this is known software where I know I can offer it for download.
My plan was the other one because then just add hash to yourself and speed up the process.
But now I have the problem that my server no longer offers your data after a while and constantly needs a new pin and restart.
Config: https://pastebin.com/iysujJds
So I want to use IPFS like “torrent” for servers. :slight_smile:
So you can offer a higher download rate the more people add the file.
Well, it works. :joy:

I also have another problem with IPFS, when I have more than 100 connections at home, the internet is turned off.
The router firewall then reports problems and notice Illegal - Dropped FORWARD pack ans critical No Ranging Response received - T3.
So I have to severely limit the connections, which IPFS doesn’t adhere to at all after a while.
After 1-2 hours I have 300-600 connections although the limit is 1-10.
I have a DS-Lite connection at home.
It all works on my server.

Both dht and dht-client should works.

You should disable the reproviding in this case (I’ve seen your own is currently all).

EDIT: I think your problem is likely your router dropping packets, so other nodes start avoiding yours as it haves a very bad connection (from their point of view).

Okay, thanks for the info, I just had the server in the dhtserver as a test.
Strangely, files are no longer found here after a while, or are only available again after minutes.
For some, not at all, only after restarting and waiting a bit.
What is the difference to dhtclient and dhtserver?
The certificate(dhtclient) only says save performance and does not take commands from other servers.

In the config you can set a time in which you determine how often the server says that it is the provider of a hash. Should I put that up?

Here some examples:
Qman5EqMB2kFWQdGNGwjH3cghXDQfEG5nkKqpfnELfcSZQ?filename=svpihole2008.zip

https://ipfs.io/ipfs/QmSsRcmma1bbcPKNvqwu8E4iW6S4uJwDtzM8X4y3W2tD54?filename=svPiholeVM2008.zip

https://ipfs.io/ipfs/QmUKAPG1MNwoJ79kXbZrLRSNtPZkivS9hFg86tgREgoW53?filename=svpihole2010.zip

QmZpDehxeSbLoqu9rKf73d8DHL3Rfcjn2epkfTtPm9bW5k

With my problem at home, I also think that the router is causing problems, because of the firewall.
I have to test it, it can also be that Unitymedia / Vodafone causes problems because of my DS-Lite 400 Mbits Internet line. With Lite IPs several use one IPv4, so I use IPv6 now only.

So slow?? 250k/s
Testing

https://ipfs.io/ipfs/QmSaxipAocDXEW7r7D7Unpuhh15ctgv5LZ4E9p7nEg9eKK

Ok so first DHT have nothing to do about exchanging files.
DHT is a distributed directory where you can find 2 thing, peers and peers about a topic.

I’m gonna compare IPFS to torrent but be aware that this comparision is limted because some torrents client do implements a dht too (e.g. transmission).

If you want a file in a p2p network you need to somehow find peoples owning this file, you can’t just ask everyone around you randomly : “do you have this ?” (you could but this would be very inefficient).

How does torrent solve this issue ?

Simple, in a torrent you have so called tracker, a tracker is a server running some kind of software. It is very simple, a client who owns a file will contact the tracker and tell something like :

Hey I own this file (Qmfoo), here are my IP and port too if someone need them.

Then when someone want this file, he is gonna contact the server asking :

Hey, do you know about Qmfoo?

And the server answer

Yes, here is the list of nodes and their IP that I know have this file.

At this moment the second node contact the first one and they start exchanging the file.

And how does IPFS fix this issue ?

Ipfs have a DHT, in the end the DHT does the same thing than trackers do but they don’t have any central server, only lots of random node.
Basicaly in a DHT everyone is a tracker, everyone may store the list of people owning the file and the list of people wanting it.
But we have a new issue, let’s say we have 50k nodes (ipfs probably have more), how do we know wich tracker to use ? Ipfs CID (Qm…) doesn’t incorporate any metadata, just id of the hashing algorithm and the hash.

So DHT have something called distance the idea will be to somehow rank nodes seeing how far they are from your hash and we will use some of the closets nodes as trackers.
In IPFS the solution is to take the peer id (hash of the public key) of the node (Qm...) and the hash you are searching and XOR them together, the smaller is the resulting number the smaller is the distance beetween your hash and this node.

So when a node owns a file it will first find in his peerstore the node wich is the closets to his file. Then he will ask him:

Hi, could you store that I have Qmfoo for me pls ?

The node now answer:

No sorry, I’m only Qmbor and I know Qmboo wich is even closer.

Now your node restart the process with Qmboo wich may redirect to Qmfoa.
Qmfoa is not perfectly the hash of your file but as hashs are distributed randomly this is expected.
Qmfoa doesn’t know a better node and now stores that your node have the file Qmfoo.

Now lets say a node wants to find Qmfoo (e.g. you request it on the gateway), he will repeat the search process but might take an other route (ask Qmfej wich redirect him to Qmfoi and Qmfoi to Qmfoa).
Now Qmfoa tell to the node searching Qmfoo that he knows your node have it.

And now the gateway will contact your node and they will start exchanging the file together.

Basicaly in this whole process that I just told you, they were 2 types of nodes:

  • the clients (your node and the gateway), they only ask or store information in other nodes
  • and the server (all other nodes), these one guided you through the DHT and stored the information for you (in torrent it’s trackers who have this role)

and the dht-client option just disable the server part (by default both are active), so you still ask other node, they will store wich file you have but you will not do this for them.

2 Likes

Wonderful description @Jorropo! It may be worth stressing that the only thing the DHT stores is the LOCATION(s) of a particular hash, not the actual CONTENT of a particular hash. CONTENT is only reliably available on nodes that have it pinned and are actually running and reachable.

And “file” in your usage is actually “chunk” and actual files, particularly large files, may be composed of several (or many) chunks, each of which is identified by a unique CID which must be queried from the DHT to locate a node that (reportedly) has the content of that chunk.

At least, that’s my understanding of the way things work.

2 Likes

Thanks for the explanation, now I understand. :slight_smile:

So you have to control your swarm

I tried one way using scuttlebutt friend relationship
2 years struggling with IPFS… And how I succeed using it with ScuttleButt!

Now I am using https://cesium.app WOT and LIKE relations as their wallet identity is the same ed25519 key pair

1 Like