IPFS repair mechanism for durable storage

Let’s assume that you want to store some data that is very important for a long time. IPFS has some pretty good preconditions for ensuring data integrity, with content-addressed storage with secure hashes as identifiers.

So how would I set up a number of IPFS nodes to actually do store important data? Assuming that IPFS nodes are physically distributed in different AWS availability zones / regions as well as different physical machines.

First of all, IPFS nodes should be configured in such a way that they validate data on lookup, so you have a guarantee that you will never get wrong data for a hash. I guess this is covered by the HashOnRead config option, correct?

Second, there should be some kind of background process that regularly validates blocks, discards invalid blocks, and requests them again over the network. Basically a background repair mechanism similar to e.g. what is available for cassandra. Ideally, corrupted blocks should be stored somewhere (lost+found) in case you can’t recover the block from the network.

Does such a repair mechanism exist? If so, how would I configure it?

So how would I set up a number of IPFS nodes to actually do store important data? Assuming that IPFS nodes are physically distributed in different AWS availability zones / regions as well as different physical machines.

You’ll probably want to use something like ipfs-cluster. It’s currently alpha-quality but we are already running it ourselves and it works quite well (and uses the same CLI/HTTP interface):

All IPFS nodes will validate data received from other peers when they receive it. HashOnRead tells the node to re-hash when reading from the local datastore.

Does such a repair mechanism exist? If so, how would I configure it?

Not at the moment.

Note 1: In this case, we probably wouldn’t want repair to be fully automated as such failures usually mean that a disk is failing (and should be replaced before recovery is attempted.

Note 2: If you’re willing to make the failure-domain a single IPFS node, you can use IPFS cluster (I believe it has a repair mechanism although I’m not sure how automated it is.

1 Like

Thanks. We are definitely going to use ipfs-cluster on the cloud side of things.

Yes, this is exactly what I want, to protect against bitflips. If you store sufficiently large quantities of data, they do happen, even if your storage medium is not close to end of life.

The overhead should not be that big on go-ipfs, right?

Automated + logging errors to alert of impending trouble would be the best option.

These will be individual devices (android devices and raspberry pi), with pretty low specs, so it would be great if this was built in to the ipfs binary.

It seems to me that writing this from the outside is not possible because the ipfs repo api is not rich enough. What will ipfs repo verify do in case of a bad block? I could not find any docs.

Verifying (cryptographic) hashes on every read is expensive however, we can (and probably should) randomly sweep parts of the datastore.

It seems to me that writing this from the outside is not possible because the ipfs repo api is not rich enough. What will ipfs repo verify do in case of a bad block? I could not find any docs.

It returns a list of corrupted blocks. The missing pieces are:

  1. A nice machine readable output. One can easily parse the output of this command but it should have been machine readable from the start.
  2. A way to remove the corrupted blocks (so they can be re-fetched).

Hi @rklaehn, I’m the lead developer for ipfs-cluster. While we’re always improving, things are a bit rough to get started, so feel free to reach out if you need any help.

Can you put numbers on it? Will it actually slow things down, or will it just increase CPU load a bit on systems without extremely good persistent storage?

As mentioned above, I would prefer to have the block repair functionality in the IPFS binary. I mean, ensuring data integrity is a core function of ipfs for me.

However, I can understand if you decide to not have this functionality built in to keep things simple. But then it would be great if at least it was possible to implement it via some kind of script using the api.

Thanks for the offer. You will probably hear from us once we set up a cluster on AWS. You do support “cluster of cluster”, right? That would be helpful if you want to be extra paranoid and spread the data not just over availability zones but also over regions.

It will significantly increase CPU load (I don’t have numbers but hashing is already shows up pretty high on our CPU profiles). You can, of course, turn HashOnRead on if you’re willing to pay the price but most users won’t need it.

As mentioned above, I would prefer to have the block repair functionality in the IPFS binary.

I would actually like this (for pinned data, at least). However, it likely won’t happen in IPFS itself in the near future (unless someone steps up to implement it).

You can distribute your peers over regions while keeping them in the same cluster too. Cluster-of-cluster is not perfectly supported yet (but you can test it!)