Advice for starting a new ipfs-cluster

[luizirber]

Hello,

I’m about to create a new ipfs-cluster for storing ~9TB of data from 3.5 million files. Previously I was running an IPFS node and adding new files from a local dir, but I started having problems to keep it updated over time (the collections is growing by ~1k files a day, ~6GB). So this time I wanted to describe the problem here first, and ask for some advice on how to maintain it properly =]

More details:

• Since it’s a lot of files, what is the best store to use in this case? BadgerDS? Before I was using ipfs add --pin --nocopy --fscache {file} to pin new datasets and avoid creating two copies of the data, but it was a bit annoying to manage when I had to move data around (like when I needed larger HDs for storage).
• The initial node in the cluster has 16GB of RAM, a 256GB SSD and 16TB HDD (2x8TB disks). I was planning to store the blocks in the HDDs, but keep other metadata in the SSD. Is this enough/makes sense?
• Use crdt as consensus, since additional nodes will be followers and only the initial node will pin more data to the cluster.
• What options should I use with ipfs add to have good future compatibility? (which hash algorithm, CIDv1, …)
• The dir structure looks like this:

  .
  ├── wort-genomes
  │   └── <1.16M files>
  ├── wort-img
  │   └── <65k files>
  └── wort-sra
      └── <2.3M files>
  

  Mirroring this structure in IPFS is useful because it is easy to point IPNS to the root, and then accessing a different file can be done by changing wort-sra/ERR4020100.sig to wort-sra/SRR11555563. But… It never really worked, because that wort-sra/ dir has way too many files beneath it, and just never finishes loading. I was considering not building these dirs for now (and using another API for providing the mapping from dataset ID to IPFS hash), but is there any solution to this that doesn’t involve splitting the directory into smaller dirs (like git objects are represented, for example)? Is IPLD a viable solution for building these dirs with millions of children?

Thanks =]

P.S: this cluster will support the storage/distribution of the data from https://wort.oxli.org/, a system for indexing public genomic databases to allow large scale sequencing data search and comparison. I wrote more about it on chapter 5 of my dissertation.

[hector]

Hey

@luizirber

Since it’s a lot of files, what is the best store to use in this case? BadgerDS? Before I was using ipfs add --pin --nocopy --fscache {file} to pin new datasets and avoid creating two copies of the data, but it was a bit annoying to manage when I had to move data around (like when I needed larger HDs for storage).

Yes, for so many objects per node you probably have no other option given how flatfs degrades at high number of blocks.

@luizirber

The initial node in the cluster has 16GB of RAM, a 256GB SSD and 16TB HDD (2x8TB disks). I was planning to store the blocks in the HDDs, but keep other metadata in the SSD. Is this enough/makes sense?

You will need to define several mounts in go-ipfs config, a bit like flatfs does. Badger does not do this by default, but I think it should be possible.

For cluster, keep it on the SSD.

@luizirber

Use crdt as consensus, since additional nodes will be followers and only the initial node will pin more data to the cluster.

If the cluster is fully controlled by you, and you don’t expect your cluster peers to come and go regularly, use raft. The amount of state crdt will need to keep for such a fast-growing cluster will be significant and will force you to manually compact from time to time. Raft is better if you control the cluster.

@luizirber

What options should I use with ipfs add to have good future compatibility? (which hash algorithm, CIDv1, …)

Use --raw-leaves (which implies cid-version=1 I think). How big are the files? I would recommend bumping the chunker size to 1MiB. You may want to set the announce strategy in the dht to roots (ipfs config, see docs).

@luizirber

But… It never really worked, because that wort-sra/ dir has way too many files beneath it, and just never finishes loading. I was considering not building these dirs for now (and using another API for providing the mapping from dataset ID to IPFS hash), but is there any solution to this that doesn’t involve splitting the directory into smaller dirs (like git objects are represented, for example)?

Did you enable https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#directory-sharding--hamt ? If not, your directory blocks are over the 2MB limit, so they will not be sent around, and will never load.

Thanks for using ipfs and cluster for cool projects!

[luizirber]

Thanks for the awesome feedback, this is great!

@hector

Yes, for so many objects per node you probably have no other option given how flatfs degrades at high number of blocks.

Cool, so BadgerDS it is =]

@hector

You will need to define several mounts in go-ipfs config, a bit like flatfs does. Badger does not do this by default, but I think it should be possible.

If it gets too gnarly I might just keep it all in the HDDs, let’s see.

@hector

For cluster, keep it on the SSD.

@hector

If the cluster is fully controlled by you, and you don’t expect your cluster peers to come and go regularly, use raft. The amount of state crdt will need to keep for such a fast-growing cluster will be significant and will force you to manually compact from time to time. Raft is better if you control the cluster.

But with raft I can’t make a collaborative cluster, right?

(I have this dream that people will help host this data, although realistically it is only going to be me managing to grab some computational resources in different places from assorted academic grants )

@hector

Use --raw-leaves (which implies cid-version=1 I think). How big are the files? I would recommend bumping the chunker size to 1MiB. You may want to set the announce strategy in the dht to roots (ipfs config, see docs).

There are some large outliers (3.8GB, 1kb), but they are mostly in the 1-10MB range. And from what I found there is no problem from setting the chunker size to be larger than some file I’m adding, so I’ll do the 1MiB chunker.

@hector

Did you enable https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#directory-sharding--hamt ? If not, your directory blocks are over the 2MB limit, so they will not be sent around, and will never load.

Hah, that explains why I can’t load them =]

@hector

Thanks for using ipfs and cluster for cool projects!

Thank you for all the great work on it!

I’ll report back when I have something running (and more questions =P)

[hector]

@luizirber

But with raft I can’t make a collaborative cluster, right?

(I have this dream that people will help host this data, although realistically it is only going to be me managing to grab some computational resources in different places from assorted academic grants )

If it comes to that, you can always export, switch consensus type, and import.