In the case of multiple small files, which one should I use
“prefix”: “flatfs.datastore”,
“prefix”: “leveldb.datastore”,
“prefix”: “badger.datastore”,
50 ~ 80 KB file
In total 100TB
1 Like
The currently recommended option is to use flatfs with ssd/nvme disks.
Since you want to store many small files, I suggest spending some time studying the go-ds-flatfs library and considering whether it may be appropriate to use a different sharding suffix length. Performance will also probably depend on your filesystem configuration.
The main issue with flatfs is that many files in a single folder imply potentially multiple directory block-reads to open them, which reduces performance. More folders with less files, or bigger directory blocks on disk might help. In any case, 100TB is unexplored territory and you are going to need very good understanding of the pieces involved: ipfs, filesystem choice and options and hardware.