Go-ds-flatfs vs go-ds-leveldb

1

I understand that there is some connection/relation between these two, but I’m not sure how they are connected/related.

My initial understanding was that the metadata of “what blocks are locally available” is stored in the leveldb inside $IPFS_PATH/datastore, and the actual block data is stored in the $IPFS_PATH/blocks.

But, after digging around a bit, I noticed two things,
1 - Blockstore uses prefix "blocks"
2 - After adding a bunch of files to my IPFS, I looked at the keys in the leveldb, and could not find a single key containing "blocks" prefix. I could only see /pins and /providers

This leads me to think that, in the default profile, both the leveldb and flatfs datastores are used - leveldb for metadata like pins, provider records etc. and flatfs for actually storing the blocks. And the leveldb does not contain ANY data/metadata about what is stored in the flatfs datastore.

2

(Apologies for the poor way in which I asked the above question)

Here is what I understand about flatfs and leveldb datastores now. Please correct me if any part of below statements are wrong.

  • go-ds-flatfs is a datastore implementation that is used only to store blocks. The data is actually stored as files in the filesystem, specifically in the $IPFS_PATH/blocks folder.
  • go-ds-leveldb is a datastore implementation that uses leveldb to store everything (like pins, provider records) except blocks. It does not store any data/metadate about the blocks at all. The data is stored as leveldb key value pairs. The leveldb location is $IPFS_PATH/datastore.

Thank you :slight_smile:

1 Like
3

Your understanding is mostly correct.

Really, they’re just two different datastores:

  • go-ds-flatfs is a very simple flat-file datastore that can only
    store blocks and doesn’t really implement all the features we expect
    datastores to implement (e.g., sorted queries, etc.).
  • go-ds-leveldb is a general-purpose datastore that can store
    anything. However, it doesn’t perform very well when storing large
    blobs of data (e.g., blocks).

By default, go-ipfs uses flatfs for blocks, and leveldb for everything
else. You can also replace one or both with go-ds-badger or mix and
match by “mounting” datastores on top of each other.

1 Like