Persistent P2P Serverless Multi-Party Shared Indexing Capability

The portion of persistency and internet censorship resistance is for IPFS is only partially implemented in my opinion.

A scenario which is most compiling to look into is censorship resistant resource indexing and one good example are torrent sites. Lately, a few torrent search engines have mysteriously disappeared and that got me thinking how best to make full use of IPFS to create some sort of a persistent online filesystem so that censorship is rendered almost implausible.

In the event that the hosting machine or host has been compromised and prevented from going online anymore, some sort of a distributed indexing system could be created to ensure that there is no reliance on a single host on the IPFS network for content.

The ability to copy files to create duplicates on other host are available but it will not be convenient to frequently copy and sync files and writing shell scripts to periodically sync and copy files from hosts can be completed.

A “distributed bookmark” feature can be created by not only creating a physical duplicate from a target host but also to create a sort of “Round Table” format where others who are interested in participating in the “distributed bookmark” can “bookmark” and make a copy and everyone whom have made a “distributed bookmark” will “sit at the round table” and check each other’s “bookmark” to ensure that everyone have a sync-ed copy and no one is tampering with the “distributed bookmark”.

In the even some authorities or some circumstances occurs to cause the original host or a few members of the “distributed bookmark” to be taken offline forever, the “distributed bookmark” will still be alive as long as a few members are still active and available. When more members join in to become members of the “distributed bookmark”, it will repeat the same procedures of automatically sync-ing and maintaining intergrity of the “distributed bookmark” in a “round table” style voting of some sorts to see if the bookmarks are maintained via majority votes of some sorts.

Also, this introduce the problem of distributed integrity management of the “distributed bookmarks” in some sort of lightweight ledger which I have not yet thought of and also the “what-ifs” that could happen if some agency decided to use all their computing powers to attempt to create a majority (kind of like the Bitcoin ledger problem where someone holds majority of shares of Bitcoin processing machines of sorts).

It will be interesting if IPFS can be equipped with such distributed and persistent indexing capabilities as such mentioned above.

3 Likes

I was thinking about similar problems and found OrbitDB provides great building blocks for this.

Instead of IPNS this relies on PUBSUB to share hashes as they are updated by changes in the database. Meaning you don’t rely on some authority with a keypair that could be compromised.

On top of that you can employ sharding strategies to balance between resilience and performance. For comparison, you wouldn’t want every person who wants to find a torrent or a wikipedia page to have to mirror a full database with potentially millions of entries. Even if it’s just metadata.

If you want to dig deep into these topics I recommend checking out the reference info in these repositories:

  1. About pubsub: https://github.com/libp2p/research-pubsub/
  2. About CRDTs: https://github.com/ipfs/research-CRDT
1 Like

Also @pgte is working on y-ipfs-connector, which uses yjs to put CRDTs on IPFS pubsub. This is very new work – @pgte has only had a few days to work on it so far but here is a quick demo of it in action: https://www.youtube.com/watch?v=8xE1MEyZsbo.

You might want to follow the work @daviddias and I are doing with @edsilv and @aescylus for the IIIF (International Image Interoperability Framework) conference in June, our goal is to use Dynamic Content over IPFS to synchronise IIIF document annotations. To track the development of that work, follow this issue https://github.com/ipfs/notes/issues/240

1 Like

More links with current work around IIIF on IPFS, which uses pubsub and CRDTs on IPFS:

We intend for these code bases to be useful references that show you how to do pubsub with CRDTs on IPFS

1 Like