lindner
February 12, 2019, 9:58pm
1
Hi there, just a note that I’m working with the Known developers to add IPFS support for data storage:
opened 02:12PM - 15 Jan 19 UTC
closed 05:35PM - 01 Apr 19 UTC
The current implementation of Known stores most media assets in a local filesyst… em and creates UUIDs to reference them. As an alternate I would like to propose supporting Content-Addressable storage. That is, storage based on the Hash of the content. This is used by [IPFS](https://ipfs.io) and other systems. It can be applied to both user uploaded content and especially for embed content that is fetched and rendered.
With this in place it will be possible to use a public IPFS gateway instead of a CDN (Content Delivery Network) Read more on how CloudFlare supports this in their [blog post](https://www.cloudflare.com/distributed-web-gateway/).
Ideal scenario is that core has the right hooks and then an IPFS plugin (or DAT, or Swarm, or other future system) can use these for to do their job.
## Proposed Solution
1. Replace or augment the UUID handling for uploaded content/images to use consistent hashing of the content for URLs.
- IPFS uses the hash of the content, or the hash of a container that holds references to other content.
- The same content always has the same hash values, this means that this content is deduplicated between instances and scales with usage.
2. Allow for an alternate URL prefix for content serving.
- Internal file-based, using the /file/ prefix, but use consistent hashing for paths that 'look like' hashes, due to encoding they will start with `Qm`
- Internal path-mapped IPFS, use /ipfs/ prefix on same server
- External IPFS content service like https://cloudflare-ipfs.com/ipfs/
3. On upload allow for hashing the content, storing content in an external system, and storing the resulting consistent hash code
4. Handle Deletes, since there is no deleting content once it's out there.
5. Support Content-Addressable assets in imageproxy. Instead of storing the encoded URL, store the content hash and pin the content locally. This allows for the same image to be shared among multiple instances.
6. Bonus points: store generated js/css assets in IPFS as well. On server startup find the static assets used and add them to the content-addressable backend and use the resulting hash codes to construct the serving URLs.
I’m trying to standardize blob storage in IPFS for the data I create. Feel free to drop in and make suggestions. And look forward to an IPFS enabled Known!
what is the benefit do you want to get from this.
The first plugin is ready to go:
Uploaded images are stored and served from IPFS.
Independently I’ve cleaned up references to static assets so they too can be served from IPFS.
@rogerlzp - the benefit I want to see is Known instances should reinforce each other by using IPFS. For example each instance creates thumbnails for comments with the same set of images. Using IPFS means that duplicated data gets served more effectively.
Eventually there’s more that can be done with storing some of the raw post data inside IPLD, but we’ll explore that later…