IPFS, zsync, ZIM and maximizing deduplication

yeehi · August 22, 2021, 8:29am

zsync, the advanced file download tool that helps keep things in sync, is having a rewrite: zsync2

ZIM is a binary file format in which Wikipedia snapshots are produced by the Kiwix project and published at Index of /zim/wikipedia

One problem with .zim snapshots is that they are typically very large, hundreds of gigabytes for wikipedia, for example.

github.com/ipfs/distributed-wikipedia-mirror

ZIM on IPFS: maximizing deduplication

opened 07:35PM - 16 Apr 20 UTC

lidel

help wanted kind/question

> See parent issues first: #42 / https://github.com/ipfs/distributed-wikipedia-m…irror/issues/69 > **tldr** we are looking at low hanging fruits when it comes to deduplication during `ipfa add` of files from https://download.kiwix.org/zim/wikipedia/ ## Context ### :package: Each ZIM snapshot is a big file [ZIM is a binary file format](https://wiki.openzim.org/wiki/ZIM_file_format) in which Wikipedia snapshots are produced by the Kiwix project and published at https://download.kiwix.org/zim/wikipedia/ ### :arrow_down: Import to IPFS can be customized See relevant parameters (`ipfs add --help`): ``` -t, --trickle bool - Use trickle-dag format for dag generation. -s, --chunker string - Chunking algorithm, size-[bytes], rabin-[min]-[avg]-[max] or buzhash. Default: size-262144. --raw-leaves bool - Use raw blocks for leaf nodes. (experimental). --nocopy bool - Add the file using filestore. Implies raw-leaves. (experimental). --fscache bool - Check the filestore for pre-existing blocks. (experimental). --cid-version int - CID version. Defaults to 0 unless an option that depends on CIDv1 is passed. (experimental). --hash string - Hash function to use. Implies CIDv1 if not sha2-256. (experimental). Default: sha2-256. --inline bool - Inline small blocks into CIDs. (experimental). --inline-limit int - Maximum block size to inline. (experimental). Default: 32. ``` #### Low hanging fruit: `--chunker` When we import a file to IPFS it is chunked into pieces, and this chunking algorithm can be customized. The default one is just a static block of some fixed size (iirc 256KB), but in go-ipfs 0.5+ we support parameterized Rabin and experimental Buzhash chunkers. ``` -s, --chunker string - Chunking algorithm, size-[bytes], rabin-[min]-[avg]-[max] or buzhash. Default: size-262144. ``` ## :moneybag: Goal: maximizing deduplication with `ipfs add` The key problem with keeping ZIM around is their size: Turkish Wikipedia is tens of gigabyes, English wikipedia is 650 GB. Keeping multiple snapshots of the same language is expensive. Any level of deduplication that IPFS could produce would be highly beneficial. Kiwix project considered supporting something like zsync (https://github.com/kiwix/overview/issues/12) however I believe `ipfs add` with custom rabin or buzhash could produce even better results. We need to do some benchmarks to validate that claim. ### Questions - How much space (%) we can save on average within a single language? - This is important if we want to create collaborative pinning cluster per language - ..or across different languages? - This is important for Kiwix project, because if the numbers are significant, they could switch entire https://download.kiwix.org/zim/ to IPFS and save resources or be able to provide more snapshots. ## :chart_with_upwards_trend: Benchmarking plan (TBD) TBD, below is a doc with my initial ideas – needs refinement before we invest time, as it will be time-consuming to run it. - https://hackmd.io/5Ga2Z-usSDO7toY5wlDb1A I wonder if we could leverage [ipfs/DAGger](https://github.com/ipfs/DAGger) for automating benchmarks – thoughts @ribasushi? > **tldr** we are looking at low hanging fruits when it comes to deduplication during `ipfa add` of files from https://download.kiwix.org/zim/wikipedia/

Topic		Replies	Views
(draft) Common Bytes - standard for data deduplication Ecosystem	14	2556	December 27, 2022
IPFS with master and secondary nodes, disable unpin Coding files	3	262	May 1, 2022
Building Version Control System Over IPFS Help	1	170	November 22, 2023
Syncing MFS data between nodes? Ecosystem files	4	1192	March 7, 2020
Tutorial: Building a Versioning System With Blockstack and IPFS Coding	0	330	September 13, 2019

IPFS, zsync, ZIM and maximizing deduplication

Related Topics