Following situation: you have a large number of big files stored on IPFS and have some function to compute derived data. For the sake of an example, let’s assume that you have large videos and a function that computes thumbnail videos. Creating a thumbnail video takes a long time, so you want the result to be persisted.
How would you store this derived data in such a way that it is easy to find and to add to? We assume that the deriving function is a pure function. A node that wants to show a thumbnail video would first check if one already exists, and if not compute it itself.
One solution would be to have an IPNS name that points to a “thumbnail directory” containing the thumbnails, indexed by the hash of the original file. But the problem with this approach is that only one node can add things at the same time.
Is there a better way to do this? E.g. is it possible to use the DHT directly for something like this? I think this is a very generic mechanism (memoization) that might be worthy of direct support.