I’ve been wondering about this as well. An app I would love to see is a video streaming platform where my videos don’t get pulled because I walk by a radio playing something copyrighted.
I’ve been wondering about the cluster management and how intelligent it would be about reasonably distributing the data so peers aren’t overloaded.
I dug around the repo a bit and found this in allocate.go:
This file gathers allocation logic used when pinning or re-pinning to find which peers should be allocated to a Cid. Allocation is constrained by ReplicationFactorMin and ReplicationFactorMax parameters obtained from the Pin object.
The allocation process has several steps:
- Find which peers are pinning a CID
- Obtain the last values for the configured informer metrics from the monitor component
- Divide the metrics between “current” (peers already pinning the CID) and “candidates” (peers that could pin the CID), as long as their metrics are valid.
- Given the candidates:
- Check if we are overpinning an item
- Check if there are not enough candidates for the “needed” replication factor.
- If there are enough candidates:
- Call the configured allocator, which sorts the candidates (and may veto some depending on the allocation strategy).
- The allocator returns a list of final candidate peers sorted by order of preference.
- Take as many final candidates from the list as we can, until ReplicationFactorMax is reached.
- Error if there are less than ReplicationFactorMin.
I’ve been trying to understand how the preference order is determined. It has something to do with “
Metrics” coming from the pub/sub network. I’m still fuzzy on it.