I want to host files on IPFS and I have thousands of servers that will host the content. I have been attempting to use IPFS Cluster to make a server of mine act as cluster leader (the only trusted peer within the cluster) and define the pins. Then, I have all of my nodes join as cluster followers.
In testing, it works fine for the first 20-30 nodes. They all join the cluster, set their pins, and pin the files locally. However, after 30+ nodes the cluster leader becomes unresponsive even though there is no excessive CPU/RAM/Disk/Network that I can see. The main thing I can see is hundreds of open network connections, many of which are ESTABLISHED, which leads me to believe that it is trying to reach cluster followers that may have gone offline.
My best guess is that the cluster leader becomes saturated with open network connections, possibly because the default settings results in very frequent pings, check-ins, etc. I also guess that this is made worse by some of the nodes regularly going offline and coming back online, and their connections potentially still being retried from previous sessions.
Once it gets into this “bad state”, I am unable to list peers or pins - both commands simply hang for 5-10 minutes and never return. I guess this might be because it is trying to iterate through peers to establish the state and the peers are changing too frequently to make this possible so it is never able to complete.
So there are a few questions in here:
Does what I’m describing above sound like an issue of misconfiguration? Or is this simply not how Clustering is supposed to be used?
Are there limits to how many members can be in a cluster?
Should I bother using cluster since I will have a single source of truth for the pins anyway? Would it be better for me to make my own tool that tells my thousands of nodes which pins to keep?
Any help would be sincerely appreciated!