Ipfs-cluster pinning question

#1

Hi
Is the below scenario do-able with ipfs or ipfs cluster?

  • I want to have 3 devices on ipfs (ipfs-cluster) namely A,B,C, which are allowed to send files to get pined
  • I want to have 3 other devices on ipfs (ipfs-cluster) namely X,Y,Z, which pin any file sent for pinning from any of the devices in previous step
  • When device A,B,C send a file to be pinned, it should only be pinned on itself and on the devices in second step(X,Y,Z) and not on other devices in step 1.

As an example:
“A” marks a file sample.txt to be pinned. the file sample.txt gets pinned on A, X, Y, Z and NOT on B,C. Then device “B” sends file test.txt to be pinned. file test.txt gets pinned on B,X,Y,Z and NOT on A,C. the same for C.

P.S1: Devices X,Y,Z do not need to be able to send any file for pinning(doesn’t matter if they have the ability though)
P.S2: It is ok if when device A sends a file for pinning it does not get pinned on itself.

#2

I found this ,but could not understand the first part of suggestion for replication factor -1. does that suggestion do what I am asking here ? can someone be more specific with node names I mentioned above?

#3

It is possible. Set replication factor and allocations explicitally: ipfs-cluster-ctl --replication-factor 4 --allocations A,X,Y,Z pin add <cid>

1 Like
#4

The problem is group A, B, C and group X, Y, Z changes. So new nodes might get added or removed. So I cannot use fixed parameters. Something I noticed is if I define first group as cluster service and second group as cluster follower, then does it mean when A sends a file to be pinned, it only gets pinned on second group?

#5

If i were doing this I’d just let X,Y,Z be a cluster, and then let A,B, or C be able to send a message to the cluster when it wants to pin something. That way A,B,C participate the way you want (not pinning each other’s stuff)

1 Like
#6

Thank you. send a message with ipfs? Can you elaborate on this?

#7

You may be wanting to do it all inside IPFS where you don’t run any of your own code (which may not be possible) but if you have actual code of your own which runs to do the pins on A,B,C then right when you do those pins you can just update an X,Y,Z cluster right then, and one way of doing that could be IPFS PubSub, where your X,Y,Z cluster is listening for (subscribed to) an PubSub event which it will use to update the PINs. This is kind of an obvious solution, and therefore may not even be a good answer, and something you were already aware you could do.

1 Like
#8

Thank you! As obvious as it was, I missed this easy approach! Glad you mentioned it!

1 Like
#9

Yes, you can of course use the API of X,Y,Z and send the pins directly there ipfs-cluster-ctl --host /ip4/X-IP/tcp/9094 ... See Security and ports - Pinset orchestration for IPFS

2 Likes
#10

I tested pubsub and the issue is, if all peers in cluster are disconnected when the message is sent, then the cluster misses it and never pins. Any idea on that?

#11

A lot of people have a kind of misconception where they think IPFS is like some “database in the cloud” (like a blockchain). It’s not. There’s no repository you’re actually uploading to. There’s no “network” you can just rely on to always exist unless you’ve purchased some pinning service capability or are hosting it yourself. Think of BitTorrent. It’s like that. If no one is seeding your file, it can’t be found, and if none of your peers are listing on PubSub, you’re messages won’t be heard by anyone.

#12

I understand the concept and I will have my own nodes to serve the files. the purpose is to have some nodes that are allowed to change the pin (without pinning the contents themselves), and some nodes to save the files when new pinning requests are received.
I am not expecting any database feature, just need consistent pinning. Currently ipfs-cluster provides this consistency, however, all nodes in the cluster pin the content and I could not find a way to exclude a trusted node from pinning the content on itself.

So basically, can we tell a trusted node in cluster to not pin a content itself and only send the pin request to others peers?

#13

So we’re back to your original question then which was basically: “Can I have some peers in a cluster that are configured to do everything but pinning.” I think the answer to that was that it’s not supported, but I’m not sure.

If there were such a thing as a “Max space used by PINs” config setting (per peer), then you could theoretically set that to zero, but I’m not sure such a setting exists, or if it would throw a wrench in the gears to set it to zero.

#14

There is a mx size, but not sure if ipfs gc respects it. Anyone knows the answer? If I set StorageMax to something, if pinned files through ipfs-cluster reach this limit, and I run garbage collector, does it remove excess files to bring it down?

#15

Well, the whole idea of IPFS cluster is essentially wrapping pubsub in diverse logic so that this an other problems don’t happen.

#16

It does not get respected.

1 Like