Ok… Just when I wrote that IPFS can’t do anything for limiting access, Peergos released a way to do access control…
Note that they are running their own forked IPFS clients (still open-source), so if you try to talk with regular IPFS clients, my understanding is that it would work except for this feature (Am I right @ianopolous ?). They are trying to upstream the feature to regular IPFS.