Private Swarms and Private Gateway Domain

[flyingzumwalt]

From @jbenet on Tue Sep 29 2015 07:58:33 GMT+0000 (UTC)

Dan Barber wrote to ipfs-users@googlegroups.com

Hi Juan,

Now I know what i’m about to ask goes against the principals of what you want from IPFS, but for some research what I would like to do is create an intranet/private-tracker type IPFS and push to my own domain with my own swarm(s) instead of ipfs.io
Before I go through and tear up the code to achieve this, has it been considered before and have provisions been made for this kind of thing if you needed to move away form ipfs.io? Is there a nice config line to edit for example, or is a lot of it embedded?

Thanks,
Dan


Copied from original issue: https://github.com/ipfs/faq/issues/56

[flyingzumwalt]

From @jbenet on Tue Sep 29 2015 08:01:33 GMT+0000 (UTC)

> Now I know what i’m about to ask goes against the principals of what you want from IPFS,

Not at all, private and otherwise-disjoint networks are an important part of the internet. Our systems must work with them.

but for some research what I would like to do is create an intranet/private-tracker type IPFS and push to my own domain with my own swarm(s) instead of ipfs.io
Before I go through and tear up the code to achieve this, has it been considered before and have provisions been made for this kind of thing if you needed to move away form ipfs.io? Is there a nice config line to edit for example, or is a lot of it embedded?

Totally doable. A few points

• Private Networks are coming soon – see https://github.com/ipfs/go-ipfs/issues/1633 for an explanation
• https://ipfs.io is just a bunch of go-ipfs nodes running the HTTP gateway behind nginx. you can our infrastructure here: https://github.com/ipfs/infrastructure/ – which means you can replace it with your own nodes.
• Still, the canonical URLs are recommended to be set to /ipfs/<hash>/... and /ipns/<name>/..., and only-if-you-must (all of this year, until we get IPFS in the browser without needing an install), use fs://ipfs/<hash>/..., fs://ipns/<name>/.... – see https://github.com/ipfs/go-ipfs/issues/1678 for a lenghty explanation of why fs:. (ipfs:// is also allowed, but then you have ipfs://ipfs/<hash>. PLEASE DO NOT do ipfs://<hash>/... or ipns://<hash>/...
• If you want http://ipfs.io/ipfs/… urls to resolve to your infrastructure, please:
  • only do so for /ipfs/...' and/ipns/…` routes. (we use the root, to serve the project website. this may be ill advised on our part in the long run)
  • be clear with users that you are doing this.
  • ideally do this from the client-side, like the extensions do:
    • https://addons.mozilla.org/en-us/firefox/addon/ipfs-gateway-redirect/
    • https://github.com/dylanPowers/ipfs-chrome-extension
  • ideally redirect to some other domain you really control, by poisoning your own DNS (easy to do with /etc/hosts)
  • (this feels sketchy but workable)

HTH

[flyingzumwalt]

From @djbarber on Tue Sep 29 2015 08:18:48 GMT+0000 (UTC)

Hi Juan,

Thanks for the response! I am trying to get it behind my own domain, I do not want to use hosts workaround or mess with the URI structure.

Yep I have deployed the inf but I think most of my problems at this point are AWS based, my docker containers seem to crash when I attempt to attach to them. But Ansible completes all OK on the gateway and ipfs and common runs, i’m ignoring cjdns and pinbot for now.

I have a question regarding bootstrap nodes/peers, if I’m starting my own swarm there is going to be a chicken and egg scenario? It’s not entirely clear what the purpose of the bootstrap nodes is, but I have modified the lists to just point to my (currently single) instance of solarnet.

Required ingress public ports are 4001, 5001, 80, 443 and 8080 only?

Thanks,
Dan

[flyingzumwalt]

From @jbenet on Tue Sep 29 2015 08:38:28 GMT+0000 (UTC)

Hmmm, maybe things are crashing because of some assumptions particular to our infra? not sure. Perhaps describe deeper what you’re doing in an issue over at https://github.com/ipfs/support/

i’m ignoring cjdns and pinbot for now.

yep, no need for them usually.

if I’m starting my own swarm there is going to be a chicken and egg scenario? It’s not entirely clear what the purpose of the bootstrap nodes is,

it’s the nodes your nodes will connect to on boot. all distributed systems have one main problem: how do you establish the first connections to some nodes in “the network”? The answer for everything, from dns to bitcoin to bittorrent to IPFS, is to keep a list of a few nodes with the clients. (e.g. the root TLDs, or the bootstrap peers here).

So set the bootstrap nodes in the configs of all your private network nodes to bootstrap to your solarnet nodes.

More here: https://ipfs.io/ipfs/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/example#/ipfs/QmThrNbvLj7afQZhxH72m5Nn1qiVn3eMKWFYV49Zp2mv9B/bootstrap/readme.md courtesy of @insanity54

but I have modified the lists to just point to my (currently single) instance of solarnet.

yep, that’s right.

[flyingzumwalt]

From @djbarber on Tue Sep 29 2015 13:03:53 GMT+0000 (UTC)

Yeah there was a bunch of stuff that was being assumed…
In the end I actually decoupled it from Docker for better visibility into what was going on.

I intend to go back and post my findings to support site for you when i get some time in a few days.

Anyway i’d just like to post:

http://planets.everywhere.avid.com/ipfs/Qmd54HbuifRurC6MsoGb4YmctMbirjLGMuUwPJDUhXQ48T

It’s certainly possible to produce an independent cluster on private domain with a private swarm with some work.

[daviddias]

For the readers, you can find a new tutorial of how to create a pnet here https://github.com/libp2p/js-libp2p/tree/master/examples/pnet-ipfs#private-networking-with-ipfs