IPFS Peer ID getting set as localhost - K8s

1

Hi everyone, I have been using the results of this thread with great success for some time but we are seeing a very odd behavior recently:

Issue description:

2 IPFS nodes

Node A: 1 inside of kubernetes
Node B: 1 standalone node outside of kubernetes

Behavior:

When Node B connects to Node A, the address resolution for Node B does NOT MATCH the expected IP address. This results in file retrieval operations from A <-> B failing

  1. Node B executes the following, successfully connecting to Node A:

     ubuntu@ip-172-31-25-18:~$ ipfs swarm connect /ip4/54.202.115.116/tcp/30882/ipfs/QmcrMUeg6nNYp2CGMmC2Q1wqcx2ohZzMkebQNx1dQaxWT2
 connect QmcrMUeg6nNYp2CGMmC2Q1wqcx2ohZzMkebQNx1dQaxWT2 success

  2. Node A has a reference to Node B, but the IP address IS INCORRECT, referencing the internal k8s IP address…

     /# ipfs swarm peers | grep QmRe12uTMbx9WRiVVuZpacYjFhB5Pm6naM6XSHJmxmPXm1
 /ip4/192.168.15.104/tcp/10570/ipfs/QmRe12uTMbx9WRiVVuZpacYjFhB5Pm6naM6XSHJmxmPXm1

How can we resolve this issue?

Node A has been configured to announce a specific IP address and port retrieved from the cluster’s service definition - with the end result being something as follows:

/ # ipfs id
{
	"ID": "QmcrMUeg6nNYp2CGMmC2Q1wqcx2ohZzMkebQNx1dQaxWT2",
	"PublicKey": "",
	"Addresses": [
		"/ip4/54.202.115.116/tcp/30882/ipfs/QmcrMUeg6nNYp2CGMmC2Q1wqcx2ohZzMkebQNx1dQaxWT2"
	],
	"AgentVersion": "go-ipfs/0.4.22/",
	"ProtocolVersion": "ipfs/0.1.0"
}

In the above case, our config has been modified with the following:

"Announce    ": [
            "/ip4/54.202.115.116/tcp/30882"
        ],

The external IP is deduced from the node IP and the service port

Node B has the following connection information:

ubuntu@ip-172-31-25-18:~$ ipfs id
{
	"ID": "QmRe12uTMbx9WRiVVuZpacYjFhB5Pm6naM6XSHJmxmPXm1",
	"PublicKey": "",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/ipfs/QmRe12uTMbx9WRiVVuZpacYjFhB5Pm6naM6XSHJmxmPXm1",
		"/ip6/::1/tcp/4001/ipfs/QmRe12uTMbx9WRiVVuZpacYjFhB5Pm6naM6XSHJmxmPXm1",
		"/ip4/34.213.124.44/tcp/4001/ipfs/QmRe12uTMbx9WRiVVuZpacYjFhB5Pm6naM6XSHJmxmPXm1"
	],
	"AgentVersion": "go-ipfs/0.4.22/",
	"ProtocolVersion": "ipfs/0.1.0"
}
2

are you deploy your service on kubernetes (are you sure to expose 4001 and other node of your ipfs node).

3

Does this means after upgrading go-ipfs? Or randomly? Actually, I see 0.4.22. You should try 0.4.23, alhtough probably the issue is not related.

This results in file retrieval operations from A ↔ B failing

I am not sure this should affect file retrieval. There is a connection established between those two nodes.

Node A has a reference to Node B, but the IP address IS INCORRECT, referencing the internal k8s IP address…

I am a bit confused because Node B is supposed to be standalone, outside of kubernetes, so I’m not sure what the internal k8s IP address means here. By all means, the connection source address as seen by Node A was /ip4/192.168.15.104/tcp/10570.

4

Thanks for the replies! Hector - we had the same suspicion that it might not be related but noticed that the file retrieval issue is immediately resolved if we peer from inside k8s -> standalone (after explicitly disconnecting any localhost addresses, this ensures neither node A or node B have an internal k8s address)

I should have clarified, we are actually in the process of migrating from a set of standalone IPFS nodes outside k8s to a mix of IPFS k8s pods within k8s and standalone nodes outside k8s - prior to this migration we did not have this issue at all (file retrieval between nodes worked fine)

So, you are right that the connection seen by Node A is /ip4/192.168.15.104/tcp/10570 - however this is not the address that Node B actually lives at - trying to re-connect to that address from inside k8s ( /ip4/192.168.15.104/tcp/10570) simply does not work. I suspect the ingress routing of the request results in the final connection to Node A coming from a k8s local address.

When traversing the DHT to retrieve a file only present on Node B, it seems like Node A’s connection string for B is either invalid or retrieval fails at some other step…

5

If there is actually a live connection between the two (as swarms peers reported), then it should not need to traverse the DHT. There might be some weirdness in that setup, so that the connection seems to come from a different place and may not even work at all so that DHT fallback is necessary (maybe traffic only flows one way?).

In any case, if /ip4/192.168.15.104/tcp/10570 is what is seen rather than the real source IP outside k8s, I must only assume that k8s is doing some sort of address translation on ingress that is not really working (?). Unfortunately I am not so familiar to networking on kubertenes to say…

6

Yeah, this is exactly what we suspect is happening - that the connection ‘seems’ to come from a different place due to ingress address translation. This would not be an issue if file retrieval worked despite this odd local connection behavior but unfortunately it does not.

7

I think some more information is needed here.

  1. Are you able to share the K8S pod, service and ingress (if used) manifests? If not, information about pod network settings, service type and configuration and if ingress is being used
  2. General environment details. Are both nodes on AWS, GCP, bare metal, etc?