Having difficulty in making local go-ipfs connect to public ipfs

I have a newbie question about IPFS Port-Forwarding.

So, I’m trying many times to upload a folder of (10,000 files) to IPFS directly, and when testing the output hashes just some small random numbers of hashes are working in ipfs public https://ipfs.io/ipfs/<hash> and the majority are not just keep loading forever without output.

After searching in IPFS NAT configurations docs, I followed the following trails:

  • Activating UPnP ==> Same results
  • Creating a virtual server from the router to make port-forwarding to 4001 ==> Same results

So really I don’t know where the issue comes from.

And I’m using a Python script for uploading all those files using py-ipfs-http-client, and here is the script:

import json
import os
import glob
import ipfshttpclient

# Hashes for all the items
ipfs_hashes = []

# A list of image hashes after uploading to IPFS
img_hashes = []

# Hashes for all the JSON items
obj_hashes = []

# A list of JSON hashes after uploading to IPFS
final_hashed = []

# Reading all images from /items directory
with ipfshttpclient.connect() as client:
    ipfs_hashes = client.add('items', pattern='*.jpg')

for obj in ipfs_hashes:
    if obj['Name'] == "items":
        break

    obj_name = obj['Name']
    split_str_1 = obj_name.split('/')
    split_str_2 = split_str_1[1].split('.')

    image = dict({
        "name": split_str_2[0],
        "description": "description",
        "image": "https://ipfs.io/ipfs/" + obj['Hash']
    })

    img_hashes.append(dict({
        "name": split_str_2[0],
        "description": "description",
        "hash": obj['Hash']
    }))

    image_json_file = open("json_items/"+split_str_2[0]+".json", "w+")
    image_json_file.write(json.dumps(image))
    image_json_file.close()

obj_hashes = client.add('json_items', pattern='*.json')

for obj in obj_hashes:
    if obj['Name'] == "json_items":
        break

    item_json_hash = obj['Hash']
    final_hashed.append(item_json_hash)

json_object_hashes = json.dumps(final_hashed, indent=4)

hashes_json_file = open("hashed.json", "w")
hashes_json_file.write(json_object_hashes)
hashes_json_file.close()

json_img_hashes = json.dumps(img_hashes, indent=4)

img_hashes_file = open("image_hashes.json", "w")
img_hashes_file.write(json_img_hashes)
img_hashes_file.close()

I will leave below some useful information, please let me know if you want to check any other configurations.

  1. ipfs --version
    ipfs version 0.7.0

  2. ipfs daemon
    Initializing daemon…
    go-ipfs version: 0.7.0-ea77213e3
    Repo version: 10
    System version: amd64/linux
    Golang version: go1.15.2
    Swarm listening on /ip4/127.0.0.1/tcp/4001
    Swarm listening on /ip4/127.0.0.1/udp/4001/quic
    Swarm listening on /ip4/172.17.0.1/tcp/4001
    Swarm listening on /ip4/172.17.0.1/udp/4001/quic
    Swarm listening on /ip4/192.168.1.104/tcp/4001
    Swarm listening on /ip4/192.168.1.104/udp/4001/quic
    Swarm listening on /ip6/::1/tcp/4001
    Swarm listening on /ip6/::1/udp/4001/quic
    Swarm listening on /ip6/fd74:ae1:c698:db00:da5c:3a2c:f128:5c03/tcp/4001
    Swarm listening on /ip6/fd74:ae1:c698:db00:da5c:3a2c:f128:5c03/udp/4001/quic
    Swarm listening on /p2p-circuit
    Swarm announcing /ip4/127.0.0.1/tcp/4001
    Swarm announcing /ip4/127.0.0.1/udp/4001/quic
    Swarm announcing /ip4/147.75.195.153/tcp/4001/p2p/QmW9m57aiBDHAkKj9nmFSEn7ZqrcF1fZS4bipsTCHburei/p2p-circuit
    Swarm announcing /ip4/147.75.195.153/udp/4001/quic/p2p/QmW9m57aiBDHAkKj9nmFSEn7ZqrcF1fZS4bipsTCHburei/p2p-circuit
    Swarm announcing /ip4/192.168.1.104/tcp/4001
    Swarm announcing /ip4/192.168.1.104/udp/4001/quic
    Swarm announcing /ip6/::1/tcp/4001
    Swarm announcing /ip6/::1/udp/4001/quic
    API server listening on /ip4/127.0.0.1/tcp/5001
    WebUI: http://127.0.0.1:5001/webui
    Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8081
    Daemon is ready

  3. ipfs swarm peers
    /ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
    /ip4/107.21.57.178/udp/4001/quic/p2p/QmVM6gGjbEHkoVj1gaWbKCoDMQDVMRWu7LFuctSdUTtNjw
    /ip4/116.203.118.195/udp/4001/quic/p2p/12D3KooWSTw4kpj3gs8a9AbGu8GZmhHNVqUqxquAwaEYgow82Rw8
    /ip4/119.192.33.32/tcp/40031/p2p/QmR431tA9TKg2jTWavQSUku7QDyN55ZinqXBMUfLc3keur
    /ip4/119.237.29.91/tcp/45165/p2p/QmdixKgLAFrWiE9KmZAh8H8ci31cz9jRD384wR4344PEGC
    /ip4/13.52.104.158/tcp/4001/p2p/QmRRrNteAKzW7NR61AKNGjYQbCZYAbSub5382agqvaz5aS
    /ip4/141.51.112.209/udp/4001/quic/p2p/12D3KooWGHU9CcCdU4Aokb6KfYdGyQY3BNpPAavLexnvsciSNVZW
    /ip4/147.75.195.153/tcp/4001/p2p/QmW9m57aiBDHAkKj9nmFSEn7ZqrcF1fZS4bipsTCHburei
    /ip4/149.28.253.31/udp/4001/quic/p2p/12D3KooWCuFbkbPLM4MeEH7tFhrQ5RD69rVsq4ABmr49zpxwVNaU
    /ip4/165.227.155.27/udp/4001/quic/p2p/QmYzENQaBXTkTnQF9e27kaTBtzRBiXMducuPFtUhMCqvYm
    /ip4/175.212.105.13/tcp/33954/p2p/QmPJdhaELkxy8HBytbzALVyJH7ZfY5S7K8GcCyq9goEzdS
    /ip4/176.126.86.152/udp/4001/quic/p2p/12D3KooWAoP4akhGRAwXLYSY4q1fNFnLoD9PKJLjNnHqR8Dyhg9P
    /ip4/209.141.43.176/udp/4001/quic/p2p/QmUjjqVy8qwKMpc6ruHjakQkJVxTtuhv8ktqysrkQXEgys
    /ip4/211.216.184.19/tcp/39417/p2p/QmXaSkRZUBT55h5uXuKqzRQBe2BKGfjnNFxGYz737FQFJ6
    /ip4/218.187.128.157/tcp/49090/p2p/QmQdmXFdDZ5EYPqAG6eBZhTn5Y4ojczJbXV4bL4qKoW7Q1
    /ip4/221.165.76.196/tcp/52990/p2p/QmTaUER2rjDc9NUFHkm5Ukxvf4mQzSQ4chPtpNcZ5kXDEi
    /ip4/3.88.223.172/tcp/4001/p2p/12D3KooWSznpKdPnzLux8m2ZfPCdKK1PrfY9LvnmqMD5UKtc5bx3
    /ip4/35.81.238.43/tcp/4001/p2p/12D3KooWQYn8PuKzhy7MRHWenaKCNfUp278oMmX3sqxrUD1cQVmY
    /ip4/37.52.166.127/tcp/38207/p2p/QmQmRW67P2qXdNojGSMAyX9Q17rmftLchZbzgUhHUY4XWn
    /ip4/49.12.68.83/tcp/4001/p2p/12D3KooWCGRHsmJCuJncrdYHqGbohnpBHvf6N7xEvVUmEGkB8WwP
    /ip4/52.148.252.249/tcp/4001/p2p/12D3KooWEHzPn97Ur4dKXLyedjADb7G5fYRjZGoUyLJdeniNV7Kb
    /ip4/60.129.77.115/udp/32068/quic/p2p/12D3KooWF6NQYGnBUyiMqppDVzvXSz4H1Wcj2uvsBDFT29trLgeF
    /ip4/77.38.146.33/tcp/45775/p2p/QmPyt5UYb5c2LJpdV9okxy31SkdWNabTLUAJMrt7vv5jVL
    /ip4/78.47.190.31/udp/4001/quic/p2p/QmcbDiSY1qZarCchfwm99JxkdisQn46BAmmM3Zb4VwxhMR
    /ip4/95.217.233.192/tcp/4001/p2p/QmRaQKLKU1tDN4C6f5G1WWmi7hEUzbSP1K2yiFXNckP7jY

First thing, can you download thoses files from your own node that is used by the python script ? If yes the python script works fine, if no it has an issue.

It’s weird that UPNP isn’t working, at least can you make it work with an other piece of software like portmapper (or an other UPNP client) ? To know if it’s your router upnp that is an issue, or just IPFS not cooperating with your router.

For info, if you do a manual port forwarding, this isn’t enough, you need to also tell you node that this port forwarding exists (so your node advertise it others).
To do so look at this : go-ipfs/config.md at master · ipfs/go-ipfs · GitHub
You likely will add something like in your config :

[
  "/ip4/<your external ip>/tcp/<your external port>",
  "/ip4/<your external ip>/udp/<your external port>/quic",
  "/ip6/<your external ip>/tcp/<your external port>",
  "/ip6/<your external ip>/udp/<your external port>/quic"
]
1 Like

Thanks so much! I really appreciate your help a lot.

Yes there is an issue here

For info, if you do a manual port forwarding, this isn’t enough, you need to also tell you node that this port forwarding exists (so your node advertise it others).
To do so look at this : go-ipfs/config.md at master · ipfs/go-ipfs · GitHub
You likely will add something like in your config :

[
  "/ip4/<your external ip>/tcp/<your external port>",
  "/ip4/<your external ip>/udp/<your external port>/quic",
  "/ip6/<your external ip>/tcp/<your external port>",
  "/ip6/<your external ip>/udp/<your external port>/quic"
]

I configured the .ipfs/config and I will make another testing again to the script
Thanks a lot