Apache as HTTPS Reverse Proxy for IPFS Node: memory leak

Help Kubo
,
1

I created the HTTPS proxy for IPFS node on my Apache server with the following settings:

Apache:

<VirtualHost *:443>
    SSLEngine on
    SSLProtocol all -SSLv2
    SSLHonorCipherOrder on
    SSLCipherSuite "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS"
    Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains"
    SSLCompression Off
    SSLCertificateFile /etc/letsencrypt/live/ipfs.mydoma.in/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/ipfs.mydoma.in/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
    ServerName ipfs.mydoma.in
    ServerAdmin admin@ad.min
    ProxyRequests On
    ProxyPreserveHost On
    ProxyPass / http://localhost:48080/
    ProxyPassReverse / http://localhost:48080/
</VirtualHost>

IPFS (fragment):

"Addresses": {
    "Swarm": [
      "/ip4/*.*.*.*/tcp/45001"
    ],
    "Announce": [
	"/dns4/ipfs.mydoma.in/tcp/45001"
    ],
    "AppendAnnounce": [],
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "API": "/ip4/127.0.0.1/tcp/65001",
    "Gateway": "/ip4/127.0.0.1/tcp/48080"
  },

The problem is that running daemon starts immediately to hang up many opened connections on Apache, and to consume a lot of memory spamming the log with the message:

2022-02-13T14:38:45.172+0300	INFO	bs:sess	session/session.go:459	No peers - broadcasting	{"session": 45, "want-count": 1}

This process consumes about 3GiB of memory when session value reaches ~1000.

The process works normally when proxying is disabled.

I tried to implement a lot of samples of HTTP proxying for IPFS, nothing works. Please help me to clarify the keystone of this situation. What could be a reason for such behaviour? How to set up Apache HTTPS-HTTP proxy for IPFS node properly?

2

Perhaps you are affected by 51814 – mod_proxy in Apache HTTP 2.2 FIN_WAIT2 in server side, it leaves as CLOSE_WAIT for a long time in mod_proxy side. ? (see apache 2.2 - Apache2 reverse proxy connections staying persistent, filling ssh channels - Server Fault)

In principle, the bitswap messages you see are not related to the reverse proxy connections not being closed though.

1 Like
3

We run our reverse proxies with nginx and it works fine btw.

4

Thanks, @hector ! These messages, however, I used as an indication point if the problem is present, and they indicate. It seems the migration to Nginx is becoming necessary in the old project to start involving IPFS…