IPFS server keeps crashing

Help Kubo
1

Hey! We have a go-ipfs server with built-in s3 support plugin. And it works well except that the sever killed regularly, many times a day.

In logs I see:

Nov 29 05:18:05 IPFS-node-03 systemd[1]: ipfs.service: Main process exited, code=killed, status=9/KILL
Nov 29 05:18:05 IPFS-node-03 systemd[1]: ipfs.service: Failed with result 'signal'.
Nov 29 05:18:05 IPFS-node-03 systemd[1]: ipfs.service: Scheduled restart job, restart counter is at 31.
Nov 29 05:18:05 IPFS-node-03 systemd[1]: Stopped IPFS Daemon.
Nov 29 05:18:05 IPFS-node-03 systemd[1]: Started IPFS Daemon.
Nov 29 05:18:05 IPFS-node-03 ipfs[419597]: Initializing daemon...
Nov 29 05:18:05 IPFS-node-03 ipfs[419597]: go-ipfs version: 0.11.0-dev-5a61bedef-dirty
Nov 29 05:18:05 IPFS-node-03 ipfs[419597]: Repo version: 11
Nov 29 05:18:05 IPFS-node-03 ipfs[419597]: System version: amd64/linux
Nov 29 05:18:05 IPFS-node-03 ipfs[419597]: Golang version: go1.16.8
Nov 29 05:18:06 IPFS-node-03 ipfs[419597]: 2021/11/29 05:18:06 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 >

I also see some S3 access errors, but they are not time-related to the moment when the server got killed:

Nov 29 05:59:28 IPFS-node-03 ipfs[419597]: 2021-11-29T05:59:28.513Z        ERROR        engine        decision/blockstoremanager.go:92        blockstore.GetS>
Nov 29 05:59:28 IPFS-node-03 ipfs[419597]: caused by: Head "https://sfo2.digitaloceanspaces.com/ipfs/ipfs/AFYJVZAXXX3EH3KXXX7OFM4BXXXOOSJQY>

Except that there are no errors in logs, but I see some frequent memory spikes (VM has 2Gb RAM), and sometimes the server has less than 20Mb of ram according to Zabbix alerts (for like <5 minutes). Could these crashes be related to memory issues? Or what else might cause them?

2

Check the system log, if it is a memory issue there should be entries about it causing it to be killed (OOM).

But yes, for 2 Gb RAM you will need to optimize the configuration a bit. Also depends what other things you are running.

3

OK, I have updated VM up to 2CPU/4GB. Keeping eye on the logs…

4

Ok, looks like the amount of RAM was the problem. Now looks good and I don’t see any crashes.

5

I remember looking around in the documentation and finding that using ‘–profile flatfs,server’ as a commandline option helped a lot with RAM. I think flatfs is also now the default and I’m not sure how flatfs and S3 storage backend work together. Hope it helps.

1 Like