Memory Requirements for Large IPFS Server

1

I have a server with the following specs:
2 CPU cores
2 GB RAM
250Mb/sec unmetered connection
100 GB free disk mounted at $IPFS_PATH
“StorageMax”: “90GB”

Please note the following screenshot:

Yeu2U3R.jpg1440×1236 241 KB

I can see that it is hitting a memory limit at 2GB, so I recently added a swapfile.

Is there a memory/disk relationship that requires more ram for my setup?
What would be the recommendation?

1 Like
2

I don’t know how IPFS’ memory consumption scales with repo size, but that screenshot makes it look like you only have 2GB of RAM with a 4GB swap file.

3

Its a VM, and I have been adjusting the hardware values. The same issue occurs with 4GB ram.

4

That makes sense.

FYI there’s an issue for go-ipfs to reduce memory usage, but I don’t see anything in there about how memory consumption scales with repo size. It sounds like the devs are expecting to see a significant reduction in memory consumption with connection closing once that’s implemented.

2 Likes
5

I haven’t done the same thing yet for memory utilization vs repo size (at least, not yet), but I collected some memory utilization data vs peer count using a simple bash script I wrote for logging ipfs daemon memory usage, repo size, and peer count over time.

At least for an idle daemon on v0.4.10, each additional peer on my idle laptop consumes on average a bit less than 0.5 MB. I suspect this graph would look very different for an actively used daemon, but it should be straightforward to reproduce different test scenarios (e.g., offline daemon while adding content to repo, online daemon while adding content, online daemon pinning content, etc) as desired while capturing the relevant metrics with the script and then analyzing them afterwards to look for trends. In case it matters, the repo size is pretty much right around 8.9 GB for all of these data points.

The connections alone don’t seem like they could account for 2 GB (or 4 GB) of memory usage since I’ve never been connected to more than about 700 peers, so I’m guessing there are some other significant factors.

2 Likes
6

The size of your repo shouldn’t matter much. Usually, memory usage:

  • Starts out at some base usage (the offset you’re seeing at peer count = 0).
  • Scales linearly with the number of connected peers.
  • Grows over time as we learn about new peers and put them in our address book. Note: we’re planning on moving this to disk to reduce memory usage but doing that without thrashing the disk will take a bit of work.
  • Grows over time as we store data for the DHT.

There are two things you can do right now to try to reduce memory usage:

  1. Update to 4.11-rc2. We’ve fixed several major memory leaks in this release.
  2. Use the experimental --routing=dhtclient flag. This will prevent your daemon from participating in the DHT as a server.
2 Likes
7

So, it turns out I was somewhat incorrect. Performance shouldn’t scale with repo size but it currently does with the flatfs datastore (the current default). However, with 4.11 (just released) you can try the new (warning, experimental) badger datastore. It should be much faster and should better support large repos.

2 Likes