Testground my journey

Hi,

I will be documenting my journey into testground. I plan to write tests in rust lang for my app.

First step was to install docker, I did but in rootless mode (more on that later) then I installed go.

Worked fine until I tried testground daemon the command was not recognised. I asked on discord and @Jorropo helped me.

When installing go they don’t tell you to add a PATH to .bashrc

export PATH="$HOME/go/bin:$PATH"

Easy fix!

Next step was to run a test plan.

Can’t connect to docker. Docker host endpoint is wrong. Oh no!
Being not super familiar with docker I thought re-intalling docker without the rootless mode would work but it didn’t. You need rootless mode to use the docker extension in VScode anyway.

Back to .bashrc!

export DOCKER_HOST=unix:///run/user/1000/docker.sock

Did the trick since it wasn’t the default.

I was finally able to run the example but some containers errored.

Jan 10 22:14:28.645340  INFO    container isn't running; starting       {"container_name": "testground-redis"}
Jan 10 22:14:29.149892  ERROR   starting container failed       {"container_name": "testground-redis", "container_name": "testground-redis", "error": "Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.core.somaxconn: open /proc/sys/net/core/somaxconn: no such file or directory: unknown"}
Jan 10 22:14:29.153360  INFO    container isn't running; starting       {"container_name": "testground-sync-service"}
Jan 10 22:14:29.656777  ERROR   starting container failed       {"container_name": "testground-sync-service", "container_name": "testground-sync-service", "error": "Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.core.somaxconn: open /proc/sys/net/core/somaxconn: no such file or directory: unknown"}
Jan 10 22:14:29.659622  INFO    container isn't running; starting       {"container_name": "testground-influxdb"}
Jan 10 22:14:30.002163  INFO    container isn't running; starting       {"container_name": "testground-sidecar"}
Jan 10 22:14:30.387401  ERROR   starting container failed       {"container_name": "testground-sidecar", "container_name": "testground-sidecar", "error": "Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting \"proc\" to rootfs at \"/proc\" caused: mount through procfd: operation not permitted: unknown"}
Jan 10 22:14:30.387479  ERROR   doRun returned err      {"err": "healthcheck fixes failed; aborting:\nChecks:\n- local-outputs-dir: ok; directory exists.\n- control-network: ok; network exists.\n- local-grafana: failed; container state: exited\n- local-redis: failed; container state: created\n- local-sync-service: failed; container state: created\n- local-influxdb: failed; container state: exited\n- sidecar-container: failed; container state: created\nFixes:\n- local-outputs-dir: unnecessary; \n- control-network: unnecessary; \n- local-grafana: ok; container created.\n- local-redis: failed; failed to start container.\n- local-sync-service: failed; failed to start container.\n- local-influxdb: ok; container created.\n- sidecar-container: failed; failed to start container.\n"}

I’ll see tomorrow, if I start writing my own test plans or if I try to fix this mess.

1 Like

I investigated the errors and they were related to unprivileged containers. So I decide to re-install docker again… Did my best to remove any trace of it before hand (removing DOCKER_HOST from my .bashrc too).

Afterwards, tried to run the network example test plan and had new errors, I had forgotten to run make install again after reinstalling docker and was missing some containers.

Then it WORKED! All containers where online and the test ran to completion.

But why?

The containers must run under the default docker context not rootless otherwise they don’t have the privilege necessary.

Yesterday I ran my first test that I made!

I had to build my own synchronisation and figured out what environment variables are used.

The docs are useful but they did not list the variables so I had to log them, then built a client for the redis db. I took way too long to figure out how to build a “barrier”, I was overthinking it. A timer checking a number to see if it’s high enough, that’s it.

The docs are not made for other language that go, I wish they were more agnostic.

I’m now trying to find what kind of syncing is done by the sync service itself. It’s not documented… They assume your using the go-sdk.

1 Like

Yesterday I was able to list the IP address of all the containers and sync it using pubsub and barriers.

50 containers all syncing up is a beautiful thing!

I still can’t figure out how to wait for the “network-initialized” state. The topic and the event probably need to be formatted exactly right for it to work. Again, the docs assume your using go and they don’t even mention how it works.

Got problem compiling static bins (for tiny containers), so I used debian:buster-slim (only 8x bigger :stuck_out_tongue: ). Works fine, 80mb is manageable.

I still can’t figure out that “network-initialized” state, so I just YOLOed and I got libp2p to ping other containers!

You need far less things than buster-slim.
The only thing you need is:

$ ldd $(which ipfs) | awk '{$1=$1};1'
linux-vdso.so.1 (0x00007fff837fd000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f00f3234000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f00f322e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f00f303c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f00f3288000)

If you are feeling adventurous you can make your own container with only thoses things, it’s gonna be 4MiB big or so and work (you don’t actually need a valid linux install, that all golang use (if you don’t use plugins or cgo)).

You can also use a reflinking capable FS, like btrfs or ZFS (I don’t actually know if docker supports btrfs, I guess so but not tested).
Then docker is gonna use a reflink to create your dockers, meaning they will be copy on write backed and all files that they share will be only stored once and share between all of them.

1 Like

Thanks I’ll keep this in mind when I want to scale up. BTW I’m not using go it’s all in rust.

ldd /path/to/your/bin

Will tell you what you really need :slight_smile: (in most languages / things that doesn’t use after link time relocations)

I don’t know if rust supports it, but you can also build a fully static binary.
Which should just work without installing everything (it basically only require a kernel which your host provides). (now that I think about it you might need the linker still, not sure)

That’s what I was doing but had problems compiling some rust crates (libs), plus I read that the compile target had multi-threaded performance issues too and so I switched to buster-slim to focus on the simulation.

If I end up having problems with container size I’ll try what you said. Thank you.

1 Like