Integration tests for docker image

[tegefaulkes]

Specification

This issue is to track the progress of PKG integration tests for the docker image.

A selection of bin tests need to be updated to support using the docker image as the target for testing.

The main tests that need to be updated in order are;

1. bin/agent/start.test.ts
2. bin/bootstrap.test.ts
3. rest of bin/agent
4. bin/polykey.test.ts

Depending on the decided scope, the rest of the tests that could be updated are listed in MatrixAI/Polykey-CLI#10

Some things specific to docker that need to be checked refereed from https://github.com/MatrixAI/js-polykey/issues/389#issuecomment-1180048178.

1. Rather than --network host, you'll need an IP address and Port mapping, make sure this works on the CI/CD too
2. Rather than --pid host you'll need handling of the signals by the PK agent
3. With --userns host and --user "$(id -u)" this may still be necessary locally, in order to allow the files written to be readable by the test code which is running as part of our local user. But on the CI/CD this both may be the root user. But experiment with this.

Additional context

Tasks

• 1. update to use testIf to explicitly run the test if testing docker. Right now we implicitly running them by disabling the tests that are not supported for docker.
• 2. convert the following test...
  • bin/agent/start.test.ts
  • bin/bootstrap.test.ts
  • rest of bin/agent
  • `bin/polykey.test.ts
  • rest of the bin tests
• 3. update .env.example with any changes to the required env vars.
• 4. ensure CI/CD job is working.

[tegefaulkes]

So we're running into the quirks of using DIND to run our container. Currently the problem is how mounting the host FS works. Since we're running the dockerd daemon in the DIND service container we don't actually have access to the FS that the docker is running on.

This means I can't actually specify the temp file directory for the docker to run since we can't actually create it for the container. I'm not sure there is a way for us to share the file system between the main container and the DIND service.

That leaves us with the following options.

1. Skip or modify tests that require accessing files created by the docker container.
2. Modify the gitlab-runner to run it's own dockerd daemon instead of using a DIND service.

The simpler solution for now is just only do tests that don't require directly accessing the contents of the agent's directory. This is a pretty big restriction however since the docker image is just running the raw node code we don't need to be as exhaustive in the testing here. We only need a limited amount of tests to act as a sanity check.

[tegefaulkes]

Nice comment about the problem Here https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227#note_52029664

a possible solution mentioned here is using

variables:
  SHARED_PATH: /builds/shared/$CI_PROJECT_PATH

I'll look into this.

Update: Yep, looks like this works.

[CMCDragonkai]

The next comment is also insightful: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227#note_52092575

[CMCDragonkai]

And also https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227#note_128388116... seems like there's a few ways.

[tegefaulkes]

Sweet, it's working. In the recent test all but 1 of the tests/bin/agent/start.test.ts succeeded.

[tegefaulkes]

Both the start.test.ts and bootstrap.test.ts is passing now. From here on I just need to convert tests to use docker and the do some cleaning up.