Bug: Active Record secrets impossible to persist, making certain GitLab features like Dependency Proxy unusable

[Gaibhne]

GitLab added a few new secrets to store operational secrets in the database via Active Records (see Operational Secrets). These three keys are the relevant pieces in the encryption:

• active_record_encryption_primary_key
• active_record_encryption_deterministic_key
• active_record_encryption_key_derivation_salt

The first two are lists, as I understand, with the third being a simple value. When GitLab starts and detects that they are unset, it generates them like so (in /home/git/gitlab/config/secrets.yml - don't worry, those are not my actual keys):

  active_record_encryption_primary_key:
  - 1OmFswg5dUL5Cgl6i3CTVm3bUFXMDC42
  active_record_encryption_deterministic_key:
  - 5e7DiXxhy3g73Nd8oo4YuZwbLIW7XnZ3
  active_record_encryption_key_derivation_salt: ZAATKYJSHgr0DgrfX5d27LEeBs6gOGbH

Unlike other entries in that file such as secret_key_base, otp_key_base or encrypted_settings_key_base, the Active Record keys currently can not be set at all by this repository, not even through the normal process of providing them as ENV vars. I've looked into how the file is populated/updated by this repo, and from my understanding, no avenue to provide your own secrets is currently possible, because the file is generated from /etc/docker-gitlab/runtime/config/gitlabhq/secrets.yml which as you can see in https://github.com/sameersbn/docker-gitlab/blob/master/assets/runtime/config/gitlabhq/secrets.yml simply has no entries for those values (and when I tried to hack entries in there, they were not populated, so further work might also be needed).

I suggest that the same process that lets us set other secret variables through ENV vars such as GITLAB_SECRETS_SECRET_KEY_BASE and friends is expanded to cover these three new variables, as without such a change, features such as the Docker Dependency proxy are currently unusable - the feature gets corrupted every time the container restarts. I will file another issue with details on how to recover from that happening.

The only workaround I can think of to use such features at all with this container right now is to start the container, display the values of what GitLab generates (once it has been running for a minute) like so:

docker exec -it gitlab cat /home/git/gitlab/config/secrets.yml

Then write those copies to a file that is then mounted in place as a volume via /your/path/secrets.yml:/home/git/gitlab/config/secrets.yml with whatever permissions and settings are appropriate in your setup. WARNING: this will break various features of this repository by hard coding what is usually set by the above mentioned variables!

[Gaibhne]

The fix is relatively trivial and I already tried it for our local setup, I can make a PR when I find some time - as the encryption framework is scheduled to be adopted for more settings in the future, it's probably sensible to adopt support for it. The system supports rolling keys (on GitLabs side), indicated by being able to provide a list of keys, but I am unsure how to do that with the templating in this repository, maybe someone else can improve upon my PR (I imagine multiple keys could be provided via ENV vars containing a shell array).

[kkimurak]

Pushed kkimurak/docker-gitlab@1098126 while waiting for the pull request to be submitted (TODO: verify and fix command line usage in README.md)

To provide a list of keys, I think we could take a similar approach to RACK_ATTACK_WHITELIST to provide a list of keys (in this case, the keys are just alphanumeric strings, so the validation for each value that I implemented for RACK_ATTACK_WHITELIST in #2846 is probably not necessary.)

[Gaibhne]

@kkimurak as usual, life/work got in the way. Your PR seems pretty complete, thank you! I see some minor issues with the language used in the description, and I'm not sure the syntax for the first two keys is right - when GitLab generates these values, those fields are lists (like in my example above). Will your code work for single key values which are not surrounded with [ and ] ?

Let me know if you'd like me to clean up the language (and how), sorry I didn't have time to push my own changes.

[kkimurak]

Will your code work for single key values which are not surrounded with [ and ] ?

If you mean to set GITLAB_SECRETS_ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=first-key, my code will not work, I think. It should be ['single-key'] as it requires array of keys.

I will test it in my local environment, but since I have never used the dependency proxy and registry, I can probably only report that it started and that I was able to create an empty project/repository and git push/pull.

As a first step, I think it would be fine to submit your first suggestion as pull request and support multiple keys in the future work.

Also, I'm not good at English, so please feel free to edit anything in the text.

[kkimurak]

As expected: it started up, I could create an empty project/repo and do git push/pull. If you have any further testing instructions please post them and I'll try.

[Gaibhne]

My computer blew up so there was a bit of a delay. I made a PR, but please give me until tomorrow to actually test the changes - building GL locally takes a bit 😄 - when you are happy, you can just squash my changes into your PR. Though one correction - your commit message mentions 'Active Directory' instead of 'Active Records'

[kkimurak]

Though one correction - your commit message mentions 'Active Directory' instead of 'Active Records'

Oops.. Thank you for correction, I hadn't noticed. It's embarrassing (as usual)

but please give me until tomorrow to actually test the changes - building GL locally takes a bit

Tips: If you're only making changes to the runtime assets, you can mount the repository's assets/runtime to /etc/docker-gitlab/runtime on the container to see it work without having to rebuild the image.

[Gaibhne]

Thanks for that tip! Unfortunately, your syntax doesn't seem to work - the config file looks like this:

active_record_encryption_primary_key: '["long-and-random-alphanumeric-string","additional-random-string","yet-another-long-string"]'

The solution is the opposite of what the text states (which I'll correct and push before posting this comment):

In docker-compose.yml, you have to quote whole value is wrong, the correct syntax is:

- GITLAB_SECRETS_ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=["long-and-random-alphanumeric-string","additional-random-string"] without additional quotes, which results in a secrets.yml file looking like this:

active_record_encryption_primary_key:
  - long-and-random-alphanumeric-string
  - additional-random-string

Which I have tested and which works. The way you can test is to start a new GitLab instance, create a group (in my example test), and go to http://:10080/groups/test/-/settings/packages_and_registries - there, you turn on Enable Dependency Proxy and enter your Docker Hub credentials, save, and restart the container - without our fixes, those settings will now be broken, impossible to turn on again without database manipulation.

I pushed the PR so you can merge/squash whenever.

[kkimurak]

In docker-compose.yml, you have to quote whole value is wrong, the correct syntax is:

- GITLAB_SECRETS_ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=["long-and-random-alphanumeric-string","additional-random-string"] without additional quotes,

Hmm, maybe it's a style difference. I created a separate (and private) repository to manage docker-compose.yml to operate gitlab, and it uses KEY: value style to list environment variables, whereas this repository uses - KEY=value style. I'll check it this weekend.

I would like to document this repository with a syntax that can be tried with the style used in this repository. I will fix it. Or maybe I can write both.

For now, I will fix the commit message of my commit as you pointed out.
Since the destination of your pull request is my repository, would it be okay to merge this and then submit a pull request using that to sameersbn/docker-gitlab?

[Gaibhne]

Yes, that was my suggestion - I only made minor changes that don't really warrant their own commit, so you can just squash my commit into yours and submit that if you'd like.

[kkimurak]

@Gaibhne Merged kkimurak#2 and submitted a PR #3121 . Thank you for your contribution!