fix mongodb connection issue when using TLS client + username/password authentication by mathiastop · Pull Request #30073 · hashicorp/vault

mathiastop · 2025-03-28T09:52:10Z

Description

This change aimed to fix a regression that was introduced by #9519, and fix #10985

Previously, when using a mongodb database with TLS and username/password authentication, the order in which Mongo config is merged worked in a way that the MongoDB x.509 auth mechanism will always win.

Step to reproduce:

Have a MongoDB instance that requires client TLS verification
Run vault secrets enable database
Run the following to provision the MongoDB secret engine config:

vault write database/config/mongo-example \
  plugin_name=mongodb-database-plugin \
  allowed_roles="*" \
  connection_url="mongodb+srv://{{username}}:{{password}}@cluster.mongo.local/admin" \
  username="example-user" \
  password="example-password" \
  tls_certificate_key=@/mongocerts/clientcertkey.pem \
  tls_ca=@/mongocerts/cacerts.pem

output

Error writing data to database/config/mongo-example: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/database/config/mongo-example
Code: 400. Errors:

* error creating database object: failed to verify connection: connection() error occurred during connection handshake: auth error: round trip error: (UserNotFound) Could not find user "CN=cluster.mongo.local" for db "$external"

mongodb logs

{"t":{"$date":"2025-03-28T11:23:28.061+01:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.16.191:36924","uuid":{"uuid":{"$uuid":"65a2d442-0fad-46af-a13c-1c390ee12ad8"}},"connectionId":11748,"connectionCount":8}}
{"t":{"$date":"2025-03-28T11:23:28.071+01:00"},"s":"I",  "c":"NETWORK",  "id":6723804, "ctx":"conn11748","msg":"Ingress TLS handshake complete","attr":{"durationMillis":9}}
{"t":{"$date":"2025-03-28T11:23:28.071+01:00"},"s":"I",  "c":"NETWORK",  "id":6723801, "ctx":"conn11748","msg":"Accepted TLS connection from peer","attr":{"peerSubject":"CN=cb0mongo1.sarbacane.local","cipher":"TLS_AES_128_GCM_SHA256"}}
{"t":{"$date":"2025-03-28T11:23:28.071+01:00"},"s":"W",  "c":"NETWORK",  "id":23236,   "ctx":"conn11748","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-03-28T11:23:28.072+01:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11748","msg":"client metadata","attr":{"remote":"192.168.16.191:36924","client":"conn11748","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.6"}}}
{"t":{"$date":"2025-03-28T11:23:28.080+01:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.16.191:36936","uuid":{"uuid":{"$uuid":"7ddeb17e-556f-412b-9976-83c53c829610"}},"connectionId":11749,"connectionCount":9}}
{"t":{"$date":"2025-03-28T11:23:28.080+01:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.16.191:36952","uuid":{"uuid":{"$uuid":"773f78ff-baf9-40f0-a093-ce5edd2dec0f"}},"connectionId":11750,"connectionCount":10}}
{"t":{"$date":"2025-03-28T11:23:28.091+01:00"},"s":"I",  "c":"NETWORK",  "id":6723804, "ctx":"conn11749","msg":"Ingress TLS handshake complete","attr":{"durationMillis":10}}
{"t":{"$date":"2025-03-28T11:23:28.091+01:00"},"s":"I",  "c":"NETWORK",  "id":6723801, "ctx":"conn11749","msg":"Accepted TLS connection from peer","attr":{"peerSubject":"CN=cb0mongo1.sarbacane.local","cipher":"TLS_AES_128_GCM_SHA256"}}
{"t":{"$date":"2025-03-28T11:23:28.091+01:00"},"s":"W",  "c":"NETWORK",  "id":23236,   "ctx":"conn11749","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-03-28T11:23:28.091+01:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11749","msg":"client metadata","attr":{"remote":"192.168.16.191:36936","client":"conn11749","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.6"}}}
{"t":{"$date":"2025-03-28T11:23:28.091+01:00"},"s":"I",  "c":"ACCESS",   "id":5286307, "ctx":"conn11749","msg":"Failed to authenticate","attr":{"client":"192.168.16.191:36936","isSpeculative":true,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cb0mongo1.sarbacane.local","db":"$external","error":"UserNotFound: Could not find user \"CN=cluster.mongo.local\" for db \"$external\"","result":11,"metrics":{"conversation_duration":{"micros":539,"summary":{}}},"extraInfo":{}}}
{"t":{"$date":"2025-03-28T11:23:28.096+01:00"},"s":"I",  "c":"NETWORK",  "id":6723804, "ctx":"conn11750","msg":"Ingress TLS handshake complete","attr":{"durationMillis":15}}
{"t":{"$date":"2025-03-28T11:23:28.096+01:00"},"s":"I",  "c":"NETWORK",  "id":6723801, "ctx":"conn11750","msg":"Accepted TLS connection from peer","attr":{"peerSubject":"CN=cb0mongo1.sarbacane.local","cipher":"TLS_AES_128_GCM_SHA256"}}
{"t":{"$date":"2025-03-28T11:23:28.096+01:00"},"s":"W",  "c":"NETWORK",  "id":23236,   "ctx":"conn11750","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-03-28T11:23:28.096+01:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11750","msg":"client metadata","attr":{"remote":"192.168.16.191:36952","client":"conn11750","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.6"}}}
{"t":{"$date":"2025-03-28T11:23:28.099+01:00"},"s":"I",  "c":"ACCESS",   "id":5286307, "ctx":"conn11749","msg":"Failed to authenticate","attr":{"client":"192.168.16.191:36936","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cb0mongo1.sarbacane.local","db":"$external","error":"UserNotFound: Could not find user \"CN=cb0mongo1.sarbacane.local\" for db \"$external\"","result":11,"metrics":{"conversation_duration":{"micros":382,"summary":{}}},"extraInfo":{}}}
{"t":{"$date":"2025-03-28T11:23:28.101+01:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn11748","msg":"Interrupted operation as its client disconnected","attr":{"opId":16261121}}
{"t":{"$date":"2025-03-28T11:23:28.101+01:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn11749","msg":"Connection ended","attr":{"remote":"192.168.16.191:36936","uuid":{"uuid":{"$uuid":"7ddeb17e-556f-412b-9976-83c53c829610"}},"connectionId":11749,"connectionCount":9}}
{"t":{"$date":"2025-03-28T11:23:28.101+01:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn11748","msg":"Connection ended","attr":{"remote":"192.168.16.191:36924","uuid":{"uuid":{"$uuid":"65a2d442-0fad-46af-a13c-1c390ee12ad8"}},"connectionId":11748,"connectionCount":8}}
{"t":{"$date":"2025-03-28T11:23:28.102+01:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn11750","msg":"Connection ended","attr":{"remote":"192.168.16.191:36952","uuid":{"uuid":{"$uuid":"773f78ff-baf9-40f0-a093-ce5edd2dec0f"}},"connectionId":11750,"connectionCount":7}}

with the fix:

vault write database/config/mongo-example \
  plugin_name=mongodb-database-plugin \
  allowed_roles="*" \
  connection_url="mongodb+srv://{{username}}:{{password}}@cluster.mongo.local/admin" \
  username="example-user" \
  password="example-password" \
  tls_certificate_key=@/mongocerts/clientcertkey.pem \
  tls_ca=@/mongocerts/cacerts.pem

output

Success! Data written to: database/config/mongo-example

mongodb logs

{"t":{"$date":"2025-03-28T11:27:40.482+01:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.16.191:54140","uuid":{"uuid":{"$uuid":"4ba110a3-ba4a-4879-873e-eaa1e99e49cf"}},"connectionId":11757,"connectionCount":8}}
{"t":{"$date":"2025-03-28T11:27:40.489+01:00"},"s":"I",  "c":"NETWORK",  "id":6723804, "ctx":"conn11757","msg":"Ingress TLS handshake complete","attr":{"durationMillis":6}}
{"t":{"$date":"2025-03-28T11:27:40.489+01:00"},"s":"I",  "c":"NETWORK",  "id":6723801, "ctx":"conn11757","msg":"Accepted TLS connection from peer","attr":{"peerSubject":"CN=cb0mongo1.sarbacane.local","cipher":"TLS_AES_128_GCM_SHA256"}}
{"t":{"$date":"2025-03-28T11:27:40.489+01:00"},"s":"W",  "c":"NETWORK",  "id":23236,   "ctx":"conn11757","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-03-28T11:27:40.490+01:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11757","msg":"client metadata","attr":{"remote":"192.168.16.191:54140","client":"conn11757","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.6"}}}
{"t":{"$date":"2025-03-28T11:27:40.497+01:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.16.191:54148","uuid":{"uuid":{"$uuid":"725b6069-9782-48fa-9f09-c616e174afd0"}},"connectionId":11758,"connectionCount":9}}
{"t":{"$date":"2025-03-28T11:27:40.498+01:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.16.191:54152","uuid":{"uuid":{"$uuid":"241c5737-f66f-4684-986f-3022f90f4ec7"}},"connectionId":11759,"connectionCount":10}}
{"t":{"$date":"2025-03-28T11:27:40.510+01:00"},"s":"I",  "c":"NETWORK",  "id":6723804, "ctx":"conn11758","msg":"Ingress TLS handshake complete","attr":{"durationMillis":10}}
{"t":{"$date":"2025-03-28T11:27:40.510+01:00"},"s":"I",  "c":"NETWORK",  "id":6723801, "ctx":"conn11758","msg":"Accepted TLS connection from peer","attr":{"peerSubject":"CN=cb0mongo1.sarbacane.local","cipher":"TLS_AES_128_GCM_SHA256"}}
{"t":{"$date":"2025-03-28T11:27:40.510+01:00"},"s":"W",  "c":"NETWORK",  "id":23236,   "ctx":"conn11758","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-03-28T11:27:40.511+01:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11758","msg":"client metadata","attr":{"remote":"192.168.16.191:54148","client":"conn11758","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.6"}}}
{"t":{"$date":"2025-03-28T11:27:40.517+01:00"},"s":"I",  "c":"NETWORK",  "id":6723804, "ctx":"conn11759","msg":"Ingress TLS handshake complete","attr":{"durationMillis":17}}
{"t":{"$date":"2025-03-28T11:27:40.518+01:00"},"s":"I",  "c":"NETWORK",  "id":6723801, "ctx":"conn11759","msg":"Accepted TLS connection from peer","attr":{"peerSubject":"CN=cluster.mongodb.local","cipher":"TLS_AES_128_GCM_SHA256"}}
{"t":{"$date":"2025-03-28T11:27:40.518+01:00"},"s":"W",  "c":"NETWORK",  "id":23236,   "ctx":"conn11759","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-03-28T11:27:40.518+01:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11759","msg":"client metadata","attr":{"remote":"192.168.16.191:54152","client":"conn11759","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.6"}}}
{"t":{"$date":"2025-03-28T11:27:40.518+01:00"},"s":"I",  "c":"ACCESS",   "id":6788604, "ctx":"conn11759","msg":"Auth metrics report","attr":{"metric":"acquireUser","micros":0}}
{"t":{"$date":"2025-03-28T11:27:40.528+01:00"},"s":"I",  "c":"ACCESS",   "id":5286306, "ctx":"conn11759","msg":"Successfully authenticated","attr":{"client":"192.168.16.191:54152","isSpeculative":true,"isClusterMember":false,"mechanism":"SCRAM-SHA-256","user":"exemple-user","db":"admin","result":0,"metrics":{"conversation_duration":{"micros":9878,"summary":{"0":{"step":1,"step_total":2,"duration_micros":256},"1":{"step":2,"step_total":2,"duration_micros":18}}}},"extraInfo":{}}}
{"t":{"$date":"2025-03-28T11:27:40.530+01:00"},"s":"I",  "c":"NETWORK",  "id":6788700, "ctx":"conn11759","msg":"Received first command on ingress connection since session start or auth handshake","attr":{"elapsedMillis":2}}

hashicorp-cla-app · 2025-03-28T09:52:23Z

All committers have signed the CLA.

jsdidierlaurent · 2025-03-28T09:56:48Z

👍

ju2tomate

👍

VioletHynes · 2025-04-08T19:53:04Z

Hi there! Thanks for this. I'm going to dig a little more into the subject area since I'm not quite an expert with Mongo. I've set the tests to run, too.

One thing I'd like to also understand: would this change be potentially breaking? For example, if a user in some way relied on the buggy behaviour, would they need to be made aware upon upgrade?

Would it be possible to add a new test to mongodb_test.go to prevent this regression from happening in the future?

mathiastop · 2025-04-15T16:36:23Z

Hello @VioletHynes :)
Thanks for your reply and sorry for the late answer.
Concerning your question if a user relied on the buggy behavior, this change does not break it. I only wanted to fix the #10985 via a "proper" way.

I've added a test TestGetTLSAndUserPasswordAuth in mongodb_test.go, and by working on the test I refactored a bit my change to made it better (in my opinion) than the first change I've made.

VioletHynes

Thanks for the unit test! This looks awesome. Going to run the tests then try and get this merged. The extreme detail in the PR helped a tonne, thank you!

VioletHynes · 2025-04-15T18:41:47Z

Oh, one more thing -- TestGetTLSAndUserPasswordAuth will need a godoc to pass our linter :)

mathiastop · 2025-04-16T07:27:26Z

Hello :)

I've added the godoc. Thanks for your help !

VioletHynes · 2025-04-16T14:02:18Z

Thanks! I've re-approved and re-ran the tests. I'd merge this now, but I need to get this approved by someone else who is a codeowner of one of the files, so expect that review/hopefully approval soon. Appreciate the patience.

benashz · 2025-04-25T15:35:10Z

since you have this testCase structure, I wonder if we should make this a table driven test. Even if there is only one testcase.

Indeed, this test is based on TestGetTLSAuth, which is a table driven test (I suppose).
In fact, instead of creating a new test, I should've update the already existing test to add a new verification.
Corrected by updating the TestGetTLSAuth function to add the new verification

benashz · 2025-04-25T15:37:06Z

It would be good if we could also check the actual error message.

As the test is now included in TestGetTLSAuth function, the error are checked in this test

benashz · 2025-04-25T15:45:16Z

Could we use the constant for this value from x/mongo/driver/auth instead? You will need to import go.mongodb.org/mongo-driver/x/mongo/driver/auth

Suggested change

authMechanism := "MONGODB-X509"

authMechanism := auth.MongoDBX509

benashz · 2025-04-25T15:48:57Z

Could we use the constant for this value from x/mongo/driver/auth instead? You will need to import go.mongodb.org/mongo-driver/x/mongo/driver/auth

Suggested change

authMechanism = "SCRAM-SHA-256"

authMechanism := auth.SCRAMSHA256

benashz · 2025-04-25T16:20:11Z

It seems odd that we are setting up the auth credentials for non-TLS auth here? Is that expected?

These option are only added when the verification line 251 is true, so only when we have at least a TLSCertificateKeyData

benashz · 2025-04-25T16:21:09Z

What if no AuthMechanism could be determined? It looks like that is possible here.

I'm not sure if I understood your question correctly 😅
These option are only added when the verification line 251 is true, and the authMecanism value is filled by default with auth.MongoDBX509 on line 253

…d authentication

blaumer · 2025-06-16T18:22:31Z

Thanks for the change @mathiastop ! This is expected to be in Vault 1.20.1

…d authentication (hashicorp#30073) * fix mongodb connection issue when using TLS client + username/password authentication * add changelog file * fix authMecanism more properly - add test for TLS and User/Password auth * add godoc on TestGetTLSAndUserPasswordAuth * update test to be included in TestGetTLSAuth - replace value by constant

mathiastop · 2025-09-25T10:15:50Z

Hello :)
"Resurrecting" this thread, when will this feature be available ? As you said it was supposed to be in the version 1.20.1, but it is still not available in 1.20.4 😅

Thanks !

fairclothjm · 2025-09-30T16:59:29Z

@mathiastop This didn't get backported to the release branch, so it won't be available until Vault 1.21. cc @hashiblaum

mathiastop · 2025-10-01T06:58:19Z

Hey @fairclothjm, thanks for the update ! I will stay tuned 👀

mathiastop requested a review from a team as a code owner March 28, 2025 09:52

ju2tomate reviewed Mar 28, 2025

View reviewed changes

mathiastop requested a review from a team as a code owner March 28, 2025 10:13

mathiastop requested a review from lane-wetmore March 28, 2025 10:13

heatherezell added the secret/database/mongodb label Mar 28, 2025

VioletHynes previously approved these changes Apr 15, 2025

View reviewed changes

mathiastop dismissed VioletHynes’s stale review via 5e8f7fb April 16, 2025 07:24

mathiastop requested a review from VioletHynes April 16, 2025 10:02

VioletHynes previously approved these changes Apr 16, 2025

View reviewed changes

benashz self-requested a review April 25, 2025 16:21

benashz reviewed Apr 25, 2025

View reviewed changes

mathiastop dismissed VioletHynes’s stale review via dd1633a April 28, 2025 09:28

mathiastop requested review from VioletHynes and benashz April 28, 2025 09:36

mathiastop added 5 commits April 28, 2025 14:12

fix mongodb connection issue when using TLS client + username/passwor…

ae4ec62

…d authentication

add changelog file

42825ab

fix authMecanism more properly - add test for TLS and User/Password auth

8ca1e84

add godoc on TestGetTLSAndUserPasswordAuth

aca0be6

update test to be included in TestGetTLSAuth - replace value by constant

9213cde

mathiastop force-pushed the fix/mongodb-tls-client-with-username-password branch from dd1633a to 9213cde Compare April 28, 2025 12:12

blaumer approved these changes Jun 16, 2025

View reviewed changes

blaumer merged commit 17054b5 into hashicorp:main Jun 16, 2025
68 of 69 checks passed

github-actions Bot mentioned this pull request Mar 9, 2026

🌱 CNCF mission generation 2026-03-09 kubestellar/console-kb#326

Closed

	authMechanism := "MONGODB-X509"
	authMechanism := auth.MongoDBX509

	authMechanism = "SCRAM-SHA-256"
	authMechanism := auth.SCRAMSHA256

Conversation

mathiastop commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

hashicorp-cla-app Bot commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsdidierlaurent commented Mar 28, 2025

Uh oh!

ju2tomate left a comment

Choose a reason for hiding this comment

Uh oh!

VioletHynes commented Apr 8, 2025

Uh oh!

mathiastop commented Apr 15, 2025

Uh oh!

VioletHynes left a comment

Choose a reason for hiding this comment

Uh oh!

VioletHynes commented Apr 15, 2025

Uh oh!

mathiastop commented Apr 16, 2025

Uh oh!

VioletHynes commented Apr 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blaumer commented Jun 16, 2025

Uh oh!

Uh oh!

mathiastop commented Sep 25, 2025

Uh oh!

fairclothjm commented Sep 30, 2025

Uh oh!

mathiastop commented Oct 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mathiastop commented Mar 28, 2025 •

edited

Loading

hashicorp-cla-app Bot commented Mar 28, 2025 •

edited

Loading