(database/mongodb) Regression: username/password auth can't be used with client TLS in Vault 1.6+

[pbar1]

Describe the bug

If a Mongo database is provisioned at database/config/mongo-example, and both username/password and TLS certificates are passed, the MongoDB x.509 auth mechanism will always win; this renders other auth mechanisms unusable. This breaks MongoDB secret engines that are using client TLS in addition to other auth mechanisms such as SCRAM-SHA-256, when upgrading from Vault 1.5 to 1.6+.

This is due to the new order in which Mongo config is merged in plugins/database/mongodb/connection_producer.go:

client, err = mongo.Connect(ctx, options.MergeClientOptions(options.Client().ApplyURI(connURL), clientOptions))

To Reproduce
Steps to reproduce the behavior:

1. Have a MongoDB instance that requires client TLS verification
2. Run vault secrets enable database
3. Run the following to provision the MongoDB secret engine config:

vault write database/config/mongo-example \
  plugin_name=mongodb-database-plugin \
  allowed_roles="*" \
  connection_url="mongodb://{{username}}:{{password}}@mongo-example.svc.cluster.local:27017/admin?authMechanism=SCRAM-SHA-256" \
  username="example-user" \
  password="example-password" \
  tls_certificate_key=@/mongocerts/clientcertkey.pem \
  tls_ca=@/mongocerts/cacerts.pem

3. Upon attempting to verify the database config object, Vault cannot authenticate via username/password and instead is forced to use the client TLS certs for authentication. Notice the user that is trying to be used, which was extracted from the TLS cert. See error:

* error creating database object: failed to verify connection: connection() : auth error: round trip error: (UserNotFound) Could not find user "CN=foo,O=foo,L=foo,ST=foo,C=foo" for db "$external"

Expected behavior
The MongoDB secret engine is provisioned successfully.

Environment:

• Vault Server Version (retrieve with vault status): v1.6.2
• Vault CLI Version (retrieve with vault version): v1.6.2
• Server Operating System/Architecture: linux/amd64

Vault server configuration file(s):

disable_mlock = true
ui            = true
log_format    = "json"

listener "tcp" {
  address                            = "[::]:8200"
  cluster_address                    = "[::]:8201"
  tls_cert_file                      = "/certs/server.pem"
  tls_key_file                       = "/certs/server-key.pem"
  tls_require_and_verify_client_cert = true
  tls_client_ca_file                 = "/certs/cacerts.pem"

  telemetry {
    unauthenticated_metrics_access = true
  }
}

telemetry {
  disable_hostname      = true
  enable_hostname_label = true
}

seal "awskms" {}

storage "dynamodb" {
  ha_enabled = "true"
}

service_registration "kubernetes" {}

Additional context
This regression was introduced by #9519, which was a change to allow passing tls=true in the Mongo connection URL properly (#9509). There should probably be some tests around this behavior.

If attempting to connect to a MongoDB instance that requires client TLS, solely with username/password set and no TLS certs, the following error is returned (formatted for readability):

Error writing data to database/config/mongo-example: Error making API request.
URL: PUT https://vault.vault.svc.cluster.local:443/v1/database/config/mongo-example
Code: 400. Errors:
* error creating database object: failed to verify connection: server selection error: server selection timeout, current topology:
{ 
    Type: Unknown,
    Servers: [
        { 
            Addr: mongo-example.svc.cluster.local:27017,
            Type: Unknown,
            State: Connected,
            Average RTT: 0,
            Last error: connection() : connection(mongo-example.svc.cluster.local:27017[-202]) incomplete read of message header: EOF
        },
    ]
}

[pbar1]

Hey - any updates here?

[jeffreyflynt]

I am facing the same issue. Has anyone come up with a workaround?

[jeffreyflynt]

@pbar1 Unsure if you are still facing this issue. But I got around it by setting tlsCertificateKeyFile= & tlsCAFile= in the uri string, instead of tls_certificate_key & tls_ca separately.