Smoke tests for Databases fail with SSL error

[ycombinator]

While trying to cut a new release, I ran all smoke tests. The Cloud Databases smoke tests immediately failed:

$ php tests/OpenCloud/Smoke/Runner.php --include=Database
Welcome to the PHP OpenCloud SmokeTest!

Authenticate
   Using identity endpoint: https://identity.api.rackspacecloud.com/v2.0/
   Using region: IAD
   Token generated: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Executing OpenCloud\Smoke\Unit\Database

   1. Get Flavors
PHP Fatal error:  Uncaught exception 'Guzzle\Http\Exception\CurlException' with message '[curl] 35: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure [url] https://iad.databases.api.rackspacecloud.com/v1.0/XXXXX/flavors/detail?limit=100' in /root/php-opencloud/vendor/guzzle/guzzle/src/Guzzle/Http/Curl/CurlMulti.php:359

[ycombinator]

curling the Cloud Databases endpoint from the command line also produces a similar error:

$ curl -v https://iad.databases.api.rackspacecloud.com
* Rebuilt URL to: https://iad.databases.api.rackspacecloud.com/
* Hostname was NOT found in DNS cache
*   Trying 207.97.219.112...
* Connected to iad.databases.api.rackspacecloud.com (207.97.219.112) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS alert, Server hello (2):
* error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
* Closing connection 0
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Note, however, that specifying the TLSv1 cipher suite makes the error go away:

 curl -v --cipher TLSv1 https://iad.databases.api.rackspacecloud.com
* Rebuilt URL to: https://iad.databases.api.rackspacecloud.com/
* Hostname was NOT found in DNS cache
*   Trying 207.97.219.112...
* Connected to iad.databases.api.rackspacecloud.com (207.97.219.112) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / RC4-SHA
* Server certificate:
*        subject: OU=Go to https://www.thawte.com/repository/index.html; OU=Thawte SSL123 certificate; OU=Domain Validated; CN=iad.databases.api.rackspacecloud.com
*        start date: 2013-06-04 00:00:00 GMT
*        expire date: 2015-06-04 23:59:59 GMT
*        subjectAltName: iad.databases.api.rackspacecloud.com matched
*        issuer: C=US; O=Thawte, Inc.; OU=Domain Validated SSL; CN=Thawte DV SSL CA
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: iad.databases.api.rackspacecloud.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Via: 1.1 Repose (Repose/2.12)
< Content-Length: 175
< Date: Fri, 13 Mar 2015 22:05:51 GMT
< Connection: close
* Server Jetty(8.0.y.z-SNAPSHOT) is not blacklisted
< Server: Jetty(8.0.y.z-SNAPSHOT)
<
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
{"versions": [{"status": "CURRENT", "updated": "2012-08-01T00:00:00Z", "id": "v1.0", "links": [{"href": "http://iad.databases.api.rackspacecloud.com/v1.0/", "rel": "self"}]}]}

Also note that this is NOT an issue with other service endpoints:

• Cloud Servers:

$ curl -v https://iad.servers.api.rackspacecloud.com/
* Hostname was NOT found in DNS cache
*   Trying 162.209.116.128...
* Connected to iad.servers.api.rackspacecloud.com (162.209.116.128) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / AES256-SHA
* Server certificate:
*        subject: OU=Go to https://www.thawte.com/repository/index.html; OU=Thawte SSL123 certificate; OU=Domain Validated; CN=iad.servers.api.rackspacecloud.com
*        start date: 2013-07-08 00:00:00 GMT
*        expire date: 2015-07-08 23:59:59 GMT
*        subjectAltName: iad.servers.api.rackspacecloud.com matched
*        issuer: C=US; O=Thawte, Inc.; OU=Domain Validated SSL; CN=Thawte DV SSL CA
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: iad.servers.api.rackspacecloud.com
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Date: Fri, 13 Mar 2015 22:04:43 GMT
< Via: 1.1 Repose (Repose/6.2.1.2)
< WWW-Authenticate: Keystone uri=https://identity.api.rackspacecloud.com/v2.0
< Content-Length: 0
* Server Jetty(9.2.z-SNAPSHOT) is not blacklisted
< Server: Jetty(9.2.z-SNAPSHOT)
<
* Connection #0 to host iad.servers.api.rackspacecloud.com left intact

• Cloud Queues:

$ curl -v https://iad.queues.api.rackspacecloud.com/
* Hostname was NOT found in DNS cache
*   Trying 162.209.114.130...
* Connected to iad.queues.api.rackspacecloud.com (162.209.114.130) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / DHE-RSA-AES128-SHA
* Server certificate:
*        subject: C=US; ST=Texas; L=San Antonio; O=Rackspace US, Inc.; OU=Product; CN=*.queues.api.rackspacecloud.com
*        start date: 2014-04-14 00:00:00 GMT
*        expire date: 2016-04-13 23:59:59 GMT
*        subjectAltName: iad.queues.api.rackspacecloud.com matched
*        issuer: C=US; O=Thawte, Inc.; CN=Thawte SSL CA
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: iad.queues.api.rackspacecloud.com
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Keystone uri='https://identity.api.rackspacecloud.com/v2.0/tokens'
< Content-type: text/plain
< Transfer-Encoding: chunked
< Connection: Keep-Alive
<
* Connection #0 to host iad.queues.api.rackspacecloud.com left intact

[ycombinator]

BTW, versions of curl and openssl on the machine where I ran the smoke tests from are:

$ curl --version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

That version of OpenSSL is bad (heartbleed), so I'm now trying to upgrade it and try the smoke tests again.

[ycombinator]

Even after upgrading OpenSSL to 1.0.2, I still get the same curl error:

$ openssl version
OpenSSL 1.0.2 22 Jan 2015

$ curl https://iad.databases.api.rackspacecloud.com
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

[ycombinator]

BTW, I'm on Ubuntu 14.04:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"

[ycombinator]

Paging @dstufft. Could you please help us figure out what might be going on here and how best we should fix it? Thanks.

[dstufft]

Almost certainly this is caused because RC4 is insecure and broken and nobody should rely on it. I'm pretty sure that Linux distros are shipping updates which default to no longer using RC4. Then the iad.databases.api.rackspacecloud.com endpoint comes along and only supports RC4.

[dstufft]

Cipher String:

ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:ECDH+RC4:DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5

This Cipher list has been carefully constructed to result in a specific set of ciphers in a specific order. This list has been constructed to exclude any unsafe ciphers with the exception of RC4, which is currently weak but still required for compatibility with many services.

The string translates to a cipher suite list which is roughly:

• AES GCM using the ECDHE Key Exchange, then the DHE Key Exchange.
• AES 256 using the ECDHE Key Exchange, then the DHE Eey Exchange.
• Any AES using the ECDHE Key Exhcange, then the DHE Key Exchange.
• Any "HIGH" security cipher using the ECDHE Key Exchange, then the DHE Key
  Exchange.
• 3DES using the EXDHE Key Exhcnage, then the DHE Key Exchange.
• Any AES GCM using the RSA Key Exchange (No PFS).
• Any AES using the RSA Key Exchange (No PFS).
• Any "HIGH" cipher using the RSA Key Exchange (No PFS).
• Any 3DES cipher using the RSA Key Exchange (NO PFS).
• Any RC4 cipher preferring those with ECDHE, then DHE, then RSA Key exchanges.

Finally it explicitly disallows:

• Any cipher that does not include authentication.
• Any cipher that does not include encryption.
• Any cipher that uses a MAC based on MD5.

[ycombinator]

We will be using the cipher suite mentioned in the previous comment until the Rackspace Cloud Servers service stops supporting RC4 and starts supporting stronger ciphers instead.