Latest Changes to How Snowflake Handles OCSP

Snowflake client drivers (JDBC, ODBC, Python, and others) connect to a variety of endpoints over HTTPS including the Snowflake service, AWS S3, Azure Blob Storage and Okta (when using Okta for authentication). When a Snowflake client driver is served a certificate by any of the endpoints as part of an HTTPS connection, it checks if the HTTPS certificate has been revoked using OCSP (Online Certificate Status Protocol).

You can learn more about why certificate revocation checking is important, and why we use OCSP for it through this earlier blog post. For maximum security, the driver performs the OCSP check for not just the endpoint’s certificate (aka leaf certificate), but also for all the certificates in the chain of trust up to the intermediate certificate issued by the Root CA. The driver fails the connection if either a) it receives a revoked status for any of the certificates in the chain, b) is unable to obtain the OCSP response, or c) receives an invalid OCSP response (such as an expired response or a response with an invalid signature). Behavior characterized by (b) and (c) is referred to as fail-close behavior in this blog.

Snowflake’s availability is dependent on the CA’s ability to be available and produce correct OCSP responses. Normally, we would assume that CAs would support such an expected behavior, but during the past year, we encountered numerous instances when Snowflake client drivers failed to connect to endpoints because either the CA OCSP responder went down, produced expired responses, or allowed the signing certificate to expire without renewing them on-time.

Many of you have built critical applications on top of Snowflake, and your application’s availability is dependent on Snowflake’s availability. To provide more reliable connectivity from Snowflake drivers, we are changing the default behavior of certificate revocation checking from fail-close to fail-open. With fail-open behavior, we will “log and continue” instead of failing the HTTPS connection for cases when the CA isn’t able to provide us with a valid response. We will continue to fail the connection when we receive an OCSP response with a revoked status. There will be a logline with the default log level included with every connection made without an OCSP check. The new (default) fail-open option is in addition to the existing option to turn-off ocsp checking. If you don’t want to assume the risk of fail-open, then you configure fail-close behavior by referring to the FAQ below.

You can monitor the client driver logs to detect when fail-open connections are made. The logline will contain detailed information including the endpoint url, certificate detail, and why the OCSP check is skipped. You can use this information to match with known issue updates posted on the Snowflake status page. If your security team detects unusual activity, you can temporarily shutdown such client access until your security team’s investigation is complete or turn on fail-close behavior according to the process listed in the FAQs section below.

"WARNING!!! Using fail-open to connect. Driver is connecting to an HTTPS endpoint without OCSP based certificate revocation checking as it couldn't obtain a valid OCSP response to use from the CA OCSP responder. Details <JSON including url, certid, ocsp responder url, reason >"

The new behavior change is available in the latest versions of:

• Snowflake SnowSQL
• Snowflake JDBC Driver
• Snowflake ODBC Driver
• Snowflake Connector for Python
• SQL Alchemy (upgrade the underlying Python connector to the latest version)
• Snowflake Spark Connector
• Snowflake NodeJS Driver

Note: Snowflake doesn’t perform OCSP checking for .NET driver which uses the underlying .NET framework for checking validity of the HTTPS certificate.

In addition, Snowflake is working closely with CA providers and cloud platforms (AWS and Azure) to provide a more reliable infrastructure for OCSP checking. We’ll continue to look at improving the robustness of certificate revocation checking while making sure the infrastructure provides the highest level of availability which our customers demand.

1. Upgrade Snowflake client drivers to the latest version.
2. If you can’t upgrade client driver for any reason, then turn-off OCSP checking (such as for partner BI and ETL/ELT applications).
3. Monitor client driver logs.
4. See FAQ below.

What is Snowflake doing to ensure partner applications pickup the latest driver version?

The Snowflake Partner Ecosystem team is working closely with all of our partners to ensure that they pick up the latest driver versions as soon as possible.

I want to preserve the current ocsp_fail_open for all drivers fail-close behavior when upgrading to the latest driver version. What should I do?

Follow the below instructions to set the parameter based on the driver type to preserve the fail-close behavior:

SnowSQL

Pass connection parameter, snowsql -o ocsp_fail_open=false  

(or)

set the ocsp_fail_open=false in the SnowSQL config file

Python

set the connection parameter ocsp_fail_open=false

JDBC

set connection property “ocspFailOpen” to false

(or)

set the system property “net.snowflake.jdbc.ocspFailOpen” to false

ODBC

Set connection parameter “OCSP_FAIL_OPEN” to false

(or)

Set configuration parameter OCSPFailOpen to false in the file pointed to by the environment variable $SIMBAINI

What are some of the best practice recommendations to keep my communications secure?

Here are a few best practices you can use to keep communications secure.

1. Use AWS PrivateLink to keep communication with Snowflake service off the public internet, and turn-off public access to Snowflake. Refer to PrivateLink documentation for further details.
2. Allow client drivers to run only on managed desktops and servers.
3. Send client driver logs to a log management system or upload them into a Snowflake database, and monitor for connections made without ocsp checking.

Certificate revocation checking is an important step for ensuring security of HTTPS connections, and OCSP is the right protocol to do such check. Snowflake will continue to work with the industry (CAs and cloud platforms) to ensure they provide maximum availability of their infrastructure for OCSP. Meanwhile, to minimize disruptions when accessing Snowflake, update Snowflake drivers in your client applications to the latest version starting May 10, 2019, where OCSP checking behavior is changing to fail-open. If you can't accept the risk of fail-open connections, then you can keep the current fail-close behavior (which may cause disruption), by referring to the FAQ above.