Snowpark has generated significant excitement and interest since it was announced. Snowpark is a developer framework that enables data engineers, data scientists, and data developers to code in their language of choice, and execute pipelines, machine learning (ML) workflows, and data applications faster and more securely. While many parts of Snowpark are in preview stages, External Functions entered General Availability earlier this year.
The External Functions feature allows developers to invoke external APIs and remote services from within their Snowflake SQL queries and blend the response into their query result, in the same way as if they were internal Snowflake functions, eliminating the need to export and reimport data. This significantly simplifies building data pipelines that require complex transformations or augmentations that require using custom code or third-party services. Now, we’re pleased to announce Public Preview support for Request/Response Translators—which will further simplify the use of the External Functions when talking to external APIs by eliminating the need for spinning up and maintaining an intermediary data translation layer.
When using external functions, the call to a remote service or third-party API is relayed through one of the supported cloud-native proxy services (for example, an AWS API gateway configured to interface with the external service API). But oftentimes the data sent may need to be translated from the Snowflake data format to the data format expected by the external service, and vice versa. For example, utilizing an external service such as Amazon Comprehend for sentiment detection requires spinning up an intermediary compute layer such as an AWS Lambda to perform the transformation between data formats.
Fig 1. Solution architecture to call a third-party API using External Functions without Translators
However, configuring and maintaining intermediary third-party compute infrastructure for data formatting can create computational and network management overhead. The Translators feature solves for this and simplifies the solution architecture by making it easier to change the format of data sent to, and received from, remote services used by external functions.
The feature allows you to conveniently:
- Convert data from Snowflake’s format to the remote service’s native input format (request translator).
- Convert data from the remote service’s native output format to Snowflake’s format (response translator).
Fig 2. Simplified Solution architecture to call a third-party API using External Functions and Translators
Request/Response Translators are implemented as JavaScript user-defined functions (UDFs) and can be linked to external functions. You almost always write a pair of UDFs: one to translate the request being sent, and one to translate the response. Snowflake calls these functions as part of each external function call. Specifically, Snowflake calls the Request Translator function, passes it the Snowflake-formatted data, then takes the returned data and sends it to the remote service. Similarly, after the remote service returns data, Snowflake calls the Response Translator function to convert the data back to the format that Snowflake understands. From the user perspective, calling an external function with a request/response translator is the same as calling any other external function. Except that now you have fewer pieces of infrastructure and services to maintain, further simplifying your data pipelines! Check out our feature documentation to learn more, and we look forward to hearing your feedback.