Using Your Existing API to Become a Snowflake Data Marketplace Provider, Part 1
Jun 22, 2020 | 5 Min Read
Author: Justin Langseth
Snowflake Data Marketplace
Many data providers who participate in Snowflake Data Marketplace are already using Snowflake Cloud Data Platform as their primary data store, and they can share secure slices of their data via Global Snowflake, Snowflake’s global data sharing feature, with any other Snowflake consumer regardless of which cloud or Snowflake region each is using. But other potential data providers, especially data enrichment companies, are not yet using Snowflake themselves.
Until now, to participate in Snowflake Data Marketplace, these vendors have gotten Snowflake accounts, loaded their data (or mapped it from their data lake using external tables), and then proceeded from there. But for large companies, this can take time and it requires some data engineering work to set up the data flows into Snowflake.
Leverage an Existing REST API to Fulfill Snowflake Data Marketplace Requests
One thing nearly all such data enrichment providers already have, however, is a REST API. Snowflake recently announced our external functions capability, which allows Snowflake accounts to call external APIs. By using external functions, data providers can fulfill requests for data enrichments from Snowflake Data Marketplace consumers by calling their own APIs on behalf of their customers, from a Snowflake account in the consumer’s region, and sharing the resulting data to the customer’s Snowflake account.
Through this approach, data providers can be quickly added to Snowflake Data Marketplace and immediately fulfill consumer requests in any Snowflake cloud or region by using their existing enrichment APIs, eliminating the need for the engineering of new data flows by the provider and eliminating the need for cross-region data replication.
Introducing Quantifind, a Data Enrichment Provider
Quantifind uses machine learning technology to help financial institutions assess risk on entities in bulk. Entity searches are performed through a variety of data sources including sanctions and blacklists, negative news, social media, and real-time search engines. Results are further refined using entity resolution and risk typology models, highlighting high-risk entities that are accurate and relevant, while eliminating any false positives.
On Snowflake Data Marketplace, a Snowflake customer can share parts of their customer or prospect data tables with Quantifind’s Snowflake account. Quantifind detects any new rows of data in the shared data, calls its own API to get the risk information about the new people or companies, retrieves the results, and shares them back to the customer’s Snowflake account.
This makes it very easy for a customer of Quantifind to access Quantifind’s data enrichment service. The customer simply shares their data with Quantifind, and Quantifind shares back the enriched data. While it would be possible for a Snowflake customer to use external functions to call Quantifind’s API directly, the customer would then be responsible for developing, testing, and monitoring the integration themselves. It’s easier for that to be done from Quantifind’s Snowflake account than from each individual customer’s account.
Benefits for Your Customers of Using This Approach
This API-based approach benefits your customers in the following ways:
- It removes costs that each data consumer would otherwise incur to load data into Snowflake through another method, such as FTP or S3. These costs can often be quite significant and can take time to enable.
- It removes costs that each data consumer would incur to develop and operate its own process to call your APIs from their database.
- It provides the ability for each data consumer, from its Snowflake account, to get more data more quickly from the provider, which reduces latency and increases time-to-value.
- It requires no changes for the data consumers if you become more Snowflake-native over time and start to fulfill all or part of the requests directly from Snowflake-stored data and Snowflake-run logic without needing to call your APIs in all cases.
- It provides a quick way for your customers to test-drive new data enrichment services.
- Data consumers have the ability to access data and data services within Snowflake regardless of their Snowflake region.
Part 2 of this blog post provides a step-by-step example that demonstrates how Quantifind uses this API-based approach.