5 Reasons to Host Your Own Data Exchange
Dic 03, 2019 | 4 Min Read
Author: Justin Langseth
Modern Data Sharing, Snowflake Technology
Our customers have been putting a lot of data into Snowflake, and many have been using Snowflake’s data sharing capabilities to share data without copying it between companies (for example, a SaaS vendor sharing data with its customers). We launched the Snowflake Data Exchange at our Summit conference in June 2019 to help our customers find and exchange data with other Snowflake customers and with external data providers.
As we briefed our customers on the Snowflake Data Exchange, we heard one consistent point of feedback: They wanted their own data exchange. They told us that finding and sharing data within their own organizations was sometimes even harder than sourcing external data. And in addition to internal data cataloging, they told us that having a walled garden where they could securely list and share data between themselves and their customers, partners, and suppliers was an urgent priority.
We listened to this feedback, and we are excited about the availability of Snowflake Data Exchange™. A data data exchange lets a Snowflake customer host and operate its own data exchange.
Customers like Coatue, a technology sector hedge fund, is planning to use the Data Exchange as a tool to explore new data assets in a “data store” where they are the only consumer.
“At Coatue, we understand the role data plays in smart investing and we aim to be at the leading edge of data management,” Coatue Co-Founder and Senior Managing Director, Thomas Laffont said. “The Snowflake Data Exchange blazes a trail for how data will be organized, discovered, augmented, and accessed in the future. It eliminates traditional data boundaries and enables real time, frictionless data sharing that will yield key benefits for our ongoing data investments.”
A data exchange can be under your company’s brand, and you control who can get in. It could just be for internal use, or you could also open it to customers, partners, and suppliers. You can control what data assets are listed and who has access to which sets of data. These capabilities allow for a seamless way to discover and share data both within your organization and with your business partners.
Here are five reasons you may want to host and operate your own Snowflake-powered data exchange:
#1: Facilitating Internal Data Discovery
Midsize and large companies have been looking for an easy way to have an internal data catalog for a long time. From enterprise search and portals to master data management and data cataloging/crawling, companies have tried it all. Yet a simple holistic solution to control and grant access to data, that is easy to deploy and maintain and actually gets widespread adoption, remains elusive.
Most of the existing solutions have attacked the problem from the bottom up, either by trying to crawl and index databases and other content sources or by having data architects explicitly wire everything up. However, indexing and crawling systems tend to gather a lot of out-of-date data or data of questionable accuracy, and they run into access control concerns. And centralized data control systems require too much work and sustained attention from data administrators.
We believe a crowdsourced, internal “marketplace” for data is a better solution. By deploying an internal data exchange, any member of the internal community can list data assets it finds valuable and can vouch for–and plans to keep current. And companies can still coordinate access controls, as well as control who can immediately see versus request access to certain data or certain types of data.
This approach combines the benefits of crowdsourcing content while ensuring data quality and providing the right level of centralized control and coordination, all of which can overcome the challenges that have slowed the adoption of other approaches to enterprise data cataloging. In addition, this approach allows users across an enterprise to contribute data, use data from other groups, and join data together to create enriched data products, for both internal use as well as potentially for external monetization.
#2: Streamlining Physical and Digital Data Supply Chains
Once you have a data exchange for use by your internal departments and divisions, you can consider opening the exchange up to your physical and digital supply chain suppliers. For example, you could share data with suppliers about your inventory levels or about consumption of things they supply, so they can better meet your needs. Or, digital data providers could provide data directly into your data exchange to make it immediately usable and joinable to your internal enterprise data, reducing the costs for you and your providers to transmit, store, and load the data.
Some companies such as hedge funds and marketing agencies bring in data from many external sources. Some hedge funds evaluate over 1,000 potential data sets per year. Interestingly, a data exchange can be used to not only connect with data that you have already purchased but also to evaluate new data assets. Therefore, a hedge fund could have potential data suppliers list their data on a data exchange, and the fund could explore and “shop” for data in a data store where it is the only customer. Such an internal data store could also “tunnel” in data assets from the public Snowflake Data Exchange.
Or, an existing supplier of marketing data could list some additional data sets that one of its customers could use via a exchange on a trial basis, and if the customer finds the data sets useful, the supplier can immediately provide full access through the same exchange. These arrangements can bring much greater depth of data, bidirectional and much fresher data, and greater trust and transparency to relationships between suppliers of data and physical goods and their customers.
#3: Creating a Data Community with your Partner and Customer Ecosystem
Almost every company has customers and other companies it partners with. And almost every company wants to make those relationships deeper, more efficient, and more profitable. A great way to improve the performance of these relationships is to enable the secure interchange of data in a bidirectional manner.
Generally this is challenging to accomplish, and companies share only very limited sets of data via APIs, FTP, dashboards, or files. Also, this type of data sharing often comes at great cost,, data latency, and some security risk.
A company with many customers and partners could instead host a data exchange, and then invite its customers and partners to participate in the exchange. The customers and partners could consume secure slices of data from the host company and also push data in the other direction. Sharing data in the other direction could involve sharing data back to the host, but it also could involve listing data so other participants in the ecosystem can securely access it as well. Data from the public Snowflake Data Exchange, or from other external sources, could also be included to complete the data offerings.
Every large company depends on other companies and on its customers. Bidirectionally sharing data not only from the company to and from these parties, but also between these external parties themselves, can allow rich, collaborative data ecosystems to develop where groups of companies can work together around data. Participants can securely discover, combine, and enrich data assets to help service a common, large customer or to form new partnerships among themselves. Some of these relationships may even lead to opportunities to sell secure views of data or functions across data to other participants of a walled garden ecosystem.
#4: Monetizing your Data
All this leads to data monetization. Raw data assets are generally too unpolished and full of potentially sensitive data to just outright sell them to other companies. Generally, the owner of the data needs to perform data cleaning, deidentification, aggregation, joining, and other forms of data enrichment before the data is salable to another party.
A data exchange can be a great place to discover, assemble, clean, and enrich data to make it more monetizable. A large company on a data exchange could assemble data from across its divisions and departments that could become valuable to another company when it is properly cleaned and processed. Or, participants in a data exchange could work together to join their data sets to create a useful data product that any one of them alone would not be able to produce.
For example, once data assets are “cooked” by participants of a data exchange, they could be listed on public data marketplaces such as the Snowflake Data Marketplace.
#5: Connecting your Whole Industry
If a data exchange really takes off, it could become so big and influential that it becomes the standard place for a whole industry to interchange, collaborate around, and monetize data. There is probably room for one or two “mega ecosystem data exchanges” in each industry. Once one exchange gains significant traction, it could become the “go to” place for that industry to interchange and monetize data. If more than one viable exchange emerges in an industry, the respective hosts of these exchanges could decide to partner and “cross-tunnel” some (but maybe not all) assets between their exchanges to achieve critical mass. Or, they could decide to remain head-to-head data ecosystem competitors.
Although it is possible that industry coalitions could host such data exchanges, it is even more likely that one or two large players in each industry will bootstrap data exchanges quickly and broadly enough to become the de facto data exchange for their industry. This provides a significant incentive for companies that want to become major players and first-movers in the data side of their industries to start as soon as they can to build their internal data exchanges and then open them up to their suppliers, customers, and partners.
How to Get Started with a Data Exchange
Even if your current data cataloging and exchanging aspirations are less than industry-wide data domination, you should get started with a data exchange. A good starting point is with your internal data assets. And you can start small. Assemble some useful assets, turn on a data exchange powered by Snowflake, open it up to your company, and start sharing. And when the time is right, open the exchange to your customers, partners, and suppliers, and see how far that takes you. For more information on hosting your own Snowflake data exchange, click here.