As traditional finance experts come to the world of cryptocurrency, they need better platforms to get insight into the blockchain ecosystem. Token Flow Insights helps them get the data and analytics they need, with a little help from the Snowflake platform.

Previously the reserve of technology enthusiasts, cryptocurrencies and blockchains, and in particular Decentralized Finance, are now attracting traditional investors looking to diversify their portfolios. Yet, even experienced market analysts and risk managers find it hard to interpret and analyze the data surrounding blockchains.

That’s where Token Flow comes in. The company  sources and contextualizes  full historical and live data from several blockchains to offer financial and business users the high-quality insights and analytics they’re used to across conventional asset classes.

Collecting and processing all this information across multiple blockchain protocols means Token Flow has to handle huge volumes of data. So, how does a startup with limited resources maintain the complex data architecture needed to drive the future of blockchain insight and analytics?

Bringing blockchain insights to a new audience demands powerful infrastructure

“Our goal is to bridge the gap for traditional finance people,” explained Tomek Mierzwa, Chief Data Wizard and Co-Founder of Token Flow. “We collect data from blockchains, but add in the context and semantics to make it understandable and useful to business users.”

“There are a number of challenges that come with that,” Mierzwa continued. “We have to process high volumes of data and usage patterns are very unpredictable. We needed a high performance architecture that could provide data to lots of users.”

Mierzwa’s prior experience meant he’d tried many different data ecosystems: “PostgreSQL is a great technology and something we’d started to use, but it can require a lot of database administration. And that demands skills and resources we didn’t have to hand as a newly-founded company.”

A straightforward data platform with minimal maintenance requirements

“In the earliest days of the company, we’d experimented with a simple setup in Snowflake’s platform,” explained Mierzwa. “So when it came time to establish our ecosystem, Snowflake was the obvious choice.

Mierzwa and the Token Flow team appreciated the platform’s high performance, elastic scalability and low maintenance requirements. “We’re a data company, not an infrastructure company,” he said. “The great thing about Snowflake is they handle all the infrastructure maintenance for us.”

Another aspect that helped Token Flow focus on its core business was a simple onboarding process with Snowflake: “We were starting from zero with Snowflake,” explained Mierzwa. “But as it externally behaves like a SQL database (including standard queries), our engineers could lean on their experiences with other technologies to get up to speed quickly.”

And by taking part in Snowflake’s Startup program, Token Flow had access to additional support. “Snowflake offered us resources that really helped us get started and avoid some of the usual setup challenges startups experience,” he said . “We also were offered a lot of support by the Snowflake team. Though we didn’t have to rely on it much, because the platform was so easy to use.”

“When it came time to establish our ecosystem, Snowflake was the obvious choice.”

Data processing pipelines based on Snowflake

With a versatile data platform at the ready, Token Flow can collect and process data at different levels from multiple blockchain protocols to offer a broad suite of data to customers.

Using a custom extraction layer built in a mix of Python, Javascript, and Go, Token Flow pulls data from its own blockchain nodes and stores it in its data platform built on Snowflake. From there, it uses a data pipeline based on Airflow ETL to move data between layers where it’s further processed to make it useful to different groups of users.

Token Flow’s architecture splits data into three layers, one layer for raw data, one for decoded data with added semantics and context, and a final layer that combines data to track transactions across multiple blockchains. “This is the most important thing for our users,” explained Mierzwa. “Because the blockchain space moves so quickly, it’s not enough to analyze a single chain—you need to connect the dots across multiple protocols and technologies.”

Once it has extracted, processed, and stored its data layers, Token Flow also shares its data with customers through the Snowflake Marketplace. “Snowflake makes sharing really simple,” said Mierzwa. “We get great visibility through Snowflake’s marketplace, and our customers keep telling us how easy it is to set up an account and start accessing our data.”

Less cost and more peace of mind with Snowflake

“The cost savings in Snowflake were an important thing for us,” said Mierzwa. “The low maintenance requirements compared to other technologies save us money. And we don’t need to hire infrastructure engineers to manage the database—on previous projects I had as many as four full time employees handling this, so saving on that headcount cost is significant.”

While the immediate cost savings were a big draw for Token Flow, there’s one benefit you can’t put a price on: peace of mind. “It’s something you get used to all too quickly, but we just never have to worry about database performance and infrastructure with Snowflake,” he explained.

The future is bright for Token Flow—and for blockchain insights

Since releasing a beta version of its Ethereum and StarkNet datasets through the Snowflake Marketplace, Token Flow has attracted several users in a short timespan. Mierzwa is happy that the beta products are off to a strong start: “In the first two weeks, we’ve seen around 35 companies access our data, which is higher than expected. As we collect feedback, we can now improve the product and start considering how to monetize with confidence.”

“Because we use Snowflake, it’s a good point for potential investors; they believe in the technology behind our business.”

Token Flow also has its sights on horizontal expansion, as it seeks to bring its analysis to more chains and protocols, and use additional features from Snowflake’s platform to achieve this. “We’ll be loading and sharing more data with Snowflake,” said Mierzwa. “And we’re also exploring how we can use features like Snowpark to process more data within Snowflake and make our pipeline even more efficient. Whatever we decide to do, we know Snowflake can scale with us as we grow.”

And in the meantime, the Token Flow team is in exciting discussions with new customers and potential investors alike. “It’s interesting that when we tell potential investors we’re on the Snowflake Data Cloud, the response is always positive,” Mierzwa elaborated. “Because we use Snowflake, it’s a good point for potential investors; they believe in the technology behind our business.”