Hello streaming enthusiasts!
I want to give a little teaser of what we’re working on in the realm of stream processing at Snowflake, because we need some help: YOUR help!
At the beginning of 2020, I joined Snowflake because I saw an opportunity to create the next generation of stream processing systems. I’ve opined for a long time that the true endgame for streaming is the seamless blending of batch and streaming together into a single, symbiotic system that transcends the capabilities of both. My goal at Snowflake is to finally make that dream a reality.
To be clear, Snowflake also happens to be an amazing company that’s firing on all cylinders across a number of dimensions: technical architecture, people, leadership, financials, culture, and work-life-balance. But none of those are why I signed up; they were just table stakes. The reason I’m here is the opportunity to further push forward the state of the art in stream processing.
Think of all of the shortcomings in stream processing systems today1: the limitations of their NoSQL heritage; the contortions customers go through to fit their use cases into the restrictive computational models; the pain of running these complex systems at scale, and the even greater pain of evolving them over time; and the difficult and manual nature of tuning a system to fit the right latency/cost/completeness profile.
We have big ideas for how to fix some of those things. But we need your ideas and your help to figure out how to fix the rest of them and then build it all out. It’s been a blast so far, and we’re just getting started.
Interested parties should apply here. I hope to see you join our team soon!
(1Or alternatively, browse through my Open Problems in Stream Processing keynote slides from DEBS 2019, which are summarized here.)