New Snowflake features released in Q2’17
Aug 16, 2017
Author: Artin Avanes
Engineering, How to Use Snowflake
It has been an incredible few months at Snowflake. Along with the introduction of self-service and numerous other features added in the last quarter, we have witnessed:
- Our customer base has grown exponentially with large numbers of applications in full production.
- Billions of analytical jobs successfully executed this year alone, with petabytes of data stored in Snowflake today, and without a single failed deployment to-date.
- A strong interest in pushing the boundaries for data warehousing even further by allowing everyone in organizations to share, access and analyze data.
Continuing to engage closely with our customers during this rapid growth period, we rolled out key new product capabilities throughout the second quarter.
Instantly sharing data without limits – Introducing the Data Sharehouse
Giving users the ability to easily and securely share data in the right format without the need for cumbersome data integration pipelines.
One of our highlights was the general availability of Snowflake’s Data Sharing feature. It allows multiple Snowflake customers to instantly share data with each other.
With data sharing, there is no need to use the expensive, insecure, and often complicated and error-prone procedure of transferring large numbers of single files from one location to another. Instead, customers can now simply share data via SQL statements and secure views fully integrated with our role-based access control model.
To learn more about how Snowflake customers are already leveraging this capability to build new products and features to drive their business, we encourage you to read:
- How PlayFab is offering instant access to the analytics of gaming data
- How Localytics removes one of the most challenging tasks marketers face in the mobile analytics space
Improving out-of-the box performance & SQL programmability
Our ongoing mission is to build the fastest database for data warehousing with the SQL you love and no concurrency limits.
- We continued to make end-to-end query execution faster with more efficient pruning for sub-columns in VARIANT types (JSON), general JOIN performance improvements, and faster global median calculations.
- We addressed popular customer requests for improved programmability via SQL by:
- Customers who are using our multi-cluster warehouses auto-scale feature for spiking workloads can now specify up to 10 compute clusters. This allows running hundreds of queries without any query queuing.
Staying ahead with enterprise-ready security and compliance
From day one, security has always been core to Snowflake’s design.
- One of our exciting new capabilities this quarter is the general availability of customer-managed keys which added an additional layer of security for customers with sensitive data. This feature is the primary component of Tri-Secret Secure, a Snowflake Enterprise Edition for Sensitive Data (ESD) feature. You can find more details in our engineering blog about customer-managed keys in Snowflake.
- We also improved the ability for our users to monitor and filter the query history Information Schema table function for more specific SQL command types.
- After its preview in Q1, secure views reached general availability in Q2.
- In terms of certification, Snowflake received PCI DSS compliance – a common requirement for customers in banking, financial services, retail, services, and more. Customers and prospects who have PCI requirements will need to subscribe to the Snowflake Enterprise Edition for Sensitive Data (ESD).
Improving our ecosystem and data loading
Enabling developers and builders to create applications with their favorite tools, drivers, and languages remains a top priority.
- For enterprise-class ETL, data integration and replication:
- Our Talend connector reached general availability status with Snowflake’s component being included in Talend Open Studio 6.4 version.
- We worked with Informatica to build a new V2 connector for Informatica’s cloud product based on their new connector SDK.
- We collaborated with Matillion to build a Snowflake-specific version of their product that entirely pushes compute into Snowflake (including complex transformations).
- Stitch released a generally available Snowflake connector.
- Attunity released a preview Snowflake connector.
- For business intelligence (BI):
- Snowflake’s connector for PowerBI reached general availability. The PowerBI team also added support for the PowerBI Service via the gateway node w/ DirectQuery mode enabled.
- We worked with the AWS QuickSight team to add a Snowflake native connector to their product.
- For data warehouse automation, Wherescape added support for Snowflake.
- We enhanced our parallel data loading & unloading via the COPY command; developers can now:
- Expanding our driver support, we announced a preview version of our open-source Go driver, available in Snowflake’s Github repo.
Increasing transparency and usability
These features are designed to strike the right balance between offering a service that is easy to operate and exposing actionable insights into the service itself.
- Snowflake users can now set up MFA via the UI; they no longer need to reach out to our support team to enable the feature.
- Building on the general availability of Query Profile in Q1, we added a number of additional usability enhancements that can be leveraged to better understand where time is spent during query execution.
- The AWS Key ID is now displayed in DESC STAGE output and interface.
- We added support for leveraging file extensions used for data unloading operations, and changed the default behavior when loading files containing byte order marks, i.e. we now detect and skip the marks instead of throwing an error because the data could not be converted to proper data types.
- To allow Snowflake users to better control consumption of compute resources, we also enhanced resource monitors (currently in preview). Users can now explicitly assign them to virtual warehouses and specify certain actions if a credit threshold is met or exceeded. Please stay tuned for a separate blog on this important capability.
Scaling and investing in service robustness
These service enhancements aren’t customer visible, but are crucial for scaling to meet the demands of our rapidly growing base of customers.
- Given our rapid growth since the beginning of this year, we continued working on product scale, reliability, and availability to ensure our readiness for the next phase of hyper-growth.
- We’re already seeing our efforts to automate all operations, particularly Snowflake deployments, pay off. We were able to quickly roll out a new region (US East – Virginia in Q2) while working on additional deployments around the globe at the same time.
Acknowledgements and conclusion
As always, we want to first thank our customers for their continuous feedback. Additionally, we want to recognize that moving rapidly while also scaling would not be possible without our mighty Engineering Team, which has proven once again that it’s possible to ship high-quality features while serving our existing users’ needs at the same time.
Now, onwards to Q3 and the next set of exciting capabilities you will hear about very soon!
For more information, please feel free to reach out to us at firstname.lastname@example.org. We would love to help you on your journey to the cloud. And keep an eye on this blog or follow us on Twitter (@snowflakedb) to keep up with all the news and happenings here at Snowflake Computing.