Breaking Free from Complexity Saves Time and Money
Author: Michael Nixon
How to Use Snowflake, Snowflake Technology
In traditional data warehousing, complexity comes in various forms, including time and labor-intensive tasks like creating clusters. Procuring a platform then managing and scaling that platform also takes time and labor, as does managing all of the software components interoperating with the environment.
Mitigating complexity (therefore saving time) is just one of the reasons businesses are turning to the cloud. In fact, we surveyed attendees during a webinar (7 Tips for a Modern Cloud Data Warehouse) we hosted with our data integration partner, Talend, and the results were not surprising. Nearly 40% of attendees indicated that eliminating manual effort and complexity were their top reasons for moving analytic workloads to the cloud.
For on-premises data warehouses, it can take days or weeks to procure hardware, plus weeks or months to get the hardware up and running into production.
For cloud-washed platforms (platforms ported to the cloud), it can take six to eight minutes per node to create and spin up clusters. Factor in more time to secure or validate software licenses and, in some cases, to create a cluster management console (which adds separate costs) and yet more time is added to the process.
As the demand placed on these systems increases, even more time delays occur when you need to scale the environment.
Data Warehouse Technology is Advancing
Snowflake’s data warehouse as a service approach eliminates the costly and time-consuming task of building and managing clusters.
Table 1 highlights the tasks Snowflake handles for you versus the manual efforts typically required with ordinary data warehouses.
Snowflake service is engineered with built-in cloud infrastructure and database management capabilities. Combined, these features enable the service to behave autonomously and with zero management. Because the Snowflake software drives all database operations (including scaling a data warehouse), database administration is not required.
Set it Once and Go Equals Zero-management
Snowflake eliminates complexity and dramatically simplifies management. There are two options for creating a data warehouse in Snowflake and enabling it to auto-scale. Option one, shown below, requires using the Snowflake UI to select the size of the warehouse, name the warehouse, set the number of required clusters to scale, set a scaling performance policy and enable auto-resume.
Option two for setting up a warehouse in Snowflake requires executing the following SQL statement using the Snowflake UI worksheet or a similar notebook tool.
CREATE WAREHOUSE executive_dashboards WITH WAREHOUSE_SIZE = 'MEDIUM' WAREHOUSE_TYPE = 'STANDARD' AUTO_SUSPEND = 300 AUTO_RESUME = TRUE MIN_CLUSTER_COUNT = 1 MAX_CLUSTER_COUNT = 5 SCALING_POLICY = 'STANDARD';
Using the UI will take about 30 seconds to complete, not counting the automatic system warehouse provisioning that follows. This can take an additional few seconds or up to a couple of minutes. If you cut and paste the SQL script above, it takes a matter of seconds to complete the process.
Either option will establish a medium-sized virtual warehouse, with one active cluster set as the default state and up to five active clusters at the ready when you need extra compute capacity. Compute resources will automatically suspend after five minutes of idle activity. This saves cost without you having to terminate the warehouse and unload data.
Snowflake eliminates complexity in other ways. For example, you don’t need Hive, MapReduce, HDFS or Java to work with and query semi-structured data like JSON. You don’t need multiple platforms to join semi-structured data with structured data. And, with live, secure data sharing (no data movement required) you save time, effort and the costs associated with manually duplicating and transferring data. Saving time saves you money.
Make managing clusters a thing of the past
Once set, the Snowflake service takes over and your team is free from manually scaling or managing a data warehouse. Eliminating remedial tasks allows you to focus on higher value activities such as developing and engineering new products.