Through this quickstart, you will learn how to get started with Snowpark Connect for Apache Spark™. Using Snowpark Connect for Apache Spark, you can run Spark workloads directly on Snowflake.

By the end of this quickstart, you will learn how to:

Snowpark is the set of libraries and code execution environments that run Python and other programming languages next to your data in Snowflake. Snowpark can be used to build data pipelines, ML models, apps, and other data processing tasks.

With Snowpark Connect for Apache Spark, you can connect your existing Spark workloads directly to Snowflake and run them on the Snowflake compute engine. Snowpark Connect for Spark supports using the Spark DataFrame API on Snowflake. All workloads run on Snowflake warehouse. As a result, you can run your PySpark dataframe code with all the benefits of the Snowflake engine.

In Apache Spark™ version 3.4, the Apache Spark community introduced Spark Connect. Its decoupled client-server architecture separates the user's code from the Spark cluster where the work is done. This new architecture makes it possible for Snowflake to power Spark jobs.

During this step you will set up a Snowflake Notebook in Workspaces and upload the PySpark notebook.

Sign up for a Snowflake Free Trial account and login to Snowflake.

Download the getting_started_with_snowpark_connect_for_apache_spark.ipynb from this GitHub repository. On the GitHub page, click the Download raw file button (download icon in the top right) and save the .ipynb file to your computer.

Snowflake Notebooks in Workspaces provide a Jupyter-compatible environment with managed compute and direct access to Snowflake data.

Follow along and run each of the cells in the notebook. Here is what each section covers:

The first cells initialize the Snowpark Connect server and verify the connection. The notebook uses get_active_session() to connect to Snowflake and then creates a Spark session via snowpark_connect.server.init_spark_session().

Next, you will generate synthetic support case data using PySpark. This demonstrates creating DataFrames with complex schemas, data generation patterns, and writing DataFrames to Snowflake tables using saveAsTable.

You will create a Spark UDF to categorize support cases based on their description, apply it to the data, and write the transformed results back to Snowflake. This shows how custom Python logic can be used within PySpark on Snowflake.

Finally, you will create a Snowflake Python function and invoke it using SQL passthrough from Spark with spark.sql(). This demonstrates how to combine native Snowflake functions with PySpark workflows.

Congratulations! You have successfully completed this quickstart and learned how to run PySpark workloads directly on Snowflake using Snowpark Connect.