Snowpark and Streamlit
Overview
While Snowflake started as a SQL-based data warehouse, it has evolved into a full programmable data cloud. Snowpark and Streamlit in Snowflake are the two primary frameworks that allow developers and data scientists to write code in non-SQL languages (Python, Java, Scala) and build applications directly within the Snowflake environment.
Key Concepts: Snowpark
Snowpark is a developer framework that brings deep integration of Python, Java, and Scala into Snowflake.
The Snowpark API (DataFrames)
- Instead of writing SQL strings, developers use Snowpark's DataFrame API (which is very similar to PySpark or Pandas).
- Under the hood, the Snowpark API translates these DataFrame operations into optimized Snowflake SQL.
- This SQL is then pushed down to the Snowflake compute layer for execution, meaning the data never leaves Snowflake.
Serverless Execution (UDFs and Stored Procedures)
Snowpark allows you to upload custom Python, Java, or Scala code and run it directly inside Snowflake's secure boundary.
- User-Defined Functions (UDFs): Used for scalar or tabular transformations (e.g., running a Python regex function on every row).
- Stored Procedures: Used for orchestration and administrative tasks.
Snowpark-Optimized Warehouses
Standard virtual warehouses are optimized for SQL. For heavy machine learning training or complex Python data processing that requires massive amounts of memory, you can use a Snowpark-optimized warehouse. They provide up to 16x the memory per node compared to standard warehouses and consume 1.5x more credits per hour.
Key Concepts: Streamlit in Snowflake
Streamlit is an open-source Python library that makes it easy to build custom web apps for machine learning and data science. Snowflake acquired Streamlit and integrated it natively.
- Native App Development: You can write a Streamlit app entirely in Python and host it directly inside Snowflake.
- No Infrastructure Management: You do not need to manage web servers, Docker containers, or front-end hosting. Snowflake handles the compute and hosting of the app.
- Secure: Because the app runs inside Snowflake, it respects existing Role-Based Access Control (RBAC). Users can only interact with the data they are permitted to see.
Architecture Visualized
Developer Machine (Python/VS Code)
│
▼ (Snowpark DataFrame API translates to SQL)
│
Snowflake Cloud Services Layer (Parses & Optimizes)
│
▼ (Executes Pushdown SQL & Python UDFs)
│
Snowflake Compute Layer (Virtual Warehouses)
Python Example (Conceptual)
# A simple Snowpark Python example showing DataFrame usage
from snowflake.snowpark import Session
# Connect to Snowflake
session = Session.builder.configs(connection_parameters).create()
# Read a table into a Snowpark DataFrame
df = session.table("sales_data")
# Filter and aggregate using Python syntax (executed as SQL in Snowflake)
summary_df = df.filter(df["region"] == "US").group_by("category").count()
# Show results
summary_df.show()
Exam Tips
⚠️ Crucial for COF-C03:
- Pushdown: Understand that Snowpark *pushes down* processing to Snowflake. It does not pull data out of Snowflake to process on your local machine.
- Languages: Snowpark supports Python, Java, and Scala.
- Streamlit: Know that Streamlit is used for building interactive data applications natively in Snowflake using Python.
- Warehouses: Snowpark-optimized warehouses are used specifically for high-memory requirements, not just general SQL queries.
Key Takeaways
- Snowpark provides a DataFrame API to write non-SQL code that translates to Snowflake SQL.
- It allows running custom Python/Java/Scala code inside Snowflake via UDFs and Stored Procedures.
- Streamlit allows for the creation of secure, hosted web apps directly within Snowflake.