Expedition. Free, on-demand.

Technical tracks for practitioners, outcomes for leaders.

Snowflake for Developers/Guides/Build a Video Search App with TwelveLabs on Snowflake
Quickstart

Build a Video Search App with TwelveLabs on Snowflake

Cortex LLM
Akhil Ramasagaram

Overview

In this guide you'll turn a folder of video files into an app that lets you:

  • Search every video by typing a description or dropping in an image, across what's seen, heard, and said, and jump straight to the matching moment.
  • Analyze any video with a plain-language prompt and get structured answers back with Pegasus 1.5.

The whole pipeline runs inside Snowflake. No video leaves your account, and every model call is a SQL function.

Architecture

What You'll Build

StepWhat it doesSnowflake surface
1. StagePoint Snowflake at your videosStage + directory table
2. EmbedTurn each video into time-stamped vectorsAI_MULTI_EMBED with Marengo 3.0
3. IndexMake visual, audio, and transcript vectors searchableCortex Search (multi-index)
4. SearchQuery one or all modalities at oncemulti_index_query
5. AnalyzeAsk questions about a videoAI_COMPLETE with Pegasus 1.5
6. AppPut it all behind a UISnowflake App Runtime (Next.js)

What You'll Learn

  • How to embed video, text, and images into one shared vector space with AI_MULTI_EMBED
  • How to build a Cortex Search service with multiple vector indexes, one per modality
  • How to search them with multi_index_query
  • How to ask questions about a video with AI_COMPLETE and Pegasus 1.5
  • How to deploy the app on Snowflake App Runtime

What You'll Need

  • A Snowflake account in a region that supports twelvelabs-marengo-embed-3-0, or with cross-region inference enabled. See AI_MULTI_EMBED regional availability.
  • A role with SNOWFLAKE.CORTEX_USER and privileges to create a database, stage, and Cortex Search service
  • A few .mp4 files (a handful of short clips is enough to start)
  • Node.js 20+ and the Snowflake CLI to run and deploy the app

Stage Your Videos

Create a database, a schema, and a stage with a directory table. The directory table is what lets you list and reference files from SQL.

CREATE DATABASE IF NOT EXISTS TWELVE_LABS_DEMO;
CREATE SCHEMA IF NOT EXISTS TWELVE_LABS_DEMO.VIDEO_INTELLIGENCE;
USE SCHEMA TWELVE_LABS_DEMO.VIDEO_INTELLIGENCE;

CREATE STAGE IF NOT EXISTS VIDEO_STAGE
  DIRECTORY  = (ENABLE = TRUE)
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');

Upload your videos to the stage in Snowsight (Data » Add Data » Load files into a Stage), or from the CLI:

snow stage copy ./videos/*.mp4 @TWELVE_LABS_DEMO.VIDEO_INTELLIGENCE.VIDEO_STAGE

Then refresh the directory table:

ALTER STAGE VIDEO_STAGE REFRESH;
SELECT RELATIVE_PATH, SIZE FROM DIRECTORY(@VIDEO_STAGE);

aside positive Already have video in S3, GCS, or Azure? Create an external stage on a storage integration instead. The rest of this guide works the same with either kind of stage.

We also need a small internal stage where the app uploads images for image-to-video search and caches clips for Pegasus:

CREATE STAGE IF NOT EXISTS SEARCH_UPLOADS
  DIRECTORY  = (ENABLE = TRUE)
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');

Embed Videos with Marengo

AI_MULTI_EMBED with Marengo watches each video and returns a list of segments. Each segment has a 512-dimensional vector for one modality (visual, audio, or transcription) plus its start and end time.

Flatten that output into one row per segment:

CREATE OR REPLACE TABLE VIDEO_EMBEDDINGS AS
WITH raw AS (
    SELECT
        REPLACE(RELATIVE_PATH, '.mp4', '') AS episode,
        AI_MULTI_EMBED(
            'twelvelabs-marengo-embed-3-0',
            TO_FILE('@VIDEO_STAGE', RELATIVE_PATH)
        ) AS emb
    FROM DIRECTORY(@VIDEO_STAGE)
)
SELECT
    episode,
    f.value['embedding']::VECTOR(FLOAT, 512) AS embedding_vec,
    f.value['embedding_option']::STRING      AS modality,
    f.value['start_sec']::FLOAT              AS start_sec,
    f.value['end_sec']::FLOAT                AS end_sec
FROM raw, LATERAL FLATTEN(input => raw.emb['value']) f;

Check what you got:

SELECT modality, COUNT(*) AS segments, COUNT(DISTINCT episode) AS videos
FROM VIDEO_EMBEDDINGS
GROUP BY modality;

aside positive Marengo puts text, images, and video in the same vector space. That's why a text query or an uploaded photo can be matched directly against video segments.

Now pivot to one row per segment with a column per modality, so each modality can get its own index:

CREATE OR REPLACE TABLE VIDEO_SEGMENT_VECTORS AS
SELECT
    episode || '_' || start_sec::STRING AS segment_id,
    episode, start_sec, end_sec,
    MAX(IFF(modality = 'visual',        embedding_vec, NULL)) AS visual_vec,
    MAX(IFF(modality = 'audio',         embedding_vec, NULL)) AS audio_vec,
    MAX(IFF(modality = 'transcription', embedding_vec, NULL)) AS transcript_vec
FROM VIDEO_EMBEDDINGS
GROUP BY episode, start_sec, end_sec;

Create one Cortex Search service with three vector indexes. Setting query_model tells the service which model produced the vectors, so you can query each index independently or together.

CREATE OR REPLACE CORTEX SEARCH SERVICE VIDEO_MULTI_SEARCH_SVC
  VECTOR INDEXES
    VISUAL_VEC     (query_model = 'twelvelabs-marengo-embed-3-0'),
    AUDIO_VEC      (query_model = 'twelvelabs-marengo-embed-3-0'),
    TRANSCRIPT_VEC (query_model = 'twelvelabs-marengo-embed-3-0')
  PRIMARY KEY (SEGMENT_ID)
  ATTRIBUTES  EPISODE
  WAREHOUSE   = <your_warehouse>
  TARGET_LAG  = '1 day'
AS (
    SELECT SEGMENT_ID, EPISODE, START_SEC, END_SEC,
           VISUAL_VEC, AUDIO_VEC, TRANSCRIPT_VEC
    FROM VIDEO_SEGMENT_VECTORS
);

Search with multi_index_query

Embed the query with the same Marengo model, then pass the vector to the index you want to search. Run these snippets in a Snowflake Notebook, where session comes from get_active_session():

from snowflake.core import Root
from snowflake.snowpark.context import get_active_session

session = get_active_session()
svc = (Root(session).databases["TWELVE_LABS_DEMO"]
       .schemas["VIDEO_INTELLIGENCE"]
       .cortex_search_services["VIDEO_MULTI_SEARCH_SVC"])

query_vec = session.sql("""
  SELECT AI_MULTI_EMBED('twelvelabs-marengo-embed-3-0',
         'Find me a scene where a ship is burning at night'):value[0]['embedding']::ARRAY
""").collect()[0][0]

results = svc.search(
    multi_index_query={"visual_vec": [{"vector": query_vec}]},
    columns=["episode", "start_sec", "end_sec"],
    limit=5,
)

To search what's seen, heard, and said in one request, pass the same vector to every index:

results = svc.search(
    multi_index_query={
        "visual_vec":     [{"vector": query_vec}],
        "audio_vec":      [{"vector": query_vec}],
        "transcript_vec": [{"vector": query_vec}],
    },
    columns=["episode", "start_sec", "end_sec"],
    limit=5,
)

aside positive Image search works the same way: upload the image to SEARCH_UPLOADS and embed it with AI_MULTI_EMBED('twelvelabs-marengo-embed-3-0', TO_FILE('@SEARCH_UPLOADS', '<image>.jpg')). multi_index_query accepts text or vectors, so always embed files first.

aside negative Raw Marengo similarity scores are on different scales per modality (transcript matches typically score much higher than visual or audio). If you want to weight modalities yourself, query each index separately, min-max normalize the scores, and combine them. The sample app does this in its MARENGO_SEARCH procedure.

Analyze Video with Pegasus 1.5

Pegasus 1.5 is a video understanding model, available in AI_COMPLETE. Give it a staged video and a natural-language prompt to summarize it, answer questions about it, or extract structured information:

SELECT AI_COMPLETE(
    'twelvelabs-pegasus-1-5',
    'Summarize this video in 3 bullets, then return JSON with the main characters and the timestamp each first appears.',
    TO_FILE('@VIDEO_STAGE', '<your_video>.mp4')
) AS analysis;

There's no frame extraction and no transcription step.

aside negative Preview: video processing with AI_COMPLETE, including twelvelabs-pegasus-1-5, is in Public Preview. See Cortex AI Functions: Multimodal for current status and regional availability.

aside positive Pegasus 1.5 accepts MP4, MOV, AVI, MKV, and WebM files up to 2 hours and 2 GB. See the release note and Cortex AI Functions: Multimodal.

aside positive Pair it with search. To analyze just one search hit, cut its start_sec/end_sec range to a clip and send the clip to Pegasus. The sample app's PEGASUS_ANALYZE procedure does this, caching clips in SEARCH_UPLOADS/clips/.

Deploy the App

The front end is a Next.js app that runs on Snowflake App Runtime (SAR). It's a thin layer over the SQL you just wrote:

  • Search: embeds the text or image query with Marengo, runs multi_index_query against VIDEO_MULTI_SEARCH_SVC, and plays each hit at its timestamp with a presigned URL.
  • Analyze: sends a video (or a search hit's time range) and a prompt to Pegasus 1.5.
Search
Analyze

Grab the app source from the src/ folder of this guide. src/setup.sql creates the stored procedures (MARENGO_SEARCH, CLIP_VIDEO, PEGASUS_ANALYZE) and the Cortex Agent the app calls; replace COMPUTE_WH in it with your warehouse, then:

npx degit Snowflake-Labs/sfquickstarts/site/sfguides/src/build-a-video-search-app-with-twelve-labs/src tl-video-studio
cd tl-video-studio
snow sql -f setup.sql  # procedures + Cortex Agent
npm install
npm run dev            # run locally against your default Snowflake connection
snow app setup --app-name=tl_video_studio   # first deploy only: creates the app's deployment config
snow app deploy        # deploy to Snowflake App Runtime

See the src/ README for the grants the app's owner role needs.

Try it

  • Search: "Find me a scene where a ship is burning at night", or upload a photo of a location or character.
  • Analyze: open a hit and ask "Return JSON with a summary, the main characters, and any historical references."

Clean Up

App Runtime apps keep consuming compute while they run. When you're done, remove the deployed app from the tl-video-studio folder, then drop the demo database (stages, tables, search service, procedures, and agent):

snow app teardown --force
DROP DATABASE IF EXISTS TWELVE_LABS_DEMO;

Conclusion and Resources

You built a video search and analysis app with SQL functions and a small front end on Snowflake App Runtime:

  • Marengo 3.0 (AI_MULTI_EMBED) turned your videos into time-stamped visual, audio, and transcript vectors.
  • Multi-index Cortex Search made every modality searchable by text or image in one service.
  • Pegasus 1.5 (AI_COMPLETE) answered questions about any video or moment.

What You Learned

  • How to embed video, text, and images into one shared vector space with Marengo
  • How to build and query a multi-index Cortex Search service with multi_index_query
  • How to analyze a video, or a single search hit, with Pegasus

Next Steps

  • Run Pegasus over your whole library with a JSON schema and store the output as a metadata table you can join to your business data.
  • Wrap search and analysis as tools on a Cortex Agent.

Related Resources

Updated Nov 17, 2026

This content is provided as is, and is not maintained on an ongoing basis. It may be out of date with current Snowflake instances