Blog/AI & ML/Cortex Can Now Watch, Hear and Understand Video: TwelveLabs on Snowflake
Nov 18, 2026/5 min readAI & ML

Cortex Can Now Watch, Hear and Understand Video: TwelveLabs on Snowflake

The problem

Video is now the most produced, most consumed and least understood data in the enterprise.

It accounts for the largest share of consumer internet traffic. Within organizations, video content includes marketing libraries with tens of thousands of creatives, sports archives spanning decades of broadcast footage, security and operations footage, training content, earnings calls, user-generated media and content libraries worth billions.

And yet, little of it is easily queryable. It sits in cold storage as a cost line, not an asset. That is a missed opportunity. Queryable video could help marketers tie the content of a creative to its campaign performance, let broadcasters monetize decades of archives, let retailers turn product videos into structured catalog data and help compliance teams locate relevant footage more efficiently. The insights are already in the video; organizations just can't ask questions of it.

The reason is simple: Video has lacked a comparable foundation-model moment. Text got transformers, embeddings and reasoning models, and an ecosystem of search, summarization and analysis emerged. Video had no equivalent. Computer vision recognized objects in frames but could not understand narrative, emotion or temporal flow. Transcription captured speech but missed visual information. The modalities stayed siloed. Enterprise video remained dark data.

That changes now. Snowflake has partnered with TwelveLabs to bring video-native AI capabilities available through Snowflake Cortex AI Functions, enabling organizations to search and analyze video using SQL, while applying Snowflake’s governance and security controls.

Why TwelveLabs

TwelveLabs builds video-native AI models to process visuals, audio and speech together across time. The models are designed for unified, temporal analysis of video content, rather than relying solely on individual-frame analysis or separate modality pipelines. 

Two models power this:

Marengo (the perception engine): A multimodal encoder that converts video into 512-dimensional embeddings capturing visual content, audio patterns and spoken language in a single representation. Think of it as what text embedding models did for documents, but for video.

Pegasus (the reasoning engine): A video-language model that watches video and generates text: summaries, structured metadata, answers to questions, scene descriptions. Think of it as what GPT did for text, but for video.

Together, they give you both sides of the video intelligence stack. Marengo finds the right moment in a massive library. Pegasus tells you what that moment means.

Why this matters on Snowflake

Your campaign data, audience segments, content metadata and spend already live in Snowflake. Until now, the video that generated all of that data was locked outside the analytics layer.

With TwelveLabs video AI capabilities accessed through Cortex AI Functions:

  • Generate video embeddings in-place with Cortex AI Functions. 
  • It’s SQL. Not a separate platform, not a Python SDK, not a new vendor to onboard. A function call in a query.
  • Video-derived signals sit in the same tables as your structured data. Content intelligence meets performance metrics in a single query.
  • It’s governed. Row-level policies, role-based access, audit logs. The controls can apply to video-derived data stored in Snowflake tables.
architecture

What it looks like

Embed: One function call to create searchable video embeddings

Videos live on a Snowflake stage, internal or external (S3, GCS, Azure).

 

SELECT AI_MULTI_EMBED(
    'twelvelabs-marengo-embed-3-0',
    TO_FILE(@my_stage, relative_path)  -- works with internal or external stages
) AS embeddings
FROM video_catalog;

Marengo processes each video end-to-end and returns a structured object containing embeddings across three modalities. Each modality produces one or more segments depending on the video’s length and content, with every segment represented as a 512-dimensional vector:

field

The number of segments varies by video length and modality. A 2-minute video might produce 10 visual segments, 8 audio segments and 12 transcription segments. Each captures a different facet of the content: what is on screen, what is heard and what is spoken (supporting 36 languages).

Your video library becomes a vector database, searchable with plain text.

Search: Describe what you want, find the exact moment

Once your video embeddings are stored, Cortex Search indexes them and makes your video library searchable in natural language, reducing reliance on manual tags, keyword matching and maintained content taxonomies.

 

query_vec = session.sql("""
  SELECT AI_MULTI_EMBED('twelvelabs-marengo-embed-3-0',
         'Find me a scene where a ship is burning at night'):value[0]['embedding']::ARRAY
""").collect()[0][0]

results = svc.search(
    multi_index_query={"visual_embedding": [{"vector": query_vec}]},
    columns=["video_file", "start_sec", "end_sec"],
    limit=5
)

In practice: Challenger School's marketing team built a footage-search tool on AI_MULTI_EMBED that surfaces relevant clips in minutes, replacing a manual search process that previously stretched across weeks.

"Using Twelve Labs' Marengo model through Snowflake's AI_MULTI_EMBED, our team can ask for footage in natural language and receive timestamps, transcript context, and Premiere review sequences linked to the original source files. This gives our editors a practical starting point for creating videos."

Yashwanth Karumanchi
Research & Development, Challenger School Foundation

Analyze: Ask any question, get structured answers

One function call can simplify video analysis. Depending on the use case, you may not need a separate frame-extraction or transcription pipeline. Pegasus can analyze video and return free-text answers or structured metadata.

 

SELECT AI_COMPLETE(
    'twelvelabs-pegasus-1-5',
    'What are the main characters doing? Identify any historical references.',
    TO_FILE(@my_stage, relative_path)  -- works with internal or external stages
) AS analysis
FROM video_catalog;

For advanced use cases, Cortex Agents can orchestrate multi-vector search across modalities combining visual, audio and transcription signals with custom ranking strategies, all through tool calling. Describe what you are looking for in plain language, and the agent retrieves the most relevant moments from across your video library.

surface

Who this is for

industry

Get started

If you are sitting on video data — content libraries, ad creatives, broadcast archives, surveillance footage and training materials — that should be generating insights, this is your opportunity to turn storage costs into strategic assets.

Marengo 3.0 and Pegasus 1.5 are now generally available. Follow this guide to get started on multimodal use cases.

 

Share this post

Subscribe to our blog newsletter

Get the best, coolest and latest delivered to your inbox each week

Where Data Does More