SQL autocomplete looks simple: Given the text around a cursor, predict what a developer is likely to type next.
In production, however, SQL autocomplete is a demanding problem of both inference speed and quality. A suggestion must arrive almost instantly, follow the Snowflake SQL dialect, use the correct objects from the available schema and align with the user's likely intent. The model must also know when to stay silent. When the available context does not support a reliable completion, or when the SQL around the cursor is already complete, showing nothing is better than interrupting the user with an unnecessary or incorrect suggestion.
At Snowflake, AI Code Suggestions — our SQL autocomplete feature in Snowsight — are now generally available, serving up to 70,000 users a day and processing up to 2 million triggers. Our previous production system used a 30B-A3B mixture-of-experts (MoE) model trained primarily on synthetic, targeted examples. We set out to determine whether a much smaller model, trained specifically for the Snowflake autocomplete experience, could improve suggestion quality while reducing serving computation.
The work delivered several production improvements:
- A 4B dense model outperformed the previous 30B-A3B MoE model in both internal precision and actual user acceptance.
- Internal test-set precision increased from 23.4% to 40.4%.
- Actual user acceptance increased from 17.80% to 26.35%, even as the model produced 20% more suggestions.
- Median inference latency decreased by 71%, to 29% of the previous model's baseline.
Two technical advances were central to these gains:
- Context-grounded training targets improved precision by more than 13 percentage points after supervised fine-tuning.
- An LLM-judge reward provided a stronger reinforcement-learning signal than a rule-based SQL-BLEU reward.
In this blog, we describe how we built a production-oriented evaluation from internal employee editing sessions, reconstructed intended completions, curated context-grounded post-training targets and used reinforcement learning to teach the model what to suggest, how much to suggest and when to remain silent.
| Model | Architecture | Internal test-set precision | Actual user acceptance rate⁺ | Relative inference latency | Relative suggestion volume |
|---|---|---|---|---|---|
| Previous model | 30B-A3B MoE | 23.4% | 17.80% | 100% (baseline) | 100% |
| New model | 4B dense | 40.4% | 26.35% | 29% of baseline | 120% of baseline |
⁺ Actual user acceptance rates were calculated over comparable seven-day windows for both models.
Autocomplete is a precision-first problem
General-purpose coding models provide a strong foundation for autocomplete. Many are pretrained with fill-in-the-middle objectives and have learned patterns across programming languages, configuration files and SQL dialects.
But Snowflake SQL represents only a small portion of that pretraining distribution. General coding ability does not guarantee familiarity with Snowflake-specific syntax, enterprise schemas or the query patterns encountered in production.
Autocomplete also differs from conventional code generation. In a chat interface, the user explicitly requests an answer and can review it before acting. Autocomplete intervenes while the user is already working. A poor suggestion can interrupt the user and reduce trust in future completions.
This leads to three core requirements.
Low latency: Suggestions must feel immediate. Moving from a 30B-A3B MoE model to a 4B dense model reduces the time required to serve each trigger.
Snowflake specialization: The model must understand the structure of Snowflake SQL and ground its suggestions in schema information, prior SQL history relevant to the current schema and the text surrounding the cursor.
Calibrated abstention: The model should remain silent when the context does not support a reliable suggestion or when the query is already complete.
While latency is easy to quantify, measuring the other two requires an evaluation set that represents not only successful completions, but also ambiguous contexts and cases where the correct response is empty.
Building evaluation from internal editing sessions
To reflect production-like behavior without using customer data, we construct our evaluation datasets from autocomplete interactions generated by Snowflake employees in internal environments.
We maintain a continuously evolving suite of evaluation datasets sampled from different internal traffic periods. Each dataset is fixed once created, allowing reproducible comparisons across model versions, while new datasets are added as product capabilities, query patterns and user behavior evolve.
We also maintain specialized datasets for specific failure modes and edge cases, but use them primarily for diagnosis rather than as the main measure of model quality.
Reconstructing the intended completion
Each sampled trigger contains the SQL prefix and suffix, relevant schema, prior SQL history, trimmed worksheet context and several queries subsequently executed by the user.
Not every subsequent query is related to the original trigger. A user may switch worksheets, execute an unrelated statement or continue editing through several intermediate versions.
We first use an LLM to identify which subsequent executed query, if any, most likely evolved from the worksheet state at the time of the trigger. We then use another LLM step to extract the completion that best represents the intended edit at the original cursor position.
This is not a simple text-diff problem. Between the trigger and execution, the user may edit both sides of the cursor, correct typos, rename aliases or restructure the query. A direct diff can therefore assign unrelated changes to the autocomplete span.
Ambiguous examples receive additional human inspection. The result is a fixed reference completion that can be used consistently across model evaluations.
During evaluation, an LLM judge receives the model suggestion together with the available context and reference completion. It classifies the suggestion as incorrect, partially useful or fully useful, and identifies the primary failure category.
Measuring useful autocomplete
Autocomplete quality cannot be determined by a single metric.
A model can achieve high precision by responding very rarely, or it can generate technically correct suggestions that are too short to add meaningful value.
We therefore track three primary behaviors.
Precision is the percentage of nonempty suggestions judged to be fully useful:
Precision = fully useful suggestions / all nonempty suggestions
For the current product experience, we generally prefer a conservative model with high precision over one that responds more often but less reliably.
Suggestion length is tracked to prevent precision from rewarding trivial completions. In earlier experiments, we trained high-precision models whose correct suggestions were often only a single word or token. Users reported that these suggestions did not meaningfully improve the experience.
Abstention must be measured in both directions. The laziness rate captures cases where the reference completion is nonempty but the model produces nothing. We also curate examples where the prefix and suffix already form the complete query and the expected response is empty.
Together, these metrics describe whether the model responds, whether the response is useful and whether it provides enough value to show.
With these behaviors defined, we next applied the same principles to the data used for post-training.
Training only on predictable completions
We use the same LLM-assisted reconstruction pipeline to curate post-training data from internal employee editing sessions. Compared with rule-based matching, this process substantially improves coverage when employees edit both sides of the cursor or execute several intermediate queries.
However, accurately recovering what the user eventually typed does not guarantee that the model could have predicted it from the context available at the trigger. This distinction led us to make context grounding a central part of training-data curation.
Context grounding
Consider a user who eventually inserts a column absent from the provided schema, perhaps because schema retrieval was incomplete. Or consider a filter containing a literal that never appeared in the SQL history, worksheet context or schema metadata.
These may be valid human edits, but they are impossible autocomplete targets. Training on them encourages guessing and penalizes the model for failing to produce information it could not have known.
We define a target as context-grounded when it can be reasonably inferred from the prefix, suffix, schema, prior SQL history and worksheet context.
For each raw reference edit, we identify the longest useful prefix that can be reasonably supported by the available context. If the full edit contains an unpredictable identifier or literal, we retain only the longest useful portion that remains predictable from the available context. If no useful portion is sufficiently grounded, the training target is empty. The model is therefore trained to generate only what it could reasonably have inferred at that moment.
| SFT Model | Internal test-set precision | Avg. judge rating |
|---|---|---|
| Without context grounding | 23.9% | 0.704 |
| With context grounding | 37.4% | 0.865 |
As shown in Table 2, context grounding improved internal test-set precision by more than 13 percentage points and raised the average judge rating from 0.704 to 0.865 after supervised fine-tuning (SFT). Removing targets unsupported by the available context substantially improved suggestion quality and reduced the incentive to guess.
Context grounding alone, however, does not fully determine the ideal training target. A predictable continuation is not necessarily useful enough to show, while an unpredictable ground truth does not imply an alternative reasonable prediction could not have been provided there. Training-data curation must therefore consider both predictability and the behavior we want from the product.
We also anonymize identifiers and other potentially sensitive information before training. Although anonymization removes semantic clues, it often acts as useful regularization by encouraging the model to rely on SQL structure and the schema included in the prompt rather than memorizing familiar identifiers.
These supervised-data improvements teach the model what can be inferred from the available context. But autocomplete also requires behavioral alignment: The model must decide whether a possible completion is useful enough to show, how much of it to generate and when to abstain. We use reinforcement learning to further strengthen those decisions.
Reinforcement learning for calibrated autocomplete
After SFT, we used reinforcement learning to improve three behaviors that are central to autocomplete: what to complete, how much to complete and when to remain silent.
We trained the model with Group Relative Policy Optimization (GRPO). For each prompt, the model generates several candidate completions, and a reward function ranks them according to their usefulness.
Reward design is particularly important for autocomplete. Unlike conventional text-to-SQL or code-generation tasks, the SQL around the cursor may be incomplete, and the best response may be empty. Execution success is therefore not a generally available reward signal. A useful reward must evaluate not only the quality of the generated SQL, but also whether the model should respond and how much it should suggest.
To support this behavior, we combine graded quality signals with deterministic checks that discourage ungrounded identifiers, unnecessary continuations and excessive abstention. These checks ensure that the model receives positive feedback only for completions that are appropriately scoped and supported by the available context.
Comparing rule-based and model-based rewards
Among the reward approaches we evaluated, two emerged as the most promising. The first was SQL-BLEU, a deterministic SQL-aware metric that compares a candidate with the reference completion using both token-level and structural similarity. It is inexpensive and reproducible, and works even when the query is incomplete. However, like other reference-based metrics, SQL-BLEU is limited by its dependence on a single target. It may under-reward a completion that differs from the reference text but still represents a valid and useful next edit.
Our second approach used an LLM judge aligned with the same notion of usefulness as our offline evaluation. The judge considers whether a completion is syntactically valid, semantically meaningful, aligned with the intended edit and coherent with the text on both sides of the cursor.
Starting from the same SFT checkpoint, the LLM-judge reward produced stronger results than the SQL-BLEU reward, as shown in Table 3.
| Reward (same SFT checkpoint) | Internal test-set precision | Avg. judge rating |
|---|---|---|
| SFT baseline⁺ | 31.8% | 0.693 |
| Gates + SQL-BLEU (rule-based) | 33.1% | 0.805 |
| Gates + LLM judge | 36.1% | 0.841 |
⁺Experiments in this table used an intermediate SFT checkpoint during reward development rather than the final deployed checkpoint. The values therefore differ from the final results reported in Table 1.
The rule-based reward remains valuable as a fast and deterministic signal during development. The LLM judge, however, better captures semantic equivalence and user intent, making it a stronger signal for final model alignment.
Reward design also requires careful balancing. In an early experiment, optimizing too heavily for precision caused the model to remain silent on most prompts. Although the resulting model appeared precise, it failed to provide completions in many cases where it could have helped. This result reinforced the importance of optimizing the full autocomplete behavior rather than a single metric. The final reward balances suggestion quality, groundedness and appropriate participation.
Results
The final 4B dense model delivered substantially higher precision than the previous 30B-A3B MoE model. More importantly, the offline improvement translated into a higher actual acceptance rate across the deployed user population.
The production deployment confirmed the gains summarized in Table 1. The new model increased internal test-set precision from 23.4% to 40.4%, increased actual user acceptance from 17.80% to 26.35% over comparable seven-day windows and reduced median inference latency to 29% of the previous baseline — all while producing 20% more suggestions.
Offline precision remains higher than observed acceptance. A meaningful portion of dismissed suggestions still align with the query the user eventually executes. Acceptance is influenced by factors beyond correctness: Users may continue typing, correct a typo, revise another part of the query or simply not act on a suggestion at that moment.
This distinction matters. A useful autocomplete system must optimize not only for correctness, but also for timing and the amount of value provided by each suggestion.
Conclusion
Production SQL autocomplete is a precision-first, latency-sensitive problem where specialization and calibrated silence matter as much as raw generation capability.
By building representative evaluation sets, reconstructing intent from internal employee editing sessions, removing unsupported training targets and aligning the model with reinforcement learning, we trained a 4B dense model that outperformed the previous 30B-A3B system in precision, acceptance and inference efficiency. No customer query data was used in evaluation or training.
AI Code Suggestions in Cortex Code are now generally available. We invite Snowflake users to try the feature and let us know where it helps, where it gets in the way and how we can make the autocomplete experience even better.
The broader lesson is that autocomplete quality does not depend on model scale alone. It comes from aligning every part of the training and evaluation pipeline with one deceptively difficult product question:
At this exact cursor position, is there something useful the model can confidently say?
