Natural Language Processing in the Age of LLMs

Human language is full of ambiguity. Words change meaning based on their surrounding context, and the same idea can be expressed countless ways. Natural language processing (NLP) has spent decades trying to make that complexity understandable to computers. Large language models (LLMs) are a powerful class of models within the field of NLP. But LLMs haven’t made other approaches obsolete. Modern NLP still draws on a range of models and techniques, each suited to different kinds of language tasks and system requirements.

Natural language processing is a field of artificial intelligence and computer science focused on enabling computers to process human language and extract or produce meaning from it.

The field includes two closely related areas:

• Natural language understanding (NLU) focuses on interpreting language: identifying intent, classifying text, extracting entities or determining how words relate to one another in context.
• Natural language generation (NLG) concerns the production of language, including summarization, translation, question answering and generative AI.

The two categories frequently overlap in modern systems. A conversational application, for example, needs to interpret a user’s request before generating an appropriate response. A document intelligence system might identify relevant passages, extract structured information and then summarize what it found.

NLP now describes a field rather than a particular technical stack. Rules, statistical models, specialized neural networks and LLMs all sit within its history and, in some cases, its current practice.

See how Booking.com accelerates travel innovation with Cortex AI:

Natural language processing is a field of AI concerned with computational approaches to process, understand and generate human language. An LLM is a particular kind of pretrained model that sits within the field: typically transformer-based, trained on very large collections of data and capable of performing a broad range of language tasks.

Task-specific NLP models continue to operate alongside LLMs. For a narrowly defined workload — particularly one running at high volume or with strict latency, cost or output requirements — a specialized classifier or smaller language model may be a better fit over an LLM, while an application requiring flexible interaction across many kinds of language, meanwhile, is more likely to benefit from an LLM.

Task-specific NLP models vs. LLMs

Modern NLP architectures often use both. A system might rely on a specialized classifier for a high-volume routing task, an embedding model for retrieval and an LLM for generating a response. The choice is rarely about whether NLP or LLMs “win.” Teams select the model or combination of models that fits each part of the application.

There’s no single NLP pipeline. A rule-based parser, a statistical classifier and a transformer-based language model process text differently.

At a high level, however, NLP systems need some way to represent language computationally, identify useful relationships within that representation and produce an output suited to the task. In modern transformer-based systems, that process typically begins with tokenization and proceeds through layers of learned contextual representation.

1. Convert text into tokens

Before a model processes language, the text is divided into units called tokens. Many modern language models use subword tokenization, which breaks text into frequently occurring words, word pieces or character sequences rather than requiring every possible word to exist as a separate vocabulary item.

This design helps models handle uncommon terms, variations of the same word and languages with large vocabularies. A word the model has never seen as one unit may be represented through smaller familiar components.

Each token receives an identifier from the model’s vocabulary. At this point, however, the model still has numbers representing symbols rather than any useful representation of their meaning.

2. Represent tokens as vectors

The model maps token identifiers into vectors, called vector embeddings: arrays of numerical values that provide a mathematical representation the network can process.

These initial embeddings serve as a starting point. As the representation passes through a transformer, information from surrounding tokens alters it. The representation of bank in river bank, for instance, develops differently from the representation of bank in savings bank because the neighboring context differs.

This contextual representation is one of the important departures from earlier methods where a word often had the same fixed representation wherever it appeared.

3. Use attention to model context

Transformers use self-attention to calculate relationships among tokens in a sequence. For each token, the model assigns different levels of relevance to other tokens as it constructs a contextual representation.

Consider the sentence: The customer deposited the check at the bank.

The words customer, deposited and check provide strong evidence for the financial meaning of bank. In another sentence — The canoe drifted toward the bank — entirely different surrounding words shape the interpretation.

Multi-head attention performs several sets of these comparisons in parallel. Individual attention heads can capture different relationships within the same sequence, allowing the network to represent multiple aspects of context at once.

4. Refine those representations through transformer layers

Attention is only one component of a transformer block. After attention combines contextual information, additional neural-network operations transform the resulting representations. Repeating this process across many layers produces increasingly rich representations of the input.

Those layers allow models to encode patterns involving syntax, semantics, references between distant parts of a passage and other relationships that are difficult to capture through local word order alone.

The architecture used to process those representations differs among model families. Encoder-oriented models build representations particularly suited to classification and language understanding. Decoder models generate sequences one token at a time. Encoder-decoder models first represent an input sequence and then generate a corresponding output sequence, an architecture long associated with tasks such as translation and summarization.

5. Produce an output for the task

What happens next depends on the NLP application. A classifier might convert the final representation into probabilities across categories such as positive, neutral and negative. An information extraction model might label particular spans of text as names, organizations or dates. A generative model predicts a distribution over possible next tokens and repeatedly selects from that distribution to produce a sequence.

For many current systems, the underlying model was pretrained on a much broader language objective before being used for the specific application. Teams then supply task instructions, retrieve external context, fine-tune the model or combine several of those methods.

That pretraining-and-adaptation workflow is one of the defining features of modern NLP. Instead of learning language from scratch for every application, systems increasingly reuse representations and capabilities learned across much larger collections of text.

Much of the information businesses work with every day takes the form of language: documents, emails, support conversations, reports, product descriptions and knowledge-base articles. NLP provides ways to interpret that material computationally, whether the goal is to assign a label, find relevant information, extract a fact or generate new text.

The boundaries between those jobs have also become less rigid. Earlier systems were frequently designed around a single task. With pretrained models and shared language representations, one application might retrieve passages from documents, extract information from them and generate a summary in response to a user’s question.

Broadly, NLP applications fall into four areas:

Understand and classify text

Some of the longest-running NLP tasks involve converting unstructured text into categories or structured information. Text classification assigns text to predefined categories. An organization might classify incoming support requests by topic or determine which documents belong to a particular business domain. Intent classification performs a related job by identifying what a user is trying to accomplish from the language of a request.

Other systems look inside the text itself. Named entity recognition (NER) identifies entities such as people, organizations, products and locations within a document. Information extraction pulls facts or relationships from documents. Sentiment analysis estimates attitudes or opinions expressed in text.

Depending on the complexity of the language and the requirements of the application, teams might use an LLM or they might use task-specific approaches ranging from statistical classifiers to specialized transformer models and general-purpose foundation models.

Find information by meaning

Keyword search works well when users know the terminology contained in the information they need. Language is less cooperative when the wording differs. A search for employees who left the company, for example, could miss documents that consistently use worker attrition terminology. Semantic search addresses that gap by representing queries and documents according to their meaning. Embedding models encode text as vectors, after which similarity calculations identify content whose representations are close together even when the wording is different.

That same mechanism supports document discovery, recommendation, clustering and retrieval for generative AI systems. In retrieval-augmented generation (RAG), for instance, a system first finds information relevant to a request and supplies that material to a generative model as context.

Transform language and information

Other NLP tasks convert language from one form into another. Machine translation is an example: an input sequence in one language produces an output sequence in another. Summarization compresses a longer document or collection of documents while preserving salient information. Information extraction turns prose into structured fields, while natural language interfaces such as text-to-SQL translate a question expressed in ordinary language into another representation that software can execute.

Although generative models have broadened this category considerably, transformation doesn’t necessarily mean free-form generation. The desired output might be a tightly constrained schema, a translated sentence or a small set of extracted values.

Generate and interact through language

LLMs have greatly expanded the generative side of NLP. Question-answering systems, conversational assistants and natural language interfaces now support interactions that extend across multiple turns and tasks rather than following a fixed set of commands.

In an enterprise setting, those applications frequently sit on top of other NLP capabilities. A user asks a question in natural language, then the system interprets the request, retrieves relevant information, incorporates structured or unstructured context and finally generates an answer. What appears to the user as a single conversation may involve several distinct language-processing operations behind the interface.

Not every NLP workflow begins with written text. Speech recognition converts spoken language into a representation that downstream language systems can process. A call-center application, for example, might transcribe a conversation before applying intent classification, information extraction, sentiment analysis or summarization to the resulting text.

Once the task is defined, the next question is architectural: what does the system need from its model and its data? For some applications, the primary requirement is a useful numerical representation of language. Others need access to information outside a model’s training data, stronger performance within a specialized domain or tighter control over how a model responds. Modern NLP systems typically combine several approaches, often within the same application.

Represent semantic meaning with embeddings

Embeddings represent pieces of information as vectors — ordered sets of numbers whose positions in a high-dimensional space reflect relationships learned by a model. For text, embeddings provide a computational way to compare meaning. Passages discussing similar concepts tend to receive representations that are closer together than unrelated passages, even when they share relatively few exact words.

Embeddings support a range of NLP workloads. Semantic search uses embeddings to match queries with conceptually related documents. Clustering groups similar text without requiring predefined categories. Recommendation and deduplication systems likewise use vector similarity to identify relationships among pieces of content.

Note that these text embeddings are distinct from the contextual token representations discussed earlier. Dedicated embedding models also produce vectors, but typically at the level of a sentence, passage or document. These vectors are designed for tasks such as semantic search, retrieval and similarity comparison.

Retrieve information with semantic and vector search

Once text has been represented as vectors, a retrieval system can compare those representations and locate relevant content. Traditional lexical search and semantic retrieval are often complementary. Exact terms remain valuable — particularly for names, identifiers, error codes and specialized terminology — while semantic similarity helps when a query expresses the same idea using different language.

Modern retrieval systems frequently combine several signals rather than relying on one method alone. For example, a search capability might use lexical matching and vector similarity to generate candidates, apply metadata filters or access rules, then rerank the results before passing them to an application.

For NLP systems built over enterprise information, retrieval quality sets an important boundary around everything that happens downstream. A capable language model has little opportunity to answer accurately from proprietary information if the retrieval layer supplies an irrelevant document or overlooks the passage containing the answer.

Note that more retrieved context doesn’t necessarily improve the result. Retrieval tuning is highly workload-dependent. In Snowflake’s internal testing on a sample IT data set, fewer retrieved chunks sometimes outperformed larger context windows — a reminder that optimal retrieval depth should be evaluated against each team’s specific data and use case.

Ground generative models with RAG

A pretrained language model reflects patterns learned during training. The training data wouldn’t have included a current, authoritative copy of an organization’s internal documents, recent transactions or changing product information.

Retrieval-augmented generation brings external information into the system and places it in the model’s context before generation. The model then produces its response with access to that material.

This is particularly useful when knowledge changes frequently or lives in proprietary systems. Updating the underlying source can make newer information available for retrieval without requiring another round of model training.

Retrieval doesn’t guarantee factual output, however. Poor retrieval, ambiguous sources, conflicting documents and failures in generation all remain possible. For this reason, RAG systems require evaluation across both stages: whether the right information was retrieved and whether the model used it correctly.

Adapt a model through fine-tuning

When a problem is a result of model behavior or task performance rather than access to information, fine-tuning offers another route. During fine-tuning, additional training updates the weights of a pretrained model using examples relevant to a narrower objective. Depending on the task, that process might teach a model a specialized classification scheme, reinforce a particular output format or improve performance on language characteristic of a domain.

The amount of adaptation varies. Some methods update all model parameters, while parameter-efficient approaches modify or add a much smaller set. Instruction tuning, meanwhile, trains models on examples of instructions and desired responses so that they follow new requests more effectively. Preference-based post-training techniques further shape how models select among plausible responses.

These methods operate differently from retrieval. Fine-tuning changes the model itself. RAG supplies information at inference time. In practice, an application may use both — a model adapted for a particular task while retrieval provides the current information needed to perform it.

Steer behavior through prompts and context

The lightest-weight form of adaptation can happen without training at all. Instructions, examples and contextual information supplied with a model call will influence how a pretrained model interprets the task and structures its output.

For sufficiently capable models, this approach gives teams room to test and refine an application before undertaking model training. The trade-off is that behavior remains sensitive to the information placed in the context, the model selected and the way the task is framed.

Choose the approach around the requirement

These methods address different parts of an NLP system, which is why treating them as competing alternatives obscures real architectural decisions.

If the application needs semantic representations, an embedding model supplies them. If it needs relevant external knowledge, retrieval and grounding provide that information at inference time. When the model itself needs stronger specialized behavior or task performance, fine-tuning offers a way to modify it. Prompts and contextual examples provide another layer of control without changing model weights.

Production constraints narrow those choices. Latency and throughput influence model size and retrieval design, while evaluation requirements affect how much flexibility an application can tolerate. Governance shapes which models have access to which data, and operational complexity influences how many components a team wants to maintain.

As modern NLP systems grow more capable, in other words, the architecture needs to reflect the problem being solved rather than a standard recipe applied to every language task.

Natural language processing predates modern machine learning by decades. Across that history, the central problem has remained stubbornly consistent: human language carries meaning through context, convention and ambiguity, while computers require representations they can manipulate explicitly.

What changed was how much of that structure engineers had to specify themselves.

Rules-based NLP

Early NLP systems relied heavily on rules written by linguists and developers. Dictionaries, grammatical patterns and manually encoded relationships told software how to recognize or transform particular forms of language.

Within narrow domains, such systems could be highly precise, but scaling them was difficult. Real language varies across speakers, contexts and domains, while exceptions accumulate quickly.

Elements of rule-based processing remain useful, especially when a requirement is narrow and explicit. Regular expressions, dictionaries and deterministic parsing rules, for example, still appear in production pipelines where predictable behavior is preferable to statistical inference.

Statistical NLP

Beginning in the late 20th century, statistical NLP shifted more of the work from authored rules to patterns learned from collections of text. Representations reduced documents to measurable features based on the terms they contained. N-gram language models estimated the probability of words or characters from the sequences that preceded them. Combined with machine-learning algorithms, those representations supported classification, search, language modeling and other NLP tasks at much greater scale.

Linguistic preprocessing was often central to the pipeline. Stemming and lemmatization reduced words to common forms; part-of-speech tagging identified grammatical roles; dependency parsing represented syntactic relationships between words.

Several of these techniques are still used today, particularly when simplicity, interpretability or computational efficiency is valuable. Their limitations are clearest where meaning depends heavily on context.

Neural NLP

Neural networks introduced representations learned directly from data rather than engineered primarily as explicit text features. Models such as word2vec and GloVe represented words as dense vectors in which geometric relationships captured aspects of semantic similarity. Instead of treating doctor and physician as unrelated vocabulary items unless rules or counts connected them, learned embeddings could place them near one another in vector space.

Recurrent neural networks (RNNs), including long short-term memory (LSTM) networks, added stronger sequence modeling. By processing tokens in order and carrying information forward through the sequence, they supported applications such as translation, speech recognition and language generation.

Long sequences remained difficult. Information had to travel through successive recurrent steps, creating both computational constraints and problems retaining relationships between distant parts of a passage.

Transformers

Introduced in the 2017 paper Attention Is All You Need, the transformer replaced the step-by-step processing used by recurrent networks with attention, allowing the model to consider relationships among tokens across a sequence more directly. That change allowed training to be parallelized much more effectively and gave models a stronger mechanism for representing long-range context. Combined with increasing compute, larger data sets and advances in training methods, transformers enabled increasingly capable pretrained language models.

Models could now learn broad language representations first and then transfer those capabilities across many downstream tasks. BERT demonstrated the power of large-scale transformer pretraining for language understanding, while decoder-based models pushed the same general architecture toward increasingly capable generation. Later foundation models expanded the range of tasks one pretrained system could address through instructions and in-context examples.

For practitioners, that progression changed the usual starting point. Building an NLP application rarely requires creating a language model from the ground up. Pretrained models, embedding models and language services provide reusable capabilities, after which teams choose how much additional retrieval, prompting or adaptation the application needs.

Earlier techniques didn’t disappear along the way, however. Their role simply narrowed as learned models took over increasingly complex language tasks, while simpler methods retained advantages in workloads where their speed, transparency or predictability fits the requirement.

Transformer models have improved NLP performance dramatically, yet many of the field’s hardest problems originate in language itself. Words change meaning with context. Speakers leave information unstated. Different communities express the same idea differently, and several responses to a question may all be reasonable. Generative systems add another challenge because fluent output is easy to mistake for a reliable response.

Ambiguity and context

Consider a simple sentence: I saw the man with the telescope. Who has the telescope? The observer may have used it to see the man, or the man may be carrying it. Grammar permits both readings. Human readers resolve ambiguities like this using context and world knowledge, often without noticing the inference. NLP systems face the same underlying problem at enormous scale. Meanings shift with the surrounding sentence, earlier conversation, domain terminology and assumptions that may never be written down.

Transformers improved contextual modeling substantially, especially compared with representations that treat words largely in isolation. Even so, additional context doesn’t always settle the interpretation. Sometimes the source itself is genuinely ambiguous.

For production applications, this challenge affects system design. A model shouldn’t manufacture certainty where the available information supports several plausible readings.

Multilingual and domain variation

Training resources vary enormously across languages, dialects and specialized domains. A model with strong performance on general English text may struggle with clinical shorthand, legal terminology, technical documentation or a language represented sparsely in its training data.

Even within one organization, terminology evolves. Product names change, abbreviations acquire local meanings and individual teams develop their own vocabulary. Systems built over enterprise language therefore need evaluation against the language they will encounter in practice rather than assumptions derived from broad benchmarks.

Evaluation

For tasks with a single well-defined target, NLP evaluation can be relatively straightforward. For example, a classification model either assigned the correct label or it didn’t.

Open-ended language tasks complicate that picture. Two summaries might differ substantially, but both preserve the important information, while a question-answering system might produce a factually correct response that nevertheless leaves out the information the user actually needed.

Consequently, evaluation typically spans several dimensions: factual accuracy, relevance, completeness, instruction following and task-specific requirements. Automated metrics help at scale, but human judgment or model-based evaluation frequently enters the process where output quality cannot be reduced to one exact answer.

The evaluation target also extends beyond the model. In a retrieval-based application, teams need to know whether the right source was retrieved, whether the model interpreted it correctly and whether the final response met the application's requirements.

Hallucination and grounding

Generative language models predict plausible continuations from the information available to them. Plausibility and factual support are very different standards, however.

The result may be a confident but unsupported answer, an incorrect combination of facts or plausible-sounding information that fills gaps in the available evidence. In enterprise applications, where users often expect answers tied to authoritative internal data, those errors are particularly consequential.

Grounding gives the model access to relevant external information at inference time and provides a basis for evaluating its response against source material. Guardrails, source attribution and application-specific evaluation add further controls.

None substitutes for testing the complete system under realistic conditions. A grounded model could still misread a retrieved passage, rely on the wrong source or generate a conclusion the evidence doesn’t support.

Bias and fairness

Language data reflects the people and institutions that produced it. Historical stereotypes, uneven representation and differences in how groups are discussed can impact models trained on this data. Aggregate benchmark performance alone may conceal substantial differences across demographic groups, languages or categories of input.

The effect varies by application. Bias in a brainstorming tool presents a different risk from bias in a system used to classify customer requests, evaluate documents or support decisions about people.

Learn how data bias enters AI systems and how to reduce it >

Privacy and governance

Enterprise NLP systems frequently operate on some of an organization’s most sensitive information: emails, contracts, support transcripts, internal documents, financial records and other unstructured text.

Once models, embedding services and retrieval layers enter the architecture, access to that information has to follow the same governance requirements that apply elsewhere in the data environment. Which users can retrieve a document? Which model can process it? Where are embeddings stored? What information is logged during inference?

Those questions sit directly inside the NLP architecture. A system that retrieves semantically relevant information without preserving the source system’s access controls has improved search while creating a security problem.

For enterprise NLP, model capability is only one part of the system. The model also needs access to the right documents, tables and operational data — with the permissions and governance attached to them.

This requirement shapes the architecture around retrieval and inference. Semantic search may surface relevant unstructured content, while structured data supplies business context that doesn’t appear in the text itself. Access controls still need to hold across both: a conversational interface shouldn’t expose information a user couldn’t retrieve through the underlying system.

Where those operations run also affects data movement. Generating embeddings, storing vectors, retrieving source material and sending it to separate model services can introduce additional copies and governance surfaces. Keeping retrieval, model processing and governed data closer together reduces those handoffs.

Snowflake ML supports NLP workflows across model inference, embeddings, retrieval and natural language interaction, allowing teams to work with structured and unstructured data in the same governed environment.

Cortex AI provides access to language and embedding models for tasks such as summarization, extraction, classification and generation. For semantic retrieval, teams can generate vector embeddings from text and store them alongside the source data and associated metadata.

Cortex Search adds a managed retrieval layer for semantic search and RAG applications. It combines vector retrieval with keyword search, helping preserve exact matches for names, codes and specialized terminology while also retrieving conceptually related content.

For applications that need to query structured data through natural language, Cortex Analyst translates user questions into SQL using semantic models that provide business context around the underlying data. That gives NLP applications access to information that may live in tables rather than documents.

Together, these capabilities support a broader pattern: models, retrieval and governed enterprise data operating within the same platform rather than across a collection of disconnected services. Teams can use language models where they fit, specialized retrieval where it’s needed and existing Snowflake governance controls across the underlying data.