What is BERT? NLP Model Explained

BERT’s ability to analyze context bidirectionally is considered a major milestone in the evolution of AI and NLP. This allowed it to achieve record-breaking results across eleven NLP tasks, including question answering, sentiment analysis and named entity recognition (automatically categorizing whether a word represents a person, product, organization or other entity). The transformer architecture used by BERT has become the foundation for virtually all modern LLMs, due to its ability to capture relationships between words across long stretches of text.

BERT's training and inference involve several sophisticated mechanisms working together: 

Tokenization

BERT breaks text into smaller pieces called tokens. For example, the word “playing” might split into “play” and “##ing.” Each token gets converted to a number, and BERT adds special markers like [CLS] at the start of sentences and [SEP] between them. This approach increases accuracy with less commonly used words and makes the size of its vocabulary more manageable.

Input embeddings

Each token receives three types of embeddings: token (what the word is), position (where it appears in a sequence) and segment (which sentence it belongs to). This gives BERT useful information about the content and structure of text. 

Attention mechanisms 

BERT uses attention mechanisms to calculate how much each word should consider every other word in a sentence. For example, when BERT processes the word “bank,” it assigns attention scores to all other words in that sentence. If “river” and “water” appear, they receive high scores, indicating that “bank” probably refers to a riverbank. If “money” and “deposit” score higher, BERT understands “bank” means a financial institution. 

Transformer encoder layers

BERT processes text through multiple stacked layers, with each layer running multiple attention calculations in parallel. Each layer captures progressively more complex patterns. Early layers might learn basic grammar, while deeper layers understand abstract relationships and semantics.

Pretraining tasks 

As part of the pretraining process, BERT randomly masks 15% of tokens and attempts to predict what they are. This aids in bidirectional understanding. It also analyzes pairs of sentences and predicts whether the second sentence comes before or after the first in the original text. This technique helps it understand the relationship between sentences.

Fine-tuning and inference

Once pretraining has been completed, developers can add a task-specific layer on top and train BERT to perform that task, such as sentiment analysis or spam detection. During inference, text flows through all the attention layers to build contextual understanding, and BERT outputs predictions based on those rich representations.

Since its introduction in 2018, BERT has been deployed across a wide range of practical use cases. These include:

Google search 

BERT powers Google's search ranking to better understand the context and intent behind complex queries, especially longer conversational searches where word order and prepositions matter. 

Virtual assistants 

BERT improves intent recognition in voice assistants like Google Assistant and Alexa, helping them understand what users actually want. It also enables more accurate responses to follow-up questions by maintaining context across a conversation.

Healthcare 

By analyzing clinical notes and medical records, BERT can extract relevant patient information, identify diagnoses and flag potential drug interactions or contradictions in treatment plans. 

Legal tech 

BERT powers contract analysis tools that identify key clauses, obligations and risks across thousands of legal documents. It enables semantic search through case law, helping lawyers find relevant precedents even when different terminology is used.

Ecommerce 

By understanding customer intent, BERT makes it easier for chatbots to respond accurately to customer service inquiries and can classify product reviews by sentiment.

Social media 

BERT helps moderate content by detecting hate speech, harassment and misinformation with better contextual understanding than keyword-based approaches. It powers social media recommendation systems that suggest relevant connections, groups or content to users.

As originally designed, BERT suffers from a handful of limitations. The primary ones include:

High computational cost

BERT requires substantial computing power for both training and inference, making it expensive and slow for real-time applications, especially on resource-constrained devices. 

Limited input length

BERT can only process sequences up to 512 tokens in length, which is problematic for long documents like legal contracts or research papers that need to be understood as a whole. 

Inability to generate text

Because it was built purely as an encoder for understanding text, BERT is unable to generate coherent responses or create new content. GPT models and later encoder-decoder architectures specifically designed to handle both understanding and generation are suitable for tasks like summarization and translation.

Sensitivity to hyperparameters

Model performance can vary significantly based on settings like learning rate, batch size and the number of times BERT takes to complete a pass through a training dataset. Extensive fine-tuning may be required.

Challenges in multilingual performance

Multilingual BERT was trained on 104 languages simultaneously, which meant each language got less attention and performance suffered compared to language-specific models. Newer models train on much larger multilingual datasets with better sampling strategies or use cross-lingual transfer learning to improve language performance.

BERT has spurred the creation of other more advanced language models. Some of the leading ones include:

GPT 

GPT uses unidirectional (left-to-right) processing and is trained to predict the next word in a sequence, making it naturally suited for generating coherent text like conversations and creative writing. Unlike BERT, it can only see previous context when understanding a word, not what comes after.

RoBERTa 

Robustly Optimized BERT Pretraining Approach (RoBERTa) employs the same bidirectional architecture as BERT but trains on 10 times more data. It uses improved techniques like dynamic masking, changing which words get masked each time the model is trained on the same sentence. As a result, RoBERTa achieves significantly better performance without changing BERT’s fundamental approach.

XLNet 

XLNet achieves bidirectional understanding like BERT but uses permutation language modeling, predicting words in random order instead of masking them. It's often more accurate than BERT but is more computationally complex and harder to train.