Image Classification: How It Works, Models and Production Considerations

Two image classifiers with similar accuracy can behave very differently in production. Why?

Overall accuracy compresses a great deal of behavior into one number. Two models can score similarly while differing in rare-class recall, calibration, robustness to changed image conditions, inference latency and GPU cost. Those differences emerge from the architecture, training data, evaluation design and serving environment around the classifier.

Building a production classifier requires teams to account for all of those trade-offs across the full workflow: how the model learns visual features, which architecture fits the task, where performance can degrade and how training and inference will run at the required scale.

Image classification is a computer vision process that uses a computational model to estimate which category or combination of categories best describes the visual content of an input image. Organizations use it for tasks such as identifying signs of disease in a medical scan, flagging images that may violate a content policy, identifying crop diseases from photographs of plant leaves, and assigning product categories or attributes to retail images.

The output depends on the classification setup:

• Binary classification selects between two classes, such as defective or acceptable.
• Multiclass classification assigns one label from several mutually exclusive options.
• Multilabel classification assigns several labels to the same image.
• Hierarchical classification organizes predictions at hierarchical levels.

For example, a quality-control system might classify a part as either defective or acceptable in a binary task. A multiclass system might assign the part one defect type, such as a crack, dent or discoloration. A multilabel system could assign both “crack” and “discoloration” when the same part contains more than one defect. A hierarchical system might first classify the issue as a surface defect or a structural defect, then assign a more specific label such as discoloration, dent or crack.

Most modern image classifiers use deep learning to learn visual features from labeled data. During supervised learning, each training image is paired with its expected class label, and the model adjusts its parameters whenever its prediction differs from the correct answer.

Learn how document AI company LandingAI helps companies use images, documents and video to support visual AI applications in manufacturing, healthcare and other industries

An image classifier converts pixel data into an internal representation, then maps that representation to one or more class scores. The way it extracts visual features depends on the model architecture.

Two architecture families dominate modern image classification:

• Convolutional neural networks (CNNs) learn spatial patterns by applying filters across local image regions.
• Vision transformers divide an image into patches and use attention to model relationships among them.

Their feature-extraction mechanisms differ, but both ultimately produce a representation that a classification head maps to the target classes. Both can also be trained through supervised learning, adapted through transfer learning and evaluated against the same production requirements.

Feature extraction with convolutional neural networks

A CNN applies learned filters, also called kernels, across small regions of an image. Each filter responds to a particular visual pattern and produces a feature map showing where that pattern appears.

Earlier CNN layers often respond to local patterns such as edges, color transitions and textures. Deeper layers combine information across larger parts of the image and can capture shapes, object parts and class-related patterns. These features are distributed across the network rather than forming a strict, easily interpreted hierarchy.

As the representation moves through the network, many CNNs reduce its spatial dimensions. Pooling layers can perform this downsampling by selecting the maximum value or calculating an average within a local region, while newer architectures often use strided convolution. Reducing resolution lowers the computational load in later layers and increases the portion of the image represented by each feature.

The downsampling pattern has to fit the task. If the classes differ through small defects, markings or anatomical details, reducing the image too aggressively can remove information the classifier needs. For example, a classifier inspecting circuit boards may need to detect a hairline crack that covers only a few pixels. If the network reduces the image resolution too early, that crack may disappear from the representation even though the rest of the board remains visible.

Feature extraction with vision transformers

The original Vision Transformer (ViT) divides an image into nonoverlapping patches, converts them into embeddings and uses attention to model the relationships among them. Newer transformer architectures may also use local attention, hierarchical representations or convolutional components.

Attention allows the model to weigh information from different parts of the image when constructing its representation. A patch containing one feature can therefore be interpreted in relation to patches elsewhere in the frame, without requiring the information to pass through a long sequence of local convolutional operations. For example, when classifying a bird, the model can relate a patch containing the beak to patches containing the wings, body and surrounding habitat. No single patch will identify the species on its own, but the relationships among them can support the prediction.

Vision transformers often benefit from large-scale pretraining because they encode fewer assumptions about local image structure than CNNs. With sufficient pretraining, however, they can adapt effectively to a broad range of classification tasks. Hybrid architectures also exist, combining convolutional layers with transformer-based attention.

Mapping the representation to class scores

After feature extraction, a classification head maps the model’s internal representation to the target classes. In a CNN, the head may use global average pooling followed by one or more fully connected layers. A vision transformer may use a dedicated classification token or a pooled representation of the patch embeddings.

For multiclass classification, the model commonly uses softmax to turn its output scores into values that sum to one. These values are often treated as probabilities, but they may not reflect the model’s true level of confidence unless the model has been calibrated.

Multilabel classification uses independent outputs because several classes can apply to the same image. A sigmoid function commonly converts each score into a separate value for each label. The system then needs a threshold for deciding when each label applies. Teams can use one threshold across all classes or tune separate thresholds by class based on validation results and the cost of different errors.

Training the classifier

During supervised training, the model receives batches of images paired with their expected labels. A forward pass produces class scores, and a loss function measures the difference between those predictions and the correct answers. Cross-entropy loss is commonly used for multiclass classification, while multilabel tasks typically use a binary cross-entropy variant.

Backpropagation calculates how the loss changes with respect to each trainable parameter. The optimizer uses those gradients to update the model’s weights, after which the model processes another batch. Repeating this sequence allows the feature extractor and classification head to learn together.

A validation set contains images that don’t contribute to weight updates. Comparing training and validation results helps teams identify overfitting, tune hyperparameters and select a model checkpoint.

The test set should remain separate until the team has selected the architecture, hyperparameters, thresholds and checkpoint. Repeatedly using test results to make those decisions effectively turns the test set into another validation set.

The same basic training loop applies to CNNs and vision transformers, although the two architectures can differ in data requirements, memory use and optimization behavior.

Adapting a pretrained model

Most production projects begin with a pretrained model rather than randomly initialized weights. Through transfer learning, teams take a CNN or vision transformer trained on a large image collection and adapt it to a narrower set of domain-specific classes.

The pretrained model already contains broadly useful visual representations. Fine-tuning updates some or all of its weights using the target images, while the classification head is configured for the new label set. For example, a model pretrained on a broad image collection may respond to edges, textures and common shapes. A manufacturer can fine-tune those representations using a smaller collection of labeled product images rather than teaching the model every visual pattern from the beginning.

When the pretrained model’s visual representations transfer well to the target domain, fine-tuning usually requires less labeled data and compute than training a model from randomly initialized weights.

Evaluating the model

For binary and multiclass tasks, accuracy usually measures how often the model selects the correct class. Multilabel tasks are more complicated because each image can have several correct labels. A prediction might get some labels right and others wrong, so teams typically evaluate precision, recall and F1 score for each label or across the full data set.

The evaluation set should reflect the images, class frequencies and operating conditions expected in production. Depending on the workload, teams may also measure per-class recall, calibration, robustness, inference latency, throughput and GPU or memory use.

After defining the classification task, teams must decide how the model will learn and which specific architecture fits the workload. Those choices affect the amount of labeled data required, the compute used during training and inference, and the level of control teams have over the classifier.

Learning from labeled and unlabeled images

Image classifiers can learn through several setups:

• Supervised learning trains directly on image-label pairs. Its objective aligns closely with the eventual task, although assembling accurate labels can require considerable time and domain expertise.
• Unsupervised learning works without predefined class labels. Clustering can group images based on similarities in their representations, helping teams explore a collection, find recurring patterns or organize data before annotation. Those groups may not match the categories the final application needs, however.
• Self-supervised learning derives a training signal from the images themselves. Depending on the method, a model might learn to match two augmented views of the same image, predict missing image regions or associate images with accompanying text. Teams can later fine-tune the resulting representations with a smaller labeled data set.
• Zero-shot classification uses a pretrained vision-language or multimodal model to assign natural-language classes without task-specific training. It’s useful for exploration and rapidly changing taxonomies, but may be less consistent than a domain-specific classifier when categories depend on subtle visual differences or specialized terminology.

Common CNN architectures

CNN architectures use the convolutional feature-extraction process described earlier, but their designs emphasize different trade-offs:

• ResNet uses residual connections that allow information and gradients to bypass one or more layers. These shortcut paths make very deep CNNs easier to train and reduce the degradation problems seen in earlier deep networks.
• EfficientNet scales network depth, width and input resolution together through a compound scaling method. The design seeks a more balanced use of compute than increasing only one dimension of the network.
• MobileNet uses lightweight convolution operations designed for environments with constrained memory and processing capacity. It’s commonly considered for mobile, embedded and edge inference.

Common vision transformer architectures

The original ViT applies a standard transformer encoder to a sequence of image patches. Its larger variants can support high-capacity classification workloads, particularly when substantial pretrained weights and centralized compute are available.

Other transformer designs modify how patches are created or how attention is calculated. Hierarchical models such as Swin Transformer progressively combine nearby patches and restrict some attention operations to local windows, improving efficiency while preserving the transformer architecture’s ability to model broader relationships.

Architecture selection depends on the task’s visual complexity, available training data, latency target, deployment hardware and accuracy requirements. A compact MobileNet may fit an inspection device that must return predictions locally, while a larger ResNet or vision transformer may suit centralized processing where additional compute produces a meaningful performance gain.

A production classifier encounters variation that a static benchmark can’t fully reproduce. Lighting changes, new equipment, different image compression or a shift in the underlying population can alter the input distribution, sometimes without changing the business definition of the task.

Training data quality and coverage

Training a deep classifier from random initialization typically requires a large volume of accurately labeled examples. Transfer learning and large-scale pretraining can reduce that requirement, but teams still need representative target-domain data for fine-tuning and evaluation. Labeling can be particularly demanding when annotations require a radiologist, engineer or another subject-matter expert.

Class imbalance creates another problem. When a large percentage of training images belong to one category, a model can achieve high overall accuracy while performing poorly on the rarer class. Teams may address the imbalance through targeted data collection, sampling strategies, class-weighted loss functions or augmentation, depending on the source and severity of the gap.

Additionally, incorrect labels introduce conflicting training signals. An isolated error may have little effect in a large data set, while systematic ambiguity — two reviewers applying a category differently, for example — can prevent the model from learning a stable decision boundary. A documented annotation policy and reviewer agreement checks help expose those inconsistencies before training.

Overfitting and generalization

A high-capacity model can memorize details from its training images, including patterns that don’t hold outside that collection. Training accuracy might continue to rise, but validation performance will stall or decline.

Image augmentation exposes the classifier to altered versions of the training images through operations such as cropping, rotation, color adjustment or noise injection. Appropriate transformations depend on the domain: A horizontal flip may preserve the label for a product photograph yet change the meaning of a medical or scientific image.

Regularization, weight decay, dropout, early stopping and careful validation can also reduce overfitting. When the available data remains limited, transfer learning narrows the amount of task-specific information the model must learn from scratch.

Compute requirements

Training deep CNNs and vision transformers involves repeated matrix operations across large image tensors. Higher image resolutions, larger batches and deeper networks increase GPU memory use and training time, while hyperparameter searches multiply the number of runs.

Inference introduces another scaling consideration. A production workload may need to handle thousands or millions of images within a defined time window. Meeting that throughput can require batching, parallel workers, GPU acceleration or a dedicated serving layer.

For latency-sensitive applications, the model’s forward pass is only part of the response time. The system must also retrieve and decode the image, apply the required resizing and normalization, transfer the resulting tensor to the model and return or store the prediction. A fast model can miss its latency target if the surrounding input pipeline can’t keep pace.

Bias, domain shift and interpretability

Image classifiers can produce systematically different error rates across classes, demographic groups, devices or operating environments. Those disparities often reflect the training data: Some groups or conditions might be underrepresented, labels might contain systematic inconsistencies, or acquisition methods might encode patterns that correlate with the target class without generalizing beyond the development set.

Domain shift is another issue that can affect accuracy. It occurs when the distribution of production images differs from the distributions used for training and validation. A manufacturing classifier trained at one facility may encounter different lighting, camera geometry or materials at another; a medical model may receive scans from devices, institutions or patient populations that were sparsely represented during development. The target classes remain the same, but the statistical properties of the inputs change.

Robustness tests based on synthetic corruptions can’t fully capture that variation. Validation needs to include images drawn from the devices, locations, populations and operating conditions expected after deployment.

Interpretability methods address a different question: which image regions or learned features influenced a particular prediction. Techniques such as saliency maps and class activation maps can help teams investigate spurious correlations or unexpected model behavior, but they don’t establish that the model is unbiased or robust.

For this reason, higher-consequence workflows typically combine interpretability with subgroup analysis, per-class metrics, domain-specific test sets, confidence thresholds and monitoring for changes in input and outcome distributions.

Snowflake supports two approaches to image classification: training a custom CNN or vision transformer with labeled data, or using Cortex AISQL for zero-shot classification with a multimodal model. The right choice depends on the available labels, accuracy requirements and need for control.

Train a custom classifier

Snowflake Notebooks on Container Runtime provide CPU or GPU environments for building image-classification workflows with frameworks such as PyTorch and TensorFlow. Teams can access image files and metadata in Snowflake, fine-tune a pretrained model, evaluate it and register the selected model in the Snowflake Model Registry.

A typical workflow is to:

1. Load and preprocess labeled images.
2. Split the data into training, validation and test sets.
3. Fine-tune a pretrained CNN or vision transformer.
4. Compare models using overall and per-class metrics.
5. Register and deploy the selected model for batch or real-time inference.

Container Runtime also supports distributed data loading and training for larger workloads. Performance depends on the model, data volume and compute configuration, and teams should confirm the availability of preview capabilities before designing around them.

This approach is best suited to stable classes, representative labeled data and applications that require control over architecture, thresholds and retraining.

Classify images with Cortex AISQL

When labeled data is unavailable, Cortex AISQL can use multimodal models to classify images stored in Snowflake. With AI_COMPLETE, teams can define a set of categories, describe the classification criteria and request a label or structured response for each image.

This approach is useful for exploration, candidate-label generation and frequently changing taxonomies. Because the model isn’t fine-tuned on the organization’s examples, teams should validate its output on representative images before using it in consequential workflows.

Choose the right approach

Use a custom classifier when classes are stable, labeled data is available and consistent performance is required. Use zero-shot classification when the task is exploratory, labels are limited or categories change frequently. Both approaches can keep image references, metadata, labels, evaluation results and model outputs within Snowflake’s governed environment.

An image classifier’s production behavior reflects the full workflow used to create and run it. The training images and labels shape what the model learns; validation data and class-level metrics reveal where that learning holds; and the inference pipeline determines whether predictions arrive at the required volume, latency and cost. Keeping those parts connected makes the system easier to evaluate, reproduce and update as the image distribution changes.