Inference
Also: Model Inference, AI Inference, Inferencing, Inference serving, Inférence, Inférence de modèle, Modell-Inferenz
The moment a trained AI model is put to work: it takes a new input and produces an answer, prediction or generated output.
What It Is
Inference is the running phase of an AI model. Training is when the model learns from data; inference is when it applies what it learned to something new. Every time a user types a question into a chatbot and gets an answer, that single request-and-response cycle is one inference. A CMO sees inference when a recommendation engine scores a visitor and picks which offer to show. A CFO sees it when a fraud model flags a transaction in real time. The model is no longer changing, it is producing output.
Why it matters
Inference is where AI meets your P&L, because it is the part you pay for repeatedly. Training happens occasionally, but inference runs every time the product is used, and the cost scales with volume. A model that is cheap to build can still be expensive to operate if millions of inferences run each day. For leaders this reframes the budget conversation: the real question is not only "what does it cost to build" but "what does each answer cost, times how many answers per month." Inference also drives customer experience directly. Slow inference means a laggy assistant; unreliable inference means wrong recommendations reaching customers. Latency, cost per call and accuracy at inference time are the metrics that decide whether an AI feature ships or stays in the lab.
How it works
At inference time the trained model sits behind an API or embedded in an application. A request arrives (a prompt, an image, a set of customer attributes), the model processes it and returns a result, usually within milliseconds to seconds. The heavier the model, the more computing power each inference consumes, which is why teams trade off model size against speed and cost. Techniques such as model distillation and smaller context windows exist largely to make inference cheaper and faster. As a leader you will meet inference in vendor pricing (often billed per token or per call), in latency service levels, and in capacity planning when a feature suddenly gets popular.
Frequently asked questions
What is inference in AI?
Inference is the running phase of an AI model: it takes a new input and produces an answer, prediction or generated output. Training is when the model learns from data; inference is when it applies what it learned to something it has never seen. Every time a user types a question into a chatbot and gets an answer, that single request-and-response cycle is one inference.
What is the difference between training and inference?
Training is the learning phase, inference is the running phase. During training the model changes as it absorbs data; during inference the model is fixed and simply produces output. The cost structure differs too: training happens occasionally, while inference runs every single time the product is used.
Why does inference matter for a budget, not just for engineers?
Because inference is the part of AI you pay for repeatedly, and its cost scales with usage volume. A model that is cheap to build can still be expensive to operate if millions of inferences run each day. The budget question becomes: what does each answer cost, multiplied by how many answers per month.
Which inference metrics should a leader track before shipping an AI feature?
Latency, cost per call and accuracy at inference time. These three decide whether an AI feature goes live or stays in the lab: slow inference produces a laggy assistant, unreliable inference pushes wrong recommendations to customers, and cost per call determines whether the feature survives at scale.
Why do teams use smaller models or shorter context windows?
Mainly to make inference cheaper and faster. The heavier the model, the more computing power each inference consumes, so teams trade model size against speed and cost. Techniques such as model distillation and reduced context windows exist largely for that reason.