Once a model is trained, it earns its keep through inference: making predictions on new data. Amazon SageMaker AI offers four ways to run inference: real-time, serverless, asynchronous and batch. This lesson shows how to pick one from three clues in a scenario: how fast an answer is needed, how big each request is, and how steady the traffic is.
Training and inference
Training is when a model learns from historical data. It happens occasionally and can take hours. Inference is when the trained model is used, for example scoring a payment for fraud or tagging a photo. Inference happens far more often than training, so how you run it has a big effect on cost and user experience. The four options below differ in whether a server is always running, how long a request can take, and how results come back.
Real-time and serverless inference
Real-time inference keeps a model on a persistent endpoint, which is always running and ready to answer. Each request gets a response within milliseconds to seconds. It suits steady traffic where a person or system is waiting, such as checking a payment at checkout. You pay for the instances while they run, even when quiet.
Serverless inference also answers requests as they arrive, but SageMaker AI manages capacity for you and scales it down when there's no traffic, so you don't pay for idle time. The trade-off is a possible delay, called a cold start, when a request arrives after a quiet spell. It suits light, intermittent or unpredictable traffic with small requests.
Asynchronous and batch inference
Asynchronous inference puts requests in a queue and processes them in the background. The result is written to storage, and the caller can be notified when it's ready. It handles large inputs, such as long audio or video files, and jobs that take minutes rather than seconds. It can scale down to zero instances between bursts.
Batch inference, called batch transform in SageMaker AI, scores a whole dataset in one go, with no endpoint left running afterwards. It suits offline work where all the data is available up front and nobody is waiting on any single answer, such as scoring every customer once a month.
Choosing an option
| If the scenario says... | Choose |
|---|---|
| Someone is waiting; traffic is steady; low latency matters | Real-time |
| Traffic is light or unpredictable; pay nothing when idle; small requests | Serverless |
| Large files or long processing; requests arrive over time; queue them | Asynchronous |
| A whole dataset at once; results not needed immediately; no endpoint | Batch transform |
On the exam
The trap is usually between two options that both sound reasonable. Serverless and asynchronous can both scale to zero, but serverless is for small, quick requests, and asynchronous is for big payloads or long jobs. Batch and asynchronous both run in the background, but batch processes a dataset you already have all at once, while asynchronous handles individual requests as they trickle in.