Inference parameters are settings you send with each request to shape how a foundation model writes its answer, without retraining or changing the model. Some control how predictable or varied the wording is; others control how long the answer can run and where it stops. Exam scenarios describe a symptom, such as answers that are too repetitive, too random or too long, and ask which setting to change and in which direction.
How a Model Picks Words
A language model writes one token at a time. For each next token, it scores every possible option with a probability, then picks from those options. Inference parameters change that picking step. They either reshape the probabilities or shrink the pool of candidates the model is allowed to choose from. Default values and allowed ranges vary from model to model.
Temperature
Temperature changes how strongly the model favors its most likely options.
- Lower temperature makes the top choices even more dominant, so answers become more predictable and consistent. Good for factual answers, extraction and anything that must be repeatable.
- Higher temperature flattens the probabilities, so less likely words get picked more often. Answers become more varied and creative, but also more prone to wander. Good for brainstorming, slogans and stories.
Temperature never makes a model know more facts. A low setting makes it consistent, not correct.
Top K and Top P
Both settings limit which candidate tokens the model may pick from.
- Top K keeps a fixed number of the most likely candidates. With Top K set to 50, the model chooses among the 50 most probable next tokens, whatever their probabilities.
- Top P keeps the smallest group of top candidates whose probabilities add up to a set share. With Top P at 0.9, the model chooses from the candidates that together make up 90% of the probability. When the model is confident, that group may be one or two tokens; when it's unsure, the group grows.
Lower values of either make output more focused. Higher values allow more unusual choices.
Length and Stopping
- Maximum length (often called max tokens) caps how many tokens the answer can contain. It controls cost and response time, because output tokens are billed and take time to generate. Set it too low and answers get cut off mid-sentence.
- Stop sequences are strings that end generation the moment the model produces them. They're useful when the model keeps writing past the point you need, for example adding extra made-up questions after answering yours.
- Penalties, offered by some models, discourage repeated words or phrases.
| Symptom | Change |
|---|---|
| Answers vary when they should be identical | Lower temperature |
| Creative output feels repetitive | Raise temperature or Top P |
| Answers run long and cost too much | Lower maximum length |
| Model keeps writing past the useful part | Add a stop sequence |
On the Exam
Match the symptom to the parameter. Consistency problems point to temperature. Length and cost problems point to maximum length. Runaway output points to stop sequences. If an option says a parameter makes the model "more accurate" or "know more", it's a trap: parameters change style and length, not knowledge.