inference time compute (or inference-time compute) refers to the strategy of allocating more computational resources at the moment the model generates the response, instead of investing them solely in the training phase. The underlying idea is that a model can "think more" for a specific query, dedicating additional cycles to explore, verify, or refine its output before delivering it.
This approach matters because it allows for improving the quality of responses without the need to retrain the model or increase its size. In complex reasoning tasks —mathematics, logic, or programming— it is often more beneficial to let the model reason for longer than to scale its parameters. Some common techniques include:
- Generating several solutions and choosing the best one by voting (self-consistency).
- Deploying extensive step-by-step reasoning chains.
- Using verification and self-correction loops.
The practical nuance is that this extra compute has a real cost: higher latency and increased expense per query, so it is advisable to reserve it for problems that truly justify it.