TOPS (Tera Operations Per Second, trillions of operations per second) is the unit used to measure the raw performance of AI accelerators, especially NPUs. A "40 TOPS" chip can execute 40 trillion elementary arithmetic operations —multiplications and additions, the raw material of neural networks— every second. It is to AI hardware what horsepower is to a car: the marketing figure that summarizes, imperfectly, how much power is under the hood.
How it is measured (and why almost always in INT8)
TOPS depend entirely on the numerical precision used for calculation. Modern inference works with models quantized to 8-bit integers (INT8), so that is the figure manufacturers announce; the same silicon performs approximately double in INT4 and half in FP16:
| Precision | Typical use | Relative performance |
|---|---|---|
| INT4 | Highly quantized LLMs, maximum speed | ~2× the INT8 figure |
| INT8 | standard quantized inference (the announced figure) | 1× |
| FP16 | unquantized models, light training | ~0.5× |
| FP32 | full precision, almost never in NPUs | ~0.25× |
Comparing TOPS from different manufacturers without looking at precision is like comparing apples to oranges. And watch out for its relative: TFLOPS measures floating-point operations (the domain of GPUs and training); TOPS, integer operations (the domain of NPUs and inference). They are not interchangeable.
Theoretical peak vs. real performance
TOPS are a laboratory maximum: they assume that calculation units never wait for data. In practice, the limiting factor is usually memory bandwidth — 200 TOPS are useless if the model weights do not arrive in time. With LLMs this is decisive: a chip with fewer TOPS but plenty of fast, well-connected memory can smoothly run a model that chokes another nominally superior one. That is why the Tiiny AI Pocket Lab boasts as much about its 80 GB of LPDDR5X as its ~190 TOPS, and why the Qualcomm AI250 bets on near-memory computing with 10× effective bandwidth.
TOPS per watt: the metric that separates generations
Energy efficiency is what has allowed AI to move from the datacenter to the pocket. A Hailo-10H delivers ~13 TOPS per watt; an Axelera Metis, ~24; a desktop GPU from a few years ago, less than 1. For a company, TOPS/watt translates directly into operating cost: more inference for every euro of the electricity bill. It is the metric upon which NAiOS Labs is designing its NPU rack unit for its own datacenters.
2026 References: how many TOPS each thing has
| Hardware | TOPS (INT8) | Power Consumption | Class |
|---|---|---|---|
| High-end mobile | 35–45 | <5 W | NPU integrated into the SoC |
| "AI PC" Laptop (Copilot+ requires 40) | 40–50 | ~10 W | Integrated NPU |
| Hailo-10H | 40 | ~3 W | M.2/USB edge module |
| Tiiny AI Pocket Lab | ~190 | ~30 W (TDP) | Pocket mini-PC |
| Axelera Metis | 214 | 3.5–9 W | M.2/PCIe accelerator |
| NVIDIA Jetson AGX Orin | 275 | 15–60 W | Embedded module |
| Qualcomm AI200/AI250 | rack scale | up to 160 kW/rack | Datacenter |
How many TOPS does your use case need?
- Voice transcription and basic vision (cameras, OCR): 5–15 TOPS are enough; this is the domain of integrated NPUs and edge modules.
- Assistant with small LLM (quantized 7B), document RAG: 40+ TOPS and, above all, 8–16 GB of memory for the model.
- Industrial vision at line speed: 100–200 TOPS with guaranteed latency (Metis, Jetson).
- Medium