The definition
Tokens per watt takes a serving system's throughput, in tokens per second, and divides it by the power drawn while producing them, in watts.
A watt is one joule per second, so the ratio is also tokens per joule. Its inverse is energy per token, usually quoted in joules. The two are one measurement written two ways, so a change in one is not a second result.
The watts were read somewhere in the electrical chain, and that place is part of the number. Watts read at the accelerator die give a higher ratio than watts read at the rack or the meter, for the same tokens.
Why it is derived, not measured
No instrument reads tokens per watt directly. One counter records tokens and another instrument reads power, and the ratio is arithmetic on the two. The two readings also have to cover the same accelerators over the same window, or the ratio divides one period's work by another period's power.
That makes it a derived figure, as firm as the weaker of its two readings. Taken on the same accelerators over the same window, both readings describe the same work. A tokens-per-watt figure should say it is derived, next to the number.
Two ways the ratio rises
The ratio rises when the same watts produce more tokens, or when the same tokens take fewer watts. On a fleet serving a fixed stream of requests, the second is what a power reduction with throughput held looks like.
A worked example with round numbers: a server producing 1,000 tokens a second while drawing 500 W makes 2 tokens per joule. Serve the same 1,000 tokens a second at 400 W and it makes 2.5 tokens per joule. The ratio rose by a quarter on the same 1,000 tokens a second, and 100 W is free for more work.
On a fixed fleet a higher ratio means the same tokens for less energy. Put the power released to work, with more accelerators inside the same power envelope or more requests on the capacity freed, and it becomes more tokens.
Batch size, model, numerical precision and serving settings all move the ratio. So does load. A fleet waiting between bursts of requests still draws power while producing few tokens, and that idle time pulls the average down.
What a tokens-per-watt claim has to carry
Before comparing two figures, check that each states the following alongside the number. Where the watts were read: die, board, rack or meter. The hardware, named. The window of time. The workload and serving stack, including the model.
It should also name the operating mode or settings. One system at two settings gives two figures, and two figures are not a range.
Then check the same run for throughput and latency. A power reduction bought with lost work can also raise tokens per watt on paper. Tokens per second and tail latency, held on the same run, are what rule that out.
Questions people ask
- What is tokens per watt?
- Throughput in tokens per second divided by power in watts, which equals tokens per joule. It measures how much inference output a system produces for the energy it uses.
- Is energy per token the same measure?
- Yes, written the other way up. Energy per token is joules divided by tokens, the inverse of tokens per joule, so a change in one is the same result as a change in the other.
- Does a higher tokens per watt mean more tokens?
- On a fixed fleet it means the same tokens for less energy. Use the power released for more accelerators or more requests, and it means more tokens too.
Where ATHLAZ fits
ADAPT changes the watts, not the work
ADAPT, AI-Driven Adaptive Power Technology, is a per-GPU software runtime that sets how each accelerator draws power while inference runs.
Measured: up to 21% less GPU die power on NVIDIA H100 NVL, over a continuous 48-hour window, with managed and baseline arms under an equal power cap. NVML GPU-die power is a lower bound on wall power.
Derived from the same run, in balanced operating mode: up to +22% tokens per watt, from die power and serving throughput, with 0% throughput change.