Where a GPU's power goes over a day
Look at a week of power readings from a busy inference fleet and three regions show up. There is an idle floor, where accelerators wait for work. There is a middle, where they run part loaded. And there is flat out.
At the floor, the accelerator still draws power while producing nothing. In the middle, its clock often runs higher than the work in hand needs. Flat out, it needs everything it has.
So the question is not how much a GPU draws at peak. It is how much of its day it spends in the first two regions. A bursty fleet has the most to recover, and a week of your own readings, set out below, shows where yours sits.
The levers that exist
Placement and scheduling pack work onto fewer accelerators, so the rest can sit at a low floor. Serving settings such as batch size and numerical precision change how much work each watt produces.
A power limit holds each accelerator under a ceiling. That suits a site that must never exceed a number, and it is a different job from matching the draw to the work.
Clock control sets the rate each accelerator runs at. The hardware vendor's own management interface accepts a locked clock target, and the vendor's command-line tool can set and reset it.
Newer silicon does more work per watt, and a refresh is a lever in its own right. Software levers apply to whatever silicon is installed, before and after a refresh.
Why a fixed setting and a changing fleet disagree
A target that suits an accelerator running flat out is the wrong target for one waiting for the next request. A real fleet moves between those states all day.
A fixed setting has to pick one. Set for the busy hour, it leaves power unused in the quiet one. Set for the quiet hour, it holds back work in the busy one. A loop that reads the work and adjusts each accelerator follows the day instead.
Will it slow the GPUs down?
Only where the work allows. Clock control runs the clock lower when the work in hand does not need the speed, so the test is whether throughput and latency hold.
The way to know is to measure both on the same run as the power. Read tokens per second and tail latency from your own serving stack, over the same window as the power readings.
How much your fleet has to recover
A week of per-accelerator power or utilisation samples settles it. An idle floor or a part-loaded middle is power to recover, and readings that sit high and flat around the clock leave little.
Two cases need a different start. A workload pinned at full load around the clock has little duty cycle to work with, so confirm it on that week of readings first. Where tenants own the accelerators, the authority to change how they run sits with each tenant, so the tenant is the one to deploy per-accelerator control.
What it takes on the host, and how it comes off
Clock control needs privilege on the host, so a service that does it holds more authority than an ordinary workload. A reviewer should see each privilege it holds, and what each one is for.
Stopping the service should release the clocks. After that, the vendor's own reset commands, nvidia-smi -rgc and nvidia-smi -rmc, return the device to its defaults.
Questions people ask
- How can I reduce GPU power consumption without losing throughput?
- Take the draw down only where the work does not need it: at the idle floor and in the part-loaded middle. Then read tokens per second and tail latency over the same window as the power.
- Is a power limit the same as reducing GPU power consumption?
- A power limit holds each accelerator under a ceiling, which caps the peak. Clock control matches the draw to the work across the day. They answer different needs.
- What do nvidia-smi -rgc and -rmc do?
- The first resets a GPU's locked clock settings to their defaults, and the second resets its locked memory clocks. Both are part of NVIDIA's own command-line tool.
Where ATHLAZ fits
ADAPT does this clock control per GPU
ADAPT, AI-Driven Adaptive Power Technology, is per-GPU software that runs this clock control continuously. It reads each accelerator's counters and sets one locked clock target per accelerator, per control tick, through NVIDIA's own management interface.
Measured: up to 21% less GPU die power on NVIDIA H100 NVL, over a continuous 48-hour window, with managed and baseline arms under an equal power cap. Derived from the same run: up to +22% tokens per watt and 0% throughput change.
The saving came from duty cycle: most of it idle-floor reduction, and most of the rest clock-down at mid utilisation.
The security review sets out the privilege ADAPT holds on the host and how it comes off.