How A lot Does a Native LLM Really Value to Run? I Measured Each Watt on Apple Silicon

by yourself {hardware} is that it’s free. You purchased the machine, the electrical energy is a rounding error, and each token after that’s gravy. A current In the direction of Information Science piece, How A lot Does It Really Value to Run a Native LLM?, put a quantity on that gravy by sampling the ability draw of an RTX 3090 and pricing it in opposition to an actual electrical energy tariff. Its discovering caught with me: the costliest mannequin to run wasn’t the largest one. Value per token tracked throughput, not parameter depend.

I don’t have a 3090. I run every part on an M3 Extremely Mac Studio with 96 GB of unified reminiscence—no discrete GPU, no VRAM, only one pool of reminiscence the CPU and GPU share. That’s a special sufficient machine that I wished to know whether or not the identical shock holds, and by how a lot. So I measured it. 5 fashions, sustained era, actual wall-socket vitality, priced at my precise utility charge. The brief model: the shock doesn’t simply maintain on Apple Silicon, it will get larger. My 120-billion-parameter mannequin is about 5 occasions cheaper per token than a mannequin 1 / 4 its measurement.

Right here’s how I bought the numbers, and why they land the place they do.

How the measurement works

The instrument is a small software I constructed known as TokenWatt. It’s an OpenAI-compatible proxy constructed only for Apple Silicon: you level it at no matter native inference server you already run, and it forwards each request byte-for-byte whereas bracketing it with an vitality measurement. It reads the whole-SoC energy rails by means of Apple’s IOReport interface—the identical counters Exercise Monitor’s vitality tab is constructed on—which suggests no sudo, no exterior instrumentation, simply the numbers the chip experiences about itself. It subtracts a rolling idle baseline so that you’re measuring the marginal value of the request, not the machine sitting there respiratory, and it costs the consequence at your electrical energy charge. Mine is a flat $0.31/kWh.

The honesty drawback with any measurement like that is that the on-die counters can drift, and there’s no approach for a reader to know whether or not to belief them. So the one factor I insisted on—and the explanation the software exists in any respect—is calibration in opposition to floor fact. I ran each measurement on this article with the Mac plugged right into a Shelly Plug US Gen4 that meters precise vitality on the wall, and TokenWatt’s built-in calibration match the SoC counters to it—no hand-tuning, and the identical process anybody with a metering plug can rerun. Each quantity under carries an actual error band from that match; they land between ±2.6% and ±4.5%. While you see “$0.109 per million tokens,” that’s a wall-verified determine with a acknowledged uncertainty, not a hopeful studying off an inner register.

For the headline numbers I ran every mannequin in a sustained era loop—the machine doing nothing however decoding tokens as quick as it might probably—at three durations (120, 360, and 720 seconds), three passes every, with a 15-minute idle baseline between runs. That’s the cleanest, most reproducible sign you will get: a saturated GPU, no gaps, no chilly begins. It’s a finest case, and I’ll come again to what occurs in messy actual visitors later.

The 5 fashions

All the things right here suits in 96 GB, which seems to matter (extra on that close to the top). The fashions span from a 4-billion-parameter dense mannequin as much as a 120-billion-parameter mixture-of-experts, throughout completely different quantizations. “Footprint” is what the weights occupy in unified reminiscence, and “tok/s” is sustained decode throughput. Value is per million output (generated) tokens, at $0.31/kWh.

I sorted that desk by value on goal, as a result of sorting it by measurement would scramble it. The most cost effective mannequin is the smallest, positive. However the most costly mannequin, by an element of 5 to 9, is the 27-billion-parameter dense one—smaller than three of the 4 fashions it’s dearer than. The 120B mannequin undercuts it 5 to 1. Parameter depend merely doesn’t predict what a mannequin prices to run right here: the costliest one is a mid-size dense mannequin, and each mannequin bigger than it’s cheaper.

Why the large mannequin wins

The mechanism is similar one the RTX-3090 piece recognized, and it’s price stating plainly: the electrical energy value of a token is watts divided by throughput. Two issues set it—how a lot energy the machine attracts whereas producing, and what number of tokens it produces per second—and the 27B dense mannequin loses on each.

Take a look at its two numbers. It attracts 138 watts, essentially the most of any mannequin right here, and it generates solely 21.5 tokens per second, the fewest. The reason being the factor that governs single-stream decode on this {hardware}: to provide one token, the machine has to learn each energetic weight out of reminiscence, and era velocity is ready by what number of bytes that’s divided by the chip’s reminiscence bandwidth. A dense 8-bit mannequin prompts all 27 billion parameters for each token—it streams its total ~28 GB of weights per token. That’s essentially the most information motion of something within the desk, which is why it’s each the slowest and essentially the most power-hungry: excessive bytes-per-token, low velocity, the worst of each worlds.

The mixture-of-experts fashions break that hyperlink between measurement and bytes-per-token. gpt-oss-120b has 120 billion parameters sitting in reminiscence, however for any given token a router prompts solely about 5 billion of them—so it reads a small fraction of its 59 GB per token, although the entire thing is resident. Much less information moved per token means extra velocity (74 tok/s) and fewer energy (94 W). Quantization compounds it: the MoEs right here run at 4 to six bits in opposition to the dense mannequin’s 8, shrinking the bytes-per-token additional. The result’s a 120B mannequin that’s genuinely cheaper and sooner to run than a 27B one, on the identical machine, measured on the similar wall socket—as a result of what you pay for per token is bytes streamed, and parameter depend solely units the bytes if the mannequin prompts all of them.

The 4B mannequin wins on pure value as a result of it’s tiny and blisteringly quick (134 tok/s), nevertheless it’s a special high quality tier. The actual lesson is in the course of the desk: amongst fashions you’d really attain for on a tough job, the well-quantized MoEs beat the dense mid-size mannequin outright. In case you have been selecting by parameter depend—”27B is smaller, it should be cheaper to run than a 120B”—you’d choose the one most costly choice on the board.

Does it maintain when the visitors is actual?

A sustained loop is a benchmark, not a workload. It retains the GPU pinned at 100% with no idle gaps, no chilly begins, no ready on a consumer. Actual serving is lumpier, and lumpier is dearer per token. So the sincere query is whether or not the ordering survives contact with precise use.

I’ve a second information supply for that: TokenWatt has been logging my actual inference visitors for a month—6,300-odd requests throughout these similar fashions, no matter I really threw at them. These numbers are the software’s estimated (not wall-calibrated) tier, so I belief their ordering greater than their third decimal, however the ordering is the entire level:

The hole is, if something, wider within the wild than on the bench: in day-to-day use the dense 27B value me roughly ten occasions as a lot per token because the MoEs. Two issues transfer these numbers relative to the managed desk—they’re priced per complete token (immediate plus output, not simply era), and actual visitors runs the machine properly under saturation, which raises the per-token value by roughly two to a few occasions versus the best-case loop. However the form is similar. The dense mid-size mannequin is the costly one irrespective of how I slice it.

The Apple Silicon asterisks

A couple of issues are particular to this machine, and so they matter if you wish to reproduce this or purpose about your personal.

Holding an enormous mannequin resident prices energy even between tokens. On a unified-memory machine the entire mannequin lives in the identical RAM the system makes use of, and maintaining tens of gigabytes of weights resident carries a standing overhead—reminiscence bandwidth, regulators, followers—separate from the arithmetic of era. I measured this overhead climb with mannequin measurement after which flatten out at roughly 21 watts as soon as a mannequin is large enough; the calibration itself additionally shifts barely for the biggest, memory-bound fashions. It’s a second-order impact in opposition to the throughput story, nevertheless it’s actual—another reason to maintain solely the fashions you’re actively utilizing resident.

96 GB is a ceiling, and it’s why the desk stops the place it does. My largest mannequin, gpt-oss-120b at MXFP4, occupies about 59 GB resident. Add the OS and a working KV cache and I’m close to the sensible restrict of a 96 GB machine—which is precisely why there’s no 200B mannequin within the desk. An MoE has to carry all its specialists in reminiscence although it solely makes use of just a few per token, so the footprint is the total mannequin measurement, not the active-parameter measurement. A 192 GB or 256 GB Mac Studio might run considerably bigger fashions, and the identical watts-over-throughput logic predicts the place they’d land. The outcomes additionally reproduce downward: on a 16, 32, or 64 GB Mac, run the smaller rows right here and you must see the identical ordering, as a result of the mechanism doesn’t depend upon my explicit machine.

That is marginal electrical energy solely. It isn’t the price of the Mac, which is by far the bigger quantity and relies upon solely on how laborious you employ it. Amortizing the machine this ran on—a Mac Studio that Apple at the moment sells for $5,299 (M3 Extremely, 96 GB, 1 TB)—over just a few thousand requests dwarfs a tenth of a cent of electrical energy. The measurement right here is the vitality ground beneath that equation—helpful for evaluating fashions in opposition to one another, and for understanding the true variable value of a token, however not a complete value of possession.

And a phrase on the cloud, since everybody asks. For scale, a budget hosted fashions individuals really attain for aren’t free both: as of mid-2026 GPT-5.4-nano lists at $1.25 per million output tokens, Gemini 2.5 Flash at $2.50, and Claude Haiku 4.5 at $5. My electrical energy for one million output tokens ranges from $0.06 on the 4B to $0.55 on the dense 27B. That reads like a rout for native, however the comparability is rigged in native’s favor: my determine is marginal electrical energy solely, on a machine I already paid a number of thousand {dollars} for, working smaller fashions than these hosted endpoints. The truthful takeaway isn’t “native is ten occasions cheaper.” It’s that after the machine is purchased, the marginal value of a token actually is low—and which mannequin you choose swings it by an order of magnitude. The cloud bundles the {hardware} you’d in any other case amortize; native unbundles it and fingers you the electrical energy invoice, which is the one quantity I can measure precisely.

The takeaway

Decide the smallest, quickest mannequin that clears your high quality bar—that’s the identical recommendation the RTX-3090 piece landed on, and Apple Silicon doesn’t change it. What Apple Silicon provides is a sharper corollary: as a result of unified reminiscence lets an enormous MoE match and since an MoE streams solely a fraction of its weights per token, a well-quantized 120B mannequin will be cheaper and sooner to run than a dense 27B one. Don’t choose by parameter depend. Decide by throughput, and let the measurement inform you which is which.

One caveat the fee numbers can’t present: value solely issues amongst fashions that may really do the job. In my very own agentic coding work—constructing a Lisp interpreter with actual tool-calling loops—the Qwen3.6 fashions held instruction-following and gear calls collectively extra reliably than a equally sized Gemma-4-31b, which is why Gemma by no means earned a slot in my rotation, or this desk. Clear the standard bar first; then let value and throughput break the tie. Which native fashions really maintain up underneath an actual tool-calling workload is the place I’m headed subsequent.

And measure your personal, as a result of your charge and your fashions aren’t mine. TokenWatt is open supply, it sits in entrance of no matter native server you already run, and it takes two strains to level at it:

uv software set up tokenwatt          # or: uvx tokenwatt · pip set up tokenwatt
tokenwatt serve --upstream http://127.0.0.1:8080 --rate 0.31

It logs a per-request, per-model ledger just like the one above, and if in case you have a metering good plug it’ll calibrate itself in opposition to the wall so your numbers carry an actual error bar as a substitute of a shrug. The repo is at github.com/mmmugh/tokenwatt. I’d genuinely wish to see these numbers reproduced on different machines and different charges—the entire level of measuring is that you simply don’t should take my phrase for it.

All figures measured on a Mac Studio (Apple M3 Extremely, 96 GB, macOS 26), wall-calibrated in opposition to a Shelly Plug US Gen4, priced at $0.31/kWh. Sustained-generation figures are plug-calibrated with ±2.6–4.5% bands; real-traffic figures are the uncalibrated estimated tier and are directional. Prices are marginal electrical energy solely and exclude {hardware}.