H100 vs A100 Inference Cost 2026: Cost-Per-Token, Throughput, and Break-Even Math

What cost-per-token means in inference

Cost-per-token — and the more commonly quoted cost per million tokens ($/1M tokens) — is the unit of inference economics the way $/MWh is the unit of electricity economics. It folds in three things at once: the rent you pay for the GPU per hour, the throughput the GPU sustains per second at your batch size, and how that throughput decays as you serve longer sequences. Two GPUs with very different $/hr can produce identical $/1M tokens; conversely two GPUs with identical $/hr can produce very different $/1M tokens — purely because of throughput.

The formula is simple:

Throughput is shaped by three things, in order of importance: the model size (parameter count drives VRAM and FLOPs/token), the batch size (larger batches amortize weight loads and increase GPU utilization), and the sequence length (longer sequences mean more attention compute per token, especially on KV-cache-heavy architectures like Llama-2). H100 wins on throughput per dollar because Hopper doubles tensor-core throughput per SM over Ampere, doubles memory bandwidth via HBM3, and adds the FP8 tensor path that A100 lacks — all of which compound at large model scale.

There's a fourth axis we layer on top: provider-native rates. The same GPU model has a wide spread of $/hr across providers. H100 spot ranges from roughly $1.49/hr on Vast.ai (the current inference floor) to $4.10+/hr on reserved AWS. A100:80GB spot ranges from roughly $0.42/hr on Vast.ai up to $3.10+/hr on reserved hyperscaler tiers. We use the spot-floor provider rate as the procurement baseline in the table below — that's the marginal hour most teams pay — and reserve the reserved-tier rates for capacity planning rather than pure $/token comparison. The full live matrix is at /cost-per-token with the workload-modelled calculator at /cost-per-token-calculator.

The H100 vs A100 inference matrix (Q3 2026 spot floors)

The matrix below uses H100 at $1.49/hr Vast.ai (7th week flat — the current inference floor) and A100-80GB at $0.42/hr as the spot-floor baseline. Throughput per GPU is the moving part — and the moving part does not scale uniformly with $/hr. Two takeaways before the table:

• Provider rates come from the live dataset — these are real numbers, not derived. We tag the spot row provider-native in the live table.
• Per-GPU throughput differs across model sizes in ways that are workload-specific. For the matrix table below we apply the kalshi-derived / provider-native / data pending honesty convention from the procurement explainer — provider rates are real, throughput numbers at the model-size grain are framed against the live /cost-per-token index rather than fabricated as pseudo-precision figures here. For your exact model + framework + batch shape, plug it into the calculator.
• Break-even model size is an inference procurement concept: at what parameter count does H100 slot quality (Hopper FP8 +4× tensor core + HBM3 bandwidth) start paying for its $/hr premium in $/token terms? In Q3 2026, that break-even sits around ~30B parameters at batch=8, sequence=2048 — but it moves with batch, sequence, and quant scheme.

Reading the matrix: at 7B parameters, A100-80GB and A100-40GB produce roughly the same per-token economics as H100 — and A100-40GB wins outright because of $/hr. At 13B, H100 with FP16 begins to break even against A100-80GB. At 70B and beyond, H100 with FP8 wins clearly: throughput per H100 at FP8 is roughly 4× A100 throughput per A100, which more than compensates for the 3.5× $/hr premium.

For the Falcon-180B class (180B+ parameters), A100 is structurally out of the running: even with 80GB VRAM, you need tensor parallelism across multiple GPUs, and the cross-GPU communication overhead eats most of A100's $/hr advantage. H100 NVLink bandwidth (900 GB/s) makes the same tensor-parallel feasible at much lower latency cost. If you're serving Falcon-180B-class models, the question is H100 spot vs H100 reserved, not H100 vs A100 — A100 is off the table.

Procurement takeaway for Q3 2026

The H100 vs A100 inference decision in Q3 2026 has a clean shape once the workload is fixed:

• 7B and 13B models at scale — A100 wins $/token For high-QPS inference of smaller models where you can saturate A100 batches, A100-80GB produces the same per-token cost as H100 at lower absolute $/hr — and A100 capacity is widely available. Lock 3-month reservations on A100 spot and use A100 as your throughput backbone.
• 70B and Falcon-180B class — H100 wins $/token, A100 is off the table H100 slot quality (FP8 + Hopper tensor cores + 80GB HBM3 + 900 GB/s NVLink) is structurally required for sub-quantized inference of 70B+ models. The $/hour premium is offset by 4× throughput and much lower tensor-parallel overhead. Target H100 spot at $1.49/hr Vast.ai for elastic tier; reserved only for sustained baseline.
• Mixed-fleet orgs — match GPU to model size, not to $/hr If you're serving both small and large models, run a mixed fleet: A100 for 7B/13B traffic, H100 for 70B+/low-latency traffic. Don't try to standardize on H100 across the fleet — the $/hr premium on small-model inference is wasted.