AI Compute and Accelerators: A Practical Reference

A GPU runs thousands of threads in lockstep groups, a model usually called single instruction, multiple threads. Threads are scheduled in warps of 32, and the hardware hides memory latency by switching between warps rather than by caching aggressively, which is why a GPU tolerates a memory system that would starve a CPU. Modern parts pair that general shader machinery with dedicated matrix units that multiply small tiles in a single instruction, and those units supply almost all the throughput on machine-learning work. The general cores remain because real models are not only matrix multiplies: normalization, activation, sampling, and the data movement between kernels all run on the same device, and a design that could only multiply matrices would hand those back to the host across a slow link.

The strength is that it is programmable and everybody already targets it. A new model architecture runs on day one, badly if necessary, and a kernel can be written to make it run well. That flexibility is why research happens on GPUs even when a fixed-function part would serve a shipped workload more cheaply. The weakness is that generality costs area and power: a large fraction of the die is scheduling, register file, and cache that a dedicated matrix engine would not need, and the power budget per package has climbed past what air cooling handles. The other weakness is economic rather than technical, in that the parts are allocated rather than simply sold, so a buyer's throughput is often set by what was delivered rather than by what was designed.

Use a GPU when the workload is still changing, when the team needs one target for both training and inference, or when the software the product depends on exists only in a vendor-specific form. Use it for training almost unconditionally at present, because the distributed training stack is mature here and immature nearly everywhere else. Move away from it for a stable, high-volume inference workload where a fixed-function part gives more tokens per watt, and for on-device work where the power budget rules it out entirely. If the model is small enough to be bandwidth-bound at batch one, check a cheaper part with comparable memory bandwidth before buying peak FLOPS that the workload cannot reach.

Warps of 32 threads as the scheduling unit · datacenter packages drawing roughly 400 W to over 1 kW, with liquid cooling usual past about 700 W · HBM bandwidth in the low single-digit TB/s on current parts · matrix units supplying the large majority of peak throughput on transformer work · model FLOPs utilization commonly 30–50% on well-tuned training runs, lower on inference · native support spanning FP32 down to FP8 and FP4 on recent generations.

NVIDIA's datacenter parts are the reference point and the bulk of the installed base; AMD's Instinct line is the closest architectural equivalent with a different software stack. Consumer parts from both vendors run the same programming model at lower memory capacity, which is why so much open-source inference work targets them.

The chip is the single largest line in a cluster's capital, and everything else is a fraction of it. What complicates the arithmetic is that peak specifications describe a machine nobody reaches: model FLOPs utilization of 30–50% on a tuned training run means half the hardware paid for is idle at any moment, and that gap is closed with engineering rather than capital. Depreciation assumption matters more than list price, because a part that keeps serving inference for years after it stops being competitive for training has a very different cost per token in year five than in year one. Supply allocation has repeatedly been the binding constraint on what an operator can deploy, which puts terms of access, rather than unit price, at the center of most real negotiations.

7 minGPUs: ExplainedIBM Technology · 100k+ views

14 minCUDA Explained - Why Deep Learning uses GPUsdeeplizard · 100k+ views

CUDA C++ Programming Guide (NVIDIA) · Efficiently Scaling Transformer Inference (arXiv)

A dataflow architecture lays the computation out in space rather than stepping through it in time. Instead of repeatedly loading weights from external memory into a fixed set of math units, the machine holds large amounts of on-chip memory next to a very large array of processing elements and streams activations through a graph that has been compiled onto the fabric. Wafer-scale integration takes the idea to its physical limit by declining to cut the wafer into dies at all, so an entire wafer becomes one processor with on-chip bandwidth that no package-level interconnect can match. The engineering problems that follow are power delivery, heat removal, and yield: a wafer will contain defects, so the design has to route around bad regions rather than discard the part.

The strength is memory bandwidth and latency that package boundaries otherwise prevent, which directly attacks the constraint that limits most inference. Holding weights in on-chip SRAM removes the external memory read that sets the token rate on conventional parts, and a model that fits entirely on-chip can run at rates a bandwidth-bound GPU cannot approach. The weaknesses are capacity and software. On-chip memory is measured in gigabytes where HBM is measured in hundreds, so a large model has to be split across many units, and the compiler that maps a graph onto a spatial fabric is doing a harder job than one emitting kernels for a conventional device. Utilization also falls sharply when a model's shape does not match the fabric's.

Consider it when latency at small batch is the product requirement rather than throughput at large batch, since that is exactly where conventional parts are weakest and this approach is strongest. It suits a stable model that fits the available on-chip memory, and a customer willing to work within one vendor's compiler. Avoid it when the model changes weekly, when the workload is training at cluster scale where the surrounding software matters more than the chip, or when the deployment cannot supply the power and cooling density that a wafer-scale part requires. If the model does not fit on-chip, most of the argument for the architecture disappears.

On-chip SRAM in the order of gigabytes per unit against hundreds of gigabytes of HBM on a conventional accelerator · on-chip bandwidth orders of magnitude above package-level memory bandwidth · wafer-scale parts drawing well over a kilowatt and needing engineered power and cooling · redundancy designed in so that a wafer with defects still yields a working part · compilation of a model graph onto the fabric taking minutes to hours rather than seconds.

Cerebras builds wafer-scale engines with the whole wafer as one device. Groq and SambaNova ship dataflow architectures at conventional die sizes with large on-chip memory and static compilation. All three sell access through their own clouds as well as as hardware, which is a direct response to the software adoption problem.

This class competes on tokens per second per dollar at small batch rather than on peak throughput, which is a genuinely different sales argument and a narrower one. The capital cost per unit is high and the addressable workload is narrower than a GPU's, so utilization risk falls on the vendor more than on the buyer, which is much of why these companies operate their own clouds instead of only selling boxes. That model also solves the software problem commercially rather than technically: a customer using a hosted endpoint never has to port anything. The strategic question for an investor is whether the latency advantage survives the next generation of conventional memory, since HBM bandwidth has been rising quickly and the gap being sold is a gap in someone else's roadmap.

Cerebras product chip overview (Cerebras) · Efficiently Scaling Transformer Inference (arXiv)

Server CPUs now carry matrix extensions that make them a usable inference device rather than only a host. The instructions operate on small tiles held in dedicated registers, which raises throughput on matrix multiplication by roughly an order of magnitude over the vector units that preceded them. What a CPU still lacks is memory bandwidth: server DRAM delivers hundreds of gigabytes per second where HBM delivers terabytes, so a CPU running a decode workload is bandwidth-starved for the same reasons any other device is, only sooner. The compensating advantage is capacity, since a server can hold far more DRAM than any accelerator holds HBM, and a large model that will not fit on one accelerator often fits in system memory without partitioning.

The strength is that the hardware is already bought and already idle. Inference on spare CPU capacity has no acquisition cost, no allocation queue, and no separate operational story, which makes it the obvious first deployment for a workload whose volume is unproven. It also runs anything, since there is no kernel coverage question on a CPU. The weakness is throughput per watt, which is poor enough that any workload with sustained volume moves to an accelerator on cost alone. Batch efficiency is limited by bandwidth rather than compute, so the usual trick of batching harder returns less than it would on a GPU.

Use it for low-volume, latency-tolerant inference, for small models, and for the period before a workload's demand is known well enough to justify buying accelerators. It is also the right answer when the model must sit next to a large amount of data already in system memory, or when a deployment cannot host accelerators at all. Move to an accelerator once the workload runs continuously, since at that point the power and rack space consumed by CPU inference costs more than the accelerator would. Avoid it entirely for training beyond toy scale.

Server DRAM bandwidth in the hundreds of GB/s against low single-digit TB/s for HBM · system memory capacity into the terabytes per socket against tens to hundreds of gigabytes of accelerator memory · matrix extensions raising matrix throughput by roughly an order of magnitude over prior vector units · typical server sockets at 200–400 W · no acquisition cost where existing fleet capacity is idle.

Intel's AMX extensions and Arm's matrix and vector extensions are the two main lines. Hugging Face and ONNX Runtime both ship CPU execution paths that are the usual way this capacity actually gets used, and llama.cpp made quantized CPU inference ordinary on consumer hardware.

The economics are unusual because the marginal cost of the hardware is often zero: a fleet already sized for peak web traffic has idle sockets most of the day, and inference is a way to fill them. That makes CPU inference cheap in exactly the situation where volume is low and unpredictable, and it stops being cheap the moment the workload is steady, because sustained CPU inference consumes power and rack space that an accelerator would use far more efficiently. The useful rule of thumb is that CPU inference wins on capital and loses on operating cost, so the crossover is set by duty cycle rather than by model size. For a vendor, this is a defensive market rather than a growth one, since it is sold as a use for hardware the customer already owns.

GPU and CPU inference optimization (Hugging Face) · ONNX open standard for model interchange (ONNX)

An FPGA is an array of configurable logic blocks, arithmetic slices, and on-chip memory joined by a programmable interconnect, so the hardware itself is defined after manufacture. For machine learning that means the arithmetic can be built at whatever precision the model actually needs rather than at the precision a fixed device happens to offer, and a pipeline can be laid out to match one specific network with no instruction fetch or scheduling overhead. The cost is that building that pipeline is a hardware design task. Vendor toolchains now accept models from ordinary frameworks and generate the configuration, which has narrowed the gap, but a design that reaches the part's potential still involves engineering closer to chip design than to software.

The strengths are deterministic latency, arbitrary precision, and the ability to put custom logic next to the model, which is why FPGAs persist in signal processing, networking, and instrumentation where the inference is one stage of a larger real-time pipeline. Because the datapath is built rather than scheduled, per-inference latency is predictable in a way that a GPU's is not. The weaknesses are throughput per dollar against a comparable ASIC or GPU, and development cost. Reconfiguration is also slow enough that switching between models is a deployment event rather than a runtime decision.

Use an FPGA when the inference has to happen inside an existing real-time pipeline, when latency has to be bounded rather than merely low on average, or when the precision or operator set is unusual enough that fixed hardware wastes most of its throughput. It suits low and medium volumes where an ASIC's mask cost cannot be justified. Avoid it for large language model serving at scale, where memory bandwidth rather than arithmetic flexibility is the constraint and an FPGA has no bandwidth advantage, and avoid it wherever a team without hardware engineers has to maintain the result.

On-chip memory in the tens of megabytes, far below any HBM-equipped accelerator · deterministic per-inference latency, typically bounded to within microseconds by design · reconfiguration taking milliseconds to seconds, so model switching is not a runtime operation · arbitrary precision, including formats no fixed accelerator supports · development effort measured in engineer-months where a GPU port is measured in days.

AMD's Vitis AI and Intel's OpenVINO FPGA paths are the two mainstream toolchains. The common production use is in network appliances, medical instruments, and defense systems where the model is one block in a pipeline that also does filtering, protocol handling, or sensor fusion.

Unit cost is high relative to throughput, and the case is almost never made on tokens per dollar. It is made on the system: an FPGA that absorbs the inference plus the signal processing plus the interface logic replaces several components, and the comparison is against that whole bill of materials rather than against an accelerator. Development cost is the number that decides the outcome, and it is large enough that the volume has to be low for the mask cost of an ASIC to remain unjustified while high enough to repay the engineering. That squeeze is narrow, which is why the FPGA share of machine-learning inference has stayed modest even as the tools improved.

Vitis AI development environment (AMD) · OpenCL open standard for heterogeneous programming (Khronos Group)

A systolic array is a grid of multiply-accumulate cells wired so that operands flow from one cell to its neighbors on each clock rather than being fetched individually from memory. A weight value loaded into a cell is reused by every activation that passes through it, so one memory read serves many operations, and that reuse is the point: it attacks the bandwidth constraint by moving less data rather than by moving data faster. The tradeoff is rigidity. The array has a fixed shape, and a matrix multiplication whose dimensions do not match it leaves cells idle, so the compiler spends much of its effort tiling and padding real model shapes onto the hardware's shape.

The strength is efficiency on the operation that dominates transformer workloads: for large, well-shaped matrix multiplications this is close to the most arithmetic per watt any programmable approach delivers, and it needs less control logic and register file than a general processor. Coupled with a purpose-built interconnect, it scales to very large training jobs. The weaknesses are utilization on awkward shapes, and everything that is not a matrix multiply. Attention, normalization, and sampling all have to run somewhere, and a design that under-provisions those units finds them limiting the whole pipeline. Availability is the other practical weakness, since the significant deployments are captive to the companies that built them.

Use it where the workload is dominated by large matrix multiplications and the model shape is known well enough to be compiled onto the array efficiently, which describes most large-scale training and much high-volume serving. It suits organizations willing to work in a framework the vendor supports well rather than at the kernel level. Avoid it for research on unusual architectures where shapes change constantly, for workloads that are mostly attention rather than dense matrix multiplication, and where the hardware is only available as a cloud service in a region the product cannot use.

Weight reuse across the array so that one memory read serves many multiply-accumulate operations · utilization falling sharply when matrix dimensions do not divide evenly into the array shape · deployments scaling to thousands of chips joined by a purpose-built fabric · precision typically BF16 and FP8 rather than the full range a GPU supports · the original datacenter TPU study reporting large efficiency gains over contemporary CPUs and GPUs on production inference.

Google's TPU is the canonical example and the most documented, with published architecture papers from the first generation through v4. Amazon's Trainium and Inferentia and several Chinese datacenter accelerators use related structures. The published TPU work is the best public description of how such a machine is actually built and scheduled.

Most of the significant systolic-array capacity was built by companies to serve their own workloads, which changes the economics from a product sale into an avoided purchase. That vertical position removes the software adoption problem, since the same organization owns the model, the framework, and the chip, and it removes the margin a merchant vendor would take. It also means published pricing reflects a cloud service rather than silicon, so comparing it against a GPU list price compares two different things. For an investor the question is whether a merchant market exists at all, since the two largest deployments are captive and the parts are rented rather than sold.

17 minTensor Processing Units (TPUs) ExplainedThe Coding Gopher · 50k+ views

16 minHow Nvidia GPUs Compare To Google’s And Amazon’s AI ChipsCNBC · 1m+ views

In-Datacenter Performance Analysis of a Tensor Processing Unit (arXiv) · Introduction to Cloud TPU (Google Cloud)

An inference ASIC is silicon designed for one job: running trained models forward, usually at low precision, usually with a memory system sized for the weights rather than for training's activation traffic. Dropping training removes a great deal of hardware, since there is no need for the wider precision, the gradient storage, or the collective operations that a training chip must support. What remains can be spent on the units that matter for serving. The design bet is that the shape of the workload holds still long enough for a chip taped out today to be relevant when it ships, which is a two-to-three year proposition in a field that has repeatedly changed direction inside that window.

The strength is efficiency: for a workload the part was designed around, tokens per watt and tokens per dollar can be substantially better than a general accelerator, because none of the die is spent on flexibility the job never uses. The weakness is exactly the same property. A model with an operator the chip lacks either runs on a fallback path or does not run, and the fallback is usually slow enough to erase the advantage. Software is the harder problem in practice: the chip has to be reachable from the frameworks customers use, and building that support costs more and takes longer than most hardware teams plan for.

Use one when the serving workload is high volume, stable, and understood well enough to be confident about the operator set two years out, and when the power or cost per token of a general accelerator is genuinely the constraint on the business. It suits inference providers at scale and product companies with one dominant model. Avoid it while model architecture is still moving, when the team cannot absorb a second software stack, and when the volume is low enough that the general-purpose part's higher unit cost is cheaper than the integration effort.

Non-recurring engineering for a leading-edge design in the tens of millions of dollars and up, with mask sets a large part of it · two to three years from design start to volume availability · precision usually FP8 and below, since training range is not required · power per part commonly in the 100–400 W band rather than the kilowatt class · efficiency advantages over general accelerators reported as multiples on matched workloads, contingent on the workload matching.

Amazon's Inferentia, Meta's MTIA, and a number of startup parts target this position, as do the inference-only configurations several vendors ship alongside training silicon. Almost all are deployed by the organization that commissioned them rather than sold on the open market.

The capital structure is the opposite of a GPU purchase: a large fixed cost up front in design and masks, then a low marginal cost per part. That only works at volume, and the volume has to be committed before the chip exists, which is why the successful examples are nearly all captive to a hyperscaler with a known internal workload. The risk that kills these projects is not fabrication but timing, since a part designed around a model shape that stops being used arrives efficient at the wrong thing. Software is the second risk and the more common one: a chip that cannot be reached from the framework customers already use has no market at any efficiency.

MLPerf Inference: Datacenter benchmark results (MLCommons) · MLPerf Inference Benchmark (arXiv)

An edge neural processing unit is a block on a phone, laptop, or embedded system-on-chip that runs models within a power budget measured in watts rather than hundreds of watts. The design constraints are different in kind from a datacenter part: there is no HBM, memory is shared with the rest of the system, and the thermal budget is whatever the enclosure can shed without a fan. Efficiency therefore comes from low precision, from keeping weights in on-chip memory where possible, and from operator sets tuned to the small models these devices actually run. Nearly every recent phone and laptop silicon carries one, so the installed base is enormous and mostly idle.

The strengths are power, latency, and privacy. Inference on the device costs no network round trip and no server capacity, keeps data local, and works without connectivity, which matters for a growing set of regulated and consumer applications. The weakness is capacity. Shared LPDDR bandwidth and a few gigabytes of usable memory bound what can run, so on-device models are small, quantized, and specialized rather than general. Fragmentation is the other real weakness: every vendor's NPU has its own runtime and quantization requirements, so shipping one model across the installed base means shipping and validating several conversions.

Use it when latency, privacy, offline operation, or per-inference server cost are what the product is about, and when a small quantized model is genuinely sufficient for the task. It is the right home for wake words, transcription, image processing, and classification. Route to a server when the task needs a large model, when quality is the product, or when the model has to be updated more often than the client can be. The common pattern is a hybrid, where the device handles what it can and escalates the rest, and the hard engineering is in deciding which is which.

Power budgets from well under a watt on embedded parts to a few watts on laptop silicon · shared system memory rather than dedicated HBM, so bandwidth is a fraction of a datacenter part's · models typically quantized to 8 bits or below to fit · a separate runtime and conversion path per vendor, so one model becomes several validated builds · installed base in the billions of devices across phones and laptops.

Apple's Neural Engine, Qualcomm's Hexagon, and the NPU blocks in recent laptop silicon from Intel, AMD, and Qualcomm are the volume examples. Embedded parts from Arm and a long tail of microcontroller vendors carry much smaller versions for always-on sensing.

The silicon is effectively free to the developer, since it ships inside a device the customer already bought, and that is the whole economic argument: an inference that runs on the handset costs the product company nothing per call, where the same inference on a server has a marginal cost that scales with usage. For a consumer product with millions of users and a high call rate, moving work on-device can be the difference between a viable margin and none. The cost sits in engineering instead, in the conversion, quantization, and per-vendor validation needed to ship across a fragmented installed base, and that cost is paid per model rather than per user.

Microscaling Data Formats for Deep Learning (arXiv) · ONNX open standard for model interchange (ONNX)

HBM stacks several DRAM dies vertically, connects them with through-silicon vias, and places the stack on the same package as the processor with a very wide interface. Where a conventional DRAM channel is 64 bits wide and runs fast, an HBM stack is thousands of bits wide and runs comparatively slowly, and the product of width and rate is what delivers bandwidth an order of magnitude beyond commodity memory. Sitting on the package rather than on a board is what makes the width possible, since routing thousands of connections across a motherboard is not practical. That packaging requirement is why HBM supply is tied to advanced packaging capacity rather than only to DRAM fabs.

The strength is that it directly relieves the constraint that limits most machine-learning work. Decode rate is set by how fast weights can be read, so bandwidth converts almost linearly into tokens per second on a bandwidth-bound workload, which is not true of added arithmetic. Power per bit moved is also better than driving signals across a board. The weaknesses are cost, capacity, and supply. HBM is far more expensive per gigabyte than commodity DRAM, capacity per stack is limited by how many dies can be stacked and cooled, and the packaging step that attaches it has been a hard bottleneck across the industry, so availability is set by advanced packaging lines rather than by wafer starts.

It is not really an option on a datacenter accelerator so much as a requirement, since a part without it cannot compete on any bandwidth-bound workload. The real decision is how much of it to buy, and that follows from the model: if weights plus KV cache fit in a smaller capacity, paying for more is wasted, and if they do not fit the model has to be split across chips, which converts a memory problem into a much harder fabric problem. For edge and cost-sensitive parts, GDDR or LPDDR is the alternative and the tradeoff is explicit bandwidth for price.

Stacks of typically 8 to 12 DRAM dies joined by through-silicon vias · interface thousands of bits wide against 64 bits for a conventional channel · per-accelerator bandwidth in the low single-digit TB/s against hundreds of GB/s for server DRAM · capacity per accelerator in the tens to low hundreds of gigabytes against terabytes of system DRAM · cost per gigabyte several times commodity DRAM · supply gated by advanced packaging capacity rather than by DRAM wafer starts.

Micron, SK hynix, and Samsung are the three suppliers, and HBM content is now a large share of the bill of materials of a datacenter accelerator. The packaging step that attaches stacks to the processor is covered on the semiconductor manufacturing sheet, where the yield and capacity constraints are described in more detail.

HBM has become one of the largest single line items in an accelerator's bill of materials, which is unusual for a memory component and is a direct consequence of bandwidth being the binding constraint. That has moved pricing power toward the three suppliers and made HBM allocation a strategic negotiation rather than a purchasing one. The second-order effect matters more for the industry: because supply is limited by advanced packaging rather than by DRAM capacity, adding HBM output is slower than adding conventional memory output, so the shortage persists through cycles where commodity DRAM is in surplus. For a buyer, the practical consequence is that memory capacity per accelerator, not arithmetic, is usually what is being rationed.

13 minThe Special Memory Powering the AI RevolutionAsianometry · 100k+ views

5 minHigh Bandwidth Memory (HBM) As Fast As PossibleTechquickie · 100k+ views

High-bandwidth memory (Micron) · Efficiently Scaling Transformer Inference (arXiv)

HBM stacksSemiconductor Manufacturing

GDDR is graphics DRAM soldered around a processor on the board, trading HBM's extreme width for higher per-pin signaling rates and far simpler packaging. LPDDR is the low-power mobile line, shared with the rest of a system-on-chip and optimized for energy per bit rather than peak throughput. Neither approaches HBM's bandwidth, and both cost a fraction as much per gigabyte and need no advanced packaging step. For a large class of inference work that tradeoff is the right one, because the workload is limited by capacity or by cost rather than by the last increment of bandwidth.

The strengths are price, availability, and capacity per dollar. A part using GDDR can ship in volume without competing for the packaging capacity that gates HBM, which in a constrained market is a commercial advantage rather than only a technical one. The weakness is bandwidth, which is the specification that sets decode rate, so a GDDR-equipped part serving a large model at small batch will be slower in direct proportion. LPDDR adds a further constraint in that bandwidth is shared with everything else on the chip, so the memory available to the model depends on what the rest of the system is doing.

Use GDDR where cost per unit of capacity matters more than peak bandwidth: smaller models, batch workloads that can amortize weight reads across many sequences, and consumer or workstation deployments. Use LPDDR wherever the power budget rules out everything else, which in practice means every phone, laptop, and embedded device. Move to HBM when the workload is genuinely bandwidth-bound at the batch size the product needs, since below that point the extra cost buys throughput the workload cannot use.

Per-part bandwidth in the hundreds of GB/s against low single-digit TB/s for HBM · cost per gigabyte a fraction of HBM's · no advanced packaging requirement, so supply follows ordinary DRAM capacity · LPDDR bandwidth shared with the rest of the system-on-chip rather than dedicated to the accelerator · capacity per board often higher than an HBM part of similar price.

Consumer and workstation graphics parts use GDDR and are widely used for inference and for small-scale fine-tuning. LPDDR sits behind every phone and laptop NPU. Several inference-focused datacenter parts have chosen GDDR deliberately to avoid the HBM supply queue.

The argument is capacity per dollar and availability, and in a market where HBM is rationed the second matters as much as the first. A part that ships because its memory is not supply-constrained can take share from a better part that cannot be bought, which is a commercial dynamic rather than a technical one and has repeatedly decided outcomes in this market. For the buyer, the honest comparison is tokens per second per dollar at the batch size actually used, since GDDR closes much of the gap under batching and closes none of it at batch one. For consumer hardware, LPDDR's economics are simply the device's economics, since the memory is bought for the whole system rather than for the model.

GPU and CPU inference optimization (Hugging Face) · Microscaling Data Formats for Deep Learning (arXiv)

Every accelerator holds a working set in static memory on the die itself: register files, shared memory or scratchpad, and caches. This memory is one to two orders of magnitude faster than anything off-chip and costs no package-crossing energy, but it is measured in tens or hundreds of megabytes where external memory is measured in tens or hundreds of gigabytes. Almost all the performance engineering on an accelerator is about this gap. A kernel that keeps its working tile resident on-chip and reuses it many times runs near peak, and the same arithmetic written to fetch operands repeatedly from external memory runs at a fraction of it, on identical hardware.

The strength is that it is the only memory fast enough to keep the math units busy, which makes it the resource that skilled kernel work is really allocating. Techniques that reorganize a computation to fit its working set into on-chip memory have produced some of the largest speedups in the field without changing the mathematics at all. The weakness is capacity, and it does not scale well: SRAM has been shrinking more slowly than logic across recent process nodes, so each generation buys proportionally less on-chip memory than logic, and the imbalance the whole system suffers from gets slightly worse rather than better.

This is not a purchasing decision so much as the thing that determines whether the hardware bought performs. The practical implication is that a workload's throughput depends on whether its kernels tile to the available on-chip memory, so a model with unusual shapes, very long sequences, or an operator without a fused implementation will underperform its specification. Where a choice does exist is architectural: dataflow parts spend far more area on SRAM and correspondingly less on external bandwidth, which suits models small enough to be held resident and suits nothing else.

On-chip SRAM in the tens to low hundreds of megabytes per accelerator against tens to hundreds of gigabytes of external memory · access latency and bandwidth one to two orders of magnitude better than external DRAM · SRAM area scaling more slowly than logic on recent process nodes, so the ratio worsens each generation · fused attention implementations reporting large speedups purely by keeping tiles resident rather than by reducing arithmetic.

FlashAttention is the clearest published case: it restructures attention so intermediate results stay in on-chip memory instead of being written out and read back, and gains substantial speed and memory savings while computing exactly the same result. Dataflow and wafer-scale architectures take the same idea to its limit by trying to hold the entire model on-chip.

On-chip memory is expensive in the only currency that matters at design time, which is die area, and it competes directly with the math units for it. That tradeoff is one of the sharpest architectural decisions a vendor makes, and it is largely invisible to buyers because no specification sheet leads with SRAM capacity. Its economic significance for a customer is indirect but large: because achieving peak requires kernels tiled to this memory, the gap between a naive implementation and a tuned one is often a factor of two or more, which means engineering time substitutes for hardware purchases at a very favorable rate. Organizations without kernel expertise pay for that gap in capital instead.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv) · FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (arXiv)

Arithmetic intensity is the ratio of operations performed to bytes moved from memory. Every machine has its own ratio of peak arithmetic to peak bandwidth, and comparing the two tells you which resource a kernel exhausts first: below the machine's ratio the kernel is bandwidth-bound and adding arithmetic capability changes nothing, above it the kernel is compute-bound and bandwidth is not the problem. Plotted, this gives the roofline, a ceiling that slopes with bandwidth at low intensity and flattens at peak throughput. It is the single most useful piece of analysis for deciding what hardware to buy, and it is routinely skipped in favor of comparing peak specifications that most workloads never approach.

The strength of the model is that it is simple, cheap to apply, and usually right about the first-order question. Knowing that a workload sits on the sloped part of the roofline immediately rules out a whole category of purchases and points at quantization and batching instead. The weakness is that it is a first-order model: it ignores cache behavior, occupancy, launch overheads, and the fact that real workloads are a sequence of kernels with different intensities. It tells you which wall you are against, not how close to that wall an implementation gets.

Apply it before any hardware comparison. Measure or estimate the bytes moved and the operations performed for the workload's dominant kernels, compare against candidate machines' ratios, and only then look at specifications. It is most valuable when a purchase decision is being made on peak FLOPS, because that is exactly the situation where it most often overturns the conclusion. It is less useful for tuning a single kernel that is already near its ceiling, where profiling tools give better information.

Transformer decoding at batch one having very low arithmetic intensity, since each token reads the whole weight set to produce one output · prefill and training having high intensity because the same weights serve many tokens at once · batching raising intensity roughly in proportion to batch size until the KV cache exhausts memory · a worked example, derived here rather than published: a 70-billion-parameter model at BF16 occupies about 140 GB, so a 3 TB/s memory system reads it roughly 21 times a second and caps an unbatched stream near 21 tokens a second regardless of arithmetic throughput.

The published analyses of transformer inference scaling apply this reasoning explicitly to decide partitioning strategies and batch sizes. The same arithmetic explains why quantization is so effective on decode and so unhelpful on prefill, and why serving systems work hard to batch requests that arrive independently.

This is analysis rather than a product, and its economic value is in purchases avoided. A buyer who establishes that a serving workload is bandwidth-bound stops paying for arithmetic that cannot be reached, and typically finds that a cheaper part with comparable memory bandwidth delivers the same tokens per second for less money. The same analysis redirects effort toward quantization and batching, both of which are engineering rather than capital. Getting this wrong is expensive in a specific and common way: fleets sized on peak FLOPS comparisons are routinely over-provisioned in arithmetic and under-provisioned in memory, and neither error is visible until the workload runs.

Efficiently Scaling Transformer Inference (arXiv) · FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv)

Generating text one token at a time would require recomputing attention over the entire preceding sequence at every step. The key-value cache avoids that by storing the key and value tensors for tokens already processed, so each new token attends against stored state rather than recomputing it. The cost is memory, and it grows with sequence length and with the number of concurrent requests. On a long-context serving workload the cache can exceed the size of the model weights, at which point the binding constraint on how many users a machine serves is not the model at all but the accumulated conversation state behind them.

The strength is that it converts a quadratic recomputation into a linear read, which is what makes autoregressive generation practical at all. The weakness is that its memory grows without bound as conversations lengthen, and naive implementations reserve a contiguous block sized for the maximum possible length, which wastes most of it. That waste is severe enough that fixing it was worth a research contribution: paging the cache into fixed-size blocks, allocated on demand and shared between requests with common prefixes, raised achievable batch sizes substantially on unchanged hardware.

The cache is not optional in production serving; the decisions are how to store it and when to discard it. Quantize it when memory is the constraint and quality has been measured, since it is more sensitive than weight quantization and degradation shows up on long context first. Page it rather than reserving contiguous blocks. Share prefixes when many requests begin with the same system prompt, which is common enough in production to be worth engineering for. Offload it to host memory only when the alternative is refusing the request, since the transfer cost is high.

Cache size growing linearly with sequence length and with concurrent requests · at long context it can exceed the model weights, becoming the limit on concurrency · naive contiguous allocation wasting a large fraction of reserved memory, since most requests do not reach the maximum length · paged allocation with prefix sharing reported to raise serving throughput by a large multiple at unchanged hardware · quantizing the cache to 8 bits roughly halving its footprint, with quality effects that have to be measured on long context.

vLLM's paged attention is the reference implementation and is now standard in most serving stacks. Prefix caching across requests that share a system prompt is a common production optimization, and grouped-query attention is the architectural response, reducing the number of key-value heads and therefore the cache directly.

This is where serving cost is quietly decided. Concurrency per accelerator sets cost per request, and on long-context workloads concurrency is limited by cache memory rather than by weights or arithmetic, so an engineering change to how the cache is allocated moves unit economics as much as a hardware upgrade would. The paged-allocation result is the clearest example available of software returning a large multiple on throughput from hardware already bought. It also explains a pricing pattern visible across inference providers: long-context requests cost disproportionately more than their token count suggests, because they occupy scarce cache memory for the whole duration of the conversation.

9 minThe KV Cache: Memory Usage in TransformersEfficient NLP · 100k+ views

11 minHow KV Cache Speeds Up LLMs for Faster AI Models on GPUsIBM Technology · 100k+ views

Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv) · vLLM documentation (vLLM)

Compute Express Link is a cache-coherent protocol that runs over the PCIe physical layer and lets memory sit behind a link rather than on a processor's own channels. That allows capacity to be added to a node without adding sockets, and allows a pool of memory to be shared between hosts and reallocated as workloads need it. For machine learning the attraction is capacity rather than speed: CXL-attached memory has bandwidth and latency closer to a distant DRAM channel than to HBM, so it is a tier below system memory rather than a competitor to on-package memory.

The strength is that it attacks stranded capacity, which is a real and measurable waste in large fleets where memory is provisioned per server for a peak that rarely arrives. Pooling lets that headroom be shared. For inference, it offers somewhere to put a KV cache or model weights that would otherwise not fit, at a cost far below refusing the request. The weakness is that the latency and bandwidth penalty is real, so anything placed there has to be genuinely cold or the performance loss exceeds the capacity gain. Adoption has also been slower than the specifications suggested, since the benefit depends on software that understands memory tiering.

Use it when capacity rather than bandwidth is what runs out, and when the data placed there is accessed rarely enough to tolerate the extra latency. Offloading a KV cache for paused or long-idle conversations is the clearest inference case, since the alternative is evicting the conversation entirely. Fleet-level pooling makes sense at a scale where stranded memory is a large enough number to justify the integration work. Avoid it for anything on the critical path of token generation, and avoid assuming a roadmap benefit that depends on tiering software that is not yet in place.

Runs over the PCIe physical layer, so bandwidth per link is far below on-package memory · latency higher than local DRAM and far higher than HBM, placing it as a distinct memory tier · capacity addable without adding CPU sockets · pooling intended to recover memory stranded by per-server provisioning · adoption still early relative to specification maturity, with the benefit gated on tiering-aware software.

The Compute Express Link Consortium publishes the specification and the member list, which now includes every major processor and memory vendor. Early deployments are mostly memory expansion within a server rather than the shared pools the specification enables, which is the simpler case and the one that needs least software support.

The pitch is recovering money already spent. In a large fleet, memory provisioned per server for peak demand sits idle most of the time, and pooling converts that stranded capacity into usable capacity without new purchases, which is a rare kind of saving in this market. Against that, the switches, controllers, and integration work are real costs, and the benefit only appears at a fleet size where stranded memory is a large number. For inference specifically the value is narrower than the general case: it provides somewhere cheap to put cold state, which raises the number of conversations a machine can hold without raising the number it can actively serve.

Compute Express Link specification and overview (CXL Consortium) · Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv)

Scale-up fabric is the high-bandwidth interconnect joining accelerators inside a single node or rack so that they behave as one larger device. It is not a network in the usual sense: links are short, the topology is fixed at build time, and bandwidth per link is an order of magnitude above what cluster networking delivers. That bandwidth is what makes tensor parallelism viable, where a single matrix multiplication is split across several chips and partial results are exchanged on every layer. Split that way, the chips have to talk constantly, so the fabric is on the critical path of every token rather than being a background transfer.

The strength is that it turns a memory capacity problem into a solvable one. A model too large for one accelerator can be split across eight without the collapse in throughput that splitting across a slower link would cause. It also lowers latency at small batch, which is exactly where serving is hardest. The weakness is that it is proprietary and it does not extend. Bandwidth of this kind is achievable over a few meters, so the coherent domain stops at a node or a rack, and beyond that boundary the machine falls back to ordinary networking with an order of magnitude less bandwidth. That discontinuity shapes how every large model is partitioned.

Use tensor parallelism across a scale-up domain when a model does not fit on one accelerator, and keep it inside the domain, since crossing to the slower fabric with the same parallelism strategy performs badly. Beyond the domain boundary, switch to pipeline or data parallelism, which exchange far less per step. When buying, treat the size of the coherent domain as a first-order specification rather than a detail, because it decides the largest model that can be served without a partitioning strategy that costs throughput.

Per-link bandwidth roughly an order of magnitude above cluster networking · coherent domains typically spanning 8 accelerators in a node, extending to rack scale on recent designs · tensor parallelism exchanging data on every layer, so fabric latency enters every token · reach limited to a few meters, which is what sets the domain boundary · vendor-specific protocols rather than an interoperable standard.

NVIDIA's NVLink and its rack-scale switched form are the widely deployed case. AMD's Infinity Fabric fills the same role in its accelerator nodes, and TPU pods use a purpose-built interconnect that extends further than most. Ultra Accelerator Link is the industry effort to produce an open equivalent.

This is the most defensible part of an accelerator vendor's position, and it is sold as a system rather than a chip. A customer buying into a proprietary fabric is buying the node and the rack topology with it, which raises switching cost far above what the silicon alone would create, and it is why competitors have organized around an open standard rather than competing link by link. For the buyer, the practical consequence is that price comparison at the chip level is misleading: the relevant unit is the coherent domain, since that is what determines which models can be served and what the partitioning strategy has to be.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning (arXiv) · Efficiently Scaling Transformer Inference (arXiv)

Recent designs extend the scale-up domain from a node to a whole rack, joining dozens of accelerators through a switched fabric so that they present as one very large pool of memory and compute. Doing so pushes several engineering problems at once: the rack draws power measured in tens of kilowatts, the fabric switches themselves consume significant power and space, and the cabling has to carry aggregate bandwidth that would be impractical over any longer distance. Liquid cooling stops being optional at this density. What is bought in exchange is a coherent domain large enough to hold models that no node-scale domain can serve without falling back to slower networking.

The strength is that it moves the discontinuity outward. Models that would have required cluster-level partitioning fit inside one fast domain, and both training efficiency and serving latency improve accordingly. Larger domains also raise the batch sizes that fit, which improves the arithmetic intensity of decode directly. The weaknesses are facility-level. A rack at this power density cannot be installed in most existing datacenter halls without changes to power distribution and cooling, so the deployable market is narrower than the addressable one, and the failure domain grows: losing a rack now means losing a much larger unit of capacity.

It suits frontier training and high-volume serving of large models, where the domain size is the binding constraint and the operator controls a facility that can host it. It suits organizations that are power-constrained rather than capital-constrained less well, since the density concentrates demand rather than reducing it. Where an existing hall cannot supply the power or the liquid cooling, a larger number of conventional nodes is the realistic option even though it performs worse per chip.

Dozens of accelerators inside one coherent domain against roughly eight in a conventional node · rack power in the tens of kilowatts, well beyond the few kilowatts a traditional rack was designed for · liquid cooling required rather than optional · fabric switches consuming meaningful power and rack space themselves · failure domain expanding from a node to a rack, which changes checkpointing strategy.

NVIDIA's rack-scale systems are the most visible, and TPU pods have used large coherent domains for longer. The Open Compute Project's rack and power standards are the vehicle through which much of the facility side gets standardized, and the data-center infrastructure sheet covers the cooling and power delivery that these racks depend on.

The cost is not only the hardware but the building that can accept it. An operator with halls designed for a few kilowatts per rack faces a retrofit before deploying any of this, and that retrofit competes for the same capital, which is why deployment has concentrated among operators building new capacity rather than filling existing space. Against that, the density improves performance per dollar of chip meaningfully, so the comparison is between two very different capital structures rather than two prices. Grid interconnection is the constraint sitting behind all of it: at these densities, a site's available power decides how much can be deployed long before the budget does.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning (arXiv) · Trends in Artificial Intelligence (Epoch AI)

Beyond the scale-up domain, accelerators are joined by a cluster network. Two options dominate: InfiniBand, a purpose-built lossless fabric with its own protocol stack, and RDMA over Converged Ethernet, which carries the same remote memory access semantics over Ethernet. Both let one machine read and write another's memory without involving the remote processor, which is what keeps latency low enough for collective operations to scale. The distinction that matters is loss behavior. Machine-learning collectives perform badly with packet loss and retransmission, so the Ethernet path depends on congestion control and priority flow control configured correctly, and getting that wrong is a common cause of clusters that underperform their specification.

InfiniBand's strength is that lossless behavior and low latency are properties of the design rather than of the configuration, which is why it has dominated large training deployments. Its weakness is a narrow supplier base and a separate operational skill set. Ethernet's strengths are the opposite: many vendors, familiar operations, and a faster commodity cost curve. Its weakness is that achieving comparable collective performance requires careful tuning, and the industry's answer to that has been to standardize the required behaviors rather than leave them to each operator.

Use InfiniBand for large training clusters where scaling efficiency is the product and operational familiarity is secondary. Use Ethernet where the fleet is mixed, where inference dominates and collectives are less central, or where supplier diversity is a procurement requirement. For inference-heavy fleets the choice matters much less than it does for training, because serving exchanges far less data between machines. Whatever the choice, size the network against the collective pattern the training strategy actually produces rather than against aggregate bandwidth.

Remote direct memory access bypassing the remote processor, which is what keeps collective latency low · lossless operation intrinsic to InfiniBand and configuration-dependent on Ethernet · network typically a meaningful minority of cluster capital on a training build and far less on an inference fleet · scaling efficiency, rather than link speed, being the number that decides value · Ultra Ethernet organized specifically to standardize the behaviors that make Ethernet competitive for this traffic.

InfiniBand carries most published large training runs to date. Several hyperscalers have built very large training fleets on Ethernet instead, using their own congestion control. The Ultra Ethernet Consortium is the multi-vendor effort to make that path standard rather than proprietary.

Network is a minority of cluster capital but a majority of the risk to scaling efficiency, which makes it a poor place to economize. A cluster that reaches 90% scaling efficiency instead of 70% has effectively bought a fifth more accelerators for the price of better networking, and that arithmetic usually dominates the price difference between the two options. The strategic argument is supplier concentration: InfiniBand comes from essentially one vendor, and much of the Ethernet effort is funded by buyers who would rather not have their cluster economics set by that vendor's pricing.

9 minInfiniBand and RoCE: Artificial Intelligence Data Centers | FiberMallFiberMall · 10k+ views

About InfiniBand (InfiniBand Trade Association) · Ultra Ethernet Consortium

Distributed training is built from a small set of collective operations, of which all-reduce is the most important: every participant contributes a tensor, the values are summed, and every participant receives the result. Implemented naively that costs bandwidth proportional to the number of participants. Implemented as a ring, where each rank sends to its neighbor in two passes, the bandwidth each rank moves is roughly twice the tensor size regardless of how many ranks there are, which is what allows training to scale to thousands of chips. Tree and hierarchical variants trade bandwidth for latency on small messages, and libraries choose between them at runtime based on message size and topology.

The strength is that a well-implemented collective library hides an enormous amount of topology-specific engineering behind a stable interface, and gets most of the achievable performance without the model author knowing the network layout. Overlapping communication with computation, so gradients for early layers transfer while later layers are still computing, is what keeps scaling efficiency high. The weakness is that the abstraction leaks under stress: a slow link, a misconfigured congestion setting, or a straggler rank shows up as a collective that takes far longer than its size suggests, and diagnosing it requires exactly the topology knowledge the library exists to hide.

Any training run past one node depends on this, so the decision is which strategy generates the collectives rather than whether to use them. Keep tensor parallelism inside the scale-up domain, where the all-reduce per layer is affordable, and use data or pipeline parallelism across the slower cluster fabric, where the exchange happens per step rather than per layer. Profile the collectives directly when scaling efficiency disappoints, since the cause is more often a network or straggler problem than a compute one.

Ring all-reduce moving roughly twice the tensor size per rank regardless of rank count, which is what makes large-scale training tractable · tree and hierarchical algorithms chosen at runtime for small messages where latency dominates · overlapping communication with computation as the main lever on scaling efficiency · one straggler rank able to hold up an entire collective, so tail latency rather than average bandwidth sets the step time.

NCCL is the standard implementation on NVIDIA hardware and the reference for how these algorithms are selected in practice; RCCL is the equivalent elsewhere. Megatron-LM and ZeRO are the two published strategies that most production training stacks build on, and each generates a distinctly different collective pattern.

Collectives are where a cluster's advertised capacity turns into delivered capacity, and the gap is large. Scaling efficiency of 70% against 90% is a difference of roughly a quarter of the fleet, and it is recovered through library tuning, topology-aware placement, and stragglers being found and removed, all of which are operational rather than capital. That makes this one of the highest-return areas for engineering effort in large-scale training, and it is also why organizations without deep systems teams get less out of identical hardware than those with them.

NVIDIA Collective Communications Library (NVIDIA) · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (arXiv)

How the switches are wired decides what the collectives can achieve. A fat tree provides uniform bandwidth between any pair of endpoints and is the general-purpose answer, at the cost of a large amount of switching in the upper layers. Machine-learning traffic is not general purpose: it is dominated by a small number of predictable collective patterns, which allows a topology tuned to those patterns to deliver comparable performance with less switching. Rail-optimized designs exploit this by connecting each accelerator's network port to a separate switch plane, so a collective across corresponding ranks stays within one plane and never contends with the others.

The strength of a tuned topology is cost: matching the wiring to the traffic removes switching that a general-purpose design would need, and the savings at cluster scale are substantial. Placement-aware scheduling then keeps a job's ranks in positions where its collectives stay local. The weakness is rigidity. A topology tuned for one parallelism strategy performs worse on another, and a cluster shared between training and inference, or between teams with different strategies, gives back much of the benefit. Reconfigurable optical switching is one answer to that, at the cost of a great deal of complexity.

Tune the topology when the cluster is dedicated to training with a known parallelism strategy and is large enough for switching cost to matter, which usually means thousands of accelerators. Keep a conventional fat tree when the fleet is shared, mixed, or expected to run workloads that are not yet defined, since the flexibility is worth more than the switching saved. In every case, match job placement to the topology, because an optimal wiring with careless scheduling performs like neither.

Fat tree providing uniform bandwidth between all endpoints at the cost of substantial upper-layer switching · rail-optimized designs keeping each collective within one switch plane, cutting the switching required · machine-learning traffic dominated by a few predictable collective patterns rather than arbitrary pairs · optically reconfigurable topologies used at large scale to reshape the network per job · placement-aware scheduling required for any tuned topology to deliver its advantage.

The published TPU v4 work describes optical circuit switching used to reconfigure the topology for individual jobs, which is the most documented example of a network that changes shape to match the workload. Rail-optimized designs are now common in large GPU clusters, and most operators publish little about their exact layouts.

Switching is a real fraction of network capital, and network is a real fraction of cluster capital, so topology decisions move the total by a visible amount at scale. The deeper economic effect is on utilization rather than capital: a topology that forces jobs into poor placements leaves accelerators waiting on collectives, and idle accelerators are the most expensive thing in the building. That is why large operators invest in placement and scheduling well beyond what the hardware cost alone would justify, and why the flexibility of a general topology is often worth its extra switching in a fleet whose workloads keep changing.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning (arXiv) · NVIDIA Collective Communications Library (NVIDIA)

Sixteen-bit floating point comes in two arrangements that divide the same budget differently. FP16 gives 5 bits to the exponent and 10 to the mantissa, so it is precise over a narrow range. BF16 gives 8 bits to the exponent and 7 to the mantissa, keeping FP32's full range with much coarser precision. For training, range matters more than precision: gradients span many orders of magnitude and values that fall outside the representable range become zero or infinity, which ends a run. BF16 therefore trains without the loss scaling that FP16 requires, which is why it became the default despite representing each value less precisely.

The strength is that halving the bits halves memory and roughly doubles effective bandwidth, and on hardware with native support it raises throughput as well, for accuracy loss that is negligible on most training. Accumulation is usually performed in FP32 inside the matrix unit, so the reduced precision applies to storage and multiplication rather than to the running sum, which is what keeps the error acceptable. The weakness of FP16 is the range problem and the loss scaling machinery it forces on the training loop. BF16's weakness is the opposite: 7 mantissa bits are coarse enough that some numerically sensitive operations still have to be kept in higher precision.

Use BF16 as the default for training and fine-tuning on any hardware that supports it, and keep normalization, softmax, and the optimizer state in higher precision unless there is evidence they tolerate less. Use FP16 where the hardware supports it and BF16 is absent, accepting loss scaling. For inference, treat 16-bit as the reference point that lower precisions are measured against rather than as the endpoint, since most serving workloads have room to go further.

FP16 with 5 exponent bits and 10 mantissa bits, BF16 with 8 and 7 · BF16 keeping FP32's exponent range, so no loss scaling is needed in training · half the memory footprint and roughly double the effective bandwidth against FP32 · accumulation typically performed in FP32 inside the matrix unit even when inputs are 16-bit · a 70-billion-parameter model occupying roughly 140 GB at 16 bits.

BF16 originated in Google's TPU work and is now supported across essentially all training hardware. Mixed-precision training, where weights are stored in 16-bit and a master copy plus optimizer state is kept wider, is the standard recipe in every major framework.

This was the first large free lunch in the field and it has already been eaten: the move from FP32 to 16-bit halved memory and roughly doubled throughput across the board, and every current cost baseline assumes it. Its remaining economic significance is as the reference against which lower precisions are judged, since claims about FP8 and FP4 are claims about degradation relative to a 16-bit run. For hardware buyers, native BF16 support is now a floor rather than a differentiator, and a part without it is not competitive for training at any price.

FP8 Formats for Deep Learning (arXiv) · Introduction to Cloud TPU (Google Cloud)

FP8 packs a floating-point value into eight bits, and because that leaves so little to divide, two variants are used rather than one: E4M3 with four exponent bits for weights and activations, and E5M2 with five for gradients, which need more range. Neither has enough range to cover a whole tensor's dynamic range on its own, so FP8 in practice always comes with scaling, where a per-tensor factor moves values into the representable window before conversion and back out afterwards. Managing those factors is most of the implementation work, and getting them wrong produces silent overflow rather than an obvious error.

The strength is another halving of memory and doubling of effective bandwidth over 16-bit, with native throughput gains on hardware that supports the format. For inference it is close to free on most models, and it is now used in training with careful scaling. The weakness is that the margin for error is small: with three or two mantissa bits, values are represented coarsely enough that a mis-scaled tensor loses information irrecoverably, and the failure shows up as a slightly worse model rather than a crash. Support also varies, so a model quantized for one vendor's FP8 path may need requantizing for another.

Use it for inference on any model where a measured evaluation shows acceptable quality, which covers most current models. Use it in training when the framework supports the scaling machinery well and the run is long enough for the throughput to matter, and keep a higher-precision master copy of the weights. Do not adopt it on the strength of an aggregate benchmark score, since degradation appears first on long context, rare tokens, and multi-step reasoning, none of which move an average much.

E4M3 with 4 exponent and 3 mantissa bits for forward tensors, E5M2 with 5 and 2 for gradients · half the memory and roughly double the effective bandwidth of 16-bit · per-tensor or finer scaling required, since neither variant covers a full tensor's dynamic range alone · a 70-billion-parameter model occupying roughly 70 GB at 8 bits against 140 GB at 16 · quality loss usually small but concentrated in long-context and rare-token behavior.

The FP8 format specification was published jointly by several hardware and software vendors, which is why the two variants are consistent across implementations. Current training hardware supports it natively, and most production inference stacks offer it as a standard option.

FP8 is the cheapest remaining throughput increase available to an operator, because it is a software change against hardware already bought. Halving the bytes per parameter roughly doubles decode rate on a bandwidth-bound workload, which is a larger improvement than most hardware upgrades deliver and costs an evaluation run rather than capital. The reason it is not universal is that the evaluation is real work and the failure mode is quiet, so organizations that lack a good task-specific evaluation harness cannot safely take the saving. That makes evaluation infrastructure, rather than the format, the thing standing between many operators and a substantial cost reduction.

FP8 Formats for Deep Learning (arXiv) · MLPerf Inference: Datacenter benchmark results (MLCommons)

Four bits allow sixteen distinct values, which is far too few to represent a tensor with a single scale factor. Microscaling formats make it work by attaching a shared exponent to each small block of values, typically 32 elements, so the scale adapts across the tensor rather than being fixed for the whole of it. The block scale is stored alongside the data and costs a small overhead, giving an effective width somewhat above four bits. This is the arrangement that has made sub-8-bit inference practical on real models rather than only on benchmarks, and recent hardware implements it natively rather than emulating it.

The strength is memory and bandwidth: a quarter of 16-bit, which on a bandwidth-bound decode translates almost directly into tokens per second and decides whether a large model fits on one accelerator at all. The weakness is accuracy, and at this width it is no longer negligible. Weight-only quantization to four bits is tolerable for many models; taking activations down as well is materially harder, and the KV cache is more sensitive again. Formats also differ between vendors and tools, so a quantized artifact is less portable than a 16-bit one.

Use it when capacity or bandwidth is the binding constraint and an evaluation on the real task shows acceptable quality: serving a large model on fewer accelerators, or fitting a model onto edge hardware that could not otherwise hold it. Prefer weight-only quantization first, and treat activation and cache quantization as separate decisions with separate evidence. Avoid it for training, where the range and precision are insufficient, and avoid it where output quality is the product and the evaluation budget is not there.

Four bits giving sixteen representable values, with a shared scale per block of typically 32 elements · effective width somewhat above 4 bits once block scales are counted · a quarter of the memory of 16-bit, so a 70-billion-parameter model occupies roughly 35 GB · weight-only quantization to 4 bits close to lossless on many models, with activation and cache quantization progressively harder · native hardware support recent rather than established.

The microscaling formats were specified by a group of hardware and software vendors and are supported in recent accelerators. GPTQ and AWQ are the two widely used weight-only quantization methods that made 4-bit serving ordinary, and both are available in mainstream inference stacks.

This is the lever that decides how many accelerators a given model needs, and the arithmetic is stark: a model that fits on one accelerator at four bits and requires two at eight bits has roughly half the serving cost, before any throughput difference. For edge deployment it is often the difference between running on the device and not running at all. The cost is evaluation and the risk of quiet quality loss, which is why the technique spread fastest among operators serving their own products, where the quality tradeoff is theirs to make, and slower among vendors serving customers who specified a model rather than a quality level.

Microscaling Data Formats for Deep Learning (arXiv) · AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv)

Post-training quantization converts a model trained at 16-bit into a lower-precision form without retraining it. The naive approach, rounding every weight to the nearest representable value, degrades quality more than it needs to, because errors in different weights matter unequally. The methods in production use exploit that. One class solves for the rounding that minimizes the error in each layer's output on a small calibration set, correcting for the interaction between weights rather than treating them independently. Another observes that a small fraction of activation channels carry outsized magnitudes and protects the weights those channels touch, or migrates the difficulty from activations into weights where it is easier to handle.

The strength is that it takes hours on a single machine and needs no training data beyond a small calibration sample, which puts it within reach of any team that can run the model. It is the cheapest large cost reduction available in serving. The weakness is that the quality effect is model-specific and task-specific, and aggregate benchmarks hide it. A quantized model can score within a point of the original while behaving noticeably worse on long context or on the specific task a product depends on, and only a task-relevant evaluation reveals that.

Use it whenever a model is being served at volume and the evaluation infrastructure exists to check the result. Start with weight-only quantization, which is the safest and covers most of the memory saving, and treat activation and KV-cache quantization as separate steps with separate evidence. Prefer quantization-aware training only when post-training methods have been tried and the loss is unacceptable, since the cost is far higher. Do not ship a quantized model on benchmark averages alone.

Calibration sets of a few hundred samples typically sufficient, against the full training set for quantization-aware training · runtime measured in hours on one machine for models of tens of billions of parameters · weight-only quantization to 4 bits reported close to lossless on many models · a small fraction of activation channels carrying outsized magnitudes, which is what the outlier-aware methods target · degradation concentrated in long-context and rare-token behavior rather than spread evenly.

GPTQ solves layer-wise for the rounding that minimizes output error. AWQ protects the weights associated with the most significant activation channels. SmoothQuant migrates quantization difficulty from activations into weights, and LLM.int8() isolates outlier channels into higher precision. All four are implemented in mainstream serving stacks.

The return on this work is unusually high: an engineer-week of quantization and evaluation can halve or quarter the memory a served model occupies, which reduces the accelerator count for the same traffic in direct proportion. Compared with any hardware purchase, the ratio of saving to cost is not close. What limits adoption is the evaluation rather than the quantization, since an organization that cannot measure quality on its own task cannot safely take the saving and will keep serving at 16-bit. That makes a good task-specific evaluation harness a piece of cost infrastructure rather than a quality nicety.

12 minOptimize Your AI - Quantization ExplainedMatt Williams · 500k+ views

5 minWhat is LLM quantization?Airtrain AI · 10k+ views

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv) · SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (arXiv)

Most trained networks contain many weights that contribute little, and removing them reduces both storage and arithmetic. The difficulty is that unstructured sparsity, where any weight may be zero, does not help hardware built around dense matrix tiles: the zeros still occupy lanes. Structured sparsity solves this by constraining the pattern so hardware can exploit it, most commonly by requiring that exactly two of every four consecutive weights are zero, which allows the hardware to store only the non-zeros plus a small index and to skip the rest. That constraint costs some accuracy relative to unstructured pruning, and buys a speedup that actually appears.

The strength is a genuine reduction in both memory and arithmetic on supported hardware, composable with quantization, so the savings multiply rather than compete. The weakness is that the accuracy recovery usually requires retraining or fine-tuning after pruning, which is far more expensive than post-training quantization and puts it out of reach for teams that only have inference infrastructure. The achieved speedup is also well below the nominal ratio, because only the matrix multiplications benefit and the rest of the model does not.

Use it where the model is stable, the volume is high enough to repay a fine-tuning run, and the hardware supports the structured pattern natively. It suits an organization serving its own model at scale far better than one serving many third-party models, since each would need its own pruning and recovery. Prefer quantization first, since it is cheaper and gives a larger saving on bandwidth-bound work, and add sparsity on top only where the additional gain justifies the retraining.

Two-of-four structured pattern the common hardware-supported form, giving nominal halving of stored weights and matrix operations · realized speedup well below the nominal figure, since only matrix multiplication benefits · accuracy recovery typically requiring fine-tuning after pruning rather than being free · composable with quantization, so the memory savings multiply · unstructured sparsity giving better accuracy at a given ratio but no speedup on dense hardware.

The two-of-four pattern supported in recent accelerator generations is the widely deployed structured form, with published recipes for pruning and recovering accuracy. Mixture-of-experts routing is a different and now more influential kind of sparsity, applied at the level of whole subnetworks rather than individual weights.

Sparsity has delivered less than it once promised, and the reason is instructive: the saving is in arithmetic, while most inference is limited by memory bandwidth, so halving the operations on a bandwidth-bound workload changes little. Its economics are therefore best on compute-bound work, which is training and prefill rather than decode, and that is exactly where the retraining cost is hardest to justify. The result is a technique with real but narrow value, and it is largely why attention has shifted to quantization and to expert-level sparsity, both of which reduce bytes moved rather than operations performed.

Accelerating Sparse Deep Neural Networks (arXiv) · Efficiently Scaling Transformer Inference (arXiv)

A mixture-of-experts layer replaces one feed-forward block with many, and a small router selects a couple of them per token. Total parameters rise while the arithmetic per token stays close to that of a much smaller dense model, which decouples model capacity from compute cost. The catch is that capacity and compute are decoupled but memory is not: every expert has to be resident somewhere, so the full parameter count still has to be held even though only a fraction is used per token. That turns a compute problem into a memory and communication problem, which is the trade the architecture actually makes.

The strength is quality per unit of compute. A sparse model with far more total parameters can match or exceed a dense model while costing similar arithmetic per token, which is why the largest deployed models use this structure. The weaknesses are systems problems. Experts are distributed across accelerators, so routing sends tokens across the fabric and back on every such layer, and the resulting all-to-all traffic is sensitive to the interconnect. Load imbalance is the other persistent problem: routers concentrate tokens on popular experts unless deliberately regularized, leaving some accelerators idle while others queue.

Use it where quality per unit of training and serving compute is the objective and the deployment has a fast interconnect and enough aggregate memory to hold all experts. It suits large-scale providers and suits constrained deployments poorly, since the memory requirement does not shrink with the compute. For serving, batch size matters more than usual, because a small batch touches few experts and wastes the memory holding the rest.

Typically one or two experts activated per token out of many, so active parameters are a fraction of total · total memory set by all experts regardless of how few are used per token · all-to-all communication on every mixture layer, making the fabric part of the critical path · load-balancing regularization required, or routing concentrates on a subset of experts · published models reaching dense-equivalent quality at substantially lower compute per token.

GShard and Switch Transformers established the approach at scale and documented the load-balancing problem directly. Mixtral is a widely available open model using the structure, and most current frontier models are believed to use some version of it.

This is the main architectural reason serving costs have not risen in step with model capability. Decoupling parameters from arithmetic means a provider can raise quality without raising compute per token proportionally, and that shows up directly in price per token. The costs move rather than disappear: memory per served model goes up, the interconnect requirement goes up, and utilization becomes harder because imbalanced routing leaves hardware idle. The net has been strongly favorable at large scale and unfavorable at small scale, which is part of why serving economics increasingly favor operators large enough to keep every expert busy.

8 minWhat is Mixture of Experts?IBM Technology · 50k+ views

20 minA Visual Guide to Mixture of Experts (MoE) in LLMsMaarten Grootendorst · 50k+ views

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (arXiv) · Mixtral of Experts (arXiv)

CUDA is a programming model, a compiler, and a very large body of libraries built over nearly two decades. Its importance is not the language, which is a modest extension of C++, but everything layered above: the linear algebra, communication, and deep-learning primitive libraries that frameworks call, and the accumulated kernels written against them. A competing accelerator does not have to reproduce the language. It has to reproduce the libraries, the framework integrations, the profiling tools, and the tacit knowledge of thousands of engineers, and that is the part that has proved hard to replicate at speed.

The strength for a user is that everything works and everything is documented, which shortens the path from research to production more than any hardware advantage would. For the vendor it is the most durable competitive asset in the industry, because it compounds: more users produce more kernels, which make the platform more valuable. The weakness for a user is exactly that dependence, since it converts a hardware purchasing decision into a platform commitment. The weakness for the ecosystem is that portability layers consistently lag, so alternatives arrive with a real performance and coverage gap even when the silicon is competitive.

For most teams this is a default rather than a decision, and the useful question is how much exposure to accept. Writing against framework-level abstractions rather than vendor primitives keeps options open at some cost in peak performance, and is worth doing where the workload is not performance-critical. Writing custom kernels is worth it where a small number of operations dominate runtime, and that work should be expected to be vendor-specific. For a buyer evaluating an alternative accelerator, the question to ask is what fraction of the stack would have to be rewritten, since that is the real switching cost.

Nearly two decades of accumulated libraries, tools, and kernels rather than a language advantage · framework-level code often portable, kernel-level code rarely so · alternative platforms typically arriving with both a performance gap and an operator-coverage gap against equivalent silicon · translation layers usually reaching a fraction of native performance on unported code · the switching cost concentrated in a minority of the code that consumes the majority of runtime.

AMD's ROCm is the closest direct equivalent and has narrowed the gap most on the frameworks rather than at the kernel level. OpenCL and SYCL are the vendor-neutral standards. Triton is the most successful practical answer, in that it lets kernels be written once at a higher level and compiled to more than one target.

The software moat is worth more than any single generation of silicon, and it is the reason competing parts have repeatedly failed to take share despite competitive specifications. For an investor the useful test is where a customer's code sits: a stack written entirely against framework abstractions can move with an engineering sprint, while one with hand-tuned kernels cannot move at any reasonable cost. That distinction, rather than benchmark performance, is what predicts whether an alternative vendor can win an account. It also explains why the most credible challenges have come from organizations large enough to fund their own compiler and kernel work rather than from customers choosing on price.

3 minNvidia CUDA in 100 SecondsFireship · 1m+ views

CUDA C++ Programming Guide (NVIDIA) · OpenCL open standard for parallel programming (Khronos Group)

A machine-learning compiler takes a model expressed as a graph of operations and lowers it to code for a specific device, applying transformations along the way that a human would otherwise apply by hand. The most valuable of these is fusion: combining several operations so intermediate results stay in on-chip memory instead of being written to external memory and read back. Since most models are bandwidth-bound, removing those round trips is often worth more than any improvement to the arithmetic. Modern stacks are built on a common intermediate representation infrastructure so that multiple front ends and multiple hardware back ends can share the same passes.

The strength is that a compiler generalizes where a hand-written kernel does not: it fuses whatever graph it is given, including shapes and combinations nobody anticipated, which is what makes new model architectures usable on day one. Higher-level kernel languages have also made it possible to write a competitive kernel in far less code than the vendor-native path requires, which broadens who can do the work. The weakness is that compilers reliably lose to expert hand-tuning on the few operations that dominate runtime, so production stacks are a mixture: compiled code for the long tail, hand-written kernels for the critical few.

Rely on the compiler for everything by default, and profile to find the small number of operations that dominate. Hand-write or specialize those, and leave the rest. Use a portable kernel language where more than one hardware target matters, accepting some peak performance for the portability. For edge deployment, expect the compiler to matter more than on servers, since the runtime and operator coverage on device NPUs is thinner and what the compiler cannot lower simply will not run.

Fusion removing external memory round trips, which on bandwidth-bound work is usually the largest single win available · compilers typically reaching a good fraction of hand-tuned performance across a whole model while losing on individual hot kernels · higher-level kernel languages reducing kernel source size substantially against vendor-native code · a shared intermediate representation letting one set of passes serve several front ends and back ends · operator coverage rather than speed being the limiting factor on newer hardware.

XLA compiles graphs from several frameworks and is the production path for TPUs. MLIR is the shared infrastructure most current compilers are built on. Triton is the higher-level kernel language now widely used to write attention and other custom kernels that reach close to hand-tuned performance.

Compiler investment is how a hardware vendor converts silicon into a usable product, and it is consistently underestimated in cost and duration. The pattern across challengers is the same: competitive hardware arrives, the compiler covers the common operators, and the workloads customers actually run include enough uncovered cases that measured performance falls short of the specification. For customers, the compiler is what determines whether a new model runs the week it is published or the quarter after, and that lag has real commercial value in a field moving this fast. Shared infrastructure has lowered the entry cost, which is the main reason more credible alternative back ends exist now than five years ago.

OpenXLA Project · Triton documentation (Triton)

Beneath every framework sits a set of hand-written kernels for the operations that dominate runtime: matrix multiplication, attention, normalization, and the collective primitives. These are written by specialists against a specific architecture's memory hierarchy and instruction set, and the difference between a competent implementation and an expert one is routinely a factor of two or more on identical hardware. Attention is the clearest case. Restructuring it so that tiles remain in on-chip memory, rather than materializing the full attention matrix in external memory, produced large speed and memory improvements while computing exactly the same result.

The strength is leverage: a handful of kernels account for most of the runtime of most models, so effort concentrated there returns more than effort spread across a framework. Because the improvements are mathematically exact, they carry no quality tradeoff to evaluate. The weakness is that this work is architecture-specific and perishable. A kernel tuned for one generation's memory hierarchy needs revisiting for the next, and the pool of engineers who can do it well is small, which is a real constraint on how fast any alternative hardware platform can close its performance gap.

Use the vendor or community library version of every standard operation rather than a naive implementation, since the gap is large and the libraries are free. Write a custom kernel when profiling shows a single operation dominating runtime and no library version fits the shape, and expect the effort to be measured in engineer-weeks and to need maintenance across hardware generations. For unusual model architectures, budget for kernel work explicitly rather than assuming the library covers it.

A small number of operations accounting for most of the runtime of a typical model · expert kernels routinely delivering a factor of two or more over competent naive implementations · fused attention reporting substantial speedups and memory savings while producing identical results · kernels needing revision each hardware generation as the memory hierarchy changes · the supply of engineers able to do this work being a practical constraint on alternative platforms.

FlashAttention and its successor are the best-documented examples of the technique and are now standard across serving and training stacks. Vendor libraries for linear algebra, deep-learning primitives, and collectives are the layer that frameworks call by default, and community kernel collections have grown to cover much of what the vendor libraries do not.

Kernel engineering is the highest-leverage spending available to a large operator, because the gains apply to hardware already bought and require no additional capital. An engineer who improves a dominant kernel by 30% has effectively added 30% to the fleet's capacity for that workload. The constraint is people rather than money, since the skill is scarce and takes years to develop, which is why this capability concentrates in a small number of organizations and why open-sourced kernels have had outsized influence on the industry's overall efficiency.

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (arXiv) · Triton documentation (Triton)

A serving engine turns a model file into a service that answers many concurrent requests efficiently. The work it does is scheduling rather than arithmetic. Requests arrive independently, at different prompt lengths, and finish at different times, so a naive server that groups them into fixed batches wastes most of its capacity waiting for the slowest member. Continuous batching instead admits new requests into the running batch as others complete, keeping the accelerator busy. Combined with paged key-value cache allocation and prefix sharing, these scheduling decisions determine throughput far more than the choice of framework underneath.

The strength is that these are pure software gains against hardware already bought, and they are large: published results for paged allocation with continuous batching report multiples of throughput improvement over earlier serving approaches at equal quality. The engines also absorb quantization, speculative decoding, and distributed serving behind one interface. The weakness is operational complexity, since the scheduler now has many interacting parameters and the failure modes are subtle: a configuration that maximizes throughput can produce latency variance that a product cannot accept, and the tradeoff is not visible in a single number.

Use one for any production serving deployment; writing a scheduler by hand is not a reasonable use of engineering time. Choose based on the features that match the workload rather than on headline throughput: prefix sharing matters when many requests share a system prompt, structured output support matters when the product needs it, and multi-node serving matters only if the model does not fit on one machine. Measure the latency distribution rather than the mean, since the scheduling decisions that raise throughput usually widen the tail.

Continuous batching admitting new requests as others finish, rather than waiting for a whole batch · paged cache allocation with prefix sharing reported to raise serving throughput by a large multiple at unchanged hardware · time to first token governed by prefill and inter-token latency by decode, so the two have different bottlenecks · throughput and tail latency trading off against each other through batch size and admission policy · quantization, speculative decoding, and parallelism all exposed as engine configuration rather than model changes.

vLLM is the widely used open implementation and the origin of paged attention. TensorRT-LLM, SGLang, and several vendor stacks occupy the same position with different strengths. Most commercial inference providers run a modified version of one of these rather than something written from scratch.

This layer decides cost per token more directly than the hardware does, because it sets how many concurrent requests a given accelerator serves. Moving from naive batching to a modern engine has repeatedly delivered improvements that would otherwise have required several times the hardware, which makes it the first place to look when serving costs are too high. The reason it is not universally exploited is that tuning it well requires understanding the workload's arrival pattern and length distribution, and that knowledge sits with the product team rather than the infrastructure team, so the gains go unclaimed where the two do not talk.

vLLM documentation (vLLM) · Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv)

Training a model larger than one accelerator's memory requires splitting it, and there are three ways to do so that combine rather than compete. Data parallelism replicates the model and splits the batch, exchanging gradients each step. Tensor parallelism splits individual layers across chips, exchanging activations within every layer. Pipeline parallelism assigns different layers to different chips and streams microbatches through them. A fourth technique shards the optimizer state, gradients, and parameters across data-parallel ranks rather than replicating them, which removes the largest memory cost in training without changing the mathematics.

The strength is that the combination makes training at cluster scale possible at all, and the frameworks encapsulate enough of it that a team can run a large job without writing distribution code. Sharding optimizer state in particular removes a memory burden that is several times the model itself under common optimizers. The weakness is that the right combination depends on the model shape, the cluster topology, and the interconnect at every level, so the configuration space is large and the penalty for a poor choice is severe. Fault tolerance is the other weakness: at thousands of chips, failures are frequent enough that checkpointing strategy becomes part of the training plan.

Keep tensor parallelism inside the scale-up domain where bandwidth supports per-layer exchange. Use pipeline parallelism to cross slower boundaries, accepting the bubble that pipelining introduces. Use data parallelism outermost, with sharded optimizer state when memory rather than bandwidth is the constraint. Beyond that, follow the framework's published guidance for the model shape rather than deriving it fresh, and validate scaling efficiency on a small run before committing a large one.

Optimizer state under common optimizers costing several times the model's own parameter memory, which is what sharding removes · tensor parallelism exchanging data per layer and data parallelism per step, a difference of orders of magnitude in communication frequency · pipeline parallelism introducing an idle bubble that shrinks as microbatch count rises · scaling efficiency of 70–90% typical on well-configured large runs · failures frequent enough at thousands of chips that checkpoint interval is a planned tradeoff.

Megatron-LM established the tensor and pipeline parallel implementations most stacks still follow, and ZeRO established optimizer state sharding. Both are integrated into the major frameworks, and JAX offers a different route in which the partitioning is expressed as annotations and the compiler generates the communication.

The configuration chosen here decides how much of a cluster's capital delivers useful work, and the spread is wide enough to dominate hardware differences: a run at 60% scaling efficiency wastes a third of a fleet that a run at 90% would use. Because that gap is closed with expertise rather than equipment, organizations with strong systems teams get materially more out of identical hardware, and this is one of the clearest places where that advantage compounds. Checkpointing cost is the other economic term, since restart time after a failure is capacity lost, and at scale failures are routine rather than exceptional.

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv) · JAX documentation (JAX)

A training cluster is a fleet built around one job rather than many. Accelerators are joined by a high-bandwidth network, storage has to sustain the read rate the job demands, and the whole thing is scheduled as a unit because a job holding thousands of chips cannot be interleaved with other work. The distinguishing property against an inference fleet is synchronization: every step waits for the slowest participant, so a single degraded link, a thermally throttled chip, or a slow storage read holds up the entire run. That makes tail behavior rather than average performance the thing that has to be engineered.

The strength is that a single coherent fleet can train a model no smaller arrangement could, and scaling laws have made the relationship between compute spent and quality obtained predictable enough to plan around. The weaknesses are utilization and fragility. A cluster sized for one large job is poorly matched to anything else, so it is idle between runs unless deliberately shared, and failures at this scale are frequent enough that the effective throughput is materially below the nominal. Power is the binding constraint on building more, more often than capital now is.

Build or rent one when the objective is a model that cannot be trained any other way, and when the organization can keep it busy. Rent rather than build when the training need is episodic, since idle time on an owned cluster is the most expensive form of waste in this field. Where the work is fine-tuning rather than pretraining, a much smaller allocation is usually correct, and the tendency to over-provision for the largest job an organization might one day run is the common and expensive mistake.

Thousands of accelerators scheduled as one unit, since a synchronous job cannot share them · scaling efficiency of 70–90% typical, so nominal capacity overstates delivered capacity · failure rates at this scale making checkpoint and restart part of the throughput calculation rather than an exception · power in the megawatts, with grid interconnection now a common gate on expansion · published scaling laws relating compute, data, and model size closely enough to size a run in advance.

The compute-optimal scaling work relating model size to data budget is the standard reference for sizing a pretraining run, and the earlier scaling law work established the relationships it refined. Public tracking of frontier training compute shows the trend in cluster scale over time.

The dominant cost is accelerator capital and the dominant risk is idleness, which makes the utilization of a training cluster the number that decides whether it was a good purchase. Renting converts that risk into someone else's problem at a premium, and the crossover depends almost entirely on how continuously the organization trains. Power availability has become the constraint that decides where and when capacity can be added at all, ahead of both chip supply and capital in many markets. Depreciation assumptions compound the uncertainty, since hardware bought for frontier training retains value mainly through a second life serving inference.

Training Compute-Optimal Large Language Models (arXiv) · Trends in Artificial Intelligence (Epoch AI)

At thousands of accelerators running for weeks, component failures are routine rather than exceptional, and a synchronous training job cannot survive losing a participant. The standard response is periodic checkpointing: the model, optimizer state, and data position are written to storage often enough that a failure costs only the work since the last write. That write is not small, because optimizer state is several times the size of the model, so checkpointing is a substantial and recurring load on the storage system. The interval is a straightforward tradeoff, between time lost writing checkpoints and time lost redoing work after a failure.

The strength of frequent checkpointing is bounded loss: a failure costs at most one interval of progress. The weakness is that writing is expensive and, in the simplest implementation, synchronous, so the whole cluster waits. Asynchronous and sharded checkpointing address that by writing in the background and distributing the write across ranks, which is now standard. The remaining weakness is detection: a chip that has failed outright is easy to find, while one producing wrong results quietly or running slowly is not, and the second kind can corrupt a run or halve its speed without an obvious alarm.

Checkpoint on any run long enough that losing its progress would matter, which at cluster scale means all of them. Set the interval from the observed failure rate and the checkpoint cost rather than by convention, since both vary widely between fleets. Use asynchronous sharded writes if the framework supports them. Invest in detecting slow and silently faulty nodes separately from failed ones, because stragglers cost throughput continuously where a hard failure costs it once.

Optimizer state several times the model's size, so a checkpoint is much larger than the parameter count suggests · checkpoint interval set by balancing write cost against expected work lost per failure · asynchronous and sharded writing removing most of the synchronous stall · silent faults and stragglers harder to detect than hard failures and more damaging over a long run · storage bandwidth for checkpointing a real sizing input for the cluster rather than an afterthought.

The optimizer state sharding techniques used to reduce training memory also shard the checkpoint, which is why the two are usually implemented together. Every major training framework now ships asynchronous checkpointing, and large operators publish little about their failure rates, which makes this one of the harder areas to benchmark externally.

Fault tolerance is unglamorous and it directly sets the delivered throughput of an expensive asset. A cluster losing a meaningful share of its hours to restarts and stragglers is buying capacity it does not receive, and the fix is engineering and monitoring rather than more hardware. Storage sized for checkpoint bandwidth is a real capital line that gets underestimated in cluster budgets, and undersizing it converts into lost accelerator hours at a poor exchange rate. The strongest operators treat failure rate as a tracked metric with an owner, which is the practical difference between a cluster that delivers most of its nominal capacity and one that does not.

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv) · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (arXiv)

The two phases of generating a response have opposite characteristics. Prefill reads the prompt and computes its key-value state in one pass, doing a great deal of arithmetic per byte fetched, so it is compute-bound. Decode then produces tokens one at a time, reading the whole weight set per token, so it is bandwidth-bound. Running both on the same machine means each phase interferes with the other: a long prefill blocks decoding for waiting users, and interleaving them leaves whichever resource the current phase does not need idle. Disaggregation runs the two on separate pools, transferring the computed cache between them.

The strength is that each pool can be sized and configured for one job. Prefill machines can be chosen for arithmetic, decode machines for memory bandwidth and capacity, and neither phase's latency is held hostage by the other, which improves both time to first token and inter-token latency at the same time. The weakness is the transfer: the key-value state computed during prefill has to move to the decode machine, and that state is large, so the interconnect between pools becomes a design constraint. The architecture also adds operational complexity and only pays back at a scale where both pools stay busy.

Use it when serving at a scale where both pools can be kept loaded, and when the workload mixes long prompts with sustained generation, which is exactly where interference is worst. It suits inference providers and large product deployments. Avoid it at small scale, where the added complexity and the transfer cost exceed the interference being avoided, and where a single well-configured pool with continuous batching captures most of the available gain.

Prefill compute-bound and decode bandwidth-bound, so the two phases stress opposite resources · time to first token set by prefill and inter-token latency by decode, allowing each to be optimized separately · key-value state transferred between pools on every request, making the link between them a sizing constraint · published results reporting substantial improvements in serving goodput under latency constraints against colocated execution · benefit contingent on both pools staying busy, so it degrades at low utilization.

Splitwise and DistServe are the two published systems that established the approach and quantified the gain. Several commercial inference providers now run some version of it, and the major open serving engines have added support.

This is a scheduling change that converts into hardware savings, and the mechanism is straightforward: separating the phases lets each pool run near its own limit rather than each holding the other back, so the same request rate needs fewer machines. It also allows genuinely different hardware in each pool, which opens the possibility of buying cheaper high-bandwidth parts for decode and compute-dense parts for prefill rather than one part that compromises on both. That flexibility is the more interesting long-term consequence, because it creates a market position for silicon that would be uncompetitive as a general accelerator.

Splitwise: Efficient generative LLM inference using phase splitting (arXiv) · DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (arXiv)

The same model serves two economically different products. Interactive serving answers a user who is waiting, so latency is the specification and batch size is limited by how long requests may be held for company. Batch serving processes work with no one waiting, so requests can be accumulated into very large batches, and the arithmetic intensity of decode rises with them. Because decode at batch one is bandwidth-bound and batching amortizes the weight read across the whole batch, the cost per token of batch work is a fraction of the interactive cost on identical hardware.

The strength of batch serving is that it uses hardware efficiently, and it can be scheduled into whatever capacity interactive traffic leaves idle, which is substantial in any fleet sized for peak. The strength of interactive serving is that it is what most products need. The weakness of batching is that it is only available where the work is genuinely asynchronous, and the weakness of interactive serving is that the latency requirement forces small batches, which leaves the accelerator underused for reasons no configuration change can fix.

Route work to batch whenever the user is not waiting: evaluations, document processing, embedding generation, synthetic data, and offline classification all qualify and are frequently run interactively out of habit. Keep interactive serving for anything with a person on the other end, and use the latency budget deliberately, since even a few hundred milliseconds of admission delay allows meaningfully larger batches. Fill idle interactive capacity with batch work rather than letting it sit, which is the single largest efficiency gain available to a mixed fleet.

Cost per token on batch work a fraction of interactive cost on the same hardware, driven by batch size rather than any hardware difference · decode arithmetic intensity rising roughly in proportion to batch size until the key-value cache exhausts memory · interactive batch size bounded by the admission delay the product tolerates · fleets sized for peak interactive traffic leaving large idle capacity off-peak · commercial batch pricing commonly set at a substantial discount to interactive, reflecting the underlying difference.

Every major inference provider offers a discounted batch tier alongside interactive endpoints, and the discount is a direct reflection of the batching economics rather than a marketing choice. Internally, most large operators backfill idle serving capacity with evaluation and data-generation work for the same reason.

The gap between these two modes is one of the largest cost differences in serving, and it is available to anyone willing to classify their workload honestly. A great deal of work that is run interactively has no one waiting for it, and moving it to a batch path reduces its cost substantially with no change to hardware, model, or quality. The second lever is temporal: interactive demand has a daily shape, and the capacity provisioned for the peak is idle for much of the day unless batch work is queued to absorb it. Operators who do both run materially cheaper fleets than those who do neither.

Efficiently Scaling Transformer Inference (arXiv) · vLLM documentation (vLLM)

On-device inference runs the model on the user's hardware rather than on a server. The constraints are memory, power, and thermal: a phone has a few gigabytes available to an application and a budget of a few watts sustained before it throttles, which bounds model size to something well under what a server would run. The compensating advantages are that the inference costs the product company nothing per call, works without connectivity, and keeps data on the device, which matters for regulated categories and is increasingly a product claim in its own right.

The strength is unit economics and privacy at once, which is a rare combination. A feature that runs locally has no marginal serving cost, so usage can grow without a cost line growing with it. The weakness is capability, since the models that fit are small and quantized, and the gap against server models is large enough to be obvious on hard tasks. Fragmentation compounds it: each vendor's accelerator has its own runtime, quantization requirements, and operator coverage, so shipping across the installed base means validating several builds of the same model.

Use it where the task is narrow enough for a small model to do well, where latency or offline operation is part of the value, or where per-call server cost would not support the usage. Wake words, transcription, image processing, and classification fit. Use a hybrid where the device handles common cases and escalates the rest, which captures most of the cost saving while preserving quality on hard inputs. Do not use it where quality is the product, or where the model must change more often than the client can be updated.

A few gigabytes of memory available to an application on a phone against tens to hundreds on a server accelerator · sustained power budgets of a few watts before thermal throttling · models quantized to 8 bits or below to fit at all · zero marginal cost per inference against a per-call server cost that scales with usage · a separate runtime and validation pass per silicon vendor, so one model becomes several shipped artifacts.

Speech recognition, keyboard prediction, and photo processing are the mature high-volume cases. Recent phone and laptop platforms ship small language models in the operating system with developer access, which moves the fragmentation problem from the application to the platform.

The saving is structural rather than incremental: a feature that runs on the device has no cost line that grows with usage, which changes what can be offered for free and at what scale. For a consumer product with high call rates, that difference frequently decides whether the feature ships at all. The costs move into engineering and are paid per model rather than per user, in conversion, quantization, and per-vendor validation, and they recur every time the model is updated. The practical consequence is that on-device works best for features stable enough to amortize that pipeline across many releases.

Microscaling Data Formats for Deep Learning (arXiv) · ONNX open standard for model interchange (ONNX)

Cost per token is the unit economic of inference, and it is built from four numbers: what the hardware cost, how long it is depreciated over, how many tokens per second it produces on the actual workload, and how many hours a day it is busy. The first is public and the other three are not, which is why published comparisons of chips rarely predict what an operator actually pays. Prefill and decode also have to be costed separately, since prompt tokens are processed in parallel and generated tokens are not, and pricing that charges the same for both is averaging over a large underlying difference.

The strength of the metric is that it aggregates every other decision on this sheet into one comparable number, which is what makes it the right target for optimization. Its weakness is that it is easy to quote misleadingly. A figure computed at full utilization, maximum batch size, and a generous depreciation life describes a machine nobody runs, and the same hardware in production can cost several times more per token. Comparisons are only meaningful when utilization, batch size, precision, context length, and depreciation are stated together.

Use it as the objective when optimizing a serving fleet, and decompose it when it looks wrong, since the fix is usually in utilization or batch size rather than in the hardware. Use it with caution when comparing vendors, and insist on the four underlying assumptions. For build-versus-buy decisions, compare against a provider's batch tier rather than its interactive price, since that is closer to the underlying cost of the work.

Four inputs: acquisition cost, depreciation life, achieved tokens per second, and utilization · a three-year against a six-year depreciation assumption changing the capital component by roughly a factor of two · a fleet at 40% utilization costing roughly double per token against the same fleet at 80% · prefill and decode differing enough in cost that pricing them identically averages over a large gap · quantization from 16 to 8 bits roughly doubling decode throughput on bandwidth-bound work, and therefore roughly halving its share of cost.

Published inference pricing across providers spans a wide range for comparable models, and much of that spread reflects utilization and batching rather than hardware or margin. MLPerf inference results are the closest thing to a standardized throughput comparison, though they measure a benchmark rather than a production workload.

The largest levers are not the ones most discussed. Utilization and batching typically move cost per token more than the choice of accelerator does, and both are operational. Depreciation assumption moves it as much again and is a finance decision rather than a technical one. Hardware selection matters, but it enters as one term among four, which is why two operators running identical chips can report costs differing by a factor of two or more without either being wrong. For anyone evaluating a business in this market, the honest first question is what utilization the quoted economics assume.

MLPerf Inference: Datacenter benchmark results (MLCommons) · Efficiently Scaling Transformer Inference (arXiv)

Accelerators are the largest capital item in this industry and they are idle a great deal of the time. Two distinct measures matter and are often confused. Model FLOPs utilization is how much of a chip's peak a running job achieves, commonly 30–50% on a well-tuned training run. Fleet utilization is what fraction of the day the chips are running anything at all, and in fleets sized for peak interactive demand it can be considerably lower. Multiplying the two gives the fraction of purchased capability that turns into work, and that product is usually a much smaller number than either factor suggests.

Utilization is the cheapest capacity available, because raising it costs engineering rather than capital and the hardware is already paid for. Backfilling idle interactive capacity with batch work, improving kernels, and scheduling jobs to fit the topology all add effective capacity at a fraction of the cost of buying more. The weakness of pursuing it is that some idleness is deliberate: headroom for demand spikes and for failures is a service-level requirement rather than waste, and a fleet run at very high utilization has no margin when something breaks.

Measure both numbers before buying hardware, since the common outcome is that a fleet has more capacity than its utilization suggests. Treat a low model FLOPs utilization as a software problem and a low fleet utilization as a scheduling and product-mix problem, because the fixes are entirely different. Depreciation assumptions should be tested against how long hardware has actually stayed in service in the fleet rather than against an accounting default.

Model FLOPs utilization commonly 30–50% on tuned training runs and often lower on inference · fleet utilization in interactive-only fleets limited by the daily demand shape · the product of the two being the fraction of purchased capability that does useful work · depreciation over three against six years changing the capital cost per hour by roughly a factor of two · older accelerators retaining usefulness for inference well past the point they stop being competitive for training.

The pattern of retiring accelerators from training into inference service is what makes longer depreciation schedules defensible, and it is now standard practice among large operators. Public tracking of frontier compute gives some visibility into how long hardware generations stay in service.

Depreciation is where the accounting and the physics disagree, and the disagreement is worth money. A three-year schedule reflects competitive obsolescence for training; a six-year schedule reflects the physical life of a part that keeps serving inference. Which is right depends on whether a second life exists for the hardware, and that in turn depends on demand for older-generation inference capacity, which has so far been strong. An operator assuming a long life while the resale and reuse market weakens is overstating margin, and this is one of the clearer places where an investor can test whether reported economics are conservative.

Trends in Artificial Intelligence (Epoch AI) · MLPerf Inference: Datacenter benchmark results (MLCommons)

Datacenter accelerators are allocated rather than simply sold. Supply is gated by advanced packaging and high-bandwidth memory capacity rather than by logic wafer starts, and both of those expand on timescales of quarters to years, so demand above supply resolves through queueing rather than through price alone. The consequence is that an operator's capacity is often set by what it was allocated rather than by what it could afford, and terms of access, including prepayment and multi-year commitment, become the substance of the negotiation.

For the vendor this is a strong position and it has held longer than most such positions do, because the constraint is physical rather than commercial. For a buyer the practical strength is that a committed allocation is itself an asset and a competitive advantage. The weakness on both sides is planning risk. Committing to multi-year volumes in a field where model architecture and precision requirements shift within a year means committing to hardware whose suitability is uncertain, and the alternative of not committing means having no capacity at all.

Treat allocation as a strategic function rather than a procurement one at any scale where it binds. Where a workload can tolerate older hardware, buying previous-generation parts that are not allocation-constrained is frequently the better economics, particularly for inference. Renting capacity converts the allocation problem into a pricing problem and is usually correct for episodic demand. Diversifying across vendors has a real software cost, so it is worth doing before it is urgent rather than after.

Supply gated by advanced packaging and high-bandwidth memory rather than by logic capacity · lead times measured in quarters, with expansions of packaging capacity measured in years · multi-year commitments and prepayment common as terms of access · previous-generation parts frequently available without allocation constraints and adequate for much inference · capacity, rather than capital, repeatedly the binding constraint on operator growth.

The concentration of high-bandwidth memory supply among three vendors, and of advanced packaging among a smaller number of providers, is the structural reason the constraint persists across cycles. The semiconductor manufacturing sheet covers the packaging steps that create it.

When supply is the constraint, the ability to buy is worth more than the ability to pay, which inverts the usual relationship between capital and capacity. That has favored operators with long-standing supplier relationships and those willing to commit early, and it has made secured allocation a component of company valuation in a way that would be unusual in most hardware markets. The risk sitting under it is that the constraint eases: an operator holding multi-year commitments into a market with available supply has bought expensive capacity that competitors can now match at spot prices.

High-bandwidth memory (Micron) · Trends in Artificial Intelligence (Epoch AI)

Advanced accelerators are subject to export restrictions that limit which parts may be sold into which countries, and the thresholds are written against measurable device characteristics such as processing performance and interconnect bandwidth. Because the rules key on specifications rather than on product names, vendors have repeatedly designed parts that fall below a threshold in order to keep serving a restricted market, and regulators have repeatedly revised the thresholds in response. The result is a moving boundary that both hardware roadmaps and deployment plans have to be built around.

For a compliance function the strength of specification-based rules is that they are testable: a part's status can be determined from its datasheet rather than from a case-by-case judgment. The weakness is instability. A design taped out against one threshold can arrive after the threshold has moved, which is a two-to-three year exposure against a rule that has changed more often than that. Restrictions also extend beyond chips to the equipment that makes them and, in some formulations, to remote access, which broadens the compliance surface well past the hardware itself.

Anyone selling, deploying, or investing in this hardware across borders has to track it. Check where a product sits relative to current thresholds and how much margin it has, since a design close to a limit carries risk that is invisible in its performance figures. For deployment planning, confirm whether restrictions apply to the hardware, the service, or both, since cloud access has been treated differently from physical export at various times. Treat compliance advice as a legal function rather than a technical one.

Thresholds written against measurable characteristics such as processing performance and interconnect bandwidth rather than product names · rules revised several times, on a cadence shorter than a chip design cycle · a two-to-three year design-to-volume timeline against that shorter revision cadence · controls extending to manufacturing equipment and, in some formulations, to remote access to controlled hardware · compliance status determinable from specifications, which is what makes design-to-threshold possible.

Successive US rules on advanced computing have adjusted the thresholds and closed gaps that earlier versions left, and vendors have shipped market-specific parts in response each time. The published guidance from the administering agency is the authoritative source, and it changes often enough that a citation is a snapshot rather than a standing description.

Export policy has become a first-order input to hardware roadmaps rather than a downstream compliance matter, and it shows up in the economics three ways. It removes addressable market, which affects volume assumptions directly. It creates design work for market-specific parts that carry the same fixed cost across smaller volumes. And it has driven substantial state-backed investment in domestic alternatives inside restricted markets, which changes the competitive landscape on a longer horizon than the restrictions themselves. For an investor the practical question is how much of a company's forecast depends on markets that policy could close.

Trends in Artificial Intelligence (Epoch AI) · MLPerf Inference: Datacenter benchmark results (MLCommons)