Benchmarking Terminology for Large Language Model Serving

End-to-end latency encompasses all processing stages: network transmission, queuing, prefill computation, decode iterations, output filtering, and return transmission. For streaming responses, end-to-end latency is measured until the final token is received.¶

End-to-end latency depends on output length. Longer responses require more decode iterations and produce higher latency. When comparing systems, testers SHOULD control for output length or report latency normalized by output token count.¶

Client-side measurement includes network round-trip time. Server-side measurement excludes external network latency but may still include internal network hops within a distributed serving system.¶

• Measurement point (client-side vs server-side)¶
• Treatment of failed or truncated responses¶
• Clock synchronization for distributed measurement¶

Time to First Token (TTFT) measures how long a user waits before any response content appears. For interactive applications, TTFT determines perceived responsiveness independent of total response length.¶

TTFT includes network transmission, authentication, admission control, queue wait time, and prefill computation. Under low load with short prompts, prefill dominates TTFT. Under high load, queue wait time may dominate. With long prompts, prefill computation scales with input token count.¶

For non-streaming responses, TTFT equals end-to-end latency because all tokens arrive together. Testers SHOULD specify whether the response mode is streaming or non-streaming.¶

Some systems emit an empty or whitespace-only first token before substantive content. Testers MUST specify whether TTFT measures time to any token or time to first non-empty token.¶

• Streaming vs non-streaming response modes¶
• Definition of "first token" when initial tokens lack content¶
• Client-side vs server-side measurement point¶

Inter-Token Latency (ITL) measures the generation interval between adjacent tokens during the decode phase. ITL reflects decode efficiency and is affected by batch size, model architecture, and memory bandwidth.¶

ITL varies across tokens within a single request due to batching dynamics. When other requests join or leave the batch, the per-request compute allocation changes. Testers SHOULD report ITL distribution statistics rather than a single value.¶

ITL is measured server-side and excludes network transmission delay. For client-observed intervals, see Time Between Tokens (Section 4.1.4).¶

When aggregating ITL across requests, token-weighted averaging counts each token equally. Request-weighted averaging counts each request equally regardless of length. These methods yield different results. Testers MUST specify the aggregation method.¶

• Token-weighted vs request-weighted aggregation¶
• Variation within a single request due to batching¶
• Exclusion of prefill-to-first-token interval¶

Time Between Tokens (TBT) measures the client-observed interval between adjacent tokens. TBT equals ITL plus network transmission variability and buffering effects.¶

Network jitter, TCP buffering, and intermediary proxies cause TBT to differ from ITL. Multiple tokens may arrive in a single network packet, producing near-zero TBT followed by a longer gap.¶

TBT directly affects user-perceived streaming smoothness. High TBT variance creates a "stuttering" appearance even when average TBT is acceptable.¶

• Network buffering causing token bunching¶
• Proxy and CDN effects on delivery timing¶
• Measurement requires client-side instrumentation¶

The average time to generate each output token after the first token, computed as:¶

TPOT = (End-to-End Latency - TTFT) / (Output Token Count - 1)¶

Time per Output Token (TPOT) summarizes decode-phase performance in a single value. Unlike ITL, which measures each interval, TPOT averages across all decode steps for a request.¶

TPOT excludes the first token because TTFT captures that interval separately. For single-token outputs, TPOT is undefined.¶

TPOT is request-weighted by construction: each request contributes one TPOT value regardless of output length. When aggregating across requests, report the distribution rather than only the mean.¶

TPOT relates to user-perceived generation speed. A TPOT of 50ms corresponds to 20 tokens per second, approximately 900 words per minute for English text.¶

• Undefined for single-token outputs¶
• Request-weighted aggregation differs from token-weighted ITL¶
• Denominator uses (Output Token Count - 1)¶

Normalized latency enables comparison across requests with different output lengths. It amortizes fixed overhead (TTFT) across all tokens.¶

For short outputs, TTFT dominates normalized latency. For long outputs, TPOT dominates. Testers SHOULD report output length distribution alongside normalized latency to enable interpretation.¶

Normalized latency obscures user-facing behavior because users experience TTFT and TPOT separately. Testers SHOULD NOT report normalized latency as the sole latency metric.¶

• Obscures distinction between TTFT and decode latency¶
• Sensitive to output length distribution¶
• Not directly interpretable as user experience¶

Prefill performs a forward pass over all input tokens to populate the key-value cache. This computation is parallelizable across the input sequence and is compute-bound on current hardware.¶

Prefill latency scales approximately linearly with input token count for uncached inputs. With prefix caching enabled, prefill processes only the uncached suffix, reducing latency for requests sharing common prefixes.¶

Prefill latency is a component of TTFT. Other TTFT components include queue wait time, network latency, and authentication overhead.¶

Chunked prefill implementations split long prefills into smaller segments interleaved with decode steps. This reduces head-of-line blocking but increases total prefill time. Testers MUST specify whether chunked prefill is enabled.¶

• Chunked vs monolithic prefill execution¶
• Prefix caching effects on effective prefill length¶
• Distinction from TTFT in reporting¶

Decode latency equals (Output Token Count - 1) multiplied by average ITL. It measures total time in the decode phase.¶

Decode is memory-bandwidth-bound on current hardware because each step reads the full model weights and growing key-value cache while producing a single token.¶

Decode latency grows linearly with output token count under constant batching conditions. Variable batch membership during generation causes decode latency to deviate from simple linear scaling.¶

• Variation due to dynamic batching¶
• Growth of key-value cache during decode¶

Output token throughput measures system-wide generation capacity. It increases with offered load until the system saturates, then plateaus or declines.¶

Throughput measurement requires specifying the token counting method. Subword tokenization produces different counts than word-level tokenization for the same text. Testers MUST identify the tokenizer used.¶

Some implementations pad sequences to uniform length within a batch. Testers MUST specify whether throughput counts include or exclude padding tokens.¶

Throughput measurement requires a defined time window. Short windows capture instantaneous throughput. Longer windows smooth over load variations. Testers MUST specify the measurement window duration.¶

• Tokenizer-dependent token counts¶
• Inclusion or exclusion of padding tokens¶
• Measurement window duration¶

Input token throughput measures prefill capacity. Systems with efficient batched prefill achieve higher input throughput than those processing prompts sequentially.¶

Input throughput differs from output throughput because prefill and decode have different computational characteristics. A system optimized for long-context prefill may show high input throughput but lower output throughput, and vice versa.¶

Prefix caching affects input throughput measurement. With caching, input tokens divide into cache hits (not processed) and cache misses (processed). Testers MUST specify whether input throughput counts all input tokens or only cache misses.¶

• Treatment of cached vs processed input tokens¶
• Different from output throughput due to phase characteristics¶

Request throughput counts completed requests regardless of token counts. A system completing many short requests achieves higher request throughput than one completing fewer long requests, even at equal token throughput.¶

Request throughput is relevant for capacity planning when requests have predictable lengths or when per-request overhead dominates.¶

Failed requests require specified handling. Testers MUST specify whether request throughput includes only successful completions or also counts failures.¶

• Treatment of failed or truncated requests¶
• Sensitivity to request length distribution¶

Batched inference pads sequences to uniform length. Padding tokens consume compute but carry no information. Non-padding throughput measures useful work.¶

Systems using variable-length batching or continuous batching may avoid padding entirely. For these systems, non-padding throughput equals total output throughput.¶

• Applicable only to padded batching schemes¶

Offered load characterizes workload intensity. Two forms exist:¶

Open-loop load specifies a request arrival rate (requests per second) independent of system response. New requests arrive according to a specified distribution regardless of outstanding request count.¶

Closed-loop load specifies a fixed concurrency level. Each completed request triggers a new request, maintaining constant outstanding requests.¶

Open-loop load reveals system behavior under overload. Closed-loop load cannot exceed system capacity by construction. Testers MUST specify the load model and its parameters.¶

• Open-loop vs closed-loop model selection¶
• Arrival distribution for open-loop (Poisson, uniform, bursty)¶
• Think time between requests for closed-loop¶

Sustainable load identifies the operating region boundary. Below sustainable load, the system meets latency and quality targets. Above sustainable load, latency increases unboundedly or requests fail.¶

Sustainable load depends on the service objectives. Stricter latency targets yield lower sustainable load. Testers MUST declare the service objectives when reporting sustainable load.¶

Sustainable load also depends on workload characteristics. Longer prompts or outputs reduce sustainable load. Testers MUST characterize the workload profile.¶

• Requires declared service objectives¶
• Sensitive to workload characteristics¶
• May differ for different SLO percentiles¶

Mean latency obscures distribution shape. A system with low mean but high variance provides inconsistent user experience. Percentiles characterize the distribution.¶

P50 (median) indicates typical experience. P99 indicates worst-case experience for most users. P99.9 indicates extreme tail behavior relevant for high-volume services.¶

Percentile computation requires sufficient sample size. For P99 accuracy, at least 1000 samples are needed. For P99.9, at least 10000 samples. Testers MUST report sample size alongside percentiles.¶

Percentiles apply to TTFT, TPOT, ITL, and end-to-end latency. Testers SHOULD report percentiles for multiple metrics rather than a single summary.¶

• Sample size requirements for tail percentiles¶
• Appropriate percentile selection for use case¶
• Confidence intervals for reported percentiles¶

Full distributions reveal structure that percentiles miss. Multimodal distributions indicate distinct operating regimes. Heavy tails indicate outlier sensitivity.¶

Histogram bin width affects resolution. Narrow bins reveal detail but require more samples. Testers SHOULD use logarithmic binning for latency distributions spanning multiple orders of magnitude.¶

• Bin width selection¶
• Sample size for distribution estimation¶

Jitter measures streaming smoothness. Low jitter indicates consistent token pacing. High jitter indicates irregular delivery that users perceive as stuttering.¶

Jitter arises from batching dynamics, memory bandwidth contention, and garbage collection pauses. Systems with continuous batching show higher jitter than static batching due to variable batch membership.¶

Jitter is computed per-request, then aggregated. Report the distribution of per-request jitter values.¶

• Aggregation method across requests¶
• Separating server-side jitter from network jitter¶

Maximum pause captures the worst interruption in streaming output. A single long pause degrades user experience even when average ITL is acceptable.¶

Long pauses arise from garbage collection, KV cache operations, batch recomputation after preemption, and request scheduling delays.¶

• Distinguishing generation pauses from network delays¶
• Threshold for "pause" vs normal variation¶

First-come-first-served scheduling causes head-of-line (HOL) blocking. A long prefill operation delays all subsequent requests regardless of their size.¶

Chunked prefill mitigates HOL blocking by limiting the maximum uninterruptible computation. Shortest-job-first scheduling eliminates HOL blocking but requires output length prediction.¶

HOL blocking is measured as the difference between observed latency and latency under an idealized scheduler with no blocking.¶

• Requires baseline comparison to quantify¶
• Depends on workload length distribution¶

Queue depth indicates load relative to capacity. Growing queue depth signals approaching overload. Stable queue depth indicates balanced load.¶

Queue depth has multiple measurement points: admission queue, prefill queue, decode batch wait queue. Testers MUST specify the queue measured.¶

• Multiple queue stages in serving systems¶
• Instantaneous vs time-averaged measurement¶

Queue wait time is a component of TTFT representing time spent waiting rather than computing. Under low load, queue wait approaches zero. Under high load, queue wait dominates TTFT.¶

Systems with multiple queues (admission, prefill, decode) have corresponding wait times. Total queue wait is the sum across stages.¶

• Multiple queue stages¶
• Inclusion of admission control delay¶

Jain's Fairness Index quantifies allocation equality:¶

J(x) = (sum(x_i))^2 / (n * sum(x_i^2))¶

where x_i is the allocation to request or tenant i, and n is the count. J ranges from 1/n (maximally unfair) to 1 (perfectly fair).¶

Fairness applies to latency (lower is better, so use reciprocal), throughput, or SLO attainment. Testers MUST specify the measured quantity.¶

Multi-tenant systems require per-tenant fairness measurement. Single-tenant systems measure fairness across concurrent requests.¶

• Choice of measured quantity (latency, throughput, SLO attainment)¶
• Tenant definition in multi-tenant systems¶

Static batching pads sequences to uniform length. Batch utilization measures the fraction of compute applied to real tokens versus padding.¶

Continuous batching achieves high utilization by avoiding padding. For continuous batching systems, batch utilization approaches 1.0 and is less informative.¶

• Not meaningful for continuous batching systems¶
• Varies with workload length distribution¶

Admission control rejects requests to prevent overload. Admission rate below 1.0 indicates load shedding.¶

Rejected requests may receive an error response or be redirected. Testers MUST specify the rejection behavior.¶

• Rejection behavior (error vs redirect)¶
• Distinction from content-based refusals¶

Memory pressure or priority policies cause request preemption. Preempted requests lose their key-value cache state and must recompute it upon resumption.¶

High preemption rates indicate memory over-subscription or aggressive scheduling. Preemption degrades latency for affected requests.¶

• Distinction between temporary preemption and permanent eviction¶
• Treatment of requests that are preempted multiple times¶

Preemption discards key-value cache state. Upon resumption, the system recomputes prefill for all tokens (input plus previously generated output). This recomputation is wasted work.¶

Preemption loss contributes to tail latency. Requests preempted late in generation lose more work than those preempted early.¶

• Measurement requires tracking recomputed tokens¶
• Multiple preemptions accumulate loss¶

Starvation occurs when scheduling policies indefinitely delay certain requests. Priority inversion and unbounded queue growth cause starvation.¶

The starvation threshold depends on application requirements. Testers MUST declare the threshold when reporting starvation rate.¶

• Threshold selection is application-dependent¶
• Distinction from queue wait time distribution tail¶

Recovery latency includes: wait time for scheduling, KV cache reload or recomputation, and prefill of previously generated tokens.¶

Systems with KV cache offloading to host memory recover faster than those requiring full recomputation. Recovery latency varies with the amount of generated output at preemption time.¶

• Variation based on progress at preemption¶
• Offloading vs recomputation recovery strategies¶

Memory-constrained systems swap KV cache to host memory when accelerator memory is exhausted. Swapping enables higher concurrency at the cost of swap latency.¶

Swap rate indicates memory pressure. High swap rates degrade latency due to PCIe transfer overhead.¶

• Granularity of swap operations (per-request vs per-block)¶
• Distinction between swap-out and swap-in rates¶

Paged attention systems fault in KV cache blocks on demand. Page fault latency includes PCIe transfer time from host memory or storage.¶

Fault latency increases ITL for affected decode steps. Prefetching strategies aim to hide fault latency.¶

• Prefetching effectiveness¶
• Storage tier latency (DRAM vs SSD)¶

Prefix caching stores KV cache state for common prompt prefixes. Subsequent requests sharing a cached prefix skip prefill for those tokens.¶

Hit rate depends on workload locality. Workloads with shared system prompts achieve high hit rates. Workloads with unique prompts achieve low hit rates.¶

Hit rate is computed as cached tokens divided by total input tokens across requests. Per-request hit rate varies; report the distribution.¶

• Granularity of cache matching (exact prefix vs subsequence)¶
• Multi-tenant cache isolation¶

Cache capacity limits the number and length of prefixes stored. Larger capacity enables more prefixes or longer prefixes at the cost of memory available for active requests.¶

Capacity is often expressed as a fraction of total accelerator memory or as maximum cacheable token count.¶

• Trade-off with memory for active requests¶
• Dynamic vs static allocation¶

Eviction occurs when cache capacity is exhausted. High eviction rates indicate insufficient capacity for the workload's prefix diversity.¶

Eviction policies (LRU, frequency-based) affect which prefixes remain cached. Testers SHOULD specify the eviction policy.¶

• Eviction policy effects¶
• Distinguishing capacity eviction from staleness eviction¶

This metric quantifies caching benefit. It depends on both hit rate and the length of cached prefixes.¶

Measurement requires comparing TTFT with caching enabled versus disabled, or estimating based on prefill latency per token.¶

• Requires baseline measurement without caching¶
• Varies with cached prefix length¶

Speculative decoding uses a smaller draft model to propose multiple tokens verified in parallel by the target model. Higher acceptance rates yield greater speedup.¶

Acceptance rate depends on draft model quality and alignment with the target model. Rates vary by domain; code and structured text achieve higher rates than creative writing.¶

Acceptance rate is computed per speculation window, then averaged. Report the distribution across windows.¶

• Variation across domains and prompts¶
• Dependence on draft model selection¶

Speedup depends on acceptance rate and the relative cost of draft and target model inference. High acceptance with low draft cost yields high speedup.¶

Speculative decoding increases TTFT due to draft model prefill. Speedup applies to the decode phase only. End-to-end speedup is lower, especially for short outputs.¶

• Excludes TTFT overhead¶
• Varies with acceptance rate¶

Draft model inference adds to prefill time and per-step decode time. This overhead must be recovered through acceptance to achieve net speedup.¶

Overhead is measured as additional latency per speculation window or as fraction of total compute.¶

• Amortization over speculation window length¶
• Memory overhead for draft model weights¶

Verification throughput measures the target model's capacity to check draft proposals. Higher verification throughput enables longer speculation windows.¶

Verification processes multiple tokens in parallel, achieving higher throughput than autoregressive generation of the same tokens.¶

• Distinction from generation throughput¶
• Variation with speculation window length¶

Retrieval-Augmented Generation (RAG) systems embed queries before searching a vector store. Embedding latency adds to TTFT.¶

Embedding models are smaller and faster than generation models. Embedding latency is a minor TTFT component for most deployments.¶

Batched embedding of multiple queries achieves higher throughput than sequential embedding.¶

• Batched vs sequential embedding¶
• Embedding model selection effects¶

Retrieval latency includes vector similarity search, optional reranking, and document fetching. It is a component of TTFT for RAG systems.¶

Retrieval latency depends on index size, search algorithm, and number of results. Approximate nearest neighbor search trades accuracy for speed.¶

• Index size and algorithm effects¶
• Reranking inclusion¶

Recall measures retrieval effectiveness. Low recall causes the generation model to lack relevant context. This is a quality metric, not a performance metric, but affects overall system evaluation.¶

Measuring recall requires ground-truth relevance labels. For benchmarking without labels, use proxy metrics such as answer correctness.¶

• Requires ground-truth relevance labels¶
• Trade-off with retrieval latency¶

Retrieved documents increase prompt length, increasing prefill computation. This overhead scales with retrieved token count.¶

The overhead is the difference between prefill latency with retrieved context and prefill latency without it.¶

• Varies with retrieval result length¶
• Interaction with context length limits¶

Not all retrieved content contributes to generation. Low utilization indicates retrieval of irrelevant content, wasting context window capacity and prefill compute.¶

Utilization is difficult to measure directly. Proxy measurements include attention weight analysis and ablation studies.¶

• Indirect measurement required¶
• Attribution challenges¶

Agentic systems perform multiple internal operations per user request. Task completion latency measures the full user-facing response time.¶

Task completion latency depends on the number of internal steps, which varies by task complexity. Simple tasks complete in one LLM call. Complex tasks require multiple calls with tool use.¶

• Variation with task complexity¶
• Definition of task completion for open-ended tasks¶

Agentic systems decompose user requests into multiple LLM calls for planning, reasoning, and action. Sub-request count indicates system complexity and affects total latency and cost.¶

High sub-request counts indicate complex reasoning chains or retry loops. Testers SHOULD examine sub-request count distributions to identify inefficient patterns.¶

• Includes retries and failed attempts¶
• Varies with task complexity¶

Agentic loops occur when the system repeats similar actions without advancing toward task completion. Loops indicate planning failures or tool errors.¶

Loop detection requires defining "progress" for the task domain. Common heuristics include action repetition count and state similarity thresholds.¶

• Progress definition is task-dependent¶
• Distinguishing loops from legitimate retries¶

Agents call external tools for information retrieval, computation, or actions. Tool latency contributes to task completion latency.¶

Tool latency varies by tool type. Local computation completes in milliseconds. External API calls require seconds.¶

• High variance across tool types¶
• External dependencies outside system control¶

Agentic goodput combines completion rate and quality. A task that completes but produces incorrect results does not count toward goodput.¶

Objective definitions are task-specific. Testers MUST declare objectives and evaluation criteria.¶

• Task-specific objective definitions¶
• Evaluation criteria for correctness¶

Safety systems filter outputs to prevent harmful content. Policy violation rate measures filter failure.¶

Violation detection requires evaluation against policy criteria. Automated classifiers or human review provide measurements.¶

Low violation rate indicates effective filtering. Very low rates may indicate overly restrictive filtering causing false refusals.¶

• Policy definition and scope¶
• Detection method reliability¶

Overly sensitive safety filters refuse legitimate requests. False refusals degrade user experience and system utility.¶

Measuring false refusals requires labeled benign requests. Testers MUST specify the evaluation dataset and labeling criteria.¶

False refusal rate trades off against policy violation rate. Stricter filtering reduces violations but increases false refusals.¶

• Requires labeled benign test set¶
• Trade-off with policy violation rate¶

Guardrails add processing before, during, or after generation. Input filters add to TTFT. Output filters add to end-to-end latency.¶

Overhead is measured by comparing latency with guardrails enabled versus disabled.¶

• Baseline measurement without guardrails¶
• Multiple guardrail stages with distinct overhead¶

SLOs specify targets such as:¶

• P99 TTFT below 500ms¶
• P95 TPOT below 50ms¶
• Error rate below 0.1%¶

SLOs derive from user experience requirements and business constraints. Different applications require different SLOs.¶

SLO definitions include the metric, percentile, threshold, and measurement window. Testers MUST fully specify SLOs when reporting attainment.¶

• Application-specific requirements¶
• Multiple SLOs may apply simultaneously¶

Attainment rate summarizes SLO compliance. Request-based attainment counts requests meeting SLOs. Time-based attainment counts measurement windows where aggregate metrics meet SLOs.¶

Attainment below 1.0 indicates SLO violations. The acceptable attainment level depends on SLA commitments.¶

• Request-based vs time-based measurement¶
• Handling of multiple simultaneous SLOs¶

Goodput equals total throughput multiplied by SLO attainment rate. It measures useful, compliant work rather than raw capacity.¶

Systems with high throughput but low attainment achieve low goodput. Goodput captures the throughput-quality trade-off.¶

• Requires defined SLOs¶
• May use request or token basis¶

Energy per token enables efficiency comparison across systems and sustainability analysis. The value depends on model size, hardware, batch size, and workload.¶

Measurement requires power monitoring integrated with token counting. GPU power is accessible via vendor APIs (NVML for NVIDIA). Total system power requires external instrumentation.¶

Energy per token differs between prefill and decode phases. Testers SHOULD report phase-separated energy when feasible.¶

• Measurement boundary (GPU vs system)¶
• Phase attribution¶

Power draw varies with load. Idle systems consume less power than systems under load. Prefill phases consume more power than decode phases due to higher compute utilization.¶

Testers MUST specify the measurement boundary: GPU only, GPU and CPU, or entire system including cooling.¶

• Measurement boundary specification¶
• Sampling rate for time-varying power¶

• Sampling rate may miss brief spikes¶
• Relationship to thermal design power (TDP)¶

Carbon intensity equals energy consumption multiplied by the grid's carbon factor (gCO2e/kWh). Grid carbon intensity varies by location and time.¶

Testers reporting carbon metrics MUST specify the grid carbon factor used.¶

• Grid carbon factor variation¶
• Scope of emissions (operational only or including embodied)¶

Cost includes compute (hardware amortization or rental), energy, and operations. Testers MUST specify included cost components.¶

Cloud pricing provides a market cost reference. Self-hosted deployments require cost modeling.¶

• Cost component inclusion¶
• Pricing model assumptions¶

GPU-hours measures resource consumption independent of hardware cost. For multi-GPU deployments, report aggregate GPU-hours.¶

GPU-hours equals end-to-end latency multiplied by GPU count used for the request.¶

• Fractional allocation in multi-tenant systems¶
• GPU count for distributed inference¶

• Cost basis selection¶
• Throughput metric selection¶

Compute utilization indicates how effectively the system uses accelerator compute resources. Low utilization suggests memory bandwidth or scheduling bottlenecks.¶

For GPUs, utilization metrics include SM occupancy and tensor core utilization. Testers MUST specify the utilization metric.¶

• Multiple utilization definitions (SM, tensor core, etc.)¶
• Vendor-specific measurement tools¶

• Measurement tool availability¶
• Cache effects on apparent bandwidth¶

KV cache memory grows with batch size and sequence length. Memory exhaustion limits concurrent request capacity.¶

KV cache memory equals: batch_size * sequence_length * num_layers * 2 * hidden_dim * precision_bytes¶

• Variation with batch composition¶
• Paged vs contiguous allocation¶

• Allocator-specific measurement¶
• Definition of "unusable" depends on minimum allocation size¶

Tensor parallelism partitions model layers across accelerators. Communication overhead reduces efficiency below ideal scaling.¶

Efficiency equals throughput with N GPUs divided by (N times single-GPU throughput).¶

• Baseline single-GPU measurement¶
• Communication topology effects¶

Pipeline parallelism partitions model layers into sequential stages. Pipeline bubbles (idle time during fill and drain) reduce efficiency.¶

Bubble fraction equals (P-1)/(P-1+M) where P is pipeline depth and M is microbatch count.¶

• Microbatch count selection¶
• Stage imbalance effects¶

• Imbalance metric selection¶
• Dynamic variation across batches¶

• Operation type specification¶
• Message size effects¶

Common precision modes include FP32, FP16, BF16, FP8, INT8, and INT4. Lower precision reduces memory and increases throughput at potential accuracy cost.¶

Mixed precision uses different precisions for different tensors. Testers MUST specify precision for weights, activations, and KV cache separately if they differ.¶

• Mixed precision specification¶
• KV cache precision separate from weights¶

• Baseline precision selection¶
• Workload effects on speedup¶

• Baseline precision selection¶
• Overhead from quantization metadata¶

Quantization may degrade accuracy. Impact is measured on task- specific benchmarks or perplexity.¶

Testers MUST specify the evaluation benchmark and baseline.¶

• Benchmark selection¶
• Task-specific variation¶

• Storage medium effects (local vs network)¶
• Parallelism in loading¶

Cold start latency exceeds steady-state latency due to JIT compilation, cache population, and memory allocation.¶

Cold start affects user experience for scale-to-zero deployments and after restarts.¶

• Definition of cold (first request vs first after idle)¶
• JIT compilation effects¶

• Stability definition¶
• Workload effects on warm-up¶

• Infrastructure-dependent (containers, VMs, bare metal)¶
• Warm traffic routing timing¶

• Architectural vs memory-constrained limits¶
• Performance degradation at maximum length¶

• Maximum length definition¶
• Performance effects at high utilization¶

Prefill latency scales linearly with input length for standard attention. Decode latency scales with total sequence length due to growing KV cache access.¶

Efficient attention mechanisms (sparse, linear) may achieve sub-linear scaling.¶

• Prefill vs decode scaling differs¶
• Architecture-dependent scaling¶