Benchmarking Methodology for Large Language Model Serving

4.1.1. Model Engine Configuration

The Model Engine configuration measures raw inference capability.¶

                +------------------+
                |   Load Generator |
                +--------+---------+
                         |
              Internal API (gRPC/HTTP)
                         |
                +--------v---------+
                |   Model Engine   |
                |  (SUT Boundary)  |
                +------------------+

Included components:¶

• Model weights and inference runtime¶
• Batching and scheduling logic¶
• KV cache management¶
• Tensor operations and kernels¶

Excluded components:¶

• External network transport¶
• Authentication and authorization¶
• Rate limiting¶
• Input/output safety filtering¶
• Load balancing¶

This configuration is appropriate for comparing inference engines (vLLM, TensorRT-LLM, SGLang) independent of deployment stack.¶

4.1.2. Application Gateway Configuration

The Application Gateway configuration measures user-observable API performance.¶

                +------------------+
                |   Load Generator |
                +--------+---------+
                         |
              External API (HTTPS)
                         |
                +--------v---------+
                | Application GW   |
                |  (SUT Boundary)  |
                |  +------------+  |
                |  |   Engine   |  |
                |  +------------+  |
                +------------------+

Included components (in addition to Model Engine):¶

• TLS termination¶
• Authentication and session management¶
• Rate limiting and quota enforcement¶
• Input validation and output filtering¶
• Safety guardrails¶

This configuration is appropriate for comparing API providers or evaluating production deployment performance.¶

4.1.3. Compound System Configuration

The Compound System configuration measures end-to-end task completion for agentic or retrieval-augmented workloads.¶

                +------------------+
                |   Task Driver    |
                +--------+---------+
                         |
                +--------v---------+
                |  Compound System |
                |  (SUT Boundary)  |
                |  +------------+  |
                |  | Retrieval  |  |
                |  +------------+  |
                |  +------------+  |
                |  |   Tools    |  |
                |  +------------+  |
                |  +------------+  |
                |  |  Gateway   |  |
                |  +------------+  |
                +------------------+

Included components (in addition to Application Gateway):¶

• Retrieval pipeline (embedding, vector search, reranking)¶
• Tool execution environment¶
• Orchestration logic¶
• Multi-turn conversation state¶

This configuration is appropriate for evaluating RAG systems or agentic applications.¶

4.2.1. Timing Resolution

The load generator MUST measure time with resolution of 1 millisecond or better. Microsecond resolution is RECOMMENDED for ITL measurement.¶

4.2.2. Streaming Support

The load generator MUST support streaming response protocols (SSE, WebSocket, or gRPC streaming). It MUST record the arrival time of each token or chunk, not only the complete response.¶

4.2.3. Open-Loop Load Generation

The load generator MUST support open-loop load generation where request arrival times are determined by a specified distribution independent of response times. Poisson arrivals MUST be supported. Uniform and bursty arrival patterns are RECOMMENDED.¶

4.2.4. Closed-Loop Load Generation

The load generator MUST support closed-loop load generation where a fixed number of concurrent requests are maintained. When a request completes, a new request is immediately submitted.¶

4.2.5. Request Isolation

The load generator MUST NOT allow slow responses to delay the submission of subsequent requests in open-loop mode. Asynchronous or multi-threaded implementation is REQUIRED.¶

4.2.6. Output Recording

The load generator MUST record for each request:¶

• Request submission timestamp¶
• First token arrival timestamp¶
• Each subsequent token arrival timestamp¶
• Final token arrival timestamp¶
• Total input token count¶
• Total output token count¶
• Request success/failure status¶

4.3.1. Workload Parameters

Each workload MUST specify:¶

Input length distribution:

Distribution type (fixed, uniform, normal, empirical), parameters (mean, std, min, max, or histogram), and unit (tokens using specified tokenizer).¶

Output length distribution:

Distribution type (fixed, uniform, normal, empirical), parameters (mean, std, min, max, or histogram), control method (max_tokens parameter, stop sequence, or both), and unit (tokens using specified tokenizer).¶

Content characteristics:

Domain (general, code, conversation, instruction), language (English, multilingual, code languages), and system prompt presence and typical length.¶

Prefix sharing:

Fraction of requests sharing common prefix and shared prefix length distribution.¶

4.3.2. Standard Workloads

This document defines five standard workloads. Full specifications appear in Appendix A.¶

4.3.3. Workload Reproducibility

For reproducible benchmarking:¶

• Testers MUST use deterministic random seeds for workload generation. The seed MUST be reported.¶
• Testers SHOULD publish the exact request sequences used, or provide generation code with fixed seeds.¶
• When using dataset-derived workloads (ShareGPT, HumanEval), testers MUST specify the dataset version, subset selection method, and any preprocessing applied.¶

4.4.1. Tokenizer Specification

The test report MUST specify:¶

• Tokenizer name and version (e.g., "cl100k_base", "Llama-3 tokenizer")¶
• Vocabulary size¶
• Source (Hugging Face model ID, tiktoken name, or custom)¶

4.4.2. Token Counting Method

For cross-system comparison where systems use different tokenizers:¶

Option A - Native tokenizer:

Count tokens using each system's native tokenizer. Report results separately with tokenizer identified. This method reflects actual system behavior but complicates comparison.¶

Option B - Reference tokenizer:

Count tokens using a declared reference tokenizer for all systems. This enables direct comparison but may not reflect actual system token counts.¶

The test report MUST declare which option is used. Option B with cl100k_base (GPT-4 tokenizer) as reference is RECOMMENDED for cross-system comparison.¶

4.4.3. Special Token Handling

The test report MUST specify handling of:¶

• BOS/EOS tokens (included or excluded from counts)¶
• System prompt tokens (counted separately or included)¶
• Tool/function call formatting tokens¶

4.5.1. Warm-up Requirements

Before measurement begins, testers MUST:¶

1. Load the model fully into accelerator memory¶
2. Process at least 100 requests or 10,000 output tokens, whichever is greater¶
3. Wait for request queue to drain completely¶
4. If prefix caching is enabled and being tested, populate the cache with representative prefixes¶

4.5.2. Warm-up Verification

Testers SHOULD verify warm-up completion by:¶

1. Measuring latency for a probe request before and after warm-up¶
2. Confirming latency stabilization (less than 10% variation across consecutive probe requests)¶

4.5.3. Cold Start Measurement

When cold start performance is being measured (Model Load Time, Cold Start Latency), warm-up MUST be skipped. The test report MUST clearly indicate cold start measurement.¶

4.6.1. Supported Protocols

This methodology supports:¶

Server-Sent Events (SSE):

HTTP-based streaming. Each event contains one or more tokens. RECOMMENDED for Application Gateway testing.¶

WebSocket:

Bidirectional streaming. Each message contains one or more tokens.¶

gRPC streaming:

Binary streaming protocol. Each message contains one or more tokens. RECOMMENDED for Model Engine testing.¶

4.6.2. Token Chunking

Streaming protocols may deliver multiple tokens per chunk due to batching or network buffering. The test report MUST specify:¶

• Protocol used¶
• Whether each chunk contains exactly one token or potentially multiple tokens¶
• How multi-token chunks are handled for ITL calculation¶

4.6.3. ITL Calculation with Chunked Delivery

When chunks contain multiple tokens:¶

Option A - Chunk timing:

Measure inter-chunk latency. Report as "Time Between Chunks" rather than ITL. Note chunk size distribution.¶

Option B - Distributed timing:

Distribute chunk arrival time across tokens. If a chunk with N tokens arrives at time T, assign arrival time T to all N tokens. This understates ITL variance.¶

Option C - Server-side timing:

Use server-reported per-token timestamps if available. This measures ITL independent of network effects.¶

The test report MUST declare which option is used. Option C is RECOMMENDED when available.¶

4.7.1. Single-Machine Testing

When load generator and SUT run on the same machine, clock synchronization is inherent. This configuration is RECOMMENDED for Model Engine testing.¶

4.7.2. Distributed Testing

When load generator and SUT are on different machines:¶

• NTP synchronization MUST achieve accuracy of 10ms or better¶
• PTP synchronization SHOULD be used when sub-millisecond accuracy is required¶
• The test report MUST state the synchronization method and estimated accuracy¶

4.7.3. Network Latency Measurement

For Application Gateway testing where network latency is significant:¶

• Testers SHOULD measure and report network RTT separately¶
• Testers MAY subtract estimated network latency from TTFT to isolate server-side processing time¶
• Any latency adjustment MUST be documented in the test report¶

4.7.4. Timestamp Format

All timestamps MUST be recorded in a format with at least millisecond precision. ISO 8601 with milliseconds (YYYY-MM-DDTHH:MM:SS.sssZ) or Unix epoch with milliseconds is RECOMMENDED.¶

4.8.1. Guardrail Disclosure

The test report MUST disclose:¶

• Whether input content filtering is enabled¶
• Whether output content filtering is enabled¶
• Names of safety systems if known (e.g., "Llama Guard")¶
• Whether any requests were refused during testing¶

4.8.2. Production-Representative Testing

For Application Gateway benchmarking intended to represent production performance:¶

• Safety systems SHOULD be enabled in their default configuration¶
• The test report MUST note if safety systems are disabled¶
• Testers SHOULD run comparative tests with safety enabled and disabled to quantify overhead¶

5.1.1. Objective

To determine the latency from request submission to first token receipt under varying load conditions. TTFT measures perceived responsiveness for interactive applications.¶

5.1.2. Setup Parameters

The following parameters MUST be defined:¶

5.1.3. Procedure

1. Configure the SUT with specified parameters.¶
2. Complete warm-up procedure (Section 4.5).¶
3. Begin load generation at the specified arrival rate or concurrency.¶

4. For each request:¶

   1. Record submission timestamp (T_submit)¶
   2. Record first token arrival timestamp (T_first)¶
   3. Calculate TTFT = T_first - T_submit¶
   4. Record input token count¶
5. Continue until request count is reached.¶
6. Compute distribution statistics.¶

5.1.4. Measurements

5.1.5. Reporting Format

The test report MUST include:¶

5.2.1. Objective

To determine the maximum rate at which the SUT can generate output tokens while maintaining acceptable latency. This test measures system capacity under load.¶

5.2.2. Setup Parameters

The following parameters MUST be defined:¶

5.2.3. Procedure

This test employs an iterative search to find maximum throughput.¶

1. Configure the SUT with specified parameters.¶
2. Complete warm-up procedure.¶

3. For each load level (arrival rate or concurrency):¶

   1. Run load for the specified test duration.¶
   2. Record all request timings.¶
   3. Compute throughput as total output tokens divided by test duration.¶
   4. Compute TTFT and TPOT percentiles.¶
   5. If latency constraint specified, check SLO compliance.¶

4. Use binary search to find maximum throughput:¶

   1. If no latency constraint: find load level where queue grows unboundedly (system saturation).¶
   2. If latency constraint: find highest load level meeting SLO.¶
5. Report throughput at the maximum sustainable load level.¶

5.2.4. Measurements

5.2.5. Reporting Format

5.3.1. Objective

To characterize the relationship between throughput and latency across the operating range of the SUT. This test produces a throughput-latency curve revealing system behavior better than point measurements.¶

5.3.2. Setup Parameters

Workload:

One of the standard workloads or fully specified custom workload.¶

Test duration per point:

Minimum 60 seconds per load level.¶

Load levels:

At least 10 load levels spanning from low load (10% of estimated capacity) to saturation.¶

Load model:

Open-loop is REQUIRED for this test. Closed-loop cannot reveal behavior beyond capacity.¶

5.3.3. Procedure

1. Estimate system capacity using a preliminary throughput test or published specifications.¶
2. Define load levels: 10%, 20%, 30%, ..., 100%, 110%, 120% of estimated capacity.¶

3. For each load level in ascending order:¶

   1. Run load for specified duration.¶
   2. Record all request timings.¶
   3. Compute achieved throughput (may differ from offered load at saturation).¶
   4. Compute latency percentiles.¶
4. Plot throughput vs latency curves.¶

5.3.4. Measurements

For each load level, record:¶

• Offered load (request rate)¶
• Achieved throughput (output tokens per second)¶
• TTFT: P50, P95, P99¶
• TPOT: P50, P95, P99¶
• End-to-end latency: P50, P95, P99¶
• Request success rate¶
• Queue growth indicator (stable/growing)¶

Derived metrics:¶

Optimal operating point:

Load level achieving highest throughput while meeting specified SLO.¶

Knee point:

Load level where P99 latency exceeds 2x the minimum P99 latency observed.¶

Saturation point:

Load level where achieved throughput first decreases from previous level.¶

5.3.5. Reporting Format

Knee point: 14 req/s (TTFT P99 exceeds 2x minimum)¶

Saturation point: 22 req/s (throughput peaks)¶

5.4.1. Objective

To characterize the variability of token delivery during the decode phase. ITL distribution determines streaming smoothness experienced by users.¶

5.4.2. Setup Parameters

Workload:

Synthetic-Uniform or Conversation workload RECOMMENDED.¶

Minimum output length:

Requests MUST generate at least 50 output tokens to provide meaningful ITL samples.¶

Request count:

At least 100 requests for per-request statistics, yielding 5000+ ITL samples.¶

Load level:

Specify as percentage of maximum throughput. Multiple load levels RECOMMENDED: 25%, 50%, 75%, 90% of saturation.¶

Measurement method:

Specify per Section 4.6.3 (chunk timing, distributed timing, or server-side timing).¶

5.4.3. Procedure

1. Configure SUT and complete warm-up.¶

2. For each load level:¶

   1. Generate requests at specified load.¶
   2. For each request, record arrival time of each token after the first.¶
   3. Calculate ITL_i = T(token_i) - T(token_{i-1}) for each consecutive token pair.¶
   4. Aggregate ITL samples across all requests.¶
   5. Calculate per-request jitter (standard deviation of ITL within each request).¶
   6. Record maximum pause duration per request.¶

The interval between request submission and first token (TTFT) MUST NOT be included in ITL calculation.¶

5.4.4. Measurements

5.4.5. Reporting Format

5.5.1. Objective

To determine the maximum number of concurrent requests the SUT can maintain while meeting latency objectives. This test measures memory capacity and scheduling limits.¶

5.5.2. Setup Parameters

Workload:

Synthetic-Uniform RECOMMENDED for controlled testing.¶

Fixed output length:

Use fixed output length (e.g., 256 tokens) to ensure all requests have similar duration.¶

Initial concurrency:

Starting number of concurrent requests (e.g., 8).¶

Maximum concurrency:

Upper bound for search (e.g., 512).¶

Success criteria:

Request completion rate >= 99%, TTFT P99 <= specified threshold, and no out-of-memory errors.¶

5.5.3. Procedure

This test employs binary search to find maximum concurrent capacity.¶

1. Configure SUT and complete warm-up.¶
2. Set concurrency = initial concurrency.¶

3. For each concurrency level:¶

   1. Submit [concurrency] requests simultaneously.¶
   2. Maintain concurrency: when a request completes, immediately submit a replacement.¶
   3. Run for at least 60 seconds or 100 request completions per slot, whichever is longer.¶
   4. Record completion rate, latency percentiles, and any errors.¶
   5. Check success criteria.¶

4. Binary search:¶

   1. If success criteria met: increase concurrency toward maximum.¶
   2. If success criteria not met: decrease concurrency.¶
   3. Continue until convergence.¶
5. Report maximum concurrency meeting success criteria.¶

5.5.4. Measurements

Maximum concurrent requests:

Highest concurrency meeting success criteria.¶

Achieved throughput at maximum:

Output tokens per second at maximum concurrency.¶

Tokens in flight at maximum:

Approximate total tokens (input + output so far) across all concurrent requests.¶

5.5.5. Reporting Format

Maximum concurrent requests meeting criteria: 64¶

5.6.1. Objective

To evaluate how equitably the SUT allocates resources across concurrent requests with different characteristics. This test reveals head-of-line blocking, starvation, and priority effects.¶

5.6.2. Setup Parameters

Workload:

Synthetic-Skewed REQUIRED. The high length variance creates fairness-sensitive conditions.¶

Request classes:

Define two or more request classes:¶

• Short requests: Input [64, 256] tokens, output [32, 128] tokens¶
• Long requests: Input [1024, 4096] tokens, output [256, 1024] tokens¶

Class mix:

Ratio of request classes (e.g., 80% short, 20% long).¶

Load level:

70-90% of saturation throughput RECOMMENDED to create contention.¶

Request count:

At least 500 requests per class.¶

5.6.3. Procedure

1. Configure SUT and complete warm-up.¶
2. Measure baseline: performance of each class in isolation at same total load.¶
3. Generate mixed workload with specified class ratio.¶
4. Run at specified load level for at least 300 seconds.¶
5. For each request, record class membership, submission time, first token time, completion time.¶
6. Compute per-class statistics and fairness metrics.¶

5.6.4. Measurements

Per-class latency:

TTFT P50, P95, P99 for each request class.¶

Latency inflation:

(Mixed workload TTFT) / (Isolated TTFT) per class.¶

Jain's Fairness Index:

J = (sum(x_i))^2 / (n * sum(x_i^2)) where x_i is normalized latency. J = 1.0 indicates perfect fairness. J < 0.9 indicates significant unfairness.¶

Starvation rate:

Fraction of requests waiting longer than 5x the median wait time for their class.¶

5.6.5. Reporting Format

5.7.1. Objective

To evaluate the performance benefit of prefix caching under workloads with shared prefixes. This test quantifies TTFT reduction from cache hits.¶

5.7.2. Setup Parameters

Workload:

Code Completion workload RECOMMENDED (high prefix sharing).¶

Shared prefix:

Define a prefix shared across requests.¶

Prefix length:

Length in tokens of shared prefix.¶

Sharing fraction:

Percentage of requests sharing the prefix.¶

Comparison mode:

Test MUST run in two configurations: cache disabled (baseline) and cache enabled.¶

5.7.3. Procedure

1. Configure SUT with cache disabled.¶
2. Complete warm-up (without populating prefix cache).¶
3. Run workload, record TTFT for all requests.¶
4. Enable prefix cache.¶
5. Optionally pre-populate cache with shared prefix.¶
6. Run identical workload, record TTFT for all requests.¶
7. Compare results.¶

5.7.4. Measurements

TTFT without cache:

P50, P95, P99 with caching disabled.¶

TTFT with cache:

P50, P95, P99 with caching enabled.¶

TTFT reduction:

(TTFT_no_cache - TTFT_cache) / TTFT_no_cache as percentage.¶

Cache hit rate:

Fraction of prefix tokens served from cache.¶

Throughput improvement:

Percentage increase from caching.¶

5.7.5. Reporting Format

5.8.1. Objective

To characterize SUT behavior when memory resources are constrained, including preemption, swapping, and degradation patterns.¶

5.8.2. Setup Parameters

Workload:

Long Context workload RECOMMENDED to create memory pressure.¶

Oversubscription level:

Percentage above maximum capacity (e.g., 110%, 125%, 150%).¶

5.8.3. Procedure

1. Determine maximum concurrent capacity from Section 5.5.¶
2. Configure SUT and complete warm-up.¶

3. For each oversubscription level:¶

   1. Submit requests at concurrency exceeding capacity.¶
   2. Run for at least 120 seconds.¶
   3. Monitor request completions, preemption events, latency.¶
   4. Record any OOM errors or system failures.¶
4. Analyze degradation patterns.¶

5.8.4. Measurements

Completion rate:

Percentage of requests completing successfully at each level.¶

Preemption rate:

Fraction of requests preempted at least once.¶

Preemption recovery rate:

Fraction of preempted requests that eventually complete.¶

Preemption loss:

Average tokens discarded per preemption event.¶

5.8.5. Reporting Format

5.9.1. Objective

To characterize how latency and throughput scale with context length.¶

5.9.2. Setup Parameters

Workload:

Long Context workload REQUIRED.¶

Context length range:

Sequence of lengths to test (e.g., 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K tokens).¶

Fixed output length:

Use consistent short output (256 tokens) to isolate prefill impact.¶

Load model:

Closed-loop with low concurrency (1-4).¶

Requests per length:

At least 20 requests per context length.¶

5.9.3. Procedure

1. Configure SUT and complete warm-up with short-context requests.¶

2. For each context length in ascending order:¶

   1. Generate requests with specified input length.¶
   2. Submit requests at low concurrency.¶
   3. Record TTFT and total latency for each request.¶
3. Analyze scaling behavior and fit to scaling models.¶

5.9.4. Measurements

Per-length latency:

TTFT Mean, P50, P95 for each context length.¶

Prefill scaling:

Time per input token (TTFT / input_length).¶

Scaling exponent:

Fit exponent k where TTFT proportional to context_length^k.¶

Throughput at length:

Maximum throughput achievable at each context length.¶

5.9.5. Reporting Format

Best fit: Linear (R^2 = 0.9987), ~68 microseconds per input token¶

5.10.1. Objective

To quantify the latency impact of safety systems and content filtering.¶

5.10.2. Setup Parameters

Workload:

Conversation workload RECOMMENDED.¶

Content mix:

Use benign content to measure processing overhead.¶

Configurations to compare:

The following configurations should be tested:¶

• Baseline: All guardrails disabled (if possible)¶
• Input filtering only¶
• Output filtering only¶
• Full filtering: All production guardrails enabled¶

Load levels:

Test at 25%, 50%, 75% of capacity.¶

5.10.3. Procedure

1. Configure SUT with baseline (no guardrails).¶
2. Complete warm-up and run workload at each load level.¶
3. Enable each guardrail configuration and repeat.¶
4. Compare results across configurations.¶

5.10.4. Measurements

Per-configuration latency:

TTFT P50, P95, P99 and End-to-end latency for each configuration.¶

Input filter overhead:

TTFT(input_filter) - TTFT(baseline)¶

Total guardrail overhead:

End-to-end(full) - End-to-end(baseline)¶

Throughput reduction:

Percentage reduction from guardrails.¶

5.10.5. Reporting Format

A.1.1. Input Specification

Distribution:

Uniform¶

Minimum:

128 tokens¶

Maximum:

512 tokens¶

Mean:

320 tokens¶

Content:

Random token IDs from vocabulary¶

A.1.2. Output Specification

Distribution:

Uniform¶

Minimum:

64 tokens¶

Maximum:

256 tokens¶

Mean:

160 tokens¶

Control:

max_tokens parameter¶

A.1.3. Other Parameters

System prompt:

None¶

Prefix sharing:

None¶

Temperature:

0.0 (deterministic)¶

Stop sequences:

None¶

A.1.4. Generation Method

Python pseudocode:¶

def generate_synthetic_uniform(n_requests, seed=42):
    rng = random.Random(seed)
    requests = []
    for i in range(n_requests):
        input_len = rng.randint(128, 512)
        output_len = rng.randint(64, 256)
        input_tokens = [rng.randint(0, 100255)
                        for _ in range(input_len)]
        requests.append({
            'input_tokens': input_tokens,
            'max_tokens': output_len,
            'temperature': 0.0
        })
    return requests

A.2.1. Input Specification

Distribution:

Log-normal¶

mu:

5.5 (in log space)¶

sigma:

1.0 (in log space)¶

Minimum:

32 tokens (floor)¶

Maximum:

4096 tokens (cap)¶

Median:

~245 tokens¶

Mean:

~405 tokens¶

A.2.2. Output Specification

Distribution:

Log-normal¶

mu:

4.5 (in log space)¶

sigma:

1.2 (in log space)¶

Minimum:

16 tokens (floor)¶

Maximum:

2048 tokens (cap)¶

A.3.1. Data Source

Dataset:

ShareGPT (vicuna_cleaned subset)¶

Version:

2023-04-12¶

Preprocessing:

Filter conversations with 1+ assistant turns¶

A.3.2. Length Statistics (Reference)

Input tokens:¶

• P50: 156¶
• P95: 892¶
• P99: 2134¶

Output tokens:¶

• P50: 234¶
• P95: 789¶
• P99: 1567¶

A.4.1. Data Source

Dataset:

The Stack (Python, JavaScript, TypeScript subset)¶

Preprocessing:

Extract function-level completions¶

A.4.2. Prefix Sharing Pattern

• 10 unique repository contexts¶
• Each 512-1024 tokens¶
• 80% of requests share one of these prefixes¶
• Distribution: Zipf with s=1.5¶

A.5.1. Input Specification

Distribution:

Uniform over target lengths¶

Target lengths:

[8192, 16384, 32768, 65536, 131072] tokens¶

Structure:

[document][question]¶

Document:

Fills target length minus 100 tokens¶

Question:

Fixed ~100 token question about document¶

A.5.2. Output Specification

Distribution:

Fixed¶

Length:

256 tokens¶

Control:

max_tokens = 256¶

B.1.1. HTTP/SSE Measurement

Client-side TTFT:¶

T_submit:

time of sending final byte of HTTP request¶

T_first:

time of receiving first data event with content token¶

T_first is when the complete "data:" line is received and parsed, not when the first byte of the response arrives.¶

B.1.2. gRPC Streaming Measurement

T_submit:

time of sending request message¶

T_first:

time of receiving first response message with token¶

B.1.3. Server-Side Measurement

If server instrumentation available:¶

T_submit:

time request enters inference queue¶

T_first:

time first token exits model forward pass¶

Server-side excludes network latency but may include internal queue time.¶

B.2.1. Recommended Approach

1. First, characterize delivery pattern (tokens per chunk)¶
2. If single-token chunks dominate (>90%): use direct measurement¶
3. If multi-token chunks common: prefer server timestamps¶
4. If server timestamps unavailable: use chunk timing and document¶

B.3.1. NTP Synchronization

1. Both machines sync to same NTP server¶
2. Verify offset: ntpq -p (check offset column)¶
3. Acceptable offset: < 10ms for most LLM benchmarking¶
4. Document NTP server and measured offset¶

B.3.2. PTP Synchronization

For sub-millisecond accuracy:¶

1. Use PTP-capable network hardware¶
2. Configure ptp4l on Linux systems¶
3. Acceptable offset: < 1 microsecond¶

B.3.3. Single-Machine Alternative

Recommended for Model Engine testing:¶

1. Run load generator on same machine as SUT¶
2. Use loopback network interface¶
3. Clock synchronization inherent¶
4. Eliminates network latency from measurement¶