Benchmarking Terminology for Output Behavior Fault Detection in Large Language Model Serving Systems
draft-gaikwad-llm-fault-detection-terminology-00

Benchmarking Methodology                                      M. Gaikwad
Internet-Draft                                               Independent
Intended status: Informational                            11 August 2026
Expires: 12 February 2027

 Benchmarking Terminology for Output Behavior Fault Detection in Large
                     Language Model Serving Systems
            draft-gaikwad-llm-fault-detection-terminology-00

Abstract

   This document defines terminology for benchmarking the fault
   detection capability of observability systems that monitor Large
   Language Model (LLM) serving deployments.  It defines terms for
   injectable output behavior fault classes, for the detection events
   and latency intervals those faults produce, and for the coverage and
   accuracy characteristics of the detector under test.

   The System Under Test is the observability system, and not the
   language model.  Faults are injected at a known time under controlled
   conditions, which makes detection latency measurable in a way it is
   not in production.

   This document defines terminology only.  A companion document defines
   the corresponding methodology.  This document specifies no test
   procedures, no acceptance thresholds, and no service-level
   objectives.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 12 February 2027.

Gaikwad                 Expires 12 February 2027                [Page 1]
Internet-Draft         Behavior Fault Terminology            August 2026

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.2.  Relationship to Other Work  . . . . . . . . . . . . . . .   3
   2.  Conventions and Definitions . . . . . . . . . . . . . . . . .   4
   3.  Test Architecture Terms . . . . . . . . . . . . . . . . . . .   4
     3.1.  Detector Under Test . . . . . . . . . . . . . . . . . . .   4
     3.2.  Observation Point . . . . . . . . . . . . . . . . . . . .   4
     3.3.  Fault Injection Point . . . . . . . . . . . . . . . . . .   5
   4.  Fault Terms . . . . . . . . . . . . . . . . . . . . . . . . .   5
     4.1.  Behavior Fault  . . . . . . . . . . . . . . . . . . . . .   5
     4.2.  Fault Class . . . . . . . . . . . . . . . . . . . . . . .   6
     4.3.  Consistency Degradation . . . . . . . . . . . . . . . . .   7
     4.4.  Fault Profile . . . . . . . . . . . . . . . . . . . . . .   7
   5.  Event Terms . . . . . . . . . . . . . . . . . . . . . . . . .   8
     5.1.  Fault Onset . . . . . . . . . . . . . . . . . . . . . . .   8
     5.2.  Detection Event . . . . . . . . . . . . . . . . . . . . .   8
     5.3.  False Detection . . . . . . . . . . . . . . . . . . . . .   9
     5.4.  Baseline Interval . . . . . . . . . . . . . . . . . . . .   9
   6.  Detection Characteristic Terms  . . . . . . . . . . . . . . .  10
     6.1.  Detection Latency . . . . . . . . . . . . . . . . . . . .  10
     6.2.  Detection Coverage  . . . . . . . . . . . . . . . . . . .  10
     6.3.  Detection Threshold Magnitude . . . . . . . . . . . . . .  11
     6.4.  Boundary Masking  . . . . . . . . . . . . . . . . . . . .  12
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  12
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  12
   9.  Normative References  . . . . . . . . . . . . . . . . . . . .  12
   10. Informative References  . . . . . . . . . . . . . . . . . . .  13
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  14
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  14

Gaikwad                 Expires 12 February 2027                [Page 2]
Internet-Draft         Behavior Fault Terminology            August 2026

1.  Introduction

   LLM serving deployments are monitored by observability systems that
   report behavioral indicators such as hallucination rate, refusal
   rate, and guardrail activation rate.  These systems are widely
   deployed.  Their detection capability is not characterized.

   Operators do not currently know how long their observability system
   takes to raise a behavioral degradation, nor what fraction of
   degradations it raises at all.  Both quantities are difficult to
   obtain from production records.  A production incident has no
   independently known start time.  The moment behavior first departed
   from intent is inferred after the fact, if it is recovered at all.

   Fault injection removes this difficulty.  When a fault is introduced
   at a known time under controlled conditions, the interval from
   introduction to detection is directly measurable.  The fraction of
   injected faults that are detected is directly measurable.  These are
   properties of the observability system, and this document provides
   the vocabulary to state them.

   The terms defined here follow the pattern established by [RFC1242]
   and [RFC8238].  Each term is given a definition, a discussion,
   measurement units where applicable, and a statement of known issues.

1.1.  Scope

   This document defines terminology only.  It does not specify test
   procedures, traffic profiles, injection mechanisms, acceptance
   thresholds, or service-level objectives.  A companion methodology
   document is intended to specify those.  That document does not yet
   exist.  Statements in this document that defer a detail to the
   methodology identify work still to be done.

   The behavioral indicators whose degradation is injected are defined
   by reference.  This document does not define hallucination, refusal,
   or policy violation.  It defines the fault classes constructed from
   them and the detection characteristics measured against them.

   Benchmarking activity described by this terminology is confined to a
   test environment.  Section 7 states this requirement.

1.2.  Relationship to Other Work

   [RFC1242] and [RFC2544] establish the terminology and methodology
   pattern this document follows.  [RFC8238] and [RFC8239] apply that
   pattern to data center benchmarking.

Gaikwad                 Expires 12 February 2027                [Page 3]
Internet-Draft         Behavior Fault Terminology            August 2026

   [LLM-TERMS] defines System Under Test boundaries for LLM serving, and
   [LLM-METHOD] defines the corresponding methodology.  Those boundaries
   are reused here to locate the observation point of the detector under
   test.  [AI-FABRIC-TERMS] defines terminology for the network fabric
   beneath those serving boundaries.

   [RFC9940] defines terms for network fault and problem management.
   This document adopts its separation between a fault and its
   detection.  It applies that separation to behavioral faults in model
   serving.

2.  Conventions and Definitions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Test Architecture Terms

3.1.  Detector Under Test

   Definition:
      The observability system whose fault detection capability is
      characterized by a benchmark using this terminology.

   Discussion:
      The Detector Under Test (DUT) comprises the classifiers, judges,
      heuristics, sampling logic, aggregation, alerting rules, and
      thresholds that together determine whether a behavioral
      degradation is raised.  The language model is not under test.  The
      model and its serving stack form part of the test environment, and
      their configuration is held constant across trials.

   Issues:
      Where the DUT includes a language model as a judge, that model is
      part of the DUT and its version is reported as DUT configuration.

   See Also:
      Observation Point; Behavior Fault.

3.2.  Observation Point

   Definition:
      The boundary within the serving deployment at which the Detector
      Under Test samples requests and responses.

Gaikwad                 Expires 12 February 2027                [Page 4]
Internet-Draft         Behavior Fault Terminology            August 2026

   Discussion:
      The available boundaries are the Model Engine, the Application
      Gateway, and the Compound System, as defined in [LLM-TERMS].  A
      detector observing at the Application Gateway sees responses after
      output filtering.  A detector observing at the Model Engine sees
      them before.  The same injected fault may be visible at one
      boundary and masked at another.

   Issues:
      A benchmark result is valid only for the Observation Point at
      which it was obtained.  Results obtained at different Observation
      Points MUST NOT be compared.

   See Also:
      Detector Under Test; Fault Injection Point.

3.3.  Fault Injection Point

   Definition:
      The location within the test environment at which an Behavior
      Fault is introduced.

   Discussion:
      Injection may occur upstream or downstream of the Observation
      Point.  An injection point downstream of the Observation Point
      produces a fault the detector cannot observe, which is a valid
      configuration for characterizing detection blind spots.  The
      relative order of the Fault Injection Point and the Observation
      Point is reported with every result.

   See Also:
      Observation Point; Behavior Fault.

4.  Fault Terms

4.1.  Behavior Fault

   Definition:
      A controlled, reversible change to the behavior of a serving
      deployment that shifts one or more behavioral indicators away from
      their baseline distribution, introduced for the purpose of
      characterizing a Detector Under Test.

Gaikwad                 Expires 12 February 2027                [Page 5]
Internet-Draft         Behavior Fault Terminology            August 2026

   Discussion:
      An Behavior Fault is defined by the indicator it shifts, the
      magnitude of the shift, and the fault profile through which the
      shift is applied.  A fault is not defined by its cause.  Two
      faults that produce the same indicator shift are equivalent for
      benchmarking purposes regardless of the mechanism used to produce
      them.

   Measurement Units:
      Magnitude is expressed in the units of the affected indicator.
      For rate-valued indicators, magnitude is expressed as an absolute
      change in the rate.

   Issues:
      A fault of sufficiently small magnitude is indistinguishable from
      baseline variation.  Reported results state the baseline
      variability of the affected indicator so that the magnitude can be
      interpreted.

   See Also:
      Fault Profile; Fault Class; Baseline Interval.

4.2.  Fault Class

   Definition:
      A category of Behavior Fault distinguished by the behavioral
      indicator it shifts.

   Discussion:
      The following classes are defined for use with this terminology.
      The list is extensible.

      *  Refusal Shift: a change in the fraction of requests declined.

      *  Groundedness Degradation: a change in the fraction of responses
         unsupported by provided context.

      *  Policy Filter Degradation: a change in the fraction of
         responses that violate a declared policy and are emitted.

      *  Version-Scoped Degradation: any of the above applied to one
         model version while sibling versions on the same deployment
         remain at baseline.

   Issues:
      Detection characteristics are not comparable across Fault Classes.
      A detector may be fast on Refusal Shift and blind to Groundedness
      Degradation.  Results are reported per class.

Gaikwad                 Expires 12 February 2027                [Page 6]
Internet-Draft         Behavior Fault Terminology            August 2026

   See Also:
      Behavior Fault; Consistency Degradation.

4.3.  Consistency Degradation

   Definition:
      A change in the dissimilarity between responses to identical
      requests issued under identical configuration.

   Discussion:
      Consistency Degradation is noted separately from the Fault Classes
      of Section 4.2 because it is produced differently.  The other
      classes shift an indicator computed over independent requests.
      Consistency is computed over repeated requests, and injecting a
      shift in it requires perturbing the sampling configuration of the
      serving deployment.  That perturbation is a configuration change
      and not a behavioral fault in the same sense.  A benchmark using
      this terminology MAY include Consistency Degradation, and results
      for it are reported separately from the Fault Classes.

   Measurement Units:
      The units of the declared dissimilarity measure.

   Issues:
      A serving deployment configured for deterministic decoding has no
      baseline consistency variation, and Consistency Degradation is not
      applicable to it.

   See Also:
      Fault Class.

4.4.  Fault Profile

   Definition:
      The function describing how the magnitude of an Behavior Fault
      varies over the duration of its application.

   Discussion:
      Three profiles are defined.  A Step profile applies the full
      magnitude at the Fault Onset and holds it.  A Ramp profile
      increases magnitude linearly from zero over a stated interval.  An
      Intermittent profile alternates between full magnitude and
      baseline at a stated duty cycle and period.

   Measurement Units:
      Ramp interval and Intermittent period are expressed in seconds.
      Duty cycle is expressed as a fraction.

Gaikwad                 Expires 12 February 2027                [Page 7]
Internet-Draft         Behavior Fault Terminology            August 2026

   Issues:
      Detection Latency is defined relative to Fault Onset for all
      profiles.  For Ramp and Intermittent profiles the detector may
      reasonably raise the fault well after onset, since the fault is
      not at full magnitude at that time.  Results for these profiles
      are interpreted with the profile parameters.

   See Also:
      Fault Onset; Detection Latency.

5.  Event Terms

5.1.  Fault Onset

   Definition:
      The time at which an Behavior Fault begins to be applied at the
      Fault Injection Point.

   Discussion:
      Fault Onset is known by construction.  It is recorded by the
      injection apparatus and is not inferred from detector output.
      This is the property that distinguishes a benchmark from an
      observation of a production incident.

   Measurement Units:
      A timestamp with resolution of one millisecond or finer.

   Issues:
      The injection apparatus and the Detector Under Test may run on
      different hosts.  Clock synchronization requirements are stated in
      the companion methodology.  Reported Detection Latency values are
      valid only to the accuracy of that synchronization.

   See Also:
      Detection Event; Detection Latency.

5.2.  Detection Event

   Definition:
      The time at which the Detector Under Test emits a signal
      attributable to an applied Behavior Fault.

   Discussion:
      The signal may be an alert, a threshold crossing, a state change,
      or a log record, according to the DUT configuration.  The form of
      signal that constitutes a Detection Event is declared before the
      trial begins.  Changing it after observing results invalidates the
      trial.

Gaikwad                 Expires 12 February 2027                [Page 8]
Internet-Draft         Behavior Fault Terminology            August 2026

   Measurement Units:
      A timestamp with resolution of one millisecond or finer.

   Issues:
      Attribution of a signal to a specific applied fault requires that
      only one fault be applied at a time, or that the signal carry
      sufficient information to identify the affected indicator.  The
      companion methodology states the requirement.

   See Also:
      Fault Onset; False Detection.

5.3.  False Detection

   Definition:
      A signal emitted by the Detector Under Test during a Baseline
      Interval, in which no Behavior Fault is applied.

   Discussion:
      False Detection is measured over Baseline Intervals of stated
      duration under stated traffic.  A detector can achieve arbitrarily
      low Detection Latency by lowering its thresholds, at the cost of
      False Detection Rate.  The two are reported together.

   Measurement Units:
      Count per unit time, with the interval duration and traffic
      profile stated.

   See Also:
      Baseline Interval; Detection Latency.

5.4.  Baseline Interval

   Definition:
      A period during which traffic is offered to the serving deployment
      and no Behavior Fault is applied.

   Discussion:
      Baseline Intervals serve two purposes.  They establish the
      baseline distribution of each affected indicator, against which
      fault magnitude is expressed.  They provide the observation period
      for False Detection.

   Measurement Units:
      Seconds.

Gaikwad                 Expires 12 February 2027                [Page 9]
Internet-Draft         Behavior Fault Terminology            August 2026

   Issues:
      A Baseline Interval that is short relative to the DUT aggregation
      window does not establish a stable baseline.  The companion
      methodology states a minimum.

   See Also:
      False Detection; Behavior Fault.

6.  Detection Characteristic Terms

6.1.  Detection Latency

   Definition:
      The interval from Fault Onset to the Detection Event attributable
      to that fault.

   Discussion:
      Detection Latency is defined per trial.  A trial in which no
      Detection Event occurs before the end of the fault application
      yields no Detection Latency value.  Such trials are counted
      separately and are reflected in Detection Coverage.

   Measurement Units:
      Seconds.

   Issues:
      Detection Latency depends on offered load.  A detector observing a
      rate indicator requires a number of requests to accumulate
      evidence, and that number takes longer to accumulate at low
      request rates.  A reported value is valid only for the offered
      load at which it was obtained.  Results are reported across a
      range of loads.

   See Also:
      Fault Onset; Detection Event; Detection Coverage.

6.2.  Detection Coverage

   Definition:
      The fraction of applied Behavior Faults of a given Fault Class and
      magnitude for which a Detection Event occurred.

Gaikwad                 Expires 12 February 2027               [Page 10]
Internet-Draft         Behavior Fault Terminology            August 2026

   Discussion:
      Detection Coverage is measurable in a benchmark because the number
      of applied faults is known.  It is not measurable from production
      records, where undetected faults leave no trace.  Detection
      Coverage is the term that gives a Detection Latency figure its
      meaning.  A detector that raises one fault in ten may report a low
      mean Detection Latency across the faults it raises.

   Measurement Units:
      A fraction between zero and one.

   Issues:
      Detection Coverage is a function of fault magnitude.  A coverage
      figure reported without the magnitude at which it was obtained is
      not interpretable.  Results are reported as coverage against
      magnitude.

   See Also:
      Detection Latency; False Detection.

6.3.  Detection Threshold Magnitude

   Definition:
      The smallest fault magnitude of a given Fault Class at which the
      Detector Under Test achieves a stated Detection Coverage.

   Discussion:
      This term characterizes detector sensitivity.  It is obtained by
      repeating trials at decreasing magnitude until coverage falls
      below the stated level.  The coverage level used MUST be reported
      with the value.

   Measurement Units:
      The units of the affected indicator.

   Issues:
      Detection Threshold Magnitude approaches the baseline variability
      of the affected indicator as a lower bound.  A reported value at
      or below baseline variability indicates that the trial was not
      correctly configured.

   See Also:
      Detection Coverage; Baseline Interval.

Gaikwad                 Expires 12 February 2027               [Page 11]
Internet-Draft         Behavior Fault Terminology            August 2026

6.4.  Boundary Masking

   Definition:
      The reduction in observed fault magnitude between the Fault
      Injection Point and the Observation Point.

   Discussion:
      A fault injected at the Model Engine may be partly or wholly
      suppressed by output filtering before it reaches an Application
      Gateway Observation Point.  Boundary Masking quantifies that
      suppression.  A masking value near one indicates that the
      intervening components absorb the fault, and that detection at the
      Observation Point characterizes those components as much as the
      detector.

   Measurement Units:
      A fraction between zero and one.

   See Also:
      Observation Point; Fault Injection Point.

7.  Security Considerations

   Benchmarking activity described by this terminology is confined to a
   laboratory test environment.  The faults defined here deliberately
   degrade the behavioral safety properties of a serving deployment.
   Applying them to a deployment serving production traffic exposes
   users to unsafe, ungrounded, or incorrectly refused responses.

   Test environments MUST be isolated from production traffic.  Fault
   injection mechanisms MUST NOT be reachable from a production control
   plane.  The reasoning of [RFC6815] applies to this document.

   Benchmark results characterize where a detector is insensitive.  A
   published Detection Threshold Magnitude states the degradation an
   operator would not observe.  Publishers SHOULD consider this when
   selecting a disclosure audience.

8.  IANA Considerations

   This document has no IANA actions.

9.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

Gaikwad                 Expires 12 February 2027               [Page 12]
Internet-Draft         Behavior Fault Terminology            August 2026

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, May 2017,
              <https://www.rfc-editor.org/info/rfc8174>.

   [RFC1242]  Bradner, S., "Benchmarking Terminology for Network
              Interconnection Devices", RFC 1242, July 1991,
              <https://www.rfc-editor.org/info/rfc1242>.

   [LLM-TERMS]
              Gaikwad, M., "Benchmarking Terminology for Large Language
              Model Serving", Work in Progress, Internet-Draft, draft-
              gaikwad-llm-benchmarking-terminology, 2026,
              <https://datatracker.ietf.org/doc/html/draft-gaikwad-llm-
              benchmarking-terminology>.

10.  Informative References

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999,
              <https://www.rfc-editor.org/info/rfc2544>.

   [RFC6815]  Bradner, S., Dubray, K., McQuaid, J., and A. Morton,
              "Applicability Statement for RFC 2544: Use on Production
              Networks Considered Harmful", RFC 6815, November 2012,
              <https://www.rfc-editor.org/info/rfc6815>.

   [RFC8238]  Avramov, L. and J. Rapp, "Data Center Benchmarking
              Terminology", RFC 8238, August 2017,
              <https://www.rfc-editor.org/info/rfc8238>.

   [RFC8239]  Avramov, L. and J. Rapp, "Data Center Benchmarking
              Methodology", RFC 8239, August 2017,
              <https://www.rfc-editor.org/info/rfc8239>.

   [RFC9940]  Davis, N., Ed. and A. Farrel, Ed., "Some Key Terms for
              Network Fault and Problem Management", RFC 9940, April
              2026, <https://www.rfc-editor.org/info/rfc9940>.

   [AI-FABRIC-TERMS]
              Calabria, F., Pignataro, C., Wu, Q., and G. Fioccola,
              "Benchmarking Terminology for AI Network Fabrics", Work in
              Progress, Internet-Draft, draft-calabria-bmwg-ai-fabric-
              terminology, April 2026,
              <https://datatracker.ietf.org/doc/html/draft-calabria-
              bmwg-ai-fabric-terminology>.

Gaikwad                 Expires 12 February 2027               [Page 13]
Internet-Draft         Behavior Fault Terminology            August 2026

   [LLM-METHOD]
              Gaikwad, M., "Benchmarking Methodology for Large Language
              Model Serving", Work in Progress, Internet-Draft, draft-
              gaikwad-llm-benchmarking-methodology, 2026,
              <https://datatracker.ietf.org/doc/html/draft-gaikwad-llm-
              benchmarking-methodology>.

Acknowledgements

   This document follows the terminology structure established by
   [RFC1242] and applied to data center benchmarking in [RFC8238].  The
   author thanks the Benchmarking Methodology Working Group for that
   foundation.

Author's Address

   Madhava Gaikwad
   Independent
   Email: gaikwad.madhav@gmail.com

Gaikwad                 Expires 12 February 2027               [Page 14]