22 terms covering Evaluation, Observability, Prompting, Agents, AI & ML, Metrics, Infrastructure, Safety.
A metric that scores how well an LLM answer actually addresses the user question it was given, independent of whether the facts are correct. Low relevancy means the model wandered off topic or padded the response.
A retrieval metric that measures what fraction of the chunks fed into an LLM were actually relevant to the question, and whether the relevant ones ranked near the top. Low precision means the model was handed noise.
A numeric vector that represents the meaning of a piece of text, image, or other data so that similar items sit close together in vector space. Embeddings are the backbone of semantic search and retrieval.
An eval dataset is a curated collection of inputs, and often expected outputs, used to score an LLM application repeatedly and consistently. It is the fixed yardstick you run every change against.
Faithfulness measures how well an LLM response is supported by its retrieved context, scoring whether every claim can be traced back to the source material. It is a core metric for grading RAG systems.
A result that flags an issue which is not actually a problem. High false positive rates train users to ignore output.
Ground truth is the known-correct answer for a test case, used as the reference to score an LLM's output against. It is the standard that defines what a right answer looks like.
Programmatic checks that validate what goes into and comes out of an LLM, blocking or rewriting unsafe content before it reaches the model or the user. They enforce rules the model cannot be trusted to follow on its own.
A hallucination is when a language model generates confident, fluent text that is factually wrong or unsupported by its inputs. It is one of the central failure modes evaluation is meant to catch.
Human in the loop (HITL) means keeping a person involved in an automated system to review, correct or approve its outputs. In LLM evaluation it refers to human annotators scoring model responses that automation cannot judge reliably on its own.
LLM-as-a-judge is a technique where one language model scores or grades the output of another model against a rubric or reference. It scales evaluation to cases where writing exact-match rules is impractical.
The practice of instrumenting an LLM application so you can see what happened inside every request - the prompts, retrieved context, tool calls, tokens, cost, and latency. It is how you debug and improve a system you cannot step through line by line.
Offline evaluation scores an LLM application against a fixed dataset before deployment. It runs in development or CI, where inputs and expected outcomes are known ahead of time.
Online evaluation scores an LLM application on live production traffic, after responses have been served to real users. It measures quality continuously rather than on a fixed test set.
OpenTelemetry (OTel) is an open standard for collecting traces, metrics and logs from software. In LLM applications it defines a vendor-neutral format for spans that capture prompts, responses, token counts and latency.
An attack where crafted input tricks an LLM into ignoring its original instructions and following the attacker's instead. It is the LLM equivalent of an injection vulnerability, exploiting the fact that instructions and data share one text channel.
A three-part evaluation framework for Retrieval-Augmented Generation that scores context relevance, groundedness, and answer relevance together. Each leg isolates a different failure point in the pipeline.
Regression testing re-runs a fixed evaluation suite after every change to confirm that prompts, models or code updates have not degraded quality on cases that previously passed. It guards against silent quality drops.
A span is a single timed unit of work inside a trace, such as one LLM call, one retrieval step, or one tool invocation. Spans nest to show how a request flowed through an application.
A token is the basic unit of text a language model processes, typically a word fragment of a few characters. Models read and generate text as sequences of tokens, and providers price API usage per token.
A trace is the complete record of a single request through an LLM application, assembled from all the spans it generated. It shows every model call, retrieval, and tool step in the order they ran.
A result that correctly identifies a real issue. The true positive rate is the core measure of a tool's usefulness.
Platform comparisons, pricing changes and eval technique deep-dives. No spam.
By subscribing you agree to receive the newsletter. Unsubscribe anytime.