Experiment Metrics

Galileo AI provides a set of built-in, preset metrics designed to evaluate various aspects of LLM, agent, and retrieval-based workflows. You can also create custom metrics using LLM-as-a-judge, or code. This guide provides a reference for using these metrics in your experiments.

Out-of-the-box metrics reference

The table below summarizes gives the constants used in code to access each metric. To use these metrics, import the relevant enum.

from galileo import GalileoMetrics

import { GalileoMetrics } from "galileo";

LLM-as-a-judge Metrics

Python
TypeScript

Metric	Enum Value
Action Advancement	`GalileoMetrics.action_advancement`
Action Completion	`GalileoMetrics.action_completion`
Agent Efficiency	`GalileoMetrics.agent_efficiency`
Agent Flow	`GalileoMetrics.agent_flow`
Completeness	`GalileoMetrics.completeness`
Context Adherence	`GalileoMetrics.context_adherence`
Context Precision	`GalileoMetrics.context_precision`
Context Relevance (Query Adherence)	`GalileoMetrics.context_relevance`
Conversation Quality	`GalileoMetrics.conversation_quality`
Correctness (factuality)	`GalileoMetrics.correctness`
Unsafe Output	`GalileoMetrics.unsafe_output`
Ground Truth Adherence	`GalileoMetrics.ground_truth_adherence`
Instruction Adherence	`GalileoMetrics.instruction_adherence`
Interruption Detection	`GalileoMetrics.interruption_detection`
PII (personally identifiable information)	`GalileoMetrics.input_pii`, `GalileoMetrics.output_pii`
Prompt Injection	`GalileoMetrics.prompt_injection`
Sexism / Bias	`GalileoMetrics.input_sexism`, `GalileoMetrics.output_sexism`
Tone	`GalileoMetrics.input_tone`, `GalileoMetrics.output_tone`
Tool Errors	`GalileoMetrics.tool_error_rate`
Tool Selection Quality	`GalileoMetrics.tool_selection_quality`
Reasoning Coherence	`GalileoMetrics.reasoning_coherence`
SQL Correctness	`GalileoMetrics.sql_correctness`
SQL Adherence	`GalileoMetrics.sql_adherence`
SQL Injection	`GalileoMetrics.sql_injection`
SQL Efficiency	`GalileoMetrics.sql_efficiency`
Toxicity	`GalileoMetrics.input_toxicity`, `GalileoMetrics.output_toxicity`
User Intent Change	`GalileoMetrics.user_intent_change`
Visual Fidelity	`GalileoMetrics.visual_fidelity`
Visual Quality	`GalileoMetrics.visual_quality`

Metric	Enum Value
Action Advancement	`GalileoMetrics.actionAdvancement`
Action Completion	`GalileoMetrics.actionCompletion`
Agent Efficiency	`GalileoMetrics.agentEfficiency`
Agent Flow	`GalileoMetrics.agentFlow`
Completeness	`GalileoMetrics.completeness`
Context Adherence	`GalileoMetrics.contextAdherence`
Context Precision	`GalileoMetrics.contextPrecision`
Context Relevance (Query Adherence)	`GalileoMetrics.contextRelevance`
Conversation Quality	`GalileoMetrics.conversationQuality`
Correctness (factuality)	`GalileoMetrics.correctness`
Ground Truth Adherence	`GalileoMetrics.groundTruthAdherence`
Instruction Adherence	`GalileoMetrics.instructionAdherence`
PII (personally identifiable information)	`GalileoMetrics.inputPii`, `GalileoMetrics.outputPii`
Prompt Injection	`GalileoMetrics.promptInjection`
Sexism / Bias	`GalileoMetrics.inputSexism`, `GalileoMetrics.outputSexism`
Tone	`GalileoMetrics.inputTone`, `GalileoMetrics.outputTone`
Tool Errors	`GalileoMetrics.toolErrorRate`
Tool Selection Quality	`GalileoMetrics.toolSelectionQuality`
SQL Correctness	`GalileoMetrics.sqlCorrectness`
SQL Adherence	`GalileoMetrics.sqlAdherence`
SQL Injection	`GalileoMetrics.sqlInjection`
SQL Efficiency	`GalileoMetrics.sqlEfficiency`
Toxicity	`GalileoMetrics.inputToxicity`, `GalileoMetrics.outputToxicity`
User Intent Change	`GalileoMetrics.userIntentChange`

Luna-2 metrics

If you are using the Galileo Luna-2 model, then use these metric values.

Python
TypeScript

Metric	Enum Value
Action Advancement	`GalileoMetrics.action_advancement_luna`
Action Completion	`GalileoMetrics.action_completion_luna`
Completeness	`GalileoMetrics.completeness_luna`
Context Adherence	`GalileoMetrics.context_adherence_luna`
PII (personally identifiable information)	`GalileoMetrics.input_pii`, `GalileoMetrics.output_pii`
Prompt Injection	`GalileoMetrics.prompt_injection_luna`
Sexism / Bias	`GalileoMetrics.input_sexism_luna`, `GalileoMetrics.output_sexism_luna`
Tone	`GalileoMetrics.input_tone`, `GalileoMetrics.output_tone`
Tool Errors	`GalileoMetrics.tool_error_rate_luna`
Tool Selection Quality	`GalileoMetrics.tool_selection_quality_luna`
Toxicity	`GalileoMetrics.input_toxicity_luna`, `GalileoMetrics.output_toxicity_luna`

Metric	Enum Value
Action Advancement	`GalileoMetrics.actionAdvancementLuna`
Action Completion	`GalileoMetrics.actionCompletionLuna`
Completeness	`GalileoMetrics.completenessLuna`
Context Adherence	`GalileoMetrics.contextAdherenceLuna`
PII (personally identifiable information)	`GalileoMetrics.inputPii`, `GalileoMetrics.outputPii`
Prompt Injection	`GalileoMetrics.promptInjectionLuna`
Sexism / Bias	`GalileoMetrics.inputSexismLuna`, `GalileoMetrics.outputSexismLuna`
Tone	`GalileoMetrics.inputTone`, `GalileoMetrics.outputTone`
Tool Errors	`GalileoMetrics.toolErrorRateLuna`
Tool Selection Quality	`GalileoMetrics.toolSelectionQualityLuna`
Toxicity	`GalileoMetrics.inputToxicityLuna`, `GalileoMetrics.outputToxicityLuna`

How do I use metrics in experiments?

The run experiment function (Python, TypeScript) takes a list of metrics as part of its arguments.

Preset metrics

Supply a list of one or more metric names into the run_experiment function as shown below:

import os
from galileo.experiments import run_experiment
from galileo.datasets import get_dataset
from galileo.openai import openai
from galileo import GalileoMetrics

dataset = get_dataset(name="fictional_character_names")

# Define a custom "runner" function for your experiment.
def my_custom_llm_runner(input):
  client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
  return client.chat.completions.create(
        model="gpt-4o",
        messages=[
          { "role": "system", "content": "You are a great storyteller." },
          { "role": "user", "content": f"Write a story about {input["topic"]}" },
        ],
    ).choices[0].message.content

# Run the experiment!
results = run_experiment(
    "test-experiment",
    project="my-test-project-1",
    dataset=dataset,
    function=my_custom_llm_runner,
    metrics=[
        # List metrics here
        GalileoMetrics.action_advancement,
        GalileoMetrics.completeness,
        GalileoMetrics.instruction_adherence
    ],
)

import { GalileoMetrics, runExperiment } from "galileo";

async function runFunctionExperiment() {
  const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

  // Define a custom "runner" function for your experiment.
  const myCustomLLMRunner = async (input: any) => {
    const result = await openai.chat.completions.create({
      model: "gpt-4.1-mini",
      messages: [
        { role: "system", content: "You are a great storyteller." },
        { role: "user", content: `Write a story about ${input["topic"]}` }
      ]
    });

    return [result.choices[0].message.content];
  };

  // Run the experiment!
  await runExperiment({
    name: `Test Experiment`,
    projectName: "my-test-project-1",
    datasetName: "fictional_character_names",
    function: myCustomLLMRunner,
    metrics: [
      // List metrics here
      GalileoMetrics.actionAdvancement,
      GalileoMetrics.completeness,
      GalileoMetrics.instructionAdherence
    ]
  });
}

runFunctionExperiment();

For more information, read about running experiments with the Galileo SDKs.

Custom metrics

You can use custom metrics in the same way as Galileo’s preset metrics. At a high level, this involves the following steps:

Create your metric in the Galileo Console (or in code). Your custom metric will return a numerical score based on its input.
Pass the name of your new metric into the run experiment, like in the example below.

For example, if you have a metric called "Compliance - do not recommend any financial actions":

You would pass this to an experiment like this:

from galileo.experiments import run_experiment

results = run_experiment(
    "finance-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=["Compliance - do not recommend any financial actions"],
    project="my-project",
)

import {
  runExperiment
} from "galileo";

await runExperiment({
  name: "finance-experiment",
  dataset: dataset,
  function: wrappedRunner,
  metrics: ["Compliance - do not recommend any financial actions"],
  projectName: "my-project",
});

Custom metrics provide the flexibility to define precisely what you want to measure, enabling deep analysis and targeted improvement. For a detailed walkthrough on creating them, see Custom Metrics.

Ground truth data

Ground truth is the authoritative, validated answer or label used to benchmark model performance. For LLM metrics, this often means a gold-standard answer, fact, or supporting evidence against which outputs are compared. The following metrics require ground truth data to compute their scores, as they involve direct comparison to a reference answer, label, or fact.

Ground Truth Adherence

These metrics are only supported in experiments, as they require the ground truth to be set in the dataset used by the experiment.

To set the ground truth, set this in the output of your dataset either in the Galileo Console, or in code.

Are metrics LLM-agnostic?

Yes, all metrics are designed to work across any LLM integrated with Galileo.

Next steps

Metrics Overview

Explore Galileo’s comprehensive metrics framework for evaluating and improving AI system performance across multiple dimensions.

Experiments Overview

Learn how to use datasets and experiments to improve your application.

Run experiments

Learn how to run experiments in Galileo using the Galileo SDKs and custom metrics.

Overview

Get Started

Observability

Evaluation Metrics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Experiment Metrics

Out-of-the-box metrics reference

LLM-as-a-judge Metrics

Luna-2 metrics

How do I use metrics in experiments?

Preset metrics

Custom metrics

Ground truth data

Are metrics LLM-agnostic?

Next steps

Metrics Overview

Experiments Overview

Run experiments

​Out-of-the-box metrics reference

​LLM-as-a-judge Metrics

​Luna-2 metrics

​How do I use metrics in experiments?

​Preset metrics

​Custom metrics

​Ground truth data

​Are metrics LLM-agnostic?

​Next steps

Metrics Overview

Experiments Overview

Run experiments

Out-of-the-box metrics reference

LLM-as-a-judge Metrics

Luna-2 metrics

How do I use metrics in experiments?

Preset metrics

Custom metrics

Ground truth data

Are metrics LLM-agnostic?

Next steps