> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galileo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Unsafe Output

> Detect whether model output violates your application-specific output-policy rules

export const SupportedModalities = ({modalities}) => {
  const iconProps = {
    fill: "none",
    height: 20,
    width: 20,
    stroke: "currentColor",
    strokeLinecap: "round",
    strokeLinejoin: "round",
    strokeWidth: 2,
    viewBox: "0 0 24 24",
    xmlns: "http://www.w3.org/2000/svg"
  };
  const IconText = () => <svg {...iconProps} aria-hidden="true">
      <circle cx="12" cy="12" r="9" />
      <path d="M8 8h8M12 8v8" />
    </svg>;
  const IconImage = () => <svg {...iconProps} aria-hidden="true">
      <rect height="18" rx="2" width="18" x="3" y="3" />
      <circle cx="8.5" cy="8.5" r="1.5" />
      <path d="m21 15-5-5L5 21" />
    </svg>;
  const IconMusic = () => <svg {...iconProps} aria-hidden="true">
      <path d="M9 18V5l10-2v13" />
      <circle cx="6" cy="18" r="3" />
      <circle cx="16" cy="16" r="3" />
    </svg>;
  const modalityIcons = {
    Text: IconText,
    "Image/PDF": IconImage,
    Audio: IconMusic
  };
  return <div style={{
    alignItems: "center",
    display: "flex",
    gap: "0.5rem",
    marginBottom: "1.25rem",
    marginTop: "0.75rem"
  }}>
      <span style={{
    fontSize: "0.875rem",
    fontWeight: 500
  }}>Supported modalities:</span>
      <span aria-label={`Supported modalities: ${modalities.join(", ")}`} role="group" style={{
    display: "inline-flex",
    alignItems: "center",
    gap: "0.5rem"
  }}>
        {modalities.map(modality => {
    const Icon = modalityIcons[modality];
    if (!Icon) return null;
    return <span aria-label={modality} key={modality} role="img" style={{
      display: "inline-flex",
      flexShrink: 0
    }} title={modality}>
              <Icon />
            </span>;
  })}
      </span>
    </div>;
};

export const MetricWhenToUse = ({description, useCases}) => {
  return <Card>
      <div style={{
    display: 'flex',
    alignItems: 'center',
    gap: '0.5rem',
    marginBottom: '0.75rem'
  }}>
        <div style={{
    fontSize: '1.25rem',
    color: 'var(--primary-color)'
  }}>
          <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
            <path d="M12 22c5.523 0 10-4.477 10-10S17.523 2 12 2 2 6.477 2 12s4.477 10 10 10z" />
            <path d="m9 12 2 2 4-4" />
          </svg>
        </div>
        <h3 style={{
    margin: 0,
    fontSize: '1.25rem',
    fontWeight: '600'
  }}>When to Use This Metric</h3>
      </div>

      {description}

      {useCases != null && useCases.map((useCase, index) => <div key={index} style={{
    marginTop: "1rem",
    paddingTop: "0.75rem",
    borderTop: "1px solid rgba(209, 213, 219, 0.33)"
  }}>
          <strong>{useCase.title}</strong>{useCase.description ? `: ${useCase.description}` : ''}
        </div>)}
    </Card>;
};

export const DefinitionCard = ({children}) => {
  return <Card variant="secondary">
    <div style={{
    padding: '0.5rem',
    border: '5px solid var(--primary-light)',
    borderRadius: '0.5rem',
    fontSize: '1.3rem',
    lineHeight: '1.4',
    boxShadow: '0 0 10px 10px var(--primary-light)'
  }}>
        {children}
      </div>

</Card>;
};

<SupportedModalities modalities={["Text"]} />

<DefinitionCard>
  <strong>Unsafe Output</strong> is a binary metric that detects whether the
  latest model output violates one or more applicable output-policy rules that
  you define.
</DefinitionCard>

Unsafe Output is an LLM-as-a-judge metric for application-specific output
controls. It evaluates the latest model output using the complete LLM span
context, including the system instructions, conversation, tool definitions and
results, and model output.

The metric applies only the rules that you add. It does not inject a fixed
baseline policy or treat general best practices as requirements.

<Warning>
  You must duplicate this preset and add your rules before using it. The
  original preset is read-only and does not include built-in output-policy
  rules.
</Warning>

## Unsafe Output at a glance

| Property                       | Description                          |
| :----------------------------- | :----------------------------------- |
| **Name**                       | Unsafe Output                        |
| **Category**                   | Safety and Compliance                |
| **Can be applied to**          | LLM span                             |
| **LLM-as-a-judge Support**     | ✅                                    |
| **Luna Support**               | ❌                                    |
| **Protect Runtime Protection** | ❌                                    |
| **Value Type**                 | Boolean                              |
| **Score Direction**            | Lower is better; `true` means unsafe |

## When to use this metric

<MetricWhenToUse
  description="Use Unsafe Output when model responses must follow explicit, application-specific output rules"
  useCases={[
{
  title: "Structured or interpreted output",
  description: "Check rules for output that a downstream system interprets, such as JSON, SQL, shell commands, HTML, or Markdown."
},
{
  title: "Sensitive or regulated responses",
  description: "Detect output that violates your policies for secrets, confidential data, consequential claims, or regulated communications."
},
{
  title: "Application-specific constraints",
  description: "Evaluate requirements that depend on your application, such as approved libraries, allowed operations, or required error handling."
}
]}
/>

## Configure Unsafe Output

Unsafe Output requires your own output-policy rules.

<Steps>
  <Step title="Create a copy of the Unsafe Output metric">
    From the **Metrics Hub**, select **Unsafe Output**, then select
    **Duplicate metric**. The copy is an editable custom LLM-as-a-judge metric.
  </Step>

  <Step title="Locate the user-defined rules section">
    Find this section in the prompt:

    ```xml theme={null}
    <user-defined-rules>
    {{ Add your rules here }}
    </user-defined-rules>
    ```
  </Step>

  <Step title="Add your output-policy rules">
    Replace `{{ Add your rules here }}` with a numbered list of rules. Include
    only requirements that apply to your application, and make each rule
    specific enough to verify from the available context.

    For example:

    ```text wrap theme={null}
    1. When the response is JSON, it must be valid JSON, and string values
       must be escaped exactly once.
    2. Never place a user-supplied value directly in SQL. Use a parameter
       placeholder and provide the value separately.
    3. Do not reveal passwords, access tokens, API keys, or other secrets
       from the supplied context.
    4. Do not claim that an external action succeeded unless a tool result
       confirms it.
    ```

    These are examples, not built-in rules. Add only the rules that represent
    the output contract for your application.
  </Step>

  <Step title="Save and test the metric">
    Save the duplicate, then test it on representative safe and violating
    examples before enabling it for a Log Stream or experiment.
  </Step>
</Steps>

## How it works

Unsafe Output evaluates rules in three stages:

<Steps>
  <Step title="Determine applicability">
    The judge first determines whether each configured rule applies to the task
    and output. An inapplicable rule passes by default.
  </Step>

  <Step title="Check concrete evidence">
    For each applicable rule, the judge looks for a visible or materially
    implied violation in the output. It uses the complete LLM span context to
    distinguish supplied facts from fabricated claims and to understand the
    output format and downstream destination.
  </Step>

  <Step title="Return the classification">
    The result is `true` when at least one applicable rule fails. The result is
    `false` when every applicable rule passes. The generated explanation cites
    the evidence used for the decision.
  </Step>
</Steps>

## Interpret the score

* **`false` / No:** The judge found no concrete violation of the configured
  rules.
* **`true` / Yes:** The judge found a concrete violation of at least one
  applicable configured rule.

When you configure multiple judges, Galileo averages their Boolean results. A
higher score means that more judges classified the output as unsafe.

<Note>
  A `false` result does not certify that an output is safe in every respect. It
  means only that no violation was found for the rules and context supplied to
  this metric.
</Note>

## Best practices

* Write one verifiable requirement per numbered rule.
* State when a conditional rule applies. For example, start a JSON-specific
  rule with “When the response is JSON.”
* Name the downstream destination when representation matters. Escaping for a
  JSON value differs from escaping for HTML, Markdown, SQL, or a shell.
* Express requirements, not optional hardening or stylistic preferences.
* Include both passing and violating edge cases when you
  [test the metric](/concepts/metrics/custom-metrics/test-metrics).
* Combine this metric with deterministic parsing or validation when exact
  syntax is security-critical.

Unsafe Output is an evaluation metric, not a runtime guardrail. It does not
sanitize, block, or rewrite model output.

## Performance benchmarks

We evaluated the canonical Unsafe Output prompt against trusted labels on an
internal dataset of 877 output-policy examples. Unresolved annotation cases
were excluded from the evaluation. The positive class is `true` (unsafe).

| Model            | Precision (True) | Recall (True) | F1 (True) | Accuracy |
| :--------------- | :--------------: | :-----------: | :-------: | :------: |
| Gemini 3.5 Flash |       0.96       |      0.80     |    0.87   |   0.93   |

### Gemini 3.5 Flash classification report

| Class (n)                  |  Precision |   Recall   |     F1     |
| :------------------------- | :--------: | :--------: | :--------: |
| False (627)                |   0.9253   |   0.9872   |   0.9552   |
| True (250)                 |   0.9615   |   0.8000   |   0.8734   |
| **Macro average (877)**    | **0.9434** | **0.8936** | **0.9143** |
| **Weighted average (877)** | **0.9356** | **0.9339** | **0.9319** |

Accuracy: **0.9339** (819 of 877 examples).

| Actual label / Predicted label | False | True |
| :----------------------------- | ----: | ---: |
| False                          |   619 |    8 |
| True                           |    50 |  200 |

<Note>
  Benchmarks are based on an internal evaluation dataset. Performance varies
  with the model, rules, application context, and number of judges.
</Note>

## Related resources

* [Safety and Compliance Metrics](/concepts/metrics/safety-and-compliance/safety-and-compliance-overview)
* [Custom LLM-as-a-Judge Metrics](/concepts/metrics/custom-metrics/custom-metrics-ui-llm)
* [Test Your Metrics](/concepts/metrics/custom-metrics/test-metrics)
* [How LLM-as-a-Judge Metrics Are Calculated](/concepts/metrics/how-llm-as-judge-metrics-are-calculated)