Supported decision models
Galileo supports the following decision model:How decision model metrics work
A decision model is not an LLM. You provide a record, a plain-language question, and the allowed answers. The model returns an answer with probabilities and does not provide a written explanation. The following table describes the supported question types and results:When to use a decision model metric
Decision model metrics classify and score large volumes of text faster and at a much lower cost than LLM-as-a-judge metrics. For example, you can use decision model metrics to monitor high-volume Log streams at full coverage and perform classification-style checks for frustration, task completion, tone, and policy adherence. Use an LLM-as-a-judge metric when you need explanations, counts or percentages, images, audio or files, or very long inputs. For more information on metric differences, see Which metric should I use?Create and use decision model metrics
Create a decision model metric
Before you begin, set up a decision model integration.1
Navigate to the Metrics page
From the Galileo main menu, select Metrics.
2
Configure the metric
Select Create metric. Follow the on-screen instructions to configure the metric.If your experiment needs an expected answer, turn on Use ground truth. For guidance on writing questions, see How to write effective questions.
3
Test the metric
Select Test metric to test the metric.You can test with manual input, current logs, or a dataset. Dataset testing scores Macro F1 for boolean and categorical, and RMSE for discrete. See Test Metrics for more information.Update the metric as needed, then select Create Metric.
Decision model metric batching
When decision model metrics use the same model and inputs, Galileo batches up to 10 metrics in a single call to reduce cost. Each metric still receives its own answer. If one metric in the batch fails, all metrics in the batch fail and show a clear error. Hover over a metric value to see how many metrics were included in the batch. The cost shown for each metric is an estimate calculated as a fraction of the total batch cost based on the batch size.How to write effective questions
Ask one question per metric. If a rubric checks several qualities, split it into separate metrics. Follow these guidelines based on your question type:- For a question with multiple applicable labels, create one boolean metric per label.
- For a categorical question, consider creating an Unclear category. The model always selects an answer, even when the record doesn’t clearly fit the other categories.
- For a discrete question, describe each level in concrete terms so adjacent levels are distinguishable.
- “Was the user frustrated?”
- “Did the agent complete the task?”
- “Did the response follow the stated policy?”
Use decision model metrics
Decision model metrics work in Log streams, experiments, and playgrounds, at every step level. To enable one, select Configure Metrics within one of those pages and turn on a metric marked with the Decision type. Composite metrics can use decision model results. For a boolean decision model metric, the composite receives its probability. Charts, filters, alerts, and roll-ups work as they do for other metrics with the same output type.Read decision model metric results
A decision model result appears like any other metric result with the same output type. Hover over a result to see its confidence, answer probabilities, model version, token usage, and estimated cost. Each result has one of four states: Success, N/A, Error, or Pending.Limits and troubleshooting
Decision models accept text only. A record containing images, audio, or documents shows N/A. The record and question must fit within a 32k-token context window. Galileo does not trim oversized input. If a shared call fails, every metric in that call shows the error. The following table describes common errors and how to troubleshoot them:
See Decision Models for more information on the decision model integration, including billing and rate limits.