// REO script
!(function () {
  var e, t, n;
  (e = "638190bf025179e"),
    (t = function () {
      Reo.init({ clientID: "638190bf025179e" });
    }),
    ((n = document.createElement("script")).src = "https://static.reo.dev/" + e + "/reo.js"),
    (n.async = !0),
    (n.onload = t),
    document.head.appendChild(n);
})();

// Hubspot script
const script = document.createElement("script");
script.type = "text/javascript";
script.id = "hs-script-loader";
script.async = true;
script.defer = true;
script.src = "//js.hs-scripts.com/23114811.js";
document.head.appendChild(script);

// RB2B script.
!(function () {
  var reb2b = (window.reb2b = window.reb2b || []);
  if (reb2b.invoked) return;
  reb2b.invoked = true;
  reb2b.methods = ["identify", "collect"];
  reb2b.factory = function (method) {
    return function () {
      var args = Array.prototype.slice.call(arguments);
      args.unshift(method);
      reb2b.push(args);
      return reb2b;
    };
  };
  for (var i = 0; i < reb2b.methods.length; i++) {
    var key = reb2b.methods[i];
    reb2b[key] = reb2b.factory(key);
  }
  reb2b.load = function (key) {
    var script = document.createElement("script");
    script.type = "text/javascript";
    script.async = true;
    script.src = "https://s3-us-west-2.amazonaws.com/b2bjsstore/b/" + key + "/8XOE9GH5EDOM.js.gz";
    var first = document.getElementsByTagName("script")[0];
    first.parentNode.insertBefore(script, first);
  };
  reb2b.SNIPPET_VERSION = "1.0.1";
  reb2b.load("8XOE9GH5EDOM");
})();


Alerts that we support today

How to request a new alert?

Galileo

Client Reference

API Reference

Examples

AI Research

Deployments

Get GenAI Studio

Utilize Galileo LLM Fine-Tune's Alerts feature to detect and address dataset issues like high Data Error Potential scores and uncertainty outputs, enhancing data quality.

Using Alerts

Evaluate, Observe, and Protect your GenAI applications

What is Galileo?

Stop experimenting in spreadsheets and notebooks. Use Evaluate’s powerful insights to build GenAI systems that just work.

Overview of Galileo Evaluate

Start using Galileo Evaluate with this quickstart guide, covering prompt engineering, AI evaluation, and integrating tools into existing workflows.

Quickstart Guide | Galileo Evaluate

Learn how to integrate Galileo Evaluate into your Python applications, featuring step-by-step guidance and code samples for streamlined integration.

Integrate Evaluate Into My Existing Application With Python

Explore UI-driven prompt engineering in Galileo Evaluate to create, test, and refine prompts with intuitive interfaces and robust evaluation tools.

Prompt Engineering From A UI

Follow step-by-step instructions in Galileo Evaluate to assess generative AI models, configure metrics, and analyze performance effectively.

How-To Guide | Galileo Evaluate

Before starting your experiments, we recommend creating an evaluation set.

Create an Evaluation Set

How to use Galileo Evaluate for prompt engineering

Evaluate and Optimize Prompts

How to use Galileo Evaluate with RAG applications

Evaluate and Optimize RAG Applications

Select and understand guardrail metrics in Galileo Evaluate to effectively assess your prompts and models, utilizing both industry-standard and proprietary metrics.

Choose your Guardrail Metrics

Learn how to turn on metrics when creating runs in your Python environment.

Enabling Scorers in Runs

How to use Galileo Evaluate to find Hallucinations

Identify Hallucinations

Galileo GenAI Studio supports Custom Metrics (programmatic or GPT-based) for all your Evaluate and Observe projects. Depending on where, when, and how you want these metrics to be executed, you have the option to choose between **Custom Scorers** and **Registered Scorers**.

Register Custom Metrics

Manage and store your AI prompts efficiently in Galileo Evaluate, with tools for organizing, versioning, and analyzing prompt performance at scale.

Prompt Management-Storage

Easily compare multiple LLM runs in a single screen for better decision making

A/B Compare Prompts

Experiment with multiple prompts in Galileo Evaluate to optimize generative AI performance using iterative testing and comprehensive analysis tools.

Experiment with Multiple Prompts

If you're building a multi-step workflow or chain (e.g. a RAG system, an Agent, or a chain) and want to experiment with multiple combinations of parameters or your versions at once, Chain Sweeps are your friend.

Experiment with Multiple Workflows

Galileo allows you to do qualitative human evaluations of your prompts and responses.

Evaluate with Human Feedback

While you are experimenting with your prompts you will probably be tuning many parameters.

Add Tags and Metadata to Prompt Runs

If you already have a dataset of requests and application responses, and you want to log and evaluate these on Galileo without re-generating the responses, you can do so via our worflows.

Log Pre-generated Responses in Python

Evaluate and Optimize Agents

If you want to fetch your logged data and metrics programmatically, you can do so via our Python clients.

Programmatically fetch logged data

Galileo Evaluate is geared for cross-functional collaboration. Most of the teams using Galileo consist of a mix of the following personas

Collaborate with other personas

All projects on Galileo can be shared with others to enable collaboration.

Share a project

To download the results of your evaluation you can use the Export function. To export your runs, simply click on _Export Prompt Data._

Export your Evaluation Runs

Gain insights into your metric values in Galileo Evaluate with explainability features, including token-level highlighting and generated explanations for better analysis.

Understanding Metric Values | Galileo Evaluate How-To

Expected outputs are a key element for evaluating LLM applications. They provide benchmarks to measure model accuracy, identify errors, and ensure consistent assessments.

Logging and Comparing against your Expected Answers

Improve metric accuracy by customizing your Chainpoll-powered metrics

Customize Chainpoll-powered Metrics

Manage user permissions and securely share projects in Galileo Evaluate using detailed access control features, including system roles and group management.

Access Control Guide | Galileo Evaluate

Learn how to use Automatic Run Ranking to find the best run

Finding the best run

Discover Galileo Evaluate's integrations with AI tools and platforms, enabling seamless connectivity and enhanced generative AI evaluation workflows.

Integrations | Galileo Evaluate

Understand project concepts in Galileo Evaluate, including organization of datasets, metrics, and workflows for AI evaluation.

Project Concepts | Galileo Evaluate

Runs in Galileo are experiments or iterations done within a [project](/galileo/gen-ai-studio-products/galileo-evaluate/concepts/project).

Leverage templates in Galileo Evaluate to standardize metrics, model assessments, and workflows for efficient generative AI evaluation.

Template

Metrics are quantitative or qualitative ways to express insights about the [run](/galileo/gen-ai-studio-products/galileo-evaluate/concepts/run).

Metrics

Learn how human ratings in Galileo Evaluate enable accurate model evaluations and improve performance through qualitative feedback.

Human Ratings

Find solutions to common errors in computing metrics within Galileo Evaluate, including missing integrations and rate limit issues, to streamline your AI evaluations.

Error Computing Metrics | Galileo Evaluate FAQ

Understand the distinctions between Context Adherence and Instruction Adherence metrics in Galileo Evaluate to assess generative AI outputs accurately.

Context vs. Instruction Adherence | Galileo Evaluate FAQ

Monitor and analyze generative AI models with Galileo Observe, using real-time data insights to maintain performance and ensure quality outputs.

Overview of Galileo Observe

How to monitor your apps with Galileo Observe

Getting Started | Galileo Observe

Learn how to use Galileo Observe to monitor and analyze generative AI models, including setup instructions, data logging, and workflow integrations.

How-To Guide | Galileo Observe

Select and understand guardrail metrics in Galileo Observe to effectively evaluate your LLM applications, utilizing both industry-standard and proprietary metrics.

Choosing Your Guardrail Metrics

Registered Metrics enable the ability for your team to define the custom metrics (programmatic or GPT-based) for your Observe projects.

Registering And Using Custom Metrics

How to set up Alerts and automatically be alerted when things go wrong

Setting Up Alerts

Once your monitored LLM app is up and running and you've selected your Guardrail Metrics, you can start monitoring your LLM app using Galileo.

Identifying And Debugging Issues

Galileo Observe allows you to monitor your Retrieval-Augmented Generation (RAG) application with out-of-the-box Tracing and Analytics.

Monitoring Your Rag Application

Learn how to manually log your data via our Python Logger

Logging Data Via Python

Fetch logged data programmatically in Galileo Observe with step-by-step instructions for seamless integration into automated workflows and analysis tools.

Programmatically Fetching Logged Data

To download your Observe Data you can use the Export function.

Exporting Your Data

Gain insights into your metric values in Galileo Observe with explainability features, including token-level highlighting and generated explanations for better analysis.

Understanding Metric Values | Galileo Observe How-To

Manage user permissions and securely share projects in Galileo Observe using detailed access control features, including system roles and group management.

How to Set Up Access Control

Learn how to manually log your data from your Langchain Chains

Logging Data Via Langchain Callback

Troubleshoot common errors in Galileo Observe's metric computations, including integration issues, rate limits, JSON parsing errors, and missing embeddings, to ensure accurate evaluations.

Error Computing Metrics | Galileo Observe FAQ

Differentiate between Context Adherence and Instruction Adherence metrics in Galileo Observe to effectively evaluate and enhance your model's responses.

Context vs. Instruction Adherence | Galileo Observe FAQ

Explore Galileo Protect to safeguard AI applications with customizable rulesets, error detection, and robust metrics for enhanced AI governance.

Overview of Galileo Protect

Get started with Galileo Protect using this quickstart guide, covering setup, ruleset creation, and integration into AI workflows for secure operations.

Quickstart Guide | Galileo Protect

Follow detailed instructions on using Galileo Protect, including setting up rulesets, monitoring workflows, and ensuring secure AI application operations.

How-To Guide | Galileo Protect

Learn to create and manage stages in Galileo Protect, enabling structured AI monitoring and progressive error resolution throughout the deployment lifecycle.

Creating And Using Stages

Invoke rulesets in Galileo Protect to apply AI safeguards effectively, with comprehensive guidance on ruleset usage, configuration, and execution.

Invoking Rulesets

Explore supported metrics and operators in Galileo Protect to configure precise rulesets and enhance AI application monitoring and decision-making.

Defining Rules

Your Protect Rules rely on [Guardrail Metrics](/galileo/gen-ai-studio-products/galileo-protect/how-to/supported-metrics-and-operators). Metrics are calculated using ML models, which can have varying latencies.

Setting A Timeout On Your Protect Requests

When you're using the Galileo Protect product, once you've created a project and a stage, you can pause and resume the stage.

Pausing Or Resuming A Stage

Edit centralized stages in Galileo Protect with this guide, ensuring accurate ruleset updates and maintaining effective AI monitoring across applications.

Editing Centralized Stages

Galileo Protect can also be used within your Langchain workflows. You can use Protect to validate inputs and outputs at different stages of your workflow. We provide a `tool` that allows you to easily integrate Protect into your Langchain workflows. 

LangChain Integration | Galileo Protect

Understand project management in Galileo Protect, focusing on ruleset organization, AI model protection, and error monitoring within structured workflows.

Project Concepts | Galileo Protect

A set of rulesets that are applied during _one_ invocation.

Stage

All of the Rules within a Ruleset are executed in parallel, and the final resolution depends on all of the rules being completed.

Ruleset

A condition or rule you never want your application to break. It's composed of three ingredients

Rule

Galileo will provide a set of action types (override, passthrough), that the user can use, along with a configuration for each action type.

Action

Utilize Galileo's Guardrail Metrics to monitor generative AI models, ensuring adherence to quality, correctness, and alignment with project goals.

Overview of Galileo Guardrail Metrics

Measure ground truth adherence in generative AI models with Galileo's Guardrail Metrics, ensuring accurate and aligned outputs with dataset benchmarks.

Ground Truth Adherence

Assess instruction adherence in AI outputs using Galileo Guardrail Metrics to ensure prompt-driven models generate precise and actionable results.

Instruction Adherence

Correctness

Understand Galileo's Chunk Relevance Luna Metric

Chunk Relevance

Uncertainty

Understanding Galileo's Prompt Perplexity Metrics

Prompt Perplexity

Private Identifiable Information

Toxicity

Understand Galileo's Tool Selection Quality Metric

Tool Selection Quality

Tool Error

Tone

Sexism

BLEU and ROUGE

Understand Galileo's Prompt Injection metric

Prompt Injection

Understand the differences between Context Adherence and Instruction Adherence metrics in Galileo's Guardrail Metrics to accurately evaluate model outputs.

Context vs. Instruction Adherence | Guardrail Metrics FAQ

Find solutions to common errors in computing metrics within Galileo's Guardrail Metrics, including missing integrations and rate limit issues, to streamline your evaluations.

Error Computing Metrics | Guardrail Metrics FAQ

Fine-tune large language models with Galileo's LLM Fine-Tune tools, enabling precise adjustments for optimized AI performance and output quality.

Overview of Galileo LLM Fine-Tune

Get started with Galileo's LLM Fine-Tune in this quickstart guide, featuring step-by-step instructions for tuning AI models effectively.

Quickstart Guide | Galileo LLM Fine-Tune

Automatic Data Insights on your Seq2Seq dataset

Configuring Dq Auto

Upon completing a run, you'll be taken to the Galileo Console.

Console Walkthrough

Finetuning an LLM often requires large datasets.

Visualizing And Understanding Your Data

Learn about Galileo LLM Fine-Tune's Data Error Potential (DEP) score to identify and address errors in your training data, improving overall data quality.

Using Data Error Potential

On dataset splits where generations are enabled (e.g. the _Test split_), you'll be seeing Uncertainty Scores and Token-level Uncertainty highlighting

Using Uncertainty

Similarity search allows you to discover **similar samples** within your datasets

Finding Similar Samples

Take actionable steps in Galileo LLM Fine-Tune to address model performance issues, refine outputs, and achieve targeted AI improvements.

Taking Action

Galileo NLP Studio supports Natural Language Processing Tasks across the life-cycle of your model development.

Training High-Quality Supervised NLP Models | Galileo

Explore Galileo NLP Studio's features, including data insights, error detection, and monitoring tools for improving NLP workflows and AI quality.

Product Features

Discover Galileo NLP Studio's access control features, including user roles and group management, to securely share and manage projects within your organization.

Access Control Features | Galileo NLP Studio

The Dataset View provides an interactive data table for inspecting your datasets.

Dataset View

The Embeddings View provides a visual playground for you to interact with your datasets.


Hard for the model	Exposes the samples we believe are hard for your model to learn. These are the samples with high Data Error Potential scores.
Hard for the model cluster	Exposes clusters of data that have a high Data Error Potential.
High Uncertainty Outputs	Surfaces samples that have High Uncertainty on the generated output (only available if generations were created for this split).
High Perplexity Samples	Identifies samples whose predictions have high Perplexity.
Empty Samples	Identifies samples that have empty Input, empty Target or empty Generations.
Low Performing Cluster	Exposes clusters that have poor BLEU or ROUGE scores (only available if generations were created for this split).

Introduction

Evaluate

Observe

Protect

Galileo Guardrail Metrics

Fine Tune

Galileo NLP Studio

Using Alerts

Alerts that we support today

How to request a new alert?

Introduction

Evaluate

Observe

Protect

Galileo Guardrail Metrics

Fine Tune

Galileo NLP Studio

​Alerts that we support today

​How to request a new alert?

Alerts that we support today

How to request a new alert?