// REO script
!(function () {
  var e, t, n;
  (e = "638190bf025179e"),
    (t = function () {
      Reo.init({ clientID: "638190bf025179e" });
    }),
    ((n = document.createElement("script")).src = "https://static.reo.dev/" + e + "/reo.js"),
    (n.async = !0),
    (n.onload = t),
    document.head.appendChild(n);
})();

// Hubspot script
const script = document.createElement("script");
script.type = "text/javascript";
script.id = "hs-script-loader";
script.async = true;
script.defer = true;
script.src = "//js.hs-scripts.com/23114811.js";
document.head.appendChild(script);

// RB2B script.
!(function () {
  var reb2b = (window.reb2b = window.reb2b || []);
  if (reb2b.invoked) return;
  reb2b.invoked = true;
  reb2b.methods = ["identify", "collect"];
  reb2b.factory = function (method) {
    return function () {
      var args = Array.prototype.slice.call(arguments);
      args.unshift(method);
      reb2b.push(args);
      return reb2b;
    };
  };
  for (var i = 0; i < reb2b.methods.length; i++) {
    var key = reb2b.methods[i];
    reb2b[key] = reb2b.factory(key);
  }
  reb2b.load = function (key) {
    var script = document.createElement("script");
    script.type = "text/javascript";
    script.async = true;
    script.src = "https://s3-us-west-2.amazonaws.com/b2bjsstore/b/" + key + "/8XOE9GH5EDOM.js.gz";
    var first = document.getElementsByTagName("script")[0];
    first.parentNode.insertBefore(script, first);
  };
  reb2b.SNIPPET_VERSION = "1.0.1";
  reb2b.load("8XOE9GH5EDOM");
})();


Galileo

Client Reference

API Reference

Examples

AI Research

Deployments

Get GenAI Studio

Invoke

Value

ActionResult

ActionType

ExecutionStatus

Error Message

Execution Time

MetricComputation

List of choices to override the response with. If there are multiple choices, one will be chosen at random when applying this action.

Choices

List of subscriptions to send a notification to when this action is applied and the ruleset status matches any of the configured statuses.

Subscriptions

Type

OverrideAction

PassthroughAction

Input

Output

Payload

Metric

Value to compare with for this metric (right hand side).

Target Value

Rule

RuleOperator

RuleResult

Action

Description

List of rules to evaluate. Atleast 1 rule is required.

Rules

Ruleset

Rule Results

RulesetResult

Project Id

Stage Id

Stage Name

Stage Version

StageMetadata

List of statuses that will cause a notification to be sent to the configured URL.

Statuses

URL to send the event to. This can be a webhook URL, a message queue URL, an event bus or a custom endpoint that can receive an HTTP POST request.

SubscriptionConfig

Execution time for the request (in seconds).

Time the request was received by the server in nanoseconds.

Received At

Time the response was sent by the server in nanoseconds.

Response At

TraceMetadata

Location

Message

Error Type

ValidationError

Request

APIKeyHeader

OAuth2PasswordBearer

HTTPBasic

Response Invoke V1 Protect Invoke Post

HTTPValidationError

Evaluate, Observe, and Protect your GenAI applications

What is Galileo?

Stop experimenting in spreadsheets and notebooks. Use Evaluate’s powerful insights to build GenAI systems that just work.

Overview of Galileo Evaluate

Start using Galileo Evaluate with this quickstart guide, covering prompt engineering, AI evaluation, and integrating tools into existing workflows.

Quickstart Guide | Galileo Evaluate

Learn how to integrate Galileo Evaluate into your Python applications, featuring step-by-step guidance and code samples for streamlined integration.

Integrate Evaluate Into My Existing Application With Python

Explore UI-driven prompt engineering in Galileo Evaluate to create, test, and refine prompts with intuitive interfaces and robust evaluation tools.

Prompt Engineering From A UI

Follow step-by-step instructions in Galileo Evaluate to assess generative AI models, configure metrics, and analyze performance effectively.

How-To Guide | Galileo Evaluate

Before starting your experiments, we recommend creating an evaluation set.

Create an Evaluation Set

How to use Galileo Evaluate for prompt engineering

Evaluate and Optimize Prompts

How to use Galileo Evaluate with RAG applications

Evaluate and Optimize RAG Applications

Select and understand guardrail metrics in Galileo Evaluate to effectively assess your prompts and models, utilizing both industry-standard and proprietary metrics.

Choose your Guardrail Metrics

Learn how to turn on metrics when creating runs in your Python environment.

Enabling Scorers in Runs

How to use Galileo Evaluate to find Hallucinations

Identify Hallucinations

Galileo GenAI Studio supports Custom Metrics (programmatic or GPT-based) for all your Evaluate and Observe projects. Depending on where, when, and how you want these metrics to be executed, you have the option to choose between **Custom Scorers** and **Registered Scorers**.

Register Custom Metrics

Manage and store your AI prompts efficiently in Galileo Evaluate, with tools for organizing, versioning, and analyzing prompt performance at scale.

Prompt Management-Storage

Easily compare multiple LLM runs in a single screen for better decision making

A/B Compare Prompts

Experiment with multiple prompts in Galileo Evaluate to optimize generative AI performance using iterative testing and comprehensive analysis tools.

Experiment with Multiple Prompts

If you're building a multi-step workflow or chain (e.g. a RAG system, an Agent, or a chain) and want to experiment with multiple combinations of parameters or your versions at once, Chain Sweeps are your friend.

Experiment with Multiple Workflows

Galileo allows you to do qualitative human evaluations of your prompts and responses.

Evaluate with Human Feedback

While you are experimenting with your prompts you will probably be tuning many parameters.

Add Tags and Metadata to Prompt Runs

If you already have a dataset of requests and application responses, and you want to log and evaluate these on Galileo without re-generating the responses, you can do so via our worflows.

Log Pre-generated Responses in Python

Evaluate and Optimize Agents

If you want to fetch your logged data and metrics programmatically, you can do so via our Python clients.

Programmatically fetch logged data

Galileo Evaluate is geared for cross-functional collaboration. Most of the teams using Galileo consist of a mix of the following personas

Collaborate with other personas

All projects on Galileo can be shared with others to enable collaboration.

Share a project

To download the results of your evaluation you can use the Export function. To export your runs, simply click on _Export Prompt Data._

Export your Evaluation Runs

Gain insights into your metric values in Galileo Evaluate with explainability features, including token-level highlighting and generated explanations for better analysis.

Understanding Metric Values | Galileo Evaluate How-To

Expected outputs are a key element for evaluating LLM applications. They provide benchmarks to measure model accuracy, identify errors, and ensure consistent assessments.

Logging and Comparing against your Expected Answers

Improve metric accuracy by customizing your Chainpoll-powered metrics

Customize Chainpoll-powered Metrics

Manage user permissions and securely share projects in Galileo Evaluate using detailed access control features, including system roles and group management.

Access Control Guide | Galileo Evaluate

Learn how to use Automatic Run Ranking to find the best run

Finding the best run

Discover Galileo Evaluate's integrations with AI tools and platforms, enabling seamless connectivity and enhanced generative AI evaluation workflows.

Integrations | Galileo Evaluate

Understand project concepts in Galileo Evaluate, including organization of datasets, metrics, and workflows for AI evaluation.

Project Concepts | Galileo Evaluate

Runs in Galileo are experiments or iterations done within a [project](/galileo/gen-ai-studio-products/galileo-evaluate/concepts/project).

Leverage templates in Galileo Evaluate to standardize metrics, model assessments, and workflows for efficient generative AI evaluation.

Template

Metrics are quantitative or qualitative ways to express insights about the [run](/galileo/gen-ai-studio-products/galileo-evaluate/concepts/run).

Metrics

Learn how human ratings in Galileo Evaluate enable accurate model evaluations and improve performance through qualitative feedback.

Human Ratings

Find solutions to common errors in computing metrics within Galileo Evaluate, including missing integrations and rate limit issues, to streamline your AI evaluations.

Error Computing Metrics | Galileo Evaluate FAQ

Understand the distinctions between Context Adherence and Instruction Adherence metrics in Galileo Evaluate to assess generative AI outputs accurately.

Context vs. Instruction Adherence | Galileo Evaluate FAQ

Monitor and analyze generative AI models with Galileo Observe, using real-time data insights to maintain performance and ensure quality outputs.

Overview of Galileo Observe

How to monitor your apps with Galileo Observe

Getting Started | Galileo Observe

Learn how to use Galileo Observe to monitor and analyze generative AI models, including setup instructions, data logging, and workflow integrations.

How-To Guide | Galileo Observe

Select and understand guardrail metrics in Galileo Observe to effectively evaluate your LLM applications, utilizing both industry-standard and proprietary metrics.

Choosing Your Guardrail Metrics

Registered Metrics enable the ability for your team to define the custom metrics (programmatic or GPT-based) for your Observe projects.

Registering And Using Custom Metrics

How to set up Alerts and automatically be alerted when things go wrong

Setting Up Alerts

Once your monitored LLM app is up and running and you've selected your Guardrail Metrics, you can start monitoring your LLM app using Galileo.

Identifying And Debugging Issues

Galileo Observe allows you to monitor your Retrieval-Augmented Generation (RAG) application with out-of-the-box Tracing and Analytics.

Monitoring Your Rag Application

Learn how to manually log your data via our Python Logger

Logging Data Via Python

Fetch logged data programmatically in Galileo Observe with step-by-step instructions for seamless integration into automated workflows and analysis tools.

Programmatically Fetching Logged Data

To download your Observe Data you can use the Export function.

Exporting Your Data

Gain insights into your metric values in Galileo Observe with explainability features, including token-level highlighting and generated explanations for better analysis.

Understanding Metric Values | Galileo Observe How-To

Manage user permissions and securely share projects in Galileo Observe using detailed access control features, including system roles and group management.

How to Set Up Access Control

Learn how to manually log your data from your Langchain Chains

Logging Data Via Langchain Callback

Troubleshoot common errors in Galileo Observe's metric computations, including integration issues, rate limits, JSON parsing errors, and missing embeddings, to ensure accurate evaluations.

Error Computing Metrics | Galileo Observe FAQ

Differentiate between Context Adherence and Instruction Adherence metrics in Galileo Observe to effectively evaluate and enhance your model's responses.

Context vs. Instruction Adherence | Galileo Observe FAQ

Explore Galileo Protect to safeguard AI applications with customizable rulesets, error detection, and robust metrics for enhanced AI governance.

Overview of Galileo Protect

Get started with Galileo Protect using this quickstart guide, covering setup, ruleset creation, and integration into AI workflows for secure operations.

Quickstart Guide | Galileo Protect

Follow detailed instructions on using Galileo Protect, including setting up rulesets, monitoring workflows, and ensuring secure AI application operations.

How-To Guide | Galileo Protect

Learn to create and manage stages in Galileo Protect, enabling structured AI monitoring and progressive error resolution throughout the deployment lifecycle.

Creating And Using Stages

Invoke rulesets in Galileo Protect to apply AI safeguards effectively, with comprehensive guidance on ruleset usage, configuration, and execution.

Invoking Rulesets

Explore supported metrics and operators in Galileo Protect to configure precise rulesets and enhance AI application monitoring and decision-making.

Defining Rules

Your Protect Rules rely on [Guardrail Metrics](/galileo/gen-ai-studio-products/galileo-protect/how-to/supported-metrics-and-operators). Metrics are calculated using ML models, which can have varying latencies.

Setting A Timeout On Your Protect Requests

When you're using the Galileo Protect product, once you've created a project and a stage, you can pause and resume the stage.

Pausing Or Resuming A Stage

Edit centralized stages in Galileo Protect with this guide, ensuring accurate ruleset updates and maintaining effective AI monitoring across applications.

Editing Centralized Stages

Galileo Protect can also be used within your Langchain workflows. You can use Protect to validate inputs and outputs at different stages of your workflow. We provide a `tool` that allows you to easily integrate Protect into your Langchain workflows. 

LangChain Integration | Galileo Protect

Understand project management in Galileo Protect, focusing on ruleset organization, AI model protection, and error monitoring within structured workflows.

Project Concepts | Galileo Protect

A set of rulesets that are applied during _one_ invocation.

Stage

All of the Rules within a Ruleset are executed in parallel, and the final resolution depends on all of the rules being completed.

A condition or rule you never want your application to break. It's composed of three ingredients

Galileo will provide a set of action types (override, passthrough), that the user can use, along with a configuration for each action type.

Utilize Galileo's Guardrail Metrics to monitor generative AI models, ensuring adherence to quality, correctness, and alignment with project goals.

Overview of Galileo Guardrail Metrics

Measure ground truth adherence in generative AI models with Galileo's Guardrail Metrics, ensuring accurate and aligned outputs with dataset benchmarks.

API Reference

health

auth

evaluate

observe

protect

Invoke

Authorizations

Body

Response