evaluate

Create a new Evaluate Run

Create a new Evaluate run with workflows.

Use this endpoint to create a new Evaluate run with workflows. The request body should contain the workflows to be ingested and evaluated.

Additionally, specify the project_id or project_name to which the workflows should be ingested. If the project does not exist, it will be created. If the project exists, the workflows will be logged to it. If both project_id and project_name are provided, project_id will take precedence. The run_name is optional and will be auto-generated (timestamp-based) if not provided.

The body is also expected to include the configuration for the scorers to be used in the evaluation. This configuration will be used to evaluate the workflows and generate the results.

POST

evaluate

runs

curl --request POST \
  --url https://api.acme.rungalileo.io/v1/evaluate/runs \
  --header 'Content-Type: application/json' \
  --header 'Galileo-API-Key: <api-key>' \
  --data '{
  "project_name": "my-evaluate-project",
  "run_name": "my-evaluate-run",
  "scorers": [
    {
      "name": "correctness"
    },
    {
      "name": "output_pii"
    }
  ],
  "workflows": [
    {
      "created_at_ns": 1743174002288939000,
      "duration_ns": 0,
      "input": "who is a smart LLM?",
      "metadata": {},
      "name": "llm",
      "output": "I am!",
      "type": "llm"
    }
  ]
}'

{
  "message": "<string>",
  "project_id": "<string>",
  "project_name": "<string>",
  "run_id": "<string>",
  "run_name": "<string>",
  "workflows_count": 123,
  "records_count": 123
}

WorkflowStep

A workflow step is the atomic unit of logging to Galileo. They represent a single execution of a workflow, such as a chain, agent, or a RAG execution. Workflows can have multiple steps, each of which can be a different type of node, such as an LLM, Retriever, or Tool.

You can log multiple workflows in a single request. Each workflow step must have the following fields:

type: The type of the workflow.
input: The input to the workflow.
output: The output of the workflow.

Examples

LLM Step

{
  "type": "llm",
  "input": "What is the capital of France?",
  "output": "Paris"
}

Retriever Step

{
  "type": "retriever",
  "input": "What is the capital of France?",
  "output": [{ "content": "Paris is the capital and largest city of France." }]
}

Multi-Step

Workflow steps of type workflow, agent or chain can have sub-steps with children. A workflow with a retriver and an LLM step would look like this:

{
  "type": "workflow",
  "input": "What is the capital of France?",
  "output": "Paris",
  "steps": [
    {
      "type": "retriever",
      "input": "What is the capital of France?",
      "output": [{ "content": "Paris is the capital and largest city of France." }]
    },
    {
      "type": "llm",
      "input": "What is the capital of France?",
      "output": "Paris"
    }
  ]
}

Authorizations

Galileo-API-Key

string

header

required

Body

application/json

workflows

object[]

required

List of workflows to include in the run.

workflows.input

required

Input to the step.

workflows.type

string

default:workflow

Type of the step. By default, it is set to workflow.

Allowed value: "workflow"

workflows.output

default:

Output of the step.

workflows.name

string

default:

Name of the step.

workflows.created_at_ns

integer

Timestamp of the step's creation, as nanoseconds since epoch.

workflows.duration_ns

integer

default:0

Duration of the step in nanoseconds.

workflows.metadata

object

Metadata associated with this step.

workflows.status_code

integer | null

Status code of the step. Used for logging failed/errored steps.

workflows.ground_truth

string | null

Ground truth expected output for the step.

workflows.steps

object[]

Steps in the workflow.

workflows.steps.input

required

Input to the step.

workflows.steps.type

string

default:workflow

Type of the step. By default, it is set to workflow.

Allowed value: "workflow"

workflows.steps.output

default:

Output of the step.

workflows.steps.name

string

default:

Name of the step.

workflows.steps.created_at_ns

integer

Timestamp of the step's creation, as nanoseconds since epoch.

workflows.steps.duration_ns

integer

default:0

Duration of the step in nanoseconds.

workflows.steps.metadata

object

Metadata associated with this step.

workflows.steps.status_code

integer | null

Status code of the step. Used for logging failed/errored steps.

workflows.steps.ground_truth

string | null

Ground truth expected output for the step.

workflows.steps.steps

object[]

Steps in the workflow.

workflows.steps.steps.input

required

Input to the step.

workflows.steps.steps.type

string

default:workflow

Type of the step. By default, it is set to workflow.

Allowed value: "workflow"

workflows.steps.steps.output

default:

Output of the step.

workflows.steps.steps.name

string

default:

Name of the step.

workflows.steps.steps.created_at_ns

integer

Timestamp of the step's creation, as nanoseconds since epoch.

workflows.steps.steps.duration_ns

integer

default:0

Duration of the step in nanoseconds.

workflows.steps.steps.metadata

object

Metadata associated with this step.

workflows.steps.steps.status_code

integer | null

Status code of the step. Used for logging failed/errored steps.

workflows.steps.steps.ground_truth

string | null

Ground truth expected output for the step.

workflows.steps.steps.steps

object[]

Steps in the workflow.

workflows.steps.steps.parent

object | null

Parent node of the current node. For internal use only.

workflows.steps.parent

object | null

Parent node of the current node. For internal use only.

workflows.steps.parent.input

required

Input to the step.

workflows.steps.parent.type

enum<string>

Type of the step. By default, it is set to workflow.

Available options:

chain,

chat,

llm,

retriever,

tool,

agent,

workflow,

trace

workflows.steps.parent.output

default:

Output of the step.

workflows.steps.parent.name

string

default:

Name of the step.

workflows.steps.parent.created_at_ns

integer

Timestamp of the step's creation, as nanoseconds since epoch.

workflows.steps.parent.duration_ns

integer

default:0

Duration of the step in nanoseconds.

workflows.steps.parent.metadata

object

Metadata associated with this step.

workflows.steps.parent.status_code

integer | null

Status code of the step. Used for logging failed/errored steps.

workflows.steps.parent.ground_truth

string | null

Ground truth expected output for the step.

workflows.steps.parent.steps

object[]

Steps in the workflow.

parent

object

workflows.parent

object | null

Parent node of the current node. For internal use only.

workflows.parent.input

required

Input to the step.

workflows.parent.type

enum<string>

Type of the step. By default, it is set to workflow.

Available options:

chain,

chat,

llm,

retriever,

tool,

agent,

workflow,

trace

workflows.parent.output

default:

Output of the step.

workflows.parent.name

string

default:

Name of the step.

workflows.parent.created_at_ns

integer

Timestamp of the step's creation, as nanoseconds since epoch.

workflows.parent.duration_ns

integer

default:0

Duration of the step in nanoseconds.

workflows.parent.metadata

object

Metadata associated with this step.

workflows.parent.status_code

integer | null

Status code of the step. Used for logging failed/errored steps.

workflows.parent.ground_truth

string | null

Ground truth expected output for the step.

workflows.parent.steps

object[]

Steps in the workflow.

workflows.parent.steps.input

required

Input to the step.

workflows.parent.steps.type

string

default:workflow

Type of the step. By default, it is set to workflow.

Allowed value: "workflow"

workflows.parent.steps.output

default:

Output of the step.

workflows.parent.steps.name

string

default:

Name of the step.

workflows.parent.steps.created_at_ns

integer

Timestamp of the step's creation, as nanoseconds since epoch.

workflows.parent.steps.duration_ns

integer

default:0

Duration of the step in nanoseconds.

workflows.parent.steps.metadata

object

Metadata associated with this step.

workflows.parent.steps.status_code

integer | null

Status code of the step. Used for logging failed/errored steps.

workflows.parent.steps.ground_truth

string | null

Ground truth expected output for the step.

workflows.parent.steps.steps

object[]

Steps in the workflow.

workflows.parent.steps.parent

object | null

Parent node of the current node. For internal use only.

parent

object

scorers

object[]

List of Galileo scorers to enable.

registered_scorers

object[]

List of registered scorers to enable.

generated_scorers

object[]

List of generated scorers to enable.

project_id

string | null

Evaluate Project ID to which the run should be associated.

project_name

string | null

Evaluate Project name to which the run should be associated. If the project does not exist, it will be created.

run_name

string | null

Name of the run. If no name is provided, a timestamp-based name will be generated.

Response

200

application/json

Successful Response

message

string

required

project_id

string

required

project_name

string

required

run_id

string

required

run_name

string

required

workflows_count

integer

required

records_count

integer

required

Was this page helpful?

Suggest edits Raise issue

Get Token Get Evaluate Run Results

curl --request POST \
  --url https://api.acme.rungalileo.io/v1/evaluate/runs \
  --header 'Content-Type: application/json' \
  --header 'Galileo-API-Key: <api-key>' \
  --data '{
  "project_name": "my-evaluate-project",
  "run_name": "my-evaluate-run",
  "scorers": [
    {
      "name": "correctness"
    },
    {
      "name": "output_pii"
    }
  ],
  "workflows": [
    {
      "created_at_ns": 1743174002288939000,
      "duration_ns": 0,
      "input": "who is a smart LLM?",
      "metadata": {},
      "name": "llm",
      "output": "I am!",
      "type": "llm"
    }
  ]
}'

{
  "message": "<string>",
  "project_id": "<string>",
  "project_name": "<string>",
  "run_id": "<string>",
  "run_name": "<string>",
  "workflows_count": 123,
  "records_count": 123
}

API Reference

health

auth

evaluate

observe

protect

Webhooks

Create a new Evaluate Run

WorkflowStep

Examples

LLM Step

Retriever Step

Multi-Step

Authorizations

Body

Response

API Reference

health

auth

evaluate

observe

protect

Webhooks

​WorkflowStep

​Examples

​LLM Step

​Retriever Step

​Multi-Step

Authorizations

Body

Response

WorkflowStep

Examples

LLM Step

Retriever Step

Multi-Step