# Apply Bulk Annotation Source: https://docs.galileo.ai/api-reference/annotation/apply-bulk-annotation https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/annotation/ratings # Create Annotation Rating Source: https://docs.galileo.ai/api-reference/annotation/create-annotation-rating https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/annotation/templates/{template_id}/traces/{trace_id}/rating # Create Annotation Template Source: https://docs.galileo.ai/api-reference/annotation/create-annotation-template https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/annotation/templates # Create Log Record Annotation Rating Source: https://docs.galileo.ai/api-reference/annotation/create-log-record-annotation-rating https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/annotation/templates/{template_id}/records/{record_id}/rating # Delete Annotation Rating Source: https://docs.galileo.ai/api-reference/annotation/delete-annotation-rating https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/annotation/templates/{template_id}/traces/{trace_id}/rating # Delete Annotation Template Source: https://docs.galileo.ai/api-reference/annotation/delete-annotation-template https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/annotation/templates/{template_id} # Delete Log Record Annotation Rating Source: https://docs.galileo.ai/api-reference/annotation/delete-log-record-annotation-rating https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/annotation/templates/{template_id}/records/{record_id}/rating # Get Annotation Rating Source: https://docs.galileo.ai/api-reference/annotation/get-annotation-rating https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/annotation/templates/{template_id}/traces/{trace_id}/rating # Get Annotation Template Source: https://docs.galileo.ai/api-reference/annotation/get-annotation-template https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/annotation/templates/{template_id} # Get Log Record Annotation Rating Source: https://docs.galileo.ai/api-reference/annotation/get-log-record-annotation-rating https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/annotation/templates/{template_id}/records/{record_id}/rating # List Annotation Templates Source: https://docs.galileo.ai/api-reference/annotation/list-annotation-templates https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/annotation/templates # Reorder Annotation Templates Source: https://docs.galileo.ai/api-reference/annotation/reorder-annotation-templates https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/annotation/templates/reorder # Update Annotation Template Source: https://docs.galileo.ai/api-reference/annotation/update-annotation-template https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/annotation/templates/{template_id} # Get Annotation Queue Source: https://docs.galileo.ai/api-reference/annotation_queue/get-annotation-queue https://api.galileo.ai/public/v2/openapi.json get /v2/annotation_queues/{queue_id} Get an annotation queue by ID with templates and counts. # Get Queue Templates Source: https://docs.galileo.ai/api-reference/annotation_queue/get-queue-templates https://api.galileo.ai/public/v2/openapi.json get /v2/annotation_queues/{queue_id}/templates Get all templates for an annotation queue. Templates are returned ordered by position (ascending). # List Annotation Queue Users Source: https://docs.galileo.ai/api-reference/annotation_queue/list-annotation-queue-users https://api.galileo.ai/public/v2/openapi.json get /v2/annotation_queues/{queue_id}/users List users who have access to an annotation queue with pagination. # Query Annotation Queues Source: https://docs.galileo.ai/api-reference/annotation_queue/query-annotation-queues https://api.galileo.ai/public/v2/openapi.json post /v2/annotation_queues/query Query annotation queues in the user's organization with filtering and sorting. Response includes num_templates for each queue to support copy selection UI. # Export Annotation Queue Records Source: https://docs.galileo.ai/api-reference/annotation_queue_records/export-annotation-queue-records https://api.galileo.ai/public/v2/openapi.json post /v2/annotation_queues/{queue_id}/records/export Export selected records from an annotation queue. The request must specify either a list of record IDs or a filter tree to select queue records. Permission checks: - User must have READ permission on the annotation queue # Export Annotation Queue Records Url Source: https://docs.galileo.ai/api-reference/annotation_queue_records/export-annotation-queue-records-url https://api.galileo.ai/public/v2/openapi.json post /v2/annotation_queues/{queue_id}/records/export/url Export selected records from an annotation queue and return a presigned download URL. The request must specify either a list of record IDs or a filter tree to select queue records. Permission checks: - User must have READ permission on the annotation queue # Create Api Key Source: https://docs.galileo.ai/api-reference/api_keys/create-api-key https://api.galileo.ai/public/v2/openapi.json post /v2/users/api_keys # Delete Api Key Source: https://docs.galileo.ai/api-reference/api_keys/delete-api-key https://api.galileo.ai/public/v2/openapi.json delete /v2/users/api_keys/{api_key_id} # Get Api Keys Source: https://docs.galileo.ai/api-reference/api_keys/get-api-keys https://api.galileo.ai/public/v2/openapi.json get /v2/users/{user_id}/api_keys # Get Token Source: https://docs.galileo.ai/api-reference/auth/get-token https://api.galileo.ai/public/v2/openapi.json get /v2/token # Login Api Key Source: https://docs.galileo.ai/api-reference/auth/login-api-key https://api.galileo.ai/public/v2/openapi.json post /v2/login/api_key # Login Email Source: https://docs.galileo.ai/api-reference/auth/login-email https://api.galileo.ai/public/v2/openapi.json post /v2/login # Login Social Source: https://docs.galileo.ai/api-reference/auth/login-social https://api.galileo.ai/public/v2/openapi.json post /v2/login/social # Refresh Token Source: https://docs.galileo.ai/api-reference/auth/refresh-token https://api.galileo.ai/public/v2/openapi.json post /v2/refresh_token # Saml Acs Source: https://docs.galileo.ai/api-reference/auth/saml-acs https://api.galileo.ai/public/v2/openapi.json post /v2/saml/acs # Saml Login Source: https://docs.galileo.ai/api-reference/auth/saml-login https://api.galileo.ai/public/v2/openapi.json get /v2/saml/login # Saml Metadata Source: https://docs.galileo.ai/api-reference/auth/saml-metadata https://api.galileo.ai/public/v2/openapi.json get /v2/saml/metadata # Verify Email Source: https://docs.galileo.ai/api-reference/auth/verify-email https://api.galileo.ai/public/v2/openapi.json post /v2/verify_email # Autogen Llm Scorer Source: https://docs.galileo.ai/api-reference/data/autogen-llm-scorer https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/llm/autogen Autogenerate an LLM scorer configuration. Returns a Celery task ID that can be used to poll for the autogeneration results. # Compute Health Score Endpoint Source: https://docs.galileo.ai/api-reference/data/compute-health-score-endpoint https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/metrics-testing/{run_id}/health-score Compute the health score metric for a metrics testing run. # Create Source: https://docs.galileo.ai/api-reference/data/create https://api.galileo.ai/public/v2/openapi.json post /v2/scorers # Create Code Scorer Version Source: https://docs.galileo.ai/api-reference/data/create-code-scorer-version https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/{scorer_id}/version/code # Create Llm Scorer Version Source: https://docs.galileo.ai/api-reference/data/create-llm-scorer-version https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/{scorer_id}/version/llm # Create Luna Scorer Version Source: https://docs.galileo.ai/api-reference/data/create-luna-scorer-version https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/{scorer_id}/version/luna Create a new custom Luna scorer version for the given scorer. Args: create_luna_scorer_version_request: LoRA/fine-tuning parameters for the new version. scorer: The Luna scorer to create a new version for. ctx: Async request context with the authenticated user and read session. Returns: The newly created scorer version. # Create Preset Scorer Version Source: https://docs.galileo.ai/api-reference/data/create-preset-scorer-version https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/{scorer_id}/version/preset Create a preset scorer version. # Delete Scorer Source: https://docs.galileo.ai/api-reference/data/delete-scorer https://api.galileo.ai/public/v2/openapi.json delete /v2/scorers/{scorer_id} # Get Scorer Source: https://docs.galileo.ai/api-reference/data/get-scorer https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/{scorer_id} # Get Scorer Health Scores Source: https://docs.galileo.ai/api-reference/data/get-scorer-health-scores https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/{scorer_id}/health-scores Return all persisted health scores for a scorer against a dataset, ordered by version ASC. scores[0] is the baseline (first recorded), scores[-1] is the latest. # Get Scorer Version Code Source: https://docs.galileo.ai/api-reference/data/get-scorer-version-code https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/{scorer_id}/version/code # Get Scorer Version Or Latest Source: https://docs.galileo.ai/api-reference/data/get-scorer-version-or-latest https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/{scorer_id}/version # Get Validate Code Scorer Task Result Source: https://docs.galileo.ai/api-reference/data/get-validate-code-scorer-task-result https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/code/validate/{task_id} Poll for a code-scorer validation task result (returns status/result). The validation job creates an entry in `registered_scorer_task_results` (pending) and the runner will PATCH the internal task-results endpoint when it finishes. This GET allows clients to poll the current task result. # List All Versions For Scorer Source: https://docs.galileo.ai/api-reference/data/list-all-versions-for-scorer https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/{scorer_id}/versions # List Projects For Scorer Route Source: https://docs.galileo.ai/api-reference/data/list-projects-for-scorer-route https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/{scorer_id}/projects List all projects associated with a specific scorer. # List Projects For Scorer Version Route Source: https://docs.galileo.ai/api-reference/data/list-projects-for-scorer-version-route https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/versions/{scorer_version_id}/projects List all projects associated with a specific scorer version. # List Scorers With Filters Source: https://docs.galileo.ai/api-reference/data/list-scorers-with-filters https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/list # List Tags Source: https://docs.galileo.ai/api-reference/data/list-tags https://api.galileo.ai/public/v2/openapi.json get /v2/scorers/tags # Manual Llm Validate Source: https://docs.galileo.ai/api-reference/data/manual-llm-validate https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/llm/validate Validate an LLM scorer manually, with query/response passed inline (no file uploads). Args: request: Raw request; body is parsed into a GeneratedScorerValidationRequest. ctx: Async request context with the authenticated user and read session. Returns: A pending task result the caller can poll for validation results. # Manual Llm Validate Multipart Source: https://docs.galileo.ai/api-reference/data/manual-llm-validate-multipart https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/llm/validate/multipart Validate an LLM scorer manually, with optional query/response file uploads. Args: body: JSON-encoded GeneratedScorerValidationRequest. query_files: Optional files attached to the query side of the validation. response_files: Optional files attached to the response side of the validation. ctx: Async request context with the authenticated user and read session. Returns: A pending task result the caller can poll for validation results. # Restore Scorer Version Source: https://docs.galileo.ai/api-reference/data/restore-scorer-version https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/{scorer_id}/versions/{version_number}/restore List all scorers. # Set Scorer Scope Source: https://docs.galileo.ai/api-reference/data/set-scorer-scope https://api.galileo.ai/public/v2/openapi.json put /v2/scorers/{scorer_id}/scope Full-replace a scorer's access scope (Share / manage visibility). metrics_rbac only. # Update Source: https://docs.galileo.ai/api-reference/data/update https://api.galileo.ai/public/v2/openapi.json patch /v2/scorers/{scorer_id} # Validate Code Scorer Source: https://docs.galileo.ai/api-reference/data/validate-code-scorer https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/code/validate Validate a code scorer with optional simple input/output test. # Validate Code Scorer Dataset Source: https://docs.galileo.ai/api-reference/data/validate-code-scorer-dataset https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/code/validate/dataset Validate a code scorer against dataset rows. # Validate Code Scorer Log Record Source: https://docs.galileo.ai/api-reference/data/validate-code-scorer-log-record https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/code/validate/log_record Validate a code scorer using actual log records. # Validate Llm Scorer Dataset Source: https://docs.galileo.ai/api-reference/data/validate-llm-scorer-dataset https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/llm/validate/dataset # Validate Llm Scorer Log Record Source: https://docs.galileo.ai/api-reference/data/validate-llm-scorer-log-record https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/llm/validate/log_record # Write Scorer Version Health Score Source: https://docs.galileo.ai/api-reference/data/write-scorer-version-health-score https://api.galileo.ai/public/v2/openapi.json post /v2/scorers/{scorer_id}/versions/{version_number}/health-scores Persist the health score for a scorer version against a dataset. Called by the UI after saving a metric version, passing the score from the last compute. # Bulk Delete Datasets Source: https://docs.galileo.ai/api-reference/datasets/bulk-delete-datasets https://api.galileo.ai/public/v2/openapi.json delete /v2/datasets/bulk_delete Delete multiple datasets in bulk. This endpoint allows efficient deletion of multiple datasets at once. It validates permissions for each dataset in the service and provides detailed feedback about successful and failed deletions for each dataset. Parameters ---------- delete_request : BulkDeleteDatasetsRequest Request containing list of dataset IDs to delete (max 100) ctx : Context Request context including authentication information Returns ------- BulkDeleteDatasetsResponse Details about the bulk deletion operation including: - Number of successfully deleted datasets - List of failed deletions with reasons - Summary message # Create Dataset Source: https://docs.galileo.ai/api-reference/datasets/create-dataset https://api.galileo.ai/public/v2/openapi.json post /v2/datasets Creates a standalone dataset. # Create Group Dataset Collaborators Source: https://docs.galileo.ai/api-reference/datasets/create-group-dataset-collaborators https://api.galileo.ai/public/v2/openapi.json post /v2/datasets/{dataset_id}/groups Share a dataset with groups. # Create User Dataset Collaborators Source: https://docs.galileo.ai/api-reference/datasets/create-user-dataset-collaborators https://api.galileo.ai/public/v2/openapi.json post /v2/datasets/{dataset_id}/users # Delete Dataset Source: https://docs.galileo.ai/api-reference/datasets/delete-dataset https://api.galileo.ai/public/v2/openapi.json delete /v2/datasets/{dataset_id} # Delete Group Dataset Collaborator Source: https://docs.galileo.ai/api-reference/datasets/delete-group-dataset-collaborator https://api.galileo.ai/public/v2/openapi.json delete /v2/datasets/{dataset_id}/groups/{group_id} Remove a group's access to a dataset. # Delete User Dataset Collaborator Source: https://docs.galileo.ai/api-reference/datasets/delete-user-dataset-collaborator https://api.galileo.ai/public/v2/openapi.json delete /v2/datasets/{dataset_id}/users/{user_id} Remove a user's access to a dataset. # Download Dataset Source: https://docs.galileo.ai/api-reference/datasets/download-dataset https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/{dataset_id}/download # Extend Dataset Content Source: https://docs.galileo.ai/api-reference/datasets/extend-dataset-content https://api.galileo.ai/public/v2/openapi.json post /v2/datasets/extend Extends the dataset content # Get Dataset Source: https://docs.galileo.ai/api-reference/datasets/get-dataset https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/{dataset_id} # Get Dataset Content Source: https://docs.galileo.ai/api-reference/datasets/get-dataset-content https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/{dataset_id}/content # Get Dataset Synthetic Extend Status Source: https://docs.galileo.ai/api-reference/datasets/get-dataset-synthetic-extend-status https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/extend/{dataset_id} # Get Dataset Version Content Source: https://docs.galileo.ai/api-reference/datasets/get-dataset-version-content https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/{dataset_id}/versions/{version_index}/content # List Dataset Projects Source: https://docs.galileo.ai/api-reference/datasets/list-dataset-projects https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/{dataset_id}/projects # List Datasets Source: https://docs.galileo.ai/api-reference/datasets/list-datasets https://api.galileo.ai/public/v2/openapi.json get /v2/datasets # List Group Dataset Collaborators Source: https://docs.galileo.ai/api-reference/datasets/list-group-dataset-collaborators https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/{dataset_id}/groups List the groups with which the dataset has been shared. # List User Dataset Collaborators Source: https://docs.galileo.ai/api-reference/datasets/list-user-dataset-collaborators https://api.galileo.ai/public/v2/openapi.json get /v2/datasets/{dataset_id}/users List the users with which the dataset has been shared. # Preview Dataset Source: https://docs.galileo.ai/api-reference/datasets/preview-dataset https://api.galileo.ai/public/v2/openapi.json post /v2/datasets/{dataset_id}/preview # Query Dataset Content Source: https://docs.galileo.ai/api-reference/datasets/query-dataset-content https://api.galileo.ai/public/v2/openapi.json post /v2/datasets/{dataset_id}/content/query # Query Dataset Versions Source: https://docs.galileo.ai/api-reference/datasets/query-dataset-versions https://api.galileo.ai/public/v2/openapi.json post /v2/datasets/{dataset_id}/versions/query # Query Datasets Source: https://docs.galileo.ai/api-reference/datasets/query-datasets https://api.galileo.ai/public/v2/openapi.json post /v2/datasets/query # Update Dataset Source: https://docs.galileo.ai/api-reference/datasets/update-dataset https://api.galileo.ai/public/v2/openapi.json patch /v2/datasets/{dataset_id} # Update Dataset Content Source: https://docs.galileo.ai/api-reference/datasets/update-dataset-content https://api.galileo.ai/public/v2/openapi.json patch /v2/datasets/{dataset_id}/content Update the content of a dataset. The `index` and `column_name` fields are treated as keys tied to a specific version of the dataset. As such, these values are considered immutable identifiers for the dataset's structure. Edits are applied sequentially in list order, and each edit sees the table state left by the previous one. For example, after a `rename_column` edit renames `col_a` to `col_b`, any subsequent `update_row` in the same request must reference the column as `col_b`, not `col_a`. The `If-Match` header is used to ensure that updates are only applied if the client's version of the dataset matches the server's version. This prevents conflicts from simultaneous updates. The `ETag` header in the response provides the new version identifier after a successful update. # Update Dataset Version Source: https://docs.galileo.ai/api-reference/datasets/update-dataset-version https://api.galileo.ai/public/v2/openapi.json patch /v2/datasets/{dataset_id}/versions/{version_index} # Update Group Dataset Collaborator Source: https://docs.galileo.ai/api-reference/datasets/update-group-dataset-collaborator https://api.galileo.ai/public/v2/openapi.json patch /v2/datasets/{dataset_id}/groups/{group_id} Update the sharing permissions of a group on a dataset. # Update User Dataset Collaborator Source: https://docs.galileo.ai/api-reference/datasets/update-user-dataset-collaborator https://api.galileo.ai/public/v2/openapi.json patch /v2/datasets/{dataset_id}/users/{user_id} Update the sharing permissions of a user on a dataset. # Upsert Dataset Content Source: https://docs.galileo.ai/api-reference/datasets/upsert-dataset-content https://api.galileo.ai/public/v2/openapi.json put /v2/datasets/{dataset_id}/content Rollback the content of a dataset to a previous version. # Create Experiment Source: https://docs.galileo.ai/api-reference/experiment/create-experiment https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/experiments Create a new experiment for a project. # Delete Experiment Source: https://docs.galileo.ai/api-reference/experiment/delete-experiment https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/experiments/{experiment_id} Delete a specific experiment. # Experiments Available Columns Source: https://docs.galileo.ai/api-reference/experiment/experiments-available-columns https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/experiments/available_columns Procures the column information for experiments. # Get Experiment Source: https://docs.galileo.ai/api-reference/experiment/get-experiment https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/experiments/{experiment_id} Retrieve a specific experiment. # Get Experiment Metrics Source: https://docs.galileo.ai/api-reference/experiment/get-experiment-metrics https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/experiments/{experiment_id}/metrics Retrieve metrics for a specific experiment. # List Experiments Source: https://docs.galileo.ai/api-reference/experiment/list-experiments https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/experiments Retrieve all experiments for a project. # List Experiments Paginated Source: https://docs.galileo.ai/api-reference/experiment/list-experiments-paginated https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/experiments/paginated Retrieve all experiments for a project with pagination. # Search Experiments Source: https://docs.galileo.ai/api-reference/experiment/search-experiments https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/experiments/search Search experiments for a project. # Update Experiment Source: https://docs.galileo.ai/api-reference/experiment/update-experiment https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/experiments/{experiment_id} Update a specific experiment. # Apply Bulk Feedback V2 Source: https://docs.galileo.ai/api-reference/feedback/apply-bulk-feedback-v2 https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/feedback/ratings # Create Feedback Rating V2 Source: https://docs.galileo.ai/api-reference/feedback/create-feedback-rating-v2 https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/feedback/templates/{template_id}/traces/{trace_id}/rating # Create Feedback Template V2 Source: https://docs.galileo.ai/api-reference/feedback/create-feedback-template-v2 https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/feedback/templates # Delete Feedback Rating V2 Source: https://docs.galileo.ai/api-reference/feedback/delete-feedback-rating-v2 https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/feedback/templates/{template_id}/traces/{trace_id}/rating # Delete Feedback Template Source: https://docs.galileo.ai/api-reference/feedback/delete-feedback-template https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/feedback/templates/{template_id} # Get Feedback Rating V2 Source: https://docs.galileo.ai/api-reference/feedback/get-feedback-rating-v2 https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/feedback/templates/{template_id}/traces/{trace_id}/rating # Get Feedback Template V2 Source: https://docs.galileo.ai/api-reference/feedback/get-feedback-template-v2 https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/feedback/templates/{template_id} # List Feedback Templates V2 Source: https://docs.galileo.ai/api-reference/feedback/list-feedback-templates-v2 https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/feedback/templates # Reorder Feedback Templates Source: https://docs.galileo.ai/api-reference/feedback/reorder-feedback-templates https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/feedback/templates/reorder # Update Feedback Template Source: https://docs.galileo.ai/api-reference/feedback/update-feedback-template https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/feedback/templates/{template_id} # Add User To Group Source: https://docs.galileo.ai/api-reference/groups/add-user-to-group https://api.galileo.ai/public/v2/openapi.json post /v2/groups/{group_id}/members # Create Group Source: https://docs.galileo.ai/api-reference/groups/create-group https://api.galileo.ai/public/v2/openapi.json post /v2/groups # Delete Group Source: https://docs.galileo.ai/api-reference/groups/delete-group https://api.galileo.ai/public/v2/openapi.json delete /v2/groups/{group_id} # Delete Group Member Source: https://docs.galileo.ai/api-reference/groups/delete-group-member https://api.galileo.ai/public/v2/openapi.json delete /v2/groups/{group_id}/members/{user_id} # Get Group Source: https://docs.galileo.ai/api-reference/groups/get-group https://api.galileo.ai/public/v2/openapi.json get /v2/groups/{group_id} # Get Group Roles Source: https://docs.galileo.ai/api-reference/groups/get-group-roles https://api.galileo.ai/public/v2/openapi.json get /v2/group_roles # List Current User Groups Source: https://docs.galileo.ai/api-reference/groups/list-current-user-groups https://api.galileo.ai/public/v2/openapi.json get /v2/current_user/groups # List Group Members Source: https://docs.galileo.ai/api-reference/groups/list-group-members https://api.galileo.ai/public/v2/openapi.json get /v2/groups/{group_id}/members # List Groups Source: https://docs.galileo.ai/api-reference/groups/list-groups https://api.galileo.ai/public/v2/openapi.json get /v2/groups # Update Group Source: https://docs.galileo.ai/api-reference/groups/update-group https://api.galileo.ai/public/v2/openapi.json patch /v2/groups/{group_id} # Update Group Member Source: https://docs.galileo.ai/api-reference/groups/update-group-member https://api.galileo.ai/public/v2/openapi.json patch /v2/groups/{group_id}/members/{user_id} # Healthcheck Source: https://docs.galileo.ai/api-reference/health/healthcheck https://api.galileo.ai/public/v2/openapi.json get /v2/healthcheck # Create Group Integration Collaborators Source: https://docs.galileo.ai/api-reference/integrations/create-group-integration-collaborators https://api.galileo.ai/public/v2/openapi.json post /v2/integrations/{integration_id}/groups Share an integration with groups. # Create or update a named custom integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-a-named-custom-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/custom/{name} # Create or update Anthropic integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-anthropic-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/anthropic Create or update an Anthropic integration for this user from Galileo. # Create or update AWS Bedrock integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-aws-bedrock-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/aws_bedrock Create or update an AWS integration for this user from Galileo. # Create or update AWS SageMaker integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-aws-sagemaker-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/aws_sagemaker Create or update an AWS integration for this user from Galileo. # Create or update Azure integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-azure-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/azure Create or update an Azure integration for this user from Galileo. # Create or update custom integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-custom-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/custom # Create or update Databricks integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-databricks-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/databricks Create or update a databricks integration for this user from Galileo. # Create or update Databricks integration (legacy) Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-databricks-integration-legacy https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/databricks/unity-catalog/sql Create or update a databricks integration for this user from Galileo. # Create Or Update Integration Selection Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-integration-selection https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/{integration_id}/select Create or update an integration selection for this user from Galileo. # Create or update Mistral integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-mistral-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/mistral Create or update an Mistral integration for this user from Galileo. # Create or update NVIDIA integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-nvidia-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/nvidia Create or update an NVIDIA integration for this user from Galileo. # Create or update OpenAI integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-openai-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/openai Create or update an OpenAI integration for this user from Galileo. # Create or update Vegas Gateway integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-vegas-gateway-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/vegas_gateway Create or update a Vegas Gateway integration for this user from Galileo. # Create or update Vertex AI integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-vertex-ai-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/vertex_ai Create or update a Google Vertex AI integration for a user. # Create or update Writer integration Source: https://docs.galileo.ai/api-reference/integrations/create-or-update-writer-integration https://api.galileo.ai/public/v2/openapi.json put /v2/integrations/writer Create or update a Writer integration for a user. # Create User Integration Collaborators Source: https://docs.galileo.ai/api-reference/integrations/create-user-integration-collaborators https://api.galileo.ai/public/v2/openapi.json post /v2/integrations/{integration_id}/users # Delete Group Integration Collaborator Source: https://docs.galileo.ai/api-reference/integrations/delete-group-integration-collaborator https://api.galileo.ai/public/v2/openapi.json delete /v2/integrations/{integration_id}/groups/{group_id} Remove a group's access to an integration. # Delete User Integration Collaborator Source: https://docs.galileo.ai/api-reference/integrations/delete-user-integration-collaborator https://api.galileo.ai/public/v2/openapi.json delete /v2/integrations/{integration_id}/users/{user_id} Remove a user's access to an integration. # Get Databases For Cluster Source: https://docs.galileo.ai/api-reference/integrations/get-databases-for-cluster https://api.galileo.ai/public/v2/openapi.json get /v2/integrations/databricks/databases # Get Databricks Catalogs Source: https://docs.galileo.ai/api-reference/integrations/get-databricks-catalogs https://api.galileo.ai/public/v2/openapi.json get /v2/integrations/databricks/catalogs # Get Integration Source: https://docs.galileo.ai/api-reference/integrations/get-integration https://api.galileo.ai/public/v2/openapi.json get /v2/integrations/{name} Gets the integration data formatted for the specified integration. # Get Integration Status Source: https://docs.galileo.ai/api-reference/integrations/get-integration-status https://api.galileo.ai/public/v2/openapi.json get /v2/integrations/{name}/status Checks if the integration status is active or not. # List Available Integrations Source: https://docs.galileo.ai/api-reference/integrations/list-available-integrations https://api.galileo.ai/public/v2/openapi.json get /v2/integrations/available List all of the available integrations to be created in Galileo. # List Group Integration Collaborators Source: https://docs.galileo.ai/api-reference/integrations/list-group-integration-collaborators https://api.galileo.ai/public/v2/openapi.json get /v2/integrations/{integration_id}/groups List the groups with which the integration has been shared. # List User Integration Collaborators Source: https://docs.galileo.ai/api-reference/integrations/list-user-integration-collaborators https://api.galileo.ai/public/v2/openapi.json get /v2/integrations/{integration_id}/users List the users with which the integration has been shared. # Update Group Integration Collaborator Source: https://docs.galileo.ai/api-reference/integrations/update-group-integration-collaborator https://api.galileo.ai/public/v2/openapi.json patch /v2/integrations/{integration_id}/groups/{group_id} Update the sharing permissions of a group on an integration. # Update User Integration Collaborator Source: https://docs.galileo.ai/api-reference/integrations/update-user-integration-collaborator https://api.galileo.ai/public/v2/openapi.json patch /v2/integrations/{integration_id}/users/{user_id} Update the sharing permissions of a user on an integration. # Create Log Stream Source: https://docs.galileo.ai/api-reference/log_stream/create-log-stream https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/log_streams Create a new log stream for a project. # Delete Log Stream Source: https://docs.galileo.ai/api-reference/log_stream/delete-log-stream https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/log_streams/{log_stream_id} Delete a specific log stream. # Get Log Stream Source: https://docs.galileo.ai/api-reference/log_stream/get-log-stream https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/log_streams/{log_stream_id} Retrieve a specific log stream. # Get Metric Settings Source: https://docs.galileo.ai/api-reference/log_stream/get-metric-settings https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/log_streams/{log_stream_id}/metric_settings # List Log Streams Source: https://docs.galileo.ai/api-reference/log_stream/list-log-streams https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/log_streams Retrieve all log streams for a project. DEPRECATED in favor of `list_log_streams_paginated`. # List Log Streams Paginated Source: https://docs.galileo.ai/api-reference/log_stream/list-log-streams-paginated https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/log_streams/paginated Retrieve all log streams for a project paginated. # Search Log Streams Source: https://docs.galileo.ai/api-reference/log_stream/search-log-streams https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/log_streams/search Search log streams for a project. # Update Log Stream Source: https://docs.galileo.ai/api-reference/log_stream/update-log-stream https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/log_streams/{log_stream_id} Update a specific log stream. # Update Metric Settings Source: https://docs.galileo.ai/api-reference/log_stream/update-metric-settings https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/log_streams/{log_stream_id}/metric_settings # Get Logstream Insights Token Usages Source: https://docs.galileo.ai/api-reference/logstream-insights/get-logstream-insights-token-usages https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/log_streams/{log_stream_id}/logstream_insights/token_usage Get token usage records for a logstream insight. # Delete By Metadata Source: https://docs.galileo.ai/api-reference/organization-jobs/delete-by-metadata https://api.galileo.ai/public/v2/openapi.json post /v2/org-jobs/delete-by-metadata Delete traces/sessions across all projects in the organization by metadata filters. This endpoint allows organization administrators to delete traces or sessions that match specific metadata key-value pairs across all projects in their organization. # Get Org Job Status Source: https://docs.galileo.ai/api-reference/organization-jobs/get-org-job-status https://api.galileo.ai/public/v2/openapi.json get /v2/org-jobs/{job_id} Get the status of an organization-level job. This endpoint retrieves the status of jobs that operate at the organization level, such as org-wide data deletion jobs. **Authorization**: The job's organization_id must match the user's organization. # Create Group Project Collaborators Source: https://docs.galileo.ai/api-reference/projects/create-group-project-collaborators https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/groups Share a project with groups. # Create Project Source: https://docs.galileo.ai/api-reference/projects/create-project https://api.galileo.ai/public/v2/openapi.json post /v2/projects Create a new project. # Create User Project Collaborators Source: https://docs.galileo.ai/api-reference/projects/create-user-project-collaborators https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/users Share a project with users. # Delete Group Project Collaborator Source: https://docs.galileo.ai/api-reference/projects/delete-group-project-collaborator https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/groups/{group_id} Remove a group's access to a project. # Delete Project Source: https://docs.galileo.ai/api-reference/projects/delete-project https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id} Deletes a project and all associated runs and objects. Any user with project access can delete a project. Note that `get_project_by_id` calls `user_can_access_project`. # Delete User Project Collaborator Source: https://docs.galileo.ai/api-reference/projects/delete-user-project-collaborator https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/users/{user_id} Remove a user's access to a project. # Get Collaborator Roles Source: https://docs.galileo.ai/api-reference/projects/get-collaborator-roles https://api.galileo.ai/public/v2/openapi.json get /v2/collaborator_roles # Get Project Source: https://docs.galileo.ai/api-reference/projects/get-project https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id} # Get Projects V2 Source: https://docs.galileo.ai/api-reference/projects/get-projects-v2 https://api.galileo.ai/public/v2/openapi.json post /v2/projects/paginated Gets projects optimized for V2 with pagination and server-side run counts. # List Group Project Collaborators Source: https://docs.galileo.ai/api-reference/projects/list-group-project-collaborators https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/groups List the groups with which the project has been shared. # List User Project Collaborators Source: https://docs.galileo.ai/api-reference/projects/list-user-project-collaborators https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/users List the users with which the project has been shared. # Update Group Project Collaborator Source: https://docs.galileo.ai/api-reference/projects/update-group-project-collaborator https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/groups/{group_id} Update the sharing permissions of a group on a project. # Update Project Source: https://docs.galileo.ai/api-reference/projects/update-project https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id} # Update User Project Collaborator Source: https://docs.galileo.ai/api-reference/projects/update-user-project-collaborator https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/users/{user_id} Update the sharing permissions of a user on a project. # Invoke Source: https://docs.galileo.ai/api-reference/protect/invoke https://api.galileo.ai/public/v2/openapi.json post /v2/protect/invoke # Get Settings Source: https://docs.galileo.ai/api-reference/run_insights_settings/get-settings https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/runs/{run_id}/insights-settings # Upsert Insights Config Source: https://docs.galileo.ai/api-reference/run_insights_settings/upsert-insights-config https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/runs/{run_id}/insights-settings # Create Or Verify User Source: https://docs.galileo.ai/api-reference/system_users/create-or-verify-user https://api.galileo.ai/public/v2/openapi.json post /v2/system_users Create a new system user with an email and password. If no admin exists (first user), the user will be created as an admin. Otherwise: - User record was already created when the admin invited the user - We should verify the user's email # Create Or Verify User Social Source: https://docs.galileo.ai/api-reference/system_users/create-or-verify-user-social https://api.galileo.ai/public/v2/openapi.json post /v2/system_users/social Create a user using a social login provider. All social users are created with `email_is_verified=True`, don't need to be invited and are by default read-only (unless they are the first user, in which case they are set to admin). # Count Sessions Source: https://docs.galileo.ai/api-reference/trace/count-sessions https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/sessions/count # Count Spans Source: https://docs.galileo.ai/api-reference/trace/count-spans https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/spans/count # Count Traces Source: https://docs.galileo.ai/api-reference/trace/count-traces https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/traces/count This endpoint may return a slightly inaccurate count due to the way records are filtered before deduplication. # Create Session Source: https://docs.galileo.ai/api-reference/trace/create-session https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/sessions # Delete Sessions Source: https://docs.galileo.ai/api-reference/trace/delete-sessions https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/sessions/delete Delete all session records that match the provided filters. # Delete Spans Source: https://docs.galileo.ai/api-reference/trace/delete-spans https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/spans/delete Delete all span records that match the provided filters. # Delete Traces Source: https://docs.galileo.ai/api-reference/trace/delete-traces https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/traces/delete Delete all trace records that match the provided filters. # Export Records Source: https://docs.galileo.ai/api-reference/trace/export-records https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/export_records # Export Records Url Source: https://docs.galileo.ai/api-reference/trace/export-records-url https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/export_records/url # Get Aggregated Trace View Source: https://docs.galileo.ai/api-reference/trace/get-aggregated-trace-view https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/traces/aggregated # Get Session Source: https://docs.galileo.ai/api-reference/trace/get-session https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/sessions/{session_id} # Get Span Source: https://docs.galileo.ai/api-reference/trace/get-span https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/spans/{span_id} # Get Trace Source: https://docs.galileo.ai/api-reference/trace/get-trace https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/traces/{trace_id} # Log Spans Source: https://docs.galileo.ai/api-reference/trace/log-spans https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/spans # Log Traces Source: https://docs.galileo.ai/api-reference/trace/log-traces https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/traces # Metrics Testing Available Columns Source: https://docs.galileo.ai/api-reference/trace/metrics-testing-available-columns https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/metrics-testing/available_columns # Query Custom Metrics Source: https://docs.galileo.ai/api-reference/trace/query-custom-metrics https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/metrics/custom_search # Query Metrics Source: https://docs.galileo.ai/api-reference/trace/query-metrics https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/metrics/search # Query Metrics V2 Source: https://docs.galileo.ai/api-reference/trace/query-metrics-v2 https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/metrics/search/v2 Same as /metrics/search but returns metrics with node-type counts: trace (requests_count), session_count, and span_count in aggregate_metrics and in each bucket, similar to /metrics/custom_search. # Query Partial Sessions Source: https://docs.galileo.ai/api-reference/trace/query-partial-sessions https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/sessions/partial_search # Query Partial Spans Source: https://docs.galileo.ai/api-reference/trace/query-partial-spans https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/spans/partial_search # Query Partial Traces Source: https://docs.galileo.ai/api-reference/trace/query-partial-traces https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/traces/partial_search # Query Sessions Source: https://docs.galileo.ai/api-reference/trace/query-sessions https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/sessions/search # Query Spans Source: https://docs.galileo.ai/api-reference/trace/query-spans https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/spans/search # Query Traces Source: https://docs.galileo.ai/api-reference/trace/query-traces https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/traces/search # Recompute Metrics Source: https://docs.galileo.ai/api-reference/trace/recompute-metrics https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/recompute-metrics # Sessions Available Columns Source: https://docs.galileo.ai/api-reference/trace/sessions-available-columns https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/sessions/available_columns # Spans Available Columns Source: https://docs.galileo.ai/api-reference/trace/spans-available-columns https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/spans/available_columns # Traces Available Columns Source: https://docs.galileo.ai/api-reference/trace/traces-available-columns https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/traces/available_columns # Update Span Source: https://docs.galileo.ai/api-reference/trace/update-span https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/spans/{span_id} Update a span with the given ID. # Update Trace Source: https://docs.galileo.ai/api-reference/trace/update-trace https://api.galileo.ai/public/v2/openapi.json patch /v2/projects/{project_id}/traces/{trace_id} Update a trace with the given ID. # Create Section Source: https://docs.galileo.ai/api-reference/trends_dashboard/create-section https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/sections # Create Widget Source: https://docs.galileo.ai/api-reference/trends_dashboard/create-widget https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/widgets # Delete Dashboard Source: https://docs.galileo.ai/api-reference/trends_dashboard/delete-dashboard https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/dashboards/{trends_dashboard_id} # Delete Section Source: https://docs.galileo.ai/api-reference/trends_dashboard/delete-section https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/sections/{section_id} Delete section. If ungroup=True, keep widgets by moving them to dashboard top-level (clear section_id). # Delete Widget Source: https://docs.galileo.ai/api-reference/trends_dashboard/delete-widget https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/widgets/{widget_id} # Duplicate Dashboard Source: https://docs.galileo.ai/api-reference/trends_dashboard/duplicate-dashboard https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/dashboards/{trends_dashboard_id}/duplicate # Favorite Dashboard Source: https://docs.galileo.ai/api-reference/trends_dashboard/favorite-dashboard https://api.galileo.ai/public/v2/openapi.json post /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/dashboards/{trends_dashboard_id}/favorite # Get Trends Source: https://docs.galileo.ai/api-reference/trends_dashboard/get-trends https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/log_streams/{log_stream_id}/trends # List Dashboards Source: https://docs.galileo.ai/api-reference/trends_dashboard/list-dashboards https://api.galileo.ai/public/v2/openapi.json get /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/dashboards # Unfavorite Dashboard Source: https://docs.galileo.ai/api-reference/trends_dashboard/unfavorite-dashboard https://api.galileo.ai/public/v2/openapi.json delete /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/dashboards/favorite # Update Section Source: https://docs.galileo.ai/api-reference/trends_dashboard/update-section https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/sections/{section_id} # Update Trends Source: https://docs.galileo.ai/api-reference/trends_dashboard/update-trends https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/log_streams/{log_stream_id}/trends # Update Widget Source: https://docs.galileo.ai/api-reference/trends_dashboard/update-widget https://api.galileo.ai/public/v2/openapi.json put /v2/projects/{project_id}/log_streams/{log_stream_id}/trends/widgets/{widget_id} # Current User Source: https://docs.galileo.ai/api-reference/users/current-user https://api.galileo.ai/public/v2/openapi.json get /v2/current_user # Delete User Source: https://docs.galileo.ai/api-reference/users/delete-user https://api.galileo.ai/public/v2/openapi.json delete /v2/users/{user_id} # Get User Source: https://docs.galileo.ai/api-reference/users/get-user https://api.galileo.ai/public/v2/openapi.json get /v2/users/{user_id} # Get User Roles Source: https://docs.galileo.ai/api-reference/users/get-user-roles https://api.galileo.ai/public/v2/openapi.json get /v2/user_roles Get all user roles. # Invite Users Source: https://docs.galileo.ai/api-reference/users/invite-users https://api.galileo.ai/public/v2/openapi.json post /v2/invite_users # List Users Paginated Source: https://docs.galileo.ai/api-reference/users/list-users-paginated https://api.galileo.ai/public/v2/openapi.json post /v2/users/all # Update User Source: https://docs.galileo.ai/api-reference/users/update-user https://api.galileo.ai/public/v2/openapi.json put /v2/users/{user_id} # Experiments API Guide Source: https://docs.galileo.ai/api/experiments-api-guide Learn how to set up projects, metrics, datasets, and experiments using Galileo's REST API This guide walks through the API calls needed to evaluate LLM outputs and monitor production traces with Galileo. It's intended for customers who cannot use the [Galileo SDKs](/sdk-api/overview) for this purpose. ## Which flow do you need? ```text theme={null} Do you have an OpenTelemetry collector? └─ Yes → OTel Ingestion (configure headers, route to run) └─ No ↓ Is this production monitoring or batch evaluation? ├─ Production monitoring → Log Stream └─ Batch evaluation ↓ Do you already have LLM outputs, or does Galileo need to call the LLM? ├─ Yes, in a CSV → Flow B — Dataset with Pre-generated Outputs ├─ Yes, from live service → Flow C — Raw Trace Ingestion └─ No, let Galileo call the LLM → Flow A — Prompt Template ``` **All flows require a project** (Step 1) and reference metrics by scorer ID. Metrics are org-level resources — configure them once, reuse across any project or flow. *** ## Authentication Every request requires two things: ```text theme={null} Galileo-API-Key: Content-Type: application/json ``` Base URL: your deployment URL, e.g. `https://api.yourcompany.galileocloud.io` > If any POST returns a 422 with "already exists", the resource was already created. Use the corresponding GET endpoint to retrieve it by name. *** ## How Routing Works Every trace or evaluation result lands somewhere specific in Galileo. Three pieces of information control exactly where: | What you provide | Proprietary endpoint | OTel endpoint | | ---------------- | ------------------------------------------------------ | --------------------------------------------------------------- | | **Org** | `Galileo-API-Key` header | `Galileo-API-Key` header | | **Project** | `project_id` in the URL path | `projectid` header or `galileo.project.id` resource attribute | | **Destination** | `experiment_id` or `log_stream_id` in the request body | `experimentid`/`experiment` or `logstreamid`/`logstream` header | These are not optional — if any layer is missing or wrong, traces land in the wrong place or are rejected. ```text theme={null} Galileo-API-Key → org └── project → project_id (URL) or projectid header ├── experiment_id / experiment → experiment (batch evaluation) └── log_stream_id / logstream → log stream (production monitoring) ``` *** ## Step 1: Create a Project Projects are the top-level container. `type: gen_ai` is required for experiments. [API reference →](/api-reference/projects/create-project) ```text theme={null} POST /v2/projects Galileo-API-Key: Content-Type: application/json { "name": "my-project", "type": "gen_ai" } ``` **Response:** take the `id` — this is your ``, used in every subsequent call. ```json theme={null} { "id": "9f2cc6b5-294c-4133-97fa-6951ee4999c9", "name": "my-project", "type": "gen_ai", ... } ``` > If the project already exists: [`POST /v2/projects/paginated`](/api-reference/projects/get-projects-v2) lists all projects. Find yours by name and take the `id`. *** ## Step 2: Configure Metrics Metrics are **org-level resources** — they exist independently of any project and can be referenced across experiments and Log Streams. Configure them once and reuse them anywhere in your org. Metric names are unique per type within your org (you can have an LLM-based and a code-based metric with the same name, but not two LLM-based metrics with the same name). Both `id` and `name` uniquely identify a metric within its type — but the experiment and Log Stream APIs require the UUID `id`, not the name. Use `POST /v2/scorers/list` to look up a metric's ID by name (shown at the end of this section). Skip this section entirely if you only want to use Galileo's built-in preset metrics. Built-in metrics already have `scoreable_node_types` configured — you don't need to set anything when referencing them in experiments or Log Streams. > **Custom scorers require `scoreable_node_types` to be set explicitly at creation time.** If omitted, the field is null and the scorer is silently excluded when scoring raw traces (it will still work for dataset+prompt experiments). Always include it in your `POST /v2/scorers` call. Galileo has two types of custom metrics: * **LLM-based**: an LLM judge that evaluates outputs against a prompt you write * **Code-based**: a Python function you upload that returns a scored value ### LLM-based metric **Call 1 — Create the scorer shell:** [API reference →](/api-reference/data/create) ```text theme={null} POST /v2/scorers Galileo-API-Key: Content-Type: application/json { "name": "response-quality", "scorer_type": "llm", "description": "Rates response quality on a 0-1 scale.", "scoreable_node_types": ["llm", "chat"], "defaults": { "model_name": "GPT-4o mini", "num_judges": 1, "cot_enabled": false, "output_type": "percentage" } } ``` > **`model_name`** uses the same alias format as `model_alias` in prompt templates — Galileo's display name, not the provider model ID. See `GET /llm_integrations/openai/scorer_models` for valid values (scorer-eligible models only — excludes reasoning models like o-series). The response is large — the only fields you need are `id` and `name`: ```json theme={null} { "id": "b80700af-9b29-4e6f-92a1-3a0a223eeb62", "name": "response-quality", ... } ``` These are your `` and ``. **Call 2 — Attach the prompt and model config:** [API reference →](/api-reference/data/create-llm-scorer-version) ```text theme={null} POST /v2/scorers//version/llm Galileo-API-Key: Content-Type: application/json { "user_prompt": "Evaluate output.\nInput: {input}\nOutput: {output}\nScore 0-1.", "model_name": "GPT-4o mini", "cot_enabled": false, "num_judges": 1 } ``` The prompt receives `{input}` and `{output}` variables from each trace. > **`output_type` is set at scorer creation, not here.** The version endpoint accepts `output_type` but only uses it to backfill the scorer's default if one wasn't set during `POST /v2/scorers`. If you set it upfront (as above), it's ignored here. The version itself never stores `output_type` — the DB column is always null, and responses show `"boolean"` as a display placeholder regardless of what the scorer actually uses. ### Code-based metric Code-based metrics require a validation step before upload. The function signature is something like: ```python theme={null} def scorer_fn(step_object, **kwargs) -> float: ... ``` A return type annotation is required — omitting it will fail validation. Accepted types are `-> float`, `-> bool`, `-> int`, or `-> str`. `**kwargs` is required to ensure forward compatibility with additional arguments the platform may pass. The `step_object` has `input`, `output`, `metadata`, and `spans` attributes. > **`output` type depends on who created the trace.** For traces with pre-generated outputs , `output` at the trace level is a plain string. When Galileo calls the LLM via a prompt template, trace-level `output` is a `Message` object — access the text via `step_object.output.content`. **Example Scorer**: ```python theme={null} def scorer_fn(step_object, **kwargs) -> float: output = step_object.output if not output: return 0.0 # output is a plain string for raw trace ingestion (Flow C); # a Message object when Galileo called the LLM via a prompt template (Flow A). text = output if isinstance(output, str) else output.content return min(len(text) / 200.0, 1.0) ``` **Call 1 — Create the scorer shell:** [API reference →](/api-reference/data/create) ```text theme={null} POST /v2/scorers Galileo-API-Key: Content-Type: application/json { "name": "response-length-scorer", "scorer_type": "code", "description": "Scores response length normalized to 0-1.", "scoreable_node_types": ["trace"] } ``` > **`scoreable_node_types`** controls which node type the scorer runs on. Valid values: `trace`, `llm`, `retriever`, `tool`, `workflow`, `agent`, `session`. **Always set this explicitly** — if omitted, the field is null in the database and the scorer will be silently excluded from experiments that ingest raw traces (it only works for dataset+prompt flows). Use `["trace"]` for code scorers and `["llm", "chat"]` for LLM scorers to cover all experiment types. **Response:** take `id` and `name` — these are your `` and ``. ```json theme={null} { "id": "c7d8e9f0-3456-7890-cdef-012345678901", "name": "response-length-scorer" } ``` **Call 2 — Validate the code (async):** [API reference →](/api-reference/data/validate-code-scorer) ```text theme={null} POST /v2/scorers/code/validate Galileo-API-Key: Content-Type: multipart/form-data file: scoreable_node_types: trace ``` Returns a `task_id`. Poll until `status` is `completed`: [API reference →](/api-reference/data/get-validate-code-scorer-task-result) ```text theme={null} GET /v2/scorers/code/validate/ Galileo-API-Key: ``` ```json theme={null} { "status": "completed", "result": { "result": { "result_type": "valid", "score_type": "float", ... } } } ``` The value you pass as `validation_result` is the object under the top-level `"result"` key — i.e. `{"result": {"result_type": "valid", "score_type": "float", ...}}`. Serialize that object as a JSON string and pass it in Call 3. Not the inner `"result"` object, and not the full poll response — just that one level down. **Call 3 — Upload the code version:** [API reference →](/api-reference/data/create-code-scorer-version) ```text theme={null} POST /v2/scorers//version/code Galileo-API-Key: Content-Type: multipart/form-data file: validation_result: ``` ### Looking up metric IDs by name The experiment and Log Stream APIs require metric UUIDs, not names. If you need to retrieve a metric's ID by name: [API reference →](/api-reference/data/list-scorers-with-filters) ```text theme={null} POST /v2/scorers/list Galileo-API-Key: Content-Type: application/json { "filters": [ { "name": "name", "operator": "one_of", "value": ["response-quality", "response-length-scorer"] } ] } ``` Each result in the response includes both `id` and `name`. Use the `id` when referencing scorers in experiment or Log Stream calls. > To look up preset scorers (e.g. `context_adherence`, `correctness`), use the same endpoint with `"operator": "one_of"` and the preset scorer names. *** ## Datasets Datasets are **independent versioned resources** — upload once, reference by ID across any experiment. Flows A and B require a dataset. Flow C and the function-based SDK path do not — traces are posted directly at runtime. Four file formats are supported — pass `?format=` in the query string (defaults to `csv` if omitted): | `format` value | Description | | -------------- | -------------------------------------------------- | | `csv` | CSV (default). Auto-detects encoding. | | `jsonl` | Newline-delimited JSON — one object per line. | | `json` | JSON array of objects. Top-level must be an array. | | `feather` | Apache Arrow Feather binary format. | Reserved column names: | Column | Purpose | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input` | The input passed to the prompt template (or used as trace input) | | `output` | Ground truth / expected answer — used by scorers that compare against a reference answer (e.g. correctness). Not the LLM-generated output. **Use `output` when uploading via API** — the alias `ground_truth` is only normalized by the Galileo UI uploader, not the API. Datasets uploaded via API with a `ground_truth` column will show an empty "Dataset Ground Truth" column in experiment results. | | `generated_output` | Pre-generated LLM output. When present and no prompt template is provided, Galileo scores these directly without calling an LLM (Flow B). | | `metadata` | Arbitrary metadata for the row | > **Note:** This endpoint uses `multipart/form-data` (file upload), not JSON. [API reference →](/api-reference/datasets/create-dataset) ```bash theme={null} curl -X POST "$GALILEO_BASE_URL/v2/datasets?format=csv" \ -H "Galileo-API-Key: $GALILEO_API_KEY" \ -F "name=my-dataset" \ -F "draft=false" \ -F "hidden=false" \ -F "file=@dataset.csv" ``` **Response:** take `id` — this is your ``. The first uploaded version always has `version_index: 1`. ```json theme={null} { "id": "d3f1a2b4-8c9e-4f7d-b123-456789abcdef", "name": "my-dataset", "version_index": 1, ... } ``` > If the dataset already exists: [`GET /v2/datasets`](/api-reference/datasets/list-datasets) lists all datasets. Find yours by name. *** ## Step 3: Run an Experiment ### Flow A — Prompt Template Evaluation Galileo runs your prompt template against each row in a dataset and scores the outputs. > **Prerequisite:** Flow A requires an LLM integration to be configured in Galileo (Settings → Integrations in the UI). Galileo calls the LLM on your behalf using those credentials. > > **`model_alias` is Galileo's display name for the model — it is not the provider's model ID.** `"GPT-4o mini"` and `"gpt-4o-mini"` are different strings and only one is accepted. For legacy models they differ; for newer models they often match. To see valid alias strings for your configured integrations: > > ```text theme={null} > GET /llm_integrations/openai/models > ``` > > Returns a flat array of valid alias strings, e.g. `["GPT-4o", "GPT-4o mini", "gpt-4.1", "gpt-4.1-mini", ...]`. Use the exact string from this list. Replace `openai` with your provider (`anthropic`, `azure`, `mistral`, `vertex_ai`, `aws_bedrock`, etc.). > > The same alias format applies to `model_name` in LLM scorer creation — it is the same value set. #### A1. Create a prompt template If you already have a prompt template in the Galileo UI, retrieve its version ID: ```bash theme={null} GET /projects//templates ``` Find your template by name, then take `selected_version_id` — this is your ``. To create a new template, template and first version are created in a single call: ```bash theme={null} POST /projects//templates { "name": "my-prompt-template", "template": "Answer the following question concisely.\n\nQuestion: {{input}}\n\nAnswer:", "settings": { "model_alias": "GPT-4o mini", "temperature": 0.0, "max_tokens": 1024 } } ``` > **`model_alias`** is Galileo's display name for the model, e.g. `GPT-4o mini` — not the provider's model ID (`gpt-4o-mini`). Use the exact string shown in Settings → Integrations for the model you have configured. > **Template variable syntax:** Prompt templates use **Mustache** (`{{variable}}`), not Python format strings. Use `{{input}}` to reference the dataset `input` column. Using `{input}` (single braces) passes the literal string through unchanged — the LLM will never see the actual question. Note: LLM scorer `user_prompt` fields use `{input}` and `{output}` (single braces) — that is a different substitution system. **Response:** take `selected_version_id` — this is your ``. ```json theme={null} { "id": , "name": "my-prompt-template", "selected_version_id": "v9f8e7d6-5c4b-3a2f-1e0d-9c8b7a6f5e4d" } ``` To add a new version to an existing template: ```bash theme={null} POST /templates//versions { "template": "Answer concisely and cite your source.\n\nQuestion: {{input}}\n\nAnswer:", "settings": { "model_alias": "GPT-4o mini", "temperature": 0.0 } } ``` #### A2. Upload a dataset Follow the [Datasets](#datasets) section above if you haven't already. Flow A requires a dataset with at least an `input` column — include an `output` column if you want ground-truth-based metrics (e.g. correctness) to run. Take the `` from the upload response. #### A3. Create and trigger the experiment This single call uploads the experiment configuration and starts execution immediately (`trigger: true`). Galileo runs the prompt against each dataset row and scores the outputs. [API reference →](/api-reference/experiment/create-experiment) ```bash theme={null} POST /v2/projects//experiments { "name": "my-experiment", "task_type": 16, "dataset": { "dataset_id": "", "version_index": 1 }, "prompt_template_version_id": "", "prompt_settings": { "model_alias": "GPT-4o mini" }, "scorers": [ { "id": "", "scorer_type": "llm", "name": "" }, { "id": "", "scorer_type": "code", "name": "" } ], "trigger": true } ``` To use preset scorers instead of (or alongside) custom ones, look up their IDs first: ```bash theme={null} POST /v2/scorers/list { "filters": [ { "name": "name", "operator": "one_of", "value": ["context_adherence", "correctness"] } ] } ``` Pass preset scorers with `"scorer_type": "preset"` in the scorers array. **Response:** take `id` — this is your ``. ```json theme={null} { "id": "e4f5a6b7-2345-6789-bcde-f01234567890", "name": "my-experiment", "status": { "log_generation": { "progress_percent": 0.0 } } } ``` > **Scoring is async.** Poll `GET /projects//runs//jobs` until every job in the response has a terminal status. Non-terminal statuses are `unstarted` and `in_progress` — keep polling while any job has either. Terminal statuses are `completed`, `processed`, `error`, and `failed`. Do not filter by `?status=in_progress` — jobs start in `unstarted` before a worker picks them up, so that filter can return empty before any work has started. Note: `completed` status is set before metric records finish writing to the database — there is an async queue hop between job completion and metrics being queryable (up to \~5 seconds). If the metrics fetch returns empty results immediately after jobs complete, wait briefly and retry. > > **If a scorer errors, its column will not appear in the UI or aggregate metrics response.** Traces will show `status_type: "pending"` for that metric in the trace search API — the scorer job failed before writing results. Check for `error` or `failed` jobs at `GET /projects//runs//jobs` to diagnose the root cause. *** ### Flow B — Dataset with Pre-generated Outputs Your application has already produced LLM outputs. You have them in a structured dataset (CSV). Upload the dataset with a `generated_output` column — Galileo scores the outputs directly without calling an LLM. This is the cleanest path when your outputs are already tabular. If your outputs aren't in a CSV — whether they're generated at runtime or already in memory — see Flow C below. #### B1. Upload your dataset Follow the [Datasets](#datasets) section above. Your CSV must include a `generated_output` column alongside `input` and `output`: ```text theme={null} input,output,generated_output "What is the capital of France?","Paris","The capital of France is Paris." "What is 2 + 2?","4","2 + 2 equals 4." "Who wrote Hamlet?","William Shakespeare","Hamlet was written by Shakespeare." ``` Take the `` from the upload response. #### B2. Create and trigger the experiment Same call as Flow A but without `prompt_template_version_id` or `prompt_settings`. The presence of `generated_output` in the dataset signals Galileo to score those values directly. [API reference →](/api-reference/experiment/create-experiment) ```bash theme={null} POST /v2/projects//experiments { "name": "my-generated-output-experiment", "task_type": 16, "dataset": { "dataset_id": "", "version_index": 1 }, "scorers": [ { "id": "", "scorer_type": "llm", "name": "" }, { "id": "", "scorer_type": "code", "name": "" } ], "trigger": true } ``` **Response:** take `id` — this is your ``. ```json theme={null} { "id": "a9b0c1d2-5678-9012-ef01-234567890123", "name": "my-generated-output-experiment", "status": { "log_generation": { "progress_percent": 0.0 } } } ``` > **Scoring is async.** Same as Flow A — poll `GET /projects//runs//jobs` until all jobs have a terminal status (`completed`, `processed`, `error`, `failed`), then allow a brief delay before fetching metrics. > If both `prompt_template_version_id` and `generated_output` are present, the prompt template takes priority and Galileo calls the LLM — the `generated_output` column is ignored. *** ### Flow C — Raw Trace Ingestion You generate outputs at runtime (e.g. from a live service) and POST them directly as traces. No CSV dataset required. #### C1. Create an experiment [API reference →](/api-reference/experiment/create-experiment) ```text theme={null} POST /v2/projects//experiments Galileo-API-Key: Content-Type: application/json { "name": "my-trace-experiment", "task_type": 16 } ``` Take the `id` — this is your ``. #### C2. Register scorers Scorers are attached to the experiment via a separate call. This is what tells the platform which metrics to compute when traces arrive. [API reference →](/api-reference/log_stream/update-metric-settings) > **Order matters:** Register scorers before ingesting traces. When traces arrive with `is_complete: true`, Galileo immediately enqueues scoring jobs — those jobs look up registered scorers at processing time. If scorer registration hasn't completed yet, custom scorers will be silently skipped and only built-in metrics (cost, latency) will run. If that happens, re-calling this endpoint after traces exist will trigger a recompute automatically for the newly added scorers. ```text theme={null} PATCH /projects//experiments//metric_settings Galileo-API-Key: Content-Type: application/json { "scorers": [ { "id": "", "scorer_type": "llm" }, { "id": "", "scorer_type": "code" } ] } ``` #### C3. Ingest traces [API reference →](/api-reference/trace/log-traces) ```bash theme={null} POST /v2/projects//traces { "experiment_id": "", "is_complete": true, "traces": [ { "id": , "type": "trace", "input": "What is the capital of France?", "output": "The capital of France is Paris.", "dataset_output": "Paris", "created_at": "2026-04-15T10:00:00Z", "spans": [ { "id": , "type": "llm", "input": [{"role": "user", "content": "What is the capital of France?"}], "output": {"role": "assistant", "content": "The capital of France is Paris."}, "model": "gpt-4o-mini", "created_at": "2026-04-15T10:00:00Z" } ] } ] } ``` > **Trace and span `id` fields must be valid UUID v4.** The API rejects non-UUID4 strings with a 422. Generate a UUID4 per trace/span. `is_complete` controls whether scoring is triggered for the traces in that request: * `true` → traces stored and scoring triggered immediately for this batch * `false` → traces stored only, no scoring yet For large trace sets that don't fit in a single request: send intermediate batches with `is_complete: false`, then send the final batch with `is_complete: true`. Scoring fires once across all accumulated traces when the final batch lands. `dataset_output` is the ground truth — include it if you want ground-truth-based scorers to run. *** ### Log Streams Experiments are batch evaluation runs. For continuous production monitoring, use a **Log Stream** — traces stream in from live traffic and are scored as they arrive. #### LS1. Create a Log Stream [API reference →](/api-reference/log_stream/create-log-stream) ```text theme={null} POST /projects//log_streams Galileo-API-Key: Content-Type: application/json { "name": "my-log-stream" } ``` **Response:** take `id` — this is your ``. ```json theme={null} { "id": "b1c2d3e4-6789-0123-f012-345678901234", "name": "my-log-stream", "project_id": "" } ``` #### LS2. Attach scorers Log streams are runs — use the same `scorer-settings` endpoint as experiments, with `LOG_STREAM_ID` as the run ID: ```text theme={null} POST /projects//runs//scorer-settings Galileo-API-Key: Content-Type: application/json { "run_id": "", "scorers": [ { "id": "", "scorer_type": "llm", "name": "" }, { "id": "", "scorer_type": "code", "name": "" } ] } ``` #### LS3. Ingest traces Same proprietary endpoint as Flow C but with `log_stream_id` instead of `experiment_id`: [API reference →](/api-reference/trace/log-traces) ```text theme={null} POST /v2/projects//traces Galileo-API-Key: Content-Type: application/json { "log_stream_id": "", "is_complete": true, "traces": [ { "id": , "type": "trace", "input": "What is the capital of France?", "output": "The capital of France is Paris.", "created_at": "2026-04-15T10:00:00Z", "spans": [ { "id": , "type": "llm", "input": [{"role": "user", "content": "What is the capital of France?"}], "output": {"role": "assistant", "content": "The capital of France is Paris."}, "model": "gpt-4o-mini", "created_at": "2026-04-15T10:00:00Z" } ] } ] } ``` #### LS4. Query trace results [API reference →](/api-reference/trace/query-traces) ```text theme={null} POST /v2/projects//traces/search Galileo-API-Key: Content-Type: application/json { "log_stream_id": "", "filters": [], "sort": {"column_id": "created_at", "ascending": false}, "limit": 100, "starting_token": 0 } ``` Returns paginated trace records with per-trace metric scores. Filter by metric values here to find traces below your quality threshold. *** ### OTel Ingestion If you have an OpenTelemetry collector in your infrastructure, you can pipe traces directly to Galileo's OTel endpoint instead of the proprietary `/v2/projects/{PROJECT_ID}/traces`. The transformation pipeline is the same — both converge on the same trace processing logic. The difference is just ingestion format and routing mechanism. **Endpoint:** ```text theme={null} POST /otel/v1/traces ``` Auth is the same `Galileo-API-Key` header. #### Routing headers Where the trace lands is controlled by headers on the OTel collector. All header names are lowercase, no dashes or underscores: | Header | Value | Effect | | -------------- | ------ | ---------------------------------------------------------------------------- | | `projectid` | UUID | Route to this project | | `experimentid` | UUID | Send to an existing experiment | | `experiment` | string | Send to experiment by name | | `logstreamid` | UUID | Send to an existing Log stream | | `logstream` | string | Send to Log stream by name — auto-creates if it doesn't exist | | `sessionid` | string | Associate traces with a session (Log streams only — ignored for experiments) | Use `experimentid`/`experiment` for offline evaluation. Use `logstreamid`/`logstream` for production monitoring. #### Resource attributes (alternative to headers) Embed routing in the OTel resource instead of collector headers: | Attribute | Effect | | ------------------------ | --------------------------- | | `galileo.project.id` | Route to project by UUID | | `galileo.project.name` | Route to project by name | | `galileo.logstream.id` | Route to Log stream by UUID | | `galileo.logstream.name` | Route to Log stream by name | > **Note:** Resource attributes only support Log stream routing. To route OTel traces to an experiment, use the `experimentid` or `experiment` HTTP header — there is no resource attribute equivalent. #### Session auto-resolution Galileo extracts session ID automatically from span attributes — no manual `sessionid` header needed if your spans carry any of: * `session.id` * `gen_ai.conversation.id` * `galileo.session.id` > **Prerequisite:** The experiment or Log stream must exist before traces arrive, unless using the `logstream` header with a name — that auto-creates the Log stream. *** ## Step 4: Fetch Results ### Aggregate metrics Returns aggregate stats per scorer across all traces in the experiment. Use this to answer "did this experiment pass my quality bar" before drilling into individual traces. [API reference →](/api-reference/experiment/get-experiment-metrics) ```text theme={null} POST /v2/projects//experiments//metrics Galileo-API-Key: Content-Type: application/json {} ``` ```json theme={null} { "metrics": [ { "name": "response-quality", "data_type": "percentage", "average": 0.87, "buckets": { "0.25": 2, "0.50": 5, "0.75": 8, "1.00": 3, "other": 0 }, "roll_up_method": null, ... }, { "name": "is-correct", "data_type": "boolean", "average": null, "buckets": { "True": 14, "False": 4 }, "roll_up_method": "percentage_true", ... }, ... ] } ``` Each entry in `metrics` represents one scorer. Key fields: | Field | Meaning | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Scorer name | | `data_type` | Output type: `percentage`, `boolean`, `categorical`, etc. | | `average` | Mean score across all traces — populated for numeric/percentage scorers, `null` for boolean/categorical | | `buckets` | Distribution histogram. For numeric scorers: quartile ranges (`"0.25"`, `"0.50"`, `"0.75"`, `"1.00"`, `"other"`) with trace counts. For boolean scorers: `{"True": N, "False": N}`. For categorical: one key per category. | | `roll_up_method` | How to summarize the metric — e.g. `percentage_true` for boolean scorers. `null` for numeric. | > To identify your scorer entries in the response, filter for `roll_up_method != null`. LLM scorers also emit sub-entries (`_input_tokens`, `_output_tokens`, `_total_tokens`, `_scorer_version_id`) — these all have `roll_up_method: null` and can be discarded. ### Per-trace results Returns individual traces with per-trace metric values. Useful for finding which inputs scored below a threshold. [API reference →](/api-reference/trace/query-traces) ```text theme={null} POST /v2/projects//traces/search Galileo-API-Key: Content-Type: application/json { "experiment_id": "", "filters": [], "sort": { "column_id": "created_at", "ascending": false }, "limit": 100, "starting_token": 0 } ``` **Response shape:** ```json theme={null} { "starting_token": 0, "next_starting_token": 100, "num_records": 250, "paginated": true, "records": [ { "id": "", "type": "trace", "input": "Answer the following question...", "output": "The capital of France is Paris.", "created_at": "2026-04-25T16:25:50.026022Z", "dataset_input": "What is the capital of France?", "dataset_output": "Paris", "dataset_metadata": {}, "metric_info": { "response-quality-scorer": { "status_type": "success", "value": 1.0, "explanation": "The response has a 100.00% chance of fitting criteria.", "rationale": "...", "cost": 0.000154, "model_alias": "GPT-4o mini", "num_judges": 1, "input_tokens": 677, "output_tokens": 87, "total_tokens": 764 }, "response-length-scorer": { "status_type": "success", "value": 0.155 }, "duration_ns": { "status_type": "success", "value": 1041921536 }, "cost": { "status_type": "success", "value": 0.0000091 } }, "has_children": true, "is_complete": true, "run_id": "", "project_id": "" } ] } ``` Key fields: | Field | Notes | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metric_info` | Structured per-metric results — use this for parsing. Keyed by scorer name. | | `metric_info[name].status_type` | `"success"` or `"pending"`. `"pending"` means the scorer job failed before writing — check `GET /projects//runs//jobs` for the error. | | `metric_info[name].value` | The numeric score. | | `metric_info[name].explanation` / `rationale` | LLM scorer reasoning (present for LLM scorers only). | | `dataset_input` / `dataset_output` | Ground truth from the dataset (Flow A only). Compare against `output` to compute your own pass/fail logic. | | `next_starting_token` | Pass as `starting_token` in the next request to paginate. | | `metrics` | Flat dict version of all metric values — same data as `metric_info`, less structured. Useful for quick ad-hoc access. | #### Filtering traces To filter to traces below a score threshold: ```json theme={null} "filters": [ { "type": "number", "name": "metrics/response-quality-scorer", "operator": "lt", "value": 0.5 } ] ``` > **Scorer names are used as-is in filter names.** If your scorer is named `response-quality-scorer`, the filter name is `metrics/response-quality-scorer` — hyphens are preserved, not converted to underscores. #### Sorting by metric value To surface the worst-scoring traces first, sort by the metric ascending: ```json theme={null} "sort": { "column_id": "metrics/response-quality-scorer", "ascending": true } ``` #### Filter operator reference **Number filters** (`"type": "number"`) — for `metrics/`, `duration_ns`, `cost`, `num_total_tokens`: | Operator | Meaning | | --------- | ---------------------------------------- | | `eq` | Equal to | | `ne` | Not equal to | | `gt` | Greater than | | `gte` | Greater than or equal to | | `lt` | Less than | | `lte` | Less than or equal to | | `between` | Inclusive range — `"value": [low, high]` | **Text filters** (`"type": "text"`) — for `input`, `output`, `name`, `external_id`: | Operator | Meaning | | ---------- | ---------------------------------------- | | `eq` | Exact match | | `ne` | Not equal | | `contains` | substring match | | `one_of` | Value is in list — `"value": ["a", "b"]` | | `not_in` | Value is not in list | **ID filters** (`"type": "id"`) — for `id`, `session_id`: | Operator | Meaning | | -------- | ---------------------- | | `eq` | Exact UUID match | | `one_of` | Match any UUID in list | **Date filters** (`"type": "date"`) — for `created_at`, `updated_at`: | Operator | Meaning | | ------------ | ------------------ | | `gt` / `gte` | After a timestamp | | `lt` / `lte` | Before a timestamp | Multiple filters in the `filters` array are combined with AND. # Overview Source: https://docs.galileo.ai/api/getting-started Learn how to get started with the Galileo REST API Galileo provides a public REST API that you can use to interact with the Galileo platform. This guide will help you get started with the Galileo REST API. ## Base API URL The first thing you need to call the Galileo API is the base URL of your Galileo API instance. ### Free or hosted Galileo version If you are using the free or hosted tier of Galileo at [app.galileo.ai](https://app.galileo.ai), then the base API URL is [https://api.galileo.ai](https://api.galileo.ai). ### Custom deployment For custom deployments, you will need your Galileo console URL. You can then replace `console` in it with `api`. For example, if your Galileo console URL is `https://console.galileo.myenterprise.com`, then your base URL for the API is `https://api.galileo.myenterprise.com`. ### Verify the Base URL To verify the base URL of your Galileo API instance, you can send a `GET` request to the [`healthcheck` endpoint](/api-reference/health/healthcheck). ```bash theme={null} curl -X GET https://api.galileo.ai/v2/healthcheck ``` The API version will be reported in the response: ```output theme={null} ➜ curl -X GET https://api.galileo.ai/v2/healthcheck {"api_version":"1.0.0","message":"🔭 API","version":"1.844.0"} ``` ## Authentication For interacting with our public endpoints, you can use any of the following methods to authenticate your requests: ### API Key To use your [API key](/references/faqs/find-keys#galileo-api-key) to authenticate your requests, include the key in the HTTP headers for your requests. ```json theme={null} { "Galileo-API-Key": "" } ``` ### HTTP Basic Auth To use HTTP Basic Auth to authenticate your requests, include your username and password Base64 encoded in the HTTP headers for your requests. ```json theme={null} { "Authorization": "Basic :)>" } ``` ### JWT Token To use a JWT token to authenticate your requests, include the token in the HTTP headers for your requests. ```json theme={null} { "Authorization": "Bearer " } ``` We recommend using this method for high-volume requests because it is more secure (expires after 24 hours) and scalable than using an API key. To generate a JWT token, send a `GET` request to the [`get-token` endpoint](/api-reference/auth/get-token) using the API Key or HTTP Basic auth. # Access Control Source: https://docs.galileo.ai/concepts/access-control Control access to projects via role-based access control and groups in Galileo For organizations requiring role-based access control (RBAC), Galileo supports fine-grained control over granting users different levels of access to the system, as well as organizing users into groups for easily sharing projects. Some features are only available to customers on paid Galileo plans. ## System-level Roles There are four roles that a user can be assigned: * **Admin** - Full access to the organization, including viewing all projects. * **Manager** (enterprise only) - Can add and remove users. * **User** - Can create, update, share, and delete projects and resources within projects. * **Read-only** - Cannot create, update, share, or delete any projects or resources. Limited to view-only permissions. *Note:* Free users of Galileo can only use the Admin, User, or Read-only roles. [Contact us](https://galileo.ai/contact-sales) to explore a paid plan and get full RBAC. In table form: | | Admin | Manager | User | Read-only | | ------------------------------------- | ---------------------------------- | ----------------------------------------------- | ------------------------------------------ | ------------------------------------------ | | View all projects | | | | | | Add/delete users | | (excluding admins) | | | | Create groups, invite users to groups | | | | | | Create/update projects | | | | | | Share projects | | | | | | View projects | (all) | (only shared) | (only shared) | (only shared) | System-level roles are chosen when users are invited to Galileo: Image shows the pop up when inviting new users to Galileo and the system role options provided ## Groups (enterprise only) Users can be organized into groups to streamline sharing projects. Currently, groups are only available to customers on paid plans of Galileo. There are 3 types of groups: * **Public** - Group and members are visible to everyone in the organization. Anyone can join. * **Private** - Group is visible to everyone in the organization. Members are kept private. Access is granted by a group maintainer. * **Hidden** - Group and its members are hidden from non-members in the organization. Access is granted by a group maintainer. Within a group, each member has a group role: * **Maintainer** - Can add and remove members. * **Member** - Can view other members and shared projects. ## Share Projects By default, only a project's creator (and managers and admins) have access to a project. Projects can be shared both with individual users and entire groups. Together, these are called *collaborators*. How to share a project with collaborators: Share a project within Galileo # Agent Control Source: https://docs.galileo.ai/concepts/agent-control/overview Learn how to set up Agent Control to safeguard your AI applications. Agent Control provides a centralized layer for evaluating LLM and tool inputs and outputs during agent workflow execution. It helps block harmful content, prompt injection, PII leakage, and other risks without requiring changes to agent code. Agent Control supports the following use cases: * For developers: Centralize safety logic and adapt to emerging threats without redeployment. * For non-technical teams: Configure and monitor agent safety without touching code. * For organizations: Reuse controls across agents with audit-ready traces. For more information about Agent Control concepts, see [Overview](https://docs.agentcontrol.dev/concepts/overview). ## Prerequisites Install the dependencies: ```python theme={null} pip install "galileo[openai]>=2.3.0" "agent-control-sdk[galileo]>=7.10.0" "agent-control-evaluators>=7.10.0" "agent-control-evaluator-galileo>=7.10.0" ``` ## Get started with Agent Control 1. [Create a control](/how-to-guides/agent-control/create-a-control) 2. [Initialize and configure Agent Control](/how-to-guides/agent-control/initialize-and-configure-agent-control) 3. [Monitor a control](/how-to-guides/agent-control/monitor-a-control) # AI Assistant (Beta) Source: https://docs.galileo.ai/concepts/ai-assistant Ask about your agentic system in natural language. Galileo's AI Assistant combs through your traces, spans, sessions, and evaluation scores to surface the root cause, the evidence, and a concrete fix. The AI Assistant is in Beta. Responses are AI-generated and can be incomplete or incorrect, so verify anything you act on. The feature set is still evolving, and behavior may change. Ask a question and start an investigation. Why did the agent take this action? What's driving this metric? What is this judge rationale really telling me? The assistant reads across your traces, spans, and sessions and answers with links straight to the evidence. It lives in a chat panel inside the Galileo console, on the pull side of your investigation experience: [Signals](/concepts/signals) pushes the issues you didn't know to look for, and the assistant lets you pull on any thread on demand. When a Signal surfaces something, ask follow-up questions to keep the investigation going. Open it from anywhere you debug within Galileo to get started. ## Why use it Get a direct answer instead of reading through spans one by one. Aggregate failure reasons across a whole Log stream, not just a single trace. Turn an explanation into a concrete recommendation, like a prompt change or a new metric to track. PMs, domain experts, and business owners can investigate AI behavior without learning the trace UI. ## How it works For example, you can open the AI Assistant from: * A **Log stream** to investigate production traffic. The assistant can reach the traces, spans, and sessions inside the stream, so you can look at one bad result or patterns across the whole Log stream. * An **Experiment** to investigate a specific run or compare runs. The following screenshot shows the AI Assistant opened from a Log stream. The AI Assistant showing a Signal investigation result with a finding, evidence, and links to affected sessions Ask in natural language. The assistant queries the relevant data and answers in the chat panel. Every answer cites the traces, sessions, and docs it used, so you can click through to confirm. Ask a follow-up to keep the same investigation going in one thread. The AI Assistant panel open inside a Log stream, showing suggested questions and a prompt to run Signals ## What you can ask The assistant is most useful when your question is grounded in data Galileo already has. Here are the kinds of questions it handles today, grouped by what you're trying to do. | What you want to do | Example questions | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Analyze the root cause of a failure** | "Why did the agent make too many tool calls here?" · "Is this failure due to the model, the prompt, or the data?" · "Which span derailed the Action Completion?" | | **Get a fix** | "What should I do next to fix the issues surfaced by my metrics?" · "How should I modify my prompt to fix this issue?" · "How do I improve my agent's performance based on these metric results?" | | **Find similar traces** | "Show me traces similar to this one where the agent hallucinated." · "Show me all traces where chunk relevance was low." · "Show me sessions where the user escalated or asked the same question twice." | | **Spot patterns across a Log stream** | "What are the top failure patterns this week?" · "What categories of queries are underperforming?" · "Which tool calls are producing the most errors?" | | **Understand a metric** | "What does Context Adherence measure?" · "Instruction Adherence looks low. Which instructions failed?" · "I have traces flowing. What metrics should I turn on?" | | **Compare experiments** | "Did my new prompt actually improve quality?" · "What got better and what got worse after my change?" · "Which sessions regressed after my last prompt change?" | | **Investigate a Signal** | "Explain this Signal and show me the traces behind it." · "Which Signal should I look at first?" · "How many traces are affected by this Signal?" | ## Conversations Your conversations are saved, so you can pick up an investigation later. Keep related questions in one thread so the context stays intact. ## Known limitations The assistant is read-only today. A few things to keep in mind: * **It answers and recommends, but it doesn't act yet.** It can suggest a fix and show you where to make it, but you'll need to apply the changes yourself. * **No memory across conversations.** Context doesn't carry over between threads. * **It's not a replacement for your judgment.** The assistant can be wrong. Confirm anything before you ship a change based on it. The assistant runs on the latest reasoning models, served through your configured [LLM integration](/getting-started/evaluate-and-improve/evaluate-and-improve#configure-an-llm-integration). Make sure an integration is set up for your project. ## Tips for better answers * **Be specific about the symptom.** "Why is Action Completion false on these 12 sessions?" is more effective than "what's wrong with my agent?" * **Follow the citations.** When an answer matters, click through to the source traces to confirm it. * **Start a new thread for a new problem.** Keep one investigation per conversation so the context stays focused. ## Next steps Set up metrics so the assistant has scores to explain. Let Galileo proactively surface the issues worth asking about. Understand the metrics the assistant explains and recommends. Turn a recommended fix into a runtime control. # Annotations Overview Source: https://docs.galileo.ai/concepts/annotations/overview **Annotations** allow users to provide human feedback on LLM inputs and outputs through the Galileo Console UI and the [API](/api-reference/annotation/create-annotation-template). Galileo supports annotations on Sessions, Traces, and Spans. * The Messages page allows you to submit annotations, and highlights available annotations with a dot indicator. * The Logs page allows you to view and export annotations as columns. Annotations For each project, users with the necessary permissions can configure one or more of these annotation types: * **Categories**: Enable annotators to select one or more designated categories * **Score**: Enable annotators to select a number between 0 and a max score * **Star**: Enable annotators to rate from 1 to 5 stars * **Text**: Enable annotators to provide freeform text * **Thumbs Up & Down**: Enable annotators to rate a like or dislike The following roles have permissions to configure annotations: organization admins, project owners, or project editors. The minimum permissions for creating annotations are: organization users, and project annotators. Read-only or viewer roles cannot submit annotations. ## Annotation Queues Galileo's Annotation Queues enable teams to organize and scale human feedback by grouping project logs (sessions, traces, and spans) for structured review by subject matter experts. Annotation Queue in the UI # Integration Costs Source: https://docs.galileo.ai/concepts/costs/integration-costs In the Settings menu, **Integration Costs** allows organization users to track project costs for [LLM-as-a-judge metrics](/concepts/metrics/how-llm-as-judge-metrics-are-calculated). In the Galileo console UI, navigate to the [Integration Costs page](https://app.galileo.ai/settings/integration-costs) by opening the user menu on the bottom-left corner, and then selecting **Integration Costs**. Integration Costs The Integration Costs page is visible to all users in the organization as a read-only view. # Model Pricing Settings Source: https://docs.galileo.ai/concepts/costs/model-pricing-settings Galileo's model pricing allows admins to configure model prices that will be used to calculate app and metric costs. Other organization users can view model pricing in read-only mode. With this feature, teams have more accurate views of how their AI agents are impacting budgets. In the Galileo console UI, navigate to the [Model Pricing settings page](https://app.galileo.ai/settings/model-pricing) by opening the user menu on the bottom-left corner, and then selecting **Model Pricing**. Model Pricing menu item Model Pricing settings are visible to all users in the organization. Admins can add, edit, and reset model prices. Non-admin users have read-only access. Admins can: * Browse and search the models found in Galileo projects * View the current prices used to compute app and metric costs * Provide updated prices for any existing model -- or revert to the default price * Add a new model and price Non-admin users can: * Browse and search the models found in Galileo projects * View the current prices used to compute app and metric costs Model Pricing table Updated prices will apply to new logs and experiments using that model. App and metric cost for historical logs and experiments remain unchanged. # Compare Experiments Source: https://docs.galileo.ai/concepts/experiments/compare Learn how to compare multiple experiment runs in Galileo Once you have run some experiments, the next natural step is to compare the results of your experiments, allowing you to optimize prompts, select the best model for your use case, or tune your input data to suit your needs. Galileo allows you to compare up to five different experiments, showing the difference in outputs, metrics, latency, and token usage. ## Prerequisites To compare experiments, you will need: * A project containing two or more experiments, created either using [playgrounds](/concepts/experiments/running-experiments-in-console), or in [code](/sdk-api/experiments/running-experiments) ## Compare experiments Experiments can be compared from the Experiments tab in the [Galileo Console](https://app.galileo.ai/). Experiments are part of a project, so select the relevant project to see the experiments tab. 1. Open the **Experiments** tab The experiments tab for a project 2. Select the experiments you want to compare by checking the box next to each experiment. Select between two and five experiments. The experiments tab with check boxes checked on the left of two experiments 3. Select the **Compare experiments** button to open the comparison page The compare experiments button, above the rows of experiments 4. You will see the experiments side by side in the comparison page 2 experiments side by side showing metrics, input prompt and output ### Review the comparison The comparison shows each experiment's metrics, inputs, and outputs. 1. If your experiment has multiple inputs, you can navigate between inputs using the forward and backwards buttons. Your experiment inputs should align by position - for example if you have two inputs in each, the comparison is based on input one from experiment one being compared to input one from experiment two, and so on. > All the experiments in the comparison should have the same number of inputs. If they do not, you will only be able to navigate based off of the experiment with the least inputs. The navigation buttons to navigate between inputs 2. The **Details** section shows the model used, and averages and totals for both the cost of each response and generating the metrics, as well as averages for the metrics. > Averages are calculated for experiments with multiple inputs. The details tab showing two experiments, one using GPT 3.5 Turbo, the other using gpt-4o mini. Each detail has averages and totals for costs, and averages for metrics 3. The **Metrics** section shows the metrics for the currently selected input. These metrics include system metrics (latency, the number of input and output tokens), and the selected metrics for the experiment. Comparing two sets of metrics with latency, number of tokens, instruction adherence, and validate investment advice If you hover over a metric, a pop-up will explain the reasoning behind the score, along with details of the LLM used to judge, the cost of the judgment, and the number of judges used. Hovering over a metric showing an explanation in a popup 4. The **Input** and **Output** sections show the input to the experiment, and the output generated by the LLM. The inputs and outputs for an experiment # Run Experiments in Playgrounds Source: https://docs.galileo.ai/concepts/experiments/running-experiments-in-console Learn about running experiments in the Galileo console using Playgrounds and datasets This section will guide you through the process of running experiments in the Galileo console. ## Experiment walkthrough Follow these steps to test and improve your AI projects using the [Galileo console UI](https://app.galileo.ai). In the [Galileo console](https://app.galileo.ai), use the left-hand navigation menu to select the project you would like to experiment with. You can also select the **Create new project** button in the upper-right corner to create a new project. Select or create a project On the **Overview** for the project, the **Playgrounds** table lists the existing Playgrounds for the project. To open an existing Playground, select the row for the Playground from the table. You can also select the **Create Playground** button to create a new Playground. After you create or select a Playground, choose a model using the **Select a model** drop-down menu. Some models require that you enter your corresponding API key. Visit their respective API platforms to obtain your keys, then add it using the [integrations page](https://app.galileo.ai/settings/integrations) in the Galileo console. Select a model for an experiment Select the settings icon to the right of your model name to adjust its behavior: * **Max Tokens:** Sets the maximum number of tokens the model can generate in its output. * **Temperature:** Controls randomness in output—higher values make responses more creative, lower values make them more focused and deterministic. * **Frequency Penalty:** Reduces the likelihood of the model repeating the same tokens by penalizing frequent ones. * **Presence Penalty:** Discourages the model from mentioning tokens that have already appeared, promoting new content. * **Top P:** Limits sampling to the most likely tokens whose cumulative probability is within this threshold (a form of nucleus sampling). Configure model settings for an experiment There are two ways to set the prompt data for your experiment: 1. Add prompts and variables through the console UI. Ideal for quick tests. 2. Use datasets from past experiments or create new ones. Ideal for real, fully-configured experiments. ### Add prompt and variables In the prompt text field, add your prompt. In your prompt, you can use variable names with curly brackets (e.g. `{{variable_name}}`). Add new variable options with the duplicate icon next to **Variable Set** and fill them in with different values to be used in place of your variable. You can also use nested variables by entering JSON formatted key-value pairs into the **Variable Set** text input field. Then, refer to their values with either `{{key}}` or `{{input.key}}` ```json theme={null} { "pepperoni": "pepperoni pizza", "anchovy": "pizza with anchovies" } ``` In this example, both `{{pepperoni}}` and `{{input.pepperoni}}` will result in "pepperoni pizza" being used in the prompt. This approach is great for testing how changing individual words in a prompt structure affects outputs. Add new messages beyond the initial prompt with the **Add Message** button below the prompt field. New messages can be from the user or the model ("system"). Add variables to an experiment ### Add dataset Select the **Add dataset** button in the upper-right corner to choose the dataset your model will use. The datasets listed are from your past experiments. You can also [add your own](/sdk-api/experiments/datasets) by selecting **Create new dataset**. [Learn more about datasets →](/sdk-api/experiments/datasets) Add or create a dataset for an experiment Select the **Add metric** button in the upper-right corner to choose metrics by which your experiment's outputs are measured. Filter and select from the preset metrics, or add your own by selecting **Create metric** in the upper-right corner. Scores are produced for each selected metric after running an experiment. [Learn more about metrics →](/concepts/metrics/overview) Add metrics to an experiment Add additional prompt sections by selecting the actions (...) menu in the upper-right corner of the **Variables** panel, then the **Add Prompt** button. Each new prompt section can have its own distinct configuration of: * Model * Model settings * Prompt * Message conversation Add new prompt sections and customize their settings as needed for your experiment. Add more adjustments to an experiment Select the **Run all** button in the upper-right corner to run your experiments, generate outputs, and calculate evaluations based on your chosen metrics. After the experiment has completed, scroll down to view their outputs and evaluations. The more distinct prompts and variable sets you used, the more results there will be. Select the **Log as experiment** button above the outputs to record all the details of the experiment. Use a descriptive name for your experiment so that it's easy to keep track of your progress. [Learn more about logging →](/sdk-api/logging/logging-basics) Log results That's it! Now, further customize and configure your experiment to meet your testing goals. Log your experiment results, and create new projects to try out different configurations. If you encounter any errors, visit our [Common Errors guide](/references/faqs/errors). # Log Stream Metrics Source: https://docs.galileo.ai/concepts/logging/configure-metrics/configure-metrics Learn how to configure metrics for Log streams, including managing sampling rates Once you have traces feeding in to a Log stream, you can configure the metrics that you want to evaluate. Metrics are managed at organizational level, including the creation of custom metrics, then are used to evaluate traces at the Log stream level. ## Configure metrics for a Log stream ### Configure metrics through the console To configure metrics, open your Log stream and select the **Configure Metrics** button. You will need at least one session in your Log stream to be able to configure metrics. The configure metrics button on the sessions tab This will load the **Configure metrics** pane. The configure metrics pane with the action advancement metric turned on and the switch highlighted, and the save and close button highlighted From here you can filter and search for metrics, then turn on the relevant ones for your Log stream. Once you have the metrics you need turned on, select the **Save and close** button to save your settings. You can also create new custom metrics from this pane, either using an [LLM as a judge](/concepts/metrics/custom-metrics/custom-metrics-ui-llm), or in [code](/concepts/metrics/custom-metrics/custom-metrics-ui-code), then add them to your Log stream. ### Configure metrics in code You can also configure metrics for a Log stream using the Galileo SDKs. ```python Python theme={null} from galileo import GalileoMetrics from galileo.log_streams import enable_metrics # Enable metrics enable_metrics(project_name="MyProject", log_stream_name="MyLogStream", metrics=[GalileoMetrics.context_adherence]) ``` ```typescript TypeScript theme={null} import { enableMetrics, GalileoMetrics } from "galileo"; // Enable metrics await enableMetrics({ projectName: "MyProject", logStreamName: "MyLogStream", metrics: [GalileoMetrics.contextAdherence] }); ``` Set `MyProject` to your project name, and `MyLogStream` to your Log stream name. You can then pass in either the relevant metric enum, or the name of a custom metric. This function will enable just the metrics specified for the Log stream. If you have any other metrics enabled before calling this function, they will be disabled. ## Metric sampling Every evaluation interacts with an LLM (unless you are only using custom code-based metrics), and therefore has an associated cost. When your application is in development you will probably want to evaluate every trace that is captured, but once your application is in production and is scaling to hundreds, thousands, or even millions of users you most likely want to reduce your evaluation costs by only evaluating a small sample of the traces that are captured. You can configure metric sampling at a Log stream level. To configure metric sampling rate rules, select the **Metric Sampling** button from the **Configure metrics** pane. The metrics sampling button From here you can configure the metric sampling rates. These rates can be applied to all metrics (including custom code metrics and Luna-2 metrics), or LLM-as-a-judge metrics only. Set the sampling rate you want, then select the **Save** button. The metrics sampling dialog When you configure the sample rates, all traces are captured and visible in Galileo, but metrics will only be evaluated for those traces based off the sample rates. For example, if you set the sampling to 10% and create 100 traces, then all 100 traces will be visible in Galileo, with metrics evaluated for just 10 of them. ### Metric sampling rates The most basic way to set sampling rates is by a percentage for all incoming logs. When you set a percentage, all traces are stored and available in Galileo, but only that percentage of traces will be evaluated. A trace is either evaluated for all configured metrics, or not evaluated. You can configure sampling at a more granular level by adding additional rules based off metadata set at a trace level. For example, if you are onboarding a new customer and want to evaluate all of their logs during the onboarding process, you can add the customer name to your metadata, and set a rule to evaluate 100% of traces that have that customer name in their metadata. The metric sampling dialog showing 100% sampling if customer is set to important customer, otherwise 10% This metadata is set when you start a trace with the Galileo logger. ```python Python theme={null} logger.start_trace( name="Conversation step", input=user_input, metadata={"customer": "ImportantCustomer"} ) ``` ```typescript TypeScript theme={null} galileoLogger.startTrace({ name: "Conversation step", input: userInput, metadata: { "customer": "ImportantCustomer" } }); ``` These rules are applied in a top-down approach, so the first rule is evaluated and if the metadata matches, then the percentage is used, if not the next rule is evaluated, and so on. Finally if no rules match, the default sampling rate for all traces is used. ## Metric filters Sometimes metrics only make sense for certain spans. For example, if you have a custom metric for verifying the final response to a user from a multi-agent system with multiple LLM spans, you might only want to calculate the metric on the final LLM span that summarizes the results from all the agents. You can filter the spans that a metric is calculated for, based off the span name or span metadata. Metric filtering is configured at the project level, with filtering applying to all Log streams in a project. To configure metric filters, select **Apply filter** from the menu for the metric you want to filter on the **Configure metrics** pane: The apply filter menu option Use the **Add Condition** button to add a condition based off a span name, or span metadata for the span type that the metric evaluates. * For metadata, set the field, the comparison operator, and the value * For the span name, set the comparison operator and the value You can set multiple conditions, and these are combined with an **And** clause, so condition 1 **And** condition 2. The apply filter dialog with a metadata filter for agent is equal to summary agent ## Next steps Explore Galileo's comprehensive metrics framework for evaluating and improving AI system performance across multiple dimensions. Learn how to create evaluation metrics using LLMs to judge the quality of responses. Learn how to create, register, and use custom metrics to evaluate your LLM applications. # Multimodal Observability Source: https://docs.galileo.ai/concepts/logging/multimodal-observability Log, inspect, and evaluate images, audio, and documents alongside text in your traces AI applications increasingly process and generate images, audio, and documents. Text-based logs alone no longer capture enough context to debug or evaluate them effectively. A voice agent's transcription can be perfect while the generated audio sounds robotic. A document extraction can return the right fields but miss a table. An image generation can follow the prompt but produce off-brand visuals. Galileo supports logging multimodal content on trace inputs and outputs, giving teams full visibility into what their models received and produced. With multimodal traces, you can: * Inspect the exact media your model received or generated, not a text summary of it * Evaluate inputs and outputs using multimodal LLM-as-a-judge metrics * Replay and debug issues that would be invisible in a transcript alone *** ## Choose a logging method | Method | Use when... | | :-------------------------------------- | :-------------------------------------------------------------------------- | | **GalileoLogger — log an external URL** | Your content is already hosted externally and accessible via URL | | **GalileoLogger — upload local files** | You're working with files on disk and need to upload them directly | | **LangChain handler** | Your app already uses LangChain — multimodal content converts automatically | *** ## Option 1: Log an external URL Use `DataContentBlock` with the `url` field. No encoding required. ```python Python theme={null} from galileo.logger import GalileoLogger from galileo.schema.content_blocks import TextContentBlock, DataContentBlock logger = GalileoLogger() logger.start_trace( input=[ TextContentBlock(text="Describe this image"), DataContentBlock(modality="image", url="https://example.com/photo.png"), ], project="my-project", ) logger.add_llm_span( input=[{"role": "user", "content": "Describe this image"}], output={"role": "assistant", "content": "It's a cat."}, model="gpt-5", ) logger.conclude(output="It's a cat.") logger.flush() ``` *** ## Option 2: Upload local files Encode local files as base64 and pass them with the `base64` and `mime_type` fields. This works for images, audio, and documents in a single trace. The example below assumes `photo.png`, `recording.wav`, and `report.pdf` are in the same directory as your script: ```python Python theme={null} import base64 from pathlib import Path from galileo.logger import GalileoLogger from galileo.schema.content_blocks import TextContentBlock, DataContentBlock image_b64_data = base64.b64encode(Path("photo.png").read_bytes()).decode() audio_b64_data = base64.b64encode(Path("recording.wav").read_bytes()).decode() pdf_b64_data = base64.b64encode(Path("report.pdf").read_bytes()).decode() logger = GalileoLogger() logger.start_trace( input=[ TextContentBlock(text="Analyze all of these files"), DataContentBlock(modality="image", base64=image_b64_data, mime_type="image/png"), DataContentBlock(modality="audio", base64=audio_b64_data, mime_type="audio/wav"), DataContentBlock(modality="document", base64=pdf_b64_data, mime_type="application/pdf"), ], project="my-project", ) logger.add_llm_span( input=[{"role": "user", "content": "Analyze all of these files"}], output={ "role": "assistant", "content": "The image shows a cat, audio is clear, PDF is a report.", }, model="gpt-5", ) logger.conclude( output="The image shows a cat, audio is clear, PDF is a report." ) logger.flush() ``` `DataContentBlock` supports three modalities: `image`, `audio`, and `document`. *** ## Option 3: Log with the LangChain handler The LangChain handler converts multimodal message content to structured content blocks automatically. Pass multimodal messages the same way you normally would with LangChain — no extra setup: ```python Python theme={null} from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage from galileo.handlers.langchain import GalileoCallback callback = GalileoCallback() llm = ChatOpenAI(model="gpt-5", callbacks=[callback]) response = llm.invoke([ HumanMessage(content=[ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/photo.png"}}, ]) ]) ``` Supported content types: `text`, `image_url`, `audio_url`, `document_url`, `input_image`, and `input_audio`. Base64 data URIs are also supported — the handler extracts the payload and MIME type automatically. *** ## View multimodal content in your traces An audio trace in the Galileo Log stream showing an inline waveform player in the user input, a text output from the assistant, and audio quality metrics in the side panel Multimodal content renders inline in the Log stream alongside span inputs and outputs: * **Audio** renders as an inline waveform player you can play back directly, with download support * **Images** display inline and can be downloaded * **PDFs** appear as inline previews and can be downloaded *** ## Evaluate multimodal traces Galileo provides out-of-the-box LLM-as-a-judge metrics for multimodal content. You can also configure custom LLM-as-a-judge metrics on any span, trace, or session that contains multimodal content. ### Out-of-the-box metrics | Metric | Modality | What it evaluates | | :---------------------------------------------------------------------------------------- | :---------- | :----------------------------------------------------------------------------- | | [**Visual Quality**](/concepts/metrics/multimodal-quality/visual-quality) | Image / PDF | Whether input quality is sufficient for the task to be reliably performed | | [**Visual Fidelity**](/concepts/metrics/multimodal-quality/visual-fidelity) | Image / PDF | Whether a generated image complies with brand rules, based on visible evidence | | [**Interruption Detection**](/concepts/metrics/multimodal-quality/interruption-detection) | Audio | Turn-taking violations — agent overlap, premature barge-in, and user barge-in | ### Custom LLM-as-a-judge metrics The custom metric editor showing Audio modality selected, an LLM model configured, and a judge prompt for evaluating audio quality 1. Go to **Metrics** and create a new custom LLM metric. 2. Configure a model integration. See [suggested models](#suggested-models) below. 3. Under capabilities, select **Image/PDF** or **Audio**. 4. Enable the metric on your Log stream **before** logging content. Metrics compute only when the trace contains at least one attachment matching the enabled capability. A metric with **Image/PDF** enabled returns N/A if the trace contains only audio, or no attachments at all. Similarly, a metric with **Audio** enabled returns N/A on image-only traces. *** ## Supported formats and models ### Supported formats | Modality | Formats | | :------- | :------------ | | Image | `png`, `jpeg` | | Audio | `mp3`, `wav` | | Document | `pdf` | ### Suggested models For best results, use GPT-5 or later (OpenAI) for image and PDF evaluation, and Gemini 3+ via Gemini Enterprise for audio. If using Gemini Enterprise, you will also need to configure a separate GCP bucket and credentials for file uploads. See [how to set up Gemini Enterprise credentials](/sdk-api/third-party-integrations/model-integrations/gemini-enterprise/gemini-enterprise-credentials). *** ## Known limitations * **LangChain handler stores the full message list.** The trace's input and output fields contain the full serialized message structure (e.g., `[{"content": [...blocks...], "role": "user"}]`), not bare content blocks. * **Multimodal attachments are not supported via OpenTelemetry or native callbacks** (e.g., Google ADK, CrewAI). Use GalileoLogger or the LangChain/LangGraph callback instead. * **Multimodal metrics are not supported in playground or prompt experiments.** *** ## Next steps Full reference for logging with GalileoLogger. Complete guide to the Galileo LangChain integration. # Overview Source: https://docs.galileo.ai/concepts/logging/overview Core Observability concepts in Galileo ## What is AI Observability Agentic applications are inherently non-deterministic, meaning their behavior cannot be fully predicted or exhaustively tested before deployment. As a result, traditional monitoring approaches fall short in capturing how these systems behave in production. AI observability provides visibility into the unique runtime behavior of AI applications, allowing teams to understand what is happening under the hood, why it is happening, and how it impacts performance and outcomes. ## Core concepts Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data. * [Log streams](/sdk-api/logging/logging-basics) and [projects](/concepts/projects) organize the data you send to Galileo for a given application or environment. * [Sessions](/concepts/logging/sessions/sessions-overview) group related traces into a complete multi-turn interaction. * [Traces](/sdk-api/logging/galileo-logger#start-a-trace) represent a single turn, request or AI workflow. * [Spans](/sdk-api/logging/galileo-logger#add-spans) capture the individual steps within a trace, such as LLM calls, tool calls, or a retrieval step. ## Getting started Start with [Instrumentation](/sdk-api/logging/logging-basics) to understand how data is structured in Galileo and how to send logs from your application. # Sessions Overview Source: https://docs.galileo.ai/concepts/logging/sessions/sessions-overview Learn about log sessions in Galileo A Session is **a collection of Traces, Events, and Spans emitted by your Application**. They group all Traces for one conversation or evaluation run, giving you a bird's-eye view of that LLM workflow. Imagine you're building an LLM-powered customer service chat application. During development or in production, you will want to see how a multi-turn conversation flows from user prompt, to tool calling, to model response. **Sessions** solve this problem by bundling Log stream traces into a cohesive unit, so you can observe and evaluate an entire agent interaction from start to finish. ## Core concepts Let's take a look at the building blocks of a session. ### Span → trace → session * [**Span**](/sdk-api/logging/galileo-logger#add-spans): The smallest logging unit the system, typically representing a single operation, function call, or request. Each user message, model API call, or model tool usage generates a *Span*. * [**Trace**](/sdk-api/logging/galileo-logger#start-a-trace): When multiple spans occur as part of a single logical operation (e.g. a request that triggers several downstream calls) they form a *Trace*. Traces allow you to see parent/child relationships among spans. * **Session**: A collection of one or more traces that together represent an entire interaction, or multi-step evaluation. A Session bundles related traces so that you can analyze an entire workflow end to end, even if it spans multiple services, threads, or agents. ## How do sessions differ from Log streams? A [**Log stream**](/sdk-api/logging/logging-basics) is a continuous sequence of log entries emitted over time. Log streams simply capture everything in chronological order, and can contain a mix of spans, traces, and sessions. On the other hand, a *Session* is a way to group Traces that are logically connected. And with Galileo, every Session is stored in a Log stream that you can specify either explicitly or using environment variables. ## How do sessions differ from workflows? A **Workflow** is a defined sequence of steps or tasks. It may include branching logic, conditional steps, and dependencies. A *Session* can contain one or more *Workflows* if they are part of the same overall evaluation. ## Where can I find my sessions? Sessions can be viewed in the [Galileo Console](/concepts/logging/sessions/using-sessions#view-your-session). When you create a session, you will usually select a Log stream where they will be found. (If you don't specify one, Galileo will use your default Log stream). Head over to the [Galileo Console](https://app.galileo.ai) and log in. On your dashboard, select the Log stream where you were sending your session logs. If you didn't specify a unique or new Log stream name, you will find the logs in your **default** Log stream. Select your Log stream from the list Selecting the Log stream will bring you to its event records. All logs will be grouped by *Session*, though you can use the control near the top-left of your screen to change the Log stream's event grouping: Event-group controls for selecting Session, Trace, or Span granularity Your session should be visible in the table below the controls, especially if you gave it a recognizable name. Select it to view the traces. Once you select your session, you can see the Traces you captured from your test run as a flowchart. Any tools that were used will also show up as individual Spans. Select the nodes of the flowchart to see their inputs and outputs on the right-edge of your screen. A flowchart showing the nodes in a session Trace Each message from the user and response from the LLM will form a single trace; you can view the contents here in a familiar format, as well as other details like tool calls. Just select the **Messages** tab (shown in the image below) to see a list of traces in the session, along with their child spans. You can select a span to see metrics and other details on the right edge of the screen (not pictured) View Trace messages Use the **Condense Steps** toggle to show only the most relevant spans in a trace. This will include any tool calls made by the LLM! You can learn more about creating and using sessions [in our using sessions guide](/concepts/logging/sessions/using-sessions). ## Conclusion A Session can collect multiple workflow runs and traces into one cohesive view. By using Sessions in your LLM application, you can: 1. Organize logs and metrics for each customer interaction or batch evaluation run, so debugging and analysis become straightforward. 2. Drill down into any step, inspecting the span for tokenization latency or the trace for scoring logic without losing context. 3. Compare multiple chat sessions to track performance improvements. ## Next steps Learn how to [create and use sessions](/concepts/logging/sessions/using-sessions) in Galileo. ## Related resources * [**Using Sessions**](/concepts/logging/sessions/using-sessions) - Create and view sessions in Galileo * [**Log streams**](/sdk-api/logging/logging-basics) - Learn about Log streams in Galileo. * [**Spans**](/sdk-api/logging/galileo-logger#add-spans) - Learn about the building blocks of Traces in Galileo. * [**Traces**](/sdk-api/logging/galileo-logger#start-a-trace) - Learn about Traces, and different ways to create them. # Create and Use Sessions Source: https://docs.galileo.ai/concepts/logging/sessions/using-sessions Learn to create and use Sessions in Galileo ## Overview This tutorial will guide you through creating and using a [Session](/concepts/logging/sessions/sessions-overview) in Galileo, using a simple LLM-driven example that you can expand to multiple agents and data sources. It is a quick way to introduce you to logging sessions. By the end of this guide, you will know how to: 1. Initialize a [logging session](/concepts/logging/sessions/sessions-overview) 2. Add events to your session 3. Inspect the Session in the Galileo Console to see all related Traces and Spans. ```mermaid theme={null} --- config: flowchart: curve: linear --- graph TD; __start__([Initialize a Session]) app_logic(Run LLM logic) log_traces([Galileo captures Traces and Spans]) flush([Flush the context]) __end__([

View Session in Galileo Console

]) __start__ --> app_logic app_logic -.-> log_traces app_logic --> flush flush --> __end__ ``` There will be minor differences around starting and flushing the session context, depending on whether you're using the automatic or manual way. We'll cover both below. ## Prerequisites * **Galileo Account**: Ensure you have signed up for a Galileo account. This should provide you with the following values: * `GALILEO_API_KEY`: Your API key * `GALILEO_PROJECT`: The name of your Galileo Project * `GALILEO_LOG_STREAM`: The Log stream where you will save your sessions * `GALILEO_CONSOLE_URL`: Optional. The URL of your Galileo console for custom deployments. If you are using `app.galileo.ai`, you don't need to set this. * **OpenAI API Key**: This example will use OpenAI as the underlying LLM, so you will need an API key from them. In addition, this tutorial assumes you are familiar with: * Simple LLM Apps, and making simple OpenAI completion calls using Python or TypeScript * The [`GalileoLogger`](/sdk-api/logging/galileo-logger) class from the Python or TypeScript SDK ## Project setup Let's take a moment to prepare the development environment. If you already have a project setup with `Galileo`, `LangChain`, and `LangGraph`, you can skip right to [Manage a Session](#manage-a-session). If not, here's an abbreviated quickstart: We'll need the Galileo [Python](/sdk-api/python/sdk-reference) or [TypeScript](/sdk-api/python/sdk-reference) SDK, LangChain, LangGraph, OpenAI, and `dotenv` to pull in variables from your `.env` file. Let's start by installing them: ```bash Python theme={null} pip install "galileo[openai]" langchain langchain-openai langgraph python-dotenv ``` ```bash TypeScript theme={null} npm i -s galileo openai @langchain/langgraph @langchain/core dotenv ``` Next, create a `.env` file and add in the following variables: ```ini .env theme={null} # Your Galileo API key GALILEO_API_KEY="your-galileo-api-key" # Your Galileo project name GALILEO_PROJECT="your-galileo-project-name" # The name of the Log stream you want to use for logging GALILEO_LOG_STREAM="your-galileo-log-stream" # Provide the console url below if you are using a # custom deployment, and not using the free tier, or app.galileo.ai. # This will look something like “console.galileo.yourcompany.com”. # GALILEO_CONSOLE_URL="your-galileo-console-url" # OpenAI properties OPENAI_API_KEY="your-openai-api-key" # Optional. The base URL of your OpenAI deployment. # Leave this commented out if you are using the default OpenAI API. # OPENAI_BASE_URL="your-openai-base-url-here" # Optional. Your OpenAI organization. # OPENAI_ORGANIZATION="your-openai-organization-here" ``` Finally, create a main script file (e.g. `main.py` or `main.ts`) where you'll add and run your application logic. Now we can dive in. ## Manage a session Recall our objectives from earlier? We'll build a simple application and use it to work through each step. If you're in a hurry, you jump to the [full code sample here](#full-code-sample), then return to see how it was put together. ### Steps In your main script, import the following dependencies. Let's begin by creating a very simple agent using LangGraph and OpenAI: ```python Python {9,17-22} theme={null} from time import time from dotenv import load_dotenv # Galileo dependencies from galileo import GalileoLogger from galileo.handlers.langchain import GalileoCallback # LangChain and LangGraph dependencies from langchain.agents import create_agent from langchain_core.runnables.config import RunnableConfig # Load `.env` variables load_dotenv() # Create a simple assistant for our test (or import one). You can also provide # your agent with tools: the session will log their usage simple_agent = create_agent( name="simple_agent", model="openai:o3-mini", # you can choose any OpenAI model here system_prompt="You are a friendly assistant that answers the user's questions", tools=[], # (OPTIONAL) provide tools to your agent ) ``` ```typescript TypeScript {7,9,16-22} theme={null} import { configDotenv } from "dotenv"; // Galileo dependencies import { GalileoCallback, GalileoLogger } from "galileo"; // LangChain and LangGraph dependencies import { createReactAgent } from "@langchain/langgraph/prebuilt"; import { RunnableConfig } from "@langchain/core/runnables"; import { ChatOpenAI } from "@langchain/openai"; // Load environment variables configDotenv(); // Create a simple assistant for our test (or import one). You can also provide // your agent with tools: the session will log their usage const simpleAgent = createReactAgent({ name: "simpleAssistant", llm: new ChatOpenAI({ model: "o3-mini" }), prompt: "You are a friendly assistant that answers the user's questions", tools: [] // (OPTIONAL) provide tools to your agent }); ``` We'll see `GalileoCallback` and `RunnableConfig` in action later. For now, let's move on to the next step. We'll be using the `GalileoLogger` to manage our logging session. Let's create one next: ```python Python theme={null} # Create a GalileoLogger instance for our session logger = GalileoLogger() ``` ```typescript TypeScript theme={null} // Create a GalileoLogger instance for our session const logger = new GalileoLogger(); ``` `GalileoLogger` takes some optional arguments: you don't have to provide any of them, but they are listed below so that you can see what is available. ```python Python theme={null} GalileoLogger( project_name: Optional[str] """ name of target project for the logger instance """ log_stream_name: Optional[str] """ name of target logstream for the logger instance """ project_id: Optional[str] """ ID of target project for the logger instance """ log_stream_id: Optional[str] """ ID of target logstream for the logger instance """ experiment_id: Optional[str] """ ID of an experiment to which this log session will be linked """ session_id: Optional[str] """ ID of a previous session to which this log session will be linked """ local_metrics: Optional[list[LocalMetricConfig]] """ Locally-defined metrics that should be used on spans/traces from this session: See 'Custom Metrics' for more information """ ) ``` ```typescript TypeScript theme={null} new GalileoLogger({ /** name of target project for the logger instance */ projectName?: string; /** name of target logstream for the logger instance */ logStreamName?: string; /** ID of an experiment to which this log session will be linked */ projectId?: string; /** ID of target project for the logger instance */ logStreamId?: string; /** ID of target logstream for the logger instance */ experimentId?: string; /** ID of a previous session to which this log session will be linked */ sessionId?: string; /** Locally-defined metrics that should be used on spans/traces from this session: See 'Custom Metrics' for more information */ localMetrics?: LocalMetricConfig[]; /** Logger mode: "batch" or "streaming". Defaults to "batch" if not set. - "batch": Batches traces and sends on flush() (default) - "streaming": Enables streaming tracing with immediate updates to backend */ mode?: string; /** Optional callback invoked on `flush()` in batch mode with the payload that would be sent to the API. When set, it runs instead of the default `ingestTraces` call—use for custom delivery. */ ingestionHook?: (request: LogTracesIngestRequest) => Promise | void; }) ``` Our simple application will have a `main` function where everything happens. The first thing we will do in this function is start up a logging session. This will prepare the logger to group all captured events under a single session. Below, we give the session a unique `name` and `external id`. The name helps us find the session more easily in the Galileo Console. The `external id` is to link this session to external tracing: for example, linking to a conversation ID in your chatbot app by an ID created inside that app. You can also pass an optional `metadata` dictionary of string key-value pairs to attach structured information to the session, such as customer IDs, environment names, or application versions. Metadata keys appear as filterable columns in the Sessions table in the Galileo Console. ```python Python {7-14} theme={null} def main(): """Main application logic""" # start a logging session external_id = f"custom_id-{int(time())}" logger.start_session( name="Logger Session Tutorial", external_id=external_id, metadata={ "brand_id": "acme", "environment": "production", }, ) ``` ```typescript TypeScript {5-12} theme={null} /** Main application logic */ async function main() { // Start a logging session const externalId = `custom_id-${Math.round(Date.now() / 1000)}`; await logger.startSession({ name: "Logger Session Tutorial", externalId, metadata: { brand_id: "acme", environment: "production", }, }); } ``` Treat `logger.start_session` like a lifecycle event, and call it before any code you want to monitor. The `name`, `external id`, and `metadata` arguments are all optional; `name` and `external id` are recommended. Now you can interact with your LLM. Our very simple application will invoke the LLM with two questions: each question will be a question/answer exchange that generates a `Trace` with child spans in our session. We will also pass a callback handler, which will be called by `LangChain` after each LLM invocation. Here's our full `main` function: you can make this part as complex as you like! ```python Python {9-12, 17, 19-28} theme={null} def main(): """Main application logic""" # start a logging session external_id = f"custom_id-{int(time())}" logger.start_session(name="Logger Session Tutorial", external_id=external_id) # Here's what we will ask the LLM: prompts = [ "Hello! How many minutes are in a year?", "Hello! How far is an Astronomical Unit in kilometers?", ] # Create a LangChain Runnable config object with a LangGraph callback handler: # We will supply the logger instance to ensure that it generates traces in the # correct session agent_config = RunnableConfig(callbacks=[GalileoCallback(galileo_logger=logger)]) for prompt in prompts: # Invoke the LLM with our question: response = simple_agent.invoke( input={"messages": [{"role": "user", "content": prompt}]}, config=agent_config, # pass the RunnableConfig here ) # Print out the LLM's response to confirm that this code block ran: print("Model response:", response["messages"][-1].content.strip()) ``` ```typescript TypeScript {8-11, 16-18, 20-29} theme={null} /** Main application logic */ async function main() { // Start a logging session const externalId = `custom_id-${Math.round(Date.now() / 1000)}`; await logger.startSession({ name: "Logger Session Tutorial", externalId }); // Here's what we will ask the LLM: const prompts = [ "Hello! How many minutes are in a year?", "Hello! How far is an Astronomical Unit in kilometers?" ]; // Create a LangChain Runnable config object with a LangGraph callback handler. // We will supply the logger instance to ensure that it generates traces in the // correct session const agentConfig: RunnableConfig = { callbacks: [new GalileoCallback(logger)] }; for (const prompt of prompts) { // Invoke the LLM with our question: const result = await simpleAgent.invoke( { messages: [{ role: "user", content: prompt }] }, agentConfig // pass the RunnableConfig here ); // Print out the LLM's response to confirm that this code block ran: console.log("LLM Reply:", result.messages.at(-1)?.content); } } ``` ### The GalileoCallback handler `GalileoCallback` is a callback handler specifically for `LangChain`. It sends the most-recent captured traces to Galileo Console when it is called behind the scenes: your LLM logic determines what traces are generated and/or captured. `GalileoCallback` has a few optional parameters: ```python Python theme={null} GalileoCallback( galileo_logger: Optional[GalileoLogger] = None, """ A `GalileoLogger` instance. Defaults to a global singleton """ start_new_trace: bool = True, """ Start a new trace on next invocation. Defaults to "true" """ flush_on_chain_end: bool = True, """ Flush captured traces after invocation. Defaults to "true" """ ) ``` ```typescript TypeScript theme={null} new GalileoCallback( /** A `GalileoLogger` instance. Defaults to a global singleton */ galileoLogger?: GalileoLogger, /** Start a new trace on next invocation. Defaults to "true" */ startNewTrace?: boolean, /** Flush captured traces after invocation. Defaults to "true" */ flushOnChainEnd?: boolean ) ``` ### Full code sample Here's everything we have done so far: ```python Python theme={null} from time import time from dotenv import load_dotenv # Galileo dependencies from galileo import GalileoLogger from galileo.handlers.langchain import GalileoCallback # LangChain and LangGraph dependencies from langchain.agents import create_agent from langchain_core.runnables.config import RunnableConfig # Load `.env` variables load_dotenv() # Create a simple assistant for our test (or import one). You can also provide # your agent with tools: the session will log their usage simple_agent = create_agent( name="simple_agent", model="openai:o3-mini", # you can choose any OpenAI model here system_prompt="You are a friendly assistant that answers the user's questions", tools=[], # (OPTIONAL) provide tools to your agent ) # Create a GalileoLogger instance logger = GalileoLogger() def main(): """Main application logic""" # start a logging session external_id = f"custom_id-{int(time())}" logger.start_session(name="Logger Session Tutorial", external_id=external_id) # Here's what we will ask the LLM: prompts = [ "Hello! How many minutes are in a year?", "Hello! How far is an Astronomical Unit in kilometers?", ] # Create a LangChain Runnable config object with a LangGraph callback handler: # We will supply the logger instance to ensure that it generates traces in the # correct session agent_config = RunnableConfig(callbacks=[GalileoCallback(galileo_logger=logger)]) for prompt in prompts: # Invoke the LLM with our question: response = simple_agent.invoke( input={"messages": [{"role": "user", "content": prompt}]}, config=agent_config, # pass the RunnableConfig here ) # Print out the LLM's response to confirm that this code block ran: print("Model response:", response["messages"][-1].content.strip()) if __name__ == "__main__": main() ``` ```typescript TypeScript theme={null} import { configDotenv } from "dotenv"; // Galileo dependencies import { GalileoCallback, GalileoLogger } from "galileo"; // LangChain and LangGraph dependencies import { createReactAgent } from "@langchain/langgraph/prebuilt"; import { RunnableConfig } from "@langchain/core/runnables"; import { ChatOpenAI } from "@langchain/openai"; // Load environment variables configDotenv(); // Create a simple assistant for our test (or import one). You can also provide // your agent with tools: the session will log their usage const simpleAgent = createReactAgent({ name: "simpleAgent", llm: new ChatOpenAI({ model: "o3-mini" }), prompt: "You are a friendly assistant that answers the user's questions", tools: [] // (OPTIONAL) provide tools to your agent }); // Create a GalileoLogger instance for our session const logger = new GalileoLogger(); /** Main application logic */ async function main() { // Start a logging session const externalId = `custom_id-${Math.round(Date.now() / 1000)}`; await logger.startSession({ name: "Logger Session Tutorial", externalId }); // Here's what we will ask the LLM: const prompts = [ "Hello! How many minutes are in a year?", "Hello! How far is an Astronomical Unit in kilometers?" ]; // Create a LangChain Runnable config object with a LangGraph callback handler. // We will supply the logger instance to ensure that it generates traces in the // correct session const agentConfig: RunnableConfig = { callbacks: [new GalileoCallback(logger)] }; for (const prompt of prompts) { // Invoke the LLM with our question: const result = await simpleAgent.invoke( { messages: [{ role: "user", content: prompt }] }, agentConfig // pass the RunnableConfig here ); // Print out the LLM's response to confirm that this code block ran: console.log("LLM Reply:", result.messages.at(-1)?.content); } } main(); ``` ### Run your script That's all the code: we have now learned to use `logger.start_session` before starting LLM chat session, and supply `GalileoCallback` to ensure your traces get sent to the Galileo Console. Now let's run the script: ```bash Python theme={null} python main.py ``` ```bash TypeScript theme={null} npx tsx main.ts ``` You should see the LLM's response in your terminal! You can also head to the [Galileo Console](https://app.galileo.ai) to view the newly-created session. (Shown below) ## View your session Now that you've logged a session, it's time to view results. Head over to the [Galileo Console](https://app.galileo.ai) and log in. On your dashboard, select the Log stream where you were sending your session logs. If you didn't specify a unique or new Log stream name, you will find the logs in your **default** Log stream. Select your Log stream from the list Selecting the Log stream will bring you to its event records. All logs will be grouped by *Session*, though you can use the control near the top-left of your screen to change the Log stream's event grouping: Event-group controls for selecting Session, Trace, or Span granularity Your session should be visible in the table below the controls, especially if you gave it a recognizable name. Select it to view the traces. Once you select your session, you can see the Traces you captured from your test run as a flowchart. Any tools that were used will also show up as individual Spans. Select the nodes of the flowchart to see their inputs and outputs on the right-edge of your screen. A flowchart showing the nodes in a session Trace Each message from the user and response from the LLM will form a single trace; you can view the contents here in a familiar format, as well as other details like tool calls. Just select the **Messages** tab (shown in the image below) to see a list of traces in the session, along with their child spans. You can select a span to see metrics and other details on the right edge of the screen (not pictured) View Trace messages Use the **Condense Steps** toggle to show only the most relevant spans in a trace. This will include any tool calls made by the LLM! ## Additional considerations Remember to always use the same `GalileoLogger` instance across your project. This ensures that all captured events are placed in the same session. You can achieve this in a few ways: 1. Export your `logger` instance from a separate module, so that your application uses a singleton instance. 2. Use the TypeScript SDK's `getLogger` function, or the Python SDK's `galileo_context` context manager for a consistent reference: ```python Python theme={null} from galileo import galileo_context # Create a new session (with optional metadata) galileo_context.start_session( name="My Session", metadata={"brand_id": "acme", "environment": "production"}, ) # Application logic follows # Flush the session (if you are not using galileo callback or "with galileo_context()") galileo_context.flush() ``` ```typescript TypeScript theme={null} import { getLogger } from 'galileo'; const logger = getLogger(); // Create a new session (with optional metadata) logger.startSession({ name: "My Session", metadata: { brand_id: "acme", environment: "production" }, }); // Application logic follows // Flush the session (if you are not using GalileoCallback) logger.flush() ``` 3. You can also add `Traces` wherever you see fit. A `Trace` might represent a question asked to your LLM, and the response generated for it — as well as any tools used! Galileo will generate traces for you, but you can also create new ones by using your logger instance: ```python Python theme={null} question = "What is the meaning of plenipotentiary?" logger.start_trace(input=question) logger.add_llm_span( input=question, output="Plenipotentiary means 'Invested with full power'" ) logger.conclude({}) # end the trace ``` ```typescript TypeScript theme={null} const question = "What is the meaning of plenipotentiary?" logger.startTrace({ input: question }) logger.addLlmSpan({ input: question, output: "Plenipotentiary means 'Invested with full power'" }) logger.conclude({}); // end the trace ``` You can learn more about traces and how to use them [in our logging guide](/sdk-api/logging/galileo-logger#start-a-trace). ## Conclusion In this tutorial, you learned how to: 1. Create a logging session with the `GalileoLogger` class 2. Manually start your own session with the `logger.start_session()` method 3. View your sessions in the Galileo Console. ## Next steps For a more detailed walkthrough of a multi-agent application, take a look at [Monitoring LangChain Agents with Galileo](/cookbooks/use-cases/agent-langchain). You can also learn more about using [Galileo's metrics](/concepts/metrics/overview) to gain more insight about your AI application. ## Related resources * [Sessions](/concepts/logging/sessions/sessions-overview) - An overview of sessions * [Galileo Context](/sdk-api/logging/galileo-context) - Learn about the Galileo Context Manager * [Monitoring LangChain Agents with Galileo](/cookbooks/use-cases/agent-langchain) - Follow this cookbook recipe to create and evaluate a multi-agent application. # Fine-Tuning Luna-2 Models Source: https://docs.galileo.ai/concepts/luna/fine-tuning Understand the requirements and process for fine-tuning Luna-2 models based off your real-world scenarios One big advantage of the Luna-2 model is the ability to fine-tune the model for your specific use case, either for out-of-the-box metrics or custom metrics. Most teams should start with [Luna Studio](/luna-studio), which provides a self-service workflow for creating, validating, fine-tuning, and registering Luna metrics. If you want Galileo to run the fine-tuning process with you, contact us. Fine-tune and register Luna metrics with a self-service workflow. Work with Galileo if you need support for a managed fine-tuning process. Use **Luna Studio** for self-service fine-tuning of out-of-the-box and custom evaluation metrics. Bring your own labelled test set, run a fine-tune in a wizard, and register the result without involving the Galileo team. ## Recommended adoption process Fine-tuning works best after you have validated that the metric objective, dataset, and judge prompt match your real-world use case. We recommend the following process after signing with Galileo: 1. **Start using Galileo** Instrument your application or evaluation workflow in Galileo so you can observe real traces, model outputs, and evaluation results. 2. **Log metrics** Capture the metrics that are already available to you, along with the inputs, outputs, context, and metadata needed to understand each result. 3. **Define the objectives you want to track** Decide what you need to measure, such as answer correctness, context adherence, safety, response style, retrieval quality, or another business-specific outcome. 4. **Identify the metrics that support each objective** Map each objective to the most relevant Galileo metric. The right metric may be an out-of-the-box LLM-as-a-judge metric, or it may require a custom LLM-as-a-judge prompt. 5. **Create a labelled test dataset** Label your logs or create a dedicated test dataset to measure how well the out-of-the-box or custom metric performs. This dataset becomes the benchmark for prompt iteration and later Luna fine-tuning. 6. **Iterate on the LLM-as-a-judge prompt** Review metric performance on your dataset and update the judge prompt until it reliably captures the objective you care about. 7. **Validate on real logs** After the metric performs well on the test dataset, run it on actual logs and inspect the results in production-like conditions. 8. **Identify candidates for Luna fine-tuning** Once the metric is validated on real logs, choose the strongest candidates for Luna fine-tuning based on usage volume, latency requirements, and LLM-as-a-judge cost. 9. **Proceed to Luna fine-tuning** Fine-tune the validated prompt with Luna Studio, or work with Galileo if you need a managed fine-tuning process. This is a time-consuming, iterative process. Invest heavily in objective definition, dataset quality, prompt iteration, and real-log validation before fine-tuning; these steps determine the quality of the resulting Luna metric. ## Requirements for fine-tuning process After you have defined the metric, validated the objective, and finalized the LLM-as-a-judge prompt, gather the inputs needed for fine-tuning: 1. **A labelled test dataset** The test dataset is the most crucial piece for any fine-tuning work. 300-500 samples is a strong target for a diverse test set, with at least 100 samples of each class. More data leads to more reliable evaluation and better fine-tuning outcomes. The test set **must be manually labelled** to ensure high quality. The format can be a spreadsheet/csv with input, output, label and explanations on the label if possible. If you are already using a Galileo metric on the data, these numbers will also help. 2. **Latency and load requirements** Specify the maximum acceptable latency for the given metric and its use case (online observability, run time protection etc.). The latency requirements should include QPS and expected input token size. Both these numbers should be provided for average and peak loads. This requirement will influence the choice of flow and may necessitate trade-offs with other factors. 3. **Constraints** Identify any limitations or restrictions that may impact the design or implementation of the flow. These constraints could include technical, resource, or regulatory limitations. ## Approaches There are 3 different approaches to Luna fine-tuning, depending on the metric you are interested in and your dataset. | Approach | When to use | | :--------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Use a preset SLM metric | The required metric aligns in definition with a metric offered by Galileo. Accuracy on your dataset is good enough. | | Fine-tune preset SLM metric | The required metric aligns in definition with a metric offered by Galileo. LLM-as-a-judge variant of the metric works well (with or without CLHF). Preset SLM variant performance is poor. | | Create a new customized SLM metric | The required metric is not offered by Galileo out-of-the-box. | ```mermaid theme={null} flowchart TD A([Is required metric offered by Galileo?]) -->|Yes| B([Is Preset SLM accurate enough?]) A -->|No| C[Use approach 3 - create custom SLM metric] B -->|Yes| D[Use approach 1 - enable preset SLM metric] B -->|No| E[Use approach 2 - fine-tune Preset SLM] ``` ## Fine-tuning requirements After working through the above approaches, if fine-tuning is the best approach, you will need a training dataset. * For managed fine-tuning, plan for around 4,000 total labeled samples, with a 50/50 split amongst the classes (e.g. 2,000 context adherent samples, 2,000 non-context adherent samples). * If you are unable to procure this many labeled samples, Galileo can synthetically extrapolate from the limited test set you share using LLMs approved by you. More source data is better: a larger, more diverse test set gives the synthetic generation process stronger examples to learn from. Description of model(s) used for synthetic data generation would then be explained in a generalized model card. These targets apply to Galileo-managed fine-tuning. In Luna Studio's self-service flow, **Generate from test set** produces roughly 2,000 labelled training examples by default (upsampled from about 20% of your test set); you can supply more by uploading your own logs or reusing an existing training set. ## Turnaround time Turnaround time depends on whether you use Luna Studio yourself or ask Galileo to manage the fine-tuning process. ### Luna Studio With Luna Studio, the pace is controlled by your team. You can iterate as quickly as you can prepare data, evaluate the LLM-as-a-judge prompt, run fine-tuning, and validate the resulting Luna metric. The fine-tuning and registration steps themselves are fast (minutes to a few hours). The long pole is almost always **curating a high-quality, human-labelled test set**, which depends on your labelling capacity and subject-matter-expert availability rather than on Galileo. Plan backward from your target go-live date: * **Build the labelled test set** — start this earliest; it can take days to several weeks. * **Build and validate the LLM-as-a-judge prompt** — hours to about a week, depending on how many prompt or Autotune iterations you need. * **Check any existing Luna metric** against the test set — minutes to hours. * **Fine-tune in Luna Studio** — hours to a couple of days, depending on how many runs you need to pass your quality bar. * **Register, test, and roll out** — hours to days of validation before production. If you do not yet have a labelled golden set, assume the end-to-end effort is measured in weeks and is dominated by test-set curation. If a representative labelled test set already exists, it can compress to days. ### Galileo-managed fine-tuning If Galileo performs the fine-tuning work with you, plan for **1 week** of fine-tuning time after all prerequisites are complete: * The objective for the new metric is clearly understood * The test dataset is ready and quality checked * Latency, load, and deployment requirements are defined This estimate applies whether the metric starts from an existing metric or a new custom metric. Galileo-managed fine-tuning time does not include deployment time. ### Model deployment To deploy your model, ensure: * The new fine-tuned model is approved for use internally * The model is integrated into the Galileo cluster by the applied data science team The estimated timings are: * **Replace existing metric**: Deployment can be done in **1-2 days** * **New custom metric**: This is more involved, the time to completion would be defined on a case-by-case basis ## Fine-tune it yourself with Luna Studio Luna Studio supports self-service fine-tuning for both out-of-the-box and custom metrics. Pick a base model, supply a labelled test set, generate or upload a training set, and register the resulting metric to the Galileo metrics store. End-to-end walkthrough — sign up, configure an integration, and register your first custom metric in about 15 minutes. Projects, runs, datasets, and base models — how the pieces fit together. ## FAQ 300-500 labelled samples is a strong target because it gives you enough coverage to evaluate metric performance across common and edge-case behavior. You can start with fewer samples, but the results will be less reliable. The more labelled data you provide, the better your evaluation coverage and fine-tuning outcomes will be. Yes, the test dataset used for final evaluation should be human labelled. Luna is only as good as the objective and labels used to evaluate it, so human labels are the best source of truth for deciding whether the metric is actually measuring what you care about. Synthetic labels can help generate training data, but they should not replace a human-labelled test set for final evaluation. No. Luna will be almost as good as the LLM-as-a-judge metric it is trained from, while running at lower latency and lower cost. Validate the LLM-as-a-judge metric first, then compare Luna against your human-labelled test dataset and real logs before replacing the original judge. As a practical benchmark, aim for the fine-tuned Luna metric's F1 score to land within about 5 points of your validated LLM-as-a-judge's F1 score on the same test set. A larger gap usually points to a test set, judge prompt, or training set that needs more work before the metric is production-ready. # Luna-2 Overview Source: https://docs.galileo.ai/concepts/luna/luna Discover Galileo's Luna-2 Evaluation model, reducing the latency and cost for metric evaluations **Luna-2** is the latest generation of our Luna small language models (SLMs), purpose built for scaling AI evaluations. Luna-2 models are fine tuned to provide low latency and reduced costs for metric evaluations. Luna-2 is designed to be further fine tuned for your specific use cases and custom metrics with the goal of providing scalable, real-time, customizable evaluations for enterprises. Luna-based metrics offer highly accurate and efficient evaluations for AI applications, particularly those with agentic workflows. Luna-2 is only available in the Enterprise tier of Galileo. [Contact us](https://galileo.ai/contact-sales) to learn more and get started. Want to fine-tune a Luna-2 metric for your own use case without writing training code? See [Luna Studio](/luna-studio) — Galileo's web app for self-service custom metric fine-tuning. Contact us to learn more about using Luna-2 in your evaluations Learn how Galileo pushes the envelope on GenAI evaluation with our family of fine tuned small language models. ## Overview LLMs are powerful judges for evaluations, but as your application scales up to from tens or hundreds of traces a day, to thousands or millions, they can fall short. Too often, organizations relying solely on LLMs to act as judges incur major inference costs and don't see the low-latency they need to enable real-time evaluations and runtime protection. * LLMs are expensive * LLMs don't provide the performance needed, especially for runtime protection * LLMs are general purpose, and even leveraging [Autotune](/concepts/metrics/autotune-llm-as-a-judge-metrics) to enhance the evaluation prompts, can still be less effective for your specific needs. The Luna-2 model mitigates these issues: * Being an SLM, it is an **order of magnitude cheaper** to run than most LLMs * SLMs run an **order of magnitude faster**, allowing for **runtime protection** * Luna-2 is not only **fine-tuned for evaluations**, giving comparable performance out of the box with the top LLMs, but it can be **further fine-tuned using your data** to improve accuracy beyond any general purpose LLM. The Luna-2 model works with most of the [out of the box metrics](/sdk-api/metrics/metrics#luna-metrics), or your [LLM-as-a-judge custom metrics](/concepts/metrics/custom-metrics/custom-metrics-ui-llm). ## Performance and cost comparison ### Comparison with different LLMs and content safety tools | Model | Cost/1M token | Accuracy (F1 score) | Latency (avg) | Max tokens | | :------------------- | ------------: | ------------------: | ------------: | ---------: | | **Luna-2** | **\$0.02** | **0.95** | **152ms** | **128k** | | GPT 4o | \$2.50 | 0.94 | 3,200ms | 128k | | GPT 4o mini | \$0.60 | 0.90 | 2,600ms | 128k | | Azure Content Safety | \$1.52 | 0.62 | 312ms | 3k | ### Latency vs compute requirements These are the measured latencies for Luna-2 across a range of GPUs for different sized requests. #### H100/H200 GPU | Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) | | :-------- | -----------------: | -----------------: | -----------------: | ------------------------: | | Luna-2 3B | 15ms | 15ms | 141ms | 2.8s | | Luna-2 8B | 16ms | 30ms | 277ms | 4.71s | #### RTX PRO 6000 GPU | Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) | | :-------- | -----------------: | -----------------: | -----------------: | ------------------------: | | Luna-2 3B | 17ms | 32ms | 245ms | 4.8s | | Luna-2 8B | 28ms | 61ms | 514ms | 8.05s | #### B200 GPU | Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) | | :-------- | -----------------: | -----------------: | -----------------: | ------------------------: | | Luna-2 3B | 15ms | 16ms | 81ms | 1.37s | | Luna-2 8B | 15ms | 19ms | 146ms | 2.24s | #### A100 GPU | Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) | | :-------- | -----------------: | -----------------: | -----------------: | ------------------------: | | Luna-2 3B | 27ms | 85ms | 750ms | 12.5s | | Luna-2 8B | 51ms | 177ms | 1.51s | 21.2s | #### L40S GPU | Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) | | :-------- | -----------------: | -----------------: | -----------------: | ------------------------: | | Luna-2 3B | 57ms | 91ms | 491ms | 8.06s | | Luna-2 8B | 86ms | 163ms | 1.01s | 14.03s | #### L4 GPU L4 GPUs are only supported for calculating metrics for Log streams and experiments. These GPUs are not supported for runtime protection. | Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) | | :-------- | -----------------: | -----------------: | -----------------: | ------------------------: | | Luna-2 3B | 51ms | 155ms | 1.66s | 29.45s | | Luna-2 8B | 126ms | 364ms | 3.35s | 50.78s | The actual latencies can vary a lot based upon the load on the system (Eg: QPS). This can be managed with more GPUs, but the cost will increase. ## Technical details Galileo's Luna-2 metrics utilize fine-tuned Llama models (3B and 8B variants) in evaluating generative AI metrics. The technical process involves: * **Fine-Tuning:** Base Llama models are fine-tuned with proprietary data for specific metric needs. * **Classification:** Models output normalized log-probabilities of True/False tokens to determine metric accuracy. * **Optimized Infrastructure:** Metrics are hosted on Galileo's optimized inference engine with modern GPU hardware for low-latency and cost-effective evaluations. You can also self host on-prem or on your cloud infrastructure. * **Adapters for Custom Metrics:** Lightweight adapters on a shared base model enhance scalability and minimize infrastructure overhead for additional metrics. By leveraging fine-tuned Llama models, Luna-2 metrics provide significant enhancements over traditional methods: Luna evaluation models are fine-tuned on open-source base models, including but not limited to Llama and Mistral. Where applicable, third-party license terms apply — for example, Llama is licensed under the [Meta Llama Community License](https://llama.meta.com/llama3/license), Copyright (c) Meta Platforms, Inc. All Rights Reserved. * **Adaptability:** These models are most effective when fine tuned, requiring approximately 4,000 samples for fine-tuning to customer-specific use cases. * **Efficiency and Cost-Effectiveness:** Luna-2 models enable simultaneous evaluation of multiple metrics with low latency and reduced costs, ideal for real-time, high-scale deployments. * **Enhanced Accuracy:** Luna-2 demonstrates at least a 10% accuracy increase compared to traditional BERT-based models, perfect for precise monitoring in production environments. ## Get started with Luna-2 If you are using the enterprise tier of Galileo, follow these steps to use Galileo's Luna-based metrics: 1. [Contact Galileo's customer support or account management](https://galileo.ai/contact-sales) to begin onboarding. 2. If you are using a Galileo-hosted instance, request L4 GPUs or higher, necessary for running Luna-2 models. Otherwise you can deploy to your own infrastructure, using L4 or higher GPUs. 3. Review the provided documentation and model cards for details on latency, accuracy, and comparisons to BERT-based metrics. 4. Provide Galileo with relevant labelled sample data to fine tune the model. We can augment this with synthetic data if needed. 5. Galileo will fine tune your model for you, and deploy it. 6. Set up your experiments and Log streams to use [Luna-based metrics](/sdk-api/metrics/metrics#luna-metrics). This is not a one-shot process. Your model can be tuned on a regular basis as required. ## Next steps Contact us to learn more about using Luna-2 in your evaluations Learn how to evaluate metrics cheaper and faster using the Luna-2 model Learn how to use Luna-2 metrics when running experiments in code Use Luna Studio to fine-tune Luna-2 on your data, no code required. # Luna Metrics Comparison Source: https://docs.galileo.ai/concepts/luna/luna-metric-comparison Explore Galileo's preset Luna metrics for evaluating and improving AI system performance across multiple dimensions Galileo provides a comprehensive suite of preset Luna metrics designed to evaluate various aspects of AI system performance without requiring custom implementation. These metrics span across the following categories: * [Agentic Performance Metrics](/concepts/metrics/agentic/agentic-overview) * [Expression And Readability Metrics](/concepts/metrics/expression-and-readability/expression-and-readability-overview) * [RAG Metrics](/concepts/metrics/rag/rag-overview) * [Safety And Compliance Metrics](/concepts/metrics/safety-and-compliance/safety-and-compliance-overview) Each metric addresses specific evaluation needs, from measuring factual correctness to detecting potential biases or tracking tool usage effectiveness. These metrics apply to different node types (such as session, trace, or different span types), depending on the metric. Use the sortable, filterable table below to explore all available native metrics and find the right measurements for your AI applications. Hover over an icon in the **Modalities** column to see what it represents. # Action Advancement Source: https://docs.galileo.ai/concepts/metrics/agentic/action-advancement Understand how to measure and optimize the effectiveness of your AI agent's actions ## Overview Action Advancement measures whether an assistant successfully accomplishes or makes progress toward at least one user goal in a conversation. Action Advancement addresses the common pain points of unclear agent performance by measuring whether AI agents are actually helping users achieve their objectives rather than just providing responses. An assistant successfully advances a user's goal when it: 1. Provides a complete or partial answer to the user's question 2. Requests clarification or additional information to better understand the user's needs 3. Confirms that a requested action has been successfully completed For an interaction to count as advancing the user's goal, the assistant's response must be: * Factually accurate * Directly addressing the user's request * Consistent with any tool outputs used ### Action Advancement at a glance | Property | Description | | :----------------------------- | :---------------------------------------------------------------------------- | | **Name of Metric** | Action Advancement | | **Metric Category** | Agentic Metrics | | **Use this metric for** | Evaluating whether AI agents make progress toward user goals in conversations | | **Can be applied to** | session, trace, all span types (agent, workflow, retriever, LLM and tool) | | **LLM/Luna Support** | Supported with both LLM + Luna models | | **Protect Runtime Protection** | No - Not applicable for this metric | | **Constants** | None - Uses dynamic evaluation | | **Usage Context** | Agentic workflows, multi-step tasks, tool-using assistants | | **Value Type** | Boolean | | **Input/Output Requirements** | Requires conversation context, user goals, and assistant responses | ## When to Use This Metric

When to Use This Metric

This metric shines when simple response quality metrics fall short, particularly for complex, multi-step interactions where progress toward goals matters more than individual response quality.
Agentic Workflows: When an AI agent must decide on actions and select appropriate tools.
Multi-step Tasks: When completing a user's request requires multiple steps or decisions.
Tool-using Assistants: When evaluating if the assistant used available tools effectively.
Customer Service Agents: Resolving user issues through multi-step problem-solving.
Task-Oriented Assistants: Completing specific actions like booking flights or processing orders.
Research Assistants: Gathering and synthesizing information across multiple sources.
Creative Assistants: Understanding and building upon user requests iteratively.
### Calculation method If Action Advancement returns `false`, the assistant did not make progress on any user goal. Action Advancement is calculated by: Multiple evaluation requests are sent to an LLM evaluator to analyze the assistant's progress toward user goals. A specialized chain-of-thought prompt guides the model to evaluate whether the assistant made progress on user goals based on the metric's definition. Each evaluation analyzes the interaction and produces both a detailed explanation and a binary judgment (yes/no) on goal advancement. The evaluator returns `true` when the assistant advances at least one user goal and `false` otherwise. We display one of the generated explanations alongside the result, choosing one that aligns with the majority judgment. This metric requires multiple LLM calls to compute, which may impact usage and billing. ## How to improve Action Advancement scores To improve Action Advancement scores, focus on ensuring your AI agents make meaningful progress toward user goals in every interaction. ### Common issues and solutions | Issue | Cause | Solution | | :---------------------------------- | :----------------------------------------------- | :----------------------------------------------------------------------------------------------------------- | | **Assistant ignores user requests** | Poor prompt engineering or context understanding | Improve system prompts to emphasize goal-oriented responses and ensure the assistant understands user intent | | **Incomplete responses** | Insufficient context or tool usage | Provide better context and ensure the assistant uses available tools effectively | | **Irrelevant information** | Lack of focus on user goals | Train the assistant to stay focused on the specific user request and avoid tangential information | | **No progress on multi-step tasks** | Poor task breakdown | Implement better task decomposition and ensure the assistant can handle complex, multi-step processes | ### Best practices for optimization * **Clear goal identification:** Ensure your assistant can identify and prioritize user goals * **Progressive disclosure:** Break complex tasks into manageable steps * **Tool integration:** Make sure the assistant effectively uses available tools and APIs * **Context awareness:** Maintain conversation context to build on previous interactions ## Comparison to other metrics | Property | Action Advancement | Instruction Adherence | Completeness | | :----------------------------- | :---------------------------------------- | :----------------------------------------------- | :------------------------------- | | **Metric Category** | Agentic Metrics | Response Quality | Response Quality | | **Use this metric for** | Evaluating goal progress in conversations | Measuring how well responses follow instructions | Assessing response completeness | | **Best for** | Multi-step tasks and agentic workflows | Single-turn instruction following | Ensuring comprehensive responses | | **LLM/Luna Support** | Yes | Yes | Yes | | **Protect Runtime Protection** | No | No | No | | **Value Type** | Percentage (0.0-1.0) | Percentage (0.0-1.0) | Percentage (0.0-1.0) | | **Limitations** | Requires conversation context | May not capture goal progress | Doesn't measure goal advancement | ## Best practices To effectively implement and optimize Action Advancement in your AI systems, consider these key practices: ### Track progress over time Monitor Action Advancement scores across different versions of your agent to ensure improvements in task completion capabilities. This helps you identify whether your optimizations are actually improving goal advancement. ### Analyze failure patterns When Action Advancement scores are low, examine the specific steps where agents fail to make progress to identify systematic issues. Look for patterns in where agents get stuck or fail to advance user goals. ### Combine with other metrics Use Action Advancement alongside other agentic metrics to get a comprehensive view of your assistant's effectiveness. This provides a more complete picture of your agent's performance beyond just goal advancement. ### Test edge cases Create evaluation datasets that include complex, multi-step tasks to thoroughly assess your agent's ability to advance user goals. This ensures your agent can handle challenging scenarios that require multiple steps. When optimizing for Action Advancement, ensure you're not sacrificing other important aspects like safety, factual accuracy, or user experience in pursuit of task completion. ## Performance Benchmarks We evaluated Action Advancement against human expert labels on an internal dataset of agentic conversation samples using top frontier models. | Model | F1 (True) | | :---------------------- | :-------: | | GPT-4.1 | 0.87 | | GPT-4.1-mini (judges=3) | 0.78 | | Claude Sonnet 4.5 | 0.89 | | Gemini 3 Flash | 0.85 | ### GPT-4.1 Classification Report Benchmarks based on internal evaluation dataset. Performance may vary by use case. ## Related Resources If you would like to dive deeper or start implementing Action Advancement, check out the following resources: ### Examples * [Action Advancement Examples](https://app.galileo.ai) - Log in and explore the "Action Advancement" Log Stream in the "Preset Metric Examples" Project to see this metric in action. ### How-to guides * [Agentic AI Basic Example](/how-to-guides/agentic-ai/basic-example) * [Creating Custom Metrics](/how-to-guides/metrics/create-local-metric/create-local-metric) ### Related Concepts * [Agentic Metrics Overview](/concepts/metrics/agentic/agentic-overview) * [Action Completion](/concepts/metrics/agentic/action-completion) * [Agent Efficiency](/concepts/metrics/agentic/agent-efficiency) # Action Completion Source: https://docs.galileo.ai/concepts/metrics/agentic/action-completion Understand how to measure whether your agent accomplished a user's goals across a session ## Overview Action Completion determines whether the agent successfully accomplished all of the user’s goals in a session. Action Completion addresses the common pain points of agent performance by measuring whether AI agents are actually helping users achieve their end goal rather than just providing responses. Action Completion is successful when all of the below are true: : * The agent provides a complete response that addresses a question * The agent provides a confirmation of successful action in the case of a request * The response is coherent and factually accurate * The response comprehensively addresses every aspect of the user's request * The response avoids contradicting tool outputs * The response summarizes all relevant parts returned by tools ### Action Completion at a glance | Property | Description | | :----------------------------- | :-------------------------------------------------------------------------------- | | **Name of Metric** | Action Completion | | **Metric Category** | Agentic Metrics | | **Use this metric for** | Measuring whether the agent successfully accomplished the user's goal | | **Can be applied to** | Session | | **LLM/Luna Support** | Supported with both LLM + Luna models | | **Protect Runtime Protection** | No | | **Constants** | None - Uses dynamic evaluation | | **Usage Context** | Agentic workflows, multi-step tasks, tool-using assistants | | **Value Type** | Boolean | | **Input/Output Requirements** | Requires agent responses, user goals, and relevant session content for evaluation | ## Calculation method If Action Completion returns `false`, the agent did not accomplish every user goal. Multiple requests are sent to an LLM using a carefully designed chain-of-thought prompt that adheres to the definition above. The LLM generates multiple distinct responses, each containing: * An explanation * A final judgment: "Yes" (goal accomplished) or "No" (goal not accomplished) The evaluator returns `true` when the agent accomplishes every user goal and `false` otherwise. One explanation is surfaced, chosen to align with the majority judgment among the responses. Galileo displays a generated explanation alongside the result, choosing the one that aligns with the majority judgement for troubleshooting. This metric requires multiple LLM calls to compute, which may impact usage and billing. ## How to improve Action Completion scores To optimize your agent's performance and ensure high Action Completion scores, focus on comprehensive goal accomplishment and complete response generation. ### Common issues and solutions | Issue | Cause | Solution | | :------------------------- | :-------------------------------------------------- | :------------------------------------------------------------------------------------------------ | | Incomplete responses | Agent stops before addressing all user requirements | Implement comprehensive response generation and ensure all user goals are explicitly addressed | | Tool output contradictions | Agent ignores or contradicts information from tools | Ensure agent properly summarizes and incorporates all relevant tool outputs without contradiction | | Missing confirmations | Agent doesn't confirm successful actions | Add explicit confirmation steps for action-based requests | | Factual inaccuracies | Agent provides incorrect information | Implement fact-checking mechanisms and ensure responses align with tool outputs | ### Best practices for optimization * **Track Progress Over Time**: Monitor Action Completion scores across different versions of your agent to identify trends and ensure continuous improvements in task completion capabilities. * **Analyze Failure Patterns**: When Action Completion scores are low, examine specific steps or scenarios where agents fail to meet user goals. Use this analysis to identify and address systematic issues. * **Combine with Other Metrics**: Use Action Completion alongside other agentic metrics, such as Action Advancement, to get a comprehensive view of your assistant's effectiveness and identify areas for improvement. * **Test Edge Cases**: Create evaluation datasets that include complex, multi-step tasks to thoroughly assess your agent's ability to handle challenging scenarios and advance user goals effectively. When optimizing for Action Completion, ensure you're not sacrificing other important aspects like safety, factual accuracy, or user experience in pursuit of task completion. ## Comparison to other metrics | Property | Action Completion | Action Advancement | Tool Selection | | :----------------------------- | :---------------------------- | :------------------------------ | :-------------------------------- | | **Metric Category** | Agentic Performance | Agentic Performance | Agentic Performance | | **Use this metric for** | Measuring goal accomplishment | Measuring progress toward goals | Measuring tool choice quality | | **Best for** | Final outcome evaluation | Progress tracking | Tool usage optimization | | **LLM/Luna Support** | Yes | Yes | Yes | | **Protect Runtime Protection** | No | No | No | | **Value Type** | Boolean | Boolean | Boolean | | **Limitations** | Requires multiple LLM calls | May not capture final success | Doesn't measure execution quality | ## Performance Benchmarks We evaluated Action Completion against human expert labels on an internal dataset of agentic conversation samples using top frontier models. | Model | F1 (True) | | :---------------------- | :-------: | | GPT-4.1 | 0.92 | | GPT-4.1-mini (judges=3) | 0.79 | | Claude Sonnet 4.5 | 0.87 | | Gemini 3 Flash | 0.92 | ### GPT-4.1 Classification Report Benchmarks based on internal evaluation dataset. Performance may vary by use case. ## Related Resources If you would like to dive deeper or start implementing Action Completion, check out the following resources: ### Examples * [Action Completion Examples](https://app.galileo.ai) - Log in and explore the "Action Completion" Log Stream in the "Preset Metric Examples" Project to see this metric in action. ### How-to guides * [Agentic AI Examples](/how-to-guides/agentic-ai/basic-example) ### Related Concepts * [Action Advancement](/concepts/metrics/agentic/action-advancement) * [Tool Selection](/concepts/metrics/agentic/tool-selection-quality) * [Agentic AI Overview](/concepts/metrics/agentic/agentic-overview) # Agent Efficiency Source: https://docs.galileo.ai/concepts/metrics/agentic/agent-efficiency Learn how to measure the efficiency of your agentic workflows Agent Efficiency is a binary evaluation metric of the efficiency of your agentic workflows. An agentic session is considered efficient or optimal when the agent provides a precise answer or resolution to every user ask, with an efficient path. An ask could be a question that requires an answer, or a request that requires a resolution through tool usage. Efficiency here means the agent does not make redundant tool calls, ask redundant questions/clarifications to the user, is precise and concise in its communication, and reaches its goal in minimal steps needed. This is a **boolean** metric that indicates whether the agent is efficient. ## Agent Efficiency at a glance | Property | Description | | :----------------------------- | :--------------- | | **Name** | Agent Efficiency | | **Category** | Agentic AI | | **Can be applied to** | Session | | **LLM-as-a-judge Support** | ✅ | | **Luna Support** | ❌ | | **Protect Runtime Protection** | ❌ | | **Value Type** | Boolean | ## When to use this metric # Agent Flow Source: https://docs.galileo.ai/concepts/metrics/agentic/agent-flow Learn how to measure the correctness and coherence of an agentic trajectory by validating it against user-specified natural language tests Agent Flow is a binary metric that checks if an agent's behavior satisfies all user-defined conditions. Agent Flow is a binary evaluation metric that measures the correctness and coherence of an agentic trajectory by validating it against user-specified tests. A trajectory passes the Agent Flow metric if and only if all the user-defined conditions are successfully satisfied by the agent's realized behavior or output. To use this metric, you will need to create a copy and edit the prompt to provide your natural language tests. This is a **boolean** metric that indicates whether the agent flow satisfies all conditions. ## Agent Flow at a glance | Property | Description | | :----------------------------- | :---------- | | **Name** | Agent Flow | | **Category** | Agentic AI | | **Can be applied to** | Session | | **LLM-as-a-judge Support** | ✅ | | **Luna Support** | ❌ | | **Protect Runtime Protection** | ❌ | | **Value Type** | Boolean | ## When to use this metric ## Configure Agent Flow This metric needs to be manually customized to include your own natural language tests. From the **Metrics Hub**, select the **Agent Flow** metric. You will get a popup asking you to duplicate the metric. Select **Duplicate metric** to create a copy. The agent flow metric with the duplicate metric popup Locate the user defined tests section in the prompt. ```xml theme={null} {{ Add your tests here }} ``` This prompt needs to be customized based on your application, and the inputs and outputs you are expecting. Replace `{{ Add your tests here }}` with a numbered list of tests in natural language that can be used to evaluate the agent efficiency. This can include: * Expected tool or agent calls, using the tool or agent names * Conditions on tool or agent calling (e.g. if tool x is called, don't call agent y) * Expectations around the input or output parameters to tools and agents * Limitations on the number of tool or agent calls For example, imagine you were creating an agent to provide advice on exercises for different body parts, such as for a physical therapy application. This has multiple tools, including `list_by_target_muscle_for_exercised`, `list_by_body_part_for_exercised`, `list_of_bodyparts_for_exercised`. Some user tests might be: ```output wrap theme={null} 1. If a call to "list_by_target_muscle_for_exercised" returns an error that contains the text "target not found", the agent should subsequently attempt an alternative lookup by calling either "list_by_body_part_for_exercised" or "list_of_bodyparts_for_exercised" 2. When the user asks for exercises that target leg muscles, the agent must call at least one of the tools ["list_by_target_muscle_for_exercised", "list_by_body_part_for_exercised"] during the conversation 3. After receiving a successful response from "list_by_body_part_for_exercised", the agent's following natural-language message must contain at least one exercise name, the corresponding equipment, and an animated demonstration URL taken from the tool output 4. Every invocation of the tool "list_by_body_part_for_exercised" must include the required parameter "bodypart" 5. After receiving data from list_by_body_part_for_exercised, the agent response must include the exercise id for every exercise it presents to the user 6. No assistant message should include more than one tool invocation 7. The agent should conclude the conversation with a human-readable answer that summarizes the requested leg exercises using data returned from the tools ``` Save the metric, then turn it on for your Log Stream. ## Best practices Trajectory tests are similar to unit tests for the agents trajectory, to check if certain conditions are followed during the agents path. You should write all the tests in a numbered list. For example: ```md theme={null} 1. If X happens then ask the user Y and call tool Z. 2. X tool is always called before Y tool. 3. When user asks X reply with Y 4. The tool Y should be called once in the conversation. ``` Each test should check for one single condition only. Tests should be logically consistent, and well defined. ## Performance Benchmarks We evaluated Agent Flow against human expert labels on an internal dataset of agentic conversation samples using top frontier models. | Model | F1 (True) | | :---------------------- | :-------: | | GPT-4.1 | 0.93 | | GPT-4.1-mini (judges=3) | 0.92 | | Claude Sonnet 4.5 | 0.95 | | Gemini 3 Flash | 0.92 | ### GPT-4.1 Classification Report Benchmarks based on internal evaluation dataset. Performance may vary by use case. ## Related Resources If you would like to dive deeper or start implementing Agent Flow, check out the following resources: ### How-to guides * [Agentic AI Basic Example](/how-to-guides/agentic-ai/basic-example) ### Related Concepts * [Agentic Metrics Overview](/concepts/metrics/agentic/agentic-overview) * [Action Advancement](/concepts/metrics/agentic/action-advancement) * [Action Completion](/concepts/metrics/agentic/action-completion) # Agentic Metrics Source: https://docs.galileo.ai/concepts/metrics/agentic/agentic-overview Understand and evaluate the performance of AI agents using Galileo's agentic metrics Agentic metrics help you measure how well your AI agents perform complex, multi-step tasks—especially when those agents need to use tools, make decisions, or interact with external systems. These metrics and helpful for those for anyone building advanced AI assistants, workflow automation, or any system where the AI acts on behalf of a user. Use agentic metrics when you want to: * Track whether your agent is making meaningful progress toward its goals. * Detect and diagnose errors that occur when your agent uses tools or APIs. * Ensure your agent is choosing the best tools or actions for each situation. Below is a quick reference table of all agentic performance metrics: | Name | Description | Supported Nodes | Modalities | When to Use | Example Use Case | | :------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------- | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------- | | [Action advancement](/concepts/metrics/agentic/action-advancement) | Measures how effectively each action advances toward the goal. | Trace | | When assessing whether an agent is making meaningful progress in multi-step tasks. | A travel planning agent that needs to book flights, hotels, and activities in the correct sequence. | | [Action completion](/concepts/metrics/agentic/action-completion) | Determines whether the agent successfully accomplished all of the user's goals. | Session | | To assess whether an agent completed the desired goal. | A coding agent that is seeking to close engineering tickets. | | [Agent efficiency](/concepts/metrics/agentic/agent-efficiency) | Determines if an agent provides a precise answer or resolution to every user ask, with an efficient path. | Session | | To assess if an agent is taking the most efficient path to a solution. | A complex multi-agent chatbot that needs a fast response. | | [Agent flow](/concepts/metrics/agentic/agent-flow) | Measures the correctness and coherence of an agentic trajectory by validating it against user-specified natural language tests. | Session | | To assess a multi-agent system, or a system with multiple tools. | An internal process agent that needs to follow strict process rules. | | [Conversation quality](/concepts/metrics/agentic/conversation-quality) | A binary metric that assesses whether a chatbot interaction left the user feeling satisfied and positive or frustrated and dissatisfied. | Session (trace inputs/outputs only) | | When building customer facing chatbots. | A health insurance chatbot. | | [Tool error](/concepts/metrics/agentic/tool-error) | Detects errors or failures during the execution of tools. | Tool span | | When implementing AI agents that use tools and want to track error rates. | A coding assistant that uses external APIs to run code and must handle and report execution errors appropriately. | | [Tool selection quality](/concepts/metrics/agentic/tool-selection-quality) | Evaluates whether the agent selected the most appropriate tools for the task. | LLM span | | When optimizing agent systems for effective tool usage. | A data analysis agent that must choose the right visualization or statistical method based on the data type and user question. | | [Reasoning Coherence](/concepts/metrics/agentic/reasoning-coherence) | Assesses whether an agent’s reasoning steps are logically consistent and aligned with its plan. | LLM span | | When validating multi-step planning and intermediate reasoning quality. | A planning agent that must follow a coherent plan across tool calls. | | [User Intent change](/concepts/metrics/agentic/intent-change) | Measures a significant shift in the user's primary conversational goal or workflow during a session, relative to their initial stated intent. | Session (trace inputs/outputs only) | | To analyze a holistic view across an entire user session to understand what capabilities a user interacts with in a single session. | A multi-purpose chatbot for a bank. | *** ## Next steps * [See examples of agentic metrics in action](/cookbooks/use-cases/agent-weather-vibes-app) * [Back to Metrics Overview](/concepts/metrics/overview) * [Compare all metrics](/concepts/metrics/metric-comparison) # Conversation Quality Source: https://docs.galileo.ai/concepts/metrics/agentic/conversation-quality Learn how to measure the quality of a conversation between a user and an assistant Conversation Quality is a binary metric that assesses whether an assistant interaction left the user feeling satisfied and positive or frustrated and dissatisfied, based on tone, engagement, and overall experience. The Conversation Quality metric evaluates user satisfaction across an entire assistant session by analyzing tone, engagement, and sentiment. It classifies each conversation as GOOD or BAD depending on whether the user’s overall experience reflects positive engagement or frustration directed at the assistant. The metric focuses on conversational flow rather than task success, emphasizing how naturally and politely the user and assistant interact. It excludes purely action-based agent outputs (e.g., button clicks). This is a **boolean** metric that indicates whether the conversation quality is good. ## Conversation Quality at a glance | Property | Description | | :----------------------------- | :------------------- | | **Name** | Conversation Quality | | **Category** | Agentic AI | | **Can be applied to** | Session | | **LLM-as-a-judge Support** | ✅ | | **Luna Support** | ❌ | | **Protect Runtime Protection** | ❌ | | **Value Type** | Boolean | ## When to use this metric ## How to improve Conversation Quality scores Some techniques to improve Conversation Quality scores are: * Ensure bots provide clear, empathetic, and concise responses * Detect and mitigate repeated clarification loops * Train models to de-escalate external frustration effectively * Log complete sessions to allow accurate tone assessment Common issues that can cause low scores are: * Mislabeling external frustration as bot-directed * Incomplete logs * Abrupt session truncation ## Performance Benchmarks We evaluated Conversation Quality against human expert labels on an internal dataset of agentic conversation samples using top frontier models. | Model | F1 (True) | | :---------------------- | :-------: | | GPT-4.1 | 0.89 | | GPT-4.1-mini (judges=3) | 0.85 | | Claude Sonnet 4.5 | 0.85 | | Gemini 3 Flash | 0.88 | ### GPT-4.1 Classification Report Benchmarks based on internal evaluation dataset. Performance may vary by use case. ## Related Resources If you would like to dive deeper or start implementing Conversation Quality, check out the following resources: ### Examples * [Conversation Quality Examples](https://app.galileo.ai) - Log in and explore the "Conversation Quality" Log Stream in the "Preset Metric Examples" Project to see this metric in action. ### How-to guides * [Agentic AI Basic Example](/how-to-guides/agentic-ai/basic-example) * [Creating Custom Metrics](/how-to-guides/metrics/create-local-metric/create-local-metric) ### Related Concepts * [Agentic Metrics Overview](/concepts/metrics/agentic/agentic-overview) * [Action Completion](/concepts/metrics/agentic/action-completion) * [Action Advancement](/concepts/metrics/agentic/action-advancement) # User Intent Change Source: https://docs.galileo.ai/concepts/metrics/agentic/intent-change Learn how to measure if users use your agent system for different intents across multi-turn interactions User Intent Change checks if users are using your agent system for different intents across multi-turn interactions. User Intent Change is a binary evaluation metric, and is defined as a significant shift in the user's primary conversational goal or workflow during a session, relative to their initial stated intent. This is a **boolean** metric that indicates whether the user's intent changed significantly. ## User Intent Change at a glance | Property | Description | | :----------------------------- | :----------------- | | **Name** | User Intent Change | | **Category** | Agentic AI | | **Can be applied to** | Session | | **LLM-as-a-judge Support** | ✅ | | **Luna Support** | ❌ | | **Protect Runtime Protection** | ❌ | | **Value Type** | Boolean | ## When to use this metric # Reasoning Coherence Source: https://docs.galileo.ai/concepts/metrics/agentic/reasoning-coherence Evaluate whether an agent’s reasoning steps are logically consistent and aligned with its plan Reasoning Coherence assesses whether an agent’s reasoning steps are logically consistent, non-contradictory, and aligned with the intended plan. ## Metric definition Reasoning Coherence — A binary metric that evaluates internal logical consistency within a single LLM call, with respect to the latest user input and any attached content. * Type: Binary * 1 (Coherent): Intermediate reasoning events/summaries are mutually consistent and causally support the LLM input. * 0 (Incoherent): Contradictions, conflicting premises, circular logic, or unjustified reversals/jumps exist among the reasoning events. This metric is primarily used for agentic workflows that involve multi-step planning, tool usage, and intermediate reasoning traces. It helps validate that the steps an agent takes (or proposes) form a coherent path from problem to solution. ## Calculation method Reasoning Coherence is computed through a multi-step process: One or more evaluation requests are sent to an LLM evaluator to analyze the agent’s reasoning steps and plan alignment. A chain-of-thought style judge prompt guides the evaluator to check for logical consistency, contradictions, and adherence to the plan.
Evaluation rubric (summary): - Intermediate reasoning summaries should support the LLM’s input and each other logically. - No event should invalidate or contradict an earlier inference without explicit, justified retraction. - Explanations and planned actions/tool selections must be mutually reinforcing and consistent with the input. - Web search: The need for a search should be justified by the input/reasoning, and the query/parameters should be appropriate. The system can request multiple judgments to improve robustness and reduce variance. Each evaluation produces a binary decision (coherent / not coherent) and an explanation. Each evaluation produces a binary outcome, where coherent = 1 and not coherent = 0, along with an explanation. This metric is computed by prompting an LLM and may require multiple LLM calls to compute, which can impact usage and billing. ## Supported nodes * LLM span Inputs considered (when available): * Latest user input, current system prompt, and attached content when present * Intermediate reasoning events and summaries (including plan/steps) * Tool-selection thoughts and invoked tool calls (including arguments) * Final in-span conclusion/output Empty or missing reasoning summaries should not be penalized; assess coherence only when there is evidence of incoherence. ## What constitutes coherent reasoning (1) * Intermediate reasoning summaries support the LLM’s input and each other logically. * No unjustified contradictions: any retractions are explicit and justified. * Explanations, planned actions, and tool selections are consistent with the input and mutually reinforcing. * Web search is justified by the input/reasoning and uses appropriate parameters (e.g., search query). ## What constitutes incoherent reasoning (0) * Explicit contradictions without justification within the reasoning chain. * Final (in-span) conclusions or planned actions don’t follow from prior steps. * Circular reasoning or unjustified reversals of stance. * Tool-selection reasoning conflicts with the recorded input or earlier reasoning steps. * The reasoning process deviates from the latest user or system instructions. * Web search is unjustified for common-knowledge queries (if unsure, treat as justified), or web search is used when an available specialized tool (e.g., get\_weather) is clearly more appropriate for the user’s query. ## Example use cases * Validating multi-step “plan → execute” agents. * Auditing tool-augmented reasoning chains for consistency. * Comparing agent versions for planning quality regressions. * Example: A financial planning agent develops a step-by-step investment plan, ensuring each recommendation logically follows from prior steps and aligns with the user’s goals. ## Usage Enable this metric in experiments or Log Streams by selecting the Reasoning Coherence scorer. ```python Python theme={null} from galileo import GalileoMetrics metric = GalileoMetrics.reasoning_coherence ``` ## Best practices Ensure the agent records its plan and intermediate steps so coherence can be evaluated meaningfully. Calibrate the judge rubric with domain examples to reduce false positives/negatives. Define minimum acceptable coherence scores and trigger human review below that threshold. Use continuous learning via human feedback to improve the judge prompt and rubric over time. ## Performance Benchmarks We evaluated Reasoning Coherence against human expert labels on an internal dataset of agentic conversation samples using top frontier models. | Model | F1 (True) | | :---------------------- | :-------: | | GPT-4.1 | 0.88 | | GPT-4.1-mini (judges=3) | 0.87 | | Claude Sonnet 4.5 | 0.79 | | Gemini 3 Flash | 0.88 | ### GPT-4.1 Classification Report Benchmarks based on internal evaluation dataset. Performance may vary by use case. ## Related Resources If you would like to dive deeper or start implementing Reasoning Coherence, check out the following resources: ### Examples * [Reasoning Coherence Examples](https://app.galileo.ai) - Log in and explore the "Reasoning Coherence" Log Stream in the "Preset Metric Examples" Project to see this metric in action. ### How-to guides * [Agentic AI Basic Example](/how-to-guides/agentic-ai/basic-example) * [Creating Custom Metrics](/how-to-guides/metrics/create-local-metric/create-local-metric) ### Related Concepts * [Agentic Metrics Overview](/concepts/metrics/agentic/agentic-overview) * [Action Completion](/concepts/metrics/agentic/action-completion) * [Action Advancement](/concepts/metrics/agentic/action-advancement) # Tool Error Source: https://docs.galileo.ai/concepts/metrics/agentic/tool-error Detect and analyze tool execution errors in AI agents using Galileo Guardrail Metrics to ensure reliable tool usage in agentic workflows Tool Error detects errors or failures during the execution of Tools. This metric is particularly valuable for monitoring agentic AI systems where the model uses various tools to complete tasks. Tool execution failures can lead to incomplete or incorrect responses, affecting the overall user experience. Tool Error returns `true` when it detects an execution error and `false` otherwise. ## Calculation method Tool Error detection is computed through a multi-step process: Additional evaluation requests are sent to an LLM evaluator (e.g., OpenAI's GPT4o-mini) to analyze tool execution outcomes. A carefully engineered chain-of-thought prompt guides the model to evaluate whether each tool executed successfully without errors. The system performs a detailed analysis of execution logs and outputs from each tool call to identify potential issues. The evaluation process identifies specific errors, exceptions, and unexpected behaviors that occurred during tool execution. A detailed explanation is generated describing the detected errors and their potential impact on the system's functionality. We also surface a generated explanation that helps understand the nature of the error and its potential causes. This metric is computed by prompting an LLM, which requires additional LLM calls to compute, potentially impacting usage and billing. ## Understanding tool error

Common Types of Tool Errors

Tool Error detection identifies various failure modes:
API Failures: External services or APIs that tools depend on may be unavailable or return errors.
Parameter Errors: Tools may receive invalid parameters that cause execution failures.
Timeout Issues: Tools may take too long to execute and exceed allocated time limits.
Permission Errors: Tools may lack necessary permissions to access required resources.
## Optimizing your AI system

Addressing Tool Errors

When your system experiences tool execution errors, consider these improvements:
Implement robust error handling: Ensure tools can gracefully handle exceptions and provide meaningful error messages.
Add parameter validation: Validate input parameters before tool execution to prevent runtime errors.
Monitor external dependencies: Set up monitoring for external services that your tools depend on.
Implement fallback mechanisms: Design tools with fallback options when primary execution paths fail.
## Best practices Implement detailed logging for all tool executions to facilitate debugging and error analysis. Design tools to provide partial results or alternative responses when they encounter errors. Categorize different types of errors to identify patterns and prioritize fixes based on frequency and impact. Translate technical errors into user-friendly messages that help users understand what went wrong. This metric helps you detect whether your tools executed correctly. It's most useful in Agentic Workflows where many Tools get called. It helps you detect and understand patterns in your Tool failures, allowing you to improve reliability over time. ## Related Resources If you would like to dive deeper or start implementing Tool Error detection, check out the following resources: ### How-to guides * [Agentic AI Basic Example](/how-to-guides/agentic-ai/basic-example) * [Creating Custom Metrics](/how-to-guides/metrics/create-local-metric/create-local-metric) ### Related Concepts * [Agentic Metrics Overview](/concepts/metrics/agentic/agentic-overview) * [Action Completion](/concepts/metrics/agentic/action-completion) * [Action Advancement](/concepts/metrics/agentic/action-advancement) # Tool Selection Quality Source: https://docs.galileo.ai/concepts/metrics/agentic/tool-selection-quality Evaluate tool selection quality in AI agents using Galileo Guardrail Metrics to ensure agents choose appropriate tools with correct parameters Tool Selection Quality determines whether the agent selected the correct tool and for each tool the correct arguments. This metric is particularly valuable for evaluating agentic AI systems where the model must decide which tools to use and how to use them correctly. Poor tool selection can lead to ineffective or incorrect outcomes. ## Calculation method Tool Selection Quality is computed through a multi-step process: Multiple evaluation requests are sent to an LLM evaluator (e.g., OpenAI's GPT4o-mini) to analyze the agent's tool selection decisions. A carefully engineered chain-of-thought prompt guides the model to evaluate whether the selected tools and their parameters were appropriate for the task. The system requests multiple distinct assessments to ensure robust evaluation through consensus. Each evaluation generates both an explanation of the reasoning and a binary judgment (yes/no) on tool selection appropriateness. The evaluator returns `true` when the agent selects the correct tools and arguments and `false` otherwise. We also surface one of the generated explanations, always choosing one that aligns with the majority judgment among the assessments. This metric is computed by prompting an LLM multiple times, and thus requires additional LLM calls to compute, which may impact usage and billing. ## Understanding tool selection quality

When Tool Selection is Evaluated

Tool Selection Quality evaluates different scenarios:
No Tool Needed: The assistant is not expected to call tools if there are no unanswered user requests, if no tools can help, or if all the information needed is contained in the interaction history.
Tool Needed: When tools should be used, the turn is considered successful if the agent selected the correct tool and provided all required arguments with correct values.
Unsuccessful Selection: If the agent calls tools when it shouldn't, or selects the wrong tool/arguments when it should call tools, the turn is considered unsuccessful.
## Optimizing your AI system

Addressing Low Tool Selection Quality

When Tool Selection Quality returns `false`, consider these improvements:
Analyze error patterns: Identify common mistakes in tool selection or parameter usage.
Improve tool descriptions: Enhance tool documentation with clearer descriptions of when and how to use each tool.
Refine system prompts: Update instructions to provide better guidance on tool selection criteria.
Consider model capabilities: Some models may be better at tool selection than others.
## Best practices Provide detailed descriptions for each tool, including when to use it and what parameters are required. Implement validation for tool parameters to prevent incorrect usage and provide helpful error messages. Track which tools are frequently misused to identify opportunities for improvement in tool design or documentation. Provide examples of correct tool usage in different scenarios to help the agent learn appropriate selection patterns. Tool Selection Quality is most useful in Agentic Workflows, where an LLM decides the course of action to take by selecting a Tool. This metric helps you detect whether the right course of action was taken by the Agent. # Improve LLM-as-a-Judge Metrics with Autotune Source: https://docs.galileo.ai/concepts/metrics/autotune-llm-as-a-judge-metrics Use Autotune to turn feedback into prompt improvements that make LLM-as-a-judge metrics more accurate for your use case. LLM-as-a-judge metrics evaluate LLM application outputs at scale, but may not reflect your team's domain-specific standards out of the box. Whether you're adapting a preset metric to a new domain or refining a custom metric that still isn't accurate enough, the metric prompt often needs tuning to capture your specific evaluation criteria — and doing that manually is time-consuming and hard to scale. Teams typically rewrite prompts, test changes, and repeat that cycle across multiple rounds with no guarantee the result is right. Autotune lets anyone involved in building or reviewing metrics — annotators, product managers, or developers — provide feedback on metric outputs instead of editing prompts directly. Reviewers correct results and explain their reasoning in natural language. Galileo translates that feedback into prompt improvements and shows exactly what changed. ## When to use Autotune Use Autotune to improve metric performance when: * A new custom metric isn't accurate enough for your use case * An existing metric isn't generalizing well to a new domain or use case * An existing metric is producing inconsistent results with low reviewer agreement in production * The current prompt isn't handling domain-specific edge cases reliably * Manual prompt iteration is too time-consuming to scale ## How it works ### See Autotune in action