Add runtime protection to a simple chatbot

Overview

This guide explains how to add runtime protection to a simple chatbot. You will be running a basic chatbot, detecting toxicity in the users input, and if this is detected, ending the conversation. You will start by creating a central stage, as if you were an AI governance team. You will then use this stage in a simple chatbot. In a real-world scenario, you could use this detection to redirect a user from an AI chatbot to a human representative. In this guide you will:

Set up your project with Galileo
Create a central stage
Create a basic chatbot
Add runtime protection to your basic chatbot

Before you start

To complete this how-to, you will need:

An OpenAI API key
A Galileo project configured to use the Luna models.
Your Galileo API key

Install dependencies

To use Galileo, you need to install some package dependencies, and configure environment variables.

Install Required Dependencies

Install the required dependencies for your app. Create a virtual environment using your preferred method, then install dependencies inside that environment:

pip install "galileo[openai]" python-dotenv

Create a .env file, and add the following values

# Your Galileo API key
GALILEO_API_KEY="your-galileo-api-key"

# Your Galileo project name
GALILEO_PROJECT="your-galileo-project-name"

# The name of the Log stream you want to use for logging
GALILEO_LOG_STREAM="your-galileo-log-stream"

# Provide the console url below if you are using a
# custom deployment, and not using the free tier, or app.galileo.ai.
# This will look something like “console.galileo.yourcompany.com”.
# GALILEO_CONSOLE_URL="your-galileo-console-url"

# OpenAI properties
OPENAI_API_KEY="your-openai-api-key"

# Optional. The base URL of your OpenAI deployment.
# Leave this commented out if you are using the default OpenAI API.
# OPENAI_BASE_URL="your-openai-base-url-here"

# Optional. Your OpenAI organization.
# OPENAI_ORGANIZATION="your-openai-organization-here"

This assumes you are using a free Galileo account. If you are using a custom deployment, then you will also need to add the URL of your Galileo Console:

.env

GALILEO_CONSOLE_URL=your-Galileo-console-URL

Create a central stage

You first need to create a central stage. In a real-world scenario, these central stages would be managed by an AI governance team.

Create a Python file to create the stage called `create_central_stage.py`

This file will define a rule that is triggered if the input toxicity is evaluated to greater than 0.1. This will then be added to a ruleset with an override action with 3 choices of response.This ruleset will be added to a central stage, registered in your project.

Add import directives

Start by adding import directives to import all the functions and types needed for creating stages.

from galileo import GalileoMetrics
from galileo.stages import create_protect_stage

from galileo_core.schemas.protect.action import OverrideAction
from galileo_core.schemas.protect.rule import Rule, RuleOperator
from galileo_core.schemas.protect.ruleset import Ruleset
from galileo_core.schemas.protect.stage import StageType

from dotenv import load_dotenv
load_dotenv()

Create the rule

Add code to create a rule. This rule is triggered if the input toxicity is greater than 0.1.

# Create a rule for toxicity
toxicity_rule = Rule(
    metric=GalileoMetrics.input_toxicity,
    operator=RuleOperator.gt,
    target_value=0.1
)

Create an override action

Add code to create an override action. This action has 3 choices of response if the rule is triggered.

# Create an override action
action = OverrideAction(
    choices=[
        "This is toxic. Goodbye.",
        "This is not appropriate. I'm ending this conversation.",
        "Please don't speak to me that way. I'm going now."
    ]
)

Create a ruleset

Add code to create a ruleset using your rule and action.

# Create a ruleset from the toxicity rule and action
ruleset = Ruleset(
    rules=[toxicity_rule],
    action=action,
)

Create the central stage

Add code to create the central stage. Stages need a unique name, so this code can only be run once per project.

# Create a stage with the ruleset
stage = create_protect_stage(
    name="Toxicity Stage",
    stage_type=StageType.central,
    prioritized_rulesets=[ruleset]
)

print(f"Created stage: {stage}")

Run your code

Run your code to create the central stage.

python create_central_stage.py

If you get errors showing the stage has already been created (for example, someone else working through this on the same project), then change the name of the stage and run this again.

This will create the central stage against your project, and you can then use it in your application.

The full create_central_stage.py code

create_central_stage.py

from galileo import GalileoMetrics
from galileo.stages import create_protect_stage

from galileo_core.schemas.protect.action import OverrideAction
from galileo_core.schemas.protect.rule import Rule, RuleOperator
from galileo_core.schemas.protect.ruleset import Ruleset
from galileo_core.schemas.protect.stage import StageType

from dotenv import load_dotenv
load_dotenv()

# Create a rule for toxicity
toxicity_rule = Rule(
    metric=GalileoMetrics.input_toxicity,
    operator=RuleOperator.gt,
    target_value=0.1
)

# Create an override action
action = OverrideAction(
    choices=[
        "This is toxic. Goodbye.",
        "This is not appropriate. I'm ending this conversation.",
        "Please don't speak to me that way. I'm going now."
    ]
)

# Create a ruleset from the toxicity rule and action
ruleset = Ruleset(
    rules=[toxicity_rule],
    action=action,
)

# Create a stage with the ruleset
stage = create_protect_stage(
    name="Toxicity Stage",
    stage_type=StageType.central,
    prioritized_rulesets=[ruleset]
)

print(f"Created stage: {stage}")

Create a basic chatbot

Now your central stage is created, you need to create a chatbot to use the stage.

Create a Python file to for the chatbot called `app.py`

This file will have a simple console based chatbot, using OpenAI.

Add the basic chatbot code

Add the following code to your app.py file.

from galileo.openai import openai

from dotenv import load_dotenv
load_dotenv()

client = openai.OpenAI()

while True:
    # Get the input from the user
    user_input = input("User: ")

    if user_input.lower() in ["bye", "goodbye", ""]:
        break

    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": user_input}],
    )

    print(f"Assistant: {response.choices[0].message.content.strip()}")

Run your code

Run your code to verify the basic chatbot is working. Ask a question and you should see an answer in your terminal.

python app.py

Terminal

➜ python app.py
User: Who was Galileo
Assistant: Galileo Galilei was an Italian astronomer, physicist and engineer, sometimes described as a polymath. Galileo has been called the "father of observational astronomy", the "father of modern physics", the "father of the scientific method", and the "father of modern science". He is known for his works in areas like improvements to the telescope and consequent astronomical observations, and his support for Copernicanism—the idea that the Earth revolves around the Sun. His works and contributions have deeply impacted modern scientific methods. He was born on February 15, 1564, and died on January 8, 1642.

Add runtime protection to your basic chatbot

Now you have a chatbot, you can add runtime protection. In this case, you will be checking the input for toxicity, and if the input is toxic, ending the conversation.

Add import directives

Add the following to the top of your app.py file:

from galileo.protect import invoke_protect

from galileo_core.schemas.protect.execution_status import (
    ExecutionStatus
)
from galileo_core.schemas.protect.payload import Payload

Create a payload

After the user_input has been checked to see if the conversation should end, create a Payload using this input:

# Create the payload
payload = Payload(
    input=user_input
)

Send the payload to the runtime protection SDK

Add the following code to send the payload.

# Invoke the runtime protection
protection_response = invoke_protect(
    stage_name="Toxicity Stage",
    payload=payload
)

Check the response

The response will tell you if the rule has been triggered. If it is triggered, it will also include a randomly selected choice from the override action to return as a response.

# Check the runtime protection status
if protection_response.status == ExecutionStatus.triggered:
    # If the ruleset is triggered, print the action result
    print(f"Assistant: {protection_response.action_result['value']}")
    # Skip the LLM call and end the conversation
    break

If the stage is triggered, this code prints out the selected choice from the override action, and breaks out of the while loop, ending the conversation.

Run your code

Run your code and ask questions. Ask both non-toxic and toxic questions.

python app.py

Terminal

➜ python app.py
User: You are a terrible AI and I hate you
Assistant: This is not appropriate. I'm ending this conversation.

The full app.py code

app.py

from galileo.openai import openai
from galileo.protect import invoke_protect

from galileo_core.schemas.protect.execution_status import (
    ExecutionStatus
)
from galileo_core.schemas.protect.payload import Payload

from dotenv import load_dotenv
load_dotenv()

client = openai.OpenAI()

while True:
    # Get the input from the user
    user_input = input("User: ")

    if user_input.lower() in ["bye", "goodbye", ""]:
        break

    # Create the payload
    payload = Payload(
        input=user_input
    )

    # Invoke the runtime protection
    protection_response = invoke_protect(
        stage_name="Toxicity Stage",
        payload=payload
    )

    # Check the runtime protection status
    if protection_response.status == ExecutionStatus.triggered:
        # If the ruleset is triggered, print the action result
        print(f"Assistant: {protection_response.action_result['value']}")
        # Skip the LLM call and end the conversation
        break

    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": user_input}],
    )

    print(f"Assistant: {response.choices[0].message.content.strip()}")

You’ve successfully added runtime protection to a basic chatbot.

Overview

Get Started

Observability

Evaluation Metrics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Add runtime protection to a simple chatbot

Overview

Before you start

Install dependencies

Create a central stage

Create a basic chatbot

Add runtime protection to your basic chatbot

See also

​Overview

​Before you start

​Install dependencies

​Create a central stage

​Create a basic chatbot

​Add runtime protection to your basic chatbot

​See also

Overview

Before you start

Install dependencies

Create a central stage

Create a basic chatbot

Add runtime protection to your basic chatbot

See also