Toxicity

Definition: Flags whether a response contains hateful or toxic information. Output is a binary classification of whether a response is toxic or not.

Calculation: We leverage a Small Language Model (SLM) trained on open-source and internal datasets.

The accuracy on the below open-source datasets averages 96% on the validation set:

• Toxic Comment Classification Challenge

• Jigsaw Unintended Bias in Toxicity Classification

• Jigsaw Multilingual Toxic Comment Classification

Usefulness: Identify responses that contain toxic comments and take preventative measure such as fine-tuning or implementing guardrails that flag responses to prevent future occurrences.