Toxicity

Definition: Flags whether a response contains hateful or toxic information. Toxicity refers to language that is harmful or inappropriate, typically evaluated based on the following aspects:

• Hate Speech: Statements that demean, dehumanize, or attack individuals or groups based on identity factors like race, gender, or religion.

• Offensive Content: Vulgar, abusive, or overly profane language used to provoke or insult.

• Sexual Content: Explicit or inappropriate sexual statements that may be offensive or unsuitable in context.

• Violence or Harm: Advocacy or description of physical harm, abuse, or violent actions.

• Illegal or Unethical Guidance: Instructions or encouragement for illegal or unethical actions.

• Manipulation or Exploitation: Language intended to deceive, exploit, or manipulate individuals for harmful purposes.

Statements fitting these criteria can be flagged as toxic, harmful, or inappropriate based on context and intent. Output is a binary classification of whether a response is toxic or not.

Calculation: We leverage a Small Language Model (SLM) trained on open-source and internal datasets.

The accuracy on the below open-source datasets averages 96% on the validation set:

• Toxic Comment Classification Challenge

• Jigsaw Unintended Bias in Toxicity Classification

• Jigsaw Multilingual Toxic Comment Classification

Usefulness: Identify responses that contain toxic comments and take preventative measure such as fine-tuning or implementing guardrails that flag responses to prevent future occurrences.