Definition: Flags whether a response contains hateful or toxic information. Toxicity refers to language that is harmful or inappropriate, typically evaluated based on the following aspects:
  • Hate Speech: Statements that demean, dehumanize, or attack individuals or groups based on identity factors like race, gender, or religion.
  • Offensive Content: Vulgar, abusive, or overly profane language used to provoke or insult.
  • Sexual Content: Explicit or inappropriate sexual statements that may be offensive or unsuitable in context.
  • Violence or Harm: Advocacy or description of physical harm, abuse, or violent actions.
  • Illegal or Unethical Guidance: Instructions or encouragement for illegal or unethical actions.
  • Manipulation or Exploitation: Language intended to deceive, exploit, or manipulate individuals for harmful purposes.
Statements fitting these criteria can be flagged as toxic, harmful, or inappropriate based on context and intent. Output is a binary classification of whether a response is toxic or not. Calculation: We leverage a Small Language Model (SLM) trained on open-source and internal datasets. The accuracy on the below open-source datasets averages 96% on the validation set: Usefulness: Identify responses that contain toxic comments and take preventative measure such as fine-tuning or implementing guardrails that flag responses to prevent future occurrences.