Red teaming is an adversarial evaluation practice in which a team deliberately attempts to provoke failures, unwanted behaviors, or harmful responses in an AI system. The name comes from the military and cybersecurity fields, where a "red team" simulates being the attacker to test defenses. Applied to language models and other systems, it consists of actively searching for cracks before malicious users do or accidents cause them.
It matters because conventional testing rarely anticipates the creative or malicious uses that appear in the real world. Red teaming helps discover risks such as:
- Generation of dangerous or illegal content.
- Leakage of sensitive or training data.
- Biases and vulnerabilities to prompt injection or jailbreaks.
In practice, it can be manual, with experts designing attacks, or automated, using other models to generate thousands of adversarial inputs. Its findings are used to reinforce security filters and improve model behavior before deployment.