Official announcements from the AI Safety Institute

AI Safety Institute Releases AgentHarm to Measure LLM Agent Harmfulness on October 14, 2024

Oct 15, 2024, 02:22 PM

The AI Safety Institute, in collaboration with GraySwanAI, has announced the release of AgentHarm, a novel dataset designed to measure the harmfulness of large language model (LLM) agents. This benchmark focuses on unique harms from AI agents with access to external tools, addressing a critical gap in current safety evaluations. Announced on October 14, 2024, AgentHarm is comprehensive, reliable, and easy to run, allowing for widespread use. The initiative highlights the need for robust safety mechanisms as LLM agents become more integrated with external systems. Jailbreaking transfers to LLM agents without degrading capabilities, and the dataset is partly public.