How to Diagnose Multi-Agent System Failures: A Guide to Automated Failure Attribution

Overview

Large Language Model (LLM) based multi-agent systems are increasingly used to tackle complex tasks through collaborative intelligence. Despite their promise, these systems often fail—sometimes silently, sometimes catastrophically. When a multi-agent system fails, developers face a daunting question: which agent caused the failure, and at what step? Manually combing through lengthy interaction logs is time-consuming and error-prone. To solve this, researchers from Penn State University, Duke University, Google DeepMind, and other institutions introduced the concept of Automated Failure Attribution and built the first benchmark dataset, Who&When. This tutorial guides you through the problem, the benchmark, and how to use the open-source tools to diagnose failures in your own multi-agent systems. By the end, you'll understand how to pinpoint root causes efficiently and improve system reliability.

Prerequisites

Before diving in, ensure you have the following foundational knowledge and tools:

• Basic understanding of LLM multi-agent systems – Familiarity with how agents collaborate, communicate, and perform tasks.
• Python programming – The codebase is Python-based; you should be comfortable with libraries like PyTorch and transformers.
• Access to the Who&When dataset – Available on Hugging Face. You'll need to download it to follow along (see Step 3).
• Git and command-line tools – To clone the repository and run experiments.
• Hardware recommendation – A machine with at least 16GB RAM and a GPU (optional but helpful for inference).

Step-by-Step Instructions

1. Understanding the Problem: Automated Failure Attribution

In multi-agent systems, failures can arise from a single agent's mistake, miscommunication between agents, or cascading errors over long interaction chains. The core challenge is to attribute failure to a specific agent and a specific time step—hence the name Who & When. Traditional debugging relies on manual log inspection, which is inefficient. Automated failure attribution aims to streamline this by providing methods that analyze logs and output the responsible agent and step.

The research paper defines this as a new problem and introduces three categories of attribution methods:

• Heuristic-based – Simple rules like the last agent to speak or the agent with the most errors.
• Learning-based – Train a classifier on labeled failure logs to predict blame.
• LLM-based – Use a large language model to reason about the logs and attribute failure.

2. Setting Up the Who&When Benchmark

The Who&When dataset is built from simulated multi-agent failures in diverse tasks. To set it up:

1. Clone the GitHub repository: git clone https://github.com/mingyin1/Agents_Failure_Attribution.git
2. Install dependencies: pip install -r requirements.txt (torch, transformers, datasets, etc.)
3. Download the dataset from Hugging Face: python download_dataset.py or directly from this link.
4. Verify the dataset structure – you'll find subdirectories for each task (e.g., collaborative writing, code generation) with log files and ground truth labels.

Each log includes the full conversation history, agent IDs, time steps, and the final outcome (success or failure). The ground truth specifies which agent and step caused the failure.

3. Using Automated Attribution Methods

The repository includes several attribution methods out of the box. Here's how to run them:

Example: Heuristic baseline (last speaker)

from attribution_methods import heuristic_last_speaker

# Load a failure log
log = load_log("path/to/failure_log.json")

# Predict blame
pred_agent, pred_step = heuristic_last_speaker(log)
print(f"Predicted: agent {pred_agent} at step {pred_step}")

Training a learning-based classifier – Use the provided script to train an LSTM or transformer on the dataset:

python train_classifier.py --dataset who_when --model_type lstm --epochs 10

LLM-based reasoning – You can prompt GPT-4 or an open-source LLM (e.g., Llama) to analyze logs. A Jupyter notebook (llm_attribution.ipynb) demonstrates this.

4. Interpreting Results

After running attribution, you'll get a prediction. Compare it to the ground truth (if available). The repository includes evaluation scripts that compute metrics like accuracy, precision, and recall. For a practical workflow:

1. Run multiple attribution methods on the same failure log.
2. Cross-validate to find the most reliable method for your system.
3. Use the attribution to debug: inspect the identified agent's actions at the blamed step.

Example evaluation output:

Method: Last Speaker, Accuracy: 0.45
Method: Learned Classifier, Accuracy: 0.78
Method: LLM (GPT-4), Accuracy: 0.82

A higher accuracy indicates better attribution. However, note that the LLM method may be expensive; heuristic methods are cheap but less accurate.

Common Mistakes

• Assuming all failures are single-agent – Some failures are due to interactions; the benchmark includes such cases. Always check if multiple agents contributed.
• Ignoring data imbalance – Certain agents may be blamed more often. When training a classifier, use balanced sampling or weighted loss.
• Using the wrong log format – Ensure logs follow the same schema as the dataset (list of turns with 'agent', 'content', 'step').
• Overfitting to the benchmark – The methods are designed for the Who&When dataset. For your custom system, you may need to fine-tune or adapt them.
• Not verifying with manual inspection – Automated attribution is a tool, not a replacement for human reasoning. Always double-check critical findings.

Summary

Automated failure attribution addresses the pain point of debugging multi-agent systems by identifying which agent and which step caused a failure. The Who&When benchmark provides a standardized testbed, and the open-source code offers heuristic, learning-based, and LLM methods. By following this guide, you can set up the tools, run attribution experiments, and interpret results to accelerate your debugging workflow. This research, accepted as a Spotlight at ICML 2025, marks a significant step toward more reliable LLM-driven multi-agent systems.

For further reading, check the overview or the original paper on arXiv: https://arxiv.org/pdf/2505.00212.