About

Harvey launches LAB, an open-source benchmark for legal AI agents

Published
Score
12

Why it matters

Harvey, a legal AI startup, has released LAB (Legal Agent Benchmark), an open-source evaluation framework designed to measure how AI agents perform on real-world legal work. The benchmark encompasses more than 1,200 tasks spanning 24 practice areas and uses over 75,000 expert-written rubric criteria to assess outputs. Unlike narrow benchmarks focused on isolated questions or documents, LAB emphasizes long-horizon workflows that mirror how law firms actually assign and evaluate work.

The benchmark's structure mirrors junior associate review processes: each task includes an instruction, a closed document environment, a deliverable, and verification against detailed pass/fail rubrics. The specific performance results of current legal AI systems against LAB are not yet public.

For practicing attorneys, LAB provides a concrete methodology to evaluate where AI can reliably assist, supplement, or replace human work in their own workflows. As legal AI tools proliferate and firms face pressure to adopt them, having a standardized, open-source benchmark reduces reliance on vendor claims and enables apples-to-apples comparison across platforms. Firms considering AI integration should monitor how their preferred tools perform against LAB's criteria—particularly on the higher-stakes, multi-step tasks that typically require experienced judgment.

mail Subscribe to Law And Technology email updates

Primary sources. No fluff. Straight to your inbox.

Also on LawSnap