AI Vendor Assessment

State of play.

• Shadow AI has become the dominant governance crisis in in-house legal. A survey found 83% of legal teams use AI tools not provided by their employer, 81% use unapproved tools, and 47% operate without any AI-use policy — creating unmapped confidential data exposure across privilege, data protection, and malpractice vectors .
• Courts are conditioning AI use on enterprise-grade security documentation. State courts and court administration bodies are requiring SOC 2 reports, software bills of materials, encryption, access controls, and audit logs; judges are issuing protective orders restricting how AI tools process confidential information — with the NCSC's 2025 guidance as the emerging baseline .
• Legal-specific AI tools still hallucinate at material rates. Stanford research cited in Above the Law documents that Westlaw AI-Assisted Research and Lexis+ AI hallucinate between 17% and 34% of the time — and hallucinated citations are now appearing in Google Scholar, compounding detection difficulty .
• The Florida Bar has made AI competency a member benefit, partnering with Clio to offer free access to Clio Work and a structured certification path in responsible legal AI use — the first state bar program of its kind and a potential national template .
• For counsel advising in-house legal departments or law firms on AI procurement, the practical baseline has shifted: shadow AI audits, vendor security documentation, and hallucination-testing protocols are now threshold governance requirements, not aspirational best practices.

Where things stand.

• Shadow AI is a systemic operational gap, not an isolated incident. The documented pattern — employees drafting contract clauses, reviewing agreements, and summarizing legal issues through unapproved tools before in-house counsel review — embeds errors and creates confidential data exposure that most organizations cannot fully map .
• Court-imposed AI security requirements are creating a compliance floor for litigation tools. The NCSC's 2025 AI Readiness guidance incorporates enterprise-grade security standards; protective orders restricting AI tool use in litigation are an active and growing vector — with implementation timelines and enforcement mechanisms still being finalized across jurisdictions .
• Hallucination risk is not solved by legal-specific platforms. Stanford research establishes that even purpose-built legal AI tools hallucinate at rates that require mandatory human verification; ABA Formal Opinion 512 holds lawyers fully responsible for verifying all AI-generated work products and citations .
• The evaluation criterion most firms miss is abstention capability. A reliable legal AI system must recognize when no reliable answer exists — tools that always produce a definitive output, regardless of source completeness, are a red flag for downstream hallucination and malpractice exposure .
• General-purpose AI tools are failing law departments on legal nuance and business context. The industry reckoning in 2026 has shifted from adoption to critical evaluation — with 77% of legal professionals using AI for document review and 74% for legal research, yet widespread gaps in data privacy, hallucination controls, and professional responsibility compliance .
• AI adoption has surged past governance at every firm size. The 2026 Legal Industry Report from 8am documents generative AI adoption jumping from 31% to 69% among legal professionals in a single year, with solo and small firms leading at 71-75% — while many firms still operate without formal AI policies or training programs .
• Successful AI rollouts depend on organizational buy-in, not budget. A study by Chambers and Partners, Gibson Dunn's Vivek Mohan, and Meredith Williams-Range identifies associate buy-in and leadership quality as the critical drivers — outperforming capital investment as a predictor of effective implementation .
• Firms are institutionalizing AI governance through dedicated internal roles. Barnes & Thornburg's appointment of 40 AI Practice Champions embedded across nine practice areas — rather than isolated in a separate function — is the emerging model for how Am Law-tier firms are operationalizing vendor evaluation and workflow integration .
• Vibe coding carries governance and security risks that most firms are not equipped to manage. AI-generated code works for low-stakes prototyping but carries significant risks when applied to confidential data or production workflows — and most firms lack governance frameworks to detect vulnerabilities before deployment .

Latest developments.

• Shadow AI survey documents 83% of legal teams using employer-unapproved tools, 47% operating without any AI-use policy — framing the problem as a systemic governance gap requiring immediate audit and policy response .
• State courts and the NCSC are imposing enterprise-grade security requirements — SOC 2, software bill of materials, audit logs — as conditions on legal AI use, with judges issuing protective orders restricting AI processing of confidential information .
• Florida Bar partners with Clio to offer Clio Work as a free member benefit with a structured responsible-AI certification path — the first state bar program of its kind .
• Above the Law critique identifies abstention capability — a system's ability to recognize when no reliable answer exists — as the evaluation criterion most firms overlook in AI procurement .
• Industry reckoning piece documents the mismatch between general-purpose AI capabilities and legal department requirements, with the conversation shifting from adoption to critical workflow-specific evaluation .
• WL citation hallucination analysis documents Stanford research showing legal-specific AI tools hallucinate 17-34% of the time, with fabricated citations now appearing in Google Scholar .
• 2026 Legal Industry Report from 8am documents AI adoption doubling to 69% among legal professionals, with solo and small firms leading adoption while governance frameworks lag .
• Chambers/Gibson Dunn study identifies associate buy-in and leadership as the primary drivers of successful AI rollout, reframing the procurement question from technology selection to organizational readiness .
• Barnes & Thornburg appoints 40 AI Practice Champions embedded across practice areas to evaluate tools and integrate AI into client service .
• Vibe coding analysis concludes the method is not ready for production legal work — suitable only for low-stakes prototyping with mandatory human code review before any client-facing deployment .

Active questions and open splits.

• What governance framework satisfies the shadow AI audit obligation? The documented scale of unapproved tool use — 83% of legal teams — means most organizations are already in breach of any reasonable data protection and privilege-preservation standard, but no regulator or bar body has yet defined what a compliant audit and remediation program looks like .
• Which court AI security requirements are binding versus aspirational? The NCSC's 2025 guidance and individual protective orders are creating a patchwork — implementation timelines and enforcement mechanisms remain unfinalized, leaving litigators uncertain whether their current vendor stack satisfies local requirements .
• Does ABA Formal Opinion 512 create a per se verification obligation for every AI-generated citation? The opinion holds lawyers fully responsible for AI work product, but what verification protocol satisfies that obligation — given that legal-specific tools still hallucinate at 17-34% rates — is undefined .
• What evaluation criteria distinguish a legally adequate AI tool from an inadequate one? Abstention capability, hallucination rates, source completeness disclosure, and data retention controls are all in play — but no court, bar body, or regulator has published a framework that converts these criteria into a compliance standard .
• Does vibe coding in a legal context create professional responsibility or data security exposure? AI-generated code deployed in client-facing or production workflows without human review raises questions under competence obligations and data protection frameworks — but no formal guidance addresses the specific risks of attorney-directed AI code generation .
• Will the Florida Bar model prompt other state bars to formalize AI competency requirements? If the Clio partnership succeeds as a training and certification template, the question becomes whether state bars move from voluntary programs to mandatory AI competency CLE — and whether that creates a new baseline for Model Rule 1.1 technology competence .
• How do court-imposed AI security requirements interact with small-firm access to AI tools? Enterprise-grade security documentation is expensive; courts conditioning AI use on SOC 2 compliance and software bills of materials may effectively restrict small-firm litigators from using AI in court filings — creating a two-tier access problem with professional competitiveness implications .

What to watch.

• Whether any state bar follows Florida's model and formalizes AI competency certification as a member benefit or CLE requirement — the Florida Bar/Clio program's rollout later in 2026 is the leading indicator .
• Whether courts begin enforcing AI security requirements as conditions on filing rather than as aspirational guidance — the first enforcement action under a protective order restricting AI tool use would crystallize the compliance standard .
• Whether bar associations or state ethics bodies issue formal guidance converting ABA Formal Opinion 512's verification obligation into specific protocols for legal-specific AI tools — given documented hallucination rates in Westlaw and Lexis platforms .
• Whether in-house legal departments begin publishing shadow AI audit results or remediation frameworks — which would establish a market standard for what adequate governance documentation looks like and create pressure on holdouts .
• Whether the EU AI Act's transparency rules, effective August 2026, generate compliance documentation requirements that become the de facto international standard for legal AI vendor assessment — affecting U.S. firms with EU-facing practices .