Independent Audit of Artificial Intelligence
The report examines independent auditing of artificial intelligence systems and asks under what conditions a conclusion about a specific, legally relevant property of a system can be substantiated and reproduced. Its starting point is that the statement “the system has passed an audit” says nothing about what was actually established unless the object, version, criteria and limitations are specified. The audit is therefore built from the claim to the procedure: the object, the criterion, the user of the result and the evidence capable of confirming or refuting the conclusion are defined first.
The study compares the grounds for audits in the EU, the United States and the United Kingdom: conformity assessment procedures under the EU Artificial Intelligence Act, independent audits of very large platforms under the European digital services regime, and New York’s rules on bias audits of automated employment decision tools. It also addresses standards and the choice of criteria, independence and conflicts of interest, team competence, conditions of access and the supply chain.
A large part of the report is devoted to evidentiary methodology: the provenance and preservation of evidence, data quality, sampling and test design, statistical uncertainty, bias, explainability and human oversight, information security, personal data protection and reproducibility. Using a generative legal assistant as an example, it sets out which errors count as material: presenting a party’s allegation as an established fact, relying on an inapplicable rule, or citing a non-existent source. The report stresses that finding no incident, having no access to information and confirming that an action did not occur are different states, and an audit opinion must report them separately.
The main conclusion is that a sound audit requires consistency between measurement, statistical inference and the decision on how the system may be used, and that a positive average trend does not offset an unresolved critical error. The authors see the main market risk in the gap between a narrow audit result and a broad promise: an organisational certificate, a single test and a legal review have different subjects and cannot substitute for one another.
For ARGA, the report proposes a limited pilot — testing a legal assistant that summarises case files with source references — and a minimum quality system with registers of assignments, evidence, conflicts of interest and findings. It states explicitly that this is a proposal: ARGA has not yet carried out such tests or obtained any official status, and the effectiveness of the methodology remains to be verified.
Citation Rules
- Mandatory source attributionWhen using ARGA Observatory materials, the full name must be cited.
- Date and version indicationFor analytical reports, the year of publication is mandatory.
- Link to the originalElectronic materials must be accompanied by an active link to the official website.
- Context preservationCitations must not be shortened or altered in a way that distorts the original meaning.
- Note on adaptationIf the text is abridged or translated, state: "adapted from ARGA Observatory report".
- No commercial use without written permission from the organization
- Data accuracy preservationCharts and tables must be reproduced without changes.
Ask the ARGA experts
Four short steps. We reply within one business day.