What should I look for when evaluating AI supplier tenders?

Answered

Use a multidisciplinary evaluation team. Look for ethics approaches, accountability for algorithm outputs, bias mitigation, reproducibility, testing under varied conditions, defined performance, security, and the skills and diversity of the delivery team. Expect approaches to evolve — AI delivery is iterative.

From the guidance

Primary (how) Guidelines for AI procurement

Robust practices may include, but are not limited to: Having an internal AI ethics approach, with examples of how it has been applied to design, develop, and deploy AI-powered solutions; Processes to ensure accountability over outputs of algorithms; Avoiding outputs that could be unfairly discriminatory; Designing for reproducibility; Testing the model under a range of conditions; Defining acceptable model performance; Robust and proportionate security provision. As part of the evaluation process, also review the specialist skills, qualifications and diversity of the team that will develop and deploy the AI system.

Section: 3. Selection, Evaluation and Award

Read this in Guidelines for AI procurement (opens in new tab)

Secondary (normative) Guidelines for AI procurement

Have suppliers highlighted and/or addressed any issues of bias within the data? Do they clearly explain why their strategies are appropriate and proportionate? Does their governance approach meet your requirements? Have the appropriate technical standards been adhered to?

Section: 9. Focus on the need to address technical and ethical limitations of AI deployment during your evaluation

Read this in Guidelines for AI procurement (opens in new tab)

Related questions