What should I require from AI suppliers on transparency, bias and training data?

Answered

Before you contract, vendors must be able to explain how they build tools, the logic and assumptions, and how outputs are generated. Ask how they identified, evaluated and mitigated risks; what data the model was trained on; and whether there are known biases. If you buy a pre-trained model, you still need to ensure any personal data used to train it meets your privacy standards. Make ATRS-level transparency an expectation in your invitation to tender — suppliers selling tools that affect the public should be comfortable with that.

From the guidance

Primary (how) Data and AI Ethics Framework

Vendors of AI systems must be able to clearly explain: the steps they take to build tools; the logic and assumptions built into their tools; how their tools generate outputs. Before entering into a contract, consider asking your supplier how they identified, evaluated and mitigated risks; what data the model has been trained on; if there are any known biases in the data or the trained model.

Section: If you’re using a third-party supplier

Read this in Data and AI Ethics Framework (opens in new tab)

Secondary (normative) Data and AI Ethics Framework

Your project might not itself gather or process personal data. For example, if you procure an AI model from a third party or use a pre-trained model. Here you must still ensure that any personal data used to train these models meets your privacy standards and complies with relevant legislation.

Section: Scoping and data acquisition

Read this in Data and AI Ethics Framework (opens in new tab)

Secondary (normative) ATRS: Mandatory scope and exemptions policy

Commercial suppliers that wish to sell algorithmic solutions to public bodies that are then used in processes that impact members of the public should be comfortable with this level of transparency that is expected of the public sector. Public bodies that are procuring solutions from vendors should make this expectation clear in their invitation to tender or other route to market.

Section: 5.3 Dealing with commercially sensitive information

Read this in ATRS: Mandatory scope and exemptions policy (opens in new tab)

Related questions