You can't rely on an AI Ethics Board

Here's why.

Okay, provocative title. So what’s my claim? I’m not suggesting well designed boards can’t purposefully serve an institution in its attempt to develop and deploy advanced information technologies at scale. What I am suggesting is that it’s a mistake to ‘over rely’ on a board. Not everything can be outsourced to such a function. It’s better (another claim I am making and have made n times before) to ‘embed’ ethical decision making into your organising structure so that it simply becomes part of how you think (shared mental model) and operate daily (organising principles, incentive structures, workflows, practices, specific tools etc.).

With the basics covered, let’s dive in (high level video here covering the ways in which I use quadratic voting, one of many potentially relevant topics for how a board might function well, providing it exists within - or external to - a broader operating structure).

Much has been written about AI Ethics Boards, what has worked well and what has worked less well. I’d say that, at least as a heuristic assessment, most work falls into the latter category.

A recent paper goes a little further, exploring more concrete recommendations for ‘how to design’ an AI ethics board.

This work covers important context and is a worthy read.

With that said, in my opinion, there’s still plenty of room to better propose how a board might ‘serve’ a meaningful purpose within a broader system for ethical decision making.

I’ve covered this topic already and will continue to do so given its importance.

An ethical decision making framework, such as the one diagrammatically represented above, serves the purpose of integrating ethical decision making workflows into an institutions organising structure. It makes the process of ethical decision making fairly consistent and relatively repeatable (recognising what can be significant variance in terms of the outputs of such a process).

Decision-making processes within an ethical decision-making system can be: 

1. Proactive as part of doing something new or making a change

2. Reactive as part of a triggering event that requires a more formal process to be undertaken

3. Proactive as part everyday work (issue backlog grooming workflow etc.)

4. Retrospective or retroactive as part of a variety of potential workflows the system describes

The general sequence is (light detail, proactive example only):

1. Something new is identified. This can be a new initiative (i.e. we want to build X) or an ‘ethical issue’ (something identified that requires attention)

2. The parties responsible for the decision-making process review existing evidence in the form of the Key Decision Log (have we explored an issue like this in the past? What deicsion was made? How, by whom etc.) and Knowledgebase (what theories might help us ground this decision making process?). This acts to both speed up, and ensure consistency of ethically oriented decisions

3. If there is clear precedent, the decision will be made (or, depending on structure, advice will be given, which might lead to a business decision), added to the Key Decision Log and used to inform future business activities (deploy, don’t, change this, alter that etc.)

4. If significant uncertainty remains, with answers not found in existing resources, additional work is done

5. If the core team (the folks stewarding the system for ethical decision making) is party to the decision, the next step is likely an internal process using a decision-making template (Consequence Scanning can be one of them. Many others exist). This template will draw on fit for purpose decision making tools (defined in the Knowledgebase and Toolkit)

5.1 A tool / process will be selected to support the decision. If decision clarity emerges from this structured process (say Consequence Scanning combined with a principles assessment, or more deliberative moral analysis led by genuine ‘experts’), the decision is logged in the Key Decision Log. Evidence of the process accompanies the decision and anything new and/or useful is added to the Knowledgebase. 

5.2 If decision clarity is not reached, the issue is added to the Issue Backlog.

6. If the core team is not party to the decision, the issue might be flagged (based on weightings, consistent criteria etc.) for review by the core team as part of their everyday workflows (this might be a team comprised with specific expertise, including moral analysis and applied ethics in the relevant context. But could also include folks adept in sociotechnical theory, organisational psychology, systems science, behavioural science etc. etc.)

7. As part of the core team’s everyday workflows, backlog grooming helps prioritise unresolved issues

8. Unresolved issues given priority can be flagged for Social Preferability Experiments (or other ‘augmenting’ workflows)

9. Social Preferability Experiments are then defined, which forms a clear picture of the requirements (work to be done, type of cohort that needs to be recruited for participation, pass/fail parameters etc.)

10. The Social Preferability Experiments are then conducted

11. The results are analysed 

11.1 If the result is a pass, the issue is resolved, the decision is logged and the data from the experiment is added to both the Key Decision Log and Knowledgebase

11.2 If the result is a fail, the issue circulates back through the same issue grooming process and will likely inform another experiment (unless this is expedited due to the issue remaining a priority. If this is the case, the learnings from the previous SPE can inform another SPE, which then follows the same process in terms of design, recruitment, running, analysis etc.)

12. The decision that’s been inclusively made and logged to the key decision log informs specific future actions

13. Then the cycle repeats, continuously, forever…

Just like Social Preferability Research, triggering events based on certain thresholds might escalate a certain decision proposal or issue up to a board (this could occur in conjunction with SP, instead of etc.). The board might then execute a process of deliberation based on various designed constraints (basically the way the board has been set up to operate and contribute). This could result in collective deliberation that leads to guidance. The process could also be blinded (the quadratic voting stuff I propose can be particularly useful here), with a collective process to follow.

The board can also serve the purpose of, if well designed and truly external to the organisation, auditing the decision making system, either in the formative stages to provide design feedback or on some ongoing basis to help assess its overall integrity and efficacy.

The board could also review and critically assess a random sample of decisions from the Key Decision Log on some regular basis (quarterly or annually). The results of such an audit could be published openly and of course, provide direct guidance back to the organisation.

The board can also engage in various activities relating to corporate governance (setting strategy, identifying and mitigating corporate risk etc.), as an external voice providing clear, documented and open guidance that can then be acted upon or not (lots of important stuff here in terms of how an org uses such guidance, which speaks directly back to the system for ethical decision making).

Such board functions exists within, and help improve, the system for ethical decision making that the organisation designs.

This is light in detail, but I hope it offers a taste as to how a board can play a significant and valued role in a broader organisational context.

The take home is that we should not expect a board to solve all of our problems. They have a role to play, but what really matters is the work we do to effectively integrate rigorous ethical decision making into our organising structure. This doesn’t happen by chance, but is possible by design.

I hope this has encouraged some useful deliberation. If you found it useful, you know what to do.