AdapData

Established 2015 Notes

Notes · I Strategy advisory

What an AI readiness assessment should test

In short

An AI readiness assessment establishes whether an organisation can adopt artificial intelligence in a specific process and operate it afterwards. A useful assessment tests five things: whether the task can be specified precisely enough to be judged right or wrong, whether the required data exists and may lawfully be used, who reviews the output, what the work costs to run at volume, and who answers for errors. An assessment that produces only a maturity score has tested none of them.

Note

What is an AI readiness assessment?

It is an examination of a named process, conducted before commitment, to establish whether artificial intelligence can be applied to it and sustained afterwards. The unit of assessment matters. An organisation is not ready or unready in general; it is ready for a particular task, with particular data, under a particular review arrangement.

The related term, artificial intelligence due diligence, describes the same examination performed for a third party in a transaction rather than for oneself. The questions are largely the same. The difference is that the party asking has less access and less time.

Can the task be specified precisely enough?

This is the question most assessments skip, and the one that determines the outcome. A task can be automated with confidence when two competent people, given the same input and the same instructions, would produce the same output, and a third could say which of them was wrong.

Where that condition does not hold, the difficulty is not technical. It is that the organisation has never written down what a correct result looks like. Discovering this is a valuable outcome in itself. The specification usually has to be written before anything else proceeds, and writing it improves the process whether or not any technology is subsequently adopted.

Does the data exist, and may it be used?

Data readiness has three components, and organisations tend to assess only the first. Does the material exist in sufficient volume. Is it accurate enough for the purpose, which is a different question from whether it is complete. And is it lawful to use for this purpose, given the terms under which it was collected and the consents on record.

The third component defeats more projects than the first two. Customer records gathered for the delivery of a service are not automatically available for training a model, and a supplier’s standard terms may commit the organisation to less than it assumes. This should be established in writing before work begins rather than during a later review.

Who reviews the output, and on what basis?

Every process of this kind produces a proportion of results that are wrong. Readiness means having decided, in advance, who examines them, against what standard, and with what authority to reject.

Two arrangements fail predictably. Review by the same person who would otherwise have performed the task, who has no time budget for it and quickly begins approving by default. And review by a person with no authority to stop the process, whose objection is recorded and overruled. A workable arrangement gives the reviewer time, a written standard, and the ability to halt. It also samples the results that were approved, since the errors that matter are the ones nobody flagged.

What does it cost to serve at volume?

A pilot is cheap. The relevant figure is the cost of one unit of output at the volume the process actually runs, including the model, the retrieval, the storage, the failed attempts and the human review attached to it.

Two effects are commonly missed. Review time is a real and recurring cost, and it does not fall as volume rises unless the sampling rate is deliberately reduced. And provider pricing is not fixed; a system whose economics work only at present prices carries a commercial exposure that belongs on the risk register rather than in a footnote.

Why do most readiness assessments produce nothing?

Because they assess the organisation rather than a process, and they conclude with a score. A score cannot be acted upon. It generates a training programme, a centre of excellence and a further assessment in twelve months.

An assessment worth commissioning ends with a small number of named processes, each marked ready, not ready, or ready once a stated condition is met, and each accompanied by the arithmetic that would make it worth doing. That document is shorter than the usual deliverable and considerably harder to write.

Questions

What is an AI readiness assessment?
An examination of a named process, conducted before commitment, establishing whether artificial intelligence can be applied to it and operated afterwards. It tests specification precision, data availability and lawfulness, review arrangements, cost to serve and accountability.
What is a readiness assessment?
An examination conducted before commitment, establishing whether an organisation can adopt a defined change and sustain it afterwards. Applied to artificial intelligence, it should be scoped to named processes rather than to the organisation as a whole.
What are the three pillars of AI readiness?
Published frameworks usually give data, technology and people, sometimes with governance as a fourth. The headings are unobjectionable and insufficient, since none of them establishes whether the task can be specified precisely enough for an output to be judged right or wrong.
What is AI due diligence?
The same examination conducted on a third party, usually in a transaction. It establishes what the target’s systems do, on what data, under whose licences, at what cost per unit of output, and with what accountability for results that are wrong.
How long does a readiness assessment take?
Three to five weeks for a small number of processes. Most of the time is spent establishing what a correct output looks like and whether the data may lawfully be used, rather than examining technology.
What is the most common blocking finding?
That the task has never been specified well enough for anyone to judge an output right or wrong. The second most common is that the data required exists but was collected under terms that do not permit this use.
Is a maturity score useful?
Rarely. It describes an organisation in general when readiness is a property of a specific process with specific data. A score tends to produce further assessment rather than a decision.
What is the 30 per cent rule in artificial intelligence?
There is no established rule of that name. The phrase is used loosely, most often for the claim that roughly a third of the tasks in a role might be automated, and sometimes as an error threshold. Neither use has an agreed definition, and neither belongs in a business case.

Engagement

We are usually engaged by a board or an owner who has received a readiness report and wants to know whether it establishes anything. Research and technical origination are carried out by our research partner. Engagements are senior-led and conducted in confidence. We reply personally to every enquiry; write to [email protected].

Related