Solving AI’s ROI problem. It’s not that easy.

What do we mean by ROI for AI?

Most people probably think they know what AI is and does, but it’s a term that encompasses many technologies, processes and functions, so it’s difficult to pin down. It’s definitely not a one-size-fits all field. That can make it challenging to determine a return on investment.

In its simplest form, ROI is a financial ratio of an investment’s gain or loss relative to its cost. In other words, when you invest in AI, the benefits of your investment should outweigh the costs.

Normally, the cost is incurred in the present or the near future, while the benefits accrue at some nonspecific point in the future. However, the uncertain timing around benefit accrual is greater than the uncertainty around the expenditure’s timing. So, be sure your ROI calculation accounts for both the time value of the money invested and the uncertainty of the benefits.

That is a standard textbook definition of ROI in financial terms, known as “hard ROI.”

In the case of AI, the hard ROI you achieve can come from a number of different sources:

• Time savings: Automated intelligence that automates repetitive manual and cognitive tasks will reduce the time needed to process those tasks (e.g., invoice processing using AI).

• Productivity increase: Assisted intelligence enhances human decision-making, which can increase employee productivity. This productivity gain can come from effectiveness and efficiency (faster execution of tasks) or better decision-making (e.g., anti-money laundering compliance using AI).

• Cost savings: In some cases, time and productivity savings can reduce the number of employees required to perform the same amount of work, which can lower costs (e.g., reduced data entry operators as forms get digitized). However, this is not universal. A number of organizations directly transfer the number of hours saved into hiring fewer people and hope to see a reduction in labor costs. This may not always happen: If you reduce the time it takes an employee to perform a task by 20%, that person may find something else to do during that time. However, if you reduce the time by 80%, it will be easier to aggregate those savings into a reduction in headcount.

• Revenue increase: Assisted and augmented intelligence can provide new services that increase both the number of customers and their willingness to pay more for those services (e.g., customers buying digital subscriptions to curated and personalized music, videos and information). 

In addition to these hard returns, AI can provide a number of soft returns. They include:

• Better experience: Assisted and augmented intelligence typically provide personalization that can result in a better customer experience. Even when companies cannot monetize their personalization efforts, they can provide a better experience, and over time that can become a cost of staying in business.

• Skills retention: Acquisition, retention and an increased need for data science and AI talent are major challenges for many organizations. Nevertheless, to ensure better utilization of fast-changing AI technology, it’s becoming necessary to have data scientists who constantly explore new ideas and solutions.

• Agility: Even though some AI projects may not generate an adequate return, exploring a number of AI projects can help your data science team build new capabilities — and be agile when responding to new opportunities and challenges.

Differentiating hard and soft AI investments

You should review your company’s AI expenditures in terms of hard and soft investments. The hard investments are the cash and financial value of the resources involved in building the AI project. The soft investments that are essential to get a good return on AI investments include: 

• Data investments: The availability, permissibility, quality and accessibility of labeled data are critical factors in building your machine-learning models. Be careful not to underestimate data challenges.

• Compute and storage investments: Your data science groups can quickly run up a huge compute bill if they don’t select the right approach or don’t budget for the right computational needs of the AI project. The compute investment becomes critical as your company moves from simple AI models to more sophisticated and complex deep-learning models.

• SME investments: Subject matter experts play a crucial role in ensuring that an AI project can deliver on its promise. SME time investments are required at all stages of an AI project: scoping, building, deployment and monitoring phases. Don’t give the go-ahead for launching an AI project without considering the need for, and availability of, the right SMEs. 

• Data science training: Data science and AI are growing rapidly with a dynamic academic, open-source and industry community of practitioners. When you don’t provide the right environment, training, coaching and mentoring for your data science and AI talent, your employees’ skills can quickly become outdated. Beyond that, it’s important for all employees to be given a basic education in AI so they can understand its opportunities, benefits and challenges.

To gain a good understanding of a planned AI project, you should map out both the hard and soft aspects of your investment. Once you’ve done such a two-by-two mapping, you can quantify some of the likely benefits.

Three big mistakes when computing ROI

When companies compute the ROI on AI initiatives, they frequently make three big mistakes — ones you should guard against. 

1. Discounting the uncertainty of benefits

Some organizations do a simple ROI calculation for each AI project, taking into account the hard investments and the hard returns, but fail to consider the uncertainty associated with realizing the benefits. 

For example, let’s say you are evaluating the potential ROI of an AI system that can take a customer complaint in the form of a free-form text and predict the severity of the complaint as high, medium or low. To compute the return, you first need to know the value of each prediction and how many will be made in a year. The value is likely to come from the number of minutes saved by your customer service representative (CSR) in moving from a manual to an AI-assisted solution. 

A complicating factor is that AI models are likely to have errors, and their accuracy is probably less than 100%. So you need to estimate both the error rate and the cost of making mistakes. In order to compute the error rate, you need to compare a baseline of human performance with the AI model’s performance. Also, since the real world is messier than a training environment, any errors could be more pronounced in production.

The task of computing the cost of any errors is even more challenging. Imagine that your entry-level CSR miscategorizes a high-severity complaint — one that needs to be handled immediately — as a low-severity complaint. Did any high-value customers defect as a result, or did the mischaracterization just result in customer dissatisfaction that was later forgotten?

Another challenge is that organizations rarely have all the data they need or the processes to capture that data. Worse, they often don’t even think along these lines to compute the ROI for AI.

Hard AI investments are typically captured by estimating the number of resources, along with the hours and rate of the resources.