Visualizing AI Tool Effectiveness. A Practical Guide to Designing Marketing KPIs

The use of AI in marketing—from AI chatbots to image generation—is on the rise. However, many marketers grapple with the same question after implementation: “How do we measure the results?”
This article explains the basics of setting KPIs to accurately measure the impact of AI tools, along with key points to consider specifically for AI adoption.

What is a KPI?

Definition of KPI

A KPI is a numerical target that serves as the key to making your marketing strategies successful.

Where KPIs Fit In

Marketing is the activity of “making products and services sell.” Furthermore, selling products and services is essentially the core of business activity itself.

To understand KPIs, it is important to grasp the following three elements and how they relate to each other.

How to Create KPIs

Overview of the Steps

There are five steps to creating KPIs.

1. Confirm the KGI
2. Examine the CSF
3. Determine the KPI
4. Set Targets
5. Operation

Confirming the KGI

It is effective to confirm your KGI—the goals directly tied to business results—by looking at them from the following perspectives.

• Public Business Goals and Strategies
  • Sales targets and basic strategies are often explained in publicly available materials, such as integrated reports. You should check these documents to reaffirm your understanding.
• Business Purpose
  • Confirm the business objective of the product or service for which you are implementing marketing measures. Clarify what value you provide to customers and what your unique differentiation points are.
• Stakeholders and Distribution Channels
  • If you sell directly to the final buyer, the process is simple. However, many cases involve multiple stakeholders, such as selling through agencies or retailers. Clarify who is involved in your marketing initiatives and what their intentions are.
• Purpose of the Initiative
  • Confirm the purpose of the initiative by asking: Who are we talking to? What and how are we communicating? What do we want the target audience to do? While objectives are often achieved through a combination of various elements, it is important to focus specifically on the purpose of this initiative. Asking yourself, “What would happen if this initiative didn’t exist?” helps clarify the value of carrying it out.
• Quantification
  • Based on these confirmations, clearly define the KGI as a numerical value. Once done, verify that you are aligned with all stakeholders.

Examining the CSF

To achieve your KGI through marketing activities, you need your target audience to reach the “state desired by the company.”
It is vital to spark a change in their mindset or behavior—whether that implies recognizing your brand name, becoming interested in a product, visiting a retail store, or making a purchase on an e-commerce site.
Organize your customers’ actions into a process. Create a “customer journey map” that outlines their purchasing behavior to clarify their pain points and needs.
From there, narrow down the critical processes required to achieve the KGI.

Determining the KPI

Once the CSF is determined, translate it into measurable KPI metrics.

For instance, if your CSF is to “increase purchase intent,” your KPIs might be the “landing page click-through rate” or the “number of whitepaper downloads.”

Setting Targets

Next, determine the numerical targets for your KPIs.

For example, if a B2B inside sales team is working to secure appointments with clients, the activity process and the method for calculating targets would look like this:

• Process
  • Select Target → Call → Appointment → Sales Qualification (Opportunity) → Sales Close (Order)

• Formula for Sales Generated by Inside Sales
  • Sales from Inside Sales = Number of Appointments * Qualification Rate * Close Rate * Unit Price

• Formula for Number of Appointments
  • Number of Appointments = Sales from Inside Sales / (Qualification Rate * Close Rate * Unit Price)

• Sample Calculation for Appointments
  • Conditions
    • KGI: 100 million yen in sales generated by inside sales
    • Unit Price: 1 million yen
    • Appt to Opportunity Rate: 20%
    • Opportunity to Close Rate: 20%
  • Number of Appointments
    • 100 million yen / (20% * 20% * 1 million yen) = 2,500 appointments

Operation

Measure your KPIs, and based on the results, review your CSF processes and resource allocation.

Key Points for Planning KPIs

As you work on your KPIs, things may not always go according to theory, and you might face some difficulties. When that happens, keep the following points in mind.

Select Actionable KPIs

Decide clearly and specifically in advance “who will take what action using this report.” This ensures you create practical KPIs that are actually useful, rather than just nice-looking plans that never get used.

Set Targets Before Launching

Setting targets for a new activity is difficult. You might be tempted to think it is okay to start without targets and decide on metrics later. However, doing so makes it impossible to evaluate whether the initiative was a success or a failure. Even if you lack sufficient data, use hypotheses to set target numbers before you begin.

Narrow Down Your Focus

To take action based on your KPI data, it is better to limit the number of metrics you watch. Select just one to three KPIs to focus on specifically.

Prioritize Outcome Metrics

“Action metrics” measure whether you did something, while “outcome metrics” measure whether you achieved something.

Since KPIs measure the “process toward achieving the KGI,” technically, either type is acceptable.

However, in digital marketing, you can measure the results of almost every activity. Outcome metrics allow you to gauge the situation more objectively than action metrics, making them the better choice.

Clarity

It is also important to choose metrics that make it easy to visualize problems and solutions.

For example, when you first start email marketing, customers aren’t used to receiving promotional emails from you yet, and your staff is likely still experimenting with writing styles. In this situation, tracking the percentage of customers who unsubscribe (opt-out rate) can help you clearly identify early-stage issues, such as problems with the email content or delivery timing.

Key Considerations When Setting KPIs for AI Tools

Don’t Get Distracted by “AI Effectiveness”—Return to the Basics

Regardless of whether you use AI or not, the most important thing is to confirm: “Does introducing AI actually improve the results for our original goal?”

For instance, if your main goal is to strengthen customer engagement, it would defeat the purpose if using AI caused a negative reaction.

When planning your KPIs, set aside the fact that you are using an AI tool for a moment. Start by focusing on the original purpose of your strategy.

What Metrics Accurately Measure “AI Effectiveness”?

AI has specific areas where it excels, such as generating and analyzing large volumes of content. To assess the true value of adopting AI, select KPIs that confirm whether you are effectively utilizing these strengths.

Unique Risks of AI

Current AI usage often serves as an experiment for future Digital Transformation (DX). Therefore, projects often start with the understanding that there will be some level of risk involved.

To further advance DX in the future, it is crucial to measure these risks. For example, setting accuracy targets for Generative AI responses and monitoring whether they are being met is an effective approach.

Case Study: Creating KPIs for AI Tools (AI Chatbots)

With the basics of KPI planning in mind, let’s explore how to determine KPIs when implementing AI tools.
Let’s consider what specific KPIs to set when operating an AI chatbot.

Step 1: KPIs to Measure the Chatbot’s Purpose

Regardless of whether you are using AI or not, you should consider KPIs based on the purpose of the chatbot itself. Follow the basic steps and start by confirming the KGI.
For example, if you have a chatbot handling registrations for events like seminars, the following items would be good candidates for KPIs.

• Customer Satisfaction (CSAT)
  • The degree of customer satisfaction with the chat support (e.g., average rating from surveys).

• Conversation Success Rate
  • The percentage of times the bot accurately understood the user’s request and provided a satisfactory answer.
    (Formula: Number of conversations where user goal was achieved / Total conversations)

Step 2: KPIs to Measure AI-Specific Impact

Consider the benefits you can expect specifically because you are using AI.

• Deflection Rate
  • The percentage of inquiries resolved solely by the AI chatbot without escalating to a human operator.
    (Formula: Number of inquiries resolved by chatbot / Total inquiries)

Note that the “Conversation Success Rate” from Step 1 is also effective for measuring the effectiveness of AI.

Step 3: KPIs to Measure AI-Specific Risks

The biggest risk with AI chatbots is the tendency to tell “plausible lies” (hallucinations). Furthermore, issues such as whether the bot actually provided an answer or if the response speed was too slow may also arise. Therefore, you should consider the following items for your KPIs.

• False Answer Rate
  • The percentage of times the AI gave an answer contrary to the facts. Conduct periodic fact-checks of responses to measure the number of false answers.
    (Formula: Number of false or misinformation answers / Total answers)

• Failure Rate (No-Answer Rate)
  • The percentage of times the AI chat could not answer the user’s question appropriately (e.g., it gave up or switched to human support).
    This serves as an indicator for improving scenarios and training the AI.

• Response Speed
  • How quickly the system reacts to user input.

List of KPIs for AI Chatbots

Based on the points discussed above, here is a summary of example KPIs for AI chatbots.

Examples of AI Tool KPIs

Let’s consider KPIs to measure the AI tools you are using in your company. Below are examples of KPI settings for AI tools used in social media content generation and advertising analysis.

Use the three perspectives introduced earlier to set the KPIs that best suit your company.