A roadmap for adopting generative AI without failure: how to build the “system” that transforms your operations

In this column, we explain the key points you should know when introducing generative AI, the implementation steps, and the risks and solutions you need to consider.

What Does Implementing Generative AI Mean?

Implementing generative AI is not “adopting a tool” — it is “transforming business processes.”

Implementing generative AI is not simply about starting to use tools like ChatGPT. It is an initiative to redesign “which tasks to improve, and how,” using generative AI — and to transform the business processes themselves.

When you interact with generative AI such as ChatGPT, its intelligence can make it easy to assume it is a “thinking tool” that can solve anything. However, in real business settings, simply delegating the thinking to generative AI and expecting a “good enough” answer rarely works.

It is important to view generative AI not as a “thinking tool,” but as a component that delivers value only within a properly designed business process.

Why Generative AI Implementations Often Fail

When a generative AI implementation fails, the root cause is often not the AI itself. In many cases, the issue is that teams have not clarified “how to break down human work and what to hand off to AI.”

While a lot of attention is focused on “how to use” generative AI and “what to make it do,” problems are likely to arise when analysis and design are not given enough weight—such as the actual business tasks, the knowledge and data the AI should reference, and the operational rules. As a result, the way generative AI is used can become overly dependent on individual users, and misalignment with real work can easily occur.

Implementing generative AI is not merely adopting a tool; it is a business process transformation — and designing the overall mechanism is critical.

The Two Core Goals of Using Generative AI: “Knowledge Acquisition” and “Automation”

At its core, using generative AI in business tends to pursue two directions: “acquiring knowledge and insights” and “automating work.” Rather than focusing on only one, many use cases aim to achieve both.

Here are some examples of how generative AI can be used.

• Automatically generating job postings for HR (Work automation)
  • To reduce HR workload, the HR representative simply selects relevant keywords, and the company’s job posting is generated automatically.
• Financial market analysis and insights delivery (Knowledge and insights acquisition)
  • A media company that provides financial information trains generative AI on a vast repository of accumulated financial data to deliver advanced market analysis and insights.
• Automating inspections in an automotive factory (Work automation)
  • By analyzing product images to identify defects or anomalies, quality control is improved.
• AI answering questions in live commerce (Work automation / Knowledge and insights acquisition)
  • In live commerce (selling products via social media), when viewers ask questions in chat during a live stream, AI answers in real time.
• Creating promotional emails (Work automation)
  • Generative AI automatically drafts email copy that is more likely to capture interest, tailored to customer preferences.
• Market research and campaign planning (Work automation / Knowledge and insights acquisition)
  • For marketing teams, generative AI analyzes market trends, consumer reactions, and competitor activities to develop marketing plans.

Two Key Outcomes: “Cost Reduction” and “Revenue Growth”

By leveraging AI, you can achieve cost reduction or revenue growth.

Because many use cases focus on automating complex tasks previously handled by people, it is common to target cost reduction through reduced work time and effort.

On the other hand, examples of revenue growth include using generative AI in marketing to analyze information and personalize content, thereby increasing the number of prospective customers acquired.

A Fail-Safe Implementation Process and Roadmap for Generative AI

Generative AI implementation typically follows these steps.

1. Analyze challenges and narrow down where to apply generative AI
2. Set objectives and targets
3. PoC (Proof of Concept)
4. Design and implement the system
5. Operations

Key Risks and Considerations When Implementing Generative AI

The risks of implementing generative AI can be broadly divided into two categories: security and legal risks, and risks related to system quality and operations.

Security and legal risks

• Information leakage and security risks
  • When individuals use generative AI by freely entering prompts, internal information, customer data, and non-public materials can easily become mixed in without the person realizing it. This is not a technical problem; it is a design problem—information flows are not visualized or controlled.
• Copyright infringement and compliance risks
  • Some risks cannot be fully solved by generative AI alone. Especially in areas that lead to legal liability—such as “copyright infringement” and “compliance”—it is important to design processes that include human review.

Risks Related to System Quality and Operations

• Risks of incorrect answers and poor judgment
  • Hallucinations—when generative AI produces false answers—occur when “which knowledge to base the answer on” is not defined. To reduce hallucinations, it becomes necessary to specify internal rules, up-to-date materials, and the domains where judgment is required.
• Risks of individual dependency and black-boxing in generative AI usage
  • In the operational phase, a gap tends to emerge between people who can “use generative AI well” and those who cannot. If left unaddressed, the work itself may end up depending on specific individuals, so caution is required.
• Risk of the system not being used
  • From a cost-effectiveness standpoint, a system that is implemented but not used is a business risk. One cause of “unused systems” is the lack of “operational design” at the time of implementation.

How to Think About Risk Mitigation

For issues such as information leakage, individual dependency, incorrect answers, and lack of adoption, it is important to examine causes and countermeasures from both perspectives: “the generative AI itself” and “how generative AI is used.”

In the following sections, we will explain “how to use generative AI,” including the risk mitigation perspective.

Three Elements Needed to Use Generative AI Reliably in Business

The Three Essential Elements

To use generative AI reliably in business, the following three elements are required.

• Generative AI model
• Knowledge data
• Workflow

Element 1: Generative AI Model

A generative AI model is the “brain” of AI. Well-known consumer services include OpenAI’s GPT, Google’s Gemini, and Anthropic’s Claude.

Generative AI models are characterized by their versatility: a single model can handle a wide range of tasks such as answering questions, writing, and translation. The foundation of generative AI models is the technique of “predicting the text that follows a given text.”

When leveraging generative AI models in a company’s business processes, it is common to use enterprise-oriented services. For business use, it is important to choose a service that meets the following three criteria.

Three strong candidates that meet these criteria are Microsoft Azure OpenAI, Amazon Bedrock, and Google Vertex AI. Their characteristics are as follows.

Even with these three services, the same “brains” (AI models) used in consumer services like ChatGPT and Gemini are available. For example, the “GPT” used in ChatGPT can be used as the brain of Azure OpenAI.

Element 2: Knowledge Data

When using generative AI in business, there are many cases where you want the AI to respond based on knowledge and information specific to the company and the target work.

For example, you may want to build a chatbot that answers employee questions in line with internal policies, or you may want to create emails that clearly communicate the features of your products. In such cases, you need to provide the generative AI with your own knowledge and data.

One way to do this is to embed knowledge into the prompt along with instructions, enabling unique responses. Because you can do it simply by writing it into the prompt, the difficulty is low, and you can also expect highly accurate answers.

However, this approach also has drawbacks: you need to provide the information in the prompt every time, making outcomes dependent on each individual’s approach, and prompts become longer, increasing the cost of using generative AI.

One solution is to use a technique called RAG. This method searches external data relevant to the user’s question, then provides both the search results and the user’s question to the generative AI to obtain an answer.

Element 3: Workflow

In real business settings, not all tasks can be completed with a simple “question” → “answer” flow—in other words, not everything is a task where “as long as it answers the question, it’s fine.”

For example, when a customer support representative answers a user’s question, they often go through multiple steps: understanding the question, gathering necessary information, selecting what matters, and drafting a response. In manufacturing inspections, the required follow-up may also vary depending on the type of defect found—such as sorting defective products or notifying a supervisor.

If you want generative AI to handle work that involves multiple steps or different actions depending on decisions, you need to specify task order and conditions. To achieve this, you design and define workflows that control how the generative AI behaves.

Practical Ways to Implement Generative AI

Two Implementation Approaches Based on Generative AI Maturity

There are two broad approaches to implementing generative AI in day-to-day operations.

• Approach 1: Use generative AI as-is (e.g., ChatGPT, Gemini)
  • Users enter instructions into chat-based tools such as ChatGPT or Gemini. While these tools can respond to many types of questions and requests, each user must input prompts every time.
• Approach 2: Build a system that combines a generative AI model, knowledge data, and workflows
  • By combining a generative AI model with knowledge data and workflows that control behavior, even complex tasks that previously required humans can be handled automatically. You can also standardize and automate specific work and roll it out company-wide as a tool anyone can use.

Challenges and Countermeasures for Approach 1: “Use Generative AI as-is”

With Approach 1, users need to enter prompts each time. Because prompting styles differ by user, there is a risk that know-how becomes dependent on individuals. There is also concern that users may include personal data that should not be entered, potentially leaking information to the generative AI model.

This approach can work well in early stages of adoption when usage is limited to people with high IT literacy and flexibility is valuable. However, when moving to full-scale implementation, you should also consider designing mechanisms that embed generative AI into business processes.

Challenges and Countermeasures for Approach 2: “Build a system that combines a generative AI model, knowledge data, and workflows”

When building a system that combines a generative AI model, knowledge data, and workflows, incorporating knowledge data can become a key challenge.

RAG, often used to incorporate knowledge data, is a search technology that is separate from the prompt used to instruct generative AI, and it requires its own unique design considerations.

Simply feeding “existing data” does not necessarily improve retrieval accuracy. It is essential to carefully map business tasks to the information required, and to design for your data characteristics—such as refining prompts to guide results toward the intended outcome.

Example: Using Generative AI for Internal Inquiry Handling

Let’s Use Generative AI to Streamline the Task of “Searching for and Reading Various Policies”

Let’s walk through a practical example to explain how to use generative AI.

In any company, employees have a wide range of questions and requests—for example, “I want to know the rules for taking paid leave,” “I need to understand the process for expense reimbursement on business trips,” or “My PC isn’t working well, so please fix it.” A common flow is to first read internal policies and then ask the inquiry desk. When policies are hard to understand or it is unclear where to ask, it can feel frustrating.

Let’s consider how to use generative AI to make inquiries to HR, Accounting, and IT more efficient.

We will first introduce “using generative AI as-is,” and then “building a system that combines a generative AI model, knowledge data, and workflows.”

Approach 1: “Use Generative AI as-is”

Setup and Usage Flow

The flow for using “generative AI as-is” for internal inquiries is as follows.

Pros of Approach 1

You can start immediately using existing services.
You can avoid the effort of manually referencing and carefully reading policy documents to find the sections relevant to your question.
You can use it flexibly—for example, uploading multiple documents at once, or asking follow-up questions within the same chat thread to explore solutions.

Cons of Approach 1

“Find the document → attach it in chat and ask a question” takes time and effort.
Because usage methods vary by person, there is a risk that usage becomes dependent on individuals.

Many internal documents are confidential. If you use generative AI such as ChatGPT in a typical consumer environment, that document may be used for training.
By changing the generative AI settings, you can prevent inputs from being used for training. Even so, you should avoid entering privacy-related information such as personal data. If individuals are free to use generative AI on their own, it becomes difficult to prevent such inputs—or to monitor what is being entered.

Approach 2: “Build a system that combines a generative AI model, knowledge data, and workflows”

Setup and Usage Flow

The setup and usage flow for building “a system that combines a generative AI model, knowledge data, and workflows” for internal inquiries is as follows.

Pros of Approach 2

General users within the company can use it easily.
By limiting and monitoring what can be entered, you can reduce the risk of information leakage.
By analyzing usage logs, you can improve the quality of internal services.

Cons of Approach 2

You need business process analysis and design, as well as implementation of knowledge data and the application.
Ongoing maintenance is required, such as updating the knowledge data.

Comparison of Approaches 1 and 2

What Matters Most for Successful Generative AI Implementation

The key to successful generative AI implementation lies not in “generative AI,” but in “the overall system.” By designing the end-to-end process to match your business and embedding knowledge data and AI appropriately, you can make generative AI implementation a success.