QKTech Helps Leading Indian Insurer Evaluate its Gen AI-powered Chatbot
Industry & Segment
Insurance
Objective
To evaluate and enhance the accuracy, reliability, and performance of the insurer’s GenAI‑powered chatbot by rigorously testing its query understanding, context retrieval, response quality, and RAG‑based SQL generation to ensure a seamless, factual, and customer‑centric experience.
Platform
Nimbus AI
Overview
QKTech, a global leader in AI-powered reliability and quality engineering, collaborated with a leading Indian insurer to optimize the performance of its customer-facing AI-powered chatbot. The chatbot, powered by generative AI complemented with Retrieval-Augmented generation (RAG), aimed to enhance customer experience by providing instant, accurate, and personalized responses to inquiries related to our client’s insurance products and services.
Client Overview
A leading Indian life insurance provider offering term and whole life protection, savings plans for education/marriage goals, retirement income products, and health coverage. The company drives digital transformation through AI and machine learning, deploying chatbots for real-time customer support, automated claims processing, and personalized policy recommendations. Its multi-channel network includes agent partnerships, online platforms, and bancassurance collaborations, supported by regional-language tools and mobile apps to ensure accessibility across urban and rural markets. Focused on innovation, it prioritizes inclusive growth and operational agility.
Business Objectives & Challenges
To improve customer experience and personalization for their product, our client designed an AI-powered chatbot harnessing generative AI and RAG to instantly and accurately answer customer queries about its range of products. Our client wanted to failproof the effectiveness and accuracy of the AI-powered chatbot while identifying areas of improvement to ensure it delivers value to their end users and augments their digital-first business vision.
The QKTech Strategy: AI-Augmented QE Framework for Chatbot Testing
After thoroughly assessing our client’s requirements and building on our in-house quality engineering of AI framework, we developed a comprehensive plan to assess the chatbot across six key performance dimensions, then deploy a two-phase testing approach to ensure peak performance, accuracy, and continuous improvement.
Chatbot Testing Performance Dimensions: QKTech’s chatbot testing framework defines six key areas to assess the quality engineering of AI-powered chatbot performance.
Query Understanding
This facet of quality engineering of AI deals with assessing the chatbot’s ability to interpret and classify user queries accurately, forming the foundation for delivering accurate and personalized responses.
Information Retrieval
The RAG-powered AI chatbot was tested for seamless data retrieval from knowledge bases and RDBMS tables, with rigorous evaluation of system integrations like CRM and AI response accuracy for both customer-facing and internal queries.
Response Generation
The final response generated through the interaction of RAG and LLM models required thorough evaluation to assess its accuracy, quality, and relevance to user queries.
Conversation Flow
Along with its responses, the AI chatbot was expected to maintain a natural and engaging flow, aligned with the brand’s customer-centric tone and style of communication.
Error Handling
To ensure optimal performance, the AI chatbot needed to be tested for managing exceptional cases, involving unexpected inputs, ambiguous queries, and out-of-scope requests.
Integration with Insurer’s Information Database
The AI chatbot’s success depended on its effective integration with the insurer database to pull and display correct data for generating relevant and accurate responses to user queries.
Two-phase testing approach
QKTech tested the functional performance of the Generative AI-powered chatbot using a two-phase testing approach, including test data generation and test evaluation.
Phase 1: Test Data Generation
The test data generation phase was divided into 3 different stages:
Data Asset Creation
This step focused on creating a comprehensive dataset to test the Generative AI-powered chatbot. To accomplish this, we created a test dataset that comprised documents with annotated data points on questions, their corresponding expected answers, and validation data created by an AI data specialist. To ensure comprehensive test coverage, we used appropriate LLM models to generate multiple variations of the initial questions.
The human-preferred validation dataset, designed to provide the guardrails for the chatbot to refine its responses, was structured into three distinct classifications:
- Positive examples showcasing questions with relevant answers in the documents
- Negative examples showcasing questions with no corresponding answers in the documents
- Edge cases tested the chatbot’s ability to handle ambiguous, open-ended, or reasoning-based queries.
Context Retrieval & Response Generation
The generated test data (user queries) was leveraged by an AI engineer to interact with the chatbot. For each user query, the AI engineer performed the following activities:
- Triggered the chatbot to retrieve relevant document chunks (context) from the provided documents
- Captured the retrieved context for later analysis
- Instructed the chatbot to generate a response to the query
Test Data Preparation
Conducting the activities to test the chatbot for its retrieval and response efficiencies, the AI engineer compiled the test data that included:
- Original query
- Retrieved contexts
- Generated responses
- Expected answers
Phase 2: Evaluation
The evaluation phase of the AI chatbot testing involved assessing the generated responses using intelligent automation in conjunction with manual verification to ensure scalable, accelerated, and accurate quality engineering. This phase also included testing the SQL queries generated by the chatbot’s RAG system to ensure optimal and reliable AI performance.
AI-driven Evaluation
The first step in this phase used an AI model evaluation platform to calculate the following metrics for each validation set:
- Context Relevance: The metric assessed whether the retrieved document chunks (contexts) were relevant to the query and contained information to answer the question. Using the platform each output was labelled as relevant or irrelevant.
- Groundedness/Hallucination: This metric determined if the generated response was based on factual information from retrieved contexts or fabricated information created due to AI hallucination. The platform marked each output as factual or hallucinated in this context.
- Answer Correctness: The metric evaluated whether the generated response matched the expected answer and labeled each output as correct or incorrect.
Results
424
total test cases executed
89.2%
answer correctness accuracy
93.9%
factual accuracy (grounded responses)
94.3%
context relevance accuracy
6.1%
hallucination rate
Our Impact
Correctness Level Below Benchmark
The chatbot’s answer correctness accuracy was benchmarked below target, at 89.2%.
High Factual Accuracy
93.9% of the responses generated were grounded in facts.
Pertinent Context Relevance
The AI chatbot registered a 94.3% accuracy in retrieving relevant contexts from the documents.
Low Hallucination Rates
Only 6.1% of responses contained hallucinated information.
Recommendations
Based on the evaluation we made the following recommendations to our client:
- Improvement in Answer Correctness: The chatbot generated 5.9% incorrect and 5% partially correct answers, missing the industry benchmark of greater than 90% correctness.
- Elimination of Hallucination: We identified several instances of AI hallucination during our evaluation, where the chatbot provided fabricated information outside the scope of the provided documents. This creates a serious risk of customer misinformation, impacting satisfaction and potentially resulting in financial losses due to misrepresentation. We recommended our client prioritize the elimination of these hallucinations to mitigate these risks.
- Accelerate Query Responses: The chatbot API took 10 seconds to respond to user queries, missing the industry-set benchmark of 5 seconds. We recommended our client reduce the query response time to 5 seconds or lower to avoid user frustration and drop-offs.
This case study showcases how QKTech’s comprehensive and intelligent quality engineering framework empowered a leading insurance company to failproof their AI chatbot performance. By providing detailed recommendations and insights on key chatbot performance parameters, we helped the client deliver enhanced customer experiences and optimize AI deployment to meet their business goals. The success of the project underscores the need for rigorous evaluation of RAG systems before deployment.
At QKTech, chatbot quality engineering sits under Reliability of AI, one of the six pillars of our Reliability OS for Enterprises, which helps enterprises ship AI agents and modern platforms without the risk.
Preparing to launch a GenAI chatbot? Schedule a free consultation with our experts.
Let’s build experiences that matter. Connect with our experts today.