Agentic AI in action: Lessons from building the Advanced Voice Assistant

9-minute read

As organizations look for more adaptive, context-aware systems to improve decision-making and efficiency, agentic AI is emerging as a key inflection point. Organizations across industries are rapidly adopting these systems to cut costs, improve service, and free teams from repetitive work.

At Logic20/20, we’ve extended our AI strategy and implementation work to demonstrate how agentic systems perform in real-world situations. Our Advanced Voice Assistant (AVA) prototype combines natural conversation, contextual reasoning, and real-time data access to simplify complex reporting workflows. Through AVA, we’re exploring how agentic systems can handle ambiguity, apply domain knowledge, and deliver consistent results.

This article explores AVA’s development as a practical case study in agentic AI innovation. We share proven practices in use case selection, system design, data preparation, and testing that can help organizations move confidently from experimentation to enterprise-scale success.

The challenge: Rethinking complex documentation workflows

In highly regulated industries, accurate documentation underpins compliance, safety, and operational consistency. Yet completing proper incident reports, inspection summaries, and compliance forms is overwhelmingly complex. Workers must determine what details to capture, which regulations apply, and how to properly document events while navigating hundreds of pages of dispersed regulatory guidance. The outcome is inconsistent records and lost insights that never make it into the system of record.

That challenge led to the development of AVA. The idea took shape after our COO spoke with an enterprise leader who described the burden of preparing incident and compliance reports—lengthy submissions that sometimes required entire committees to prepare. Workers often filled in only brief notes, adding verbal explanations later and leaving supervisors to reconstruct critical information. The process was inefficient and full of gaps.

Digital forms and workflow tools offer only partial relief. They may standardize data entry, but they still depend on users to know what to include and how to describe it. These tools also miss valuable context, such as reasoning or tone, that supports incident analysis and compliance review.

AI offers a practical path to improving documentation accuracy and efficiency. A conversational, domain-aware system can guide employees through documentation in real time, ask clarifying questions, recognize patterns, and ensure each record is both complete and contextual.

Voice interaction transforms this into a natural dialogue: workers describe incidents conversationally while the AI asks follow-up questions, probes for missing details, and confirms understanding—all through spoken conversation rather than forms and typing. By replacing blank forms with guided voice interviews, AI cuts documentation time from hours to minutes while dramatically reducing mental burden. Workers no longer waste cognitive energy searching through regulations or struggling with forms. They simply have a conversation and return to their core operational responsibilities, while the AI handles the complex documentation and compliance requirements.

Article continues below.

Use case selection: Aligning business needs with AI strengths

From the start, our team saw guided incident reporting as a strong fit for agentic AI. The process demanded detailed documentation, domain-specific knowledge, and timely accuracy—conditions well suited to an intelligent, conversational system.

But understanding what “intelligent” truly meant took iteration. Our first prototype was straightforward: a voice interface that asked questions and filled out forms as users spoke. When we demonstrated this to the client, their feedback was clear: They didn’t need a voice-enabled form; they needed a smart helper that could understand their regulatory requirements, identify missing information, and guide workers through complex documentation. Without this intelligence, workers would still be left searching through documents and potentially missing critical details.

This feedback transformed our approach. The client wanted a system smart enough to understand context from a worker’s initial description, search through their regulatory documents, and ask intelligent follow-up questions based on what actually happened. Not a rigid questionnaire, but adaptive questioning that could identify gaps and probe for missing information specific to each incident. This requirement shaped AVA from a simple voice interface into a truly agentic system—one that actively thinks about what information is needed rather than just transcribing what’s said.

Our collaboration affirmed that guided incident reporting reflects the core conditions for effective agentic AI:

• Complex and ambiguous information: Hundreds of pages of regulatory documents that workers can’t realistically memorize or quickly reference
• Domain-specific knowledge requirements: Incident-specific documentation where different situations demand different questions and compliance standards
• Tasks requiring real-time interaction: Capturing details while they’re fresh, with adaptive questioning that improves speed and accuracy
• Workflows that depend on human expertise and benefit when AI supports and amplifies informed judgment

When these elements align, agentic AI streamlines knowledge-intensive workflows, strengthens data quality, and empowers teams to focus on higher-value tasks.

Building the prototype: Speed, flexibility, and iteration

AVA’s development began with a focused goal: to prove that voice could simplify form filling. In two weeks, we built V1: a functional system where users could speak their answers while watching forms fill in real-time. It used the OpenAI Whisper model for speech recognition and OpenAI’s APIs for language processing, creating a working voice interface for structured data entry.

When we demonstrated V1 to the client, they were excited about the voice capabilities, but their feedback reaffirmed the need for true intelligence—guidance, not transcription—driving the shift toward a more adaptive second version. This insight pushed us toward V2, which took another two weeks for the initial version. V2 introduced agentic RAG capabilities, expanding AVA’s role into a context-aware assistant.

The team chose to write code directly for this prototype to allow full control over data flows and model behavior. While low-code platforms are ideal for rapid prototyping and integration, AVA’s multi-agent architecture requires deep customization. We needed agents that could adapt to conversations, search documents, and formulate intelligent questions based on context. With code, we could evolve this multi-agent system quickly and flexibly, pivoting in any direction without fighting platform limitations.

As the prototype matured, AVA advanced from a basic question-and-answer format to a context-aware interview tool. It learned to probe for missing details, clarify ambiguous responses, and adjust its questions based on both the conversation and the relevant regulatory requirements it had discovered. Each iteration improved AVA’s responsiveness and contextual accuracy.

Article continues below.

Screenshot of Logic20/20’s Advanced Voice Assistant (AVA) in action

Integration and architecture: Keeping it lightweight and scalable

AVA uses a multi-agent architecture where each agent handles a specific task: speech recognition, language understanding, document searching, and question formulation. This design delivers two key advantages: agents can be tested independently for reliability, and focused agents perform better than generalist ones. When agents try to handle too many tasks, their performance degrades due to attention limitations. By keeping each agent specialized, we maintained consistent performance across the system.

For speech recognition, we used Whisper, an open-source model that reliably transcribes natural conversation. For natural language processing and generation, we integrated OpenAI APIs for both text and voice to enable smooth dialogue. Managing natural voice conversations required careful orchestration, ensuring the system knew when to listen and when to respond and avoiding the awkwardness of talking over users. Together, these tools support multi-turn interactions while keeping performance efficient and costs manageable.

Processing regulatory data presented a separate challenge. AVA needed to read large, inconsistent documents while preserving context. To achieve that, we built a custom data-cleaning pipeline that extracted text from PDFs, including tables and diagrams, and converted it into structured content. Images containing critical information were automatically captioned to ensure nothing was lost.

Once the data was standardized, AVA applied a multi-agent approach using retrieval-augmented generation (RAG) where specialized agents handle different aspects: searching documents, ranking relevance, and formulating context-aware questions based on the conversation and findings. This design allows the system to locate relevant content quickly and understand relationships across multiple documents, whether it is matching terms semantically or by keyword.

These decisions kept AVA lightweight and scalable. Its architecture minimized dependencies, stored data locally during development, and abstracted key components so future upgrades—like connecting to enterprise databases or swapping models—could happen with minimal rework. The result is a flexible foundation built to evolve with new AI capabilities.

Testing and iteration: Challenges and insights

Testing AVA proved as instructive as building it. Early testing relied on manual reviews—running repeated conversations, checking transcripts, and refining prompts by hand. With conversations running 15 questions or more, manual testing quickly became impractical. This approach helped surface bugs and design issues quickly but also highlighted a limitation: manual QA can only go so far when testing systems capable of complex reasoning. In highly regulated domains like incident reporting, even skilled testers couldn’t predict every question AVA might ask or every possible user response.

To address that gap, the team developed a lightweight AI-driven testing framework that could simulate realistic conversations. Using the same documentation library available to AVA, the framework generated test cases with hypothetical incidents, varied responses, and context changes. This synthetic testing system could adapt its responses and hold meaningful conversations with AVA, allowing us to validate the system’s behavior at scale. These synthetic exchanges made it possible to stress-test AVA’s logic, evaluate its handling of specialized language, and identify areas where additional context or clarification was needed.

The multi-agent architecture helped here: we could test individual agents in isolation before validating the complete system. Even with those advances, testing an agentic system remains inherently complex. Unlike traditional software, AI agents can generate multiple valid answers to the same input, depending on context, model version, or randomness in output. This variability means traditional pass/fail tests can’t fully capture performance quality. Future versions of AVA will use formalized evaluation frameworks (“evals”) to measure factors such as conversational depth, factual accuracy, adaptability, and user satisfaction, offering a fuller picture of real-world performance.

These lessons reinforced a key principle of agentic AI development: testing doesn’t end once the system reaches functional stability. As models and data evolve, continuous validation is key to maintaining accuracy, reliability, and trust.

Lessons learned

Building AVA advanced our approach to applying agentic AI from concept to practical use. While each implementation will differ, several principles emerged that can help teams design and deploy effective systems.

Start with a focused use case

Agentic AI delivers the greatest value when applied to a clearly defined problem. Beginning with a contained workflow—such as guided incident reporting—makes it easier to show results early, refine quickly, and scale with confidence.

Understand the problem deeply through rapid delivery and fast iteration

Rapid iteration with AI enabled us to deliver V1 in minimal timeframe, which generated crucial client feedback. This quick delivery revealed that voice transcription alone wasn’t enough – they needed intelligent guidance. Equally important was examining the actual data: naive PDF parsing would have lost critical tables, diagrams, and text within images. Understanding both the user’s need and the data complexity shaped our approach. Fast iteration cycles let us discover requirements we couldn’t have anticipated upfront.

Specialized agents outperform generalists

Limiting each agent’s scope improved accuracy and made troubleshooting easier.

Design for modularity and flexibility

Design with adaptability in mind so components can evolve as models and frameworks change.

Prioritize clean data and domain context

Reliable outcomes depend on high-quality input. Well-structured data and strong domain alignment give the AI the context it needs to generate accurate, useful results.

Design for the human experience

In voice-based systems, usability drives adoption. Accounting for pauses, clarifications, tone, and timing helps interactions feel intuitive and natural instead of mechanical.

Validate early and often

Early testing uncovers usability and performance issues before they become costly. Ongoing validation ensures the system remains accurate, relevant, and aligned with changing user needs.

Testing agentic systems requires new approaches

Traditional testing methods fall short for adaptive systems, so we developed AI-driven simulations and plan to add formal evaluation frameworks in future releases.

Anticipate the realities of voice AI

Real-world conditions introduce complications such as background noise, overlapping speech, and variable model behavior. Building in guardrails and fallback options early prevents confusion and builds user trust.

Together, these lessons offer a practical framework for developing agentic systems that deliver measurable business value—solutions that are flexible, dependable, and designed around the people who use them.

Where intelligence meets intention

Agentic AI signals a shift in how we design technology to work alongside people. It enables the creation of systems that interact, interpret context, and adapt to complex situations, extending human capability in the process. Developing AVA shows that realizing this potential takes both imagination and discipline, balancing experimentation with structured design, and maintaining focus on human experience.

The next challenge for organizations lies in guiding these systems responsibly. Effective adoption depends on setting clear boundaries and shared accountability—defining which decisions AI should make, how transparent those decisions must be, and where human judgment remains essential.

Ultimately, we’ll define the future of AI through the choices we make in aligning intelligent systems with human goals and values.

You might also be interested in …