Build a Voice AI Agent From Scratch

Voice AI Agent From Scratch

Voice AI Agents are becoming one of the most useful ways to apply Generative AI. When I started learning how to build a Voice AI Agent from scratch, I noticed that most tutorials relied on paid APIs or only covered a single part of the system, like speech recognition or text-to-speech.

A true Voice AI Agent does more than just add a microphone to a chatbot. It listens to the user, understands what they say, reasons with an LLM, decides if it needs a tool, uses that tool, and then replies in a natural voice.

In this article, I’ll walk you through how I built the whole workflow using only free and open-source tools.

How to Build a Voice AI Agent

Before I write any code, I like to first understand how data moves through the whole system.

Our Voice AI Agent will have five main components:

  1. Microphone Input: Records the user’s voice.
  2. Speech-to-Text: Converts audio into text.
  3. LLM Reasoning: Understands the request and decides what to do.
  4. Tool Calling: Executes Python functions when external information is required.
  5. Text-to-Speech: Converts the AI response back into voice.

For this project, I will use the following open-source tools:

  1. Faster Whisper for speech recognition
  2. Ollama for running an LLM locally
  3. Llama 3.2 as our language model
  4. Piper TTS for text-to-speech
  5. SoundDevice for recording audio

You could use LangChain or LangGraph for this project, but learning the basic workflow first will make it much easier to pick up agent frameworks later.

Building a Voice AI Agent From Scratch

First, install the Python libraries:

pip install faster-whisper sounddevice scipy ollama

We also need Ollama to run the LLM locally.

After installing Ollama, download the Llama 3.2 model:

ollama pull llama3.2

For text-to-speech, we will use Piper. You can install Piper using:

pip install piper-tts

Step 1: Recording the User’s Voice

The first task is to capture audio from the microphone. Here is the function I used:

import sounddevice as sd
from scipy.io.wavfile import write

def record_audio(filename="audio.wav", duration=5):
    sample_rate = 16000

    print("Listening...")

    audio = sd.rec(
        int(duration * sample_rate),
        samplerate=sample_rate,
        channels=1,
        dtype="int16"
    )

    sd.wait()

    write(filename, sample_rate, audio)

    print("Recording complete.")

The sounddevice library records audio straight from your microphone. I use a sampling rate of 16,000 Hz since it works well with speech recognition models.

Right now, the agent listens for five seconds at a time. I kept it simple on purpose.

For production Voice AI systems, I usually prefer Voice Activity Detection (VAD). VAD records only when the user is speaking, instead of using a fixed time. But for a first version, fixed-duration recording is much easier to follow.

Step 2: Converting Speech to Text with Whisper

Now we need to convert the recorded audio into text. For this, I used Faster Whisper:

from faster_whisper import WhisperModel

whisper_model = WhisperModel(
    "base",
    device="cpu",
    compute_type="int8"
)

def transcribe_audio(filename="audio.wav"):
    segments, info = whisper_model.transcribe(filename)

    text = " ".join(
        segment.text for segment in segments
    )

    return text.strip()

Faster Whisper is a faster, optimized version of OpenAI’s Whisper model.

Here, I am using the base model. You can also use models such as:

tiny
base
small
medium
large-v3

Bigger models usually give better transcription quality, but they also need more computing power.

If you’re just starting out and using a regular laptop, I suggest beginning with the base or small model.

Another important parameter is:

compute_type="int8"

This setting uses less memory and makes it easier to run on CPUs.

When I use local AI models, I always start with the smallest one that gets the job done. Many people download the biggest model right away and then wonder why their app runs slowly.

You should always pick your model based on your hardware and what your app needs.

Want to build more practical AI Agents? My book Hands-On GenAI, LLMs & AI Agents teaches Agentic AI through real-world, hands-on projects.

Step 3: Creating Tools for the AI Agent

An AI Agent is most helpful when it can actually do things. Let’s make two simple tools:

from datetime import datetime

def get_current_time():
    return datetime.now().strftime("%I:%M %p")


def calculate(expression):
    try:
        return str(eval(expression))
    except Exception:
        return "Unable to calculate the expression."

Our agent can now:

→ Check the current time
→ Perform calculations

In a real application, tools could connect to:

  • Databases
  • Weather services
  • Email systems
  • Calendars
  • Search engines
  • Internal company APIs

Learning how to call tools is one of the most important parts of working with AI Agents.

The LLM doesn’t run these actions itself. It just decides which tool to use, and then our Python app runs the right function.

Step 4: Connecting the Voice Agent to an LLM

Now we will connect our application to Llama 3.2 using Ollama:

import ollama
import json

tools_description = """
Available tools:

1. get_current_time
Use this when the user asks for the current time.

2. calculate
Use this for mathematical calculations.

If a tool is required, respond ONLY with JSON:

{
    "tool": "tool_name",
    "argument": "tool_argument"
}

If no tool is required, respond normally.
"""

This prompt explains the available tools to the model. Now let us create the reasoning function:

def ask_llm(user_text):
    prompt = f"""
You are a helpful Voice AI Agent.

{tools_description}

User request:
{user_text}
"""

    response = ollama.chat(
        model="llama3.2",
        messages=[
            {
                "role": "user",
                "content": prompt
            }
        ]
    )

    return response["message"]["content"]

The LLM gets the user’s request as text and decides if it can answer directly or if it needs to use a tool.

Step 5: Executing the Tool

Now we need to process the LLM response:

def process_response(response):
    try:
        tool_call = json.loads(response)

        tool_name = tool_call.get("tool")
        argument = tool_call.get("argument")

        if tool_name == "get_current_time":
            result = get_current_time()

        elif tool_name == "calculate":
            result = calculate(argument)

        else:
            return response

        return f"The result is {result}"

    except json.JSONDecodeError:
        return response

The function first tries to read the LLM’s response as JSON. If it finds a tool request, we run the matching Python function.

If not, we just return the LLM’s response.

Step 6: Converting the AI Response to Voice

Now our agent can listen and think. The last step is to give it a voice.

We will use Piper TTS:

import subprocess

PIPER_MODEL = "en_US-lessac-medium.onnx"

def speak(text):
    command = [
        "piper",
        "--model",
        PIPER_MODEL,
        "--output_file",
        "response.wav"
    ]

    process = subprocess.Popen(
        command,
        stdin=subprocess.PIPE,
        text=True
    )

    process.communicate(text)

    subprocess.run([
        "python",
        "-c",
        """
import sounddevice as sd
from scipy.io.wavfile import read

rate, audio = read('response.wav')
sd.play(audio, rate)
sd.wait()
"""
    ])

Piper turns the text into speech and saves it as an audio file. Then we play the audio with sounddevice.

One thing I learned from working with Voice AI is that response time matters a lot more in voice apps than in regular chatbots.

Waiting five seconds for a chatbot reply is usually fine. But waiting five seconds for a voice response feels slow.

That’s why production Voice AI Agents often use streaming for speech recognition, LLM responses, and text-to-speech.

Step 7: Building the Complete Voice AI Agent

Now we can connect every component:

def run_voice_agent():
    while True:
        record_audio()

        user_text = transcribe_audio()

        print("You:", user_text)

        if user_text.lower() in ["exit", "quit", "stop"]:
            print("Voice Agent stopped.")
            break

        llm_response = ask_llm(user_text)

        final_response = process_response(llm_response)

        print("Agent:", final_response)

        speak(final_response)


if __name__ == "__main__":
    run_voice_agent()

Run the application:

python main.py

Now you’ve built a complete Voice AI Agent without needing any paid APIs.

Final Thoughts

Building a Voice AI Agent from scratch showed me that Voice AI isn’t just one model. It’s a system where several AI parts work together.

Speech recognition lets the agent listen. The LLM does the reasoning. Tools help it take action. Text-to-speech makes the conversation possible.

I hope you enjoyed this article on building a Voice AI Agent from scratch.

For more AI and machine learning tips, follow me on Instagram. My book, Hands-On GenAI, LLMs & AI Agents, can also help you advance your AI career.

Aman Kharwal
Aman Kharwal

AI/ML Engineer | Published Author. My aim is to decode data science for the real world in the most simple words.

Articles: 2200

Recommended For You

Leave a Reply

Discover more from AmanXai by Aman Kharwal

Subscribe now to keep reading and get access to the full archive.

Continue reading