I’ve been working on a really interesting AI project: a computer-use agent that can interact with a browser much like a person. Instead of just generating text, this agent can open websites, look at pages, click buttons, fill out forms, take screenshots, and even recover if something goes wrong.
In this article, I’ll show you how to build a computer-use AI agent with Python.
What a Computer-Use Agent Actually Does
A normal chatbot receives a prompt and generates a response.
A computer-use agent adds another layer between the model and the user, which is the computer environment.
For example, I can give my agent a task like:
Open a website, search for Python tutorials, and take a screenshot of the results.
The agent has to decide what to do next. Its basic process works like this:
Goal → Observe → Decide → Act → Observe again
I found that understanding this loop is the most important part.
The model doesn’t have to solve the whole task at once. It just needs to pick the next useful action based on what the browser is showing right now.
The Architecture I Used
I made sure to keep the architecture simple. The Python app has four main parts:
- Browser controller: Playwright controls Chromium.
- Observation layer: The agent extracts page text, URLs, titles, and screenshots.
- LLM: A local open-source model decides what action to take.
- Agent loop: Python executes the action and sends the new state back to the model.
For browser automation, I chose Playwright because it gives reliable control over Chromium and lets you click, type, navigate, and take screenshots.
For the AI part, I used Ollama so the model runs on my own computer instead of needing a paid API.
This matters because computer-use agents might expose everything you see in your browser. Running the model locally makes experimenting feel much safer.
If you’re learning how AI agents work, I recommend looking beyond single tutorials and focusing on the main ideas behind them. That’s why I wrote Hands-on GenAI, LLMs and AI Agents.
I created it as a practical guide to help you understand LLMs, generative AI, and AI agents with clear concepts, examples, and hands-on projects.
Now, let’s dive into the practical steps using Python.
Building a Computer-Use AI Agent with Python
Before building the agent, I installed a few free tools. You won’t need a paid API or any commercial browser automation platform.
First, I installed Playwright and the Python package for Ollama:
pip install playwright ollama
playwright install chromium
Then I installed Ollama and downloaded a local vision-language model:
ollama pull qwen3-vl:8b
Here’s the simplified version I made:
from playwright.sync_api import sync_playwright
import ollama
import json
import time
MODEL = "qwen3-vl:8b"
def ask_agent(goal, page_text, screenshot_path):
prompt = f"""
You are a browser automation agent.
Goal:
{goal}
Current page text:
{page_text[:12000]}
Choose ONE action.
Return JSON only:
{{
"action": "goto|click|type|screenshot|done",
"selector": "CSS selector or empty string",
"text": "text to type or URL",
"reason": "short explanation"
}}
"""
response = ollama.chat(
model=MODEL,
messages=[
{
"role": "user",
"content": prompt,
}
],
)
return json.loads(response["message"]["content"])
def run_agent(goal):
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.goto("https://www.google.com")
for step in range(15):
page_text = page.locator("body").inner_text()
screenshot = f"screen_{step}.png"
page.screenshot(path=screenshot, full_page=False)
action = ask_agent(
goal,
page_text,
screenshot
)
print("ACTION:", action)
if action["action"] == "goto":
page.goto(action["text"])
elif action["action"] == "click":
page.locator(action["selector"]).first.click()
elif action["action"] == "type":
page.locator(action["selector"]).first.fill(
action["text"]
)
elif action["action"] == "screenshot":
page.screenshot(
path="final_screenshot.png"
)
elif action["action"] == "done":
print("Task completed.")
break
time.sleep(2)
browser.close()
run_agent(
"Open a website and search for Python tutorials."
)The key part isn’t how much code there is; it’s the agent loop.
After each action, I gather the new browser state and ask the model what to do next.
Adding Screenshots Makes the Agent More Capable
At first, I tried giving the model just the DOM text.
That works pretty well for simple websites, but it doesn’t help much when the visual layout is important. A screenshot gives the agent extra information.
The model can potentially identify things such as:
- buttons
- menus
- forms
- popups
- visual errors
- page layouts
- elements that are difficult to describe through text alone
This is where vision-language models are especially helpful for computer-use agents.
Instead of making the model understand just HTML, I can give it both structured browser data and visual info.
The Takeaway
The biggest thing I learned is that a computer-use agent isn’t some far-off idea. It’s really just a continuous feedback system:
Observe → Reason → Act → Verify → Recover
The LLM handles the reasoning, Python takes care of the control logic, and Playwright connects to the computer. This setup is enough to build powerful browser agents without paying for an API or commercial automation tools.
If you’re learning about AI agents, I suggest building something like this yourself. You’ll pick up tool calling, agent loops, browser automation, vision models, error handling, and safety much faster by seeing an agent fail and fixing it, instead of just reading about how agents work.
I hope you enjoyed this article on building a computer-use AI agent with Python. For more AI and machine learning tips, you can follow me on Instagram.
