AI and VoIP Blog

I’ve run Gemma 4 on a Raspberry Pi for fun. This weekend I wanted it doing real work on my Mac: a local AI coding agent that reads my files, runs commands in my terminal, and writes code. No cloud, no API key, no per-token bill.

By the end of a few sessions it built me a working browser game, entirely offline, on an M3 Pro. It also failed in interesting ways, which taught me more than the wins did. I have added the game in this article you can give it a try.

Here’s the full setup, and the honest limits.

Table of Contents

1. Why run a local AI coding agent
2. What you need
3. How the pieces fit
4. Step 1: Install Ollama and pull Gemma 4
5. Step 2: Install Pi
6. Step 3: Point Pi at Ollama
7. Step 4: Set your defaults
8. Step 5: Give the agent a system prompt
9. Step 6: Hand it every tool
10. Putting it to work
11. Real results, honestly
12. The same test on a bigger model
13. Play it below
14. The prompts that make a local AI coding agent useful
15. Summary

Why run a local AI coding agent

Three reasons pushed me to run a local AI coding agent instead of reaching for a hosted one.

• Privacy. My code and my commands never leave the laptop.
• Cost. It is free. No subscription, no per-token meter running in the background.
• Offline. It works on a plane, on bad hotel wifi, or with the network unplugged.

I already trust local models for smaller jobs. If you want the background on that, I wrote about how to run Gemma 4 on a Raspberry Pi 4B and how LLM tool calling actually works under the hood. This post takes the same model and turns it into a working agent.

What you need

• A Mac with Apple silicon. I used an M3 Pro with 36 GB of RAM.
• Ollama, to serve the model locally.
• Pi, the coding agent that drives the model.
• Node.js, which Pi installs through.
• About 8 GB of free disk for the model weights.

Pro tip: the model must support tool calling. A model that only chats cannot act as an agent. Run ollama show <model> and check that tools is listed under capabilities.

How the pieces fit

Ollama is just a local server that speaks the OpenAI API. Pi is just a terminal app that sends your request to that server and runs whatever tools the model asks for. That is the whole trick.

You type in the terminal
      ↓
Pi (agent: reads files, edits, runs commands)
      ↓  OpenAI-compatible request
Ollama  (http://localhost:11434)
      ↓
Gemma 4  (12B, MLX build, on your GPU cores)

The model decides what to do. Pi runs the tools and feeds the results back. Ollama is the engine in the middle. No magic.

Step 1: Install Ollama and pull Gemma 4

Ollama runs the model and exposes it on localhost:11434. I used the Gemma 4 12B MLX build because MLX is Apple’s own framework and it runs fast on Apple silicon. Any tool-capable local model works, so swap the tag for whatever you have.

Install Ollama and pull the model:

brew install ollama
ollama serve &
ollama pull gemma4:12b-mlx

The weights are about 7.7 GB, so the first pull takes a few minutes. Go make a coffee. After that the model loads from disk in seconds.

Step 2: Install Pi

Pi is a small, extensible coding agent that lives in your terminal. It ships with the tools an agent needs: read files, run shell commands, edit and write files. Install it globally with npm:

npm install -g @earendil-works/pi-coding-agent

The project lives on GitHub if you want to read the source or the docs.

Step 3: Point Pi at Ollama

Pi needs to know your local model exists. You register it as a provider in ~/.pi/agent/models.json. The apiKey is a placeholder because Ollama ignores it.

{
  "providers": {
    "ollama": {
      "api": "openai-completions",
      "apiKey": "ollama",
      "baseUrl": "http://127.0.0.1:11434/v1",
      "models": [
        {
          "id": "gemma4:12b-mlx",
          "contextWindow": 262144,
          "input": ["text", "image"],
          "reasoning": true
        }
      ]
    }
  }
}

Pro tip: set contextWindow to what the model card lists, not a guess. The Ollama card for these MLX builds advertises a 256K window (262144 tokens) and Text plus Image input, so that is what goes in the config. Declaring more than the card supports does not expand the model, it just lets Pi overflow it and silently truncate.

Step 4: Set your defaults

So you don’t pick the model every time, set it as the default in ~/.pi/agent/settings.json. I keep thinking off for speed and turn it on only when a task needs it.

{
  "defaultProvider": "ollama",
  "defaultModel": "gemma4:12b-mlx",
  "defaultThinkingLevel": "off"
}

Confirm Pi sees the model with pi --list-models. It should list your model with the context window and a tools column set to yes.

Step 5: Give the agent a system prompt

This is the step most people skip, and it matters more than the model choice. Pi reads a file called AGENTS.md from ~/.pi/agent/ and loads it into every session as standing instructions. Think of it as the agent’s job description.

A small model follows blunt, explicit rules far better than a clever, subtle one. Keep it short and direct:

# Global coding agent guide
You are a senior engineer. Write correct, simple, working code.
## Environment
- Machine: Apple M3 Pro, macOS. Shell: fish (not bash/zsh).
- Assume BSD/macOS versions of CLI tools unless GNU is confirmed.
## Method
- Read the relevant files before editing. Never guess at APIs.
- Fix the root cause, not the symptom. Make the smallest change that works.
- Keep functions small and single-purpose. Prefer early returns.
## Verify before claiming done
- Run the build, lint, or test that proves it works. Report the output.
- Never say it works unless you ran it and saw it pass.

Drop a project-specific AGENTS.md in any repo and Pi layers it on top of this global one.

Step 6: Hand it every tool

Pi enables four tools by default: read, bash, edit, write. It also ships with grep, find, and ls, which make it faster at locating code. There is no settings key to turn them all on, so I wrap the command in a shell function. My shell is fish:

function pi --description 'pi with all built-in tools'
    set -l subcommands install remove update list config
    if test (count $argv) -gt 0; and contains -- $argv[1] $subcommands
        command pi $argv
    else
        command pi --tools read,bash,edit,write,grep,find,ls $argv
    end
end

Save that as ~/.config/fish/functions/pi.fish. It passes management subcommands like pi install straight through, and adds all seven tools to every real run. On bash or zsh, an alias with the same --tools flag does the job.

Putting it to work

Run pi in any folder and start typing. For a one-shot task without the interactive UI, use print mode. One quirk to know: in a pipe or script, print mode waits on standard input, so close it with < /dev/null.

pi -p "explain what this repo does" < /dev/null

For a real test I asked my local AI coding agent to build a browser Snake game in a single HTML file. It created the file, wrote the JavaScript, and checked its own syntax before handing it back.

Real results, honestly

The numbers from my M3 Pro:

Now the honest part. The Snake game ran, but the first version had a bug: the snake passed straight through the food and never grew. The cause was a grid mismatch. The snake sat on coordinates like 150, 170, 190, while the food landed on 160, 180, 200. Two grids offset by half a cell, so the head could never land exactly on the food.

The agent fixed it, but by rewriting the file with aligned coordinates rather than naming the actual cause. It also burned thousands of tokens circling the problem. That is the ceiling of a 12B model. It is not Claude or GPT. It fumbles multi-step reasoning and second-guesses itself.

The lesson: its own syntax check passed while the game was broken. A syntax check proves the code parses, not that it works. That gap is where a small model will bite you.

The same test on a bigger model

The next day I pulled the 26B build of Gemma 4 and gave it the exact same prompt. It built a working game on the first try, with no grid bug.

The prompt did not change, so what did? The bigger model picked a better structure. It kept the snake and the food on the same integer grid and multiplied by the cell size only when drawing. On one shared grid the collision always registers, so the bug the 12B hit could not happen. Better model, better instinct for the right shape of the code.

It was not flawless. It still verified with a syntax check only, never a behavioral test, and it wasted a step chasing a typo that did not exist. But the game worked, and the reason was raw capability, not clever wording. That is the real lever: a bigger local model moved the needle more than any prompt tweak did.

Play it below

This is the exact game the 26B model wrote, running right here. Click the board, then use the arrow keys. Press Space to start or restart. Desktop and keyboard only.

The prompts that make a local AI coding agent useful

A better prompt is the cheapest way to get more out of a local AI coding agent. It will not buy reasoning the model does not have, but it removes most of the self-inflicted failures. Three rules carry the weight.

• Scope small. One concrete task beats a vague feature. A 12B nails “write this one function” and flails at “build me an app.”
• Force a real test. Ask for a behavioral check, not just a syntax check. Make it prove the thing works, then report the output.
• Keep it in one file when you can. Fewer moving parts means fewer ways for a small model to wire things up wrong.

Here is the prompt that worked for the game:

Build a browser Snake game in one self-contained index.html.
Arrow keys to move, food grows the snake, game over on collision.
Before you finish, verify it works:
1. Run node --check on the script. Fix any errors and re-check.
2. Write a tiny test that simulates moves and asserts the score
   rises when the head reaches the food. Run it and show the output.
Only say it is done after both pass.

Summary

A capable coding agent, running entirely on my laptop, for the price of the electricity. The model is not a genius yet. The setup already is.

Leave a ReplyCancel reply

Type your email…

Subscribe

Join 925 other subscribers

Akash Gupta
Senior VoIP Engineer and AI Enthusiast

AI and VoIP Blog

Thank you for visiting the Blog. Hit the subscribe button to receive the next post right in your inbox. If you find this article helpful don't forget to share your feedback in the comments and hit the like/clap button. This will helps in knowing what topics resonate with you, allowing me to create more that keeps you informed.

Thank you for reading, and stay tuned for more insights and guides!

• Mail
• LinkedIn
• X

---