An Architecture to Run AI Agents Safely and Efficiently in a Linux VM on Apple Silicon

The problem

If you want to run an AI coding agent locally on your Mac today, you have two options, and neither is good.

Option A: run it directly on your Mac. Most agent frameworks, like LangChain, AutoGen, SWE-Agent and Open Interpreter, execute LLM-generated code via subprocess or exec() with full access to your filesystem, network, and credentials. Docker's engineering blog has documented AI coding agent security incidents that came from exactly this. Indirect prompt injection makes it worse than it sounds: an instruction hidden in a file the agent reads can run commands with all your permissions. Four 2026 incidents show what that looks like in practice, including cases where the agent's own permission prompts and allowlists were what got bypassed.

Option B: run it in a VM or container for isolation. That's the right answer on paper, and it runs into a wall specific to Apple silicon: Apple's Virtualization framework doesn't give a Linux guest the Mac's GPU, as it does for macOS guests. Apple's own container team has confirmed this in their public repositories, answering people who asked for exactly that: GPU passthrough availability? and Paravirtualized Graphics via virtio-gpu. A macOS guest gets a purpose-built Metal-based GPU driver; a Linux guest gets virtio-gpu, a paravirtualized 2D device with no path to the host's compute GPU. A 7B model running inside the VM is dramatically slower than the same model on the Mac.

Update, September 2026: the wall is current, not necessarily permanent. The Cua project has published a user-space "Metal capability shim" that changes the GPU capability flags llama.cpp checks, and reports 11–16× speedups inside a Linux guest with no change from Apple. It's a workaround for one inference engine, not official passthrough, and it doesn't change the approach below, which keeps the model on the Mac's GPU. But it suggests the wall is "not cracked yet" rather than "impossible".

The root cause: on Apple silicon, "safe" (inside a VM) and "fast" (on the GPU) pull in opposite directions, because for a Linux VM the GPU exists only on the host side.

The solution: split the agent from the model

The fix isn't to pick a side. It's to stop treating "run the agent" and "run the model" as one problem. The dangerous part (running generated code) and the expensive part (running the model) don't have to live in the same place:

• The agent runs inside an isolated, disposable Linux VM. When you're done, the VM and everything it wrote to disk are gone.
• The model runs on the host, on Ollama or MLX, with full native Metal GPU access.
• AI Bridge connects them over a direct vsock channel, so the agent in the VM talks to the model as it would to any local server.

Since Ollama 0.19 (March 2026), Ollama runs on Apple's MLX on Macs with 32 GB or more of unified memory, instead of the llama.cpp-based engine benchmarked below, so the two engines are less independent than they were when I ran these tests.

┌───────────────────────────────────────────────────────────────────┐
│                            macOS HOST                             │
│                                                                   │
│      Ollama / MLX Engine   ───►  Storage: ~/.ollama/models        │
│      (Native Metal GPU)          (single 10GB–50GB copy)          │
│              ▲                                                    │
│              │  Velo Workspaces AI Bridge                         │
│              │  proxies to localhost:<port>                       │
│              ▼                                                    │
│      ┌─────────────────────────────────────────────────────┐      │
│      │                     LINUX VM                        │      │
│      │  socat 127.0.0.1:<port>  ⇄  vsock CID 2             │      │
│      │  Env: OPENAI_API_BASE=http://127.0.0.1:<port>/v1    │      │
│      │  Dev apps / Docker containers / VS Code / Python    │      │
│      └─────────────────────────────────────────────────────┘      │
└───────────────────────────────────────────────────────────────────┘

A relay connects a local TCP port inside the guest to the model server's port on the Mac over vsock (CID 2 is the host in vsock addressing). The port isn't tied to Ollama, so it works for MLX or any other OpenAI-compatible server on whatever port it listens on. Inside the VM you set OPENAI_API_BASE (or your tool's equivalent) to http://127.0.0.1:<port>/v1, and any OpenAI-compatible client works: Open Interpreter, LangChain, or plain curl. No code changes, and nothing needs to know there's a VM boundary.

What this is for: real isolation for the risky part (running generated code), no GPU-virtualization tax on the expensive part (inference), and no copying a 10–50 GB model into every VM you create.

vsock is just another Virtualization.framework device, VZVirtioSocketDeviceConfiguration, following the same paravirtualized pattern as every VM's network and storage devices. The device inventory shows where it fits.

The results

I wanted to know how close this gets to "no tax", so I measured it rather than asserting it.

Setup:

• Host: Mac mini, Apple M4, 10 cores, 16GB RAM, macOS 26.6.2.
• Guest: Ubuntu Server 26.04 (minimized), 4 vCPU, 4GB RAM.
• Model: qwen2.5-coder:7b (Q4_K_M) served by Ollama on the host.

Bridge tuned with a larger socat buffer and nodelay set. Both sides on Python 3.12.

Three tests, scripts here:

1. ai_perf_test.py: single-request time-to-first-token and throughput, 5 runs, deterministic
2. ai_load_test.py: 8 concurrent clients (OLLAMA_NUM_PARALLEL=2)
3. oi_perf_test.py: full agentic-loop latency via Open Interpreter: prompt → generated code → execution → result, 5 runs × 3 tasks

The methodology lesson that mattered most: the first time, I switched between host and VM tests without fully stopping the VM and restarting Ollama in between. The host's agentic-loop numbers were a mess, wildly bimodal, with some runs 10× slower than others. The fix was to stop all VMs, stop Ollama and start ollama serve fresh before every run, host and VM alike. Once I did that consistently, the bimodal pattern disappeared. A leftover process competing for CPU with the one being measured was enough to produce dramatic numbers that meant nothing. Everything below comes from the controlled re-run.

Single-request speed: the real "tax", and it's small

This is the true cost of the setup: one extra hop over vsock. The model never leaves the Mac's hardware, so there's no GPU emulation to pay for, and what's left is a single-digit percentage.

Eight concurrent clients: the VM won

The VM finished about 15% faster through the bridge, which surprised me. Part of it is CPU contention: on the host, the benchmark client and the Ollama server compete for the same cores, while in the VM the client has its own vCPUs. My best explanation for the rest is the path each request takes. In the VM, requests arrive over vsock at Velo Workspaces' native Swift bridge, which uses Grand Central Dispatch to pass connections on to Ollama; on the host, eight Python clients hit Ollama's port directly. I haven't isolated which effect dominates, so treat this as a hypothesis, not a finding. The point that holds either way: the bridge didn't make concurrent load worse.

A real agentic loop: the VM won consistently

The VM was 17–28% faster, with similar, tight variance on both sides once background contention was controlled. A VM beating native macOS on file I/O and system info looks odd until you look at how agent frameworks run code. Open Interpreter constantly spawns shells and Python processes, writes temporary files and reads their output. On macOS, new executables and scripts can be checked by the system's security services, a small delay per launch that adds up over an agent loop, and process creation is generally heavier than Linux's fork(). A minimal Ubuntu guest has none of that. Again, this is my reading of the numbers rather than a profiled breakdown, but the vsock hop is clearly small enough to disappear into it.

How close to the goal?

The goal was close to no virtualization tax: not literally zero, since the extra hop is unavoidable, but small enough that you wouldn't notice. A single-digit difference on the raw inference path meets that. What I didn't expect was for the system-level numbers, concurrent load and real agent loops, to come out ahead in the VM rather than merely close. The isolation you take on for safety also shields the agent from contention on the host, which is a bonus on top of the safety case.

The biggest lesson: keeping the environment clean between benchmark runs matters as much as the architecture you're testing. One leftover process can produce numbers dramatic enough to mislead you completely about which side won.

The scripts are at github.com/xgz2025/veloworkspace-tools if you want to reproduce this on your own hardware or model. This is the architecture Velo Workspaces ships as AI Bridge; see it under Features, or download it and try it yourself.

Further reading: I later re-ran the same tests with MLX and got a different answer under load; what sandboxing costs across both engines has the numbers and my best theory. Velo Workspaces vs. Parallels Desktop vs. UTM compares what each does with a Linux guest's GPU, and what determines VM performance on ARM explains why the overhead is this low. AI Bridge is a Linux-guest answer to a limit that doesn't apply to macOS guests. It supports two more backends than Ollama and MLX. What leaves your Mac when you use it covers exactly what the channel can reach, and local vs. cloud LLMs for coding agents helps if you're still deciding whether a local model is right for your agent.