swampcy gives a coding agent a Linux desktop it can see and use. It's an MCP
server, started by the agent over stdio, that drives a headless
sway session with no GPU and no monitor. The agent takes
screenshots, clicks, types and starts apps, and it can find an app's buttons
and text fields through the accessibility bus and press them by name instead of
by pixel. swampcy web puts the same screen in a browser, so I can watch from
my phone and take over when I need to.
It's on GitHub now, MIT licensed.
Why
I run Claude Code and Codex on a Linux VM over SSH. In early October, when I started this, Claude Code's own computer use only worked on macOS. The MCP servers that did work on Linux were X11, mostly driving an Xvnc display. A Wayland one that started its own KWin session couldn't take a screenshot without a GPU. Nothing maintained drove a headless wlroots compositor.
An MCP tool can return an image and Claude Code shows it to the model, so a plain stdio server was all it needed to be.
A desktop nobody sits at
WLR_BACKENDS=headless WLR_RENDERER=pixman sway runs sway in software, with
outputs of whatever size you ask for. wlroots has a protocol for each thing
swampcy needs: screencopy for screenshots, a virtual pointer, a virtual
keyboard. All of it is in ordinary distro packages.
The first surprise: a headless seat has no input devices, so apps ignore
everything sent to it, sway's own seat cursor commands included. swampcy
keeps its own virtual pointer and keyboard on the seat for as long as the
session lives. It speaks the Wayland wire protocol with Go's standard library,
no libwayland and no cgo.
Holding the keyboard turned out to matter for more than typing. I started out typing with wtype, which creates a keyboard, types and destroys it. That left the seat with no active keyboard, and Chrome crashed the next time it took focus. swampcy now types on its own keyboard. For characters off the US layout it swaps in a keymap that holds them for a moment, then swaps back.
Coordinates are always pixels of swampcy's own screenshots, which are scaled to at most 1568 on the long edge. The agent never has to think about scale. The example config sets the agents' screen to 1280×800, which costs about 1,330 image tokens per screenshot.
By name, not by pixel
GTK, Qt, Firefox, LibreOffice and Chromium (when asked) describe their widgets
on AT-SPI, the accessibility bus. find_elements lists them with a short ref
like e14, and click_element e14 performs that widget's action. That beats
guessing at pixels, and it has to be an action rather than a click: GTK 4
reports some widgets' bounds up to 12 pixels off.
Wayland doesn't tell a client where its window is, so AT-SPI's screen coordinates mean nothing. swampcy takes each widget's position within its window and adds the window's origin from sway's tree, matched by process ID.
Making the video turned up a bug. On the first takes, the agent pressed the calculator's "0" twice as two parallel tool calls and only one landed, so it got 1280 × 80. It noticed in its next screenshot, said "Parallel button clicks dropped a digit", and typed the sum again one press at a time.
Being parallel had nothing to do with it. A GTK button that's activated shows
as pressed for 250 ms and only then emits clicked, ignoring any other
activation until it has. click_element returned 6 ms after asking, so a
second press of the same button within that quarter second disappeared, even
one sent after the first had finished. Three presses 100 ms apart gave one
digit. It also explains why the agent twice thought its screenshots were
lagging: it took them before the click had happened. The
fix waits 300 ms after a click in
a GTK app before returning. The video is a fresh run on the fixed build. Both
zeros land, and the agent gets 1024000 the first time.
Watching from a phone
The first viewer was wayvnc on loopback through an SSH tunnel. That's fine
from a laptop and no good from a phone, and wayvnc leaked a seat per
connection. So swampcy web sends the screen as a WebRTC video track and takes
taps and keys back on a data channel.
I encoded a scripted 22-second desktop workload with six encoders to pick one.
libaom's AV1 with its screen-content tools gave near-lossless text, 52 dB
PSNR, at 320 kbit/s. OpenH264 in screen mode used 586 kbit/s for 43 dB, and
x264 did worst of the H.264 encoders. Safari only decodes AV1 on devices with a hardware decoder, so
swampcy sends AV1 when the browser offers it and H.264 otherwise. Both
libraries are loaded at run time through purego, so the build stays
CGO_ENABLED=0. The price is calling C structs at hand-checked offsets. Each
encoder reads the library's defaults back at those offsets before it starts and
refuses to encode if they don't match.
sway reports which parts of the screen changed, and only those get converted and encoded. An idle desktop costs no CPU and sends nothing.
It listens on loopback behind tailscale serve and lets in exactly one
Tailscale login. One viewer at a time.
Two desktops
Watching wasn't enough; I wanted a desktop of my own next to the agents', so I could open something there without stealing their focus. sway can have several seats, so there are two outputs and two seats, and windows move between them from the phone. My desktop resizes itself to whatever device I'm viewing it on.
Chrome broke this. It binds the first seat the compositor offers and ignores
the rest, so my taps never reached it. swampcy web now borrows the agents'
seat while I'm using a Chrome window and hands it back after 1.5 seconds idle.
While I'm typing into Chrome, an agent's keystrokes land there too. That's in
the README's list of things that don't work yet.
Replays
When an agent says it did something on the screen, I'd rather see it. After
every click, key or launch, swampcy keeps a 640-pixel frame in memory, and the
replay tool turns the last few dozen into an animated WebP saved locally.
The agent in the video made one at the end.
How the video was made
I didn't record the VM I actually use. Its browser is signed in to my own
accounts, and that's not something to put on a blog. The demo ran in a
throwaway Fedora 44 container on my Mac instead: headless sway, swampcy built
from the public repo with the click fix, a user named demo and nothing
signed in. Claude Code
ran with only swampcy's tools allowed. A loop of grim screenshots recorded
the agents' screen. A headless Chromium sized like a phone opened swampcy web
and recorded the real stream. Its Tailscale address was a dummy interface.
The rest is the same pipeline as the trivai and Drift Sector videos: an HTML page rendered frame by frame, with the recordings laid in by ffmpeg. The tool-call list is the agent's actual stream, placed at the moment each call arrived.