We Designed Our Security Backwards (On Purpose)

Most software companies secure their infrastructure by locking the front door. We locked the front door, bricked up the windows, and then — just to be safe — made sure we don't even have a key. Here's the story of how we built Mechanic, and why the weirdest architectural decision we made turned out to be the most important one.

The question nobody likes to ask

When you run a platform where AI agents live inside your customers' computers, you face an uncomfortable question: what happens if someone breaks into your servers?

The boring answer — the one most companies quietly adopt — is: "That would be bad. Let's make sure it doesn't happen."

The honest answer is: "We should assume it will. And design so it doesn't matter."

We kept coming back to this during the early days of building Mechanic. Our AI needs to get inside customers' machines. It needs to monitor them, fix them, restart services, read logs. That requires some kind of access. And access is the thing attackers are after.

So we asked a different question: what's the worst-case scenario, and how bad would it actually be?

The key-on-the-doormat problem

Our first design was simple. Obvious, even. Our hub server would hold SSH keys for every customer machine. When something needed fixing, we'd SSH in, do the work, leave. Clean. Familiar. Standard practice.

Then we pulled the thread. If an attacker gets into our hub server and our hub stores SSH keys to every customer machine — we've just handed them the keys to every house in the neighbourhood. Not one breach. Not one customer affected. Everyone. All at once.

This is a problem with a name: lateral movement. It's how one compromised server becomes a catastrophic, cascading failure. It's how breaches go from "a bad day" to "existential crisis." We scrapped the design.

Flipping the connection inside out

Here's the insight that changed everything. Instead of our server reaching into customer machines, what if customer machines reached out to our server?

Think about it like a phone call. The customer's machine dials us. The call connects. Now both sides can talk. But if you want to listen in on that call, you need to be on the line right now. There are no keys to steal. There's no door to pick. There's just a live conversation — and conversations end.

We call this the phone-home model. Every customer machine runs a small agent we install. On startup, it opens a persistent encrypted connection to our hub and says: "I'm here. I'm listening. What do you need?" Mechanic can send instructions down that connection at any time.

But here's the critical part: the agent is not a shell. It doesn't blindly execute whatever it receives. It has a strict whitelist of exactly the operations it will perform. Health checks. Log streaming. Service restarts. Config updates. Nothing else. Ever.

If an attacker compromises our hub and tells every agent to do something destructive — nothing happens. That command isn't on the list. The agent ignores it and logs the attempt.

The blast radius of a hub compromise shrinks from "every customer, full access" to "can trigger health checks on people who are currently online."

But what about complex problems?

You might spot the flaw. A whitelist of operations is great for security. It's terrible for actually fixing things. Real problems are messy. They require investigation — poking around log files, trying something, seeing what breaks, trying something else.

We needed a way to get real access when genuinely needed, without leaving that access open all the time. The answer is the escalated session.

When automated repair fails, Mechanic can request a full shell session on the customer's machine. But this isn't SSH. It's a PTY session — a real bash terminal, piped through the same encrypted connection that's already open. Think of it as the intercom suddenly supporting a full conversation, but only because both sides agreed to it.

Here's what makes it secure:

The request must be cryptographically signed. Our signing key never lives on the hub server. It's stored in a dedicated secrets vault — a separate system whose entire job is holding secrets. The hub retrieves the key on startup and holds it in memory only. It's never written to disk. An attacker who breaks into the hub server finds: no key.

The session is time-limited. It expires automatically. Mechanic fixes the issue, the session closes, access is gone. No lingering open door.

Everything is logged. Every keystroke. Every character of output. Timestamped, chained together cryptographically so historical entries can't be silently altered. The full session is replayable.

What the threat model actually looks like now

Let's be concrete. Here's what an attacker can and can't do in various scenarios:

Scenario	What the attacker can do
Hub server compromised, no signing key	Trigger health checks. Read logs. That's it. No shell access to any customer machine.
Hub server + signing key compromised	Open shell sessions — but one at a time, only on online machines, all fully logged and audited.
Individual customer token stolen	One customer affected. No lateral movement to others.
Customer machine compromised	Completely isolated. Cannot reach other customers through the agent.

None of these scenarios are "fine." But they're all containable. Detectable. Recoverable. That's the goal: not unbreakable, but designed to fail gracefully.

The silent guardian you already have

There's one more piece worth talking about — the unsexy part that's also kind of brilliant.

We don't run our own watchdog process to keep customer bots alive. We use the one that already exists on every Linux machine: systemd. It's the first process that starts when Linux boots. It's managed everything else for over a decade. And it has one feature we care about deeply: two lines of config that say "if this process dies, restart it in 5 seconds."

Our agent sits on top of systemd, doing the intelligent work — detecting why something crashed, understanding log patterns, deciding what to try first. systemd just makes sure the agent is always there to do that work. The dumb layer is reliable. The smart layer is intelligent. Neither tries to do the other's job.

Detection that actually works

Most monitoring systems poll: they check every minute or every five minutes whether things are still running. That's 30-second average detection lag for something that failed the moment after the last check.

We watch instead of polling, wherever possible. The agent subscribes to systemd events — the instant OpenClaw crashes, systemd fires an event. The agent uses inotify to watch log files — the instant an error is written, the agent sees it. No polling, no lag. For things that can only be polled (CPU, memory, disk) we check every two minutes.

The timeline in practice: gateway crashes → detected in ~1 second → hub notified in +1-2 seconds → automated repair begins. That's not a product claim. That's just how event-driven systems work when they're designed correctly.

Why we're publishing this

Security through obscurity is not security. If our architecture only holds up because attackers don't know how it works, it's not actually secure — it just hasn't been tested yet.

We're publishing this because we want customers to understand what we built, why we built it, and what the real tradeoffs are. We're not claiming this is unbreakable. We're claiming it's designed to fail gracefully — where one bad day doesn't become everyone's bad day.

The companies we trust most — Tailscale, Cloudflare, HashiCorp — publish their architecture openly. It's a feature, not a vulnerability. We're trying to build something worth trusting. That starts with being honest about how it works.