Jairus

Why I built it

I wanted hands-free music control that runs on my own PC and a modest GPU, without sending every request to a cloud service. Jairus is also my testbed for a bigger idea: a personal desktop agent that can do more than play music, but only once each new ability has proven itself.

From wake word to music

This is the workflow as designed and used during development. In the latest supervised test, after the security changes, it got as far as the YouTube search but nothing played; see Testing and validation.

1. 1

   Wake word

   Listens for a custom “hey Jairus” model, trained with an open-source pipeline and my own voice samples.

2. 2

   Record

   Records until I stop talking, instead of a fixed window.

3. 3

   Transcribe

   Turns speech into text on the GPU with faster-whisper.

4. 4

   Understand

   Simple commands like “pause” or “not that one” are matched instantly. Everything else goes to a local language model through Ollama, which returns an action such as play, queue, pause or skip.

5. 5

   Act

   Checks my local music library first, otherwise searches regular YouTube for the request and opens the top result in my own browser. System media keys are the last fallback.

6. 6

   Follow up

   “Not that one” moves to the next of up to five results, “play that again” uses the conversation so far, and playlist requests generate five to eight songs.

Asking by remembered lyrics isn’t a separate feature: the request goes to YouTube search as spoken, so it only works when YouTube matches the lyrics.

Architecture

The original assistant still does the work; a new structure grows around it.

• Command gateway

  Gives every command an identity and a lifecycle, whether it came from my voice or my phone.

• Permissions

  Actions carry permission levels, and risky ones require confirmation.

• Shadow-mode agent

  An agent with 39 tools (apps, browser, files, games, media, Windows UI) evaluates every live command and logs what it would do. It doesn’t act yet: only the music controls actually run.

• Local first

  Wake word, speech recognition and intent parsing all run on the PC. An optional cloud model is a fallback for desktop voice only.

Prototypes, not features

Two pieces are built separately from the everyday assistant and aren’t part of it yet. Neither has been retested by hand since the phone gateway was hardened.

• Orb renderer

  A Godot 4 visual meant to react to the assistant’s state, launched as a separate, supervised process. It was built and measured on its own; the live window shows a plain placeholder instead, since the earlier orb was removed for a redesign.

• Android client

  A text-only Kotlin app that sends commands to the desktop. It is meant to connect only through a USB cable to this PC, never over Wi-Fi; that USB mode hasn’t been tried on a real phone since the hardening.

Engineering decisions

• Grow, don’t rewrite

  The working assistant was wrapped step by step with a command gateway, task tracking and permissions, instead of being replaced, so the existing music path stayed in place.

• Shadow mode before trust

  The new agent watches real commands and records its decisions before it is allowed to act on anything.

• Fallbacks at every step

  Local library, then YouTube, then media keys; an instant matcher before the language model.

• Fail closed on security

  A security review of the phone gateway led me to switch off wireless use and have the gateway rebuilt to fail closed: off by default and reachable only from this PC, with a strong generated code, browser connections refused, attempt and rate limits, music-only commands, phone command text kept out of logs, and no paid AI calls for phone requests. Encrypted wireless pairing is designed but not built.

Testing and validation

Automated testing so far covers the phone gateway and the command path behind it, not the whole assistant. The voice pipeline has no automated suite yet, and the assistant’s response time hasn’t been measured.

• 70

  Phone gateway security tests

  Local-only test server; outside services mocked

• 11 / 11

  Gateway security fixes caught by mutation checks

  Each fix undone once; the tests failed every time

• ≈4.4–6.5 ms

  Orb prototype frame time

  Development GPU, uncapped; not assistant response time

A supervised desktop voice test (October 2026; local model only, paid AI blocked) gave mixed results. The wake word was detected four times across three runs, and the local model parsed requests in about 3–8 s once loaded. But speech recognition misheard “Clair de Lune by Debussy” in all three runs, and YouTube playback failed: once with a connection reset, once with no search results found. Pause, resume and “not that one” weren’t reached. There’s no accuracy measurement for the custom wake word either, and a phone over USB and the orb renderer still need manual retesting.

Limitations and next steps

• Playback not verified

  Playback failed in the latest test and the cause hasn’t been diagnosed. The wider desktop agent exists as architecture and shadow-mode code; it doesn’t control the PC.

• Phone control is limited

  Text only, and only over USB. Wireless use waits for the encrypted pairing design to be built and reviewed.

• Little measurement

  No latency data, no wake-word accuracy, no persistent logs; the console is the only log.

• Next, when work resumes

  Diagnose the playback failure and the recognition errors, retest by hand, then measure response times. The encrypted wireless design and letting the agent act come after that.

How I built it

I’m responsible for Jairus as a whole: what it should do, its architecture, how the pieces are integrated and tested, the technical decisions, and the order it grows in. I check each phase before the next one starts. Claude Code and other AI coding assistants have implemented substantial parts of the code under that direction, so I don’t claim to have typed it all myself. Most commits in the recorded history list Claude as a co-author, and the latest security work was AI-implemented too, even where a commit doesn’t say so. Co-author lines show the assistant’s involvement; they don’t measure who wrote what.

The security work followed the same pattern: an independent review identified problems, the fixes were made, and a mutation check undid each of the 11 security fixes in turn to confirm the tests caught every one.

Clips and screenshots

• PlaceholderReal capture pending approval

• PlaceholderReal capture pending approval

• PlaceholderReal capture pending approval

Captures will use scripted, non-personal commands, with no other browser tabs, accounts or notifications on screen. Recording waits for a passing voice test and an approved capture plan.