Remote runner

theis commented

2026-08-23 19:58:49 +00:00

Owner

Recommended Design: Registered Pull Runners
Rather than exposing every runner directly to the internet, each remote machine should run a small runner agent that connects outbound to the backend.
Flow:
Remote runner
-> registers with backend
-> maintains heartbeat / long-poll connection
-> receives a TTS job
-> calls its local TTS service
-> streams results back to backend
This avoids firewall, NAT, and TLS complications. The backend does not need to initiate connections to arbitrary remote machines.
Runner Model
Add a persistent tts_runners table containing:

Runner ID
Name
Authentication token hash
Status: online, offline, disabled
Last heartbeat
Active job ID
Engine: qwen, kokoro
Device: CPU, CUDA, ROCm
Supported languages
Maximum segment size
Created and updated timestamps
The backend should never store the raw runner token, only a hash.
Runner Registration
Provide an authenticated admin flow:

Admin creates a runner in the app.
Backend generates a one-time registration token.
Admin installs/configures the runner agent remotely.
Agent exchanges the registration token for a long-lived runner credential.
Backend marks the runner online after its first heartbeat.
The token should be shown only once and revocable from the app.
Job Assignment
The current worker claims a TTS job directly from the queue. This would change to:
Claim a queued TTS job.
Select an eligible runner.
Assign the job to that runner.
Persist the assignment and lease.
Send the prepared TTS segments to the runner.
Receive and persist results exactly as today.
Runner selection could initially be simple:

Explicit runner selected by the user, or
First available compatible runner
Later it could support:
Preferred engine
GPU capability
Language support
Runner priority
Per-runner concurrency limits
User or tenant restrictions
Runner Communication
Two viable approaches exist:

Outbound pull or long-poll, recommended

Runner asks the backend for work.
No inbound port required on the remote host.
Easier to secure and operate behind NAT.
Slightly more complex because the backend must provide a runner protocol.

Backend-to-runner HTTP

Runner exposes an authenticated HTTPS endpoint.
Backend calls its registered URL.
Simpler conceptually.
Requires public reachability, TLS, firewall rules, URL validation, and protection against SSRF-style abuse.
For a personal or controlled deployment, direct HTTPS may be acceptable. For a general application, outbound runner connections are safer.
Failure Handling
Runner jobs need leases:
Assignment includes a lease expiration timestamp.
Runner sends heartbeats while processing.
If the runner disappears, the backend expires the lease and requeues the job.
Already received segments remain stored.
With the current retry behavior, the job would restart from the beginning.
For true resume support, the runner should receive only missing segment indexes.
The runner should be idempotent:
A segment result includes the job ID and segment index.
Duplicate results are safely ignored or upserted.
A job cannot be completed by an unauthorized runner.
A disabled or revoked runner loses access immediately.
Security Requirements
Use TLS for all runner communication.
Authenticate every runner request with a token or signed credentials.
Store only token hashes.
Support token revocation and rotation.
Validate runner URLs if using backend-to-runner HTTP.
Do not allow users to register arbitrary runner URLs without authorization.
Restrict runner access to assigned jobs.
Avoid sending unrelated user data to runners.
Log runner assignments, failures, and disconnects.
Frontend Changes
The app could add an admin runner page showing:
Runner name
Online/offline status
Last heartbeat
Engine and device
Current job
Queue or utilization
Enable/disable controls
Token generation/revocation
The Speech page could optionally expose a runner selector. Initially, automatic runner selection would avoid changing the user workflow.
Recommended Implementation Phases

Extract the current static TTSServiceURL into a runner abstraction.
Add runner persistence, authentication, heartbeat, and admin endpoints.
Add a local runner mode that wraps the existing TTS service.
Assign TTS jobs to one configured runner.
Add runner failure detection and lease recovery.
Add automatic capability-based scheduling.
Add resumable segment processing.
The cleanest long-term architecture is:
Backend

owns jobs, segmentation, progress, persistence, scheduling

Runner agent

owns connection to backend and local TTS process

TTS service

owns only model loading and audio synthesis
This keeps the TTS service simple while allowing multiple machines, different hardware, and future runner types.

Recommended Design: Registered Pull Runners Rather than exposing every runner directly to the internet, each remote machine should run a small runner agent that connects outbound to the backend. Flow: Remote runner -> registers with backend -> maintains heartbeat / long-poll connection -> receives a TTS job -> calls its local TTS service -> streams results back to backend This avoids firewall, NAT, and TLS complications. The backend does not need to initiate connections to arbitrary remote machines. Runner Model Add a persistent tts_runners table containing: - Runner ID - Name - Authentication token hash - Status: online, offline, disabled - Last heartbeat - Active job ID - Engine: qwen, kokoro - Device: CPU, CUDA, ROCm - Supported languages - Maximum segment size - Created and updated timestamps The backend should never store the raw runner token, only a hash. Runner Registration Provide an authenticated admin flow: 1. Admin creates a runner in the app. 2. Backend generates a one-time registration token. 3. Admin installs/configures the runner agent remotely. 4. Agent exchanges the registration token for a long-lived runner credential. 5. Backend marks the runner online after its first heartbeat. The token should be shown only once and revocable from the app. Job Assignment The current worker claims a TTS job directly from the queue. This would change to: 1. Claim a queued TTS job. 2. Select an eligible runner. 3. Assign the job to that runner. 4. Persist the assignment and lease. 5. Send the prepared TTS segments to the runner. 6. Receive and persist results exactly as today. Runner selection could initially be simple: - Explicit runner selected by the user, or - First available compatible runner Later it could support: - Preferred engine - GPU capability - Language support - Runner priority - Per-runner concurrency limits - User or tenant restrictions Runner Communication Two viable approaches exist: 1. Outbound pull or long-poll, recommended - Runner asks the backend for work. - No inbound port required on the remote host. - Easier to secure and operate behind NAT. - Slightly more complex because the backend must provide a runner protocol. 2. Backend-to-runner HTTP - Runner exposes an authenticated HTTPS endpoint. - Backend calls its registered URL. - Simpler conceptually. - Requires public reachability, TLS, firewall rules, URL validation, and protection against SSRF-style abuse. For a personal or controlled deployment, direct HTTPS may be acceptable. For a general application, outbound runner connections are safer. Failure Handling Runner jobs need leases: - Assignment includes a lease expiration timestamp. - Runner sends heartbeats while processing. - If the runner disappears, the backend expires the lease and requeues the job. - Already received segments remain stored. - With the current retry behavior, the job would restart from the beginning. - For true resume support, the runner should receive only missing segment indexes. The runner should be idempotent: - A segment result includes the job ID and segment index. - Duplicate results are safely ignored or upserted. - A job cannot be completed by an unauthorized runner. - A disabled or revoked runner loses access immediately. Security Requirements - Use TLS for all runner communication. - Authenticate every runner request with a token or signed credentials. - Store only token hashes. - Support token revocation and rotation. - Validate runner URLs if using backend-to-runner HTTP. - Do not allow users to register arbitrary runner URLs without authorization. - Restrict runner access to assigned jobs. - Avoid sending unrelated user data to runners. - Log runner assignments, failures, and disconnects. Frontend Changes The app could add an admin runner page showing: - Runner name - Online/offline status - Last heartbeat - Engine and device - Current job - Queue or utilization - Enable/disable controls - Token generation/revocation The Speech page could optionally expose a runner selector. Initially, automatic runner selection would avoid changing the user workflow. Recommended Implementation Phases 1. Extract the current static TTSServiceURL into a runner abstraction. 2. Add runner persistence, authentication, heartbeat, and admin endpoints. 3. Add a local runner mode that wraps the existing TTS service. 4. Assign TTS jobs to one configured runner. 5. Add runner failure detection and lease recovery. 6. Add automatic capability-based scheduling. 7. Add resumable segment processing. The cleanest long-term architecture is: Backend - owns jobs, segmentation, progress, persistence, scheduling Runner agent - owns connection to backend and local TTS process TTS service - owns only model loading and audio synthesis This keeps the TTS service simple while allowing multiple machines, different hardware, and future runner types.

theis added the

feature

label

2026-08-23 19:58:49 +00:00