Speech providers

About 2769 wordsAbout 9 min

A speech provider is a class that turns recorded audio bytes into a text transcript. Spora’s composer recording button and the /api/v1/speech/transcribe endpoint both call into the same SpeechToTextRegistry, which picks one provider per request via a four-tier preference cascade (the historical fifth “first-registered-wins” tier was removed in spora-core PR #253 — tier 4 now returns [null, null, null] so the SPA capability badge stops claiming a working provider when none is configured). Providers are contributed by spora-core (the built-in OpenAI-compatible transcriber) and by plugins (e.g. spora-plugin-minimax, which contributes MiniMax’s asr-1.0).

This page is the operator / agent-developer view: how the cascade works, what each preference row means, and how to read the capability endpoint’s effective_* fields. The plugin-author contract lives in Plugin author guide → Speech providers. The auto-generated REST surface is in API reference → Speech.

The components

                          ┌──────────────────────────────────────┐
                          │ SpeechToTextRegistry                 │
                          │  • list of providers (core + plugin) │
                          │  • SpeechToTextCascadeResolver       │
                          │  • configuredProvider(userId, agent) │
                          └──────────┬───────────────────────────┘
                                     │
                                     ▼ one provider per request
┌─────────────────┐    ┌────────────────────────────────────────────┐
│ providers[]     │    │ transcribe(bytes, mime, language?, agent?)│
│  ┌───────────┐  │    │                                            │
│  │OpenAI-Com │◄─┼────┤  → POST {base_url}/audio/transcriptions   │
│  ├───────────┤  │    │                                            │
│  │MiniMax    │◄─┼────┤  → POST {base_url}/v1/speech_to_text      │
│  ├───────────┤  │    │                                            │
│  │Muse      │◄─┼────┤  → bespoke multipart + ffmpeg pipeline    │
│  └───────────┘  │    └────────────────────────────────────────────┘
└─────────────────┘
LayerLives inRole
Provider classesSpora\Speech\SpeechToTextProviderInterfaceOne per vendor / wire shape. Sync-only.
RegistrySpeechToTextRegistryDiscovers providers (core + plugins), binds the resolved config’s settings, gates on isConfigured().
Cascade resolverSpeechToTextCascadeResolverWalks the four preference tiers and returns (class, source, config_id).
Configuration tablespeech_provider_configurations + principal_preferences.preferred_speech_config_id + agents.speech_driver_config_idOperator-managed rows that drive tiers 1-3.
REST surface/api/v1/speech/*10 endpoints for CRUD, defaults, capability, transcribe.

The cascade

The cascade answers one question: which provider class should this call go to? It runs on every capability probe and every transcribe call — both paths share the resolver so the recording button and the actual transcription can never disagree.

For a per-agent request (agent_id > 0), the resolver walks 4 tiers against the agent’s principal (not the caller’s):

1. Agent override         agents.speech_driver_config_id → speech_provider_configurations.provider_class
2. Principal preference   principal_preferences.preferred_speech_config_id for the AGENT's principal
                           (user-principal or group-principal, depending on agents.principal_id)
3. Global default         speech_provider_configurations WHERE is_global = true AND is_default = true
                           ORDER BY updated_at DESC, id DESC
4. No config → null       if tiers 1–3 resolved nothing, the cascade returns [null, null, null]

For a caller-scoped request (the composer recording button has no agent context), the resolver walks 4 tiers against the caller:

1. User preference        principal_preferences.preferred_speech_config_id for the caller's user-principal
2. Group preference       every group_memberships row for the caller, ordered by joined_at ASC
                           first match wins
3. Global default         same as above
4. No config → null       same as above — tier returns [null, null, null]

Both paths walk the same four-tier shape; per-agent requests use the agent’s principal for tier 2, caller-scoped requests split user + group preferences into tiers 1 and 2.

Every tier validates that the resolved provider_class is currently registered. If the operator deletes a plugin without first clearing the FK references, the resolver treats the row as unset and falls through to the next tier — never to a class the registry doesn’t have.

The “agent’s principal” pin in tier 2 is the critical subtlety operators miss. A group-owned agent must consult its group’s preference, not the caller’s user-principal. Otherwise the badge in the recording button would say “user default” when the agent’s principal is actually a group — see the SpeechToTextCascadeResolver rank-1 source for the principal lookup.

Configuration rows

Every tier (except tier 4, which returns null when nothing resolved) reads from the speech_provider_configurations table:

ColumnTypePurpose
provider_classstring (FQCN)Stable identity of the implementation — Spora\Speech\OpenAiCompatibleTranscriber, Spora\Plugins\MiniMax\MiniMaxTranscribeProvider, etc.
display_namestringOperator-facing label. When bound to a row, the registry calls bindLabel() so getDisplayName() returns this rather than the class-level default.
settingsencrypted JSONDecoded at request time and pushed into the provider via bindSettings(). Password fields are per-row encrypted via SecurityManager; the wire response masks them as '***'.
is_globalbooleanGlobal rows are admin-managed and visible to every principal. XOR with principal_id — enforced in SpeechProviderConfiguration::validateGlobalXor().
is_defaultbooleanOnly meaningful when is_global = true. Multiple rows can be global; at most one is the default. The POST /api/v1/speech/provider-configs/{id}/set-default endpoint demotes the previous default inside a lockForUpdate transaction.
principal_idFK → principals.idSet when the row is principal-scoped (user or group). The scope field on the wire response is derived from the principal’s type.

The CRUD surface is in /api/v1/speech/provider-configs. Schema-level validation (required keys, regex) walks the provider’s #[ToolSetting] attributes via SpeechProviderConfigValidator::assertSettingsAgainstSchema(), so adding a new field to the attribute list automatically widens the dynamic form the SPA renders.

Configuring a provider

POST /api/v1/speech/provider-configs carries a scope that decides who the row belongs to, and a group_id that names the group when it does:

scopeThe row is stored forWho may write it
globaleveryoneA global admin.
userthe calling principalThe caller. No extra permission needed.
groupthe named groupAn owner or admin of that group, or a global admin. group_id is required and positive.

Supplying group_id without scope: 'group', or scope: 'group' without a group_id, is a 422 SPEECH_PROVIDER_CONFIG_INVALID — the two cannot drift apart. A caller who is not a global admin and not an admin of the named group gets 403 SPEECH_PROVIDER_CONFIG_FORBIDDEN; a provider_class that is not a registered provider gets 404 SPEECH_PROVIDER_CONFIG_NOT_FOUND.

Reading a group’s configuration is narrower than writing it. Other members of the group can read it — the capability endpoint’s configured: true still shows, because the group cascade has at least one valid key — but a member who is not an admin cannot edit it. A user who is not a member of the group does not see the config at all: the list is membership-gated so that a non-member cannot tell whether the group has an STT config.

The two OpenAI-compatible vendors

Both Mistral Voxtral and OpenAI Whisper speak the same multipart wire shape (POST {base_url}/audio/transcriptions with file, model, an optional language, Bearer auth, and a JSON response carrying text), so Spora serves them through one provider class, OpenAiCompatibleTranscriber. Only three fields differ:

FieldMistral VoxtralOpenAI Whisper
Base URLhttps://api.mistral.ai/v1https://api.openai.com/v1
Modelvoxtral-mini-latestwhisper-1
Language hint (BCP-47)blank for auto-detect, or en-USblank for auto-detect, or en-US
HTTP timeout (seconds)6060
API keyyour Mistral key from console.mistral.aiyour OpenAI key

Everything else — the display name, the encrypted api_key, the shared-superset MIME list — is the same. Pick a display name that tells two configurations apart when you run more than one (Mistral Voxtral (prod) beside Mistral Voxtral (staging)).

Meta Muse is the third option, but it arrives differently: spora-plugin-muse ships a bespoke provider that handles Meta’s own multipart shape and its ffmpeg-based preprocessing, so it appears as a second card on the provider picker once the plugin is installed, with no migration to run. A provider the plugin declares through #[ToolSetting] is picked up live by the schema endpoint, so a new field on the plugin’s side widens the form without a core change.

Sending api_key: "***"

settings.api_key comes back masked as "***" on every read — the operator never sees the stored key. Sending "***" back keeps the existing value rather than overwriting it with three literal asterisks. That is the convention ToolController uses for tool API keys, and it is what makes the form safe to submit without re-typing the secret. To rotate a key, submit the new one.

Display names and collisions

Two configurations can carry the same display_name — one global, one user-scoped. The capability endpoint resolves to one of them: the user-scoped row wins and the global row becomes invisible to that caller. The uniqueness enforcement is a known follow-up, so the naming convention is the mitigation: suffix configurations that are meant to coexist ((prod), (staging)).

Reading the capability endpoint

GET /api/v1/speech/capability returns one row per registered provider, plus the resolved-class / source / config-id triple the cascade produced. The SPA reads this to decide whether to render the recording button at all and which preferred_audio_mimes[] to feed the MIME picker:

{
  "data": {
    "available": true,
    "configured": true,
    "providers": [
      {
        "name": "minimax",
        "class": "Spora\\Plugins\\MiniMax\\MiniMaxTranscribeProvider",
        "display_name": "MiniMax (prod)",
        "configured": true,
        "effective_class": "Spora\\Plugins\\MiniMax\\MiniMaxTranscribeProvider",
        "effective_source": "agent",
        "effective_config_id": 17,
        "preferred_audio_mimes": ["audio/ogg;codecs=opus", "audio/mp4", "audio/webm;codecs=opus"]
      }
    ]
  }
}

name is the stable key the provider’s getName() returns — MiniMax hardcodes 'minimax', OpenAI-compatible returns the bound label. display_name is what the SPA shows next to the dropdown; it’s the operator’s per-config display_name setting when bound, otherwise the class-level default. The fields operators care about:

FieldMeaning
availableAt least one provider class is loaded. false means every plugin that contributed a provider has been removed or disabled.
configuredThe cascade resolved a provider AND that provider’s isConfigured() returned true (its bound api_key is non-empty in v2-cascade mode). When false, the recording button hides itself — the transcribe endpoint would otherwise return 503.
classThis row’s provider FQCN. Used by the SPA to pick the matching preferred_audio_mimes[] for the resolved row. Distinct from effective_class below.
effective_classThe cascade-resolved class (per principal / agent). Identical across every row in the response — same answer, different vantage points.
effective_sourceWhich cascade tier produced the answer. See the table below.
effective_config_idThe configuration row that backed the choice. null when no FK config exists at any tier (cascade returned [null, null, null]).
preferred_audio_mimesThe MIMEs the SPA should offer the browser in order. Provider-declared via #[AcceptedAudioMime]; falls back to the common-superset default when the provider opts out.

Two more fields exist for the settings screens rather than for the caller:

  • has_global_default — true when the operator has a global settings row for this provider class.
  • config_id — the id of that global settings row, or null when there is none. It is a thin lookup against tool_configurations.id for OpenAiCompatibleTranscriber rows; a class-level provider (Muse) keeps config_id: null because it never writes to that table. The SPA deep-links a capability row into the provider’s edit form by this id.

effective_source values

The string is the tier label the cascade resolver emitted. Each maps to one row in the cascade:

ValueSource tierConfigurable by
agentPer-agent tier 1 — agent overrideAgent editor (per-agent speech_driver_config_id).
user_preferencePer-agent tier 2 (agent’s principal is a user-principal) or caller tier 1Per-user speech preferences.
group_preferencePer-agent tier 2 (agent’s principal is a group-principal) or caller tier 2Per-group speech preferences.
global_defaultTier 3 — global defaultAdmin (Settings → Speech, or POST .../set-default).
null (no source)Tier 4 — no config resolvedCascade returns [null, null, null]; SPA renders “No speech provider configured”.

null only when no provider class is registered at all — the SPA treats that as “install a speech provider” rather than “configure one”.

MIME negotiation

The browser’s MediaRecorder produces whatever its codec table says it can. Spora can’t dictate the wire — it can only steer the picker. Each provider declares the MIMEs it accepts via #[AcceptedAudioMime] attributes, in preference order:

// MiniMax rejects the Matroska container (HTTP 502 / error 2013) even
// though the underlying Opus codec is identical to OGG-over-Opus. We
// steer Chrome ≥ 105 onto audio/ogg;codecs=opus first.
#[AcceptedAudioMime('audio/ogg;codecs=opus')]
#[AcceptedAudioMime('audio/mp4')]
#[AcceptedAudioMime('audio/webm;codecs=opus')]
final class MiniMaxTranscribeProvider implements SpeechToTextProviderInterface {}

The SPA walks the resolved row’s preferred_audio_mimes[] and picks the first one whose MediaRecorder.isTypeSupported() returns true. When the provider doesn’t declare any (or doesn’t exist), the registry falls back to the common-superset default:

private const DEFAULT_PREFERRED_AUDIO_MIMES = [
    'audio/webm;codecs=opus',
    'audio/ogg;codecs=opus',
    'audio/mp4',
    'audio/webm',
    'audio/wav',
];

The full mime → container mapping the providers themselves must support lives in MediaAllowedTypesService::AUDIO_MIME_TYPES (upload allow-list) and OpenAiCompatibleTranscriber::extensionFor() (multipart filename mapping). The two deliberately include video/webm and video/mp4 to cover Safari’s MediaRecorder quirk of labelling audio-only recordings with the video container type — the byte sniffer can’t tell them apart.

How the registry calls a provider

When the transcribe endpoint runs, the flow is:

  1. Decode body — { media_id, language?, agent_id? }. The controller validates the media_id is a non-empty UUID; language is a BCP-47 hint or null; agent_id is optional and the controller validates ownership.
  2. Resolve provider — SpeechToTextRegistry::configuredProvider($userId, $agentId) walks the cascade, picks a class, binds the resolved SpeechProviderConfiguration row’s label and settings into the provider via bindLabel() and bindSettings(), and gates on isConfigured(). A null return → HTTP 503 SPEECH_PROVIDER_UNAVAILABLE.
  3. Read asset bytes — MediaAssetReader::readAsset(). data_url / local rows return bytes; external rows (just a source_url pointer) → 422 INVALID_AUDIO; missing / unauthorized / legacy storage mode → 404 MEDIA_NOT_FOUND.
  4. Transcribe — $provider->transcribe($bytes, $mime, $language, $agentId=0, $userId). InvalidAudioException → 422; SpeechToTextException (provider-side failure) → 502 with the sanitised message. API keys MUST NOT appear in the message — both SpeechToTextException and the provider’s classifyTransportFailure()/classifyHttpFailure() helpers enforce this.
  5. Cache — write text / language back to media_assets.transcript / media_assets.transcript_language so chat re-renders re-use the cached value without re-billing. (Duration is not persisted.)

The $agentId argument the controller passes to transcribe() is always 0. The cascade resolves the class from agent preferences (so the badge next to the dropdown reflects the agent’s tier), but the provider reads user / group / global settings from the resolved SpeechProviderConfiguration directly. Per-agent STT overrides via agent_tool_settings are out of scope.

Error model

StatusCodeWhen
200—Success. Body includes text, language?, duration_ms?.
401UNAUTHORIZEDNo session bound to the request.
404MEDIA_NOT_FOUNDmedia_id not visible to the caller (missing, unauthorized, or legacy storage mode). No existence leak between reasons.
422INVALID_AUDIOAsset is storage_mode=external (URL only, no bytes) or the provider rejected the MIME.
422VALIDATION_ERRORMalformed body, or agent_id was supplied but doesn’t belong to the caller.
502SPEECH_PROVIDER_FAILEDProvider raised SpeechToTextException — transport, HTTP 4xx/5xx, empty transcript. The full message is logged server-side; the wire message is sanitised.
503SPEECH_PROVIDER_UNAVAILABLEThe cascade found no provider, or the resolved provider’s isConfigured() returned false. The 503 message lists the remediation: “Add an API key in Settings → Tools for OpenAI-compatible STT (Mistral, OpenAI Whisper, Groq, etc.) or install a speech plugin.”

The /provider-configs/* endpoints use their own error codes — SPEECH_PROVIDER_CONFIG_NOT_FOUND (404), SPEECH_PROVIDER_CONFIG_FORBIDDEN (403), SPEECH_PROVIDER_CONFIG_INVALID (422).

Operator lifecycle

The expected operator flow when adding a second vendor:

  1. Install the plugin (or add a second SpeechProviderConfiguration row for a built-in provider like OpenAI-compatible). The capability endpoint now lists two rows.
  2. Create a configuration via POST /api/v1/speech/provider-configs with {provider_class, display_name, is_global: true, settings: {...}}. Admins only for global rows; users can create principal-scoped rows under a principal they own.
  3. Bind a preference — either PUT /api/v1/speech/preference (the caller’s user-principal) or PATCH /api/v1/agents/{id} setting speech_driver_config_id (per-agent override).
  4. Mark a default for global rows via POST /api/v1/speech/provider-configs/{id}/set-default (admin only; demotes the previous default in a lockForUpdate transaction).

Deleting a configuration detaches every FK reference first — agents.speech_driver_config_id is nulled and principal_preferences.preferred_speech_config_id is nulled (the preference row itself stays put, but with no FK behind it). A stale reference can never resolve to a deleted row, and SpeechProviderConfigPersistence::detachConfigurationReferencesStatic() is a belt-and-braces pass on top of the ON DELETE SET NULL FK semantics on the agent column.

What’s next