UTF-8 sanitizer

About 970 wordsAbout 3 min

Spora\Services\Text\Utf8Sanitizer is a stateless utility that repairs non-UTF-8 byte sequences in user-supplied or upstream-supplied text. Every write path in spora-core that lands a string in a TEXT column runs the incoming value through this utility first, so the bytes that end up on disk are always valid UTF-8 and any subsequent JsonResponse can serialize the row without throwing.

The problem

The connection default is utf8mb4 / utf8mb4_unicode_ci (see Spora\Core\Database), but that only governs new bytes. A row that was written before the column was utf8mb4, or that arrived through a code path that dropped raw bytes without validating them, can still contain stray 0x80 / 0xC0 / Latin-1 sequences.

The first symptom is a 500 from the HTTP layer:

Invalid value for "json" option: Malformed UTF-8 characters, possibly incorrectly encoded

The exception originates in vendor/symfony/http-foundation/JsonResponse.php when json_encode returns false with JSON_ERROR_UTF8. It happens long after the row was persisted — every controller that returns the corrupt row as JSON will trip the same wire.

Why: Closing the gap at the write site means there is exactly one place to audit (“does this code path call Utf8Sanitizer::scrubString?”), instead of trying to scrub at every JSON serialization point across the admin UI.

The utility

namespace Spora\Services\Text;

final class Utf8Sanitizer
{
    public static function scrub(mixed $value): mixed;
    public static function scrubString(string $value): string;
    public static function isValid(string $value): bool;
}

scrubString(string) — returns valid UTF-8, salvaging Windows-1252 / ISO-8859-1 bytes, dropping anything unrecognisable via iconv //IGNORE.
scrub(mixed) — dispatches on type: strings get scrubbed, arrays get recursed, everything else passes through. Use this when the shape of the value isn’t known at compile time (request bodies, Eloquent ->fill([...]) payloads).
isValid(string) — cheap, allocation-free check for callers that want to gate on UTF-8 without mutating.

What the algorithm does

Three-step escalation, each step short-circuited by the previous one:

mb_check_encoding($value, 'UTF-8') — passes valid UTF-8 through unchanged. This is the hot path; every tool result that flows through the orchestrator is already valid UTF-8 and exits here in ~0.5 µs.
iconv('UTF-8', 'UTF-8//IGNORE', $value) — cheap drop pass tried first when the input is mostly valid UTF-8 with a few stray invalid bytes. Those bytes are dropped and the surrounding text is preserved. Only fall through if iconv dropped everything or returned an invalid result.
mb_convert_encoding($value, 'UTF-8', $encoding) — salvage via Windows-1252 (covers smart quotes, em-dashes, the Euro sign in the 0x80–0x9F range) then ISO-8859-1 as the universal fallback. Together these cover every byte 0x00–0xFF, so the salvage chain always produces a valid UTF-8 string for any PHP string input. A final iconv //IGNORE pass in scrubString strips any bytes that somehow survived — a defensive guard, not a reachable fallback.

Why: iconv //IGNORE first because it’s the cheapest of the three paths (one C-level syscall) and handles the common case of “mostly valid UTF-8 with a stray bad byte” without reinterpreting the rest of the input. Windows-1252 second because it covers 0x80–0x9F (smart quotes, em-dashes, the Euro sign) that ISO-8859-1 leaves as control characters. ISO-8859-1 last as the universal fallback — every byte 0x00–0xFF is defined in either Windows-1252 or ISO-8859-1, so the salvage chain always succeeds for any PHP string input.

For plugin authors

If your plugin writes through a spora-core service — for example MediaArchiveService::ingest, ToolCallExecutor’s wrapped executeAndRecordResult, or ApprovedBatchExecutor — the core service already scrubs the inbound field. You don’t need to opt in. Add the wrap yourself only if you write Eloquent directly to a TEXT column with no spora-core wrapper in front of it (for example, the memories table in spora-plugin-memories — see MemoryService).

When you do need the utility, import it the same way you import any other core class:

use Spora\Services\Text\Utf8Sanitizer;

$content = Utf8Sanitizer::scrubString(
    (string) ($arguments['content'] ?? '')
);

For Eloquent ->fill([...]) payloads, use scrub() so nested arrays recursively scrub every string leaf:

$asset->fill(Utf8Sanitizer::scrub([
    'filename' => $request->filename,
    'tags'     => $request->tags,
    'metadata' => $request->metadata,
]));

PSR-4 import: the utility lives in app/Services/Text/Utf8Sanitizer.php inside spora-ai/spora-core. Any plugin that already requires spora-core gets the class for free — no composer.json change.

Caveats

Utf8Sanitizer does not:

Detect homograph attacks (visually-confusable characters from different scripts). That’s a separate concern — see the SSRF allowlist in core.
Repair mojibake (a string that was double-encoded UTF-8 → Latin-1 → UTF-8). The salvager interprets raw bytes as the first non-UTF-8 encoding it recognises, never reverses a known re-encoding.
Normalise Unicode (NFC / NFD / NFKC). A string like café written as the four code points é stays that way; the utility makes bytes valid UTF-8, it does not canonicalise.

Why: Each of these is a different problem with a different algorithm. The scraper has one job — make bytes that don’t break json_encode — and stays out of the higher-level normalisation questions that belong in their own utilities.