Diagnostics

Tokenizer's diagnostics system records every matching decision, generates contextual hints for failures, and renders visual diffs between templates and input. Use it to understand why a pattern isn't matching.

Enabling Diagnostics

Diagnostics are disabled by default for performance. Enable them via options:

var tokenizer = new Tokenizer(new TokenizerOptions
{
    EnableDiagnostics = true
});

var result = tokenizer.Tokenize(template, input);

if (result.Diagnostics is not null)
{
    Console.WriteLine(result.Diagnostics.Verdict);
    Console.WriteLine(result.Diagnostics.RenderAlignment());

    foreach (var token in result.Diagnostics.Tokens)
    {
        Console.WriteLine($"{token.TokenName}: {token.Outcome}");
        foreach (var issue in token.Issues)
        {
            Console.WriteLine($"  {issue.Type}: {issue.Description}");
            if (issue.Hint is not null)
                Console.WriteLine($"    Hint: {issue.Hint}");
        }
    }
}

Or via DI:

services.AddTokenizer(new TokenizerOptions { EnableDiagnostics = true });

TokenizationDiagnostics

When diagnostics are enabled, result.Diagnostics contains:

Property	Type	Description
`Tokens`	`IReadOnlyList<TokenDiagnostic>`	Per-token diagnostic narratives
`Verdict`	`string`	Human-readable summary, e.g. "Matched 3 of 5 tokens (2 missed)."
`MatchedCount`	`int`	Number of matched tokens
`MissedCount`	`int`	Number of missed tokens
`TotalCount`	`int`	Total token count
`RawEvents`	`IReadOnlyList<TokenizationEvent>`	Full event trace of every matching decision
`RenderAlignment()`	`string`	Visual alignment diff between template and input (cached)
`RenderProcessingOrder()`	`string`	Chronological engine decision walkthrough (cached)

TokenDiagnostic

Each token in the template gets a TokenDiagnostic record describing what happened during matching:

Property	Type	Description
`TokenName`	`string`	Token name from the template
`TokenId`	`int`	Internal token identifier
`Outcome`	`TokenOutcome`	`Matched`, `Rejected`, `NeverFound`, or `Blocked`
`Attempts`	`IReadOnlyList<TokenAttempt>`	Each match attempt with outcome and reason
`AssignedValue`	`string?`	Final assigned value (if matched)
`AssignedLocation`	`FileLocation?`	Location in input where value was found
`BlockedBy`	`string?`	Name of the required token that blocked this one
`Issues`	`IReadOnlyList<DiagnosticIssue>`	Issues with contextual hints

TokenAttempt

Each attempt to assign a value to a token is recorded:

Property	Type	Description
`Location`	`FileLocation?`	Position in the input
`Value`	`string?`	Attempted value
`Outcome`	`AttemptOutcome`	`Assigned`, `ValidatorRejected`, `TransformerFailed`, or `Backtracked`
`DecoratorName`	`string?`	Decorator that caused rejection or failure
`Reason`	`string?`	Human-readable explanation

Alignment Rendering

The alignment renderer produces a visual diff showing exactly how the template mapped (or failed to map) to the input. Call result.Diagnostics.RenderAlignment():

var tokenizer = new Tokenizer(new TokenizerOptions { EnableDiagnostics = true });
var compiled = tokenizer.Compile("Name: {Name}, Email: {Email}");
var result = tokenizer.Tokenize(compiled.Template, "Name: Alice, Email: alice@example.com");

Console.WriteLine(result.Diagnostics?.RenderAlignment());

Try the playground with a broken pattern to see alignment in action:

Processing Order

The processing order renderer produces a chronological walkthrough of every engine decision — which tokens were attempted, in what order, and why each succeeded or failed. Call result.Diagnostics.RenderProcessingOrder():

var tokenizer = new Tokenizer(new TokenizerOptions { EnableDiagnostics = true });
var compiled = tokenizer.Compile("Name: {Name}, Email: {Email}");
var result = tokenizer.Tokenize(compiled.Template, "Name: Alice, Email: alice@example.com");

Console.WriteLine(result.Diagnostics?.RenderProcessingOrder());

Hint Generators

The diagnostics system includes nine hint generators that provide contextual suggestions when tokens fail to match:

Generator	Detects	Example Hint
PreambleNearMiss	Preamble text that almost matches (wrong case, substring match)	"Preamble 'Email:' not found — did you mean 'email:'?"
DateFormat	DateTime parsing failures with format suggestions	"Value '25/12/2025' doesn't match format 'yyyy-MM-dd'. Try 'dd/MM/yyyy'."
ValidatorValue	Why a specific validator rejected a value	"IsInRange(1,100) failed: value '200' is outside range."
RepeatingToken	Repeating token behavior that may be unexpected	"Token 'Items' matched 3 times. Use {Items*} for repeating tokens."
BlockedToken	Token blocked because a prior required token failed	"Token 'Email' was blocked because required token 'Name' failed."
ChainedDecorator	Failure partway through a decorator chain	"Transformer 'ToDate' failed after 'Trim' succeeded — check input format."
MultipleRejection	Value rejected by multiple validators	"Value rejected by 2 validators: IsEmail, HasLength."
OptionalToken	Optional token not found (informational, not an error)	"Optional token 'Title' was not found — this is not an error."
ValueMismatch	Greedy capture consumed too much input	"Token 'Name' captured 'Alice, Age: 30' — the next preamble may be missing."

Using Diagnostics in Your Code

A practical pattern for debugging tokenization in your application:

var result = tokenizer.Tokenize(template, input);

if (!result.Success)
{
    // Log the alignment for visual debugging
    logger.LogWarning("Tokenization failed:\n{Alignment}",
        result.Diagnostics?.RenderAlignment());

    // Log per-token diagnostics
    if (result.Diagnostics is not null)
    {
        foreach (var token in result.Diagnostics.Tokens)
        {
            if (token.Issues.Count == 0) continue;

            foreach (var issue in token.Issues)
            {
                logger.LogWarning("[{Token}] {Type}: {Description}. {Hint}",
                    token.TokenName,
                    issue.Type,
                    issue.Description,
                    issue.Hint ?? "");
            }
        }
    }
}

For production, consider enabling diagnostics only when a tokenization fails, by re-running with diagnostics on:

var result = tokenizer.Tokenize(template, input);

if (!result.Success)
{
    // Re-run with diagnostics for the log
    var diagTokenizer = new Tokenizer(new TokenizerOptions { EnableDiagnostics = true });
    var diagResult = diagTokenizer.Tokenize(template, input);
    logger.LogWarning("Diagnostics:\n{Alignment}", diagResult.Diagnostics?.RenderAlignment());
}