Advanced

Async Operations

Tokenizer supports async compilation and tokenization from streams, useful for processing files or network data without loading everything into memory:

// Compile from a file
using var reader = new StreamReader("template.txt");
var compiled = await tokenizer.CompileAsync(reader);

// Tokenize from a stream
using var inputReader = new StreamReader("input.txt");
var result = await tokenizer.TokenizeAsync<MyClass>(
    compiled.Template, inputReader, cancellationToken);

All async methods accept a CancellationToken for cooperative cancellation:

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));

try
{
    var result = await tokenizer.TokenizeAsync(template, reader, cts.Token);
}
catch (OperationCanceledException)
{
    Console.WriteLine("Tokenization timed out");
}

Safety Limits

Tokenizer v3.0 includes built-in safety limits to prevent resource exhaustion from malformed or adversarial input:

Limit	Default	Description
`MaxInputLength`	1,048,576 (1 MB)	Maximum input text length in characters
`MaxTemplateLength`	65,536 (64 KB)	Maximum template pattern length
`MaxTokenCount`	500	Maximum number of tokens per template
`MaxIterations`	Auto (2× input length)	Maximum engine iterations before aborting

Configure limits via options:

var tokenizer = new Tokenizer(new TokenizerOptions
{
    MaxInputLength = 5_000_000,  // 5 MB
    MaxTokenCount = 100
});

Real-World Examples

WHOIS data parsing

var template = @"---
TerminateOnNewLine: true
---
   Domain Name: {DomainName}
   Registrar: {Registrar}
   Updated Date: {UpdatedDate:ToDateTime('yyyy-MM-ddTHH:mm:ssZ')}
   Creation Date: {CreationDate:ToDateTime('yyyy-MM-ddTHH:mm:ssZ')}
   Registry Expiry Date: {ExpiryDate:ToDateTime('yyyy-MM-ddTHH:mm:ssZ')}
   Name Server: {NameServers*}";

Log file extraction

var template = "[{Timestamp$}] [{Level$}] {Message}";
var input = "[2025-01-15 14:30:22] [ERROR] Connection refused to database server";

var result = tokenizer.Tokenize(compiled.Template, input);
// Timestamp = "2025-01-15 14:30:22", Level = "ERROR", Message = "Connection refused..."

Structured email parsing

var template = @"---
TerminateOnNewLine: true
---
From: {From}
To: {To}
Subject: {Subject}
Date: {Date:ToDateTime('ddd, dd MMM yyyy HH:mm:ss')}";

Performance

Templates are compiled once and can be reused across many tokenization operations. The compilation step (parsing, AST generation, decorator resolution) is the expensive part — tokenization itself is a single-pass operation.

// Compile once
var compiled = tokenizer.Compile(pattern);

// Reuse many times
foreach (var line in inputLines)
{
    var result = tokenizer.Tokenize(compiled.Template, line);
    // ...
}

For TokenMatcher, templates are compiled on registration and cached internally. The matcher itself is thread-safe for concurrent reads.

Error Handling

Tokenizer uses a specific exception hierarchy:

Exception	When
`LexerException`	Template pattern has invalid syntax (includes line/column)
`ParsingException`	Template AST cannot be constructed (includes line/column)
`TokenAssignmentException`	Extracted value cannot be assigned to target property
`TypeConversionException`	Transformer cannot convert the value (includes target type)
`ValidationException`	Validator setup error (not validation failure — those are handled gracefully)
`TokenMatcherException`	Error during multi-template matching

All inherit from TokenizerException. Normal validation failures (e.g., IsEmail() rejecting an invalid address) do not throw — they cause the engine to skip and continue searching.

try
{
    var compiled = tokenizer.Compile(pattern);
    var result = tokenizer.Tokenize(compiled.Template, input);
}
catch (Tokens.Exceptions.LexerException ex)
{
    Console.WriteLine($"Template syntax error at line {ex.Line}, column {ex.Column}: {ex.Message}");
}
catch (Tokens.Exceptions.TokenizerException ex)
{
    Console.WriteLine($"Tokenizer error: {ex.Message}");
}