Configuration

Tokenizer has two configuration surfaces: TokenizerOptions (set at construction time) and YAML front matter (set per-template). Front matter overrides constructor options, which override defaults.

Options Reference

Pass a TokenizerOptions instance to the Tokenizer or TemplateMatcher constructor:

var tokenizer = new Tokenizer(new TokenizerOptions
{
    OutOfOrderTokens = true,
    TrimTrailingWhiteSpace = false,
    EnableDiagnostics = true
});

Matching Behaviour

Option	Default	Description
`OutOfOrderTokens`	`false`	Allow tokens to match in any order rather than strictly left-to-right
`TerminateOnNewLine`	`false`	Extract token values up to the first newline only
`TokenStringComparison`	`InvariantCulture`	String comparison used when matching token names to object properties
`IgnoreMissingProperties`	`false`	Silently ignore tokens that don't map to a property on the target object

Whitespace

Option	Default	Description
`TrimTrailingWhiteSpace`	`true`	Trim trailing whitespace from each extracted token value
`TrimLeadingWhitespaceInTokenPreamble`	`true`	Trim leading whitespace in the static text before a token
`TrimPreambleBeforeNewLine`	`false`	Discard preamble text that appears before a newline

Safety Limits

Option	Default	Description
`MaxInputLength`	`1,048,576` (1 MB)	Maximum input text length in characters (0 to disable)
`MaxTemplateLength`	`65,536` (64 KB)	Maximum template pattern length (0 to disable)
`MaxTokenCount`	`500`	Maximum tokens per template (0 to disable)
`MaxIterations`	`0` (auto)	Maximum tokenization loop iterations. Auto-calculated as input length × 2
`MaxRegexTimeout`	`1 second`	Maximum time for a single regex evaluation in user-supplied patterns

Date/Time

Option	Default	Description
`Culture`	`null` (invariant)	Culture for parsing date/time values (month names, day names)
`DefaultOffset`	`null`	Static UTC offset for date/time values with no offset info. Takes precedence over `DefaultTimezone`
`DefaultTimezone`	`null`	IANA or Windows timezone ID (e.g. `"Europe/Berlin"`) for DST-aware offset resolution

Other

Option	Default	Description
`EnableDiagnostics`	`false`	Include structured diagnostic trace in results. No performance impact when disabled
`AllowStreamBuffering`	`false`	Buffer non-seekable streams for multi-template matching

Front Matter

You can override options per-template using YAML front matter between --- markers at the start of the template. Lines starting with # are comments.

var pattern = @"---
# Template-level configuration
name: my-template
OutOfOrder: true
TrimTrailingWhitespace: false
CaseSensitive: false
TerminateOnNewLine: true
culture: en-GB
defaultOffset: +00:00
---
First Name: {FirstName}
Last Name: {LastName}";

Available Directives

Directive	Maps To
`OutOfOrder`	`OutOfOrderTokens`
`TrimLeadingWhitespace`	`TrimLeadingWhitespaceInTokenPreamble`
`TrimTrailingWhitespace`	`TrimTrailingWhiteSpace`
`TrimPreambleBeforeNewLine`	`TrimPreambleBeforeNewLine`
`TerminateOnNewLine`	`TerminateOnNewLine`
`IgnoreMissingProperties`	`IgnoreMissingProperties`
`CaseSensitive`	`TokenStringComparison` (`true` = `InvariantCulture`, `false` = `InvariantCultureIgnoreCase`)
`culture`	`Culture`
`defaultOffset`	`DefaultOffset`
`defaultTimezone`	`DefaultTimezone`
`name`	Template name (for `TemplateMatcher`)
`tag`	Template tag (repeatable)
`hint` / `hint?`	Matching hint (required / optional)

Boolean directives accept true/false, yes/no, or on/off (case-insensitive).

Precedence

When the same option is set in multiple places, the most specific value wins:

Front matter (per-template) — highest priority
Constructor options (TokenizerOptions) — instance-level defaults
Built-in defaults — the values shown in the tables above

This means you can set sensible defaults on the Tokenizer instance and override them for specific templates using front matter.