Pattern Syntax

Tokenizer uses template patterns to extract values from text. A pattern combines static text (preambles) with token placeholders that capture data.

Preambles

A preamble is the static text immediately before a token. The tokenizer scans the input for each preamble and extracts the text that follows it.

// Pattern:  "Name: {Name}, Age: {Age}"
//            ^^^^^^        ^^^^^^
//            preamble      preamble
//
// Input:    "Name: Alice, Age: 30"
//                  ^^^^^      ^^
//                  extracted  extracted

Text is extracted from after the preamble until the next preamble is found, a newline is reached, or the input ends.

Token Syntax

Tokens are enclosed in curly braces. The token name maps to a property on your target object.

Plain text

{TokenName}

Token names are case-sensitive and must match your C# property names exactly.

Modifiers

Append a modifier character to change how a token behaves:

Modifier	Syntax	Behavior
Required	`{Token!}`	Tokenization fails if this token is not matched
Optional	`{Token?}`	Skipped if not found — tokenization continues
Repeating	`{Token*}`	Matches multiple times, values collected into a `List<string>`
Line-terminated	`{Token$}`	Extraction stops at the next newline

Required tokens

Mark a token as required with !. If the token is not found in the input, result.Success will be false.

var template = "Name: {Name!}, Email: {Email!}";
// Both tokens must be present for success

Optional tokens

Mark a token as optional with ?. The tokenizer will skip it if the preamble is not found.

var template = "Name: {Name}\nTitle: {Title?}\nEmail: {Email}";
// Title is optional — works even if "Title:" line is missing

Repeating tokens

Mark a token as repeating with *. The tokenizer matches every occurrence and collects the values into a list. Repeating tokens are implicitly optional.

var template = "Item: {Items*}";
var input = "Item: Apple\nItem: Banana\nItem: Cherry";
// Items = ["Apple", "Banana", "Cherry"]

Line-terminated tokens

The $ modifier stops extraction at the next newline, useful for multi-line input where each value occupies its own line.

var template = "Name: {Name$}\nBio: {Bio}";
// Name captures only the first line, Bio gets the rest

Decorators

Decorators transform or validate token values. Add them after the token name, separated by a colon:

Plain text

{TokenName:Decorator()}
{TokenName:Decorator('argument')}
{TokenName:First(),Second(),Third()}

Decorators are chained left-to-right. There are two types:

Transformers — modify the extracted value (e.g., Trim(), ToInt())
Validators — accept or reject the value (e.g., IsEmail(), IsInRange(1,100))

See the Transformers and Validators pages for complete references.

// Trim whitespace, convert to lowercase, validate as email
var template = "Contact: {Email:Trim(),ToLower(),IsEmail()}";

Front Matter

Templates can include a YAML front matter block to configure tokenization behavior:

Plain text

---
OutOfOrder: true
TrimTrailingWhitespace: true
TerminateOnNewLine: true
---
Name: {Name}
Email: {Email}

Option	Type	Default	Description
`OutOfOrder`	`bool`	`false`	Allow tokens to appear in any order in the input
`CaseSensitive`	`bool`	`true`	Case-sensitive preamble matching
`TrimTrailingWhitespace`	`bool`	`true`	Trim trailing whitespace from extracted values
`TrimLeadingWhitespaceInTokenPreamble`	`bool`	`true`	Trim leading whitespace in preamble text
`TrimPreambleBeforeNewLine`	`bool`	`false`	Trim preamble text before newlines
`TerminateOnNewLine`	`bool`	`false`	All tokens terminate at newline (like `$` on every token)
`ThrowExceptionOnMissingProperty`	`bool`	`false`	Throw if token doesn't match a property on target object
`IgnoreMissingProperties`	`bool`	`false`	Silently ignore tokens that don't map to properties

Token Ordering

By default, tokens must appear in the input in the same order as the template (in-order matching). If a preamble is found before the previous token's preamble, it's skipped.

Enable OutOfOrder to match tokens regardless of their position in the input:

Plain text

---
OutOfOrder: true
---
Name: {Name}
Email: {Email}
Age: {Age}

With out-of-order matching, the input "Age: 30, Name: Alice, Email: alice@example.com" will successfully match all three tokens even though they appear in a different order than the template.