Extensibility

You can extend Tokenizer with your own transformers and validators. Implement the interface, register via options, and use your decorator in templates just like the built-in ones.

Custom Transformers

A transformer modifies an extracted token value before it's assigned. Implement ITokenTransformer with a single method:

public interface ITokenTransformer : ITokenDecorator
{
    bool TryTransform(object value, string[] args, out object transformed);
}

Return true and set transformed to the new value on success. Return false to leave the value unchanged (the engine treats this as a failed transformation and skips the current match).

Example: ReverseTransformer

using Tokens.Transformers;

public sealed class ReverseTransformer : ITokenTransformer
{
    public bool TryTransform(object value, string[] args, out object transformed)
    {
        if (value is string s)
        {
            transformed = new string(s.Reverse().ToArray());
            return true;
        }

        transformed = value;
        return false;
    }
}

Use it in a template as {Name:Reverse()} after registering it.

Custom Validators

A validator accepts or rejects an extracted value. If it rejects, the engine skips the current match and continues searching for the next one. Implement ITokenValidator:

public interface ITokenValidator : ITokenDecorator
{
    bool IsValid(object value, params string[] args);
}

Example: IsUpperCaseValidator

using Tokens.Validators;

public sealed class IsUpperCaseValidator : ITokenValidator
{
    public bool IsValid(object value, params string[] args)
    {
        return value is string s && s == s.ToUpperInvariant();
    }
}

Use it in a template as {Code:IsUpperCase()} — any value that isn't all uppercase will be skipped.

Registration

Register custom decorators via TokenizerOptions using the fluent WithTransformer<T>() and WithValidator<T>() methods:

var options = new TokenizerOptions()
    .WithTransformer<ReverseTransformer>()
    .WithValidator<IsUpperCaseValidator>();

var tokenizer = new Tokenizer(options);

Decorators must be registered before templates are compiled — decorator resolution happens at compile time, not at tokenization time. The decorator name used in templates is derived from the class name: ReverseTransformer becomes Reverse, IsUpperCaseValidator becomes IsUpperCase.

Options-Aware Decorators

If your decorator needs access to culture, timezone, or other options at runtime, implement IOptionsAwareTransformer or IOptionsAwareValidator instead. These extend the base interfaces with an overload that receives TokenizerOptions:

public interface IOptionsAwareTransformer : ITokenTransformer
{
    bool TryTransform(object value, string[] args, TokenizerOptions options, out object transformed);
}

public interface IOptionsAwareValidator : ITokenValidator
{
    bool IsValid(object value, string[] args, TokenizerOptions options);
}

The engine calls the options-aware overload when available. You still need to implement the base interface method — it acts as a fallback:

using System.Globalization;
using Tokens.Transformers;

public sealed class CultureAwareTransformer : IOptionsAwareTransformer
{
    public bool TryTransform(object value, string[] args, out object transformed)
    {
        // Fallback: use invariant culture
        return TryTransform(value, args, new TokenizerOptions(), out transformed);
    }

    public bool TryTransform(object value, string[] args, TokenizerOptions options, out object transformed)
    {
        var culture = options.Culture ?? CultureInfo.InvariantCulture;
        if (value is string s)
        {
            transformed = s.ToUpper(culture);
            return true;
        }

        transformed = value;
        return false;
    }
}