Tokenizer

A .NET library that extracts structured data from unstructured text using template patterns. Define a pattern with token placeholders, pass in text, and Tokenizer pulls out the values — with built-in type conversion, validation, and diagnostics.

C# 
var tokenizer = new Tokenizer();
var template = tokenizer.Compile("Name: {FirstName}, Age: {Age:ToInt()}");
var result = tokenizer.Tokenize<Person>(template.Template, "Name: Alice, Age: 30");

Console.WriteLine(result.Value.FirstName); // "Alice"
Console.WriteLine(result.Value.Age);       // 30

Installation

NuGet
Bash 
dotnet add package Tokenizer
Package Manager
Bash 
Install-Package Tokenizer

Supports .NET Standard 2.0, .NET 8.0, and .NET 10.0.

Quick Start

The core workflow has three steps: create a tokenizer, compile a template pattern, and tokenize input text against it.

C# 
using Tokens;

// 1. Create a tokenizer
var tokenizer = new Tokenizer();

// 2. Compile a template pattern (reusable)
var compiled = tokenizer.Compile("Email: {Email}, Role: {Role}");

// 3. Tokenize input text
var result = tokenizer.Tokenize(compiled.Template, "Email: alice@example.com, Role: Admin");

if (result.Success)
{
    Console.WriteLine(result.First("Email")); // "alice@example.com"
    Console.WriteLine(result.First("Role"));  // "Admin"
}

You can also tokenize directly onto a typed object:

C# 
public class User
{
    public string Email { get; set; }
    public string Role { get; set; }
}

var result = tokenizer.Tokenize<User>(compiled.Template, input);
var user = result.Value; // User { Email = "alice@example.com", Role = "Admin" }

Dependency Injection

Register Tokenizer in your DI container using the extension method:

C# 
using Tokens.Extensions;

// Default options
services.AddTokenizer();

// From configuration
services.AddTokenizer(configuration.GetSection("Tokenizer"));

// With explicit options
services.AddTokenizer(new TokenizerOptions
{
    EnableDiagnostics = true,
    TrimTrailingWhiteSpace = true
});

Then inject ITokenizer wherever you need it:

C# 
public class MyService
{
    private readonly ITokenizer _tokenizer;

    public MyService(ITokenizer tokenizer)
    {
        _tokenizer = tokenizer;
    }
}