uCalc API Version: 2.1.3-preview.2 Released: 6/17/2026

Warning

uCalc API Preview Release Notice:The documentation describes the intended behavior of the API. The current preview build contains incomplete features, unoptimized performance, and is subject to breaking changes.

TokenTransformer = [Transformer]

Property

Product: 

Fast Math Parser

Class: 

uCalcBase

Retrieves the dedicated transformer for defining token-level transformations, enabling custom syntax like string interpolation or hex literals.

Remarks

The TokenTransformer() method returns the dedicated Transformer object used for pre-processing specific tokens during the lexical analysis phase of an expression. This allows for powerful, targeted syntax transformations like string interpolation, custom numeric notations (hex/binary), or special escape sequences.

⚙️ How It Works: The Trigger Mechanism

Unlike the global ExpressionTransformer, which processes every expression string, the TokenTransformer is only activated when the parser encounters a token that has been specifically registered with the type TokenType::TokenTransform. This makes it exceptionally efficient.

The workflow is:

  1. Define a Trigger: Register a regex pattern with ExpressionTokens().Add() and assign it the type TokenType::TokenTransform. For example, a pattern for C-style hex numbers: uc.ExpressionTokens().Add("0x[0-9A-F]+", TokenType::TokenTransform);
  2. Define the Rule: Add a FromTo rule to the TokenTransformer that matches the token's structure and replaces it with a valid uCalc expression. uc.TokenTransformer().FromTo("{'0x'}{val:'[0-9A-F]+'}", "BaseConvert(val, 16)");
  3. Evaluate: When Parse or EvalStr encounters text matching the trigger (e.g., "0xFF"), the TokenTransformer runs, replaces it with "BaseConvert('FF', 16)", and the parser continues with the transformed result.

TokenTransformer vs. ExpressionTransformer

Choosing the right tool is critical for performance and maintainability.

AspectTokenTransformer (This Topic)ExpressionTransformer
ScopeSurgical: Only runs when a specific TokenTransform token is found.Global: Runs on the entire expression string for every call to Parse or Eval.
Performance🚀 High. Zero overhead if the trigger tokens are not present.🐌 Lower. Introduces overhead on every parse, even if no rules match.
Use CaseIdeal for implementing new low-level syntax (literals, interpolation, escapes).Ideal for high-level syntactic sugar or transpiling language constructs (e.g., x^y to pow(x,y)).

Rule of Thumb: Use TokenTransformer for things that feel like new types of literals or tokens. Use ExpressionTransformer for things that feel like new operators or function call syntax.

💡 Comparative Analysis

  • vs. Manual String Parsing/Regex: A standard regex approach struggles with context (e.g., is $ an interpolation marker or part of a variable name inside a string?). Because TokenTransformer is integrated with the lexer, it operates with full awareness of existing language structures.
  • vs. Lexer Generators (ANTLR, Flex): Traditional compiler tools require an offline, static build step. uCalc's engine is fully dynamic. You can add or modify token transformation rules at runtime, allowing an application to extend its own syntax on the fly.

Examples

Extends the parser to support C-style `0x` hex and `0b` binary notations using a token transformer.
				
					using uCalcSoftware;

var uc = new uCalc();
// Define a token for C-like 0x hex notation
uc.ExpressionTokens.Add("0x[0-9A-Fa-f]+", TokenType.TokenTransform);
uc.TokenTransformer.FromTo("{'0x'}{Num:'[0-9A-Fa-f]+'}", "BaseConvert('{Num}', 16)");
Console.WriteLine($"0xFF is evaluated as: {uc.EvalStr("0xFF")}");

// Define a token for C++-style 0b binary notation
uc.ExpressionTokens.Add("0b[01]+", TokenType.TokenTransform);

// Using {@Eval} is more efficient as the conversion happens once during the token transform pass.
uc.TokenTransformer.FromTo("{'0b'}{Num:'[01]+'}", "{@Eval: BaseConvert(Num, 2)}");
Console.WriteLine($"0b1011 is evaluated as: {uc.EvalStr("0b1011")}");

// Note: uCalc has built-in support for hex/binary using # notation (e.g., #hFF, #b1011)
Console.WriteLine($"uCalc's built-in #hFF is: {uc.EvalStr("#hFF")}");
				
			
0xFF is evaluated as: 255
0b1011 is evaluated as: 11
uCalc's built-in #hFF is: 255
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // Define a token for C-like 0x hex notation
   uc.ExpressionTokens().Add("0x[0-9A-Fa-f]+", TokenType::TokenTransform);
   uc.TokenTransformer().FromTo("{'0x'}{Num:'[0-9A-Fa-f]+'}", "BaseConvert('{Num}', 16)");
   cout << "0xFF is evaluated as: " << uc.EvalStr("0xFF") << endl;

   // Define a token for C++-style 0b binary notation
   uc.ExpressionTokens().Add("0b[01]+", TokenType::TokenTransform);

   // Using {@Eval} is more efficient as the conversion happens once during the token transform pass.
   uc.TokenTransformer().FromTo("{'0b'}{Num:'[01]+'}", "{@Eval: BaseConvert(Num, 2)}");
   cout << "0b1011 is evaluated as: " << uc.EvalStr("0b1011") << endl;

   // Note: uCalc has built-in support for hex/binary using # notation (e.g., #hFF, #b1011)
   cout << "uCalc's built-in #hFF is: " << uc.EvalStr("#hFF") << endl;
}
				
			
0xFF is evaluated as: 255
0b1011 is evaluated as: 11
uCalc's built-in #hFF is: 255
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// Define a token for C-like 0x hex notation
      uc.ExpressionTokens.Add("0x[0-9A-Fa-f]+", TokenType.TokenTransform)
      uc.TokenTransformer.FromTo("{'0x'}{Num:'[0-9A-Fa-f]+'}", "BaseConvert('{Num}', 16)")
      Console.WriteLine($"0xFF is evaluated as: {uc.EvalStr("0xFF")}")
      
      '// Define a token for C++-style 0b binary notation
      uc.ExpressionTokens.Add("0b[01]+", TokenType.TokenTransform)
      
      '// Using {@Eval} is more efficient as the conversion happens once during the token transform pass.
      uc.TokenTransformer.FromTo("{'0b'}{Num:'[01]+'}", "{@Eval: BaseConvert(Num, 2)}")
      Console.WriteLine($"0b1011 is evaluated as: {uc.EvalStr("0b1011")}")
      
      '// Note: uCalc has built-in support for hex/binary using # notation (e.g., #hFF, #b1011)
      Console.WriteLine($"uCalc's built-in #hFF is: {uc.EvalStr("#hFF")}")
   End Sub
End Module
				
			
0xFF is evaluated as: 255
0b1011 is evaluated as: 11
uCalc's built-in #hFF is: 255
Previous
Next