uCalc API Version: 2.1.3-preview.2 Released: 6/17/2026

Warning

uCalc API Preview Release Notice:The documentation describes the intended behavior of the API. The current preview build contains incomplete features, unoptimized performance, and is subject to breaking changes.

Tokens

Product: 

Class: 

Remarks

uCalc uses a tokenizer (also known as a lexer) to split raw text into meaningful units called tokens. Unlike Regex which sees a stream of characters, uCalc sees a stream of words, numbers, and symbols.

Token Categories:Every token in uCalc belongs to a specific category (defined in TokenType Enum). These categories dictate parsing behavior:

  • Generic: The default category if a token is defined without a specific type.
  • Alphanumeric: Standard identifiers and words (e.g., Variable, Function_Name).
  • Literal: Represents data with an associated type (e.g., Numbers, Quoted Strings). These allow the parser to interpret values (integer vs float) immediately.
  • Bracket / BracketMatch: Defined in pairs (e.g., ( and )). If BracketSensitive is enabled (default), uCalc enforces nesting rules automatically.
  • StatementSep: (Statement Separator) Acts as a "Stop" condition for variable captures and delimits statements (e.g., ; or \n).
  • ArgSeparator: Separates items in a list (e.g., ,).
  • MemberAccess: Used for object navigation (e.g., . in Object.Property).
  • Whitespace: Ignored by default during parsing (e.g., Space, Tab).
  • TokenTransform: Special tokens that modify interpretation, such as String Interpolation markers or Hex/Binary prefixes (0x).
  • Reducible: Tokens capable of reduction logic (See Topic 802).
  • Eof: Matches the end of the input string.
  • Empty: Not a real token in text, but returned by API calls when querying a non-existent token definition.

Core Token Concepts:

  • Category: As listed above, this defines behavior.
  • Token Name: While categories group tokens, you can also match a specific token by its unique name (e.g., _token_newline or _token_myCustomSymbol). Names always start with _token_. For details, see Matching by token name.
  • Value: The actual text content (e.g., 123, Hello, "my text").

Why Tokenization Matters:

  • Precision: Matching {@Alpha} ensures you match Apple but not the App inside Apple.
  • Speed: The parser operates on integer token IDs rather than scanning string characters repeatedly.
  • Statefulness: Literals like strings ("my string") are handled as single units, preventing "partial matches" inside quotes.

Comparative Analysis: Why uCalc?

  • vs. Regex (Character-Based):
    • Boundary Hell: In Regex, matching a word requires \bword\b. In uCalc, {@Alpha} automatically respects boundaries defined by the language's syntax, making patterns safer by default.
    • Stateful Parsing: Regex doesn't know that 123 inside a quote "123" is a string. uCalc's tokenizer handles quotes first, allowing you to distinguish {@Number} from {@String} reliably without complex look-aheads.
    • Context Awareness: Regex struggles to distinguish a parenthesis used for grouping (a+b) from a literal parenthesis in a string "(a+b)". uCalc's Literal and Bracket categories handle this distinction natively.
  • vs. String.Split - Intelligence:
    • Split(' ') breaks on spaces inside quotes ("New York" -> New, York). uCalc's tokenizer respects quotes, keeping "New York" as a single token.
    • Split(',') breaks inside function calls fn(a,b). uCalc's logic respects the nesting depth of Brackets, splitting only at the top level (depending on the uCalc.Rule.BracketSensitive configuration)

Examples

Transformer: Matching by tokens vs match by character; also whitespace sensitivity
				
					using uCalcSoftware;

var uc = new uCalc();
// This examples shows the default match by
// token mode, as well as how to reconfigure
// it in order to do match by character
// along with a whitespace variation

var t = uc.NewTransformer();
var txt = "This is an island test, I said.";
t.FromTo("is", "<is>");

Console.WriteLine(t.Transform(txt));
Console.WriteLine("");

t.Tokens.Description = "Match by character";
t.Tokens.Add("."); // This overrides existing tokens
t.FromTo("is", "<is>");
Console.WriteLine(t.Tokens.Description);
Console.WriteLine(t.Transform(txt));
Console.WriteLine("");

// Note: whitespace sensitivity is off by default
// Whitespace token is re-introduced
// (after being overridden in the previous Token Add())
t.Tokens.Description = "By char + whitespace ignored";
t.Tokens.Add("[\\t\\v ]+", TokenType.Whitespace);
t.FromTo("is", "<{@Self}>");
Console.WriteLine(t.Tokens.Description);
Console.WriteLine(t.Transform(txt));
				
			
This <is> an island test, I said.

Match by character
Th<is> <is> an <is>land test, I said.

By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // This examples shows the default match by
   // token mode, as well as how to reconfigure
   // it in order to do match by character
   // along with a whitespace variation

   auto t = uc.NewTransformer();
   auto txt = "This is an island test, I said.";
   t.FromTo("is", "<is>");

   cout << t.Transform(txt) << endl;
   cout << "" << endl;

   t.Tokens().Description("Match by character");
   t.Tokens().Add("."); // This overrides existing tokens
   t.FromTo("is", "<is>");
   cout << t.Tokens().Description() << endl;
   cout << t.Transform(txt) << endl;
   cout << "" << endl;

   // Note: whitespace sensitivity is off by default
   // Whitespace token is re-introduced
   // (after being overridden in the previous Token Add())
   t.Tokens().Description("By char + whitespace ignored");
   t.Tokens().Add("[\\t\\v ]+", TokenType::Whitespace);
   t.FromTo("is", "<{@Self}>");
   cout << t.Tokens().Description() << endl;
   cout << t.Transform(txt) << endl;
}
				
			
This <is> an island test, I said.

Match by character
Th<is> <is> an <is>land test, I said.

By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// This examples shows the default match by
      '// token mode, as well as how to reconfigure
      '// it in order to do match by character
      '// along with a whitespace variation
      
      Dim t = uc.NewTransformer()
      Dim txt = "This is an island test, I said."
      t.FromTo("is", "<is>")
      
      Console.WriteLine(t.Transform(txt))
      Console.WriteLine("")
      
      t.Tokens.Description = "Match by character"
      t.Tokens.Add(".") '// This overrides existing tokens
      t.FromTo("is", "<is>")
      Console.WriteLine(t.Tokens.Description)
      Console.WriteLine(t.Transform(txt))
      Console.WriteLine("")
      
      '// Note: whitespace sensitivity is off by default
      '// Whitespace token is re-introduced
      '// (after being overridden in the previous Token Add())
      t.Tokens.Description = "By char + whitespace ignored"
      t.Tokens.Add("[\\t\\v ]+", TokenType.Whitespace)
      t.FromTo("is", "<{@Self}>")
      Console.WriteLine(t.Tokens.Description)
      Console.WriteLine(t.Transform(txt))
   End Sub
End Module
				
			
This <is> an island test, I said.

Match by character
Th<is> <is> an <is>land test, I said.

By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
Matching a certain number of tokens
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
t.FromTo("is {TokenCount:4}", "is <{TokenCount}>");
Console.WriteLine("This example captures 4 tokens");
Console.WriteLine(t.Transform("This is just a small token match test"));

// Quoted text counts as one token
Console.WriteLine(t.Transform("This is a 'really really' small test with quoted text"));
				
			
This example captures 4 tokens
This is <just a small token> match test
This is <a 'really really' small test> with quoted text
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   t.FromTo("is {TokenCount:4}", "is <{TokenCount}>");
   cout << "This example captures 4 tokens" << endl;
   cout << t.Transform("This is just a small token match test") << endl;

   // Quoted text counts as one token
   cout << t.Transform("This is a 'really really' small test with quoted text") << endl;
}
				
			
This example captures 4 tokens
This is <just a small token> match test
This is <a 'really really' small test> with quoted text
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      t.FromTo("is {TokenCount:4}", "is <{TokenCount}>")
      Console.WriteLine("This example captures 4 tokens")
      Console.WriteLine(t.Transform("This is just a small token match test"))
      
      '// Quoted text counts as one token
      Console.WriteLine(t.Transform("This is a 'really really' small test with quoted text"))
   End Sub
End Module
				
			
This example captures 4 tokens
This is <just a small token> match test
This is <a 'really really' small test> with quoted text
Displays the complete list of default token definitions, showing their type, internal name, and regex pattern.
				
					using uCalcSoftware;

var uc = new uCalc();
// Lists all tokens currently defined for the expression evaluator.

Console.WriteLine($"Token Count: {uc.ExpressionTokens.Count}");
Console.WriteLine("");
Console.WriteLine("Index  Type  Name: regex");
Console.WriteLine("========================");
var Tokens = uc.ExpressionTokens;
foreach(var token in Tokens) {
   Console.Write(Tokens.IndexOf(token));
   Console.WriteLine($"  {token.Description}  {token.Name}: {token.Regex}");
}

// Note that the expression evaluator token list has a few extra tokens,
// related to hex/bin/oct notation, string interpolation, and imaginary number
// notation, which are not found in the default Transformer token list.
				
			
Token Count: 30

Index  Type  Name: regex
========================
0  generic  _token_line: .*
1  generic  _token_catchall: .
2  generic  _token_catchall_utf8_other: [\xf0-\xf7][\x80-\xbf][\x80-\xbf][\x80-\xbf]|[\xe0-\xef][\x80-\xbf][\x80-\xbf]|[\xc0-\xdf][\x80-\xbf]
3  generic  _token_punctuation: (--|\.{3}|\xE2\x80\xA6|[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]|\xE2\x80[\x90-\x95])
4  generic  _token_quotechar: ("){3}|"|'
5  generic  _token_quotechar_single: '
6  generic  _token_quotechar_double: "
7  generic  _token_quotechar_tripledouble: """
8  memberaccess  _token_memberaccess: \.
9  generic  _token_variableargs: \.\.\.
10  reducible  _token_reducible2: [-:|+/*^&=%@!`\\<>?#$~]+
11  bracket  _token_parenthesis: \(
12  bracketclose  _token_parenthesis_close: \)
13  bracket  _token_curlybrace: \{
14  bracketclose  _token_curlybrace_close: \}
15  bracket  _token_squarebracket: \[
16  bracketclose  _token_squarebracket_close: \]
17  argseparator  _token_argseparator: ,
18  statementseparator  _token_newline: (?:\r?\n)|\r
19  statementseparator  _token_semicolon: ;
20  literal  _token_string_singlequoted: '([^']*(?:''[^']*)*)'
21  literal  _token_string_doublequoted: "([^"]*(?:""[^"]*)*)"
22  literal  _token_string_tripledoublequoted: """([\s\S]*?)"""
23  whitespace  _token_whitespace: [\t\v ]+
24  reducible  _token_reducible: [-:|+/*^&=%@!`\\<>?]+
25  literal  _token_floatnumber: [0-9]*\.?[0-9]+([eE][+-]?[0-9]+)?
26  alphanumeric  _token_alphanumeric: [a-zA-Z_][a-zA-Z0-9_]*
27  literal  _token_imaginaryunit: #i
28  tokentransform  _token_binaryhexoctalnotation: #[bho][0-9A-F]+
29  tokentransform  _token_stringinterpolationquote: \$['"]
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // Lists all tokens currently defined for the expression evaluator.

   cout << "Token Count: " << uc.ExpressionTokens().Count() << endl;
   cout << "" << endl;
   cout << "Index  Type  Name: regex" << endl;
   cout << "========================" << endl;
   auto Tokens = uc.ExpressionTokens();
   for(auto token : Tokens) {
      cout << Tokens.IndexOf(token);
      cout << "  " << token.Description() << "  " << token.Name() << ": " << token.Regex() << endl;
   }

   // Note that the expression evaluator token list has a few extra tokens,
   // related to hex/bin/oct notation, string interpolation, and imaginary number
   // notation, which are not found in the default Transformer token list.
}
				
			
Token Count: 30

Index  Type  Name: regex
========================
0  generic  _token_line: .*
1  generic  _token_catchall: .
2  generic  _token_catchall_utf8_other: [\xf0-\xf7][\x80-\xbf][\x80-\xbf][\x80-\xbf]|[\xe0-\xef][\x80-\xbf][\x80-\xbf]|[\xc0-\xdf][\x80-\xbf]
3  generic  _token_punctuation: (--|\.{3}|\xE2\x80\xA6|[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]|\xE2\x80[\x90-\x95])
4  generic  _token_quotechar: ("){3}|"|'
5  generic  _token_quotechar_single: '
6  generic  _token_quotechar_double: "
7  generic  _token_quotechar_tripledouble: """
8  memberaccess  _token_memberaccess: \.
9  generic  _token_variableargs: \.\.\.
10  reducible  _token_reducible2: [-:|+/*^&=%@!`\\<>?#$~]+
11  bracket  _token_parenthesis: \(
12  bracketclose  _token_parenthesis_close: \)
13  bracket  _token_curlybrace: \{
14  bracketclose  _token_curlybrace_close: \}
15  bracket  _token_squarebracket: \[
16  bracketclose  _token_squarebracket_close: \]
17  argseparator  _token_argseparator: ,
18  statementseparator  _token_newline: (?:\r?\n)|\r
19  statementseparator  _token_semicolon: ;
20  literal  _token_string_singlequoted: '([^']*(?:''[^']*)*)'
21  literal  _token_string_doublequoted: "([^"]*(?:""[^"]*)*)"
22  literal  _token_string_tripledoublequoted: """([\s\S]*?)"""
23  whitespace  _token_whitespace: [\t\v ]+
24  reducible  _token_reducible: [-:|+/*^&=%@!`\\<>?]+
25  literal  _token_floatnumber: [0-9]*\.?[0-9]+([eE][+-]?[0-9]+)?
26  alphanumeric  _token_alphanumeric: [a-zA-Z_][a-zA-Z0-9_]*
27  literal  _token_imaginaryunit: #i
28  tokentransform  _token_binaryhexoctalnotation: #[bho][0-9A-F]+
29  tokentransform  _token_stringinterpolationquote: \$['"]
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// Lists all tokens currently defined for the expression evaluator.
      
      Console.WriteLine($"Token Count: {uc.ExpressionTokens.Count}")
      Console.WriteLine("")
      Console.WriteLine("Index  Type  Name: regex")
      Console.WriteLine("========================")
      Dim Tokens = uc.ExpressionTokens
      For Each token In Tokens
         Console.Write(Tokens.IndexOf(token))
         Console.WriteLine($"  {token.Description}  {token.Name}: {token.Regex}")
      Next
      
      '// Note that the expression evaluator token list has a few extra tokens,
      '// related to hex/bin/oct notation, string interpolation, and imaginary number
      '// notation, which are not found in the default Transformer token list.
   End Sub
End Module
				
			
Token Count: 30

Index  Type  Name: regex
========================
0  generic  _token_line: .*
1  generic  _token_catchall: .
2  generic  _token_catchall_utf8_other: [\xf0-\xf7][\x80-\xbf][\x80-\xbf][\x80-\xbf]|[\xe0-\xef][\x80-\xbf][\x80-\xbf]|[\xc0-\xdf][\x80-\xbf]
3  generic  _token_punctuation: (--|\.{3}|\xE2\x80\xA6|[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]|\xE2\x80[\x90-\x95])
4  generic  _token_quotechar: ("){3}|"|'
5  generic  _token_quotechar_single: '
6  generic  _token_quotechar_double: "
7  generic  _token_quotechar_tripledouble: """
8  memberaccess  _token_memberaccess: \.
9  generic  _token_variableargs: \.\.\.
10  reducible  _token_reducible2: [-:|+/*^&=%@!`\\<>?#$~]+
11  bracket  _token_parenthesis: \(
12  bracketclose  _token_parenthesis_close: \)
13  bracket  _token_curlybrace: \{
14  bracketclose  _token_curlybrace_close: \}
15  bracket  _token_squarebracket: \[
16  bracketclose  _token_squarebracket_close: \]
17  argseparator  _token_argseparator: ,
18  statementseparator  _token_newline: (?:\r?\n)|\r
19  statementseparator  _token_semicolon: ;
20  literal  _token_string_singlequoted: '([^']*(?:''[^']*)*)'
21  literal  _token_string_doublequoted: "([^"]*(?:""[^"]*)*)"
22  literal  _token_string_tripledoublequoted: """([\s\S]*?)"""
23  whitespace  _token_whitespace: [\t\v ]+
24  reducible  _token_reducible: [-:|+/*^&=%@!`\\<>?]+
25  literal  _token_floatnumber: [0-9]*\.?[0-9]+([eE][+-]?[0-9]+)?
26  alphanumeric  _token_alphanumeric: [a-zA-Z_][a-zA-Z0-9_]*
27  literal  _token_imaginaryunit: #i
28  tokentransform  _token_binaryhexoctalnotation: #[bho][0-9A-F]+
29  tokentransform  _token_stringinterpolationquote: \$['"]
Matching tokens by category.
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
// Match a word (Alpha) followed by equals and a Number (Literal)
t.FromTo("{@Alpha} = {@Number}", "Assignment Detected");

Console.WriteLine(t.Transform("x = 10")); 
				
			
Assignment Detected
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   // Match a word (Alpha) followed by equals and a Number (Literal)
   t.FromTo("{@Alpha} = {@Number}", "Assignment Detected");

   cout << t.Transform("x = 10") << endl;
}
				
			
Assignment Detected
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      '// Match a word (Alpha) followed by equals and a Number (Literal)
      t.FromTo("{@Alpha} = {@Number}", "Assignment Detected")
      
      Console.WriteLine(t.Transform("x = 10"))
   End Sub
End Module
				
			
Assignment Detected
Customizing token definitions (e.g., treating hyphens as part of a word).
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
// Now capture it using the {@Alpha} category
t.FromTo("{@Alpha:w}", "<{w}>");

Console.WriteLine("Before:");
Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."));

// Define a new token pattern for hyphenated words
// We assign it to the 'AlphaNumeric' category so it behaves like a word
t.Tokens.Add("[a-zA-Z-]+", TokenType.AlphaNumeric);
Console.WriteLine("After:");
Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."));
				
			
Before:
1. <Start>-<Up> 'big ideas' <well>-<knwon>.
After:
1. <Start-Up> 'big ideas' <well-knwon>.
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   // Now capture it using the {@Alpha} category
   t.FromTo("{@Alpha:w}", "<{w}>");

   cout << "Before:" << endl;
   cout << t.Transform("1. Start-Up 'big ideas' well-knwon.") << endl;

   // Define a new token pattern for hyphenated words
   // We assign it to the 'AlphaNumeric' category so it behaves like a word
   t.Tokens().Add("[a-zA-Z-]+", TokenType::AlphaNumeric);
   cout << "After:" << endl;
   cout << t.Transform("1. Start-Up 'big ideas' well-knwon.") << endl;
}
				
			
Before:
1. <Start>-<Up> 'big ideas' <well>-<knwon>.
After:
1. <Start-Up> 'big ideas' <well-knwon>.
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      '// Now capture it using the {@Alpha} category
      t.FromTo("{@Alpha:w}", "<{w}>")
      
      Console.WriteLine("Before:")
      Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."))
      
      '// Define a new token pattern for hyphenated words
      '// We assign it to the 'AlphaNumeric' category so it behaves like a word
      t.Tokens.Add("[a-zA-Z-]+", TokenType.AlphaNumeric)
      Console.WriteLine("After:")
      Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."))
   End Sub
End Module
				
			
Before:
1. <Start>-<Up> 'big ideas' <well>-<knwon>.
After:
1. <Start-Up> 'big ideas' <well-knwon>.
Handling Brackets and Literals.
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
// Capture a function call.
// {@Alpha} matches the name.
// '(' and ')' are Bracket tokens that ensure the content is captured correctly.
t.FromTo("{@Alpha:func} ( {args} )", "Call: {func} with {args}");

Console.WriteLine(t.Transform("myfunc('Hello World', (x + y) * 2 )"));
				
			
Call: myfunc with 'Hello World', (x + y) * 2
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   // Capture a function call.
   // {@Alpha} matches the name.
   // '(' and ')' are Bracket tokens that ensure the content is captured correctly.
   t.FromTo("{@Alpha:func} ( {args} )", "Call: {func} with {args}");

   cout << t.Transform("myfunc('Hello World', (x + y) * 2 )") << endl;
}
				
			
Call: myfunc with 'Hello World', (x + y) * 2
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      '// Capture a function call. 
      '// {@Alpha} matches the name. 
      '// '(' and ')' are Bracket tokens that ensure the content is captured correctly.
      t.FromTo("{@Alpha:func} ( {args} )", "Call: {func} with {args}")
      
      Console.WriteLine(t.Transform("myfunc('Hello World', (x + y) * 2 )"))
   End Sub
End Module
				
			
Call: myfunc with 'Hello World', (x + y) * 2
Previous
Next