uCalc API Version: 2.1.3-preview.2 Released: 6/16/2026

Warning

uCalc API Preview Release Notice:The documentation describes the intended behavior of the API. The current preview build contains incomplete features, unoptimized performance, and is subject to breaking changes.

Add(Tokens)

Method

Product: 

Transformer Library

Class: 

Tokens

Imports a list of token definitions from another Tokens collection, enabling the reuse of lexical configurations.

Syntax

Add(Tokens)

Parameters

sourceTokens
Tokens
A pre-existing `Tokens` object whose definitions will be copied into the current collection.

Return

Tokens

The current Tokens object, allowing for a fluent, chainable syntax.

Remarks

🔄 Reusing Token Configurations

The Add(Tokens) method provides a powerful way to reuse token configurations by importing all token definitions from one Tokens collection into another. This is essential for creating modular, maintainable, and consistent parsing logic across different components of an application.

Core Functionality

  • Append, Don't Replace: This method appends the imported tokens to the end of the current list. It does not clear existing tokens. To start with a clean slate, you must call Clear() before importing.
  • Copy, Don't Reference: The import process creates a copy of each token definition. After the import, the two Tokens collections are independent; modifying a token in one will not affect the other.

💡 Precedence and Order

Token precedence is determined by a Last-In, First-Out (LIFO) order. Since this method appends tokens to the end of the list, imported tokens will have higher precedence than any tokens that already exist in the collection. The tokenizer will check for matches against the imported tokens first.

🎯 Primary Use Cases

  1. Sharing Base Definitions: Create a single Tokens object that defines the complete syntax for a language or data format. Other transformers can then import this base set, ensuring consistency.
  2. Extending Syntax: Import a base set of tokens and then add additional, more specific tokens to create a specialized dialect or extension of a language.
  3. Cross-Component Consistency: Ensure that an ExpressionTransformer, a TokenTransformer, and a general-purpose Transformer all share the same fundamental understanding of syntax by importing from a common source.

🆚 Comparative Analysis: Why uCalc?

In traditional compiler development with tools like ANTLR or Flex/Bison, lexical rules are defined in static grammar files. To share or reuse these rules, you typically rely on file-based imports managed by the build system. This process is entirely static and occurs at compile-time.

uCalc's approach is fundamentally different and more flexible:

  • Programmatic & Dynamic: Token sets are objects that can be created, modified, and shared entirely at runtime. You can build a library of Tokens configurations and programmatically compose them based on application logic, user settings, or other dynamic conditions.
  • No Build Step: There is no code generation or recompilation step. This allows for rapid prototyping and enables applications that can adapt or extend their own syntax on the fly.

This runtime flexibility makes Add(Tokens) a cornerstone feature for building modular and dynamically configurable parsing systems.

Examples

A succinct example showing how to define a custom token in one transformer and reuse it in another.
				
					using uCalcSoftware;

var uc = new uCalc();
// Create a source transformer and define a custom token.
var t_source = new uCalc.Transformer();
var customToken = t_source.Tokens.Add("###", TokenType.Generic);

// Create a destination transformer.
var t_dest = new uCalc.Transformer();

// Import all token definitions from the source.
t_dest.Tokens.Add(t_source.Tokens);
t_dest.FromTo("###", "MATCH");

Console.WriteLine($"t_dest can now find '###': {t_dest.Transform("Test ### Test")}");
				
			
t_dest can now find '###': Test MATCH Test
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // Create a source transformer and define a custom token.
   uCalc::Transformer t_source;
   auto customToken = t_source.Tokens().Add("###", TokenType::Generic);

   // Create a destination transformer.
   uCalc::Transformer t_dest;

   // Import all token definitions from the source.
   t_dest.Tokens().Add(t_source.Tokens());
   t_dest.FromTo("###", "MATCH");

   cout << "t_dest can now find '###': " << t_dest.Transform("Test ### Test") << endl;
}
				
			
t_dest can now find '###': Test MATCH Test
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// Create a source transformer and define a custom token.
      Dim t_source As New uCalc.Transformer()
      Dim customToken = t_source.Tokens.Add("###", TokenType.Generic)
      
      '// Create a destination transformer.
      Dim t_dest As New uCalc.Transformer()
      
      '// Import all token definitions from the source.
      t_dest.Tokens.Add(t_source.Tokens)
      t_dest.FromTo("###", "MATCH")
      
      Console.WriteLine($"t_dest can now find '###': {t_dest.Transform("Test ### Test")}")
   End Sub
End Module
				
			
t_dest can now find '###': Test MATCH Test
Importing a list of tokens from one transformer to another
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();

// This creates a simpler (not very useful) set of tokens for sake of example
var MyTokens = t.Tokens;
MyTokens.Clear(); // removes the default list of tokens
MyTokens.Add("."); // Should always have a fallback token like this one
MyTokens.Add(";", TokenType.StatementSep);
MyTokens.Add("<", TokenType.Generic, ">");
MyTokens.Add("[0-9]+", TokenType.Literal);
MyTokens.Add(" +", TokenType.Whitespace);
MyTokens.Add("[a-z]+", TokenType.AlphaNumeric);

t.FromTo("{ This | That} {etc}", "[{@Self}]"); // {etc} stops at the semicolon ";" TokenType.StatementSep

Console.WriteLine(t.Transform("This is a test; That < 123.456; abc >; ABC"));
Console.WriteLine("");

var OtherTransform = new uCalc.Transformer(uc); // This is an alternative way of constructing a new transformer
OtherTransform.Tokens.Add(MyTokens); // Imports the entire token list from the other transformer
OtherTransform.Pattern("{word:1}");

Console.WriteLine(OtherTransform.Filter("This is a test; That < 123.456; abc >; ABC").Matches.Text);

				
			
[This is a test]; [That < 123.456; abc >]; ABC

This
is
a
test
;
That
<
123
.
456
;
abc
>
;
ABC
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();

   // This creates a simpler (not very useful) set of tokens for sake of example
   auto MyTokens = t.Tokens();
   MyTokens.Clear(); // removes the default list of tokens
   MyTokens.Add("."); // Should always have a fallback token like this one
   MyTokens.Add(";", TokenType::StatementSep);
   MyTokens.Add("<", TokenType::Generic, ">");
   MyTokens.Add("[0-9]+", TokenType::Literal);
   MyTokens.Add(" +", TokenType::Whitespace);
   MyTokens.Add("[a-z]+", TokenType::AlphaNumeric);

   t.FromTo("{ This | That} {etc}", "[{@Self}]"); // {etc} stops at the semicolon ";" TokenType.StatementSep

   cout << t.Transform("This is a test; That < 123.456; abc >; ABC") << endl;
   cout << "" << endl;

   uCalc::Transformer OtherTransform(uc); // This is an alternative way of constructing a new transformer
   OtherTransform.Tokens().Add(MyTokens); // Imports the entire token list from the other transformer
   OtherTransform.Pattern("{word:1}");

   cout << OtherTransform.Filter("This is a test; That < 123.456; abc >; ABC").Matches().Text() << endl;

}
				
			
[This is a test]; [That < 123.456; abc >]; ABC

This
is
a
test
;
That
<
123
.
456
;
abc
>
;
ABC
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      
      '// This creates a simpler (not very useful) set of tokens for sake of example
      Dim MyTokens = t.Tokens
      MyTokens.Clear() '// removes the default list of tokens
      MyTokens.Add(".") '// Should always have a fallback token like this one
      MyTokens.Add(";", TokenType.StatementSep)
      MyTokens.Add("<", TokenType.Generic, ">")
      MyTokens.Add("[0-9]+", TokenType.Literal)
      MyTokens.Add(" +", TokenType.Whitespace)
      MyTokens.Add("[a-z]+", TokenType.AlphaNumeric)
      
      t.FromTo("{ This | That} {etc}", "[{@Self}]") '// {etc} stops at the semicolon ";" TokenType.StatementSep
      
      Console.WriteLine(t.Transform("This is a test; That < 123.456; abc >; ABC"))
      Console.WriteLine("")
      
      Dim OtherTransform As New uCalc.Transformer(uc) '// This is an alternative way of constructing a new transformer
      OtherTransform.Tokens.Add(MyTokens) '// Imports the entire token list from the other transformer
      OtherTransform.Pattern("{word:1}")
      
      Console.WriteLine(OtherTransform.Filter("This is a test; That < 123.456; abc >; ABC").Matches.Text)
      
   End Sub
End Module
				
			
[This is a test]; [That < 123.456; abc >]; ABC

This
is
a
test
;
That
<
123
.
456
;
abc
>
;
ABC
Previous
Next