uCalc API Version: 2.1.3-preview.2 Released: 6/16/2026

Warning

uCalc API Preview Release Notice:The documentation describes the intended behavior of the API. The current preview build contains incomplete features, unoptimized performance, and is subject to breaking changes.

Remove

Method

Product: 

Transformer Library

Class: 

Tokens

Dynamically removes a token definition from the collection, altering the parser's lexical rules at runtime.

Syntax

Remove(Item)

Parameters

tokenItem
Item
The `Item` object representing the token definition to remove. This is typically obtained via `ByName()` or `ByType()`.

Return

void

Remarks

🗑️ Dynamically Removing Token Definitions

The Remove method allows you to dynamically alter the behavior of the tokenizer at runtime by deleting a specific token definition. Once a token is removed, the parser will no longer recognize that pattern, which can fundamentally change how an input string is processed.

This method is a key part of uCalc's dynamic lexical analysis engine, enabling you to restrict or customize a language's syntax on the fly.

⚙️ How It Works

To remove a token, you must first get a handle to its Item object. This is typically done by querying the Tokens collection using methods like ByName() or ByType().

// Get the token for single-quoted stringsvar singleQuoteToken = myTransformer.Tokens().ByName("_token_string_singlequoted");// Remove it from the collectionmyTransformer.Tokens().Remove(singleQuoteToken);

After removal, any text that previously matched the token's pattern will be tokenized by the next-highest-precedence token that matches it, often a more generic "catch-all" token.

🎯 Primary Use Cases

  • Restricting Syntax: Create a "safe" or simplified version of a language by removing tokens for features you want to disable. For example, you could remove the token for comments to disallow them.
  • Replacing a Built-in Token: The standard pattern for modifying a built-in token is to Remove the default version and then Add() a new one with a custom regex pattern. This allows you to extend the language, for instance, by allowing new characters in identifiers.
  • Resolving Ambiguity: If two token patterns are conflicting, removing one can resolve the ambiguity and ensure the desired token is matched.

💡 Why uCalc? (Comparative Analysis)

In traditional compiler development with tools like ANTLR or Flex/Bison, lexical rules are static and defined in a separate grammar file. Changing a token definition requires modifying that file, regenerating parser code, and recompiling the application. This is a compile-time, static process.

uCalc's token system is entirely dynamic and programmatic. Remove is a key part of this dynamism. It allows your application to:

  • Modify Syntax at Runtime: Your code can re-categorize or remove tokens on the fly, effectively switching between different language "dialects" without a restart.
  • Create Adaptable Parsers: Build tools that can be configured by the end-user to support different comment styles or literal formats by enabling or disabling specific tokens.

This runtime flexibility provides a significant advantage over static parsing systems. For clearing all tokens at once, see the Clear() method.

Examples

Practical: Demonstrates how removing the token for single-quoted strings causes the parser to treat the quote and its contents as individual generic tokens.
				
					using uCalcSoftware;

var uc = new uCalc();
// This example removes the single-quoted string token.
var t = new uCalc.Transformer();
string txt = "This is a test, 'This is a test'";
t.FromTo("{token:1}", "<{@Self}>");

Console.WriteLine("--- Before Removing Token ---");
// Initially, 'This is a test' is treated as a single token.
Console.WriteLine(t.Transform(txt).Text);

// Now, find and remove the token definition for single-quoted strings.
var singleQuoteToken = t.Tokens.ByName("_token_string_singlequoted");
t.Tokens.Remove(singleQuoteToken);

// Re-run the transform. The text must be set again to be re-tokenized.
t.Text = txt;
Console.WriteLine("");
Console.WriteLine("--- After Removing Token ---");
// Now, the single quote is a generic token, as are the words inside it.
Console.WriteLine(t.Transform().Text);
				
			
--- Before Removing Token ---
<This> <is> <a> <test><,> <'This is a test'>

--- After Removing Token ---
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // This example removes the single-quoted string token.
   uCalc::Transformer t;
   string txt = "This is a test, 'This is a test'";
   t.FromTo("{token:1}", "<{@Self}>");

   cout << "--- Before Removing Token ---" << endl;
   // Initially, 'This is a test' is treated as a single token.
   cout << t.Transform(txt).Text() << endl;

   // Now, find and remove the token definition for single-quoted strings.
   auto singleQuoteToken = t.Tokens().ByName("_token_string_singlequoted");
   t.Tokens().Remove(singleQuoteToken);

   // Re-run the transform. The text must be set again to be re-tokenized.
   t.Text(txt);
   cout << "" << endl;
   cout << "--- After Removing Token ---" << endl;
   // Now, the single quote is a generic token, as are the words inside it.
   cout << t.Transform().Text() << endl;
}
				
			
--- Before Removing Token ---
<This> <is> <a> <test><,> <'This is a test'>

--- After Removing Token ---
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// This example removes the single-quoted string token.
      Dim t As New uCalc.Transformer()
      Dim txt As String = "This is a test, 'This is a test'"
      t.FromTo("{token:1}", "<{@Self}>")
      
      Console.WriteLine("--- Before Removing Token ---")
      '// Initially, 'This is a test' is treated as a single token.
      Console.WriteLine(t.Transform(txt).Text)
      
      '// Now, find and remove the token definition for single-quoted strings.
      Dim singleQuoteToken = t.Tokens.ByName("_token_string_singlequoted")
      t.Tokens.Remove(singleQuoteToken)
      
      '// Re-run the transform. The text must be set again to be re-tokenized.
      t.Text = txt
      Console.WriteLine("")
      Console.WriteLine("--- After Removing Token ---")
      '// Now, the single quote is a generic token, as are the words inside it.
      Console.WriteLine(t.Transform().Text)
   End Sub
End Module
				
			
--- Before Removing Token ---
<This> <is> <a> <test><,> <'This is a test'>

--- After Removing Token ---
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Internal Test: Removes the core alphanumeric token to verify that the tokenizer falls back to character-by-character tokenization.
				
					using uCalcSoftware;

var uc = new uCalc();
var t = new uCalc.Transformer();
string text = "word game";
t.FromTo("{token:1}", "[{@Self}]");

Console.WriteLine("--- Before Remove ---");
// By default, 'word' is a single alphanumeric token.
Console.WriteLine(t.Transform(text));

// Remove the alphanumeric token definition.
var alphaToken = t.Tokens.ByName("_token_alphanumeric");
t.Tokens.Remove(alphaToken);

// The transformer must be reset with the original text for the change to apply.
t.Text = text;

Console.WriteLine("");
Console.WriteLine("--- After Remove ---");
// Now, 'word' is no longer a single token. The fallback '.' token
// matches each character individually.
Console.WriteLine(t.Transform());
				
			
--- Before Remove ---
[word] [game]

--- After Remove ---
[w][o][r][d] [g][a][m][e]
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   uCalc::Transformer t;
   string text = "word game";
   t.FromTo("{token:1}", "[{@Self}]");

   cout << "--- Before Remove ---" << endl;
   // By default, 'word' is a single alphanumeric token.
   cout << t.Transform(text) << endl;

   // Remove the alphanumeric token definition.
   auto alphaToken = t.Tokens().ByName("_token_alphanumeric");
   t.Tokens().Remove(alphaToken);

   // The transformer must be reset with the original text for the change to apply.
   t.Text(text);

   cout << "" << endl;
   cout << "--- After Remove ---" << endl;
   // Now, 'word' is no longer a single token. The fallback '.' token
   // matches each character individually.
   cout << t.Transform() << endl;
}
				
			
--- Before Remove ---
[word] [game]

--- After Remove ---
[w][o][r][d] [g][a][m][e]
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t As New uCalc.Transformer()
      Dim text As String = "word game"
      t.FromTo("{token:1}", "[{@Self}]")
      
      Console.WriteLine("--- Before Remove ---")
      '// By default, 'word' is a single alphanumeric token.
      Console.WriteLine(t.Transform(text))
      
      '// Remove the alphanumeric token definition.
      Dim alphaToken = t.Tokens.ByName("_token_alphanumeric")
      t.Tokens.Remove(alphaToken)
      
      '// The transformer must be reset with the original text for the change to apply.
      t.Text = text
      
      Console.WriteLine("")
      Console.WriteLine("--- After Remove ---")
      '// Now, 'word' is no longer a single token. The fallback '.' token
      '// matches each character individually.
      Console.WriteLine(t.Transform())
   End Sub
End Module
				
			
--- Before Remove ---
[word] [game]

--- After Remove ---
[w][o][r][d] [g][a][m][e]
Token Remove
				
					using uCalcSoftware;

var uc = new uCalc();
// This example removes the single quote pattern from the list

var t = uc.NewTransformer();
var txt = "This is a test, 'This is a test'";
t.FromTo("{token:1}", "<{@Self}>");
Console.WriteLine(t.Transform(txt).Text);

// Now the ' character will no longer be special
// (see the Name() example for list of token names
t.Tokens.Remove(t.Tokens["_token_string_singlequoted"]);
Console.WriteLine(t.Transform(txt).Text);
				
			
<This> <is> <a> <test><,> <'This is a test'>
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // This example removes the single quote pattern from the list

   auto t = uc.NewTransformer();
   auto txt = "This is a test, 'This is a test'";
   t.FromTo("{token:1}", "<{@Self}>");
   cout << t.Transform(txt).Text() << endl;

   // Now the ' character will no longer be special
   // (see the Name() example for list of token names
   t.Tokens().Remove(t.Tokens()["_token_string_singlequoted"]);
   cout << t.Transform(txt).Text() << endl;
}
				
			
<This> <is> <a> <test><,> <'This is a test'>
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// This example removes the single quote pattern from the list
      
      Dim t = uc.NewTransformer()
      Dim txt = "This is a test, 'This is a test'"
      t.FromTo("{token:1}", "<{@Self}>")
      Console.WriteLine(t.Transform(txt).Text)
      
      '// Now the ' character will no longer be special
      '// (see the Name() example for list of token names
      t.Tokens.Remove(t.Tokens("_token_string_singlequoted"))
      Console.WriteLine(t.Transform(txt).Text)
   End Sub
End Module
				
			
<This> <is> <a> <test><,> <'This is a test'>
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Previous
Next