uCalc API Version: 2.1.3-preview.2 Released: 6/16/2026

Warning

uCalc API Preview Release Notice:The documentation describes the intended behavior of the API. The current preview build contains incomplete features, unoptimized performance, and is subject to breaking changes.

TypeOfToken = [TokenType]

Property

Product: 

Fast Math Parser

Class: 

Item

Gets or sets the lexical category (TokenType) for a specific token Item.

Remarks

This method functions as a dynamic getter/setter for a token's lexical category, allowing you to change how the parser interprets a token at runtime.

⚙️ Getter/Setter Behavior

  • Getter: When called without a parameter (or with the default TokenType::None), TypeOfToken() returns the token's current TokenType without changing it.
  • Setter: When you pass a specific TokenType enum member, the method re-categorizes the token. For instance, you can change the newline token from a StatementSep to Whitespace, fundamentally altering the parsing behavior for multi-line text.

💡 Why Is This Useful? (Comparative Analysis)

In traditional compiler design using tools like Lex/Flex or ANTLR, token definitions are static and defined in a separate grammar file. Changing a token's type requires modifying the grammar and regenerating/recompiling the parser.

uCalc's TypeOfToken() method provides a powerful runtime mechanism to alter the lexer's behavior on the fly. This is a significant advantage for:

  • Syntax Dialects: Easily switch between language dialects that treat characters differently (e.g., where a newline is sometimes significant and sometimes not).
  • Context-Sensitive Lexing: Programmatically change token types based on the application's state. For example, you could treat * as multiplication in a math context and as a wildcard character in a search context.
  • Extensibility: Allows for creating highly adaptable parsers that can be reconfigured without a full restart or recompilation.

The most common practical use is to re-categorize the newline token (_token_newline) from a statement separator to whitespace, which is essential for parsing languages like XML or HTML where newlines are not structurally significant.

Examples

Succinct: Demonstrates getting a token's initial type and then changing it using the setter overload.
				
					using uCalcSoftware;

var uc = new uCalc();
using (var t = new uCalc.Transformer()) {
   var myToken = t.Tokens.Add("###", TokenType.Generic);
   Console.Write("Initial Type: ");
   Console.WriteLine(myToken.TypeOfToken == TokenType.Generic);

   // Change the type
   myToken.TypeOfToken = TokenType.Reducible;
   Console.Write("New Type is Reducible: ");
   Console.WriteLine(myToken.TypeOfToken == TokenType.Reducible);
}
				
			
Initial Type: True
New Type is Reducible: True
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

#define tf(IsTrue) ((IsTrue) ? "True" : "False")

int main() {
   uCalc uc;
   {
      uCalc::Transformer t;
      t.Owned(); // Causes t to be released when it goes out of scope
      auto myToken = t.Tokens().Add("###", TokenType::Generic);
      cout << "Initial Type: ";
      cout << tf(myToken.TypeOfToken() == TokenType::Generic) << endl;

      // Change the type
      myToken.TypeOfToken(TokenType::Reducible);
      cout << "New Type is Reducible: ";
      cout << tf(myToken.TypeOfToken() == TokenType::Reducible) << endl;
   }
}
				
			
Initial Type: True
New Type is Reducible: True
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Using t As New uCalc.Transformer()
         Dim myToken = t.Tokens.Add("###", TokenType.Generic)
         Console.Write("Initial Type: ")
         Console.WriteLine(myToken.TypeOfToken = TokenType.Generic)
         
         '// Change the type
         myToken.TypeOfToken = TokenType.Reducible
         Console.Write("New Type is Reducible: ")
         Console.WriteLine(myToken.TypeOfToken = TokenType.Reducible)
      End Using
   End Sub
End Module
				
			
Initial Type: True
New Type is Reducible: True
Practical: Re-categorizes the newline token as whitespace to allow a pattern to match across multiple lines, a common requirement for parsing HTML or XML.
				
					using uCalcSoftware;

var uc = new uCalc();
using (var t = new uCalc.Transformer()) {
   var source = """
<data>
  content spans
  multiple lines
</data>
""";
   t.FromTo("<data>{body}</data>", "Body: [{body}]");

   Console.WriteLine("--- Default (Newline is a Separator) ---");
   // This fails because {body} stops at the first newline
   Console.WriteLine(t.Transform(source).Text);
   Console.WriteLine("");

   Console.WriteLine("--- Modified (Newline is Whitespace) ---");
   // Find the newline token and change its type
   var newlineToken = t.Tokens["_token_newline"];
   newlineToken.TypeOfToken = TokenType.Whitespace;
   // Now the transform succeeds
   Console.WriteLine(t.Transform(source).Text);
}
				
			
--- Default (Newline is a Separator) ---
<data>
  content spans
  multiple lines
</data>

--- Modified (Newline is Whitespace) ---
Body: [content spans
  multiple lines]
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   {
      uCalc::Transformer t;
      t.Owned(); // Causes t to be released when it goes out of scope
      auto source = R"(<data>
  content spans
  multiple lines
</data>)";
      t.FromTo("<data>{body}</data>", "Body: [{body}]");

      cout << "--- Default (Newline is a Separator) ---" << endl;
      // This fails because {body} stops at the first newline
      cout << t.Transform(source).Text() << endl;
      cout << "" << endl;

      cout << "--- Modified (Newline is Whitespace) ---" << endl;
      // Find the newline token and change its type
      auto newlineToken = t.Tokens()["_token_newline"];
      newlineToken.TypeOfToken(TokenType::Whitespace);
      // Now the transform succeeds
      cout << t.Transform(source).Text() << endl;
   }
}
				
			
--- Default (Newline is a Separator) ---
<data>
  content spans
  multiple lines
</data>

--- Modified (Newline is Whitespace) ---
Body: [content spans
  multiple lines]
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Using t As New uCalc.Transformer()
         Dim source = "<data>
  content spans
  multiple lines
</data>"
         t.FromTo("<data>{body}</data>", "Body: [{body}]")
         
         Console.WriteLine("--- Default (Newline is a Separator) ---")
         '// This fails because {body} stops at the first newline
         Console.WriteLine(t.Transform(source).Text)
         Console.WriteLine("")
         
         Console.WriteLine("--- Modified (Newline is Whitespace) ---")
         '// Find the newline token and change its type
         Dim newlineToken = t.Tokens("_token_newline")
         newlineToken.TypeOfToken = TokenType.Whitespace
         '// Now the transform succeeds
         Console.WriteLine(t.Transform(source).Text)
      End Using
   End Sub
End Module
				
			
--- Default (Newline is a Separator) ---
<data>
  content spans
  multiple lines
</data>

--- Modified (Newline is Whitespace) ---
Body: [content spans
  multiple lines]
Internal Test: Changes a token's type and then reverts it to ensure the state is managed correctly.
				
					using uCalcSoftware;

var uc = new uCalc();
using (var t = new uCalc.Transformer()) {
   var alphaToken = t.Tokens["_token_alphanumeric"];

   Console.WriteLine($"1. Initial Type is Alphanumeric: {alphaToken.TypeOfToken == TokenType.AlphaNumeric}");

   // Change it to a literal
   alphaToken.TypeOfToken = TokenType.Literal;
   Console.WriteLine($"2. Type is now Literal: {alphaToken.TypeOfToken == TokenType.Literal}");
   Console.WriteLine($"3. Type is no longer Alphanumeric: {alphaToken.TypeOfToken != TokenType.AlphaNumeric}");

   // Change it back
   alphaToken.TypeOfToken = TokenType.AlphaNumeric;
   Console.WriteLine($"4. Reverted Type is Alphanumeric: {alphaToken.TypeOfToken == TokenType.AlphaNumeric}");
}
				
			
1. Initial Type is Alphanumeric: True
2. Type is now Literal: True
3. Type is no longer Alphanumeric: True
4. Reverted Type is Alphanumeric: True
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

#define tf(IsTrue) ((IsTrue) ? "True" : "False")

int main() {
   uCalc uc;
   {
      uCalc::Transformer t;
      t.Owned(); // Causes t to be released when it goes out of scope
      auto alphaToken = t.Tokens()["_token_alphanumeric"];

      cout << "1. Initial Type is Alphanumeric: " << tf(alphaToken.TypeOfToken() == TokenType::AlphaNumeric) << endl;

      // Change it to a literal
      alphaToken.TypeOfToken(TokenType::Literal);
      cout << "2. Type is now Literal: " << tf(alphaToken.TypeOfToken() == TokenType::Literal) << endl;
      cout << "3. Type is no longer Alphanumeric: " << tf(alphaToken.TypeOfToken() != TokenType::AlphaNumeric) << endl;

      // Change it back
      alphaToken.TypeOfToken(TokenType::AlphaNumeric);
      cout << "4. Reverted Type is Alphanumeric: " << tf(alphaToken.TypeOfToken() == TokenType::AlphaNumeric) << endl;
   }
}
				
			
1. Initial Type is Alphanumeric: True
2. Type is now Literal: True
3. Type is no longer Alphanumeric: True
4. Reverted Type is Alphanumeric: True
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Using t As New uCalc.Transformer()
         Dim alphaToken = t.Tokens("_token_alphanumeric")
         
         Console.WriteLine($"1. Initial Type is Alphanumeric: {alphaToken.TypeOfToken = TokenType.AlphaNumeric}")
         
         '// Change it to a literal
         alphaToken.TypeOfToken = TokenType.Literal
         Console.WriteLine($"2. Type is now Literal: {alphaToken.TypeOfToken = TokenType.Literal}")
         Console.WriteLine($"3. Type is no longer Alphanumeric: {alphaToken.TypeOfToken <> TokenType.AlphaNumeric}")
         
         '// Change it back
         alphaToken.TypeOfToken = TokenType.AlphaNumeric
         Console.WriteLine($"4. Reverted Type is Alphanumeric: {alphaToken.TypeOfToken = TokenType.AlphaNumeric}")
      End Using
   End Sub
End Module
				
			
1. Initial Type is Alphanumeric: True
2. Type is now Literal: True
3. Type is no longer Alphanumeric: True
4. Reverted Type is Alphanumeric: True
TypeOfToken
				
					using uCalcSoftware;

var uc = new uCalc();
// See uCalc.Tokens.Count() example for list of token names

var t = uc.NewTransformer();
var MyToken = t.Tokens.Add("###");
var Alpha = t.Tokens["_token_alphanumeric"];

Console.WriteLine(MyToken.TypeOfToken == TokenType.AlphaNumeric);
Console.WriteLine(MyToken.TypeOfToken == TokenType.Generic);
Console.WriteLine(Alpha.TypeOfToken == TokenType.AlphaNumeric);
Console.WriteLine(Alpha.TypeOfToken == TokenType.Generic);


				
			
False
True
True
False
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

#define tf(IsTrue) ((IsTrue) ? "True" : "False")

int main() {
   uCalc uc;
   // See uCalc.Tokens.Count() example for list of token names

   auto t = uc.NewTransformer();
   auto MyToken = t.Tokens().Add("###");
   auto Alpha = t.Tokens()["_token_alphanumeric"];

   cout << tf(MyToken.TypeOfToken() == TokenType::AlphaNumeric) << endl;
   cout << tf(MyToken.TypeOfToken() == TokenType::Generic) << endl;
   cout << tf(Alpha.TypeOfToken() == TokenType::AlphaNumeric) << endl;
   cout << tf(Alpha.TypeOfToken() == TokenType::Generic) << endl;


}
				
			
False
True
True
False
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// See uCalc.Tokens.Count() example for list of token names
      
      Dim t = uc.NewTransformer()
      Dim MyToken = t.Tokens.Add("###")
      Dim Alpha = t.Tokens("_token_alphanumeric")
      
      Console.WriteLine(MyToken.TypeOfToken = TokenType.AlphaNumeric)
      Console.WriteLine(MyToken.TypeOfToken = TokenType.Generic)
      Console.WriteLine(Alpha.TypeOfToken = TokenType.AlphaNumeric)
      Console.WriteLine(Alpha.TypeOfToken = TokenType.Generic)
      
      
   End Sub
End Module
				
			
False
True
True
False
Change newline from statement separator to whitespace using TypeOfToken
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();

t.Str("""

<div>a b c</div>
<div>
x
y
z
</div>
<div>1 2 3</div>

""");

t.Pattern("<div>{body}</div>");
var NewLineToken = t.Tokens["_token_newline"];

Console.WriteLine("Newline as statement separator (default)");
Console.WriteLine("----------------------------------------");
Console.WriteLine(t.Find().Matches.Text);
Console.WriteLine("");

Console.WriteLine("Newline as whitespace");
Console.WriteLine("---------------------");
NewLineToken.TypeOfToken = TokenType.Whitespace;
Console.WriteLine(t.Find().Matches.Text);
				
			
Newline as statement separator (default)
----------------------------------------
<div>a b c</div>
<div>1 2 3</div>

Newline as whitespace
---------------------
<div>a b c</div>
<div>
x
y
z
</div>
<div>1 2 3</div>
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();

   t.Str(R"(
<div>a b c</div>
<div>
x
y
z
</div>
<div>1 2 3</div>
)");

   t.Pattern("<div>{body}</div>");
   auto NewLineToken = t.Tokens()["_token_newline"];

   cout << "Newline as statement separator (default)" << endl;
   cout << "----------------------------------------" << endl;
   cout << t.Find().Matches().Text() << endl;
   cout << "" << endl;

   cout << "Newline as whitespace" << endl;
   cout << "---------------------" << endl;
   NewLineToken.TypeOfToken(TokenType::Whitespace);
   cout << t.Find().Matches().Text() << endl;
}
				
			
Newline as statement separator (default)
----------------------------------------
<div>a b c</div>
<div>1 2 3</div>

Newline as whitespace
---------------------
<div>a b c</div>
<div>
x
y
z
</div>
<div>1 2 3</div>
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      
      t.Str("
<div>a b c</div>
<div>
x
y
z
</div>
<div>1 2 3</div>
")
      
      t.Pattern("<div>{body}</div>")
      Dim NewLineToken = t.Tokens("_token_newline")
      
      Console.WriteLine("Newline as statement separator (default)")
      Console.WriteLine("----------------------------------------")
      Console.WriteLine(t.Find().Matches.Text)
      Console.WriteLine("")
      
      Console.WriteLine("Newline as whitespace")
      Console.WriteLine("---------------------")
      NewLineToken.TypeOfToken = TokenType.Whitespace
      Console.WriteLine(t.Find().Matches.Text)
   End Sub
End Module
				
			
Newline as statement separator (default)
----------------------------------------
<div>a b c</div>
<div>1 2 3</div>

Newline as whitespace
---------------------
<div>a b c</div>
<div>
x
y
z
</div>
<div>1 2 3</div>
Previous
Next