How to tokenizing a syntax from source code based on operators C#

user_10849082 0
Viewed 187

I'm reading all the lines from a TextBox and I am trying to remove all the whitespace that will be in the list.

I need to be able to tokenize the following expression:

if(x==0)
{
    cout<<x;
}

into

if
(
x
==
0
)
{
cout
<<
x
;
}

My code:

public static string[] Tokenize(string sourceCode)
{
    Regex RE = new Regex(@"([\s+\+\-\*\%\,\;\&\|\<\>\=\!\{\}])");
    string[] x = RE.Split(sourceCode);

    var list = new List<string>(x);
    list.Remove(" ");

    for (int m = 0; m < list.Count(); m++)
    {
        Console.WriteLine(list[m]);
    }

    return (RE.Split(sourceCode));
}

My output:

if(x
=

=
0)






{








 

 

 
cout
<

<
x
;







}

How can I split with symbols like == << && and how to remove spaces from the list? Is there a better way of achieving what I want?

lexer compiler-construction c#
2 Answers

I agree to @juharr's comment. But if you really want to use regex, it would be better to use the Match method instead of Split because it allows you to specify the tokens you are looking for instead of the token boundaries:

 Regex RE = new Regex(@"\w+|\(|\)|\++|-+|\*|%|,|;|&+|\|+|<+|>+|=+|!|\{|\}");
 foreach (Match m in RE.Matches(sourceCode))
 {
  Console.WriteLine(m.Value);
 }

Result:

if
(
x
==
0
)
{
cout
<<
x
;
}
user_21429500

You could do something like:

var rx = new Regex(@"([\p{L}_][\p{L}\p{N}_]*|[+-]?[0-9]+|==|!=|>=|<=|<<|>>|\|\||&&|[!=+\-*/%{}();]|\s+)*");
Match match = rx.Match(str);
Group g = match.Groups[1];
foreach (var capture in g.Captures)
{
    Console.WriteLine(capture);
}

(I've included many other operators compared to your example). It is still a bad idea.

Now... This is still a bad idea, but you could make it something more complex:

string str = @"if(x==0)
{
    cout<<x;
    var x1 = '\a';
    var x2 = '\'';
    var x3 = 'X';
    var x4 = ""He\""llo\n"";
}";

var fragments = new[]
{
    // The order of these pattern is important! Longer patterns should go first (so += before + for example)
    new { Name = "Keyword", Pattern = @"(?:if|for|while|var|int|long|string|char|return)\b", Escape = false },
    new { Name = "Symbol", Pattern = @"[\p{L}_][\p{L}\p{N}_]*\b", Escape = false },
    new { Name = "Number", Pattern = @"[+-]?[0-9]+(?:\.[0-9]+)?\b", Escape = false },
    new { Name = "OperatorAssign", Pattern = @"<<=|>>=|&&=|\|\|=|[+\-*/%&|^]=", Escape = false },
    new { Name = "Operator", Pattern = @"==|!=|>=|<=|>|<|<<|>>|&&|\|\||[+\-*/%&|^!]", Escape = false },
    new { Name = "Space", Pattern = @"\s+", Escape = false },
    new { Name = "Assign", Pattern = @"=", Escape = true },
    new { Name = "OpenBrace", Pattern = @"{", Escape = true },
    new { Name = "CloseBrace", Pattern = @"}", Escape = true },
    new { Name = "Semicolon", Pattern = @";", Escape = true },
    new { Name = "OpenRoundParenthesis", Pattern = @"(", Escape = true },
    new { Name = "CloseRoundParenthesis", Pattern = @")", Escape = true },
    new { Name = "OpenSquareParenthesis", Pattern = @"[", Escape = true },
    new { Name = "CloseSquareParenthesis", Pattern = @"]", Escape = true },
    new { Name = "Char", Pattern = @"'(?:\\.|.)'", Escape = false },
    new { Name = "String", Pattern = @"\""(?:\\.|[^""])*""", Escape = false },
};

string allPatterns = string.Join('|', fragments.Select(x => $"(?<{x.Name}>{(x.Escape ? Regex.Escape(x.Pattern) : x.Pattern)})"));
var rx = new Regex(@"\G(?:" + allPatterns + ")");

int ix = 0;

while (ix < str.Length)
{
    var match = rx.Match(str, ix);

    if (!match.Success)
    {
        Console.WriteLine($"Error starting at: {str.Substring(ix)}");
        break;
    }

    var group = match.Groups.OfType<Group>().Skip(1).Single(x => x.Success);

    string name = group.Name;
    string value = match.Value;

    if (name != "Space")
    {
        Console.WriteLine($"Match: {name}: {value}");
    }
    else
    {
        Console.WriteLine("Skipping some space");
    }

    ix += value.Length;
}
user_6131300
Related