Using Regex to split a message with escaping characters (that can self-escape)

user_2099249 0
Viewed 139

My server is running Java 1.8, which is the language I also have programmed the server side as well. I was recently debugging an issue I had with my server failing to parse an incoming data packet from a legacy application, and I realized the issue is with the legacy implentation of a (very old) protocol.

That protocol has, among other things, three special characters. [] brackets for grouping a "tupple", | as a delimiter between columns, and the ~ as an escape character.

When we started our application, we assumed that the special characters will be respected, but did include the provision of escaping them if required.

Today I found that some of the older versions have no such respect. The escape character can escape itself, which creates a series of issues since a multitude of the old data contains datasets with ~ as their part (as a way to denote uncertainity on some entries).

I have no way to interact with most of the clients, so fixing it there is not an option (plus, data has been saved as is, and it would be an impossible task to go through all it in a reasonable amount of time)

I have seen the following cases of data coming (columns vary according to type, but are well defined for the specific type):

abc|def|hij <ul><li>
  not including brackets, but it is permitted for a single element

[abc|def|hij]
    single tupple

[abc|def|hij][abc|def|hij][abc|def|hij]
    multiple tupples

[abc|as~[da~]sd|hij][~[abcasdas~]|def|hij][abc|def|hij]
    multiple tupples, some with escaping characters. It should look like:
        abc,as[da]sd,hij , [abcasdas],def,hij,abc,def,hij
    when parsed

abc|~~~[sfe~]|asds]
    escaping the escape character. Data should look like:
         abc,~[sfe],asds 
    after parsing

I have managed to use Regex to split the data using |, and I am escaping it properly if there are even number of ~ behind it, up to 9 max, using the following Regex:

(?<![^~]\\~)(?<![^~]\\~{3})(?<![^~]\\~{5})(?<![^~]\\~{7})(?<![^~]\\~{9})\\|

However, try as I might, I cannot do the same for the [] tupples.

My best attempt is as follows:

(?<![^~]\\~)(?<![^~]\\~{3})(?<=\[).+?(?<![^~]\~)(?<![^~]\\~{3})(?=\])

which does not work, and given that I am not experienced with Regex, I am not that surprised.

I will provide some (fake, but proper) examples below:

[solid|surprise|32|menu|retain|45.4|rugby][solid|surprise|32|menu|retain|45.4|rugby]
    Clean example, should return the two tupples as they are.

[solid|surprise|~~32|menu|retain|45.4|rugby][solid|~[surprise match~]|32|menu|retain|45.4|~~~[rugby match~]]
    Should return:
        solid,suprise,~32,menu,retain,45.4,rugby
        AND
        solid,[surprise match],32,menu,retain,45.4,~[rubgy match]

There are can be an arbitary number of tupples permessage, so I cannot parse them with specific numbers in mind. Message length is variable too, although defined at the start of each individual message (not included in the examples, as it is not needed).

Can someone help me with a Regex that would return the contents of each tuple, sanitized where needed (~~ --> ~ and ~[ kept as part of the data, and not as the tupple start etc)?

regex java string
1 Answers

Completing this task is essentially writing a fairly complex parser, in regex, which is very error-prone. It also cannot be done with only one regex, since:

  1. Regexes can only output a flat list of matches and the output you require has one extra level of nesting.
  2. Regexes cannot replace and match in one call.

So it will need extra code (loops, etc.). @WiktorStribiżew shows how this can be done. I'll add to his answer by explaining how you can complete a task like this while staying sane.

First define your (regular) grammar:

char = '~[' | '~]' | '~~' | '~|' | ^('~' | '[' | ']' | '|')
value = char*
values = '' | value ('|' value)*
tpl = '[' values ']'
tpls = tpl*

Using the grammar to guide you, compile, by hand, two regexes:

  1. One regex to match tpls, capturing each values as a string
  2. One regex to match values, capturing each value as a string

Then use the first regex to find the content of each tuple, and for each tuple match the second regex to get each value. Finally, do the "sanitization" as @WiktorStribiżew shows.

Make sure to leave no mention in your code of the above grammar, as that would be "Not Using Regexes" and might be misconstrued as going against your manager's wishes.

user_13296150
Related