Whatsapp parse history to dialog json

user_10382384 0
Viewed 450

So i have this Whatsapp log file that looks like this: (Timestamps are not acurate)

[15.11.16, 16:13:29] Person A: Hello
[15.11.16, 16:13:45] Person B: This is
a multiline
message
[15.11.16, 16:14:12] Person A: Wow, so cool

I want to parse them into a json format that would look like that:

{
  "msg": [
    {
      "sender": "Person A",
      "message": "Hello",
      "time": 1479205511000,
      "response": {
        "sender": "Person B",
        "message": "This is\na multiline\nmessage",
        "time": 1479205536000
      }
    },
    ...
  ]
}

Whats hapening is, first of all the message will be split in three parts:

  • Time (As a timestamp)
  • Name of the Person
  • Message (If it is a multiline message, the newline characters should be escaped)

The other thing thats happening is the response thing, which chooses one Main Person (in this case "Person A"). Then it looks through each message starting at the Top, looking for the first message by that person (Called "Main-Message"). If it finds one, it will look for the response of the other person (The next message (Called "Response")). If the next message is from the main person again, it will choose that message as the "Main-Message" and look for the next one.

Example:

Person A: Hi
Person B: Hi
Person A: This is cool
Person A: I am cool
Person B: Ah ok

In this example, we will choose "Person A" as our Main-Person. The first message is written by our Main-Person, so it will be our Main-Message. The next message is from the other Person, so it will be our Response.

{
    "sender":"Person A",
    "message":"Hi",
    "time":1479205511000,
    "response":{
        "sender":"Person B",
        "message":"Hi",
        "time":1479205536000
    }
}

The next message is from our Main-Person again, so we will take it as our Main-Message. But the message after that is not by the other Person so we will skip the message before. The message after that is from our opponent again, so it will be our Response.

{
    "sender":"Person A",
    "message":"I am cool",
    "time":1479205511000,
    "response":{
        "sender":"Person B",
        "message":"Ah ok",
        "time":1479205536000
    }
}

We now have a JSON like that:

{
    "msg":[
        {
            "sender":"Person A",
            "message":"Hi",
            "time":1479205511000,
            "response":{
                "sender":"Person B",
                "message":"Hi",
                "time":1479205536000
            }
        },
        {
            "sender":"Person A",
            "message":"I am cool",
            "time":1479205511000,
            "response":{
                "sender":"Person B",
                "message":"Ah ok",
                "time":1479205536000
            }
        }
    ]
}

There are a couple Problems:

  • Multiline Messages
  • Non UTF-8 Characters like '' that should be escaped -> Text Encoding
  • Huge Logs (up to 100.000 lines) -> Performance
  • Sometimes the lines start with control characters so something like String.startsWith not always works perfectly

As you might've guessed by my tags, the language im using is Java.

If there are any questions or unclear information, feel free to draw my attention to it.

whatsapp json java
1 Answers

I just did it myself. First of all, I made a Class named "Message":

public class Message {

    public String sender;

    public String time;

    public String message;

    public String plainMessage;

    public Message(String line) {
        String[] array = line.split("]");

        time = clean(array[0].replace("[", ""));

        array = Main.removeElements(array, array[0]);

        line = Main.join("]", array);

        array = line.split(":");

        sender = clean(array[0]);

        array = Main.removeElements(array, array[0]);

        line = Main.join(":", array);

        plainMessage = line;

        line = StringEscapeUtils.escapeJava(line);
        line = escapeUmlaut(line);

        message = StringEscapeUtils.escapeJson(line);
    }

    private String escapeUmlaut(String input) {
        String output = input.replace("ü", "ue")
                .replace("ö", "oe")
                .replace("ä", "ae")
                .replace("ß", "ss");

        output = output.replace("Ü(?=[a-zäöüß ])", "Ue")
                .replace("Ö(?=[a-zäöüß ])", "Oe")
                .replace("Ä(?=[a-zäöüß ])", "Ae");

        output = output.replace("Ü", "UE")
                .replace("Ö", "OE")
                .replace("Ä", "AE");

        return output;
    }

    public static String clean(String what) {
        char[] chars = what.toCharArray();
        what = "";
        char[] allowed = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234567890[](){}=?!\\§$%&/'#*+;,:.-_<>|".toCharArray();
        for (char c : chars) {
            for (char cc : allowed) {
                if (cc == c) {
                    what += c;
                }
            }
        }

        return what;
    }

}

This Class has the task to parse the message.

This is how we combine multiline Messages:

ArrayList <String> parsed = new ArrayList <String> ();

for (int x = 0; x < size; x++) {
    String line = list.get(x);

    if (startsWith(line, '[')) {
        parsed.add(line);
    } else {
        int lastn = parsed.size() - 1;
        String last = parsed.get(lastn);
        last += "\\n" + line;
        parsed.set(lastn, last);
    }
}

This is how the Class is used:

Message MainMessage = null;

String json = "";

boolean firstWrite = false;

final Pattern pattern = Pattern.compile("[a-zA-Z0-9]+", Pattern.MULTILINE);

for (int x = 0; x < size; x++) {
    result = "";
    progressBar.setValue(x);

    String line = parsed.get(x);

    if (config.debug) {
        System.out.println(line);
    }

    Message message = new Message(line);
    final Matcher matcher = pattern.matcher(message.plainMessage);

    if (!matcher.find()) {
        continue;
    }

    if (x == 0 && mainSender == null) {
        mainSender = message.sender;
        MainMessage = message;
        continue;
    }

    if (mainSender.equals(message.sender)) {
        MainMessage = message;
    } else if (message.sender != mainSender && MainMessage != null) {
        json = "\n{\"sender\": \"" + MainMessage.sender + "\", \"message\": \"" + MainMessage.message + "\", \"response\": ";
        json += "{\"sender\": \"" + message.sender + "\", \"message\": \"" + message.message + "\"}}";
        if (firstWrite) {
            json = "," + json;
        }

        Files.write(Paths.get(
            export), json.getBytes(), StandardOpenOption.APPEND);
        json = "";
        MainMessage = null;

        firstWrite = true;
    }
}
user_103823840
Related