SplitTokens issue in Processing

user_15418542 0
Viewed 68

Here is a short code below to count words and more, but I face a strange problem:

when I run it some words of the original string don’t appear on the screen!? How come?

Did I forgot something using splitTokens ?! Thanks a lot for your help in advance.

All the best,

L

IntDict counts;
String[] tokens;

void setup() {
    size(1000, 1000);
    background(0);
    counts = new IntDict();
    String[] lines = {
        "If only you were paying a bit attention to me sometimes.",
        "I am not just a care giver I also need soemone to hug me tenderly."
    };
    String alltext = join(lines, " ");
    tokens = splitTokens(alltext, "\n\";.?!'():\n ");

    for (int i = 0; i < tokens.length; i++) {
        String word = tokens[i].toLowerCase();

        if (counts.hasKey(word)) {
            counts.increment(word);
        } else {
            counts.set(word, 1);
        }
        println(word);
    }

    String[] keys = counts.keyArray();

    for (int i = 0; i < keys.length; i++) {
        textSize(20);
        float x = 50;
        float y = 50 + 15 * i;
        text(keys[i], x, y);
    }
}

void draw() {}
token processing arrays split string
1 Answers

What you're rendering on the screen is not the split tokens, it's only the unique words(counts' keys).

If you call printArray(tokens); you'll see in console that each word appears, including duplicates:

IntDict counts;
String [] tokens;

void setup() {
  size(1000, 1000);
  background(0);
  counts = new IntDict();
  String [] lines = {"If only you were paying a bit attention to me sometimes.",
    "I am not just a care giver I also need soemone to hug me tenderly."};
  String alltext = join(lines, " ");
  tokens = splitTokens(alltext, "\n\";.?!'():\n ");
  
  println("with duplicates"); 
  printArray(tokens);
  
  for (int i =0; i<tokens.length; i++) {
    String word = tokens[i].toLowerCase();

    if (counts.hasKey(word)) {
      counts.increment(word);
      println("\tfound duplicate", word);
    } else {
      counts.set(word, 1);
    }
    //println(word);
  }

  String []keys = counts.keyArray();
  
  println("without duplicates"); 
  printArray(keys);
  
  for (int i=0; i<keys.length; i++) {

    textSize(20);
    float x = 50;
    float y= 50+15*i;
    text(keys[i], x, y);
  }
}

void draw() {
}

If you want to render all the words simply iterate (and access) tokens instead of keys in the second for loop where you call text()

user_897660
Related