Tokenizing a String but ignoring delimiters within quotes

user_406091 0
Viewed 31072

I wish to have have the following String

!cmd 45 90 "An argument" Another AndAnother "Another one in quotes"

to become an array of the following

{ "!cmd", "45", "90", "An argument", "Another", "AndAnother", "Another one in quotes" }

I tried

new StringTokenizer(cmd, "\"")

but this would return "Another" and "AndAnother as "Another AndAnother" which is not the desired effect.

Thanks.

EDIT: I have changed the example yet again, this time I believe it explains the situation best although it is no different than the second example.

java
13 Answers

Recently faced a similar question where command line arguments must be split ignoring quotes link.

One possible case:

"/opt/jboss-eap/bin/jboss-cli.sh --connect --controller=localhost:9990 -c command=\"deploy /app/jboss-eap-7.1/standalone/updates/sample.war --force\""

This had to be split to 

/opt/jboss-eap/bin/jboss-cli.sh
--connect
--controller=localhost:9990
-c
command="deploy /app/jboss-eap-7.1/standalone/updates/sample.war --force"

Just to add to @polygenelubricants's answer, having any non-space character before and after the quote matcher can work out.

"\\S*\"([^\"]*)\"\\S*|(\\S+)"

Example:

import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class Tokenizer {

    public static void main(String[] args){

        String a = "/opt/jboss-eap/bin/jboss-cli.sh --connect --controller=localhost:9990 -c command=\"deploy " +
                "/app/jboss-eap-7.1/standalone/updates/sample.war --force\"";
        String b = "Hello \"Stack Overflow\"";
        String c = "cmd=\"abcd efgh ijkl mnop\" \"apple\" banana mango";
        String d = "abcd ef=\"ghij klmn\"op qrst";
        String e = "1 2 \"333 4\" 55 6    \"77\" 8 999";

        List<String> matchList = new ArrayList<String>();
        Pattern regex = Pattern.compile("\\S*\"([^\"]*)\"\\S*|(\\S+)");
        Matcher regexMatcher = regex.matcher(a);
        while (regexMatcher.find()) {
            matchList.add(regexMatcher.group());
        }
        System.out.println("matchList="+matchList);
    }
}

Output:

matchList=[/opt/jboss-eap/bin/jboss-cli.sh, --connect, --controller=localhost:9990, -c, command="deploy /app/jboss-eap-7.1/standalone/updates/sample.war --force"]

user_74685380

This is what I myself use for splitting arguments in command line and things like that.

It's easily adjustible for multiple delimiters and quotes, it can process quotes within the words (like al' 'pha), it supports escaping (quotes as well as spaces) and it's really lenient.

public final class StringUtilities {
    private static final List<Character> WORD_DELIMITERS = Arrays.asList(' ', '\t');
    private static final List<Character> QUOTE_CHARACTERS = Arrays.asList('"', '\'');
    private static final char ESCAPE_CHARACTER = '\\';

    private StringUtilities() {

    }

    public static String[] splitWords(String string) {
        StringBuilder wordBuilder = new StringBuilder();
        List<String> words = new ArrayList<>();
        char quote = 0;

        for (int i = 0; i < string.length(); i++) {
            char c = string.charAt(i);

            if (c == ESCAPE_CHARACTER && i + 1 < string.length()) {
                wordBuilder.append(string.charAt(++i));
            } else if (WORD_DELIMITERS.contains(c) && quote == 0) {
                words.add(wordBuilder.toString());
                wordBuilder.setLength(0);
            } else if (quote == 0 && QUOTE_CHARACTERS.contains(c)) {
                quote = c;
            } else if (quote == c) {
                quote = 0;
            } else {
                wordBuilder.append(c);
            }
        }

        if (wordBuilder.length() > 0) {
            words.add(wordBuilder.toString());
        }

        return words.toArray(new String[0]);
    }
}
user_147282740

Another old school way is :

public static void main(String[] args) {

    String text = "One two \"three four\" five \"six seven eight\" nine \"ten\"";
    String[] splits = text.split(" ");
    List<String> list = new ArrayList<>();
    String token = null;
    for(String s : splits) {

        if(s.startsWith("\"") ) {
            token = "" + s; 
        } else if (s.endsWith("\"")) {
            token = token + " "+ s;
            list.add(token);
            token = null;
        } else {
            if (token != null) {
                token = token + " " + s;
            } else {
                list.add(s);
            }
        }
    }
    System.out.println(list);
}

Output : - [One, two, "three four", five, "six seven eight", nine]

user_22388590 
private static void findWords(String str) {
    boolean flag = false;
    StringBuilder sb = new StringBuilder();
    for(int i=0;i<str.length();i++) {
        if(str.charAt(i)!=' ' && str.charAt(i)!='"') {
            sb.append(str.charAt(i));
        }
        else {
            System.out.println(sb.toString());
            sb = new StringBuilder();
            if(str.charAt(i)==' ' && !flag)
                continue;
            else if(str.charAt(i)=='"') {
                if(!flag) {
                    flag=true;
                }
                i++;
                while(i<str.length() && str.charAt(i)!='"') {
                    sb.append(str.charAt(i));
                    i++;
                }
                flag=false;
                System.out.println(sb.toString());
                sb = new StringBuilder();
            }
        }
    }
}
user_133521920
Related