Remove duplicates when adding new data

If your Collection should not contain duplicates then you should specify a Set.

Set<String> wordSet = new HashSet<>(words);
wordSet.addAll(rawWords);

The typical option is a HashSet. This assumes that your data objects have implemented hashCode and their hashCode and equals methods are consistent. Since you are working with String you do not have to do anything since this class obeys those requirements.

If you require some sort of ordering to your Collection then consider TreeSet or LinkedHashSet depending on your use case. Search for information on the Java Collections framework for more details.

You have added your words twice in the words list and wordsToExport, which might slow down things. If you want to avoid duplicated words, you can simply use Set implementation. Example implementation which will print [word1, word3, word2] in the console:

public static void main(String[] args) {
    List<String> rawWords = Arrays.asList("word1", "word2", "word3", "word1", "word2");
    Set<String> words = new HashSet<>();
    Set<String> wordsToExport = new HashSet<>();
    for (String word : rawWords) {
        words.add(word);
        wordsToExport.add(word);
    }
    System.out.println(words);
}

As you have found out, your code becomes slow, when you have many words. Why is that? Because to find out if something is in a list, you need to iterate over all items in the list and compare one by one. There are different data structures with different complexity characteristics for different tasks:

• ArrayList#contains is O(n): inspect all N elements for a list with N elements. You are doing this for every item, so your algorithm's/method's overall complexity is O(n²).
• HashSet#contains is O(1): always takes the same time, regardless of number of elements in the set. For a low number of items, O(n) might be faster than O(1), but as your collections grow in size, O(n) will become slower and slower.

You have several options:

1. If you are only using words to keep track of "words already seen", then the solution is to simply switch it to a HashSet and still keep two collections (one to export, one to keep track of known words). This will use double the memory (two collections), but the order of words in wordsToExport is the same as in rawWords (minus the duplicates of course):

   final Set<String> words = new HashSet<>(rawWords.size()/0.75f+1);
   for(final String word : rawWords) {
     if(!words.contains(word)) {
       words.add(word);
       wordsToExport.add(word);
     }
   }
   

   Another minor optimization is also possible: Set#add will return true if the element was newly added, and false if it already existed in the set:

   final Set<String> words = new HashSet<>(rawWords.size()/0.75f+1);
   for(final String word : rawWords) {
     if(words.add(word)) {
       wordsToExport.add(word);
     }
   }
   

2. Change the wordsToExport collection to be a LinkedHashSet, which allows iterating its elements in insertion-order. This requires only a single collection and not two collection to keep synced:

   final Collection<String> wordsToExport = new LinkedHashSet<>(rawWords.size()/0.75f+1);
   for(final String word : rawWords) {
     wordsToExport.add(word);
   }
   

   The loop is now so trivial, that Java provides its own method for it:

   final Collection<String> wordsToExport = new LinkedHashSet<>(rawWords.size()/0.75f+1);
   wordsToExport.addAll(rawWords);
   

   And since this is now basically copying one collection into a collection of a different kind, this can be shortened even further, by utilizing Java's collection API to its fullest:

   final Collection<String> wordsToExport = new LinkedHashSet<>(rawWords);
   

   If you need a List to be returned from your method, consider copying the LinkedHashSet's elements to a new List instance in the end. This will be O(n) again, but only executed once and not for every single item:

   final Collection<String> words = new LinkedHashSet<>(rawWords.size()/0.75f+1);
   for(final String word : rawWords) {
     words.add(word);
   }
   final List<String> wordsToExport = new ArrayList<>(words);
   

   Or shortened:

   final List<String> wordsToExport = new ArrayList<>(new LinkedHashSet<>(rawWords));
   

3. Use the Stream API introduced with Java 8 to write an expressive pipeline. This comes with some additional overhead, because the pipeline needs to be built, executed, and collected. Behind the scenes, the logic is not that different from option #1:

   final List<String> wordsToExport = rawWords.stream()
           .distinct()
           .collect(Collectors.toList());
   

4. If the order of the elements in the result collection is not relevant, simply use a HashSet and be done with it:

   final Set<String> wordsToExport = new HashSet<>(rawWords.size()/0.75f+1);
   for(final String word : rawWords) {
     wordsToExport.add(word);
   }
   

   Short version:

   final Set<String> wordsToExport = new HashSet<>(rawWords);
   

   Or, as with option #2, if a List interface is required (but you don't care for the order of elements inside the list), copy to List at the end of the method:

   final Set<String> words = new HashSet<>(rawWords.size()/0.75f+1);
   for(final String word : rawWords) {
     words.add(word);
   }
   final List<String> wordsToExport = new ArrayList<>(words);
   

   Again, as a simple one-liner:

   final List<String> wordsToExport = new ArrayList<>(new HashSet<>(rawWords));
   

5. If you require an immutable set (with arbitrary order) and know that all your words are non-null, Set.copyOf comes in handy:

   final Set<String> wordsToExport = Set.copyOf(rawWords);