What's the most efficient way to erase duplicates and sort a vector?

I agree with R. Pate and Todd Gardner; a std::set might be a good idea here. Even if you're stuck using vectors, if you have enough duplicates, you might be better off creating a set to do the dirty work.

Let's compare three approaches:

Just using vector, sort + unique

sort( vec.begin(), vec.end() );
vec.erase( unique( vec.begin(), vec.end() ), vec.end() );

Convert to set (manually)

set<int> s;
unsigned size = vec.size();
for( unsigned i = 0; i < size; ++i ) s.insert( vec[i] );
vec.assign( s.begin(), s.end() );

Convert to set (using a constructor)

set<int> s( vec.begin(), vec.end() );
vec.assign( s.begin(), s.end() );

Here's how these perform as the number of duplicates changes:

Summary: when the number of duplicates is large enough, it's actually faster to convert to a set and then dump the data back into a vector.

And for some reason, doing the set conversion manually seems to be faster than using the set constructor -- at least on the toy random data that I used.

std::unique only removes duplicate elements if they're neighbours: you have to sort the vector first before it will work as you intend.

std::unique is defined to be stable, so the vector will still be sorted after running unique on it.

I'm not sure what you are using this for, so I can't say this with 100% certainty, but normally when I think "sorted, unique" container, I think of a std::set. It might be a better fit for your usecase:

std::set<Foo> foos(vec.begin(), vec.end()); // both sorted & unique already

Otherwise, sorting prior to calling unique (as the other answers pointed out) is the way to go.

std::unique only works on consecutive runs of duplicate elements, so you'd better sort first. However, it is stable, so your vector will remain sorted.

Here's a template to do it for you:

template<typename T>
void removeDuplicates(std::vector<T>& vec)
{
    std::sort(vec.begin(), vec.end());
    vec.erase(std::unique(vec.begin(), vec.end()), vec.end());
}

call it like:

removeDuplicates<int>(vectorname);

Assuming that a is a vector, remove the contiguous duplicates using

a.erase(unique(a.begin(),a.end()),a.end()); runs in O(n) time.

Efficiency is a complicated concept. There's time vs. space considerations, as well as general measurements (where you only get vague answers such as O(n)) vs. specific ones (e.g. bubble sort can be much faster than quicksort, depending on input characteristics).

If you have relatively few duplicates, then sort followed by unique and erase seems the way to go. If you had relatively many duplicates, creating a set from the vector and letting it do the heavy lifting could easily beat it.

Don't just concentrate on time efficiency either. Sort+unique+erase operates in O(1) space, while the set construction operates in O(n) space. And neither directly lends itself to a map-reduce parallelization (for really huge datasets).

You need to sort it before you call unique because unique only removes duplicates that are next to each other.

edit: 38 seconds...

unique only removes consecutive duplicate elements (which is necessary for it to run in linear time), so you should perform the sort first. It will remain sorted after the call to unique.

You can do this as follows:

std::sort(v.begin(), v.end());
v.erase(std::unique(v.begin(), v.end()), v.end());

With the Ranges v3 library, you can simply use

action::unique(vec);

Note that it actually removes the duplicate elements, not just move them.

Unfortunately, actions weren’t standardized in C++20 as other parts of the ranges library were you still have to use the original library even in C++20.

As already stated, unique requires a sorted container. Additionally, unique doesn't actually remove elements from the container. Instead, they are copied to the end, unique returns an iterator pointing to the first such duplicate element, and you are expected to call erase to actually remove the elements.

More understandable code from: https://en.cppreference.com/w/cpp/algorithm/unique

#include <iostream>
#include <algorithm>
#include <vector>
#include <string>
#include <cctype>

int main() 
{
    // remove duplicate elements
    std::vector<int> v{1,2,3,1,2,3,3,4,5,4,5,6,7};
    std::sort(v.begin(), v.end()); // 1 1 2 2 3 3 3 4 4 5 5 6 7 
    auto last = std::unique(v.begin(), v.end());
    // v now holds {1 2 3 4 5 6 7 x x x x x x}, where 'x' is indeterminate
    v.erase(last, v.end()); 
    for (int i : v)
      std::cout << i << " ";
    std::cout << "\n";
}

ouput:

1 2 3 4 5 6 7

void removeDuplicates(std::vector<int>& arr) {
    for (int i = 0; i < arr.size(); i++)
    {
        for (int j = i + 1; j < arr.size(); j++)
        {
            if (arr[i] > arr[j])
            {
                int temp = arr[i];
                arr[i] = arr[j];
                arr[j] = temp;
            }
        }
    }
    std::vector<int> y;
    int x = arr[0];
    int i = 0;
    while (i < arr.size())
    {
        if (x != arr[i])
        {
            y.push_back(x);
            x = arr[i];
        }
        i++;
        if (i == arr.size())
            y.push_back(arr[i - 1]);
    }
    arr = y;
}

std::set<int> s;
std::for_each(v.cbegin(), v.cend(), [&s](int val){s.insert(val);});
v.clear();
std::copy(s.cbegin(), s.cend(), v.cbegin());

If your class is easily convertible to an int, and you got some memory, unique can be done without sorting before, and it's much faster :

#include <vector>
#include <stdlib.h>
#include <algorithm>
int main (int argc, char* argv []) {
  //vector init
  std::vector<int> v (1000000, 0);
  std::for_each (v.begin (), v.end (), [] (int& s) {s = rand () %1000;});
  std::vector<int> v1 (v);
  int beg (0), end (0), duration (0);
  beg = clock ();
  {
    std::sort (v.begin (), v.end ());
    auto i (v.begin ());
    i = std::unique (v.begin (), v.end ());
    if (i != v.end ()) v.erase (i, v.end ());
  }
  end = clock ();
  duration = (int) (end - beg);
  std::cout << "\tduration sort + unique == " << duration << std::endl;

  int n (0);
  duration = 0;
  beg = clock ();
  std::for_each (v1.begin (), v1.end (), [&n] (const int& s) {if (s >= n) n = s+1;});
  std::vector<int> tab (n, 0);
  {
    auto i (v1.begin ());
    std::for_each (v1.begin (), v1.end (), [&i, &tab] (const int& s) {
      if (!tab [s]) {
        *i++ = s;
        ++tab [s];
      }
    });
    std::sort (v1.begin (), i);
    v1.erase (i, v1.end ());
  }
  end = clock ();
  duration = (int) (end - beg);
  std::cout << "\tduration unique + sort == " << duration << std::endl;
  if (v == v1) {
    std::cout << "and results are same" << std::endl;
  }
  else {
    std::cout << "but result differs" << std::endl;
  }  
}

Typical results : duration sort + unique == 38985 duration unique + sort == 2500 and results are same

Most of the answer seems to be using O(nlogn) but with the use of the unordered_set we can decrease it to O(n). I saw some of the solutions using sets, but I found this one and it seems more elegant to use set and iterators.

using Intvec = std::vector<int>;

void remove(Intvec &v) {
    // creating iterator starting with beginning of the vector 
    Intvec::iterator itr = v.begin();
    std::unordered_set<int> s;
    // loops from the beginning to the end of the list 
    for (auto curr = v.begin(); curr != v.end(); ++curr) {
        if (s.insert(*curr).second) { // if the 0 curr already exist in the set
            *itr++ = *curr; // adding a position to the iterator 
        }
    }
    // erasing repeating positions in the set 
    v.erase(itr, v.end());
}

Depends on use-case. If you expect less than 100 amount of positive integer unique values and if you have a cpu capable of avx512f instruction set, then you can insert elements at a rate of ~15 clock cycles per element or 300-500 million inserts per second, by using simple comparison against a small look-up-table.

Following implementation uses CPU registers to do value lookups for ~50 unique values and L1 cache for ~1000 unique values. For L1 cache version, it takes about 160 clock cycles per insertion which is equivalent to about ~25M insertions per second and becomes slower than using std::set. For only 4 unique values, it inserts at a rate of 5.8 cycles per element which is higher than 500M/s.

//g++  7.4.0
// time measurement taken from another answer
// valid C99 and C++

#include <stdint.h>  // <cstdint> is preferred in C++, but stdint.h works.

#ifdef _MSC_VER
# include <intrin.h>
#else
# include <x86intrin.h>
#endif

// optional wrapper if you don't want to just use __rdtsc() everywhere
inline
uint64_t readTSC() {
     _mm_lfence();  // optionally wait for earlier insns to retire before reading the clock
    uint64_t tsc = __rdtsc();
     _mm_lfence();  // optionally block later instructions until rdtsc retires
    return tsc;
}

// requires a Nehalem or newer CPU.  Not Core2 or earlier.  IDK when AMD added it.
inline
uint64_t readTSCp() {
    unsigned dummy;
    return __rdtscp(&dummy);  // waits for earlier insns to retire, but allows later to start
}



#include <iostream>

template<int n>
struct FastUnique
{
    public:
    FastUnique()
    {
         it=0;
         for(int i=0;i<n;i++)
             dict[i]=-1;
    }

    void insert(const int val)
    {
        if(!test(dict,val))
            dict[it++]=val;
    }

    const int get(const int index)
    {
        return dict[index];
    }

    const int size()
    {
        return it;
    }

    private:
    int dict[n];
    int it;
    bool test(const int * dict, const int val)
    {
        int c=0;
        for(int i=0;i<n;i++)
            c+=(dict[i]==val);
        return c>0;
    }
};

int main()
{
    std::cout << "Hello, world!\n";
    const int n=500000000;

    FastUnique<64> fastSet;

    auto t= readTSC();

    for(int i=0;i<n;i++)
        fastSet.insert(i&63);

    auto t2=readTSC();

    std::cout<<(t2-t)/(double)n<<"cycles per iteration"<<std::endl;
   
    for(int i=0;i<fastSet.size();i++)
        std::cout<<fastSet.get(i)<<std::endl;
    
    return 0;
}