C++ Writing const vectors with pointers to non const

I am writing a CustomVector class, internally storing data using a standard vector:

template <class T>
class CustomVector {
  friend class CustomVector_ref<T>;

  public:
  ...

  private:
    std::vector<T> _data;
};

Then, in order to extract subvectors from CustomVector, I use a class storing pointers to each element of data:

template <class T>
class CustomVector_ref {
  public:
    //Returns the value stored in CustomVector 
    //and pointed-to by _data_ref
    T& operator[] (size_t id) { return *_data_ref[id] } 
    const T& operator[] const (size_t id) { return *_data_ref[id] } 
  ...
  private:
    std::vector<T*> _data_ref;
};

Now, to illustrate my problem it is sufficient to consider the simple costructor building a reference to all the elements of CustomVector

template<class T>
CustomVector_ref<T>::CustomVector_ref(CustomVector<T>& cv)
{
  for (T& el : cv._data)
    _data_ref.push_back(&el);
}

That works fine, but if I have a const CustomVector, I need also to define the constructor:

template<class T>
CustomVector_ref<T>::CustomVector_ref(const CustomVector<T>& cv)
{
  for (const T& el : cv._data)
  _data_ref.push_back(const_cast<T*>(&el));
}

That works too, but if the CustomVector_ref object is not declared as const, then with the non-const operator [] it is possible to write data into the const CustomVector object.

const CustomVector<int> cv(...) //CostumVector is somehow constructed, 
                                //that does not matter now

std::cout<<cv[0]<<std::endl;  //output 1 for example

CustomVector_ref<int> cvr(cv)

cvr[0] = 2;

std::cout<<cv[0]<<std::endl;  //now cv[0] stores 2

It is possible to avoid this behavior?

I have noticed that this happens also with standard vectors, for example

const std::vector<int> v(1,1);
std::vector<int*> vp;

vp.push_back(const_cast<int*>(&v[0]));

*vp[0] = 2;

std::cout<<v[0]<<std::endl;  // now v[0] stores 2, not 1

So, since this is standard C++, I do not bother too much to fix my CustomVector, but it would be nice to know if there is a (not too convoluted) solution.