using a disk allocator for std::deque

Admin 1
Viewed 155

How can I store the contents of std::deque onto disk? I found that you can specify an allocator i.e.

std::deque<std::unique_ptr<my_struct>>, disk_allocator<std::unique_ptr<my_struct>>> q;

How can I write a custom allocator "disk_allocator" to store on disk? Does one already exist in the std library?

std memory c++
2 Answers

You could go with metal. It's a bit awkward to plainly suggest a library since this is a huge topic, and to be sure that a solution does what you want/need it's important to dive into the concepts of individual problems solved by allocators, memory mapping, persistence, serialization and friends (I have linked sources on those topics in the question comments).

That been said, metal would be used like so:

// Example for deque<int>
using namespace metall;
namespace mc = metall::container;
using deque_t = deque<int, metall::allocator<int>>;

void create_data()
{
  manager mg(create_only, "/tmp/dir");
  // storage location      ^^^^^^^^ 

  deque_t* p = mg.construct<deque_t>("deque")(mg.get_allocator());
  p->push_back(10); // Can use it normally
}

void reattach_data()
{
  manager mg(create_only, "/tmp/dir");
  // storage location      ^^^^^^^^ 

  deque_t* p = mg.find<deque_t>("deque").first;
  p->push_back(10); // Can resume work
}

You can check the related examples for Standard Library interoperability.

If you are willing to go with serialization, cereal has bindings for most standard library containers including deque.

user_25676830

You can't allocate from disk in order to be used with std::deque. At least it's not how we store STL Container data into the disk. The STL Allocators' job is to encapsulate strategies for access/addressing, allocation/deallocation, and construction/destruction of objects.

For storing simple container data in the disk we could do:
for copy/paste and test: serialize_deserialize_example 

template<typename Container>
void save_to_disk(Container const& container, std::filesystem::path const& path)
{
    std::ofstream output(path);
    if(!output.is_open())
    {
        throw std::runtime_error("Couldn't open file: " + path.string());
    }

    for(auto const& item : container)
    {
        output << item << std::endl;
    }
}

But obviously, we don't have always that must simple data structure to simply converted to string and back. For serialization of STL Containers I suggest using Boost.Serialization:
for copy/paste and test: boost_serialize_deserialize 

struct my_structure final
{
    friend class boost::serialization::access;

    std::deque<std::string> data;

    template<class Archive>
    void serialize(Archive& ar, unsigned int const)
    {
        ar& data;
    }
};
user_92484660
Related