Embedded non-relational (nosql) data store

Igor_Brejc_55408 0
Viewed 8741

I'm thinking about using/implementing some kind of an embedded key-value (or document) store for my Windows desktop application. I want to be able to store various types of data (GPS tracks would be one example) and of course be able to query this data. The amount of data would be such that it couldn't all be loaded into memory at the same time.

I'm thinking about using sqlite as a storage engine for a key-value store, something like y-serial, but written in .NET. I've also read about FriendFeed's usage of MySQL to store schema-less data, which is a good pointer on how to use RDBMS for non-relational data. sqlite seems to be a good option because of its simplicity, portability and library size.

My question is whether there are any other options for an embedded non-relational store? It doesn't need to be distributable and it doesn't have to support transactions, but it does have to be accessible from .NET and it should have a small download size.

UPDATE: I've found an article titled SQLite as a Key-Value Database which compares sqlite with Berkeley DB, which is an embedded key-value store library.

nosql sqlite embedded-database .net
8 Answers

Windows has a built-in embedded non-relational store. It is called ESENT and is used by several Windows applications, including the Active Directory and Windows Desktop Search.

http://blogs.msdn.com/windowssdk/archive/2008/10/23/esent-extensible-storage-engine-api-in-the-windows-sdk.aspx

If you want .NET access you can use the ManagedEsent layer on CodePlex.

http://managedesent.codeplex.com/

That project has a PersistentDictionary class that implements a key-value store that implements the IDictionary interface, but is backed by a database.

user_2605550

Personally I would go for SQLite with NHibernate (and Fluent NHibernate). NHibernate can generate the database schema automatically for your classes, so you just need to specify what classes you want to persist, and that's quite easy with Fluent NHibernate. Furthermore, you can search for specific objects and you don't need to load all data to memory.

user_1287090

Applying the KISS principle to your problem I would recommend you use files.

As in filename is the key. File contents is the value. Windows folder is the index.

Simple, quick, efficient, flexible, and foolproof (providing the fools have low intelligence).

user_382070

This is an old question, but I thought I'd add an answer in case anyone stumbles on it. My company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb

XML works well for this type of data store given that as long as the content you're trying to store is serializable to text you can store complex hierarchical trees. In fact, if you access the database directly, you don't ever even have to touch "XML". It can also be queried with XQuery, a powerful and complete query language.

user_8070640

Thanks for your kind mention of y_serial... more precisely, it is a Python module:

warehouse Python objects with SQLite

"Serialization + persistance :: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data."

http://yserial.sourceforge.net

In my experience, SQLite is a faster and more reliable choice than most databases (including PostgresQL and Berkeley DB) for the majority of projects -- and of course, it does not need a server daemon.

yserial is very easy to implement (and far faster than the "filename is the key / file contents is the value" approach ;-)

user_2855040
Related