nw.js/electron best embedded database for large datasets?

user_3987354 0
Viewed 1458

Im trying to make simpe desktop app with electron or nw.js (not decided yet). But I see a big problem right now with databases.

I need to handle something about 1-2 millions of records, so i started to search and test. Sqlite looks ok, it can handle it, but its hard to compile for both - nw and electron. Always some errors and I gived up for now. So I tested NeDB. Fast, little, nice. After inserting 200k records on nw.js, database was never fully loaded, and count operation was not possible. So i never have any chance to test it with million records. Key / value databases to much work, need SQL or collections like MongoDb.

Do you have any ideas to make fast, embedded database for Nw.js/Electron desktop apps?

nw.js electron node.js javascript database
2 Answers

After some research I think that only way for this is sqlite3. It was impossible to compile for me, but sqlite3 for node have nice feedback. I opened issue and after few hours there was new version.

Few problems you may have when you compile sqlite3 for nw.js:

x64 or x86

Check architecture or every component you have. My first problem was that I was using x64 and I installed node-gyp with I option. So i used:

npm install --global --production windows-build-tools

Better dont do it. I had python before, but windows-build-tools installed other anyway, and my x64 python was switched to x86. Install everything you need your self, using same architecture or you will have problems.

Sqlite3 compiled, I added new module and cannot update

Its known issue. So if you want to add any module to your nw app, remove all node_modules and compile sqlite3 again.

user_39873540

Have you tried breaking up the storage in NeDB in a way that distributes that data across multiple file stores? I handle millions of records in NeDB by creating a "root" database that tracks where records are being stored, and then breaking my records up into smaller files, it's not pretty but it works.

You could try Loki.js - http://lokijs.org - I have not yet myself so I cannot say for sure that it's performance claims are honest.

user_28395380
Related