Why can't open a file via emscriptens fopen() saved in IndexedDB?

user_9705404 0
Viewed 1111

I'm trying out the Emscriptens IndexedDB, but can't get it running. The file can't be loaded, "cannot open file". With EMSCRIPTEN_FETCH_LOAD_TO_MEMORY everthing works fine.

  1. Download file via emscripten's emscripten_fetch_t
  2. Save file directly in IndexedDB via EMSCRIPTEN_FETCH_PERSIST_FILE
  3. Load it later into memory
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <iostream>

#include <emscripten.h>
#include <emscripten/fetch.h>

using namespace std;

void downloadSucceeded(emscripten_fetch_t *fetch)
{
    printf("URL %s\n", fetch->url);
    printf("bytes %llu\n", fetch->numBytes);

    FILE *file = fopen("/data/test.txt", "r"); // also tried it with 'test.txt', '/test.txt', 'data/test.txt'
    if (!file) {
        printf("cannot open file\n");
        return;
    }

    fclose (file);

    emscripten_fetch_close(fetch);
}

void downloadFailed(emscripten_fetch_t *fetch) {
  printf("Downloading %s failed, HTTP failure status code: %d.\n", fetch->url, fetch->status);
  emscripten_fetch_close(fetch);
}

extern "C" {
void EMSCRIPTEN_KEEPALIVE download() {
    cout << "download" << endl;

    emscripten_fetch_attr_t attr;
    emscripten_fetch_attr_init(&attr);
    strcpy(attr.requestMethod, "GET");
    attr.attributes = EMSCRIPTEN_FETCH_PERSIST_FILE;
    attr.onsuccess = downloadSucceeded;
    attr.onerror = downloadFailed;
    emscripten_fetch(&attr, "http://localhost/test.txt");
}
}

int main() {
    cout << "main" << endl;
    EM_ASM(
        FS.mkdir('/data');
        FS.mount(IDBFS, {}, '/data');

        FS.syncfs(true, function (err) {
            console.log("syncfs");
            Module._download();
            assert(!err);
        });
    );

    emscripten_exit_with_live_runtime();
}

Output

main
syncfs
download
URL http://localhost/test.txt
bytes 185
cannot open file

Built

emcc code.cpp -o index.js --shell-file shell.html -lidbfs.js -s EXPORTED_FUNCTIONS="['_main','_download']" -s FETCH=1 -O3

If an application wants to download a file for local access, but does not immediately need to use the file, e.g. when preloading data up front for later access, it is a good idea to avoid the EMSCRIPTEN_FETCH_LOAD_TO_MEMORY flag altogether, and only pass the EMSCRIPTEN_FETCH_PERSIST_FILE flag instead. This causes the fetch to download the file directly to IndexedDB, which avoids temporarily populating the file in memory after the download finishes. In this scenario, the onsuccess() handler will only report the total downloaded file size, but will not contain the data bytes to the file.

But I can still read the data bytes, why?

Can some help me, what I'm doing wrong?

emscripten fetch c c++ javascript
2 Answers

EMSCRIPTEN_FETCH_PERSIST_FILE actually does two things:

  1. Check whether the file is already stored in the IndexDB. If yes, retrieve it from there and do not bother the server at all.
  2. If the file is not in the local IndexDB, download it from the server and cache there.

In particular, there is very little cache control: you can use EM_IDB_DELETE to delete the cached version, but that's it.

Answer

So, it looks like you're not supposed to use IDBFS to access fetched files. Use the Fetch library instead and it will use the cached version without any network roundtrips.

It may also be useful to add -s FETCH_DEBUG to the compilation line.

But why it did not work?

Prerequisite for understanding: read through IndexDB Basic Concepts to understand "database" and "object storage".

Looking into Emscripten's source code as well as the "Storage" tab in my brower's developer console:

  • IDBFS mounted at /data-foo-bar will provide access to a database called /data-foo-bar. Specifically, its FILE_DATA object storage (see here). Naming here is arbitrary and hard-coded. Filess store in the /data-foo-bar folder are stored in keys like /data-foo-bar/my-file.txt inside that object storage.
  • In particular, you cannot access arbitrary IndexDB databases or object storages.
  • On the other hand, the Fetch library stores files to the emscripten_filesystem database, inside the FILES object storage. Again, names look arbitrary and are hard-coded.

So, you don't get access to Fetch's cache through IDBFS simply because they access different storage objects in different databases with different naming conventions.

As an example, here is what FS.writeFile('/data/my-file.txt', 'hello') results in in my Firefox:

Firefox Developer Tools storage tab screenshot showing <code>/data</code> mount

And here is where the Fetch's cache lives:

Firefox Developer Tools storage tab screenshot showing Fetch's cache

Unfortunately, I don't know why does the content of http://localhost:8000/test.txt is shown as an empty object.

user_7676320

@yeputons: Thxs for your explanation! This helped me to understand my observations...

  1. I read the content of IDBFS, and there were /tmp, /home, /dev, /proc and /data folders, but no test.txt file
EM_ASM(
    var l = FS.readdir("/");
    for(var i in l) { 
        var item = l[i];
        console.log(item);
    }
    );
  1. If I clear the browser cache and then reload the website, the fetch->numBytes=0 and with the next reload the data of test.txt are available.

So the conclusion is, as you already mentioned, don't mix the 'Fetch Cache' and IDBFS. If using EMSCRIPTEN_FETCH_PERSIST_FILE you have to fetch the file two times, 1. to download and save it in the 'Fetch Cache' and sometime later 2. to read it into the memory and access it via fetch->data.

user_97054040
Related