Why file is not filled yet after writing ? [NodeJS] [Filesystem]

user_4367301 0
Viewed 37

I have problem at reading a file just after appending lines in it. It seems that when inserting less than 913742 characters, it's ok, but when inserting more than 914534 characters, the next lines are read as blank lines (the buffer still empty).

import { promises as fs } from "fs";

const filepath = 'any.file';

const readChunk = async (start: number, end: number) => {
  const buffer = Buffer.from("".padStart(end, " "));
  const fd = await fs.open(filepath, "r");
  await fd.read({ buffer, offset: 0, position: start, length: end });
  await fd.datasync();
  await fd.close();
  return buffer.toString();
};

const append = async (line: string) => {
  const fd = await fs.open(filepath, "a");
  await fd.appendFile(line);
  await fd.datasync();
  await fd.close();
};

// count lines of a document
const count = async () => ...

My jest test:

import faker from "faker";

const fakeUser = (): UserDatas => ({
  id: Math.round(Math.random() * MAX_SAFE_INTEGER),
  message: faker.lorem.paragraphs(),
});

const fakeUsers = (count: number) =>
  Array.from({ length: count }, fakeUser);

describe("large amount of data", () => {
  it("should accept thousand lines", async () => {
    await append(
      fakeUsers(faker.datatype.number({ min: 9000, max: 10000 }))
    );
    expect(await count()).toBeGreaterThan(8000);
  });
});

The output:

  ● BsonFile › large amount of data › should accept thousand lines

    expect(received).toBeGreaterThan(expected)

    Expected: > 8000
    Received:   1364

      178 |         fakeUsers(faker.datatype.number({ min: 9000, max: 10000 }))
      179 |       );
    > 180 |       expect(await count()).toBeGreaterThan(8000);
          |                                      ^
      181 |     });
      182 |   });
      183 | });

The reason why I can not count more 1364 is that the file does not output any data.

Also, I hoped that fd.datasync() would resolve my problem, but it don't: should I let it there? Is this usefull?

typescript node.js filesystems file javascript
1 Answers

In the code below, you are appending "chunk", but I cannot see that you are defining or setting "chunk" anywhere in the code, should this be "line" which is being passed into the function? Could this be the issue?

const append = async (line: string) => {
  const fd = await fs.open(filepath, "a");
  await fd.appendFile(chunk);
  await fd.datasync();
  await fd.close();
};

Should it be:

const append = async (line: string) => {
  const fd = await fs.open(filepath, "a");
  await fd.appendFile(line);
  await fd.datasync();
  await fd.close();
};

I don't understand your test code. What does getInstance() do?

filehandle.datasync() forces all currently queued I/O operations associated with the file to the operating system's synchronized I/O completion state.

You should not need to run this function to see the data, it is only there to ensure that buffers are emptied so that you do not lose data on a power or hardware failure.

See also https://nodejs.org/api/fs.html#filehandledatasync and https://man7.org/linux/man-pages/man2/fdatasync.2.html for more information.

user_112095730
Related