Question

What is the most efficient way to read large file using Node.JS(LTS)?

I'm looking for the most efficient way to read large files in Node.js (LTS version). I know there are options like fs.readFile() and streams, but I'm unsure which method would be the best for performance when handling very large files. Could someone explain the best practices for efficiently reading large files, especially in terms of memory usage and speed?

Answer 1

Using Streams

Steps to read a large file using streams:

Use the fs.createReadStream() method to create a readable stream from the file.

Pipe the stream to other streams (e.g., to a writable stream or process data chunk by chunk).

Example:

const fs = require('fs');

// Create a readable stream

const readStream = fs.createReadStream('largefile.txt', { encoding: 'utf8', highWaterMark: 16 * 1024 }); // 16 KB per chunk

// Handle 'data' event

readStream.on('data', (chunk) => {

  console.log('Received chunk:', chunk);

});

// Handle 'end' event

readStream.on('end', () => {

  console.log('File reading finished.');

});

// Handle error event

readStream.on('error', (err) => {

  console.error('Error reading file:', err);

});

HighWaterMark: This option allows you to control the size of each chunk.

Streams in Node.js are event-driven. You handle data, end, and error events to manage the reading process, making the operation non-blocking.