Convert Figma logo to code with AI

isaacs logonode-graceful-fs

fs with incremental backoff on EMFILE

1,268
147
1,268
45
View on GitHubView on NPM

Top Related Projects

jprichardson's avatar

node-fs-extra

9,441

Node.js: extra methods for the fs object like copy(), remove(), mkdirs()

nodejs's avatar

readable-stream

1,031

Node-core streams for userland

paulmillr's avatar

chokidar

10,813

Minimal and efficient cross-platform file watching library

Quick Overview

node-graceful-fs is a drop-in replacement for the Node.js fs module that aims to improve error handling and resilience in file system operations. It adds automatic retries for certain errors and limits the number of open files to prevent EMFILE errors, making file system operations more robust in high-concurrency scenarios.

Pros

  • Improves error handling for file system operations
  • Automatically retries on certain errors (e.g., EMFILE, ENFILE, EPERM)
  • Limits the number of open files to prevent EMFILE errors
  • Compatible with the native fs module, allowing for easy integration

Cons

  • May introduce a slight performance overhead due to additional error handling
  • Could potentially mask underlying system issues by automatically retrying operations
  • Might behave differently from the native fs module in edge cases
  • Requires careful consideration when used in production environments

Code Examples

  1. Reading a file with graceful-fs:
const fs = require('graceful-fs');

fs.readFile('example.txt', 'utf8', (err, data) => {
  if (err) {
    console.error('Error reading file:', err);
    return;
  }
  console.log('File contents:', data);
});
  1. Writing to a file with graceful-fs:
const fs = require('graceful-fs');

const content = 'Hello, graceful-fs!';
fs.writeFile('output.txt', content, (err) => {
  if (err) {
    console.error('Error writing file:', err);
    return;
  }
  console.log('File written successfully');
});
  1. Using graceful-fs with promises:
const fs = require('graceful-fs').promises;

async function readAndWriteFile() {
  try {
    const data = await fs.readFile('input.txt', 'utf8');
    await fs.writeFile('output.txt', data.toUpperCase());
    console.log('File read and written successfully');
  } catch (err) {
    console.error('Error:', err);
  }
}

readAndWriteFile();

Getting Started

To use graceful-fs in your Node.js project, follow these steps:

  1. Install the package:

    npm install graceful-fs

  2. Replace the native fs module with graceful-fs in your code:

    const fs = require('graceful-fs');

  3. Use fs methods as you normally would. graceful-fs will automatically handle retries and error mitigation:

    fs.readFile('example.txt', 'utf8', (err, data) => {
  if (err) throw err;
  console.log(data);
});

That's it! You can now benefit from the improved error handling and resilience provided by graceful-fs.

Competitor Comparisons

jprichardson logo

node-fs-extra

9,441

Node.js: extra methods for the fs object like copy(), remove(), mkdirs()

Pros of fs-extra

  • Provides a wider range of file system operations, including copy, move, and remove directories
  • Offers promise-based versions of all methods, allowing for easier async/await usage
  • Includes additional utility functions like ensureDir and outputFile

Cons of fs-extra

  • Larger package size due to additional functionality
  • May have a slightly higher performance overhead for basic operations
  • Requires learning new API methods for extended functionality

Code Comparison

fs-extra:

const fs = require('fs-extra')

async function copyFile() {
  await fs.copy('/path/to/source', '/path/to/destination')
  console.log('File copied successfully')
}

graceful-fs:

const fs = require('graceful-fs')

function copyFile(callback) {
  fs.copyFile('/path/to/source', '/path/to/destination', (err) => {
    if (err) throw err
    console.log('File copied successfully')
    callback()
  })
}

fs-extra provides a more straightforward async approach with promises, while graceful-fs focuses on enhancing the core fs module's reliability. fs-extra offers additional methods like copy, whereas graceful-fs primarily improves existing fs functionality.

nodejs logo

readable-stream

1,031

Node-core streams for userland

Pros of readable-stream

  • Provides a consistent stream implementation across different Node.js versions
  • Offers enhanced performance and efficiency for stream operations
  • Includes additional features and improvements not found in core Node.js streams

Cons of readable-stream

  • Requires an additional dependency in your project
  • May have slight differences in behavior compared to native Node.js streams
  • Potential for version conflicts with other packages using different stream implementations

Code Comparison

graceful-fs:

var fs = require('graceful-fs')

fs.readFile('file.txt', 'utf8', function (err, data) {
  if (err) throw err
  console.log(data)
})

readable-stream:

var Readable = require('readable-stream').Readable

var stream = new Readable()
stream.push('Hello, world!')
stream.push(null)

stream.pipe(process.stdout)

Key Differences

  • graceful-fs focuses on improving file system operations, while readable-stream enhances stream functionality
  • graceful-fs is a drop-in replacement for the fs module, whereas readable-stream provides an alternative stream implementation
  • graceful-fs addresses specific issues like EMFILE errors, while readable-stream offers broader stream enhancements

Use Cases

  • Use graceful-fs when dealing with file system operations and potential concurrency issues
  • Choose readable-stream for consistent stream behavior across Node.js versions and additional stream features
paulmillr logo

chokidar

10,813

Minimal and efficient cross-platform file watching library

Pros of Chokidar

  • More comprehensive file watching capabilities, including recursive directory watching
  • Cross-platform consistency, especially for macOS and Linux
  • Better performance and lower CPU usage for large directory trees

Cons of Chokidar

  • Larger package size and more dependencies
  • May be overkill for simple file system operations
  • Slightly more complex API compared to graceful-fs

Code Comparison

Chokidar:

const chokidar = require('chokidar');

const watcher = chokidar.watch('path/to/directory', {
  persistent: true,
  ignoreInitial: true
});

watcher.on('change', (path) => console.log(`File ${path} has been changed`));

graceful-fs:

const fs = require('graceful-fs');

fs.watch('path/to/file', (eventType, filename) => {
  if (eventType === 'change') {
    console.log(`File ${filename} has been changed`);
  }
});

Summary

Chokidar is a more powerful and flexible file watching solution, ideal for complex applications requiring extensive file system monitoring. It offers better cross-platform consistency and performance for large directory structures. However, it comes with a larger footprint and may be unnecessary for simpler use cases.

graceful-fs, on the other hand, is a lightweight solution that enhances Node's built-in fs module with error handling and queue limiting. It's suitable for basic file system operations and provides a simpler API, but lacks the advanced watching capabilities of Chokidar.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

graceful-fs

graceful-fs functions as a drop-in replacement for the fs module, making various improvements.

The improvements are meant to normalize behavior across different platforms and environments, and to make filesystem access more resilient to errors.

Improvements over fs module

  • Queues up open and readdir calls, and retries them once something closes if there is an EMFILE error from too many file descriptors.
  • fixes lchmod for Node versions prior to 0.6.2.
  • implements fs.lutimes if possible. Otherwise it becomes a noop.
  • ignores EINVAL and EPERM errors in chown, fchown or lchown if the user isn't root.
  • makes lchmod and lchown become noops, if not available.
  • retries reading a file if read results in EAGAIN error.

On Windows, it retries renaming a file for up to one second if EACCESS or EPERM error occurs, likely because antivirus software has locked the directory.

USAGE

// use just like fs
var fs = require('graceful-fs')

// now go and do stuff with it...
fs.readFile('some-file-or-whatever', (err, data) => {
  // Do stuff here.
})

Sync methods

This module cannot intercept or handle EMFILE or ENFILE errors from sync methods. If you use sync methods which open file descriptors then you are responsible for dealing with any errors.

This is a known limitation, not a bug.

Global Patching

If you want to patch the global fs module (or any other fs-like module) you can do this:

// Make sure to read the caveat below.
var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)

This should only ever be done at the top-level application layer, in order to delay on EMFILE errors from any fs-using dependencies. You should not do this in a library, because it can cause unexpected delays in other parts of the program.

Changes

This module is fairly stable at this point, and used by a lot of things. That being said, because it implements a subtle behavior change in a core part of the node API, even modest changes can be extremely breaking, and the versioning is thus biased towards bumping the major when in doubt.

The main change between major versions has been switching between providing a fully-patched fs module vs monkey-patching the node core builtin, and the approach by which a non-monkey-patched fs was created.

The goal is to trade EMFILE errors for slower fs operations. So, if you try to open a zillion files, rather than crashing, open operations will be queued up and wait for something else to close.

There are advantages to each approach. Monkey-patching the fs means that no EMFILE errors can possibly occur anywhere in your application, because everything is using the same core fs module, which is patched. However, it can also obviously cause undesirable side-effects, especially if the module is loaded multiple times.

Implementing a separate-but-identical patched fs module is more surgical (and doesn't run the risk of patching multiple times), but also imposes the challenge of keeping in sync with the core module.

The current approach loads the fs module, and then creates a lookalike object that has all the same methods, except a few that are patched. It is safe to use in all versions of Node from 0.8 through 7.0.

v4

  • Do not monkey-patch the fs module. This module may now be used as a drop-in dep, and users can opt into monkey-patching the fs builtin if their app requires it.

v3

  • Monkey-patch fs, because the eval approach no longer works on recent node.
  • fixed possible type-error throw if rename fails on windows
  • verify that we never get EMFILE errors
  • Ignore ENOSYS from chmod/chown
  • clarify that graceful-fs must be used as a drop-in

v2.1.0

  • Use eval rather than monkey-patching fs.
  • readdir: Always sort the results
  • win32: requeue a file if error has an OK status

v2.0

  • A return to monkey patching
  • wrap process.cwd

v1.1

  • wrap readFile
  • Wrap fs.writeFile.
  • readdir protection
  • Don't clobber the fs builtin
  • Handle fs.read EAGAIN errors by trying again
  • Expose the curOpen counter
  • No-op lchown/lchmod if not implemented
  • fs.rename patch only for win32
  • Patch fs.rename to handle AV software on Windows
  • Close #4 Chown should not fail on einval or eperm if non-root
  • Fix isaacs/fstream#1 Only wrap fs one time
  • Fix #3 Start at 1024 max files, then back off on EMFILE
  • lutimes that doens't blow up on Linux
  • A full on-rewrite using a queue instead of just swallowing the EMFILE error
  • Wrap Read/Write streams as well

1.0

  • Update engines for node 0.6
  • Be lstat-graceful on Windows
  • first

NPM DownloadsLast 30 Days