through2-concurrent
Like through2 except runs in parallel with limited concurrency
Last updated 2 years ago by almost .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install through2-concurrent 
SYNC missed versions from official npm registry.

through2-concurrent

NPM

A simple way to create a Node.JS Transform stream which processes in parallel. You can limit the concurrency (default is 16) and order is not preserved (so chunks/objects can end up in a different order to the order they started in if the transform functions take different amounts of time).

Built using through2 and has the same API with the addition of a maxConcurrency option.

Non-objectMode streams are supported for completeness but I'm not sure they'd be useful for anything.

Written by Thomas Parslow (almostobsolete.net and tomparslow.co.uk) as part of Active Inbox (activeinboxhq.com).

Build Status

Install

npm install --save through2-concurrent

Examples

Process lines from a CSV in paralel. The order the results end up in the all variable is not deterministic.

var through2Concurrent = require('through2-concurrent');

var all = [];

fs.createReadStream('data.csv')
  .pipe(csv2())
  .pipe(through2Concurrent.obj(
    {maxConcurrency: 10},
    function (chunk, enc, callback) {
      var self = this;
      someThingAsync(chunk, function (newChunk) {
        self.push(newChunk);
        callback();
      });
  }))
  .on('data', function (data) {
    all.push(data)
  })
  .on('end', function () {
    doSomethingSpecial(all)
  })

Contributing

Fixed or improved stuff? Great! Send me a pull request through GitHub or get in touch on Twitter @almostobsolete or email at tom@almostobsolete.net

Current Tags

  • 2.0.0                                ...           latest (2 years ago)

9 Versions

  • 2.0.0                                ...           2 years ago
  • 1.1.1                                ...           4 years ago
  • 1.1.0                                ...           5 years ago
  • 1.0.0                                ...           5 years ago
  • 0.4.0                                ...           5 years ago
  • 0.3.1                                ...           6 years ago
  • 0.3.0                                ...           6 years ago
  • 0.2.0                                ...           6 years ago
  • 0.1.0                                ...           6 years ago
Maintainers (1)
Downloads
Today 7
This Week 1,055
This Month 1,937
Last Day 275
Last Week 1,357
Last Month 6,008
Dependencies (1)
Dev Dependencies (3)

Copyright 2014 - 2016 © taobao.org |