json-stream-reduce
Based on Mapbox's tile-reduce, a framework for map-reduce jobs in Node from streaming input
Last updated 2 years ago by jenningsanderson .
ISC · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install json-stream-reduce 
SYNC missed versions from official npm registry.

JSON-Stream-Reduce

This is a simplified framework built off Mapbox's tile-reduce to perform map-reduce functions against large files of line-delimited JSON. This simply removes all of the tile processing and instead passes each new line in the file to the map script.

This is currently used for geometry-reconstruction of historical OSM objects as output from the OSM-Wayback utility

Install

npm install json-stream-reduce

Example Implementation

File: example/index.js

var count = 0;

streamReduce({
  map: path.join(__dirname, 'map.js'),             //Map function
  file: path.join(__dirname, 'file.jsonseq'),      //Input file (lines of JSON)
  maxWorkers:10 // The number of cpus you'd like to use
})
.on('reduce', function(res) {
  count+=res
})
.on('end', function() {
  console.log("Finished with count value: "+count)
});

File: example/map.js

module.exports = function(line, writeData, done) {
  var object = JSON.parse(line.toString());
  done(object.count)
})

file.jsonseq

{"count":10}
{"count":10}
{"count":10}
{"count":10}
{"count":10}

Use

$ node examples/index.js

>>Starting up 2 workers... Job started.
>>Processing lines from file: examples/file.jsonseq
>>5 lines processed in 0s.
>>Finished, value of count: 50

Current Tags

  • 1.0.1                                ...           latest (2 years ago)

2 Versions

  • 1.0.1                                ...           2 years ago
  • 1.0.0                                ...           2 years ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 0
Dependencies (9)
Dev Dependencies (0)
None
Dependents (0)
None

Copyright 2014 - 2017 © taobao.org |