Stream large collections

[erik-stephens]

Summary

As a developer of an API, I would like large collections streamed in chunks in order to minimize the memory footprint required to serve such requests.

Notes

Looks like Transfer-Encoding: chunked is already baked into express.js or node.js. However, looks like express.js is serializing json responses like this:

var body = JSON.stringify(obj, replacer, spaces);

Fortunately, looks like there is a stream-friendly mongoose api. Will just need to handle the serialization in crudify. JSONStream looks promising for that, but if not, can do it in crudify with something like this in sendResult.coffee:

res.send('[');
delim = '';
entity for entity in data
  res.send(delim + JSON.stringify(entity))
  delim = ','
res.send(']');

Tasks

• Use the Mongoose.QueryStream API
• Handle the serialization of ourselves instead of relying on express.js.

References

• http://mongoosejs.com/docs/2.7.x/docs/querystream.html
• https://github.com/dominictarr/JSONStream

[yocontra]

We had this before but removed it when we added the authorization in. I need to figure out a way to do streaming auth without clogging the CPU