Currently the approach I am taking is a little brute-force, and won't scale well in terms of memory usage as the number of cases grows.
A better approach would be to use event-streams to emit data to the output file as we scrape it. This won't necessarily be faster, but it will significantly reduce the memory needed, and is a much more elegant and scalable solution.
Currently the approach I am taking is a little brute-force, and won't scale well in terms of memory usage as the number of cases grows.
A better approach would be to use
event-streamsto emit data to the output file as we scrape it. This won't necessarily be faster, but it will significantly reduce the memory needed, and is a much more elegant and scalable solution.