Unite parse metadata and save to mongo jobs

[CodeJjang]

Inserting to mongo is more efficient than passing through rabbitmq, so additional job is just useless overhead.

[motyros]

What about persistence?
If the Mongo is down the RabbitMQ will save the parsed Metadata and won't need to run the parser all over again.
(You need to remember that parser process is include also downloading of the data file from the S3.)
I really don't think it's a good idea to combine all of that in one big consumer.

[CodeJjang]

"If the Mongo is down" is less likely to happen than "rabbit is down". Mongo has 99.9999% uptime.
Plus, insertion to mongo is more robust and efficient than insertion to a rabbit queue.
Therefore, we do not gain anything from passing through rabbit in this issue.

In the worst and passimistic case, if we have no connection to mongo, the data is still saved and will be re-parsed because the job will be requeued. But in the common normal daily behaviour we should earn a significant advantage in speed and efficiency.

[motyros]

I may agree on the Mongo vs RabbitMQ, but why we need to do the parser all over again?
Why we beed to download the data file again from the S3?
In my opinion, parsering is a separate job from the saving the Metadata to the Mongo.
I still think is ok that it's in a separate consumer job.