[BUG] corrupt deflate stream #131
Description
Describe the Bug
When running the Ungoliant pipeline, everything proceeds smoothly initially as the JSONL files for each language are built. However, after a couple of hours, an error suddenly appears in the logs, and thereafter, only this error persists. I am curious as to why this occurs and whether it could be resolved by skipping the problematic inputs.
[2024-03-27T23:49:00Z ERROR ungoliant::pipelines::oscardoc::pipeline] ReadData(Custom { kind: InvalidInput, error: "corrupt deflate stream" })
To Reproduce
Nothing specific to mention, just the routine: downloading and pipelining.
Expected Behavior
The expected behavior is for the pipeline to function as it did earlier or to skip the corrupt inputs.
Screenshots
at first:
later:
Desktop (Please Complete the Following Information):
uname -a
Linux delta 5.14.21-150500.55.36-default #1 SMP PREEMPT_DYNAMIC Tue Oct 31 08:37:43 UTC 2023 (e7a2e23) x86_64 x86_64 x86_64 GNU/Linux