Best practice to ingest large amount of data?

[geekboood]

Hi, I was inspired by your article on processing Common Crawl data (https://quickwit.io/blog/commoncrawl) and I'm working on a similar, smaller-scale project.
I'm having trouble with quickwit index ingest directly from my S3-compatible object store due to its instability. The import process often fails and causes data duplication upon retry.
Is it possible to generate the index and data files locally first and then upload them to the S3 store? This would help me avoid the ingestion issues.