Skip to content

Conversation

tilmanschweitzer
Copy link

Add parameter --maxConcurrency to prevent different load errors.

Depending on the content different load errors like 'Bandwidth exhausted'
or 'Received RST_STREAM with code 2' occur and cause the import to be incomplete.
Limiting the concurrent imported batches seems to prevents issues.
The default behaviour is still without any limit.

see #411

Tilman Schweitzer added 2 commits December 7, 2021 22:09
Depending on the content different load errors like 'Bandwidth exhausted'
or 'Received RST_STREAM with code 2' occur and cause the import to be incomplete.
Limiting the concurrent imported batches seems to prevents issues.
The default behaviour is still without any limit.

see jloosli#411
@nonoumasy
Copy link

does this fix the bandwidth exhausted issue?

@tilmanschweitzer
Copy link
Author

@nonoumasy A max concurrency of 45 worked for my use case. Around a concurrency of 48 the issue started again. But I don't know if there are other aspects that contribute to the issue.

@nonoumasy
Copy link

nonoumasy commented Dec 19, 2022

@nonoumasy A max concurrency of 45 worked for my use case. Around a concurrency of 48 the issue started again. But I don't know if there are other aspects that contribute to the issue.

Thanks @tilmanschweitzer.
btw how do I use this parameter?
Do I just do something like?

npx -p node-firestore-import-export firestore-import -a credentials.json -b -maxConcurrency=45 backup.json

Also, when you get this issue after importing, is there possible data loss(some data doesn't get imported into firestore)? is data corruption a possibility?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants