Open
Conversation
Contributor
📝 WalkthroughSummary by CodeRabbit
WalkthroughSix e2e API test runner scripts have been updated to modify Newman timeout configuration. The --timeout-script parameter was reduced from 300000ms to 120000ms, and a new --timeout parameter set to 900000ms was added to each Newman invocation. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🧪 Test Suite AvailableThis PR can be tested by a repository admin. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/e2e/api/run-newman-composite-integration.sh (1)
175-206:⚠️ Potential issue | 🟠 MajorAvoid
evalinrun_newman; this is injection-prone.Lines [175-206] build a shell command string from interpolated values (including
--folder) and execute it witheval, which can lead to command injection and quoting bugs.🔒 Safer array-based rewrite
run_newman() { - local cmd="newman run $COLLECTION" + local -a cmd=(newman run "$COLLECTION") if [ -n "${2:-}" ] && [ -f "${2}" ]; then - cmd="$cmd -e ${2}" + cmd+=(-e "${2}") else local base_url="${BIFROST_BASE_URL:-http://localhost:8080}" local provider="${BIFROST_PROVIDER:-openai}" local model="${BIFROST_MODEL:-gpt-4o}" local embedding_model="${BIFROST_EMBEDDING_MODEL:-text-embedding-3-small}" local speech_model="${BIFROST_SPEECH_MODEL:-tts-1}" local transcription_model="${BIFROST_TRANSCRIPTION_MODEL:-whisper-1}" local image_model="${BIFROST_IMAGE_MODEL:-dall-e-3}" if [ -n "$ENV_FLAG" ]; then - cmd="$cmd $ENV_FLAG" + cmd+=(-e "$ENVIRONMENT") fi - cmd="$cmd --env-var \"base_url=$base_url\" --env-var \"provider=$provider\" --env-var \"model=$model\" --env-var \"embedding_model=$embedding_model\" --env-var \"speech_model=$speech_model\" --env-var \"transcription_model=$transcription_model\" --env-var \"image_model=$image_model\"" + cmd+=(--env-var "base_url=$base_url" --env-var "provider=$provider" --env-var "model=$model" --env-var "embedding_model=$embedding_model" --env-var "speech_model=$speech_model" --env-var "transcription_model=$transcription_model" --env-var "image_model=$image_model") fi - [ -n "$FOLDER" ] && cmd="$cmd $FOLDER" - cmd="$cmd --timeout-script 120000 --timeout 900000 -r $REPORTERS" + [ -n "$FOLDER" ] && cmd+=(--folder "$FOLDER") + cmd+=(--timeout-script 120000 --timeout 900000 -r "$REPORTERS") if [[ "$REPORTERS" == *"html"* ]]; then - cmd="$cmd --reporter-html-export $REPORT_DIR/report_${1:-run}.html" + cmd+=(--reporter-html-export "$REPORT_DIR/report_${1:-run}.html") fi if [[ "$REPORTERS" == *"json"* ]]; then - cmd="$cmd --reporter-json-export $REPORT_DIR/report_${1:-run}.json" + cmd+=(--reporter-json-export "$REPORT_DIR/report_${1:-run}.json") fi - [ -n "$VERBOSE" ] && cmd="$cmd $VERBOSE" - [ -n "$BAIL" ] && cmd="$cmd $BAIL" + [ -n "$VERBOSE" ] && cmd+=("$VERBOSE") + [ -n "$BAIL" ] && cmd+=("$BAIL") if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then - cmd="$cmd --env-var \"CI=1\"" + cmd+=(--env-var "CI=1") fi - - eval $cmd + "${cmd[@]}" }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/e2e/api/run-newman-composite-integration.sh` around lines 175 - 206, The current run_newman code builds a single string in cmd and executes it with eval, which is injection-prone; replace this with a safely quoted array-based invocation: create an array (e.g., cmd_args) instead of the string cmd, append program name "newman" then push each argument as separate elements (use cmd_args+=( "-e" "$2" ) when a file is provided, otherwise append each --env-var as separate elements with their values quoted, and handle ENV_FLAG, FOLDER, REPORTERS, REPORT_DIR, VERBOSE, BAIL, and ci_normalized by appending appropriate elements like "--timeout-script" "120000" and reporter export flags), and finally run newman with: "${cmd_args[@]}" (no eval). Ensure you stop using eval $cmd and keep all variable expansions quoted to prevent injection.
🧹 Nitpick comments (1)
tests/e2e/api/run-newman-anthropic-integration.sh (1)
188-188: Make timeout values configurable once to reduce stack-wide drift.Line [188] hardcodes values that are duplicated across multiple runner scripts. Consider env-backed defaults so future stack changes are single-point updates.
♻️ Proposed refactor
+# Timeout configuration (override in CI if needed) +NEWMAN_TIMEOUT_SCRIPT_MS="${NEWMAN_TIMEOUT_SCRIPT_MS:-120000}" +NEWMAN_TIMEOUT_MS="${NEWMAN_TIMEOUT_MS:-900000}" + run_newman() { @@ - cmd+=(--timeout-script 120000 --timeout 900000) + cmd+=(--timeout-script "$NEWMAN_TIMEOUT_SCRIPT_MS" --timeout "$NEWMAN_TIMEOUT_MS")As per coding guidelines, "always check the stack if there is one for the current PR. do not give localized reviews for the PR, always see all changes in the light of the whole stack of PRs (if there is a stack, if there is no stack you can continue to make localized suggestions/reviews)".
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/e2e/api/run-newman-anthropic-integration.sh` at line 188, The hardcoded timeouts in the cmd array (the tokens '--timeout-script' and '--timeout') inside run-newman-anthropic-integration.sh should be replaced with env-backed defaults so all runner scripts share a single configuration point; update the script to read NEWMAN_TIMEOUT_SCRIPT and NEWMAN_TIMEOUT (or a single NEWMAN_TIMEOUTS JSON/CSV) with sensible defaults and use those variables when appending to cmd, and extract the same env defaults into a common sourced file (or CI variable) so other runner scripts reference the same values.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@tests/e2e/api/run-newman-composite-integration.sh`:
- Around line 175-206: The current run_newman code builds a single string in cmd
and executes it with eval, which is injection-prone; replace this with a safely
quoted array-based invocation: create an array (e.g., cmd_args) instead of the
string cmd, append program name "newman" then push each argument as separate
elements (use cmd_args+=( "-e" "$2" ) when a file is provided, otherwise append
each --env-var as separate elements with their values quoted, and handle
ENV_FLAG, FOLDER, REPORTERS, REPORT_DIR, VERBOSE, BAIL, and ci_normalized by
appending appropriate elements like "--timeout-script" "120000" and reporter
export flags), and finally run newman with: "${cmd_args[@]}" (no eval). Ensure
you stop using eval $cmd and keep all variable expansions quoted to prevent
injection.
---
Nitpick comments:
In `@tests/e2e/api/run-newman-anthropic-integration.sh`:
- Line 188: The hardcoded timeouts in the cmd array (the tokens
'--timeout-script' and '--timeout') inside run-newman-anthropic-integration.sh
should be replaced with env-backed defaults so all runner scripts share a single
configuration point; update the script to read NEWMAN_TIMEOUT_SCRIPT and
NEWMAN_TIMEOUT (or a single NEWMAN_TIMEOUTS JSON/CSV) with sensible defaults and
use those variables when appending to cmd, and extract the same env defaults
into a common sourced file (or CI variable) so other runner scripts reference
the same values.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
tests/e2e/api/run-newman-anthropic-integration.shtests/e2e/api/run-newman-api-tests.shtests/e2e/api/run-newman-bedrock-integration.shtests/e2e/api/run-newman-composite-integration.shtests/e2e/api/run-newman-openai-integration.shtests/e2e/api/run-newman-tests.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adjusted Newman timeout configuration for e2e API tests to improve test reliability by reducing script timeout while increasing overall request timeout.
Changes
--timeout-scriptfrom 300000ms (5 minutes) to 120000ms (2 minutes) across all Newman test scripts--timeoutparameter set to 900000ms (15 minutes) for overall request timeoutThe change allows individual scripts to fail faster while giving more time for the overall test collection to complete, which should reduce hanging tests while accommodating longer-running integration scenarios.
Type of change
Affected areas
How to test
Run the e2e API tests to verify the new timeout configuration works correctly:
Verify that tests complete within expected timeframes and don't hang indefinitely on slow responses.
Screenshots/Recordings
N/A
Breaking changes
Related issues
N/A
Security considerations
No security implications - this is purely a test configuration change.
Checklist
docs/contributing/README.mdand followed the guidelines