Skip to content

Commit 27cb3e0

Browse files
sabrennerbrettlangdon
authored andcommitted
ci(llmobs): skip flaky tests hitting faulty endpoint (#14480)
some of our tests hit our real staging endpoint, which itself could sometimes be flaky. following up soon with a real fix to record a response one time and serve it back (since these tests are just checking logging behavior). ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) (cherry picked from commit 5b0b7c6)
1 parent 6932159 commit 27cb3e0

File tree

2 files changed

+3
-0
lines changed

2 files changed

+3
-0
lines changed

tests/llmobs/test_llmobs_eval_metric_agentless_writer.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ def test_buffer_limit(mock_writer_logs):
6363
)
6464

6565

66+
@pytest.mark.skip(reason="Skipping due to flakiness in hitting the staging endpoint")
6667
def test_send_metric_bad_api_key(mock_writer_logs):
6768
llmobs_eval_metric_writer = LLMObsEvalMetricWriter(1, 1, is_agentless=True, _site=DD_SITE, _api_key="<bad-api-key>")
6869

@@ -142,6 +143,7 @@ def test_send_multiple_events(mock_writer_logs):
142143
)
143144

144145

146+
@pytest.mark.skip(reason="Skipping due to flakiness in hitting the staging endpoint")
145147
def test_send_on_exit(mock_writer_logs, run_python_code_in_subprocess):
146148
env = os.environ.copy()
147149
pypath = [os.path.dirname(os.path.dirname(os.path.dirname(__file__)))]

tests/llmobs/test_llmobs_evaluator_runner.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ def test_evaluator_runner_multiple_evaluators(llmobs, mock_llmobs_eval_metric_wr
9292
]
9393

9494

95+
@pytest.mark.skip(reason="Skipping due to flakiness in hitting the staging endpoint")
9596
def test_evaluator_runner_on_exit(mock_writer_logs, run_python_code_in_subprocess):
9697
env = os.environ.copy()
9798
pypath = [os.path.dirname(os.path.dirname(os.path.dirname(__file__)))]

0 commit comments

Comments
 (0)