You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This repository is used to manager openshift-virtualization-tests strategy & design docs, such as test plans (STPs) allowing enhanced SIG improvement and collaboration
3
4
4
5
# Openshift Virtualization Quality Engineering Artifacts and Software Test Planning (STP) Repository
@@ -14,7 +15,7 @@ This process is mandatory for formal QE sign-off and feature acceptance, as **au
|**GA*|**General Availability** - Feature is available for general use |
27
+
|\**GA*|**General Availability** - Feature is available for general use |
27
28
28
29
## Process Flow: VEP to QE Sign-off
29
30
30
31
The QE process is initiated upon feature definition, usually correlating with an approved VEP.
31
32
32
-
1.**Feature Review (Pre-STP)**: The QE owner reviews VEPs (Openshift Virtualization / OCP) and requirements to **understand the value** for RH customers. This stage confirms that requirements are **testable and unambiguous** and that acceptance criteria are clearly defined. Non-functional requirements (NFRs) like Downtime, Connectivity, Performance, Security, Monitoring (metrics/alerts), Portability, and Documentation must be covered.
33
-
2.**STP Creation**: The QE owner writes the STP, detailing the scope of testing (functional and non-functional requirements), the overall Test Strategy (including types like Functional, Regression, Upgrade, and Compatibility testing), Test Environment requirements, and Risk Management. The STP must include a clear plan to address risks, including calling out an explicit **list of functionalities that cannot be tested**.
34
-
3.**STD Creation**: Once the developer begins actual development, the QE owner writes the STD. STDs must create test cases **with the customer's perspective in mind**. Each step in the test case must be a test and **verify one thing**, and each step must include the expected results.
35
-
4.**Tiering and Automation**: QE works with the Development team to define coverage for **Tier 1 and Tier 2 tests**. Automation is crucial, and tests must be running as part of one of the **release checklist jobs**.
36
-
5.**Sign-off**: Upon meeting all **Exit Criteria** (high-priority defects resolved, test coverage achieved, test automation merged), the QE owner reviews documentation and formally signs off the feature. Jira tasks are used to **block the epic** until sign-off is complete.
33
+
1.**Feature Review (Pre-STP)**: The QE owner reviews VEPs (Openshift Virtualization / OCP) and requirements to **understand the value** for RH customers. This stage confirms that requirements are **testable and unambiguous** and that acceptance criteria are clearly defined. Non-functional requirements (NFRs) like Downtime, Connectivity, Performance, Security, Monitoring (metrics/alerts), Portability, and Documentation must be covered.
34
+
2.**STP Creation**: The QE owner writes the STP, detailing the scope of testing (functional and non-functional requirements), the overall Test Strategy (including types like Functional, Regression, Upgrade, and Compatibility testing), Test Environment requirements, and Risk Management. The STP must include a clear plan to address risks, including calling out an explicit **list of functionalities that cannot be tested**.
35
+
3.**STD Creation**: Once the developer begins actual development, the QE owner writes the STD. STDs must create test cases **with the customer's perspective in mind**. Each step in the test case must be a test and **verify one thing**, and each step must include the expected results.
36
+
4.**Tiering and Automation**: QE works with the Development team to define coverage for **Tier 1 and Tier 2 tests**. Automation is crucial, and tests must be running as part of one of the **release checklist jobs**.
37
+
5.**Sign-off**: Upon meeting all **Exit Criteria** (high-priority defects resolved, test coverage achieved, test automation merged), the QE owner reviews documentation and formally signs off the feature. Jira tasks are used to **block the epic** until sign-off is complete.
37
38
38
39
## Artifacts and Templates
39
40
40
41
This repository hosts the templates for the core QE deliverable:
41
42
42
43
**Software Test Plan (STP) Template**: Outlines the scope, approach, resources, and schedule for all testing activities.
43
-
***Entry Criteria**: Requirements/design approved, environment set up, test cases reviewed.
The QE Owner is responsible for the overall quality assurance process for a feature, which includes:
51
53
52
-
* Writing the STP and STD.
53
-
* Reviewing the design and technology to identify potential testing challenges.
54
-
* Ensuring the plan addresses non-functional requirements (NFRs).
55
-
* Managing risk, including adding an explicit list of untestable aspects to the STP.
56
-
* Making sure tests are running as part of one of the **release checklist jobs**.
57
-
* Performing the final **Sign off the feature as QE**.
54
+
- Writing the STP and STD.
55
+
- Reviewing the design and technology to identify potential testing challenges.
56
+
- Ensuring the plan addresses non-functional requirements (NFRs).
57
+
- Managing risk, including adding an explicit list of untestable aspects to the STP.
58
+
- Making sure tests are running as part of one of the **release checklist jobs**.
59
+
- Performing the final **Sign off the feature as QE**.
58
60
59
61
### Stakeholders (SIGs/Approvers)
60
62
61
63
All relevant stakeholders, including SIG representatives, must be aware of and agree to the test plan.
62
64
63
-
* Stakeholders must **approve** the STP.
64
-
* Stakeholders must be aware of and **agree to take the risk** associated with any explicit list of functionalities that **cannot be tested**.
65
-
* SIGs are responsible for coordinating reviews and approvals of VEPs and STPs.
65
+
- Stakeholders must **approve** the STP.
66
+
- Stakeholders must be aware of and **agree to take the risk** associated with any explicit list of functionalities that **cannot be tested**.
67
+
- SIGs are responsible for coordinating reviews and approvals of VEPs and STPs.
66
68
67
69
## Deadlines and Check-ins
68
70
69
71
Deadlines are governed by the Openshift Virtualization release cycle, focusing on:
70
72
71
73
The QE process is governed by Openshift Virtualization and KubeVirt release milestones, focusing on test planning completion and automation delivery.
72
74
73
-
1.**VEP Planning / Feature Review:**
74
-
* At the **beginning of every release cycle**, each SIG prioritizes VEPs and decides which ones are being tracked for the upcoming release. This centralized prioritization focuses community efforts on associated pull requests.
75
-
* QE involvement starts immediately with the **feature review**, ensuring the requirements are reviewed. The review includes verifying that requirements are **testable and unambiguous** and that the value of the feature for RH customers is understood.
75
+
1.**VEP Planning / Feature Review:**
76
+
77
+
- At the **beginning of every release cycle**, each SIG prioritizes VEPs and decides which ones are being tracked for the upcoming release. This centralized prioritization focuses community efforts on associated pull requests.
78
+
- QE involvement starts immediately with the **feature review**, ensuring the requirements are reviewed. The review includes verifying that requirements are **testable and unambiguous** and that the value of the feature for RH customers is understood.
79
+
80
+
2.**Enhancement Freeze (EF):**
81
+
82
+
- The Software Test Plan (STP) must be written, reviewed, and approved by stakeholders. Testing cannot begin until the requirements and design documents are approved and stable, and test cases are reviewed.
83
+
84
+
3.**Code Freeze (CF):**
85
+
86
+
- QE sign-off requires that **test automation must be merged for GA**. The Exit Criteria for testing requires that test automation is merged.
87
+
- Features that do not land in the release branch prior to CF will need to file for an exception.
88
+
89
+
> \[!NOTE\] QE sign-off is contingent on automation being merged. Features not landing in the release branch prior to CF will need to file for an exception.
90
+
91
+
## Development Setup
92
+
93
+
### Markdown Linting
94
+
95
+
This repository uses automated linting to maintain consistent Markdown formatting across all documentation.
96
+
97
+
#### Prerequisites
98
+
99
+
Install the required tools:
100
+
101
+
```bash
102
+
# Install markdownlint-cli
103
+
npm install -g markdownlint-cli
104
+
105
+
# Install pre-commit (for automated checks)
106
+
pip install pre-commit
107
+
108
+
# Install pre-commit hooks
109
+
pre-commit install
110
+
```
111
+
112
+
#### Running Linters
113
+
114
+
**Manual linting:**
115
+
116
+
```bash
117
+
# Lint all Markdown files
118
+
markdownlint '**/*.md'
119
+
120
+
# Lint specific file
121
+
markdownlint stps/stp-template/stp.md
122
+
123
+
# Auto-fix issues where possible
124
+
markdownlint '**/*.md' --fix
125
+
```
126
+
127
+
**Automated linting (via pre-commit):**
128
+
129
+
Once pre-commit hooks are installed, linting runs automatically on every commit:
130
+
131
+
```bash
132
+
# Run all pre-commit hooks manually
133
+
pre-commit run --all-files
134
+
135
+
# Run only markdownlint
136
+
pre-commit run markdownlint --all-files
137
+
```
76
138
77
-
2.**Enhancement Freeze (EF):**
78
-
* The Software Test Plan (STP) must be written, reviewed, and approved by stakeholders. Testing cannot begin until the requirements and design documents are approved and stable, and test cases are reviewed.
139
+
#### Linting Rules
79
140
80
-
3.**Code Freeze (CF):**
81
-
* QE sign-off requires that **test automation must be merged for GA**. The Exit Criteria for testing requires that test automation is merged.
82
-
* Features that do not land in the release branch prior to CF will need to file for an exception.
141
+
The repository uses `.markdownlintrc` with the following key settings:
83
142
84
-
> [!NOTE]
85
-
> QE sign-off is contingent on automation being merged. Features not landing in the release branch prior to CF will need to file for an exception.
143
+
-**Line length**: Disabled (no line length restrictions)
144
+
-**Bare URLs**: Allowed
145
+
-**Hard tabs**: Allowed (for code examples)
146
+
-**Inline HTML**: Allowed
147
+
-**Trailing whitespace**: Allowed
148
+
-**Emphasis style**: Flexible
86
149
150
+
See `.markdownlintrc` for complete configuration.
87
151
88
152
## **Common Questions**
89
153
90
154
This section addresses frequently asked questions regarding VEP ownership, approvals, and managing delays, which impact the QE process and feature readiness.
|**Do PRs need to be approved by STP approvers?**| No, the approval of Pull Requests (PRs) is the **whole SIG's responsibility** for their code. The approver should be aware of the STP and approve based on its content, and a defined process exists to ensure this occurs.|
95
-
|**What to do in case all PRs didn't make it on time?**| The author of the STP needs to **file for an exception**. The outcome will be determined individually by maintainers based on the context of the delay.|
96
-
|**How to raise attention for my STP?**| The QE has bi-weekly **recurring meetings**. The STP owner is encouraged to join the meeting and introduce the STP to the community to raise awareness.|
97
-
|**What if the feature is relates to more than one SIG?**| While features can relate to **multiple SIGs**, a single SIG must always own it. STP owners should reach out to other SIGs that are relevant for the feature and make sure that they also review the STP.|
|**Do PRs need to be approved by STP approvers?**| No, the approval of Pull Requests (PRs) is the **whole SIG's responsibility** for their code. The approver should be aware of the STP and approve based on its content, and a defined process exists to ensure this occurs. |
159
+
|**What to do in case all PRs didn't make it on time?**| The author of the STP needs to **file for an exception**. The outcome will be determined individually by maintainers based on the context of the delay. |
160
+
|**How to raise attention for my STP?**| The QE has bi-weekly **recurring meetings**. The STP owner is encouraged to join the meeting and introduce the STP to the community to raise awareness. |
161
+
|**What if the feature is relates to more than one SIG?**| While features can relate to **multiple SIGs**, a single SIG must always own it. STP owners should reach out to other SIGs that are relevant for the feature and make sure that they also review the STP. |
0 commit comments