You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This repository is used to manager openshift-virtualization-tests strategy & design docs, such as test plans (STPs) allowing enhanced SIG improvement and collaboration
3
+
This repository manages Quality Engineering (QE) artifacts, test plans, and testing strategies for OpenShift Virtualization enhancements, ensuring features meet defined quality standards before release.
4
4
5
-
#Openshift Virtualization Quality Engineering Artifacts and Software Test Planning (STP) Repository
5
+
## Purpose
6
6
7
-
This repository is used to manage and track Quality Engineering (QE) artifacts for Openshift Virtualization enhancements, ensuring that features meet defined quality standards and user requirements before release.
7
+
Centralize QE documentation and test planning to ensure:
8
8
9
-
## WHY
9
+
-**Clear visibility** of test coverage, resources, and risks
10
+
-**Systematic quality assurance** for all OpenShift Virtualization features
11
+
-**Formal QE sign-off** requirements are met
12
+
-**Automation is merged for GA** (mandatory)
10
13
11
-
The purpose of this repository and the associated process is to ensure the **quality and reliability** of the software. By centralizing these artifacts, we ensure clear visibility of test coverage, required resources, and risks.
14
+
## Quick Start
12
15
13
-
This process is mandatory for formal QE sign-off and feature acceptance, as **automation must be merged for GA**.
16
+
### For QE Engineers
14
17
15
-
## Glossary
18
+
1.**New Feature?** → Start with the [STP Guide](./docs/stp-guide.md)
The QE process is initiated upon feature definition, usually correlating with an approved VEP.
28
+
## Documentation
32
29
33
-
1.**Feature Review (Pre-STP)**: The QE owner reviews VEPs (Openshift Virtualization / OCP) and requirements to **understand the value** for RH customers. This stage confirms that requirements are **testable and unambiguous** and that acceptance criteria are clearly defined. Non-functional requirements (NFRs) like Downtime, Connectivity, Performance, Security, Monitoring (metrics/alerts), Portability, and Documentation must be covered.
34
-
2.**STP Creation**: The QE owner writes the STP, detailing the scope of testing (functional and non-functional requirements), the overall Test Strategy (including types like Functional, Regression, Upgrade, and Compatibility testing), Test Environment requirements, and Risk Management. The STP must include a clear plan to address risks, including calling out an explicit **list of functionalities that cannot be tested**.
35
-
3.**STD Creation**: Once the developer begins actual development, the QE owner writes the STD. STDs must create test cases **with the customer's perspective in mind**. Each step in the test case must be a test and **verify one thing**, and each step must include the expected results.
36
-
4.**Tiering and Automation**: QE works with the Development team to define coverage for **Tier 1 and Tier 2 tests**. Automation is crucial, and tests must be running as part of one of the **release checklist jobs**.
37
-
5.**Sign-off**: Upon meeting all **Exit Criteria** (high-priority defects resolved, test coverage achieved, test automation merged), the QE owner reviews documentation and formally signs off the feature. Jira tasks are used to **block the epic** until sign-off is complete.
- Reviewing the design and technology to identify potential testing challenges.
56
-
- Ensuring the plan addresses non-functional requirements (NFRs).
57
-
- Managing risk, including adding an explicit list of untestable aspects to the STP.
58
-
- Making sure tests are running as part of one of the **release checklist jobs**.
59
-
- Performing the final **Sign off the feature as QE**.
54
+
### Testing Tiers
60
55
61
-
### Stakeholders (SIGs/Approvers)
62
-
63
-
All relevant stakeholders, including SIG representatives, must be aware of and agree to the test plan.
64
-
65
-
- Stakeholders must **approve** the STP.
66
-
- Stakeholders must be aware of and **agree to take the risk** associated with any explicit list of functionalities that **cannot be tested**.
67
-
- SIGs are responsible for coordinating reviews and approvals of VEPs and STPs.
56
+
```text
57
+
Unit Tests → Tier 1 (Functional) → Tier 2 (End-to-End)
58
+
Many Moderate Few
59
+
Isolated Feature-level Full Integration
60
+
```
68
61
69
-
## Deadlines and Check-ins
62
+
See [Testing Tiers Guide](./docs/testing-tiers.md) for details.
70
63
71
-
Deadlines are governed by the Openshift Virtualization release cycle, focusing on:
64
+
## Process Flow: Feature to QE Sign-off
72
65
73
-
The QE process is governed by Openshift Virtualization and KubeVirt release milestones, focusing on test planning completion and automation delivery.
66
+
```mermaid
67
+
graph TD
68
+
A[Feature Defined] --> B[Feature Review]
69
+
B --> C[STP Creation]
70
+
C --> D[STP Approval]
71
+
D --> E[Test Implementation]
72
+
E --> F[Automation Merged]
73
+
F --> G[QE Sign-off]
74
+
```
74
75
75
-
1.**VEP Planning / Feature Review:**
76
+
### Milestones
76
77
77
-
- At the **beginning of every release cycle**, each SIG prioritizes VEPs and decides which ones are being tracked for the upcoming release. This centralized prioritization focuses community efforts on associated pull requests.
78
-
- QE involvement starts immediately with the **feature review**, ensuring the requirements are reviewed. The review includes verifying that requirements are **testable and unambiguous** and that the value of the feature for RH customers is understood.
78
+
1.**Feature Review (Pre-STP)**
79
+
- QE reviews feature requirements and enhancements
80
+
- Developer Handoff/QE Kickoff meeting
81
+
- Confirms testability and acceptance criteria
79
82
80
-
2.**Enhancement Freeze (EF):**
83
+
2.**STP Creation**
84
+
- Document scope, strategy, environment, risks
85
+
- Define Tier 1 and Tier 2 test coverage
86
+
- List untestable aspects with stakeholder agreement
81
87
82
-
- The Software Test Plan (STP) must be written, reviewed, and approved by stakeholders. Testing cannot begin until the requirements and design documents are approved and stable, and test cases are reviewed.
88
+
3.**Enhancement Freeze (EF)**
89
+
- STP written, reviewed, and approved
90
+
- Entry criteria met before testing begins
83
91
84
-
3.**Code Freeze (CF):**
92
+
4.**Test Implementation**
93
+
- Develop test cases per STP
94
+
- Implement automation
95
+
- Track against exit criteria
85
96
86
-
- QE sign-off requires that **test automation must be merged for GA**. The Exit Criteria for testing requires that test automation is merged.
87
-
- Features that do not land in the release branch prior to CF will need to file for an exception.
97
+
5.**Code Freeze (CF)**
98
+
-**Test automation must be merged for GA**
99
+
- All high-priority defects resolved
100
+
- Exit criteria met
88
101
89
-
> \[!NOTE\] QE sign-off is contingent on automation being merged. Features not landing in the release branch prior to CF will need to file for an exception.
102
+
6.**QE Sign-off**
103
+
- Feature meets all acceptance criteria
104
+
- Automation running in release checklist jobs
105
+
- Documentation reviewed and approved
90
106
91
107
## Development Setup
92
108
93
109
### Markdown Linting
94
110
95
-
This repository uses automated linting to maintain consistent Markdown formatting across all documentation.
111
+
This repository uses automated linting to maintain consistent Markdown formatting.
96
112
97
113
#### Prerequisites
98
114
99
-
Install the required tools:
100
-
101
115
```bash
102
-
# Install markdownlint-cli
103
-
npm install -g markdownlint-cli
104
-
105
-
# Install pre-commit (for automated checks)
116
+
# Install pre-commit
106
117
pip install pre-commit
107
118
108
119
# Install pre-commit hooks
@@ -111,51 +122,62 @@ pre-commit install
111
122
112
123
#### Running Linters
113
124
114
-
**Manual linting:**
115
-
116
125
```bash
117
-
# Lint all Markdown files
118
-
markdownlint '**/*.md'
119
-
120
-
# Lint specific file
121
-
markdownlint stps/stp-template/stp.md
126
+
# Run all pre-commit hooks manually
127
+
pre-commit run --all-files
122
128
123
-
#Auto-fix issues where possible
124
-
markdownlint '**/*.md' --fix
129
+
#Run on specific files
130
+
pre-commit run --files README.md docs/stp-guide.md
125
131
```
126
132
127
-
**Automated linting (via pre-commit):**
133
+
See `.markdownlint.yaml` for complete configuration with detailed comments for each rule.
128
134
129
-
Once pre-commit hooks are installed, linting runs automatically on every commit:
135
+
## Responsibilities
130
136
131
-
```bash
132
-
# Run all pre-commit hooks manually
133
-
pre-commit run --all-files
137
+
### QE Owner
134
138
135
-
# Run only markdownlint
136
-
pre-commit run markdownlint --all-files
137
-
```
139
+
- Write and maintain STP
140
+
- Review design and identify testing challenges
141
+
- Ensure NFRs are addressed
142
+
- Manage risks and document untestable aspects
143
+
- Ensure automation runs in release checklist jobs
144
+
- Provide formal feature sign-off
138
145
139
-
#### Linting Rules
146
+
###Stakeholders (SIGs/Approvers)
140
147
141
-
The repository uses `.markdownlint.yaml` with the following key settings:
148
+
- Approve STP
149
+
- Agree to risks associated with untestable aspects
150
+
- Coordinate feature enhancement and STP reviews
142
151
143
-
-**Line length**: Disabled (no line length restrictions)
144
-
-**Bare URLs**: Allowed
145
-
-**Hard tabs**: Allowed (for code examples)
146
-
-**Inline HTML**: Allowed
147
-
-**Trailing whitespace**: Allowed
148
-
-**Emphasis style**: Flexible
152
+
## Common Questions
149
153
150
-
See `.markdownlint.yaml` for complete configuration with detailed comments for each rule.
|**Do PRs need STP approver sign-off?**| No, the whole SIG is responsible for code approval. Approvers should be aware of the STP. |
157
+
|**What if PRs miss the deadline?**| STP author files for an exception; maintainers decide based on context. |
158
+
|**How to raise attention for my STP?**| Join bi-weekly QE recurring meetings to introduce your STP. |
159
+
|**What if feature relates to multiple SIGs?**| One SIG must own it. Reach out to other relevant SIGs for review. |
160
+
161
+
## Resources
162
+
163
+
### Internal
164
+
165
+
-[STP Guide](./docs/stp-guide.md)
166
+
-[Testing Tiers Guide](./docs/testing-tiers.md)
167
+
-[STP Template](./stps/stp-template/stp.md)
168
+
169
+
## Contributing
170
+
171
+
1. Create STP using the [template](./stps/stp-template/stp.md)
172
+
2. Follow the [STP Guide](./docs/stp-guide.md) for structure and content
173
+
3. Ensure proper test tier coverage per [Testing Tiers Guide](./docs/testing-tiers.md)
174
+
4. Get stakeholder approval before testing begins
175
+
5. Ensure automation is merged before GA
151
176
152
-
## **Common Questions**
177
+
## Support
153
178
154
-
This section addresses frequently asked questions regarding VEP ownership, approvals, and managing delays, which impact the QE process and feature readiness.
|**Do PRs need to be approved by STP approvers?**| No, the approval of Pull Requests (PRs) is the **whole SIG's responsibility** for their code. The approver should be aware of the STP and approve based on its content, and a defined process exists to ensure this occurs. |
159
-
|**What to do in case all PRs didn't make it on time?**| The author of the STP needs to **file for an exception**. The outcome will be determined individually by maintainers based on the context of the delay. |
160
-
|**How to raise attention for my STP?**| The QE has bi-weekly **recurring meetings**. The STP owner is encouraged to join the meeting and introduce the STP to the community to raise awareness. |
161
-
|**What if the feature is relates to more than one SIG?**| While features can relate to **multiple SIGs**, a single SIG must always own it. STP owners should reach out to other SIGs that are relevant for the feature and make sure that they also review the STP. |
0 commit comments