Skip to content

Commit 047514c

Browse files
author
Neelanjan Manna
authored
chore(Chaos)(Faults): Adds Fault Docs for EC2, GCP, and Azure experiments (harness#113)
* added docs for aws ec2, gcp & azure experiments * added cd integration tutorial Signed-off-by: Akash Shrivastava <[email protected]>
1 parent 4b42c27 commit 047514c

File tree

366 files changed

+10462
-972
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

366 files changed

+10462
-972
lines changed

‎docs/chaos-engineering/chaos-faults/aws/aws-experiments-tunables.md‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,7 @@ metadata:
1919
name: engine-nginx
2020
spec:
2121
engineState: "active"
22-
annotationCheck: "false"
23-
chaosServiceAccount: ec2-terminate-by-tag-sa
22+
chaosServiceAccount: litmus-admin
2423
experiments:
2524
- name: ec2-terminate-by-tag
2625
spec:
@@ -32,7 +31,7 @@ spec:
3231
value: 'enable'
3332
# region for the ec2 instance
3433
- name: REGION
35-
value: '<region for instances>'
34+
value: 'us-east-1'
3635
# tag of the ec2 instance
3736
- name: INSTANCE_TAG
3837
value: 'key:value'
@@ -55,7 +54,6 @@ metadata:
5554
name: engine-nginx
5655
spec:
5756
engineState: "active"
58-
annotationCheck: "false"
5957
chaosServiceAccount: ec2-terminate-by-tag-sa
6058
experiments:
6159
- name: ec2-terminate-by-tag
@@ -69,7 +67,7 @@ spec:
6967
- name: TOTAL_CHAOS_DURATION
7068
VALUE: '60'
7169
- name: REGION
72-
value: '<region for instances>'
70+
value: 'us-east-1'
7371
- name: INSTANCE_TAG
7472
value: 'key:value'
7573
```

‎docs/chaos-engineering/chaos-faults/aws/ebs-loss-by-id.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
---
2+
id: ebs-loss-by-id
3+
title: EBS Loss By ID
4+
---
5+
6+
## Introduction
7+
- It causes chaos to disrupt state of ebs volume by detaching it from the node/ec2 instance for a certain chaos duration using volume id.
8+
- In case of EBS persistent volumes, the volumes can get self-attached and experiment skips the re-attachment step.
9+
Tests deployment sanity (replica availability & uninterrupted service) and recovery workflows of the application pod.
10+
11+
:::tip Fault execution flow chart
12+
![EBS Loss By ID](./static/images/ebs-loss.png)
13+
:::
14+
15+
## Uses
16+
17+
<details>
18+
<summary>View the uses of the experiment</summary>
19+
<div>
20+
Coming soon.
21+
</div>
22+
</details>
23+
24+
## Prerequisites
25+
26+
:::info
27+
- Ensure that Kubernetes Version > 1.16.
28+
- Ensure that you have sufficient AWS access to attach or detach an ebs volume for the instance.
29+
- Ensure to create a Kubernetes secret having the AWS access configuration(key) in the `CHAOS_NAMESPACE`. A sample secret file looks like:
30+
```yaml
31+
apiVersion: v1
32+
kind: Secret
33+
metadata:
34+
name: cloud-secret
35+
type: Opaque
36+
stringData:
37+
cloud_config.yml: |-
38+
# Add the cloud AWS credentials respectively
39+
[default]
40+
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
41+
aws_secret_access_key = XXXXXXXXXXXXXXX
42+
```
43+
- If you change the secret key name (from `cloud_config.yml`) please also update the `AWS_SHARED_CREDENTIALS_FILE` ENV value on `experiment.yaml`with the same name.
44+
:::
45+
46+
## Default Validations
47+
48+
:::info
49+
- EBS volume is attached to the instance.
50+
:::
51+
52+
## Experiment tunables
53+
54+
<details>
55+
<summary>Check the Experiment Tunables</summary>
56+
<h2>Mandatory Fields</h2>
57+
<table>
58+
<tr>
59+
<th> Variables </th>
60+
<th> Description </th>
61+
<th> Notes </th>
62+
</tr>
63+
<tr>
64+
<td> EBS_VOLUME_ID </td>
65+
<td> Comma separated list of volume IDs subjected to ebs detach chaos</td>
66+
<td> Eg. ebs-vol-1,ebs-vol-2 </td>
67+
</tr>
68+
<tr>
69+
<td> REGION </td>
70+
<td> The region name for the target volumes</td>
71+
<td> Eg. us-east-1 </td>
72+
</tr>
73+
</table>
74+
<h2>Optional Fields</h2>
75+
<table>
76+
<tr>
77+
<th> Variables </th>
78+
<th> Description </th>
79+
<th> Notes </th>
80+
</tr>
81+
<tr>
82+
<td> TOTAL_CHAOS_DURATION </td>
83+
<td> The time duration for chaos insertion (sec) </td>
84+
<td> Defaults to 30s </td>
85+
</tr>
86+
<tr>
87+
<td> CHAOS_INTERVAL </td>
88+
<td> The time duration between the attachment and detachment of the volumes (sec) </td>
89+
<td> Defaults to 30s </td>
90+
</tr>
91+
<tr>
92+
<td> SEQUENCE </td>
93+
<td> It defines sequence of chaos execution for multiple volumes</td>
94+
<td> Default value: parallel. Supported: serial, parallel </td>
95+
</tr>
96+
<tr>
97+
<td> RAMP_TIME </td>
98+
<td> Period to wait before and after injection of chaos in sec </td>
99+
<td> Eg: 30 </td>
100+
</tr>
101+
</table>
102+
</details>
103+
104+
## Experiment Examples
105+
106+
### Common and AWS specific tunables
107+
108+
Refer the [common attributes](../common-tunables-for-all-experiments) and [AWS specific tunable](./aws-experiments-tunables) to tune the common tunables for all experiments and aws specific tunables.
109+
110+
### Detach Volumes By ID
111+
112+
It contains comma separated list of volume IDs subjected to ebs detach chaos. It can be tuned via `EBS_VOLUME_ID` ENV.
113+
114+
Use the following example to tune this:
115+
116+
[embedmd]:# (./static/manifests/ebs-loss-by-id/ebs-volume-id.yaml yaml)
117+
```yaml
118+
# contains ebs volume id
119+
apiVersion: litmuschaos.io/v1alpha1
120+
kind: ChaosEngine
121+
metadata:
122+
name: engine-nginx
123+
spec:
124+
engineState: "active"
125+
chaosServiceAccount: litmus-admin
126+
experiments:
127+
- name: ebs-loss-by-id
128+
spec:
129+
components:
130+
env:
131+
# id of the ebs volume
132+
- name: EBS_VOLUME_ID
133+
value: 'ebs-vol-1'
134+
# region for the ebs volume
135+
- name: REGION
136+
value: 'us-east-1'
137+
- name: TOTAL_CHAOS_DURATION
138+
VALUE: '60'
139+
```

‎docs/chaos-engineering/chaos-faults/aws/ebs-loss-by-tag.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
---
2+
id: ebs-loss-by-tag
3+
title: EBS Loss By Tag
4+
---
5+
6+
## Introduction
7+
- It causes chaos to disrupt state of ebs volume by detaching it from the node/ec2 instance for a certain chaos duration using volume tags.
8+
- In case of EBS persistent volumes, the volumes can get self-attached and experiment skips the re-attachment step.
9+
Tests deployment sanity (replica availability & uninterrupted service) and recovery workflows of the application pod.
10+
11+
:::tip Fault execution flow chart
12+
![EBS Loss By Tag](./static/images/ebs-loss.png)
13+
:::
14+
15+
## Uses
16+
17+
<details>
18+
<summary>View the uses of the experiment</summary>
19+
<div>
20+
Coming soon.
21+
</div>
22+
</details>
23+
24+
## Prerequisites
25+
26+
:::info
27+
28+
- Ensure that Kubernetes Version > 1.16.
29+
- Ensure that you have sufficient AWS access to attach or detach an ebs volume for the instance.
30+
- Ensure to create a Kubernetes secret having the AWS access configuration(key) in the `CHAOS_NAMESPACE`. A sample secret file looks like:
31+
```yaml
32+
apiVersion: v1
33+
kind: Secret
34+
metadata:
35+
name: cloud-secret
36+
type: Opaque
37+
stringData:
38+
cloud_config.yml: |-
39+
# Add the cloud AWS credentials respectively
40+
[default]
41+
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
42+
aws_secret_access_key = XXXXXXXXXXXXXXX
43+
```
44+
- If you change the secret key name (from `cloud_config.yml`) please also update the `AWS_SHARED_CREDENTIALS_FILE` ENV value on `experiment.yaml`with the same name.
45+
:::
46+
47+
## Default Validations
48+
49+
:::info
50+
51+
- EBS volume is attached to the instance.
52+
53+
:::
54+
55+
## Experiment tunables
56+
57+
<details>
58+
<summary>Check the Experiment Tunables</summary>
59+
<h2>Mandatory Fields</h2>
60+
<table>
61+
<tr>
62+
<th> Variables </th>
63+
<th> Description </th>
64+
<th> Notes </th>
65+
</tr>
66+
<tr>
67+
<td> EBS_VOLUME_TAG </td>
68+
<td> Provide the common tag for target volumes. It'll be in form of <code>key:value</code> (Ex: 'team:devops')</td>
69+
<td> </td>
70+
</tr>
71+
<tr>
72+
<td> REGION </td>
73+
<td> The region name for the target volumes</td>
74+
<td> Eg. us-east-1 </td>
75+
</tr>
76+
</table>
77+
<h2>Optional Fields</h2>
78+
<table>
79+
<tr>
80+
<th> Variables </th>
81+
<th> Description </th>
82+
<th> Notes </th>
83+
</tr>
84+
<tr>
85+
<td> VOLUME_AFFECTED_PERC </td>
86+
<td> The Percentage of total ebs volumes to target </td>
87+
<td> Defaults to 0 (corresponds to 1 volume), provide numeric value only </td>
88+
</tr>
89+
<tr>
90+
<td> TOTAL_CHAOS_DURATION </td>
91+
<td> The time duration for chaos insertion (sec) </td>
92+
<td> Defaults to 30s </td>
93+
</tr>
94+
<tr>
95+
<td> CHAOS_INTERVAL </td>
96+
<td> The time duration between the attachment and detachment of the volumes (sec) </td>
97+
<td> Defaults to 30s </td>
98+
</tr>
99+
<tr>
100+
<td> SEQUENCE </td>
101+
<td> It defines sequence of chaos execution for multiple volumes</td>
102+
<td> Default value: parallel. Supported: serial, parallel </td>
103+
</tr>
104+
<tr>
105+
<td> RAMP_TIME </td>
106+
<td> Period to wait before and after injection of chaos in sec </td>
107+
<td> Eg: 30 </td>
108+
</tr>
109+
</table>
110+
</details>
111+
112+
## Experiment Examples
113+
114+
### Common and AWS specific tunables
115+
116+
Refer the [common attributes](../common-tunables-for-all-experiments) and [AWS specific tunable](./aws-experiments-tunables) to tune the common tunables for all experiments and aws specific tunables.
117+
118+
### Target single volume
119+
120+
It will detach a random single ebs volume with the given `EBS_VOLUME_TAG` tag and `REGION` region.
121+
122+
Use the following example to tune this:
123+
124+
[embedmd]:# (./static/manifests/ebs-loss-by-tag/ebs-volume-tag.yaml yaml)
125+
```yaml
126+
# contains the tags for the ebs volumes
127+
apiVersion: litmuschaos.io/v1alpha1
128+
kind: ChaosEngine
129+
metadata:
130+
name: engine-nginx
131+
spec:
132+
engineState: "active"
133+
chaosServiceAccount: litmus-admin
134+
experiments:
135+
- name: ebs-loss-by-tag
136+
spec:
137+
components:
138+
env:
139+
# tag of the ebs volume
140+
- name: EBS_VOLUME_TAG
141+
value: 'key:value'
142+
# region for the ebs volume
143+
- name: REGION
144+
value: 'us-east-1'
145+
- name: TOTAL_CHAOS_DURATION
146+
VALUE: '60'
147+
```
148+
149+
### Target Percent of volumes
150+
151+
It will detach the `VOLUME_AFFECTED_PERC` percentage of ebs volumes with the given `EBS_VOLUME_TAG` tag and `REGION` region.
152+
153+
Use the following example to tune this:
154+
155+
[embedmd]:# (./static/manifests/ebs-loss-by-tag/volume-affected-percentage.yaml yaml)
156+
```yaml
157+
# target percentage of the ebs volumes with the provided tag
158+
apiVersion: litmuschaos.io/v1alpha1
159+
kind: ChaosEngine
160+
metadata:
161+
name: engine-nginx
162+
spec:
163+
engineState: "active"
164+
chaosServiceAccount: litmus-admin
165+
experiments:
166+
- name: ebs-loss-by-tag
167+
spec:
168+
components:
169+
env:
170+
# percentage of ebs volumes filter by tag
171+
- name: VOLUME_AFFECTED_PERC
172+
value: '100'
173+
# tag of the ebs volume
174+
- name: EBS_VOLUME_TAG
175+
value: 'key:value'
176+
# region for the ebs volume
177+
- name: REGION
178+
value: 'us-east-1'
179+
- name: TOTAL_CHAOS_DURATION
180+
VALUE: '60'
181+
```

0 commit comments

Comments
 (0)