Merge pull request #271 from f5devcentral/develop

push to master

Author: VDI-Tech-Guy

docs/class10/_static/c_back_to_steady_state_2.png

@@ -23,13 +23,13 @@ BIG-IP integrates with MinIO health endpoints to monitor quorum readiness. With
 - **Automatically restore writes** once quorum returns.
 - **Outcome**: AI workloads remain consistent and responsive even under node failures.
 
-Task 1. Validate healthy write quorum
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Task 1. Validate healthy write quorum  Using Lab AIStor Cluster 2
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 In **BIG-IP TMUI**:
 
 - Navigate to Local **Traffic → Pools → cluster1-write-quorum**.
-- Confirm all 4 members are **green**.
+- Confirm all 4 members are **green**.  Change algorithm to "Least Connections (member) and click **Update**
 
 |lab400|
 
@@ -54,39 +54,42 @@ Task 2.  Run baseline workload (repesenting typical read/write load)
 
 Open the MinIO Warp bench tool (**UDF -> Components -> Traffic-Gen -> Access -> Firefox**)
 
-- Select the target: **BigIP-cluster-1 (healthcheck + quorum)**
-- Select **all** three buckets.
+- Select the target: **BigIP-cluster-2 (healthcheck + quorum)**
+- Select **only** cluster2-bucket-a
 - Put sliders on **Duration** of 10 mins and **Concurrency** at 20 threads
 - Make sure that the IP address in WARP Parameters is a new BIG-IP virtual server at **10.1.40.162:9000**
 
 |lab402|
 
 Click the **Run Benchmark** button to start a long, full ten minutes of high rate S3 load.
+The simplest way to reach the following screen in TMUI, is Local Traffic -> Pools -> Pool List and click on **cluster2-write-quorum**.
+Now click on the **Statistics** tab in upper right of screen.
 
 |lab403|
 
-We observe all members of the pool cluster-1-write-quorum are getting close to the same number of total HTTP (S3) requests.
+We observe all members of the pool cluster-2-write-quorum are getting close to the same number of total HTTP (S3) requests.
 
 Task 3.  Disable one node
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-In UDF, open **UDF -> Components -> Jump Host → Access → Web Shell** (be careful not to inadvertently use WIN-JUMP-HOST).
+In UDF, open **UDF -> Components -> Jump Host → Access → Web Shell**.
 
 - Check the active user: #whoami
 - If it returns **root**, switch to user ubuntu: #su - ubuntu
 
 |lab404|
 
 - Change to /home/ubuntu/minio directory
-- run the ansible playbook $ansible-playbook cluster1-stop-one-node.yml
+- Run the ansible playbook $ansible-playbook cluster2-stop-one-node.yml (**double-click image to enlarge**)
+
 
 |lab405|
 
 
 
 
 
-In the BIG-IP Pool being used, called cluster1-write-quorum, click on the **Members / Statistics** tab and observe 1 marked **red**, this is the expected behavor.
+In the BIG-IP Pool being used, called cluster2-write-quorum, click on the **Members / Statistics** tab and observe 1 marked **red**, this is the expected behavor.
 
 |lab406|
 
@@ -106,6 +109,8 @@ the entire pool will be taken out because the healthcheck no longer returns 200
 
 BIG-IP marks the entire pool as **red**.
 
+**The key point** :  two nodes are still up and running, but they show as red and **not** available in the context of this pool as the "write quorum" health check fails.   The health check, targetting a specific MinIO URL, is failing due to the total number of nodes falling below the threshold of healthy nodes.
+
 |lab408|
 
 Notice that all nodes are down, however a few TCP connections remain active.  No new S3 traffic will be proxied to these nodes by the corresponding
@@ -116,12 +121,12 @@ Task 5.  Read-only cluster & verification of failover
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 An F5 iRule or policy could be configured to shift traffic from a pool that is no longer available to another. In
-our configuration, the cluster1-write-quorum automatically fails over to the cluster1-read-quorum pool.  The iRule used can be seen on the Resources
+our configuration, the cluster2-write-quorum automatically fails over to the cluster2-read-quorum pool.  The iRule used can be seen on the Resources
 tab of the virtual server named **minio-cluster-healthcheck**.
 
 Let's look at the pool that the iRule will now be directing S3 traffic towards.
 
-In **BIG-IP TMUI** open (Traffic -> Pools -> Pool List -> *cluster1-read-quorum* -> Members)
+In **BIG-IP TMUI** open (Traffic -> Pools -> Pool List -> *cluster2-read-quorum* -> Members)
 
 Two nodes are shown as down (nodes 2 and 4), however there are **two healthy nodes** (nodes 1 and 3), which is sufficient to satisfy the
 read quorum, hence the pool can still operate and fully accept read operations.
@@ -132,7 +137,7 @@ We see in the following screen, the two healthy nodes continue to handle transac
 
 Open UDF -> AST -> Access -> Grafana; Select **Device Pools**.
 
-Enlarge the Active Pool Connections chart, and select **only** pools cluster1-write-quarum and cluster1-read-quarum.
+Enlarge the Active Pool Connections chart, and select **only** pools cluster2-write-quarum and cluster2-read-quarum.
 
 If the WARP ten minute load generator was active when the ansible disater simulation playbook ran, taking down two nodes, one will be able
 to see this moment.
@@ -165,7 +170,7 @@ In the following, one can see the original Ansible disaster simulation script be
 |lab411|
 
 **Expectation:**  Without any operator intervention, or requirements on the part of S3 client configuration, the entire S3 storage solution has recovered.
-Traffic destined for the **write-quarum pool** has automatically resumed handling reads and writes.
+Traffic destined for the **write-quarum pool** has automatically resumed successfully handling reads and writes.
 
 
 Troubleshooting
@@ -228,25 +233,25 @@ What You Learned - BIG-IP and AIStor Impact
    :width: 800px 
 .. |lab313| image:: ../_static/lab3-appworld2025-waf-block-message.png
    :width: 800px
-.. |lab400| image:: ../_static/c_write_quorom_pool.png
+.. |lab400| image:: ../_static/cluster_2_write_quorum.png
    :width: 800px
 .. |lab401| image:: ../_static/c_health_monitor.png
    :width: 800px
-.. |lab402| image:: ../_static/c_warp_params.png
+.. |lab402| image:: ../_static/warp_clsuter2_write_quorum.png
    :width: 800px
-.. |lab403| image:: ../_static/c_traffic_to_all_nodes.png
+.. |lab403| image:: ../_static/cluster_2_even_traffic_write_quorum.png
    :width: 800px
 .. |lab404| image:: ../_static/c_connect_to_linux_host.png
    :width: 800px
-.. |lab405| image:: ../_static/c_ansible_take_one_node_down_2.png
+.. |lab405| image:: ../_static/cluster_2_updated_ansible_playbook.png
    :width: 800px
-.. |lab406| image:: ../_static/c_one_node_down.png
+.. |lab406| image:: ../_static/cluster_2_one_node_failed.png
    :width: 800px
 .. |lab407| image:: ../_static/c_take_down_second_node.png
    :width: 800px
-.. |lab408| image:: ../_static/c_all_nodes_down.png
+.. |lab408| image:: ../_static/cluster2_2failed_nodes.png
    :width: 800px
-.. |lab409| image:: ../_static/c_2_healthy_nodes.png
+.. |lab409| image:: ../_static/cluster_2_read_quorum_still_okay.png
    :width: 800px
 .. |lab410| image:: ../_static/c_trans_write_to_read_quorum.png
    :width: 800px