OpenShift 4.15: kubectl mayastor get nodes Service 'app=api-rest' not found

[alonsocamaro]

Hi all

I have followed the steps indicated in https://openebs.io/docs/Solutioning/openebs-on-kubernetes-platforms/openshift. All but the callhome PODs are fine (this is expected because of firewall rules). I used this helm command which completed succesfully:

helm install openebs --namespace openebs openebs/openebs --create-namespace --set openebs-crds.csi.volumeSnapshots.enabled=false --set engines.local.zfs.enabled=false

But when I check the installation it fails with any command:

$ kubectl mayastor get nodes
Service 'app=api-rest' not found
$ kubectl mayastor get block-devices worker  -n openebs
Failed to list blockdevices for node worker. Error error in response: status code '404 Not Found', content: 'RestJsonError { details: "", message: "SvcError :: NodeNotFound: Node 'worker' not found", kind: NotFound }'
$ oc get dsp -n openebs
No resources found in openebs namespace.

I have labeled 3 nodes with kubectl label node $node openebs.io/engine=mayastor. Other nodes are automatically labeled with other openebs.io labels

Any clue what might be happening?

Many thanks

   Ulises

Full POD output next

$ oc -n openebs get pods
NAME                                              READY   STATUS             RESTARTS         AGE
openebs-agent-core-84788486c-94dt4                2/2     Running            0                71m
openebs-agent-ha-node-5kfrk                       1/1     Running            0                71m
openebs-agent-ha-node-7xpqg                       1/1     Running            0                71m
openebs-agent-ha-node-fjbbd                       1/1     Running            0                71m
openebs-agent-ha-node-g9l87                       1/1     Running            0                71m
openebs-agent-ha-node-m56x6                       1/1     Running            0                71m
openebs-agent-ha-node-qjwrk                       1/1     Running            0                71m
openebs-api-rest-7776fbcf4f-25v65                 1/1     Running            0                71m
openebs-csi-controller-6d47d4548-n4w7h            6/6     Running            0                71m
openebs-csi-node-2tlg9                            2/2     Running            0                71m
openebs-csi-node-588cb                            2/2     Running            0                71m
openebs-csi-node-bw58s                            2/2     Running            0                71m
openebs-csi-node-dxgmx                            2/2     Running            0                71m
openebs-csi-node-k4qkn                            2/2     Running            0                71m
openebs-csi-node-s4twp                            2/2     Running            0                71m
openebs-etcd-0                                    1/1     Running            0                71m
openebs-etcd-1                                    1/1     Running            0                71m
openebs-etcd-2                                    1/1     Running            0                71m
openebs-io-engine-ddc5d                           0/2     Pending            0                71m
openebs-io-engine-t2t9m                           0/2     Pending            0                71m
openebs-io-engine-t5bvv                           0/2     Pending            0                71m
openebs-localpv-provisioner-7d89ff787d-hz9wk      1/1     Running            0                71m
openebs-loki-0                                    1/1     Running            0                71m
openebs-lvm-localpv-controller-76d8c8f5bf-55vhf   5/5     Running            0                71m
openebs-lvm-localpv-node-4bh2v                    2/2     Running            0                71m
openebs-lvm-localpv-node-7k4rh                    2/2     Running            0                71m
openebs-lvm-localpv-node-g6g96                    2/2     Running            0                71m
openebs-lvm-localpv-node-v2575                    2/2     Running            0                71m
openebs-lvm-localpv-node-vst8w                    2/2     Running            0                71m
openebs-lvm-localpv-node-xs9x2                    2/2     Running            0                71m
openebs-nats-0                                    3/3     Running            0                71m
openebs-nats-1                                    3/3     Running            0                71m
openebs-nats-2                                    3/3     Running            0                71m
openebs-obs-callhome-66559f87c9-vvcnx             1/2     CrashLoopBackOff   16 (3m14s ago)   71m
openebs-operator-diskpool-6f49996679-l5d42        1/1     Running            0                71m
openebs-promtail-fzwgd                            1/1     Running            0                71m
openebs-promtail-h9ddm                            1/1     Running            0                71m
openebs-promtail-kqjvd                            1/1     Running            0                71m
openebs-promtail-kqn6m                            1/1     Running            0                71m
openebs-promtail-njrgz                            1/1     Running            0                71m
openebs-promtail-wjrcs                            1/1     Running            0                71m

[tiagolobocastro]

Yes, seems that worker node's io-engine is pending.

openebs-io-engine-ddc5d                           0/2     Pending            0                71m
openebs-io-engine-t2t9m                           0/2     Pending            0                71m
openebs-io-engine-t5bvv                           0/2     Pending            0                71m

You can kubectl describe to find out why.
(Btw you can disable callhome if you want to avoid the crash loop as I'm assuming it probably can't reach the target from your internal network)

[alonsocamaro]

Thanks! I overlooked those Pending pods

checking...

[alonsocamaro]

Turns out unexpectedly the labeled nodes didn´t have the expected hugepages, now I´m still getting problem with the io-engine because it doesn´t find the required pages:

$ oc -n openebs logs openebs-io-engine-xkxzr -c io-engine
[2025-03-14T08:11:55.006064284+00:00  INFO io_engine:io-engine.rs:253] Engine responsible for managing I/Os version 1.0.0, revision bcb7fbe89043 (v2.8.0+0)
[2025-03-14T08:11:55.006221778+00:00 ERROR io_engine:io-engine.rs:209] insufficient pages available PAGES_NEEDED=1024 nr_pages=772 nr_1g_pages=0

yet it looks fine to me

oc get node sjc-bd-host-104 -o jsonpath="{.status.allocatable.hugepages-2Mi}"
3584Mi

It´s funny I´ve got this in /proc/meminfo

HugePages_Total:     772
HugePages_Free:      772
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Even in my kernel I see this in dmesg (via Tuned resource):

[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-28af98682992630498bb0eb62633902be1a9fbd025565b98a6cb9270233aafe8/vmlinuz-5.14.0-284.104.1.el9_2.x86_64 ignition.platform.id=metal ostree=/ostree/boot.0/rhcos/28af98682992630498bb0eb62633902be1a9fbd025565b98a6cb9270233aafe8/0 root=UUID=014ba306-18c5-4ce8-b21f-ce4b63ba50d8 rw rootflags=prjquota boot=UUID=78998c89-b8bf-4774-bc87-50d152620bd4 systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all psi=1 hugepagesz=2M hugepages=1792
[...]
[    0.521331] HugeTLB registered 2.00 MiB page size, pre-allocated 1792 pages
[    0.521456] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages

This different issue now seems something specific to this deployment. I have configured hugepages before in OpenShift and never saw this...

[tiagolobocastro]

I was about to ask if you had other pods consuming the hugepages but given your /proc/meminfo I'm now stumped :)