Spike/argocd demo

[maltesander]

ArgoCD demo depolying stackable components and running a spark job via airflow.

What it does:

• Stack ArgoCD

  • sealed secrets controller (https://github.com/bitnami-labs/sealed-secrets)
  • deploys all operators (except secret, listener and commons which are currently installed via stackablectl, since there must be operators installed in a stack) via applicationset
  • generated tls certs for sealed secrets

• Demo based on the airflow demo, deployed via ArgoCD

  • deploy minio for airflow kubernetes executor logs
  • gitsync to fetch dags
  • automatically deploy minio connection to airflow
  • all secrets sealed
  • deploy spark job via UI

Clone the demos repo and checkout the branch spike/argocd-demo and run

stackablectl demo install argo-cd-git-ops -n argo-cd -s stacks/stacks-v2.yaml -d demos/demos-v2.yaml

or for custom repo to sync changes via argo (demo fork)

stackablectl demo install argo-cd-git-ops -n argo-cd -s stacks/stacks-v2.yaml -d demos/demos-v2.yaml --parameters customGitUrl=https://github.com/maltesander/demos --parameters customGitBranch=spike/argocd-demo

I currently port forward the argocd-server and airflow webserver. In the airflow webUI you can start the pyspark job manually.

Feedback would be very welcome :)

Edit - Notes:

• Airflow does not offer https for the web ui currently. Thats the only part not properly encrypted in this demo.

[adwk67]

Nice job! Mainly just minor text formatting stuff and a few suggestions.

[adwk67]

This needs to be removed for the demo to run on Openshift. See stackabletech/nifi-operator#827
Ditto for runAsUSer.

Suggested change

runAsGroup: 1000

[maltesander]

done 03292c2

[adwk67]

runAsUser needs to be removed, too. The demos are not primarily tested on Openshift, so maybe this point is moot. But maybe what is more relevant is to have a standard way of setting up Minio - the templated way is probably the way to go and so it would be good to have something we can copy/use between demos and tests (hence the suggested change).