RBAC Manager not logging errors or early termination

[tklovett]

We've now run into two failure scenarios during rbac-manager's Reconciliation run where it failed to log anything relevant to:

• an error occurred
• a resource could not be created
• following resources in the same RBACDefinition were skipped
• reconciliation terminated, and terminated early.

For example, the log output would end with

$ kubectl logs -n rbac-manager rbac-manager-7b499c48b6-tjzj8
time="2019-08-05T19:37:37Z" level=info msg=----------------------------------
time="2019-08-05T19:37:37Z" level=info msg="rbac-manager 0.7.0 running"
time="2019-08-05T19:37:37Z" level=info msg=----------------------------------
...
time="2019-08-05T19:54:18Z" level=info msg="Reconciling RBACDefinition my-rbac-definition"
$ kubectl ...

And no reference to error scenarios, for example:

• Kubernetes RBAC "Privilege Escalation Prevention" forbids rbac-manager from binding roles when rbac-manager itself does not have the permissions in said role.
  • This case is user error, but took several days to track down due to the lack of warning or error logs from rbac-manager
  • We encountered this when attempting to bind a create-namespace role, but had not granted that permission to rbac-manager.
• rbac-manager fails because no namespaces exist which match a provided namespaceSelector.matchLabels (see Rbac-manager doesn't handle the case where a namespace does not exist and is later created #53)

My go fu is limited, but I think the Reconciler itself may have and return a useful err here, but the wrapping code does nothing with it here. Though I'm surprised I was seeing no error output from this code...

[sudermanjr]

I think we should go ahead and move to the klog package and add logging levels with a lot more output at the higher levels. A general pass over the logging could be really good here.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.