tf-job-operator CrashLoopBackOff

[toyow]

`{"filename":"app/server.go:75","level":"info","msg":"EnvKubeflowNamespace not set, use default namespace","time":"2020-02-24T06:12:54Z"}
{"filename":"app/server.go:79","level":"info","msg":"Using cluster scoped operator","time":"2020-02-24T06:12:54Z"}
{"filename":"app/server.go:85","level":"info","msg":"[API Version: v1 Version: v0.1.0-alpha Git SHA: Not provided. Go Version: go1.12 Go OS/Arch: linux/amd64]","time":"2020-02-24T06:12:54Z"}
{"filename":"tf-operator.v1/main.go:40","level":"info","msg":"Setting up client for monitoring on port: 8443","time":"2020-02-24T06:12:54Z"}
W0224 06:12:54.926349 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
{"filename":"tensorflow/controller.go:123","level":"info","msg":"Creating TFJob controller","time":"2020-02-24T06:12:54Z"}
{"filename":"tensorflow/controller.go:130","level":"info","msg":"Creating Job controller","time":"2020-02-24T06:12:54Z"}
I0224 06:12:54.972835 1 leaderelection.go:185] attempting to acquire leader lease default/tf-operator...
E0224 06:12:54.982793 1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation/validation.go:96
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation/validation.go:79
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow/informer.go:99
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow/job.go:36
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/controller.go:195
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:554
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x115d5e8]

goroutine 146 [running]:
github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x12e8880, 0x2317ec0)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation.validateV1ReplicaSpecs(0xc0000c34a0, 0x147a19f, 0x5)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation/validation.go:96 +0x208
github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation.ValidateV1TFJobSpec(...)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation/validation.go:79
github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow.tfJobFromUnstructured(0x1471920, 0xc00087c088, 0x0, 0xc0007f8180, 0x0)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow/informer.go:99 +0x1a1
github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow.(*TFController).addTFJob(0xc0001ee900, 0x1471920, 0xc00087c088)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow/job.go:36 +0x50
github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/controller.go:195
github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0x42d26d, 0xc00054ade8, 0x11934d1)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:554 +0x26d
github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0xc0007bde38, 0x42cdcf, 0xc0008140d0)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203 +0xde
github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:548 +0x89
github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00054af68)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007bdf68, 0xdf8475800, 0x0, 0x1640c01, 0xc00081a060)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8
github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc000768980)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:546 +0x9c
github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0004a8bc0, 0xc000856010)
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/mnt/test-data-volume/tf-operator-release-d746bde9-kunming/go/src/github.com/kubeflow/tf-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62`

[cheyang]

Could you please let us know how to reproduce it? Thanks.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.