[VC-43753] CyberArk Discovery and Context: Upload data in the JSON format required by the API #684
Conversation
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
…unmarshaled Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
There was a problem hiding this comment.
I deleted this file because it was causing import cycle.
$ go test -v ./pkg/datagatherer/...
# github.com/jetstack/preflight/pkg/datagatherer/k8s
package github.com/jetstack/preflight/pkg/datagatherer/k8s
imports github.com/jetstack/preflight/pkg/testutil from fieldfilter_test.go
imports github.com/jetstack/preflight/pkg/client from envtest.go
imports github.com/jetstack/preflight/pkg/internal/cyberark/dataupload from client_cyberark.go
imports github.com/jetstack/preflight/pkg/datagatherer/k8s from dataupload.go: import cycle not allowed in test
FAIL github.com/jetstack/preflight/pkg/datagatherer/k8s [setup failed]
? github.com/jetstack/preflight/pkg/datagatherer [no test files]
? github.com/jetstack/preflight/pkg/datagatherer/local [no test files]
FAIL
| Resource interface{} | ||
| DeletedAt Time | ||
| Resource interface{} `json:"resource"` | ||
| DeletedAt Time `json:"deleted_at,omitempty"` | ||
| } |
There was a problem hiding this comment.
Added json annotations here so that I can unmarshal date readings from a file, for testing.
The agent already has an --input-file option, but stops decoding the input at api.DataReading.Data, leaving the actual data as interface{}.
In the test in this PR I need to decode the Data, so that it has the same types as the DataGatherer.Fetch return values.
| type DiscoveryData struct { | ||
| ServerVersion *version.Info `json:"server_version"` | ||
| } | ||
|
|
There was a problem hiding this comment.
I created a data type instead of the ad-hoc map that was previously returned by this data gatherer,
to make it easier to do a type conversion in ConverDatareadingsToCyberarkSnapshot function.
| @@ -307,14 +307,17 @@ func (g *DataGathererDynamic) WaitForCacheSync(ctx context.Context) error { | |||
| return nil | |||
| } | |||
|
|
|||
| type DynamicData struct { | |||
| Items []*api.GatheredResource `json:"items"` | |||
| } | |||
There was a problem hiding this comment.
I created a data type instead of the ad-hoc map that was previously returned by this data gatherer,
to make it easier to do a type conversion in ConverDatareadingsToCyberarkSnapshot function.
| @@ -16,6 +16,9 @@ var SecretSelectedFields = []FieldPath{ | |||
| {"metadata", "ownerReferences"}, | |||
| {"metadata", "selfLink"}, | |||
| {"metadata", "uid"}, | |||
| {"metadata", "creationTimestamp"}, | |||
| {"metadata", "deletionTimestamp"}, | |||
| {"metadata", "resourceVersion"}, | |||
There was a problem hiding this comment.
The Cyberark backend needs this extra metadata to produce its reports.
I think it makes sense to upload this metadata for TLSPK too.
I can't see any harm in it and it will be difficult to implement different field filters for the TLSPK vs CyberArk uploaded data.
| @@ -41,10 +42,7 @@ func (mds *mockDataUploadServer) ServeHTTP(w http.ResponseWriter, r *http.Reques | |||
| mds.handlePresignedUpload(w, r) | |||
| return | |||
| case "/presigned-upload": | |||
| mds.handleUpload(w, r, false) | |||
| return | |||
| case "/presigned-upload-invalid-json": | |||
There was a problem hiding this comment.
This mock endpoint was unused, and was passing exactly the same arguments to handleUpload.
Probably a legacy of a previous version of these tests.
So I deleted it.
| _, err := w.Write([]byte(`{"message":"method not allowed"}`)) | ||
| if err != nil { | ||
| panic(err) | ||
| } |
There was a problem hiding this comment.
I added some extra error checking here and below. These errors should normally be nil
| w.WriteHeader(http.StatusOK) | ||
| w.Header().Set("Content-Type", "application/json") | ||
| _, _ = w.Write([]byte(`{"url":`)) // invalid JSON | ||
| return |
There was a problem hiding this comment.
This branch wasn't used.
The AWS presigned URL endpoint only returns JSON for errors. Not for StatusOK.
| w.WriteHeader(http.StatusOK) | ||
| w.Header().Set("Content-Type", "application/json") | ||
| _, _ = w.Write([]byte(`{"success":true}`)) |
There was a problem hiding this comment.
The AWS endpoint returns an empty response body on success.
| d.DisallowUnknownFields() | ||
| if err := d.Decode(&snapshot); err != nil { | ||
| panic(err) | ||
| } |
There was a problem hiding this comment.
Verifies that the new Snapshot format is used in the request body.
Fixes: https://venafi.atlassian.net/browse/VC-43753
The client still deviates from the API requirements, as follows:
...but I prefer to address those in followup PRs, as time permits.
Testing
I ran the Cyberark dataupload client Integration test manually (it's not yet automated):
This PR touches some code shared with the TLSPK agent uploads so I ran the e2e test script manually and verified that the test certificate was uploaded and processed by the TLSPK backend:
$ ./hack/e2e/test.sh ... { "ts": 1754748136841.2498, "caller": "transport/round_trippers.go:632", "msg": "Response", "v": 6, "logger": "Run.gatherAndOutputData", "verb": "POST", "url": "https://api.venafi.cloud/v1/tlspk/upload/clusterdata/no?description=QSBraW5kIGNsdXN0ZXIgdXNlZCBmb3IgdGVzdGluZyB0aGUgdmVuYWZpLWt1YmVybmV0ZXMtYWdlbnQuCg&name=venafi-kubernetes-agent-e2e", "status": "200 OK", "milliseconds": 338 } {"ts":1754748136906.9675,"caller":"agent/run.go:460","msg":"Data sent successfully","v":0,"logger":"Run.gatherAndOutputData.postData"} ... { "count": 1, "certificates": [ { "id": "e1d70740-7529-11f0-a38a-f7545647ce40", "companyId": "756db001-280e-11ee-84fb-991f3177e2d0", "managedCertificateId": "e2cbc780-7529-11f0-995f-751da7434dbb", "fingerprint": "9FF2FE5A78E1E495465F5E4704361BBD8C304926", "certificateName": "venafi-kubernetes-agent-e2e.188a1c0f-15ae-4bd4-b4d3-4eaa36eec5a1", ... } + exit 0