plugin/decision: upload events as soon as buffer limit is reached

[sspaink]

Why the changes in this PR are needed?

resolve: #7455

What are the changes in this PR?

This introduces a new trigger mode:

decision_logs.reporting.trigger=immediate

This trigger mode will upload events either if the buffer limit is reached or after the maximum delay. Both the size and event buffer type supports this trigger mode. For the event buffer that means if the buffered channel is full, for the size buffer that means if the chunk buffer is full.

Using this trigger mode disables configuring the decision_logs.reporting.min_delay_seconds and only the decision_logs.reporting.max_delay_seconds can be changed. If using the size buffer type it also requires setting the decision_logs.reporting.buffer_size_limit_bytes, otherwise it defaults to unlimited and could cause confusion because then the uploads will only trigger on decision_logs.reporting.max_delay_seconds.

Example

Take for example this contrived scenario to illustrate the benefit of the immediate trigger.

Given the following configuration with a small size limit and large delay.

decision_logs:
  service: logeater
  reporting:
    buffer_type: event
    trigger: periodic
    buffer_size_limit_events: 10
    min_delay_seconds: 10
    max_delay_seconds: 20

Perhaps this config works well for most of the time, but then during peak traffic created using the vegeta cli:

echo 'POST http://localhost:8181/v1/data/example/allow' | vegeta attack --duration=10s -rate=500 | tee results.bin | vegeta report

Checking the metrics you see there is a ridiculous number of recorded dropped metrics, only 10 events weren't dropped (vegeta is sending 500 requests per second for 10 seconds) and those 10 events were probably still in the buffer:
"counter_decision_logs_dropped_buffer_size_limit_exceeded": 4990

Now updating the configuration to use the immediate trigger mode, keeping the same small buffer size and max delay:

decision_logs:
  service: logeater
  reporting:
    buffer_type: event
    trigger: immediate
    buffer_size_limit_events: 10
    max_delay_seconds: 20

Attacking it with the same vegeta configuration... There isn't a single dropped log! 🥳 Of course there are pros/cons but that could make a huge difference if logs are vital and the user has the resources to keep up with the uploads.

In this example the log eater service is a (simple go server to eat the logs )

Notes to assist PR review:

Updated the upload loop function to support immediate uploads. Also added a new loop function when setting manual trigger mode or if service is not configured. This helps separate the logic making it easier to read and moves the decision making during reconfiguration only and not while the loop is running. So this is why the loopType logic is there.

[netlify]

✅ Deploy Preview for openpolicyagent ready!

Name	Link
🔨 Latest commit	ea8b0af
🔍 Latest deploy log	https://app.netlify.com/projects/openpolicyagent/deploys/68e9298a88cf090008b4cf06
😎 Deploy Preview	https://deploy-preview-7516--openpolicyagent.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[sspaink]

Keeping this in draft until #7521 is merged, some overlap so rather fix any merge conflicts before this is reviewed.

[stale]

This pull request has been automatically marked as stale because it has not had any activity in the last 30 days.

[sspaink]

@johanfylling finally ready for review 🥳

[johanfylling]

Thanks!
A couple of thoughts.

[johanfylling]

If we're in immediate mode, and we still have a margin until the upload limit, I'd not expect us to be dropping events, but instead upload the current buffer and start a new buffer with the event that didn't fit. Is that happening, but just not reflected here?

[johanfylling]

Could we have dropped events on the above push, e.g. from the front of the size buffer at this point?

[johanfylling]

Should we also assert that the second event hasn't been dropped?

[sspaink]

Good idea, although at the moment there isn't a guarantee the event won't drop.

Like we discussed worth looking into a way to guarantee no events are dropped. Perhaps by flushing the buffer when full like you suggested. Will loop back to this 👍

[johanfylling]

Could we have dropped an event here when running in immediate mode?