[Feature Request] Multiple worker running on different datacenters

[aldy505]

Is your feature request related to a problem? Please describe.

It would be great if we can have each uptime job to have check reports coming from multiple (or different) datacenters in the world. Therefore if something is inaccessible from a certain country, we can easily trace it.

This is also useful for internal monitoring from a private networks that have multiple VPC (or multiple private network groups), you can find out which network is having issues.

Describe the solution you'd like

This is the current architecture diagram:

It would be great if the worker have some more modes:

• direct executor, like current implementation
• relay, exposes API endpoints for other worker to register themself and submits the check reports
• indirect executor (the leaf of the node), receives API request from relay and executes the healthcheck. reports back to relay through an API endpoint.

This assumes redis is not accessible on the leaf worker nodes, because it's on a different network. Which I believe it shouldn't connect to redis anyway if it's on a different datacenter.

It might be possible for producer to do this, but I don't think it will be scalable.

Describe alternatives you've considered

None.

Additional context

None.

[green5-zero]

Another major use case of this is if one of your nodes is being smashed/down etc, you're not going to get false notifications.

There should be some way to decide "when" to send notifications, such as saying inaccessible from "node1" + "node2 = notification, or being able to just have notifications for one instance detecting downtime. The status page should also be able to display reachability for each service from each status checking node.

That would put it on par with closed source paid commercial options, and be pretty much a first in the foss space for an uptime service.

[zitrusgelb]

This somewhat works with a Kubernetes deployment, although I think every task is executed by one worker and then handled by the Ingester. I had an issue with the API though sending the notifications on status changes multiple times, see: #241