Refactor background process metrics to be homeserver-scoped #18670

MadLittleMods · 2025-07-07T23:24:55Z

Refactor background process metrics to be homeserver-scoped

Separated out of #18656 because it's a bigger, unique piece of the refactor

Testing strategy

Add the metrics listener in your homeserver.yaml

listeners:
  # This is just showing how to configure metrics either way
  #
  # `http` `metrics` resource
  - port: 9322
    type: http
    bind_addresses: ['127.0.0.1']
    resources:
      - names: [metrics]
        compress: false
  # `metrics` listener
  - port: 9323
    type: metrics
    bind_addresses: ['127.0.0.1']

Start the homeserver: poetry run synapse_homeserver --config-path homeserver.yaml
Fetch http://localhost:9322/_synapse/metrics and/or http://localhost:9323/metrics
Observe response includes the background processs metrics (synapse_background_process_start_count, synapse_background_process_db_txn_count_total, etc) with the server_name label

Dev notes

synapse/metrics/background_process_metrics.py

SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.util.test_task_scheduler.TestTaskSchedulerWithBackgroundWorker.test_schedule_task

Log context docs: https://github.com/element-hq/synapse/blob/56f5097d1c8de968b845eda9e2abd5207ee76825/docs/log_contexts.md

Todo

Wait for Refactor cache metrics to be homeserver-scoped #18604 to merge
Fill in wrap_as_background_process

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

TODO: still need to fill in `run_as_background_process` usage

…ground_process`)

…trics-counter-server-name-label

``` SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.test_server.OptionsResourceTests.test_known_request The "poetry.dev-dependencies" section is deprecated and will be removed in a future version. Use "poetry.group.dev.dependencies" instead. tests.test_server OptionsResourceTests test_known_request ... [ERROR] =============================================================================== [ERROR] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/unittest.py", line 131, in new return code(orig, *args, **kwargs) File "/home/eric/Documents/github/element/synapse/tests/unittest.py", line 225, in setUp return orig() File "/home/eric/Documents/github/element/synapse/tests/test_server.py", line 219, in setUp self.homeserver = setup_test_homeserver( File "/home/eric/Documents/github/element/synapse/tests/server.py", line 1147, in setup_test_homeserver hs.setup() File "synapse/server.py", line 361, in setup self.datastores = Databases(self.DATASTORE_CLASS, self) File "synapse/storage/databases/__init__.py", line 104, in __init__ main = main_store_class(database, db_conn, hs) File "synapse/storage/databases/main/__init__.py", line 174, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/events_bg_updates.py", line 122, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/experimental_features.py", line 44, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/devices.py", line 1776, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/devices.py", line 95, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/roommember.py", line 2030, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/room.py", line 2449, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/room.py", line 2023, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/room.py", line 152, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/registration.py", line 2519, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/registration.py", line 2407, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/profile.py", line 53, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/presence.py", line 80, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/presence.py", line 61, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/transactions.py", line 81, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/state.py", line 974, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/state.py", line 118, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/state.py", line 765, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/event_federation.py", line 2094, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/media_repository.py", line 226, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/media_repository.py", line 119, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/filtering.py", line 49, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/event_push_actions.py", line 1894, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/metrics.py", line 75, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/event_push_actions.py", line 263, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/end_to_end_keys.py", line 1551, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/e2e_room_keys.py", line 69, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/search.py", line 436, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/search.py", line 145, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/account_data.py", line 68, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/push_rule.py", line 138, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/appservice.py", line 103, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/pusher.py", line 82, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/roommember.py", line 102, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/pusher.py", line 579, in __init__ super().__init__(database, db_conn, hs) File "synapse/storage/databases/main/receipts.py", line 142, in __init__ server_name=self.server_name, builtins.AttributeError: 'DataStore' object has no attribute 'server_name' tests.test_server.OptionsResourceTests.test_known_request ------------------------------------------------------------------------------- Ran 1 tests in 0.094s FAILED (errors=1) ```

…in `synapse/storage/databases/main/lock.py`

…in `synapse/util/caches/lrucache.py`

… of accepting `self.server_name`

MadLittleMods · 2025-07-17T01:10:56Z

synapse/app/_base.py

@@ -543,16 +544,24 @@ async def start(hs: "HomeServer") -> None:
    # Set up the SIGHUP machinery.
    if hasattr(signal, "SIGHUP"):

-        @wrap_as_background_process("sighup")


Because @wrap_as_background_process was updated to take self.server_name, we've had to refactor a few of the calls that don't use self.

Should be equivalent to before.

MadLittleMods · 2025-07-17T01:13:02Z

synapse/metrics/background_process_metrics.py

-    def wrap_as_background_process_inner(
-        func: Callable[P, Awaitable[Optional[R]]],
+    def wrapper(
+        func: Callable[Concatenate[HasServerName, P], Awaitable[Optional[R]]],


This HasServerName pattern matches what we do for @measure_func

We want to use self.server_name for the functions that people are decorating @wrap_as_background_process with.

Seems like a reasonable precedence.

…bscriptable` in with old deps ``` File "/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/synapse/util/caches/lrucache.py", line 128, in <module> ) -> defer.Deferred[None]: TypeError: 'type' object is not subscriptable ```

MadLittleMods · 2025-07-17T02:45:11Z

synapse/replication/tcp/redis.py

@@ -119,13 +124,15 @@ def __init__(self, *args: Any, **kwargs: Any):
            # capture the sentinel context as its containing context and won't prevent
            # GC of / unintentionally reactivate what would be the current context.
            self._logging_context = BackgroundProcessLoggingContext(
-                "replication_command_handler"
+                name="replication_command_handler", server_name=self.server_name


Error:

File "/home/runner/work/synapse/synapse/synapse/replication/tcp/redis.py", line 127, in __init__ name="replication_command_handler", server_name=self.server_name builtins.AttributeError: 'RedisSubscriber' object has no attribute 'server_name'

self.server_name is not set at this point since we set this after it's created, see

synapse/synapse/replication/tcp/redis.py

Lines 383 to 391 in 0610810

def buildProtocol(self, addr: IAddress) -> RedisSubscriber:

p = super().buildProtocol(addr)

p = cast(RedisSubscriber, p)

# We do this here rather than add to the constructor of `RedisSubcriber`

# as to do so would involve overriding `buildProtocol` entirely, however

# the base method does some other things than just instantiating the

# protocol.

p.server_name = self.server_name

I think I fixed this in f0ee141 but I'm not totally confident in this. Please double check.

For reference, we do have some logging context docs.

…trics-counter-server-name-label Conflicts: synapse/handlers/device.py synapse/storage/databases/main/end_to_end_keys.py

See #18670 (comment) Error: ``` File "/home/runner/work/synapse/synapse/synapse/replication/tcp/redis.py", line 127, in __init__ name="replication_command_handler", server_name=self.server_name builtins.AttributeError: 'RedisSubscriber' object has no attribute 'server_name' ``` `self.server_name` is not set in the constructor since we set this after it's created, see https://github.com/element-hq/synapse/blob/06108100c848ab3f6b9e541b7769bbc63adc2bf6/synapse/replication/tcp/redis.py#L383-L391

MadLittleMods · 2025-07-18T22:39:55Z

synapse/replication/tcp/redis.py

+    def _get_logging_context(self) -> BackgroundProcessLoggingContext:
+        """
+        We lazily create the logging context so that `self.server_name` is set and
+        available. See `RedisDirectTcpReplicationClientFactory.buildProtocol` for more
+        details on why we set `self.server_name` after the fact instead of in the
+        constructor.
+        """


Please double-check all of this logic. See #18670 (comment)

devonh

A small comment about the logging context changes.
Otherwise all these changes look appropriate to me.

Thank you for also cleaning up some of the unrelated doc strings along the way.

devonh · 2025-07-22T19:44:58Z

synapse/handlers/appservice.py

+        self.server_name = (
+            hs.hostname
+        )  # nb must be called this for @wrap_as_background_process


What does this comment refer to?

Oh, is this saying that the self.server_name variable must have that particular name in order to be picked up by the @wrap_as_background_process decorator?

Correct 👍

(this is the same language we use for @cached as well)

synapse/storage/databases/main/receipts.py

devonh · 2025-07-22T20:45:02Z

synapse/metrics/background_process_metrics.py

-    def wrap_as_background_process_inner(
-        func: Callable[P, Awaitable[Optional[R]]],
+    def wrapper(
+        func: Callable[Concatenate[HasServerName, P], Awaitable[Optional[R]]],


Seems like a reasonable precedence.

synapse/replication/tcp/redis.py

…trics-counter-server-name-label

See #18670 (comment)

…Generator`

…trics-counter-server-name-label

MadLittleMods · 2025-07-23T18:28:42Z

Thanks for the review @devonh 🐇

MadLittleMods added 2 commits July 7, 2025 18:22

Fill in synapse/metrics/background_process_metrics.py

a92f8d6

TODO: still need to fill in `run_as_background_process` usage

Fill in synapse/metrics/background_process_metrics.py (`run_as_back…

d40c244

…ground_process`)

MadLittleMods added the A-Metrics label Jul 7, 2025

This was referenced Jul 7, 2025

Refactor Counter metrics to be homeserver-scoped #18656

Open

Refactor metrics to be scoped to the homeserver #18592

Open

MadLittleMods added 16 commits July 9, 2025 16:19

Merge branch 'develop' into madlittlemods/18592-background-process-me…

cebaa2e

…trics-counter-server-name-label

Merge branch 'develop' into madlittlemods/18592-background-process-me…

03b7d9b

…trics-counter-server-name-label

Fill in get_retry_limiter

6c091b1

Fill in some missing self.server_name that are part of #18656

72b8815

Fill in synapse/replication/tcp/protocol.py

52301fe

Fixup BatchingQueue

013f607

Fixup MultiWriterIdGenerator

2306f63

Restore self.store in ApplicationServiceScheduler (still being used)

545f674

Add changelog

b758bf6

Convert @wrap_as_background_process -> run_as_background_process …

5538178

…in `synapse/storage/databases/main/lock.py`

Convert @wrap_as_background_process -> run_as_background_process …

b574a29

…in `synapse/util/caches/lrucache.py`

Update @wrap_as_background_process to match @measure_func pattern…

b6d65bf

… of accepting `self.server_name`

Bulk fill in self.server_name for @wrap_as_background_process

7aace77

Fire and forget generate_monthly_active_users

dba6d75

Fix fire and forget conversions

56b470b

MadLittleMods commented Jul 17, 2025

View reviewed changes

MadLittleMods added 3 commits July 16, 2025 20:21

Fix type and align with fire and forget pattern

0610810

Remove debug log

8830c21

MadLittleMods commented Jul 17, 2025

View reviewed changes

MadLittleMods added 2 commits July 18, 2025 17:05

Merge branch 'develop' into madlittlemods/18592-background-process-me…

2dddd06

…trics-counter-server-name-label Conflicts: synapse/handlers/device.py synapse/storage/databases/main/end_to_end_keys.py

MadLittleMods commented Jul 18, 2025

View reviewed changes

MadLittleMods marked this pull request as ready for review July 19, 2025 02:29

MadLittleMods requested a review from a team as a code owner July 19, 2025 02:29

MadLittleMods requested review from devonh and removed request for a team July 22, 2025 15:10

devonh reviewed Jul 22, 2025

View reviewed changes

MadLittleMods added 3 commits July 22, 2025 18:30

Merge branch 'develop' into madlittlemods/18592-background-process-me…

775503a

…trics-counter-server-name-label

No need to store the sentinel context

b512c37

See #18670 (comment)

Fill in server_name for the new thread subscriptions `MultiWriterId…

6b77f63

…Generator`

MadLittleMods requested a review from devonh July 22, 2025 23:55

devonh approved these changes Jul 23, 2025

View reviewed changes

Merge branch 'develop' into madlittlemods/18592-background-process-me…

0a8a6bb

…trics-counter-server-name-label

MadLittleMods merged commit b7e7f53 into develop Jul 23, 2025
74 of 78 checks passed

MadLittleMods deleted the madlittlemods/18592-background-process-metrics-counter-server-name-label branch July 23, 2025 18:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor background process metrics to be homeserver-scoped #18670

Refactor background process metrics to be homeserver-scoped #18670

Uh oh!

MadLittleMods commented Jul 7, 2025 •

edited

Loading

Uh oh!

MadLittleMods Jul 17, 2025

Uh oh!

MadLittleMods Jul 17, 2025

Uh oh!

devonh Jul 22, 2025

Uh oh!

MadLittleMods Jul 17, 2025

Uh oh!

MadLittleMods Jul 18, 2025

Uh oh!

MadLittleMods Jul 18, 2025

Uh oh!

devonh left a comment

Uh oh!

devonh Jul 22, 2025

Uh oh!

devonh Jul 22, 2025

Uh oh!

MadLittleMods Jul 22, 2025

Uh oh!

Uh oh!

devonh Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MadLittleMods commented Jul 23, 2025

Uh oh!

Uh oh!

	def buildProtocol(self, addr: IAddress) -> RedisSubscriber:
	p = super().buildProtocol(addr)
	p = cast(RedisSubscriber, p)

	# We do this here rather than add to the constructor of `RedisSubcriber`
	# as to do so would involve overriding `buildProtocol` entirely, however
	# the base method does some other things than just instantiating the
	# protocol.
	p.server_name = self.server_name

Refactor background process metrics to be homeserver-scoped #18670

Refactor background process metrics to be homeserver-scoped #18670

Uh oh!

Conversation

MadLittleMods commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing strategy

Dev notes

Todo

Pull Request Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devonh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MadLittleMods commented Jul 23, 2025

Uh oh!

Uh oh!

MadLittleMods commented Jul 7, 2025 •

edited

Loading