Use PySNMP >6 asyncio API by Emantor · Pull Request #1497 · labgrid-project/labgrid

Emantor · 2024-09-24T15:02:07Z

Description
Update pysnmp and it's users to use the new async API, this should fix CI and also ensures we can properly use pysnmp in the future.
Pushed additional fixes for no longer required pylint disables.

Checklist

PR has been tested

Fixes #1493
Closes #1456

Bastian-Krause · 2024-09-25T13:35:38Z

labgrid/util/snmp.py

+        except RuntimeError:
+            # no previous, external or running loop found, create a new one
+            self.loop = asyncio.new_event_loop()


Client code does the same, but only in start_session() when no loop is provided otherwise (and stashes the loop for later use).

How would other code retrieve the newly created event loop from here?

I think this must rely on an existing event loop, similar to EthernetPortManager:

labgrid/labgrid/resource/ethernetport.py

Line 251 in 6b54121

self.loop = asyncio.get_event_loop()

But get_event_loop is deprecated, so I chose not to use this.
I also think that get_running_loop almost always returns a runtime error here.

No, get_event_loop() is not deprecated per se: only if there is no current event loop. That's why I think we should simply expect a loop here. If there is none, that's an error that needs to be fixed outside of labgrid.

Using labgrid as a library means that there is not necessarily an available loop and we need to support this (which is exactly the case where there is no loop).

Testing this inside of labgrid client reveals that get_running_loop does not find the running client event loop and instead no loop is found and another is created for SNMP.

Using labgrid as a library means that there is not necessarily an available loop and we need to support this (which is exactly the case where there is no loop).

I disagree. There are certain "entry points" into labgrid where a loop can be passed or labgrid tries to find one or sets a new one up as a fallback. Exporter, coordinator and client are these "entry points". Client and libraries typically use labgrid.remote.client.start_session() to start a session. So that makes sure there is a loop.

For all other cases, code that uses labgrid as a library needs to make sure a loop exists. Maybe that needs to be documented better.

Using get_event_loop() in EthernetPortManager was an active decision. Using that makes sure there is an event loop. Otherwise you get a warning (and a new event loop) or an error, depending on the Python version.

Testing this inside of labgrid client reveals that get_running_loop does not find the running client event loop and instead no loop is found and another is created for SNMP.

That's because get_running_loop() does not work in sync contexts and there is no other way to achieve that.

I guess this is fine for the SNMP switch handling, but isn't the correct requirement for the simple power backend handling we had beforehand. So how do we proceed with this? Requiring an event loop for sync library usage is not really feasible.

How does the simple power backend handling work without a session?

How does the simple power backend handling work without a session?

What do you mean? You can totally describe all of the NetworkPowerPorts inside of the YAML configuration and not use a RemotePlace to retrieve the resources.

Yes, I see.

See commit messages of da95c44, 0e1e4fb and 606af35 for context.

What about calling labgrid.remote.client.ensure_event_loop() whenever we need an event loop? That would restore the original behavior of using labgrid as a library without a coordinator session.

We could add a function that allows library users to pass their own event loop into labgrid (stashing it) for use cases without labgrid.remote.client.start_session() (which already allows passing an external loop).

labgrid/util/snmp.py

pysnmp depends on pyasn1. `pyasn1.compat.octets` was removed in pyasn1 0.6.1 [1] leading to ModuleNotFoundErrors in labgrid's "eaton" and "poe_mib" power backends: _______________ TestNetworkPowerDriver.test_import_backend_eaton _______________ self = <test_powerdriver.TestNetworkPowerDriver object at 0x7f84794bfdd0> def test_import_backend_eaton(self): pytest.importorskip("pysnmp") > import labgrid.driver.power.eaton tests/test_powerdriver.py:295: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ labgrid/driver/power/eaton.py:2: in <module> from ...util.snmp import SimpleSNMP labgrid/util/snmp.py:1: in <module> from pysnmp import hlapi /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/hlapi/__init__.py:7: in <module> from pysnmp.proto.rfc1902 import * /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1902.py:8: in <module> from pysnmp.proto import rfc1155, error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1155.py:10: in <module> from pysnmp.proto import error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/error.py:9: in <module> from pysnmp import debug _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ # # This file is part of pysnmp software. # # Copyright (c) 2005-2019, Ilya Etingof <etingof@gmail.com> # License: https://www.pysnmp.com/pysnmp/license.html # import logging > from pyasn1.compat.octets import octs2ints E ModuleNotFoundError: No module named 'pyasn1.compat.octets' /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/debug.py:8: ModuleNotFoundError ______________ TestNetworkPowerDriver.test_import_backend_poe_mib ______________ self = <test_powerdriver.TestNetworkPowerDriver object at 0x7f84794bd910> def test_import_backend_poe_mib(self): pytest.importorskip("pysnmp") > import labgrid.driver.power.poe_mib tests/test_powerdriver.py:307: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ labgrid/driver/power/poe_mib.py:4: in <module> from ...util.snmp import SimpleSNMP labgrid/util/snmp.py:1: in <module> from pysnmp import hlapi /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/hlapi/__init__.py:7: in <module> from pysnmp.proto.rfc1902 import * /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1902.py:8: in <module> from pysnmp.proto import rfc1155, error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1155.py:10: in <module> from pysnmp.proto import error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/error.py:9: in <module> from pysnmp import debug _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ # # This file is part of pysnmp software. # # Copyright (c) 2005-2019, Ilya Etingof <etingof@gmail.com> # License: https://www.pysnmp.com/pysnmp/license.html # import logging > from pyasn1.compat.octets import octs2ints E ModuleNotFoundError: No module named 'pyasn1.compat.octets' /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/debug.py:8: ModuleNotFoundError The issue is documented upstream [2]. [3] limited the pysnmp version to <6. pysnmp 6.1.4, 6.2.6, and 7.1.0 are not affected. For stable, limit compatible pyasn1 versions to <0.6.1. On master, [5] switches labgrid to pysnmp's asyncio API, thereby dropping the upper bound introduced by [3]. [1] https://github.com/pyasn1/pyasn1/releases/tag/v0.6.1 [2] pyasn1/pyasn1#76 [3] labgrid-project#1332 [4] lextudio/pysnmp#113 (comment) [5] labgrid-project#1497 Signed-off-by: Bastian Krause <bst@pengutronix.de>

pysnmp depends on pyasn1. `pyasn1.compat.octets` was removed in pyasn1 0.6.1 [1] leading to ModuleNotFoundErrors in labgrid's "eaton" and "poe_mib" power backends: _______________ TestNetworkPowerDriver.test_import_backend_eaton _______________ self = <test_powerdriver.TestNetworkPowerDriver object at 0x7f84794bfdd0> def test_import_backend_eaton(self): pytest.importorskip("pysnmp") > import labgrid.driver.power.eaton tests/test_powerdriver.py:295: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ labgrid/driver/power/eaton.py:2: in <module> from ...util.snmp import SimpleSNMP labgrid/util/snmp.py:1: in <module> from pysnmp import hlapi /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/hlapi/__init__.py:7: in <module> from pysnmp.proto.rfc1902 import * /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1902.py:8: in <module> from pysnmp.proto import rfc1155, error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1155.py:10: in <module> from pysnmp.proto import error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/error.py:9: in <module> from pysnmp import debug _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ # # This file is part of pysnmp software. # # Copyright (c) 2005-2019, Ilya Etingof <etingof@gmail.com> # License: https://www.pysnmp.com/pysnmp/license.html # import logging > from pyasn1.compat.octets import octs2ints E ModuleNotFoundError: No module named 'pyasn1.compat.octets' /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/debug.py:8: ModuleNotFoundError ______________ TestNetworkPowerDriver.test_import_backend_poe_mib ______________ self = <test_powerdriver.TestNetworkPowerDriver object at 0x7f84794bd910> def test_import_backend_poe_mib(self): pytest.importorskip("pysnmp") > import labgrid.driver.power.poe_mib tests/test_powerdriver.py:307: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ labgrid/driver/power/poe_mib.py:4: in <module> from ...util.snmp import SimpleSNMP labgrid/util/snmp.py:1: in <module> from pysnmp import hlapi /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/hlapi/__init__.py:7: in <module> from pysnmp.proto.rfc1902 import * /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1902.py:8: in <module> from pysnmp.proto import rfc1155, error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/rfc1155.py:10: in <module> from pysnmp.proto import error /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/proto/error.py:9: in <module> from pysnmp import debug _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ # # This file is part of pysnmp software. # # Copyright (c) 2005-2019, Ilya Etingof <etingof@gmail.com> # License: https://www.pysnmp.com/pysnmp/license.html # import logging > from pyasn1.compat.octets import octs2ints E ModuleNotFoundError: No module named 'pyasn1.compat.octets' /opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/pysnmp/debug.py:8: ModuleNotFoundError The issue is documented upstream [2]. [3] limited the pysnmp version to <6. pysnmp 6.1.4, 6.2.6, and 7.1.0 are not affected. Limit compatible pyasn1 versions to <0.6.1 until [5] switches labgrid to pysnmp's asyncio API, thereby dropping the upper bound introduced by [3]. While at it, switch from "pysnmp-lextudio" to "pysnmp". The original author of pysnmp passed away and the lextudio folks took over maintenanc. While the request to take over the pysnmp PyPi project was pending, the maintained fork was called pysnmp-lextudio (see labgrid-project#1186, aa2549c). Now that the migration is complete, let's move back to the original package name. See: [6] [1] https://github.com/pyasn1/pyasn1/releases/tag/v0.6.1 [2] pyasn1/pyasn1#76 [3] labgrid-project#1332 [4] lextudio/pysnmp#113 (comment) [5] labgrid-project#1497 [6] etingof/pysnmp#429 Signed-off-by: Bastian Krause <bst@pengutronix.de>

Bastian-Krause · 2025-09-22T16:34:19Z

Rebased on latest master.

Bastian-Krause · 2025-09-22T16:41:59Z

Reworked:

move ensure_event_loop() to labgrid.util.loop for usage outside client code
add is_new_loop() to check whether ensure_event_loop() created a new loop
use ensure_event_loop() and is_new_loop() in labgrid.util.snmp and labgrid.resource.ethernetport

@Emantor What do you think of this?

Moving the function and the ContextVar to labgrid.util.loop will allow non-client code to use it as well. Otherwise usage will result in circular imports such as: from labgrid import Target labgrid/__init__.py:1: in <module> from .target import Target labgrid/target.py:10: in <module> from .driver import Driver labgrid/driver/__init__.py:4: in <module> from .serialdriver import SerialDriver labgrid/driver/serialdriver.py:10: in <module> from ..util.proxy import proxymanager labgrid/util/proxy.py:7: in <module> from ..resource.common import Resource labgrid/resource/__init__.py:2: in <module> from .ethernetport import SNMPEthernetPort labgrid/resource/ethernetport.py:9: in <module> from ..remote.client import ensure_event_loop labgrid/remote/client.py:41: in <module> from .. import Environment, Target, target_factory E ImportError: cannot import name 'Environment' from partially initialized module 'labgrid' (most likely due to a circular import) (labgrid/__init__.py) Signed-off-by: Bastian Krause <bst@pengutronix.de>

There is currently no way to find out whether ensure_event_loop() created a new loop. This information is useful in scenarios where a resource is created without a session in a sync context, because then it needs to shut down the newly created loop in clean up code. But this must not happen if a running loop is used. Since ensure_event_loop() is already public API people use, add a separate function checking whether the passed in loop is identical to the loop stashed in the loop creation code in ensure_event_loop(). Signed-off-by: Bastian Krause <bst@pengutronix.de>

Wrap the now asyncio based co-routines in loop.run_until_complete(). Signed-off-by: Rouven Czerwinski <r.czerwinski@pengutronix.de> Signed-off-by: Bastian Krause <bst@pengutronix.de>

With the new asyncio based handling we can unlock the pysnmp dependency. The lexstudio fork has also taken over maintenance of pysnmp on pypi, so switch back to pysnmp. Signed-off-by: Rouven Czerwinski <r.czerwinski@pengutronix.de>

Follow the sync API deprecation and use ensure_event_loop() making sure we have an asyncio loop even if no loop can be retrieved. Signed-off-by: Rouven Czerwinski <r.czerwinski@pengutronix.de> Signed-off-by: Bastian Krause <bst@pengutronix.de>

codecov · 2026-02-18T09:13:32Z

Codecov Report

❌ Patch coverage is 41.93548% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.2%. Comparing base (b94e172) to head (d6fb35c).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
labgrid/util/snmp.py	18.7%	13 Missing ⚠️
labgrid/resource/ethernetport.py	21.4%	11 Missing ⚠️
labgrid/util/loop.py	76.0%	6 Missing ⚠️
labgrid/driver/power/eaton.py	0.0%	2 Missing ⚠️
labgrid/driver/power/poe_mib.py	0.0%	2 Missing ⚠️
labgrid/driver/power/raritan.py	0.0%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #1497     +/-   ##
========================================
- Coverage    45.2%   45.2%   -0.1%     
========================================
  Files         174     175      +1     
  Lines       13752   13777     +25     
========================================
+ Hits         6224    6234     +10     
- Misses       7528    7543     +15

Flag	Coverage Δ
3.10	`45.2% <41.9%> (-0.1%)`	⬇️
3.11	`45.2% <41.9%> (-0.1%)`	⬇️
3.12	`45.2% <41.9%> (-0.1%)`	⬇️
3.13	`45.2% <41.9%> (-0.1%)`	⬇️
3.14	`45.2% <41.9%> (-0.1%)`	⬇️
3.9	`45.2% <41.9%> (-0.1%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Emantor force-pushed the topic/pysnmp-bump branch 4 times, most recently from 612455d to 413aa59 Compare September 25, 2024 06:07

Emantor requested review from Bastian-Krause and jluebbe September 25, 2024 06:11

Emantor force-pushed the topic/pysnmp-bump branch from 3f13ddf to 5bfe8b6 Compare September 25, 2024 06:14

Bastian-Krause reviewed Sep 25, 2024

View reviewed changes

labgrid/util/snmp.py Outdated Show resolved Hide resolved

Emantor force-pushed the topic/pysnmp-bump branch from 5bfe8b6 to cdf59f1 Compare September 25, 2024 13:41

This was referenced Sep 30, 2024

Test/Advertise Python 3.13 Support #1511

Merged

github: bump ubuntu runner images to 24.04 #1510

Merged

Bastian-Krause mentioned this pull request Oct 10, 2024

treewide: drop pylint disable=no-self-use comments #1521

Merged

1 task

Bastian-Krause mentioned this pull request Oct 10, 2024

pyproject.toml: limit pysnmp's pyasn1 dependency to <0.6.0 #1522

Merged

1 task

Bastian-Krause force-pushed the topic/pysnmp-bump branch from cdf59f1 to 871f484 Compare September 22, 2025 16:34

Bastian-Krause force-pushed the topic/pysnmp-bump branch from 871f484 to 9c7cb23 Compare September 22, 2025 16:36

Bastian-Krause assigned Emantor Sep 22, 2025

Bastian-Krause and others added 5 commits February 18, 2026 10:07

util: convert SimpleSNMP to asyncio

988b960

Wrap the now asyncio based co-routines in loop.run_until_complete(). Signed-off-by: Rouven Czerwinski <r.czerwinski@pengutronix.de> Signed-off-by: Bastian Krause <bst@pengutronix.de>

pyproject: allow pysnmp newer than 6.x

6eec3d0

With the new asyncio based handling we can unlock the pysnmp dependency. The lexstudio fork has also taken over maintenance of pysnmp on pypi, so switch back to pysnmp. Signed-off-by: Rouven Czerwinski <r.czerwinski@pengutronix.de>

resouce: ethernetport: convert to asyncio

d6fb35c

Follow the sync API deprecation and use ensure_event_loop() making sure we have an asyncio loop even if no loop can be retrieved. Signed-off-by: Rouven Czerwinski <r.czerwinski@pengutronix.de> Signed-off-by: Bastian Krause <bst@pengutronix.de>

Emantor force-pushed the topic/pysnmp-bump branch from 9c7cb23 to d6fb35c Compare February 18, 2026 09:07

Uh oh!

Comments

Conversation

Emantor commented Sep 24, 2024 • edited by Bastian-Krause Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Emantor Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bastian-Krause Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Bastian-Krause commented Sep 22, 2025

Uh oh!

Bastian-Krause commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Emantor commented Sep 24, 2024 •

edited by Bastian-Krause

Loading

Emantor Sep 25, 2024 •

edited

Loading

Bastian-Krause Sep 26, 2024 •

edited

Loading

Bastian-Krause commented Sep 22, 2025 •

edited

Loading

codecov bot commented Feb 18, 2026 •

edited

Loading