Skip to content

Fix multiprocessing logging with non-fork start methods - #107

Open
CyberAuth wants to merge 1 commit into
blacklanternsecurity:masterfrom
CyberAuth:fix/python314-console-logging
Open

Fix multiprocessing logging with non-fork start methods#107
CyberAuth wants to merge 1 commit into
blacklanternsecurity:masterfrom
CyberAuth:fix/python314-console-logging

Conversation

@CyberAuth

Copy link
Copy Markdown

Summary

Fixes a Python 3.14 multiprocessing compatibility issue that could prevent MANSPIDER console logging from functioning and could raise a weakref pickling error.

Root cause

The logging queue was created during module import. Under spawn and forkserver, child processes re-imported the module and created private queues without active listeners, so file logging continued while console records were lost.

MANSPIDER also passed its complete controller to spiderling processes. With multiple concurrent targets, the controller process pool contained an already-started Process. Its process finalizer contains a weak reference and cannot be serialized.

Parser subprocesses similarly received a bound Spiderling method, which pulled unnecessary live runtime state into the serialized object graph.

Fix

  • Create the logging queue explicitly from the active multiprocessing context.
  • Pass the queue to workers and configure logging within each process.
  • Exclude parent-only process and SMB cache state from controller serialization.
  • Pass parser workers only the picklable values required for their task.
  • Close the listener and queue in a defined shutdown order.
  • Preserve the process-based concurrency model without forcing fork.

Testing

  • Python 3.13
  • Python 3.14
  • Existing pytest suite
  • Regression coverage under fork, forkserver, and spawn
  • Multi-target process communication
  • Child-process console and file logging
  • Ruff lint and formatting checks

Related work

This is a serialization-safe alternative to #96 and #106. It preserves the default multiprocessing context and the existing process-based worker architecture.

Security / data note

The reproduction and regression tests use synthetic data only and do not require access to external SMB infrastructure.

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@CyberAuth

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

bls-cla-bot Bot added a commit to blacklanternsecurity/CLA that referenced this pull request Aug 20, 2026
@CyberAuth

Copy link
Copy Markdown
Author

recheck

@CyberAuth
CyberAuth marked this pull request as ready for review August 20, 2026 18:04

@singlerider singlerider left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid fix with a clear explanation of the root cause. The serialization-safe approach and test coverage across all three start methods are exactly right. Minor nit: parse_file is now a no-op wrapper that could be inlined, and the bare except: blocks in manspider.py could be more specific. Otherwise, approved.

singlerider

This comment was marked as duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants