Skip to content

feat: add run_in_parallel method and tests#256

Open
LucasAlvws wants to merge 17 commits intoautoscrape-labs:mainfrom
LucasAlvws:issue-224
Open

feat: add run_in_parallel method and tests#256
LucasAlvws wants to merge 17 commits intoautoscrape-labs:mainfrom
LucasAlvws:issue-224

Conversation

@LucasAlvws
Copy link
Collaborator

@LucasAlvws LucasAlvws commented Aug 29, 2025

Pull Request

Description

Related Issue(s)

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes, no API changes)
  • Performance improvement
  • Tests (adding missing tests or correcting existing tests)
  • Build or CI/CD related changes

How Has This Been Tested?

# Include code examples if relevant

Testing Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • All existing tests pass

Screenshots

Implementation Details

API Changes

Additional Info

Checklist before requesting a review

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have run poetry run task lint and fixed any issues
  • I have run poetry run task test and all tests pass
  • My commits follow the conventional commits style

@LucasAlvws LucasAlvws self-assigned this Aug 29, 2025
@codecov
Copy link

codecov bot commented Aug 29, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@LucasAlvws LucasAlvws marked this pull request as ready for review November 29, 2025 18:48
Comment on lines +547 to +554
if self.options.max_parallel_tasks and self._semaphore is None:
if self._semaphore_lock is None:
self._semaphore_lock = asyncio.Lock()

async with self._semaphore_lock:
# Double-check pattern to avoid creating multiple semaphores
if self._semaphore is None:
self._semaphore = asyncio.Semaphore(self.options.max_parallel_tasks)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A property would be better for this logic:

    @property
    def _semaphore(self) -> asyncio.Semaphore | None:
        if not self.options.max_parallel_tasks:
            return None
        
        # this is lazy
        if self._sem_instance is None:
            self._sem_instance = asyncio.Semaphore(self.options.max_parallel_tasks)
        
        return self._sem_instance

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a cached property is also really great for this case:

    @cached_property
    def _semaphore(self) -> asyncio.Semaphore | None:
        if not self.options.max_parallel_tasks:
            return None
        return asyncio.Semaphore(self.options.max_parallel_tasks)

if self._semaphore_lock is None:
self._semaphore_lock = asyncio.Lock()

async with self._semaphore_lock:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this lock is unecessary since the creation of the semaphore is sync. The GIL will take care of it, so you can remove all the logic for the lock

async def run_gather():
return await asyncio.gather(*wrapped, return_exceptions=False)

# Check if we're in the same event loop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove most of the comments, try to keep only the really necessary ones.
we can add more information about this in the documentation

logger.info(f'Resetting permissions (context={browser_context_id})')
return await self._execute_command(BrowserCommands.reset_permissions(browser_context_id))

async def run_in_parallel(self, *coroutines: Coroutine[Any, Any, Any]) -> list[Any]:
Copy link
Member

@thalissonvs thalissonvs Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the typing, a generic would be great:

T = TypeVar("T")

async def run_in_parallel(self, *coroutines: Coroutine[Any, Any, T]) -> list[T]: ...

that way, the return type of the coroutines will be reflected directly in the return type of the method:

async def get_number() -> int: ...
async def get_text() -> str: ...

# here, T will be resolved as Union[int, str]
results = await browser.run_in_parallel(get_number(), get_text())

# results will be resolved to list[str | int]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so i can just add T = TypeVar("T") in the TYPE_CHECKING nested imports?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or should i add it just before the class Browser like the logger = logging.getLogger(name)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, but i think it can be under the if TYPE_CHECKING, just remember to import annotations:

from future import __annotations__

return ws

@cached_property
def _semaphore(self) -> asyncio.Semaphore | None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I usually like to put the properties in the top of the class, can you do that?

Copy link
Member

@thalissonvs thalissonvs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some logs would also be great, try to keep most of them as debug

@thalissonvs thalissonvs force-pushed the main branch 4 times, most recently from d2c8b63 to 0ba0fa7 Compare December 26, 2025 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants