Skip to content

perf: Prune function bodies from source before building AST - #489

Open
pawamoy wants to merge 5 commits into
mainfrom
perf/rust-based-ast-parser
Open

perf: Prune function bodies from source before building AST#489
pawamoy wants to merge 5 commits into
mainfrom
perf/rust-based-ast-parser

Conversation

@pawamoy

@pawamoy pawamoy commented Sep 11, 2026

Copy link
Copy Markdown
Member

We use a Rust/Ruff-based lexer to prune function bodies in cases where we're sure we won't visit them. This makes compile() much faster since it has much less code to parse and compile into an AST. Line numbers are kept correct by adding blank lines before a final pass. See https://github.com/pawamoy/prune-source/.

@pawamoy

pawamoy commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

@Jazzcort @ryanpetrello @mgorny @mcepl you've all packaged Griffe in the past for various distros, so I wanted to give you a heads up. Let me know if you'd prefer not being mentioned.

This PR introduces a new extra called "faster" which adds prune-source (https://pypi.org/project/prune-source/) as dependency. It's written in Rust and exposed to Python thanks to Pyo3. Its wheels are built with Maturin. It currently vendors the ruff_python_parser code, but with next Ruff's release we should be able to get rid of it (they accepted to make two symbols public, that we needed).

Griffe stays pure Python, and doesn't change anything in its build process.

Let me know if I can help, or if just giving you time to approach this is enough. Merging and releasing can wait a bit.

@mgorny

mgorny commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Thanks for the heads up. Gentoo removed griffe along with MkDocs over 1.x being abandoned, and everyone switching to random forks, so we aren't affected. More generally, Rust dependency might be a problem but if it's optional, it's not much of a problem, and even less for a package without many reverse dependencies where we can just assume we can remove it from most platforms with no major loss.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants