Merge pull request #18145 from github/tausbn/python-add-guide-for-extending-the-parser

Python: Add guide describing how to extend the parser

Author: yoff

python/extractor/extending-the-parser.md

@@ -0,0 +1,94 @@
+# How to update the Python parser
+
+## Step 1: Add an extractor test
+
+Extractor parser tests live in the `tests/parser` directory. There are two different kinds of tests
+
+- Tests that compare the behavior of the old (Python-based) and new (`tree-sitter`-based) parsers, and verify that they yield the same output on the given source files, and
+- Tests that compare the output of the parser (old or new) against a fixed `.expected` file.
+
+What kind of test is run is determined based on the file name.
+If it ends in either `_new.py` or `_old.py`, then the test is run against an `.expected` file. If not, it is used to compare old against new.
+
+In most cases when adding new features, you'll only be interested in modifying the new parser (the old one is mostly there for legacy reasons).
+Thus, you will almost certainly want to create a test that ends in `_new.py`.
+
+It's a good habit to start by adding the parser test, as this makes it more easy to test when various bits of the parser have been added/modified successfully.
+
+The rest of this document will only concern itself with the process of extending the _new_ parser.
+
+To actually _run_ the tests, the easiest way is to use `pytest`.
+In the main `extractor` directory (i.e. where this file is located) run
+
+```sh
+pytest tests/test_parser.py
+```
+
+and wait for the tests to complete. It is normal and expected that the test seemingly freezes on the first run.
+This is simply because the `tsg-python` Rust binary is being built in the background.
+
+Once you have added a new test (or modified an old one) and start making modifications to the parser itself, it quickly becomes tedious to run _all_ the parser tests.
+To run just a single test using `pytest`, use the `::` syntax to specify specific tests.
+For instance, if you want to just run the tests associated with the file `types_new.py`, you would write
+
+```sh
+pytest tests/test_parser.py::ParserTest::test_types_new
+```
+
+## Step 2: Extend the `tree-sitter-python` grammar
+
+The new parser is based on `tree-sitter`, so the first task is to extend the existing `tree-sitter-python` grammar.
+This grammar can be found in the `grammar.js` file in the `tsg-python/tsp` subdirectory of the extractor directory.
+
+Note that whenever changes are made to `grammar.js`, you must regenerate the parser files by running
+
+```sh
+tree-sitter generate
+```
+
+inside the `tsp` directory.
+You'll need to install the `tree-sitter` CLI in order to run this command.
+One way to install it is to use `cargo`:
+
+```sh
+cargo install tree-sitter-cli
+```
+
+(This presupposes you have `cargo` available, but you'll need this anyway when compiling `tsg-python`.)
+
+Once the parser files have been regenerated, they'll get picked up automatically when `tsg-python` is rebuilt.
+
+> Pro-tip: When you're done with your parser changes, and go to commit these to a branch, put the autogenerated files in their own commit.
+> This makes it easier to review the changes, and if you need to go back and regenerate the files again, it's easy to modify just that commit.
+
+Once you have extended `grammar.js` and regenerated the parser files, you should be able to check that the grammar changes are sufficient by rerunning the parser test using `pytest`. If it fails while producing an AST that doesn't make sense, then you're probably on the right track. If it fails _without_ producing an AST, then something went wrong with the actual `tree-sitter` parse. To check if this is the case, you can run
+
+```sh
+tree-sitter parse path/to/test.py
+```
+
+and see what kind of errors are emitted (possibly as `ERROR` or `MISSING` nodes in the AST that is output).
+
+## Step 3: Extend `python.tsg`
+
+Once the grammar has been extended, we need to also tell `tsg-python` how to turn the `tree-sitter-python` AST into something that better matches the AST structure that we use in the Python extractor.
+
+For an introduction to the language of `tree-sitter-graph` (and in particular how we use it in the Python extractor), see the `README.md` file in the `tsg-python` directory.
+
+## Step 4: Extend the set of known AST nodes
+
+If you added new node types, or added fields to known node types, then you'll need to update a few files in the Python extractor before it is able to reconstruct the output from `tsg-python`.
+
+New AST nodes should be added in two places: `master.py` and `semmle/python/ast.py`. The former of these is used to automatically generate the Python dbscheme and `AstGenerated.qll`. The latter is what the parser actually uses as its internal representation of the AST.
+
+## Step 5: Rebuild the autogenerated AST and database scheme
+
+If you made changes to `master.py`, you'll need to regenerate a couple of files. This can be done from within the `extractor` directory using the `make dbscheme` and `make ast` commands. Note that for the latter, you need a copy of the CodeQL CLI present, as it is used to autoformat the `AstGenerated.qll` file.
+
+## Step 6: Add dbscheme upgrade and downgrade scripts
+
+If you ended up making changes to the database scheme inw step 5, then you'll need to add an appropriate pair of up- and downgrade scripts to handle any changes between the different versions of the dbscheme.
+
+This can be a bit fiddly, but luckily there are [tools](https://github.com/github/codeql/blob/main/misc/scripts/prepare-db-upgrade.sh) that can help set up some of the necessary files for you.
+
+See also the [guide](https://github.com/github/codeql/blob/main/docs/prepare-db-upgrade.md) for preparing database upgrades.