Cython: 26% faster on a simple benchmark. No code change.

[rchateauneu]

Cython is able to speedup pure Python code of about 20% apparently, without code change.

The only needed adaptation is to add some statements to setup.py which, if cython is installed could convert some or all Python source files to C, the compile them.

In a second stage, it would also be possible to add Cython decorators for static typing in the Python file. These decorators are ignored by the interpreter.

[FlorianLudwig]

@rchateauneu Is your 20% number based on a generic benchmark or did you try it with rdflib?

[rchateauneu]

This is a generic benchmark that I confirmed on other projects, but not on rdflib yet.
The speed-up can be much better, just by using decorators to set the type of integer variables in tight loops.

It is possible to optimize only some files and leave the others intact. This makes the installation faster (Because less files have to be compiled/linked).

The nice thing is that it is completely transparent: Cython transforms a Python file into a C file which is after that compiled and linked into a pyd or a so file, and is imported as is by normal Python mechanism.

In "Pure Python" mode - which is the less intrusive, the variables types can be specified with decorators (https://cython.readthedocs.io/en/latest/src/tutorial/pure.html#magic-attributes) or with a pxd auxiliary file. In this case, there in no source change at all.

[FlorianLudwig]

I have done some benchmarking and profiling of rdflib in the past, there are not many tight loops to optimize though. (at least in the sparql focused use cases i benchmarked)

Still I might be interested in seeing if rdflib profits of this. Are you planing to do some testing with rdflib and cython?

[rchateauneu]

I created a fork, and try to see what is possible.
You can see the impact on setup.py here:
https://github.com/rchateauneu/rdflib/blob/master/setup.py

You can generate the pyd file along the pure Python one, with:
python setup.py build_ext --inplace

... and remove it with:
python setup.py clean

Do you please have a specific and simple benchmark I could focus on, please ? A command without too many dependencies, for example ?
Thanks

[rchateauneu]

With this benchmark and more "cythonized" files (Please see setup.py in branch):

Best time interpreted:
py -3.6 rdflib_cython_benchmark.py
argv= ['rdflib_cython_benchmark.py']
time= 4.802026748657227

Best time compiled:
py -3.6 rdflib_cython_benchmark.py test
argv= ['rdflib_cython_benchmark.py', 'test']
time= 3.53002667427063

Of course, it is possible to go further:

• Compile more files.
• Add static typing in the py files or in separate pxd files.
• Many more Cython directives are possible.

This benchmark is a simple list of calls to rdflib, nothing complicated.

rdflib_cython_benchmark_py.TXT

[white-gecko]

@rchateauneu would you be willing to try out if cython can improve the parsers? #1261

[rchateauneu]

@white-gecko
Here is the bench I just ran. No code change except the list of Python files in setup.py . Speedup 14%

py -3.6 import.py english-wordnet-2020.ttl turtle
Windows 7.

Interpreted.
2021-02-25 18:05:27
2021-02-25 18:10:09

Then run:
py -3.6 setup.py build_ext --inplace

2021-02-25 18:54:05
2021-02-25 18:58:08

The script is:
import sys
import datetime
sys.path.insert(0, "rdflib")
import rdflib
print(rdflib.file)
g = rdflib.Graph()
print (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
g.parse(sys.argv[1], format=sys.argv[2])
print (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
``