Pros and Cons for more modularisation

[nicholascar]

Refactoring the RDFlib package into a core and a series of satellite repositories has been proposed for a long time (see Issue #391) and it was considered a possible goal of RDFlib's 6.0.0 release.

In order to proceed, we need at least some pros and cons of such as move, as well as a refactoring suggestions (what stays in core, what moves out) and this Issue is to document that.

Please add comments to this starting with PRO: or CON: and your suggestions for what should be in or out of core, with reference to the current repo structure.

[tgbugs]

CON: More packages means more work for maintainers. In my experience time cost is essentially constant per package per release, so if you split a single package into 10 you have 10x the work that you have to dedicate to managing releases. This is the general boilerplate overhead.

CON: Could increase the number of dependencies and complicate the process needed to get the right set of optional dependencies for some desired functionality.

Can we get more modularization internally within the rdflib repository without adding the maintenance overhead?

What release cycles currently suffer because of the current system?

Are there clear examples where some code could be used by another project? If the code is only used by rdflib then it seems like there is only a cost.

I'll add some pros at some point.

[niklasl]

PRO: If a feature depends on a third-party library, that burdens anyone who only needs to use the core features. As it stands, I doubt we have much of that though. If e.g. RDFa parsing depends on html5lib though, that's a case. One can conceive of future cases though, like efficient stream parsers or asynchronous handling of networking.

(I do think it's better for RDFLib to err on the side of pragmatic though, i.e. a bit heavier on features and dependencies is reasonably good enough. Still, third-party dependency issues have historically and will possibly continue to crop up occasionally.)

[rduivenvoorde]

@tgbugs as an example rdflib being used by a UI-based plugin for QGIS (a GIS desktop application): https://github.com/sparqlunicorn/sparqlunicornGoesGIS
Being tied to urllib2 is in this case a CON, as QGIS is Qt based and it is better to use Qt based classes then for network requests as then settings/debug etc of QGIS can be used (see: sparqlunicorn/sparqlunicornGoesGIS#18 (comment)).

As a visuable example: the 2nd WorldWar bunker in The Netherlands queried via the plugin/rdflib and shown in QGIS:

So speaking as QGIS community member: it would be nice to at least make the networking requests modular (maybe using some abstraction layer like:
https://github.com/planetfederal/lib-qgis-commons/blob/master/qgiscommons2/network/networkaccessmanager.py

I'm not sure if this is feasible or not...

[nicholascar]

@rduivenvoorde can you please try running your QGIC plugin with RDFlib master, as in built from the master branch of this repo? Since we've removed support for Python 2, I don't think we are tied to urllib2 anywhere and we've also just removed the dependency on the requests module too, so it and it would be good to know if these changes positively affect your library.

[ashleysommer]

The dependency on urllib2 is not directly from RDFLib (if you look at the code, you'll see no usages for urllib2 and urllib2 is not in our list of install requirements).
@nicholascar is right, it comes from the use of the requests library. We have recently removed the dependency on requests in favor of the newer built-in urllib.request Python3 module, so if you test on master you will see that urllib2 is no longer brought in.

[situx]

Hello, as the main developer of the QGIS plugin in question I might shed some light on the details of the aforementioned request.

1. I am using RDFLib 4.2.2 with SPARQLWrapper bundled with the QGIS plugin, as libraries not bundled with QGIS need to be added to QGIS plugins directly
2. The plugin queries a SPARQL Endpoint via SPARQLWrapper and as far as I can see I am able to configure whether SPARQLWrapper uses a POST or GET request, but I am not able to delegate the execution of this request to a different implementation such as QgsNetworkAccessManager (https://qgis.org/pyqgis/3.4/core/QgsNetworkAccessManager.html)
   So I assume SPARQLWrapper uses urllib to do the HTTP request to the SPARQL endpoint.

Is this wrong or is there indeed some way I could use QgsNetworkAccessManager instead?

[nicholascar]

Hi Timo,

I assume SPARQLWrapper uses urllib to do the HTTP request to the SPARQL endpoint

Yes it does. Both SPARQLWrapper & RDFlib, if doing web requests, use urllib without the option to delegate.

is there indeed some way I could use QgsNetworkAccessManager instead?

Not directly within SPARQLWrapper but: SPARQLWrapper was written a very long time ago before modern HTTP Python libraries like requests or httpx. I would suggest that to delegate network functions, you keep using RDFlib but just write your SPARQL requests into a requests or httpx-based script and take advantage of their better networking capabilities. There aren't so many benefits that you get from SPARQLWrapper over a few requests/httpx lines of code. I haven't looked into this myself buy presumably QgsNetworkAccessManager can be used by things like requests/httpx.

[situx]

Hi Nicolas,

I think it is probably not so easy. I believe QgsNetworkAccessManager will use Qt-builtin network handling.
The issue is rather not to use requests or httpx, because we would like to take advantage of internal QGIS proxy settings which are stored as a QSettings object.
My current workaround for this is to check for QGIS Settings and then apply them in urllib2 for SPARQLWrapper, but it appears that this is not sufficient. I probably have to write some new code using Rdflib and QgsNetworkManager skipping SPARQLWrapper to finally solve this issue.
Probably needs some more investigation.

[nicholascar]

I probably have to write some new code using Rdflib and QgsNetworkManager skipping SPARQLWrapper

This is what I meant! Skip SPARQLWrapper altogether and write your own code on top of RDFlib. If you use requests/httpx for the network calls - and they really should be able to use any proxy settings - you won’t be writing much more code than SPARQLWrapper anyway!

(I do think that eventually we will need to update SPARQLWrapper to use all the modern internet connection options)