Moving out RDF Dataset Canonicalization

[mielvds]

PyLD contains implementations for a couple of RDF dataset canonicalization algorithms from back when this was part of the JSON-LD work, as pointed out by @davidlehn in #190. Since then, this standardization work has moved to the Credentials Community Group, which then evolved into the rdf-canon Working Group. Eventually, this led to the RDF Dataset Canonicalization w3c recommendation.

All of this produced three algorithms:

• URDNA2012 implemented at

  pyld/lib/pyld/jsonld.py

  Line 6073 in 58ea0ea

  class URGNA2012(URDNA2015):

• URDNA2015 implmented at

  pyld/lib/pyld/jsonld.py

  Line 5649 in 58ea0ea

  class URDNA2015(object):

• RDFC-1.0 which is still to be implemented, but only deviates slightly from URDNA2015.

There have been some discussions/efforts (#182) on whether this should move elsewhere. RDF Dataset Canonicalization has a broader use and could therefore benefit from being a repository of its own; the PyLD code would be more focused. Digitalbazaar actually already did this for its JS counterpart. This would also highly benefit RDFLib, which does not have a RDFC-1.0 implementation, but already provides methods to compare graphs.

In light of this library eventually moving to the RDFlib ecosystem, I propose that maybe this functionality moves first?
A possible path forward would be:

1. Preparing a PR isolating the implementation and tests in a separate module (I can do this)
2. Adding the RDFC-1.0 implementation and test-suite (I can try this, but maybe with help of the broader RDFLib community)
3. Copying this part of the code to a RDFLib based repo and publish the package on pypi
4. Replacing the code with the new dependency

(CC @davidlehn @BigBlueHat @nicholascar @edmondchuc @anatoly-scherbakov )

[BigBlueHat]

@mielvds that plan seems very sane to me. 😁 @davidlehn any additional thoughts/color to add here?

[davidlehn]

• Splitting out is fine with me. That path seems ok.
• It might be helpful to look at the changes in https://github.com/digitalbazaar/rdf-canonize to see what was needed for RDFC-1.0.
• A tricky bit might be how to handle backwards compatibility, either minimally, partly, or at all.
• Specifically, people may have been using the current built in functionality, and need to document or provide some path for upgrades. The older unofficial algorithms might still be in use, hard to know. JS code grew an option to help find if that was the case to help upgrades, but that may have been overthinking things. There is a possibility of some specific data causing different results with the RDFC-1.0 algorithm, so we tried to provide some tools to help with that. But also, the only official algorithm is now RDFC-1.0.
• If I'm remembering correctly, most of the newer test suite tests will work fine. There are some serializing changes. And some high computational complexity cases that likely need some code to deal with properly. Again, the js code changes may be helpful with this.
• I suspect extracting this code will run into an issue I'm dealing with in the js code right now. I'm extracting the test runner code into it's own specialized lib. Idea is to use same code for manifest processing, EARL generation, and benchmarking for json-ld, yaml-ld, rdf-canon, and cbor-ld implementations. Can copy and paste at first, but consider also using that path.

[mielvds]

Thanks for providing those insights, @davidlehn. I have one more tricky bit to add:

What to do with the nquads parser/serializer? It seems excessive to maintain both this implementation and the one in rdflib, but you might want to keep the independence. Also, triples are stored differently and those would need to be aligned first. Looks like PyLD is modelled after RDFjs, while rdflib has its tuple/class structure.

This raises the question of how far the PyLD/rdflib integration should go. Sure, we would like to not have to import rdflib when using PyLD, but who's going to maintain all these duplicate implementations? (that's also why I included @nicholascar @edmondchuc in this conversation )

This is not blocking for isolating these parts inside PyLD, but it will need to be addressed eventually.

More answers inline.

• Splitting out is fine with me. That path seems ok.
• It might be helpful to look at the changes in https://github.com/digitalbazaar/rdf-canonize to see what was needed for RDFC-1.0.

will do!

• A tricky bit might be how to handle backwards compatibility, either minimally, partly, or at all.

Yes, although I think that would be ok if done gradually.

• Specifically, people may have been using the current built in functionality, and need to document or provide some path for upgrades. The older unofficial algorithms might still be in use, hard to know. JS code grew an option to help find if that was the case to help upgrades, but that may have been overthinking things. There is a possibility of some specific data causing different results with the RDFC-1.0 algorithm, so we tried to provide some tools to help with that. But also, the only official algorithm is now RDFC-1.0.

I would definitely keep all implementations, just for the sake of dependency/BW compatibility. The normalize() function would still be available as is, either as part of PyLD or in the split-off lib. Or do mean other parts that might be used?

• If I'm remembering correctly, most of the newer test suite tests will work fine. There are some serializing changes. And some high computational complexity cases that likely need some code to deal with properly. Again, the js code changes may be helpful with this.
• I suspect extracting this code will run into an issue I'm dealing with in the js code right now. I'm extracting the test runner code into it's own specialized lib. Idea is to use same code for manifest processing, EARL generation, and benchmarking for json-ld, yaml-ld, rdf-canon, and cbor-ld implementations. Can copy and paste at first, but consider also using that path.

I'll start with the copy paste and see. RDFLIb has a way running tests as well

[nicholascar]

Sure, we would like to not have to import rdflib when using PyLD, but who's going to maintain all these duplicate implementations?

Remember RDFLib's got very few dependencies when no optional parts are used (e.g. the new RDF4J client, BerkeleyDB Store) so having it as a dependency only grows the total dependency stack by about 4 pretty lightweight pure Python packages.