Fix protocol manifest #79

gkellogg · 2022-04-25T21:08:43Z

This is a fair rewrite of the protocol manifest (Turtle, HTML and now JSON-LD) based on the protocol_validator.

Information used in the CGI script has been extracted into a Ruby script used to generate the manifests. It should be at least as good as what was up previously, and conform to the semantics embedded in the CGI script (other than scripted result validation). It uses the http and cnt ontologies to encode the sequence of HTTP interactions, taking some liberties in the response fields to allow for matching results. For example:

:query_get a mf:ProtocolTest;
  dawgt:approval dawgt:Approved;
  dawgt:approvedBy <http://www.w3.org/2009/sparql/meeting/2012-11-20#resolution_3>;
  mf:action [
    a ht:Connection;
    ht:connectionAuthority "www.example";
    ht:requests ([
        a ht:Request;
        ht:absolutePath "/sparql?query=ASK%20%7B%7D&default-graph-uri=http%3A%2F%2Fkasei.us%2F2009%2F09%2Fsparql%2Fdata%2Fdata0.rdf";
        ht:httpVersion "1.1";
        ht:methodName "GET";
        ht:resp [
          a ht:Response;
          ht:body [
            a cnt:ContentAsText;
            cnt:characterEncoding "UTF-8";
            cnt:chars true
          ];
          ht:headers ([
              a ht:RequestHeader;
              ht:fieldName "Content-Type";
              ht:fieldValue "application/sparql-results+xml, application/sparql-results+json";
              ht:headerElements ([
                  a ht:HeaderElement;
                  ht:elementName "application/sparql-results+xml"
                ] [
                  a ht:HeaderElement;
                  ht:elementName "application/sparql-results+json"
                ])
            ]);
          ht:statusCodeValue "2XX",
            "3XX"
        ]
      ])
  ];
  mf:name "query via GET" .

cc/ @kasei

gkellogg · 2022-04-27T20:25:26Z

Another thing that might be useful is to remove the fuzzy results comparison, and instead use a specific result set (.srx, .srj, or .ttl) to compare that the result set or graph is isomorphic with the expected result. From the manifest description, it would just mean adding a mf:result which isn't currently used. Then, the ht:body within the ht:resp could be eliminated. Might benefit from desiring a negative version of mf:ProtocolTest, but this could be implied by the expected status code.

kasei · 2022-05-01T18:13:19Z

Overall I think this looks good. I'm going to try to take a closer look at the specifics of the protocol test definitions this coming week.

I'm unfamiliar with all the ruby/rake/haml stuff in here. I see that those predate this PR, but I would find it helpful to have a description of what's going on here in the PR (and maybe also somewhere in the repo) – what files are source of truth, what are supporting code to generate other files, which files are committed to git but are just generated from other files, etc.

gkellogg · 2022-05-02T16:07:14Z

The Ruby/Rake/Haml stuff is just there to produce the manifests. It happens to be what I'm familiar with, but the data structure used in the gen_manifest.rb should be pretty familiar to anyone using Python or Javascript. The key is the content of the mf:action which is produced.

gkellogg · 2022-05-02T17:37:04Z

An interesting project might be to re-write the CGI to use the interaction descriptions from the manifest, rather than as baked in. I essentially did this for my own tests (here and here), although they use a mock testing structure.

gkellogg · 2022-05-07T21:50:42Z

I would find it helpful to have a description of what's going on here in the PR (and maybe also somewhere in the repo) – what files are source of truth, what are supporting code to generate other files, which files are committed to git but are just generated from other files, etc.

Fair enough, as this mechanism is used pervasively throughout the repo. I'll make that the focus of another PR, but basically, the strategy is to generate JSON-LD based manifests from the Turtle source using a common Frame, and then use the resulting manifest.jsonld to generate the HTML. The template treats the JSON-LD as plain JSON and just iterates through the data-structures to generate HTML. There is more specific code for protocol to generate the manifest.ttl, but generally the code to generate the manifest.jsonld and index.html is common and run from the Rakefile in the root directory.

In principle, this could be done easily enough in another language, as code to parse and frame JSON-LD from Turtle is standard, and Haml is one of many different templating languages (e.g. Shpaml.

gkellogg · 2022-05-15T18:04:51Z

@kasei ping.

sparql11/data-sparql11/protocol/manifest.ttl

joka921 · 2025-06-18T16:21:03Z

@kasei @gkellogg
We are currently working on a tool that uses the manifest.ttl files to run the tests. Especially for the protocol tests this is somewhat hacky, as th manifests are a little underspecified. From our experience, every solution should use the manifest (publicly available, readable to humans and machines alike) as the sole source of ground truth (specify the inputs for the default and named graphs, specify the expected result, and only the input is not a query (like for the other test types) but a literal protocol string.

Then you can still build an official binary evaluation via CGI or something else, but it is easier to also integrate these tests and extend them into other tools.

gkellogg · 2025-06-18T21:05:37Z

Rebased to lastest, although some more follow-on may be necessary.

w3cbot · 2025-07-03T17:05:29Z

This was discussed during the #rdf-star meeting on 03 July 2025.

View the transcript

w3c/rdf-tests#79

gregg: 79 is created by me
… nothing is in manifest that drove the expected behaviour
… depended on running a script that gregg wiliams managed
… not relating to work that needs to be done soon, we should consider if that is the approach we want to do

ora: would be ok to check if people want to merge

andy: agree, worth it for moving things forward

adrian: 193 and 198 seem to be ok, on concepts doc

pchampin: will take care about this by end of week

Does not currently integrate back-links to tests. * Don't automatically add Content-Type on POSTs if not explicitly provided. * Validate and lint HTML/RDFa output. * Extract information from protocol_validator.cgi in to Ruby script used to generate manifest variations. * Uses HTTP and CNT ontologies to encode server interactions.

afs · 2025-10-05T17:51:40Z

Ping @kasei

kasei

I think there are a lot of changes encoded here from the previously approved (by the 1.1 WG) tests. If we're going to go through and re-approve them all, maybe that's not an issue, but it concerns me. Some of the changes I call out are important to the intentions of tests, while others do not impact conformance (e.g. some tests have been changed to use form-url-encoding instead of direct-POST, but in tests that are not testing the encoding mechanism). I think there's are also some questionable choices being made in the use of the http and content vocabularies in modeling the tests.

I think moving away from the 1.1-era encoding of these tests in my perl scripts is a good thing, but I'm a bit uncomfortable about moving away from them by moving towards ruby scripts as the source of truth here. It's still not clear to me why that's the direction used here instead of having a data file be the source of truth, and generating other data formats from that source. That also makes my uncomfortable, but it's something I could live with if we can get the flagged issues worked out (for the sake of moving forward).

kasei · 2025-10-06T02:42:17Z