Terms within multiple contexts may not be found

[martindholmes]

With @sydb, we're trying to debug a problem in the DHQ build where there are many labelled contexts, including contexts for many different languages. When indexing takes place, the "English" context is caught repeatedly and stored in the stem file items multiple times, but the lower-level context (e.g. div[@id='abstract']) do not get noticed, so searching in them fails.

When the language context is redefined as text()[lang('en')], the immediate problem goes away -- the abstract context works OK -- but now the language context no longer works; but this is because the XPath to find that context is not working, so it's not getting in the way.

The root question is: what should happen in the case of nested contexts (a term appears in a structure which has defined contexts at multiple levels). It should "just work", but I don't know if we have given much attention to testing this, so that will be the first thing to do. Another question would be why do we get multiple iterations of the same language context in the stem file; surely those should be unique values.

[martindholmes]

This needs to be fixed for 1.4.x, but it also needs to be added to tests in 2.0 and fixed there if necessary.

[joeytakeda]

@martindholmes and @sydb, I'm not sure I'm understanding the problem fully -- is it an issue with contexts or with processing of languages (or a problem where the two in combination are causing problems?). A sample with a config might be useful here just to make it a bit clearer as to what ought to be tested.

That said, to answer your question @martindholmes :

what should happen in the case of nested contexts (a term appears in a structure which has defined contexts at multiple levels

I believe (and thought) that contexts would always be inherited-- e.g. if you have this:

<div>
    <p>This article contends and argues...</p>
</div>
<div class="text">
   <div class="abstract">In this article, we argue</div>
    <div class="body">
       <p>This article contends...</p>
    </div>
</div>

and you have these contexts defined as text (div[@class='text']), abstract (div[@class='abstract']), and
body (div[@class='body']), then I would think that you should get the following:

term	context	hits
article	[none]	3
	text	2
	abstract	1
	body	1
argue	text	1
	abstract	1
	body	0

The way these contexts are passed through tokenization has been much improved in 2.0, so hopefully this has been improved by 2.0 (or if not it might be an easier fix?)

[martindholmes]

Until I've added a nested context test, and another one with the added complication of using the lang() function, I won't know for sure what's happening. I'm hoping to do that tomorrow, and I'll do it both in dev and in 1.4-release, so we can be clear about the issue.

[martindholmes]

Initial work on the test to confirm the problem is happening in branch issue-345-contexts which branches from release-1.4.

[martindholmes]

It turns out that our tests already include a simple instance of a nested context: in the "clues.html" file, there is a <blockquote> (context: "Quotations") which contains within it a <p class="citation"> (context: "Citations"), and searching for the word "Amending", which appears in the latter and therefore also in the former, we get (correctly) a single hit whether either or both of those contexts are selected. So the problem here is not with nesting itself, unless there is something in the code (@joeytakeda do you know of anything?) which would trigger a problem when we encounter the same context nested multiple times within itself.

So my suspicion is that the problem here is related to the lang() function, which is testing true repeatedly down the tree -- and in fact, it will test true for the very item which is also supposed to be testing true for the other context. In other words, a single element will match both contexts. My next strategy is going to be reproducing that scenario in the test branch: in other words, where a single item matches multiple contexts (as opposed to an ancestor matching one, and a descendant matching the other), do both contexts get added, or only one?

[martindholmes]

Incidentally @sydb I think an alternative and perhaps better approach might be to replace the lang()-based contexts with a set of description filters titled "Documents containing any of these languages" or something like that, so that you're not trying to constrain the search to specific bits of documents, but instead to documents which have content in the target languages (so you would add a meta tag for this filter for each language contained in the document). Most of the time, when you're searching for terms in a specific language, they're going to be unique to that language anyway, so the result would be the same, but the mechanics would be far simpler.

That would also enable someone to simply list all the documents containing a particular language, without having to search for any term at all, which is very useful in itself.

[joeytakeda]

From what I can understand (and apologies, I feel like I may be missing something here), I think this is the major bug of 1.4 that we are resolved through the new tokenization process. This is, I think, precisely the use case that we sought to fix, which is explained in much better detail in #219 (see #219 (comment)).

I won't duplicate that information here, but the long and the short of it, is that in 1.* , the tokenization code was not handling multiple matches for a single context (or any template produced by configuration elements of the same kind) since it would recreate attributes rather than tunnelling values. So I think this is already resolved in 2.0 and can't really be back-ported to 1.* since it requires the revamped tokenization logic.

[martindholmes]

@joeytakeda I'd forgotten most of that issue, but re-reading it now, it suggests (correct me if I'm wrong) that I should be able to reproduce the problem in 1.4 by creating two contexts which match the same element; one of them would be lost, so searching in that context would fail. If I can do that simply, then I can try the same thing in 2.0, and that should demonstrate that the problem is solved by the new tokenization approach.

[martindholmes]

That does sort of work -- when I create a third context <context match="blockquote/p[contains-token(@class,'citation')]" label="Blockquote citations"/>, the stem is indeed only associated with two contexts, but they are the two which match precisely on the p element; the ancestor context, the containing blockquote, is the one which is lost.