PluginPT15 do not handle accuentuate search

[JpEncausse]

Describe the bug

I have a document with a sentence "Invitation gâteau au chocolat" (chocolate cake in French) PluginPT15 and PluginQPS return wrong results because of accuentuate ?

To Reproduce

With plugin PluginPT15

1. If I search for "term=Gateau" I get NO results
2. If I search for "term=Gâteau" I get any results

With PluginQPS "Gateau" do not match "Gâteau" but "Gâteau" match.

Expected behavior

Both plugins should match Gâteau or Gateau

Environment Info

Orama 3.1.x

Affected areas

Search

Additional context

I use Oarama as a cache for a better search of AirTable data.
I'm searching for "term" along all fields with some boost.
PluginPT15 seems to be way better than default search for this usecase because of field ranking mechanism over large text.

[micheleriva]

Hi @JpEncausse,
at first it seems a tokenization problem, Orama should turn â into a and then let you match Gâteau with Gateau without any problem. I'll check on that.

I'm a bit confused about about this:

With PluginQPS "Gateau" do not match "Gâteau" but "Gâteau" match.

So are you saying that Gateau won't match Gâteau, but searching for Gâteau matches Gâteau correctly? Just to clarify if I understood correctly.

If so, it looks like a problem with removing diacritics during tokenization.

[JpEncausse]

Hi @JpEncausse, at first it seems a tokenization problem, Orama should turn â into a and then let you match Gâteau with Gateau without any problem. I'll check on that.

I'm a bit confused about about this:

With PluginQPS "Gateau" do not match "Gâteau" but "Gâteau" match.

So are you saying that Gateau won't match Gâteau, but searching for Gâteau matches Gâteau correctly? Just to clarify if I understood correctly.

If so, it looks like a problem with removing diacritics during tokenization.

Yes it seems exact match with â works but not fuzzy with a (on PluginQPS) but I switch back to PluginPT15 because got other issues with PluginQPS

[JpEncausse]

Do you confirm that PluginPT15 do not handle where filter ?
"where":{"LastModified":{"gte":1762627314999}}

Got an error

TypeError: Cannot read properties of undefined (reading 'type')
/node-red-data/node_modules/@orama/plugin-pt15/dist/index.cjs:1:3857
    at Array.filter (<anonymous>)
    at Object.searchByWhereClause (/node-red-data/node_modules/@orama/plugin-pt15/dist/index.cjs:1:3830)
    at innerFullTextSearch (/node-red-data/node_modules/@orama/orama/dist/commonjs/methods/search-fulltext.js:39:39)
    at performSearchLogic (/node-red-data/node_modules/@orama/orama/dist/commonjs/methods/search-fulltext.js:126:31)
    at fullTextSearch (/node-red-data/node_modules/@orama/orama/dist/commonjs/methods/search-fulltext.js:194:12)
    at

[Cyberheroes1010]

Biggest fruad Company !! Block me without paying my money !! Doing scam with researcher today i make a post on twitter & all social media !!