tokenize() issue with SPACE_OR_PUNCTUATION and quotes

[clibu]

SPACE_OR_PUNCTUATION has changed at some point and now is regexp: /[\n\r\p{Z}\p{P}]+/u
This matches quotation characters " and ' which results in, for me at least, unwanted and incorrect results. ie. p{P} matches all unicode punctuation characters.

For example: song's now matches "song" and "s" - so every 's' character in a document matches. Further documents which don't include "song" but do include "s' match. You can see this using: Demo

The older SPACE_OR_PUNCTUATION regexp did not use the new Unicode categories and did not match " and ' etc.

From reading: Unicode Character Categories I can't see how any of the Punctuation categories can be used for SPACE_OR_PUNCTUATION. That said I hadn't come across these before now.

[clibu]

Digging deeper I now see that the original SPACE_OR_PUNCTUATION regexp also matched " and ' etc. so had the same issue.

I've replaced the default tokenizor as follows, which for my use case works well.

function refinedTokenizer(str) {
  return str
    .toLowerCase()
    // normalize curly quotes and unicode apostrophes
    .replace(/[\u2018\u2019\u201A\u201B\u2032\u2035]/g, "'")
    // normalize hyphens and dashes
    .replace(/[\u2010-\u2015]/g, "-")
    // extract tokens:
    // - letters (unicode)
    // - numbers
    // - apostrophes inside words
    // - hyphens inside words
    .match(/[a-z0-9\u00C0-\u024F]+(?:['-][a-z0-9\u00C0-\u024F]+)*/g) 
    || [];
}

This maybe useful for other's.