Add .lrrignore support for filtering archives

[sitiyou]

This PR add .lrrignore support with plexignore based syntax, triggered during Shinobu scanning and when new files are added.

See #1120

syntax

Based on the Plex .plexignore spec, extended with some useful features from gitignore:

Syntax	Description
*	glob wildcard, matches any character except /
#	comment (line start only; \# escapes to literal #)
!	negation, overrides parent directory rules (\! escapes to literal !)
Leading /	anchors the pattern to the .lrrignore file's directory
Trailing /	matches all contents of the directory
\*	literal *
\	for any other \X, the backslash is silently dropped (\X → X)

Rules are resolved relative to the .lrrignore file's own directory, with nested rules overriding parent ones.

interface

Ignore.pm exports four methods:

• is_ignored($file, $rules): walks parent directories via dirname and matches against compiled regex rules
• build_ignore_rules($content_dir): scans the directory tree for .lrrignore files and builds a rule set
• initialize(): called at server startup, builds rules and stores them in Redis as LRR_IGNORE_RULES
• load_ignore_rules(): loads the frozen rule set from Redis with state $cached

windows compatibility

The project's existing cross-platform filesystem APIs don't quite cover some of the path operations this feature needs. I'm not familiar with Windows, so I can't say for sure it'll work there. The tests pass under MSYS2. And that is all I can guaranteen.

unit test

Added tests covering the various ignore syntax rules.

[sitiyou]

I forgot to handle escaped special characters in ignore rules.
Maybe I need to rethink the .lrrignore syntax design, since implementing full gitignore compatibility syntax is turning out to be more complex than I expected.
It also means characters like [] would require escaping to work properly, but they are extremely common in manga titles.

Maybe I should only implement a subset of gitignore instead.

[sitiyou]

@Difegue this PR is ready for review. The syntax has been simplified and I've tested some common use cases in docker.

[Difegue]

Re-updating the rules for every single incoming file feels overkill.

I think we could get by only updating the rules when Shinobu gets started.

[sitiyou]

Loading all .lrrignore files at startup would mean walking the filesystem twice: once to gather ignore rules, once to update filemaps.

Since find_path visits files in the same directory sequentially, update_ignore_rules can be simplified to a single string comparison for most calls:

return if $rules->{current_dir} && $rules->{current_dir} eq $dst_dir;

---

BTW, File::Find supports pruning to skip ignored directories, but Win32::LongPath::Find lacks this functionality so I did not implement pruning.

And Win32::LongPath::Find does not return directories, meaning we can't only update rules when entering a directory.

[Difegue]

ditto

[sitiyou]

This can be optimized by caching all ignore rules after the filemap scan. So any modifications made by the user will not take effect until Shinobu is restarted.
And newly added files can simply use the cached rules, no extra filesystem api calls are needed.

[psilabs-dev]

Couple questions/issues from me at first look

I think ignore reads are cheap compared to ID compute, shinobu can probably do a read once before every scan. Upload API on the other hand is less forgiving (maybe cache the rules in redis? but ehh)

Otherwise, users should not expect to update rules during LRR uptime, bc the results will be inconsistent per worker (unless you're reading per call).

[psilabs-dev]

All the ignored files are treated as deleted? What happens to metadata during a cleanup?

[psilabs-dev]

related (since "file" is false now):

LANraragi/lib/LANraragi/Utils/Database.pm

Line 391 in b94e480

unless ( -e $file ) {

[sitiyou]

It seems ok in my test. delete_archive will not delete the actual file since the file field was wiped, only metadata will be deleted.

# lib/LANraragi/Model/Archive.pm +426
    if ( -e $filename ) {
        my $status = unlink_path($filename);

[sitiyou]

OK, let me rethink the design:

• The ignore rules should be global and immutable, to make them consistent across all workers (upload, shinobu, and any other plugins).
• An extra scan of the filesystem to construct ignore rules is actually acceptable, considering the filesystem cache. (30ms vs 10ms, in my test with 700 files and 15 directories on HDD using find . > /dev/null)

So I should construct ignore rules only at startup, and cache them internally in ignore.pm. (but is it ok to make ignore.pm stateful? or simply put them in redis?)

[sitiyou]

UPDATE:

• Ignore rules are now built at startup and stored in redis for global access.
• upload.pm now rejects incoming files if they match any ignore rules.