main: using regex for choosing a parser for given file name #4270

masatake · 2025-06-08T23:33:56Z

main: using regex for choosing a parser for the given file name

This change extends --map- option to support regular
expression matching with the full file name.

The original --map- option supports glob based matching
and extension comparison with the file basename.
However, two methods are not enough if the file names are too
generic. See #3287 .

The regular expression passed to --map- must be surround
by % character like

--map-RpmMacros='%(.*/)?macros.d/macros.([^/]+)$%'

If you want to match in a case-insensitive way, append `i' after the second % like

--map-RpmMacros='%(.*/)?macros.d/macros.([^/]+)$%i'

If you want to use % as part of an expression, put \ before % for escaping.

TODO:

codecov · 2025-09-27T18:39:53Z

Codecov Report

❌ Patch coverage is 93.08176% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.02%. Comparing base (6bce4ae) to head (ef09439).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
main/parse.c	96.01%	8 Missing ⚠️
main/rexprcode.c	78.94%	8 Missing ⚠️
main/options.c	93.22%	4 Missing ⚠️
optlib/rpmMacros.c	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4270      +/-   ##
==========================================
+ Coverage   86.01%   86.02%   +0.01%     
==========================================
  Files         250      251       +1     
  Lines       64159    64352     +193     
==========================================
+ Hits        55187    55361     +174     
- Misses       8972     8991      +19

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull Request Overview

This PR extends the --map-<LANG> option to support regular expression matching for file names, addressing limitations where glob patterns and extension matching are insufficient for generic file names. The implementation adds a new regex-based mapping type alongside existing extension and pattern mappings.

Key Changes:

Introduced regex pattern support using %regex%[i] syntax for language file mappings
Added new rexprcode module to handle regex compilation and matching
Extended optlib2c to generate C code from regex mapping definitions

Reviewed Changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
main/rexprcode.c	New module implementing regex pattern compilation, matching, and encoding
main/rexprcode_p.h	Public interface for regex code operations
main/parse.c	Core integration of regex matching into language detection logic
main/parse.h	Added rExprSrc structure definition and REXPR_LAST_ENTRY macro
main/parse_p.h	Extended langmapType enum with LMAP_REXPR flag
main/options.c	Command-line option parsing for regex patterns with icase flag support
optlib/rpmMacros.ctags	Example usage replacing commented-out patterns with regex
optlib/rpmMacros.c	Generated C code with regex mapping definitions
misc/optlib2c	Extended Perl script to parse and generate regex mapping code
source.mak	Build system updates for new source files
win32/ctags_vs2013.vcxproj	Visual Studio project file updates
win32/ctags_vs2013.vcxproj.filters	Visual Studio filter file updates
Tmain/list-map-rexprs.d/*	Test cases for new --list-map-rexprs option
Tmain/versioning.d/stdout-expected.txt	Updated test output expectations

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

main/rexprcode.c

main/options.c

main/parse.c

main/options.c

Copilot · 2025-10-19T18:02:21Z

misc/optlib2c

 	       unless ($_[0]->{'langdef'} eq $1);
 	   my $spec = $2;
-	   if ($spec =~ /\((.*)\)/) {
+	   if ($spec =~ /%(.+)%(i)?/) {


The regex pattern %(.+)% is greedy and will match incorrectly if the expression contains '%' characters, even with escaping. For example, %a%b%c% would capture a%b%c instead of a. The pattern should be non-greedy: %(.+?)%(i)?/ or better yet, should properly handle escaped '%' characters in the capture group.

Suggested change

if ($spec =~ /%(.+)%(i)?/) {

if ($spec =~ /%(.+?)%(i)?/) {

main/rexprcode_p.h

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 1 comment.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-19T19:02:34Z

misc/optlib2c

 	       unless ($_[0]->{'langdef'} eq $1);
 	   my $spec = $2;
-	   if ($spec =~ /\((.*)\)/) {
+	   if ($spec =~ /%(.+)%(i)?/) {


The regex should use a non-greedy quantifier .+? instead of .+ to prevent matching across multiple patterns when there are multiple % characters in the input. This could cause incorrect parsing of escaped % characters.

Suggested change

if ($spec =~ /%(.+)%(i)?/) {

if ($spec =~ /%(.+?)%(i)?/) {

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

misc/optlib2c

main/parse.c

main/options.c

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

main/parse.h

Copilot · 2025-10-19T19:22:39Z

misc/optlib2c

+	   if ($spec =~ /%(.+?)%(i|\{icase\})?/) {
+	       my $rexpr = { expr => $1,
+			     iCase => (defined $2 && ($2 eq 'i' || $2 eq 'icase'))? 1: 0 };


The regex allows {icase} as an alternative to i, but this syntax is not documented in the PR description or help text. Either document this alternative syntax or remove it to avoid confusion.

Suggested change

if ($spec =~ /%(.+?)%(i|\{icase\})?/) {

my $rexpr = { expr => $1,

iCase => (defined $2 && ($2 eq 'i' || $2 eq 'icase'))? 1: 0 };

if ($spec =~ /%(.+?)%(i)?/) {

my $rexpr = { expr => $1,

iCase => (defined $2 && $2 eq 'i')? 1: 0 };

Copilot · 2025-10-19T19:22:40Z

main/options.c

+
+static flagDefinition langmapRexprFlagDef[] = {
+	{ 'i', "icase",  langmap_rexpr_icase_short,  langmap_rexpr_icase_long,
+	  NULL, "applied in a case-insensitive manner"},


The long flag name is 'icase', but the optlib2c script also accepts '{icase}' syntax (line 296 in misc/optlib2c). These should be consistent, or the alternative syntax should be documented.

Suggested change

NULL, "applied in a case-insensitive manner"},

NULL, "applied in a case-insensitive manner (accepts both 'icase' and '{icase}' syntax)"},

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

main/options.c

main/parse.c

Copilot

Pull Request Overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot

Pull Request Overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot

Pull Request Overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Signed-off-by: Masatake YAMATO <[email protected]>

baseFilenameSansExtensionNew("a.in.in", ".in.in") could not return "a" with the original code. Signed-off-by: Masatake YAMATO <[email protected]>

Signed-off-by: Masatake YAMATO <[email protected]>

…nsExtensionNew Delete baseFilenameSansExtensionNew() from the source tree. Signed-off-by: Masatake YAMATO <[email protected]>

The original code used a boolean value to toggle how filenames were mapped to the parser by glob-like pattern or by extension. To support the third way mapping a file name to a parser, by regular expression pattern, we will use an enum value instead of Boolean. Signed-off-by: Masatake YAMATO <[email protected]>

Signed-off-by: Masatake YAMATO <[email protected]>

This change extends --map-<LANG> option to support regular expression matching with the full file name. The original --map-<LANG> option supports the glob based matching and the extension comparison with the file basename. However, two methods are not enough if the file names are too generic. See universal-ctags#3287 . The regular expression passed to --map-<LANG> must be surrounded by % character like --map-RpmMacros='%(.*/)?macros\.d/macros\.([^/]+)$%' If you want to match in a case-insensitive way, append `i' after the second % like --map-RpmMacros='%(.*/)?macros\.d/macros\.([^/]+)$%i' If you want to use % as part of an expression, put \ before % for escaping. Signed-off-by: Masatake YAMATO <[email protected]>

Signed-off-by: Masatake YAMATO <[email protected]>

…options Signed-off-by: Masatake YAMATO <[email protected]>

Signed-off-by: Masatake YAMATO <[email protected]>

masatake · 2025-10-21T08:59:22Z

Updating the man page is the last task.

masatake marked this pull request as draft June 8, 2025 23:34

masatake changed the title ~~main: using regex for choosing a paser for given file name~~ [WIP] main: using regex for choosing a paser for given file name Jun 8, 2025

masatake force-pushed the main--rexpr branch 2 times, most recently from a5a3a28 to c50f467 Compare September 27, 2025 18:32

masatake force-pushed the main--rexpr branch from c50f467 to 24ae704 Compare October 14, 2025 20:14

masatake changed the title ~~[WIP] main: using regex for choosing a paser for given file name~~ [WIP] main: using regex for choosing a parser for given file name Oct 17, 2025

masatake force-pushed the main--rexpr branch from 24ae704 to 625a545 Compare October 19, 2025 18:00

masatake requested a review from Copilot October 19, 2025 18:00

Copilot AI reviewed Oct 19, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch from 625a545 to d59092b Compare October 19, 2025 19:01

masatake requested a review from Copilot October 19, 2025 19:01

Copilot AI reviewed Oct 19, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch from d59092b to 7ee055b Compare October 19, 2025 19:11

masatake requested a review from Copilot October 19, 2025 19:11

Copilot AI reviewed Oct 19, 2025

View reviewed changes

misc/optlib2c Show resolved Hide resolved

main/parse.c Outdated Show resolved Hide resolved

main/parse.c Show resolved Hide resolved

main/parse.c Show resolved Hide resolved

main/options.c Outdated Show resolved Hide resolved

masatake force-pushed the main--rexpr branch from 7ee055b to 1c50a08 Compare October 19, 2025 19:21

masatake requested a review from Copilot October 19, 2025 19:21

Copilot AI reviewed Oct 19, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch 2 times, most recently from 231a606 to f865e33 Compare October 20, 2025 02:17

masatake requested a review from Copilot October 20, 2025 02:19

Copilot AI reviewed Oct 20, 2025

View reviewed changes

main/options.c Outdated Show resolved Hide resolved

main/parse.c Outdated Show resolved Hide resolved

masatake force-pushed the main--rexpr branch from f865e33 to d2631e3 Compare October 20, 2025 05:45

masatake requested a review from Copilot October 20, 2025 05:48

Copilot AI reviewed Oct 20, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch from d2631e3 to 4470454 Compare October 20, 2025 06:36

masatake requested a review from Copilot October 20, 2025 06:46

Copilot AI reviewed Oct 20, 2025

View reviewed changes

masatake requested a review from Copilot October 20, 2025 10:07

Copilot AI reviewed Oct 20, 2025

View reviewed changes

masatake added 14 commits October 21, 2025 15:57

optlib2c,cosmetic: delete a redundant newline

8e1578b

Signed-off-by: Masatake YAMATO <[email protected]>

docs: (NEWS.rst) fix the ReStructuredText markup

1f321d8

Signed-off-by: Masatake YAMATO <[email protected]>

main: fix a potential bug in the code making filename sans extension

dd57a00

baseFilenameSansExtensionNew("a.in.in", ".in.in") could not return "a" with the original code. Signed-off-by: Masatake YAMATO <[email protected]>

main: add a new helper function filenameSansExtensionNew

52fb44c

Signed-off-by: Masatake YAMATO <[email protected]>

main,cosmetic: clean up whitespace with M-x whitespace-cleanup (Emacs)

1f8f56d

Signed-off-by: Masatake YAMATO <[email protected]>

main,refactor: use filenameSansExtensionNew instead of baseFilenameSa…

73caf7e

…nsExtensionNew Delete baseFilenameSansExtensionNew() from the source tree. Signed-off-by: Masatake YAMATO <[email protected]>

main,refactor: don't use else if if-block does return always

ae8c983

Signed-off-by: Masatake YAMATO <[email protected]>

optlib2c: support --map-<LANG>=%REXPR% option

6f34ebf

Signed-off-by: Masatake YAMATO <[email protected]>

rpmMacros: parse macros.d/macros.*

929a496

Signed-off-by: Masatake YAMATO <[email protected]>

Tmain: add a case testing --list-map-rexprs and --map-<LANG>=%REXPR% …

99792cc

…options Signed-off-by: Masatake YAMATO <[email protected]>

RpmMacros,Tmain: add a case testing the macros.d/macro.* rexpr entry

62f4838

Signed-off-by: Masatake YAMATO <[email protected]>

docs: write about --map-<LANG>=%REXPR% to NEWS file

ef09439

Signed-off-by: Masatake YAMATO <[email protected]>

masatake force-pushed the main--rexpr branch from 4470454 to ef09439 Compare October 21, 2025 07:25

masatake marked this pull request as ready for review October 21, 2025 08:59

masatake changed the title ~~[WIP] main: using regex for choosing a parser for given file name~~ main: using regex for choosing a parser for given file name Oct 21, 2025

masatake mentioned this pull request Oct 22, 2025

--languages='C' somehow disables +p ? #4290

Open

	NULL, "applied in a case-insensitive manner"},
	NULL, "applied in a case-insensitive manner (accepts both 'icase' and '{icase}' syntax)"},

main: using regex for choosing a parser for given file name #4270

Are you sure you want to change the base?

main: using regex for choosing a parser for given file name #4270

Uh oh!

Conversation

masatake commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

masatake commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

masatake commented Jun 8, 2025 •

edited

Loading

codecov bot commented Sep 27, 2025 •

edited

Loading