Add configurable delimiter detection during import

[mikezhanghaozhe]

Related issues and pull requests

Closes #12974

PR Description

Delimiters are now separate, where default delimiters IMPORT_KEYWORD_DELIMITERS are used in the importing process and user's preferred delimiters are used for displaying. Thus, the default delimiters can be flexible and accommodate ";".

Things to note:

1. For now, default delimiters only include [";", ","]. This list is flexible and can be connected/replaced by user's preferred delimiters later if needed.
2. The list of default delimiters have priority, which means if ";" is detected, "," will no longer be considered as a delimiter in the importing process. This addresses the concern of having "," as a valid part of the keyword. #12974 Comment
3. Citation like """ @Article{, Keywords={asdf,asdf,asdf}, } """ in BibtexParserTest will be deduplicated during importing process. The previous unit test preserved the duplicated keywords.

Steps to test

Please ignore what's below, please see #15521 (comment)

1. Create a new entry that has keywords field containing ";".

2. Keywords are separate properly.

3. The original BibTex source also displays the delimiter user prefers ("," in this case).

Checklist

• I own the copyright of the code submitted and I license it under the MIT license
• If AI tools were used, I disclosed them in the "AI usage" section and reviewed, understood, and take full ownership of all AI-generated code
• I manually tested my changes in running JabRef (always required)
• I added JUnit tests for changes (if applicable)
• I added screenshots in the PR description (if change is visible to the user)
• [/] I added a screenshot in the PR description showing a library with a single entry with me as author and as title the issue number
• I described the change in CHANGELOG.md in a way that can be understood by the average user (if change is visible to the user)
• [/] I checked the user documentation for up to dateness and submitted a pull request to our user documentation repository

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Heuristically detect BibTeX keyword delimiters and normalize on import

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 20-40 Minutes

AI Description

• Detect ; vs , when importing BibTeX keywords and normalize to configured separator.
• Escape embedded separators so delimiter characters inside keywords are preserved.
• Add requirements/docs and expand unit + golden-file coverage for the new behavior.

Diagram

graph TD
  A[/"BibTeX file"/] --> B["BibtexImporter.importDatabase"] --> C["BibtexParser.parse"] --> D[("ParserResult/Database")] --> E["normalizeKeywordDelimiters"] --> F["KeywordList.parseImport"] --> G["KeywordList.serializeWithSpaces"] --> H[("BibEntry.KEYWORDS")]
  P(("Configured separator")) -."from preferences".-> E

  subgraph Legend
    direction LR
    _io[/"Input"/] ~~~ _proc["Process"] ~~~ _db[("Stored field")] ~~~ _cfg(("Config"))
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Integrate delimiter detection into BibtexParser field parsing

• ➕ Keeps import semantics closer to the source parsing stage
• ➕ Avoids a second pass over all imported entries
• ➕ Could reuse parser context (e.g., tokenization/escaping) more directly
• ➖ Increases coupling/complexity in the core parser
• ➖ Harder to keep scope limited to KEYWORDS without affecting other fields
• ➖ More risk of regressions in BibTeX parsing behavior

2. Explicit escape-aware delimiter scan (choose delimiter by unescaped occurrences)

• ➕ More direct and predictable than the current 'size > 1' heuristic
• ➕ Can prefer a delimiter even when only one keyword is present but delimiter appears
• ➕ Can be extended to support additional candidate delimiters easily
• ➖ Requires new low-level scanning logic and careful handling of BibTeX escaping rules
• ➖ Risk of duplicating logic already embedded in KeywordList.parse

Recommendation: The current post-processing approach in BibtexImporter is a good trade-off: it localizes the behavior change to import-time keyword normalization without destabilizing the BibTeX parser. If delimiter detection needs to become more nuanced (e.g., support more delimiters or disambiguate single-keyword cases), consider upgrading parseImport to an escape-aware occurrence-based selection, but keep it in KeywordList to avoid parser coupling.

Files changed (11) +155 / -10

Enhancement (2) +50 / -4

BibtexImporter.javaNormalize imported KEYWORDS using heuristic delimiter detection +24/-3

Normalize imported KEYWORDS using heuristic delimiter detection

• Adds import-time post-processing that reads each entry’s raw KEYWORDS, parses them using prioritized default delimiters, and re-serializes them with the user-configured keyword separator (with proper escaping).

jablib/src/main/java/org/jabref/logic/importer/fileformat/BibtexImporter.java

KeywordList.javaAdd parseImport and serializeWithSpaces helpers for import normalization +26/-1

Add parseImport and serializeWithSpaces helpers for import normalization

• Adds KeywordList.parseImport to try multiple delimiters in priority order and fall back to comma. Extends serialization to support a configurable join string (e.g., ", ") while keeping delimiter/hierarchy escaping behavior intact.

jablib/src/main/java/org/jabref/model/entry/KeywordList.java

Tests (7) +89 / -6

MSBibExportFormatFilesTest.javaMock keyword separator preference for BibtexImporter in MSBib export tests +5/-1

Mock keyword separator preference for BibtexImporter in MSBib export tests

• Updates test setup to explicitly configure the keyword separator preference when constructing BibtexImporter, aligning with the new import-time keyword normalization behavior.

jablib/src/test/java/org/jabref/logic/exporter/MSBibExportFormatFilesTest.java

BibtexImporterTest.javaAdd importer tests for semicolon keyword import and escaping +41/-1

Add importer tests for semicolon keyword import and escaping

• Adds tests verifying that semicolon-separated keywords are normalized to the configured separator and that embedded configured separators within keywords are escaped correctly during normalization.

jablib/src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java

XmpUtilReaderTest.javaMock keyword separator preference for BibtexImporter in XMP tests +3/-1

Mock keyword separator preference for BibtexImporter in XMP tests

• Adjusts XMP reader test setup to provide a concrete mocked keyword separator via ImportFormatPreferences, matching the importer’s new post-processing dependency.

jablib/src/test/java/org/jabref/logic/xmp/XmpUtilReaderTest.java

KeywordListTest.javaAdd KeywordList.parseImport and serializeWithSpaces unit coverage +37/-0

Add KeywordList.parseImport and serializeWithSpaces unit coverage

• Adds tests for delimiter prioritization (preferring ';' over ',') and hierarchical keyword parsing through parseImport. Adds a test ensuring serializeWithSpaces escapes embedded delimiters while preserving readable spacing.

jablib/src/test/java/org/jabref/model/entry/KeywordListTest.java

MsBibExportFormatTest3.xmlUpdate MSBib golden output for normalized keyword separators +1/-1

Update MSBib golden output for normalized keyword separators

• Updates expected exported XML keywords from semicolon-separated to comma-separated to reflect normalized keyword storage/import behavior used in the test pipeline.

jablib/src/test/resources/org/jabref/logic/exporter/MsBibExportFormatTest3.xml

MsBibExportFormatTest5.xmlUpdate MSBib golden output for normalized keyword separators +1/-1

Update MSBib golden output for normalized keyword separators

• Adjusts expected XML keyword string to use commas instead of semicolons, aligning with keyword normalization behavior.

jablib/src/test/resources/org/jabref/logic/exporter/MsBibExportFormatTest5.xml

MsBibExportFormatTest6.xmlUpdate MSBib golden output for normalized keyword separators +1/-1

Update MSBib golden output for normalized keyword separators

• Changes expected XML keyword delimiter to commas, keeping test expectations consistent with normalized keyword serialization.

jablib/src/test/resources/org/jabref/logic/exporter/MsBibExportFormatTest6.xml

Documentation (2) +16 / -0

CHANGELOG.mdDocument heuristic keyword delimiter detection during BibTeX import +1/-0

Document heuristic keyword delimiter detection during BibTeX import

• Adds a changelog entry describing that imported keywords may be split on ';' or ',' and then normalized to the configured separator.

CHANGELOG.md

import.mdAdd import requirement for keyword delimiter normalization +15/-0

Add import requirement for keyword delimiter normalization

• Introduces a requirements doc stating that BibTeX keyword delimiters are detected (semicolon/comma) and stored using the configured separator while preserving delimiter characters inside keywords via escaping.

docs/requirements/import.md

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Null separator NPE ✓ Resolved 🐞 Bug ☼ Reliability

Description

BibtexImporter#normalizeKeywordDelimiters dereferences the configured keyword separator and passes
it into KeywordList.serializeWithSpaces; if the separator is null, importing any entry with a
keywords field will throw a NullPointerException. This is observable in callers/tests that construct
BibtexImporter with deep-stubbed ImportFormatPreferences that do not stub getKeywordSeparator().

Code

jablib/src/main/java/org/jabref/logic/importer/fileformat/BibtexImporter.java[R133-144]

+    private void normalizeKeywordDelimiters(ParserResult result) {
+        Character separator = importFormatPreferences.bibEntryPreferences().getKeywordSeparator();
+
+        for (BibEntry entry : result.getDatabase().getEntries()) {
+            Optional<String> rawKeywords = entry.getField(StandardField.KEYWORDS);
+            if (rawKeywords.isEmpty()) {
+                continue;
+            }
+
+            KeywordList importedKeywords = KeywordList.parseImport(rawKeywords.get(), IMPORT_KEYWORD_DELIMITERS);
+            entry.setField(StandardField.KEYWORDS, KeywordList.serializeWithSpaces(importedKeywords.stream().toList(), separator));
+        }

Evidence

normalizeKeywordDelimiters reads the keyword separator and passes it to serializeWithSpaces without
null checks; serializeWithSpaces relies on delimiter.toString(), which will NPE if delimiter is
null. There are existing code paths that create BibtexImporter with a deep-stubbed
ImportFormatPreferences and call importDatabase without stubbing the separator, making the NPE
reachable.

jablib/src/main/java/org/jabref/logic/importer/fileformat/BibtexImporter.java[124-145]
jablib/src/main/java/org/jabref/model/entry/KeywordList.java[108-132]
jablib/src/test/java/org/jabref/logic/importer/DatabaseFileLookupTest.java[27-36]
jablib/src/test/java/org/jabref/logic/search/sqlbased/SqlBasedLibrarySearcherWithBibFilesTest.java[101-120]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`BibtexImporter#normalizeKeywordDelimiters` assumes `importFormatPreferences.bibEntryPreferences().getKeywordSeparator()` is non-null and passes it to `KeywordList.serializeWithSpaces`, which ultimately calls `delimiter.toString()`. If the separator is null, import crashes with an NPE when processing entries with keywords.
### Issue Context
This method is now called unconditionally after parsing, so the null separator case is no longer avoidable when keywords exist.
### Fix Focus Areas
- jablib/src/main/java/org/jabref/logic/importer/fileformat/BibtexImporter.java[124-145]
- jablib/src/main/java/org/jabref/model/entry/KeywordList.java[108-132]
### Suggested fix
- Guard against null separators in `normalizeKeywordDelimiters`, e.g. fall back to `','` (or `BibEntryPreferences.getDefault().getKeywordSeparator()`) when `getKeywordSeparator()` returns null.
- Add/adjust a unit test to cover importing with a null/unstubbed separator and ensure import does not crash (and still normalizes).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}

[github-actions]

Your pull request conflicts with the target branch.

Please merge with your code. For a step-by-step guide to resolve merge conflicts, see https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/resolving-a-merge-conflict-using-the-command-line.

[calixtus]

Please consider qodo comments

[mikezhanghaozhe]

Thanks for the reminder and sorry for the delay! I have gone through qodo's comments, and all of them are valid concerns. I am updating the PR by replacing the manual getAsString with KeywordList.serialize.

[subhramit]

Please also look at the failing tests.
If the PR is not ready, please convert to draft.

[mikezhanghaozhe]

Got it! I have changed it to a draft PR. For future case like this, I will convert back to a draft PR if I cannot finish in three or four days.

Thanks for your guidance!

[mikezhanghaozhe]

I found some tests failing. After a brief inspection, I think many previous JUnit tests use bib that contains ";" in the keywords field, and their expected outputs keep ";". This PR will changes the parser in the importing process so that those ";" are replaced, causing the mismatched between expected and actual outputs in the unit tests.

I am a little bit uncertain of how to proceed next. Should I add a flag so that the changes only affect BibtexImporter?

[subhramit]

The keyword normalization during import will henceforth convert the separators to the configured/default separator. So the logic isn't wrong.

[koppor]

@Siedlerchr @koppor MSBibExportFormatFilesTest imports .bib fixture files through BibtexImporter first, then exports them.

Yeah, roundtrip. Should still work...

The keyword normalization during import will henceforth convert the separators to the configured/default separator. So the logic isn't wrong.

Yeah, maybe, the configured separator should be adapted then?

---

JabRef should have a sensible default. Maybe, we need to think at JabCon... Maybe, we should st ";" - or have some auto-detection in place? Meaning: If I as user open a library of someone else, the library should not be trashed...

[subhramit]

I had read up on bibtex default being ,, so didn't mess with that. I'll rename the tests for now.

[subhramit]

MS bib Tests with import+export renamed to round trip, including their xml resources that were being used in them.
Did not touch MsBibExportFormatTest as it was a pure export test.

[subhramit]

I had read up on bibtex default being ,, so didn't mess with that. I'll rename the tests for now.

Refs. plk/biblatex#241
Quoting:

BibDesk the leading FOSS *.bibeditor, defaults to ; as a separator for keywords in some circumstances.

Emphasis on "some circumstances" (although the link to those circumstances returns 404 12 years later).

Also refs. the biblatex standard - https://mirror.niser.ac.in/ctan/macros/latex/contrib/biblatex/doc/biblatex.pdf

---

Yeah, maybe, the configured separator should be adapted then?

If we adapt the separator as per every instance of import, it is no longer a user-configurable preference, and also then there is no concept of a "default".
And no matter how smartly we come up with cases to adapt our "default", there will be cases which we miss - and then due to lack of a fixed configuration it'll lead to unexpected behavior and user will not be in control. That may be frustrating. Especially if the original bib source had mixed delimiters and we try to infer "one" on our own to normalize it into.

As long as we allow it to be configurable, , as the default seems okay to me.

JabRef should have a sensible default. Maybe, we need to think at JabCon... Maybe, we should st ";" - or have some auto-detection in place? Meaning: If I as user open a library of someone else, the library should not be trashed...

It is not trashed - until you export/save.

[Siedlerchr]

Which is our existing default? We should not break users libraries!

[subhramit]

Which is our existing default? We should not break users libraries!

jabref/jablib/src/main/java/org/jabref/model/entry/BibEntryPreferences.java

Lines 10 to 14 in 03e5410

	private BibEntryPreferences() {
	this(
	',' // Keyword separator
	);
	}

[subhramit]

Which is our existing default? We should not break users libraries!

jabref/jablib/src/main/java/org/jabref/model/entry/BibEntryPreferences.java

Lines 10 to 14 in 03e5410

	private BibEntryPreferences() {
	this(
	',' // Keyword separator
	);
	}

Also even if that was not the default, this will not break anything that already exists - but normalize to that default for new imports. If the new import is from a bib file, only on save will it be modified.

[koppor]

Please try to configure JabRef to keep the ; at the MsBibFiles.

Edit: I thought, this PR would enable configuration

Regarding the renaming, I curently don't went into the testing infrastructure. We setup something 10 years ago, but I need to lookup.

[koppor]

Is it really impossible to configure JabRef to keep the ; ?

[koppor]

The ";" seem to be hand-crafted by some MS Word user / tester.

Thus, the change is correct.

Thank you all for the work.