Convert locale files from ini to json format

[lunny]

• Migrate from the current INI format to JSON for translations. JSON is widely supported, including by platforms such as Crowdin and Weblate.
• Added a check for the json format and duplicate keys

As a next step, we can consider converting to the i18next JSON format, which allows using placeholders like {name} instead of the current %[1]s syntax.

All original ini files have been converted with go run ./tools/convert_locale_ini_json.go

[silverwind]

Very good choice.

[wxiaoguang]

TBH I won't say it is a good choice. Full screen of \"

[wxiaoguang]

By the way, you also need to fix

• https://github.com/go-gitea/gitea/blob/main/.github/workflows/cron-translations.yml
• ./build/update-locales.sh

[silverwind]

Full screen of "

We should eventually remove all HTML from translations which will solve this. There are always ways to interpolate translation into templates without having HTML in the translation itself.

[wxiaoguang]

Full screen of "

We should eventually remove all HTML from translations which will solve this. There are always ways to interpolate translation into templates without having HTML in the translation itself.

The "eventually has been years, many "eventually" things have been years.

[hiifong]

I believe switching to TOML would be preferable, as existing translations can be easily converted to the TOML format, Nor will you see a screen full of \"

[silverwind]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

[wxiaoguang]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

Yes, any well-defined format is a big step.

Still think it should clean up the legacy HTML quotes before migrating, to make the code base clean and avoid low-level mistakes.

[hiifong]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

[wxiaoguang]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

Agree, that's also a problem.

[silverwind]

YAML has comments. Does crowdin support it? Thought I see no pressing need to comment translations.

[hiifong]

YAML has comments. Does crowdin support it? Thought I see no pressing need to comment translations.

Support, and also support TOML format.
ref: https://store.crowdin.com/categories/file-formats

[hiifong]

ref: #32937
I believe we should add comments in this situation.

[lunny]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

Why do we need comments for translation keys? Currently, almost none of the existing translations include comments. Although JSON has it's drawback, but the format is parsed well by all kinds of languages and platforms like crowdin, weblate and others.

I have a plan to replace all %s to {user} in the translation strings, I think it's also a suitable time to replace all the HTML tags at the same PR. Otherwise, we have to replace to %s at this PR and replace it again next time.

[hiifong]

Still I think JSON is a big step forward over INI. It has clear and well-known escaping rules, and any mistake will be found by any JSON parser (which should run on the CI, and this is trivial to add).

One drawback of JSON format is that it does not support adding comments.

Why do we need comments for translation keys? Currently, almost none of the existing translations include comments. Although JSON has it's drawback, but the format is parsed well by all kinds of languages and platforms like crowdin, weblate and others.

I have a plan to replace all %s to {user} in the translation strings, I think it's also a suitable time to replace all the HTML tags at the same PR. Otherwise, we have to replace to %s at this PR and replace it again next time.

So how do you plan to replace the HTML tags?

[lunny]

I believe switching to TOML would be preferable, as existing translations can be easily converted to the TOML format, Nor will you see a screen full of \"

TOML is also a possible choice but it seems weblate haven't support it.
Replace the html tags one by one manually. There are many examples in the template files.

A drawback of the YAML format is that incorrect indentation can easily lead to invalid or unintended keys. i.e.
error: a
vs

form:
     error: a

[silverwind]

I think JSON is a good choice. TOML and YAML are both more much complex specs. I think the downside of having to escape " is acceptable.

[lunny]

It's ready to review again.

[denyskon]

This will definitely be problematic with Crowdin. It doesn't simply support changing the file format, this will be as if we uploaded a completely new set of strings IIRC

[lunny]

This will definitely be problematic with Crowdin. It doesn't simply support changing the file format, this will be as if we uploaded a completely new set of strings IIRC

Yes, that the plan. Once this is merged, we can uploaded all the languages files and remove ini format files.

[denyskon]

The we'll loose all translations which are created but not proofread ...

[denyskon]

What do we do with cases like these?

[lunny]

What do we do with cases like these?

Maybe we can approve all the changes manually before the convention.

[silverwind]

==> options/locale/locale_cs-CZ.json
❌ Invalid JSON syntax: options/locale/locale_cs-CZ.json

Empty file is not valid json, you need to write {} or detect empty file.

[wxiaoguang]

I am sure that we should not introduce more fragile shell scripts into the Makefile.

So many $$ escaping, so many \\ multiple line string

And, ❌ and ✅ are written by AI? When we started using such characters in Makefile/shell scripts ....

[denyskon]

Blocking until we've dealt with the Crowdin question