Skip to content

Commit 1732c6e

Browse files
authored
Make GenerateConfusables generate NFKD instead of NFKC (#1205)
* Make GenerateConfusables generate NFKD instead of NFKC * Also fix a typo in the name of the method and use the singleton pattern This fixes problems described in the following issue: - unicode-org/properties#459
1 parent b3333b8 commit 1732c6e

18 files changed

+624
-608
lines changed

unicodetools/data/security/dev/IdentifierStatus.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
# IdentifierStatus.txt
2-
# Date: 2025-08-01, 18:11:48 GMT
2+
# Date: 2025-09-12, 03:25:00 GMT
33
# © 2025 Unicode®, Inc.
44
# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
55
# For terms of use and license, see https://www.unicode.org/terms_of_use.html
66
#
77
# Unicode Security Mechanisms for UTS #39
8-
# Version: 17.0.0
8+
# Version: 18.0.0
99
#
1010
# For documentation and usage, see https://www.unicode.org/reports/tr39
1111
#

unicodetools/data/security/dev/IdentifierType.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
# IdentifierType.txt
2-
# Date: 2025-08-04, 21:58:43 GMT
2+
# Date: 2025-09-12, 03:24:49 GMT
33
# © 2025 Unicode®, Inc.
44
# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
55
# For terms of use and license, see https://www.unicode.org/terms_of_use.html
66
#
77
# Unicode Security Mechanisms for UTS #39
8-
# Version: 17.0.0
8+
# Version: 18.0.0
99
#
1010
# For documentation and usage, see https://www.unicode.org/reports/tr39
1111
#

unicodetools/data/security/dev/confusables.txt

Lines changed: 94 additions & 96 deletions
Large diffs are not rendered by default.

unicodetools/data/security/dev/confusablesSummary.txt

Lines changed: 292 additions & 294 deletions
Large diffs are not rendered by default.

unicodetools/data/security/dev/data/confusablesSummaryIdentifier.txt

Lines changed: 40 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
# confusablesSummaryIdentifier.txt
2-
# Date: 2025-07-22, 05:49:37 GMT
2+
# Date: 2025-09-12, 03:24:49 GMT
33
# © 2025 Unicode®, Inc.
44
# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
55
# For terms of use and license, see https://www.unicode.org/terms_of_use.html
66
#
77
# Unicode Security Mechanisms for UTS #39
8-
# Version: 17.0.0
8+
# Version: 18.0.0
99
#
1010
# For documentation and usage, see https://www.unicode.org/reports/tr39
1111
#
@@ -69,21 +69,25 @@
6969
← (‎ І ‎) 0406 CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I # →I→
7070
← (‎ Ӏ ‎) 04C0 CYRILLIC LETTER PALOCHKA # →I→
7171

72-
# lٕ 1ٕ إ
72+
# lٴ lٔ ٴl 1ٔ 1ٴ أ اٴ ٴ1 ٴا ٲ أ
73+
(‎ 1ٔ ‎) 0031 0654 DIGIT ONE, ARABIC HAMZA ABOVE
74+
← (‎ lٴ ‎) 006C 0674 LATIN SMALL LETTER L, ARABIC LETTER HIGH HAMZA # →‎اٴ‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
75+
← (‎ lٔ ‎) 006C 0654 LATIN SMALL LETTER L, ARABIC HAMZA ABOVE # →‎أ‎→
76+
← (‎ ٴl ‎) 0674 006C ARABIC LETTER HIGH HAMZA, LATIN SMALL LETTER L # →‎ٴا‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
77+
← (‎ 1ٴ ‎) 0031 0674 DIGIT ONE, ARABIC LETTER HIGH HAMZA # →‎اٴ‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
78+
← (‎ أ ‎) 0627 0654 ARABIC LETTER ALEF, ARABIC HAMZA ABOVE
79+
← (‎ اٴ ‎) 0627 0674 ARABIC LETTER ALEF, ARABIC LETTER HIGH HAMZA # →‎ٵ‎→→‎ﺃ‎→→‎أ‎→
80+
← (‎ ٴ1 ‎) 0674 0031 ARABIC LETTER HIGH HAMZA, DIGIT ONE # →‎ٴا‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
81+
← (‎ ٴا ‎) 0674 0627 ARABIC LETTER HIGH HAMZA, ARABIC LETTER ALEF # →‎ٵ‎→→‎ﺃ‎→→‎أ‎→
82+
← (‎ ٲ ‎) 0672 ARABIC LETTER ALEF WITH WAVY HAMZA ABOVE # →‎أ‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
83+
← (‎ أ ‎) 0623 ARABIC LETTER ALEF WITH HAMZA ABOVE # →‎ٵ‎→→‎ﺃ‎→→‎أ‎→
84+
85+
# lٕ 1ٕ إ إ
7386
(‎ 1ٕ ‎) 0031 0655 DIGIT ONE, ARABIC HAMZA BELOW
7487
← (‎ lٕ ‎) 006C 0655 LATIN SMALL LETTER L, ARABIC HAMZA BELOW # →‎اٟ‎→
88+
← (‎ إ ‎) 0627 0655 ARABIC LETTER ALEF, ARABIC HAMZA BELOW # →lٕ→→‎اٟ‎→
7589
← (‎ إ ‎) 0625 ARABIC LETTER ALEF WITH HAMZA BELOW # →‎ٳ‎→→‎اٟ‎→
7690

77-
# lٴ ٴl 1ٴ اٴ ٴ1 ٴا ٲ أ
78-
(‎ 1ٴ ‎) 0031 0674 DIGIT ONE, ARABIC LETTER HIGH HAMZA
79-
← (‎ lٴ ‎) 006C 0674 LATIN SMALL LETTER L, ARABIC LETTER HIGH HAMZA # →‎اٴ‎→
80-
← (‎ ٴl ‎) 0674 006C ARABIC LETTER HIGH HAMZA, LATIN SMALL LETTER L # →‎ٴا‎→→‎ٵ‎→→‎اٴ‎→
81-
← (‎ اٴ ‎) 0627 0674 ARABIC LETTER ALEF, ARABIC LETTER HIGH HAMZA
82-
← (‎ ٴ1 ‎) 0674 0031 ARABIC LETTER HIGH HAMZA, DIGIT ONE # →‎ٴا‎→→‎ٵ‎→→‎اٴ‎→
83-
← (‎ ٴا ‎) 0674 0627 ARABIC LETTER HIGH HAMZA, ARABIC LETTER ALEF # →‎ٵ‎→→‎اٴ‎→
84-
← (‎ ٲ ‎) 0672 ARABIC LETTER ALEF WITH WAVY HAMZA ABOVE # →‎أ‎→→‎ٵ‎→→‎اٴ‎→
85-
← (‎ أ ‎) 0623 ARABIC LETTER ALEF WITH HAMZA ABOVE # →‎ٵ‎→→‎اٴ‎→
86-
8791
# 3 ३ ૩ З Ӡ
8892
(‎ 3 ‎) 0033 DIGIT THREE
8993
← (‎ ३ ‎) 0969 DEVANAGARI DIGIT THREE # →૩→
@@ -361,6 +365,12 @@
361365
(‎ ô ‎) 006F 0302 LATIN SMALL LETTER O, COMBINING CIRCUMFLEX ACCENT
362366
← (‎ ۿ ‎) 06FF ARABIC LETTER HEH WITH INVERTED V # →‎ھٛ‎→
363367

368+
# oٔ هٔ ۀ ۂ
369+
(‎ oٔ ‎) 006F 0654 LATIN SMALL LETTER O, ARABIC HAMZA ABOVE
370+
← (‎ هٔ ‎) 0647 0654 ARABIC LETTER HEH, ARABIC HAMZA ABOVE # →‎ۀ‎→
371+
← (‎ ۀ ‎) 06D5 0654 ARABIC LETTER AE, ARABIC HAMZA ABOVE
372+
← (‎ ۂ ‎) 06C2 ARABIC LETTER HEH GOAL WITH HAMZA ABOVE # →‎ﮤ‎→→‎ۀ‎→
373+
364374
# oာ ဝာ တ თ
365375
(‎ oာ ‎) 006F 102C LATIN SMALL LETTER O, MYANMAR VOWEL SIGN AA
366376
← (‎ ဝာ ‎) 101D 102C MYANMAR LETTER WA, MYANMAR VOWEL SIGN AA
@@ -610,16 +620,21 @@
610620
(‎ ձ ‎) 0571 ARMENIAN SMALL LETTER JA
611621
← (‎ ኔ ‎) 1294 ETHIOPIC SYLLABLE NEE
612622

613-
# وٴ ٴو ؤ
623+
# وٴ ؤ ٴو ؤ
614624
(‎ ؤ ‎) 0624 ARABIC LETTER WAW WITH HAMZA ABOVE
615625
← (‎ وٴ ‎) 0648 0674 ARABIC LETTER WAW, ARABIC LETTER HIGH HAMZA # →‎ٶ‎→
626+
← (‎ ؤ ‎) 0648 0654 ARABIC LETTER WAW, ARABIC HAMZA ABOVE # →‎ﺅ‎→→‎ٶ‎→
616627
← (‎ ٴو ‎) 0674 0648 ARABIC LETTER HIGH HAMZA, ARABIC LETTER WAW # →‎ٶ‎→
617628

618-
# ىٴ يٴ ٴى ئ
629+
# ىٴ ىٔ ئ يٴ ٴى ۓ ࢨ ئ
619630
(‎ ئ ‎) 0626 ARABIC LETTER YEH WITH HAMZA ABOVE
620631
← (‎ ىٴ ‎) 0649 0674 ARABIC LETTER ALEF MAKSURA, ARABIC LETTER HIGH HAMZA # →‎يٴ‎→→‎ٸ‎→
632+
← (‎ ىٔ ‎) 0649 0654 ARABIC LETTER ALEF MAKSURA, ARABIC HAMZA ABOVE # →‎ئ‎→→‎ﺉ‎→→‎ٸ‎→
633+
← (‎ ئ ‎) 064A 0654 ARABIC LETTER YEH, ARABIC HAMZA ABOVE # →‎ﺉ‎→→‎ٸ‎→
621634
← (‎ يٴ ‎) 064A 0674 ARABIC LETTER YEH, ARABIC LETTER HIGH HAMZA # →‎ٸ‎→
622635
← (‎ ٴى ‎) 0674 0649 ARABIC LETTER HIGH HAMZA, ARABIC LETTER ALEF MAKSURA # →‎ٸ‎→
636+
← (‎ ۓ ‎) 06D2 0654 ARABIC LETTER YEH BARREE, ARABIC HAMZA ABOVE # →‎ىٔ‎→→‎ئ‎→→‎ﺉ‎→→‎ٸ‎→
637+
← (‎ ࢨ ‎) 08A8 ARABIC LETTER YEH WITH TWO DOTS BELOW AND HAMZA ABOVE # →‎ئ‎→→‎ﺉ‎→→‎ٸ‎→
623638

624639
# ت ٺ
625640
(‎ ت ‎) 062A ARABIC LETTER TEH
@@ -715,11 +730,6 @@
715730
(‎ ٹ ‎) 0679 ARABIC LETTER TTEH
716731
← (‎ ڻ ‎) 06BB ARABIC LETTER RNOON # →‎ﮢ‎→→‎ﭨ‎→
717732

718-
# ىٔ ئ ࢨ
719-
(‎ ىٔ ‎) 0649 0654 ARABIC LETTER ALEF MAKSURA, ARABIC HAMZA ABOVE
720-
← (‎ ئ ‎) 064A 0654 ARABIC LETTER YEH, ARABIC HAMZA ABOVE
721-
← (‎ ࢨ ‎) 08A8 ARABIC LETTER YEH WITH TWO DOTS BELOW AND HAMZA ABOVE # →‎ئ‎→
722-
723733
# پ̆ ࢾ
724734
(‎ پ̆ ‎) 067E 0306 ARABIC LETTER PEH, COMBINING BREVE
725735
← (‎ ࢾ ‎) 08BE ARABIC LETTER PEH WITH SMALL V # →‎پٚ‎→
@@ -761,10 +771,6 @@
761771
(‎ چ̆ ‎) 0686 0306 ARABIC LETTER TCHEH, COMBINING BREVE
762772
← (‎ ࣁ ‎) 08C1 ARABIC LETTER TCHEH WITH SMALL V # →‎چٚ‎→
763773

764-
# ۀ ۂ
765-
(‎ ۀ ‎) 06C0 ARABIC LETTER HEH WITH YEH ABOVE
766-
← (‎ ۂ ‎) 06C2 ARABIC LETTER HEH GOAL WITH HAMZA ABOVE # →‎ﮤ‎→
767-
768774
# ݧ ࢩ
769775
(‎ ݧ ‎) 0767 ARABIC LETTER NOON WITH TWO DOTS BELOW
770776
← (‎ ࢩ ‎) 08A9 ARABIC LETTER YEH WITH TWO DOTS BELOW AND DOT ABOVE # →‎ݔ‎→
@@ -823,10 +829,10 @@
823829
(‎ उ ‎) 0909 DEVANAGARI LETTER U
824830
← (‎ ਤ ‎) 0A24 GURMUKHI LETTER TA
825831

826-
# एॅ ए̆
832+
# ए̆ एॅ
827833
(‎ ऍ ‎) 090D DEVANAGARI LETTER CANDRA E
828-
← (‎ एॅ ‎) 090F 0945 DEVANAGARI LETTER E, DEVANAGARI VOWEL SIGN CANDRA E
829834
← (‎ ए̆ ‎) 090F 0306 DEVANAGARI LETTER E, COMBINING BREVE # →एॅ→
835+
← (‎ एॅ ‎) 090F 0945 DEVANAGARI LETTER E, DEVANAGARI VOWEL SIGN CANDRA E
830836

831837
# एॆ ऎ
832838
(‎ ऎ ‎) 090E DEVANAGARI LETTER SHORT E
@@ -956,7 +962,7 @@
956962
(‎ র ‎) 09B0 BENGALI LETTER RA
957963
← (‎ ৰ ‎) 09F0 BENGALI LETTER RA WITH MIDDLE DIAGONAL
958964

959-
# ਅੈ ਅै
965+
# ਅै ਅੈ
960966
(‎ ਅै ‎) 0A05 0948 GURMUKHI LETTER A, DEVANAGARI VOWEL SIGN AI
961967
← (‎ ਅੈ ‎) 0A05 0A48 GURMUKHI LETTER A, GURMUKHI VOWEL SIGN AI
962968
← (‎ ਐ ‎) 0A10 GURMUKHI LETTER AI # →ਅੈ→
@@ -1086,15 +1092,15 @@
10861092
(‎ ெள ‎) 0BC6 0BB3 TAMIL VOWEL SIGN E, TAMIL LETTER LLA
10871093
← (‎ ௌ ‎) 0BCC TAMIL VOWEL SIGN AU
10881094

1089-
# െെ ெெ
1095+
# ெெ െെ
10901096
(‎ ெெ ‎) 0BC6 0BC6 TAMIL VOWEL SIGN E, TAMIL VOWEL SIGN E
10911097
← (‎ െെ ‎) 0D46 0D46 MALAYALAM VOWEL SIGN E, MALAYALAM VOWEL SIGN E
10921098
← (‎ ൈ ‎) 0D48 MALAYALAM VOWEL SIGN AI # →െെ→
10931099

1094-
# എെ ெഎ എெ െഎ ഐ
1100+
# ெഎ എெ എെ െഎ ഐ
10951101
(‎ ெഎ ‎) 0BC6 0D0E TAMIL VOWEL SIGN E, MALAYALAM LETTER E
1096-
← (‎ എെ ‎) 0D0E 0D46 MALAYALAM LETTER E, MALAYALAM VOWEL SIGN E # →ഐ→→െഎ→
10971102
← (‎ എெ ‎) 0D0E 0BC6 MALAYALAM LETTER E, TAMIL VOWEL SIGN E # →എെ→→ഐ→→െഎ→
1103+
← (‎ എെ ‎) 0D0E 0D46 MALAYALAM LETTER E, MALAYALAM VOWEL SIGN E # →ഐ→→െഎ→
10981104
← (‎ െഎ ‎) 0D46 0D0E MALAYALAM VOWEL SIGN E, MALAYALAM LETTER E
10991105
← (‎ ഐ ‎) 0D10 MALAYALAM LETTER AI # →െഎ→
11001106

@@ -1260,7 +1266,7 @@
12601266
(‎ ര് ‎) 0D30 0D4D MALAYALAM LETTER RA, MALAYALAM SIGN VIRAMA
12611267
← (‎ ർ ‎) 0D7C MALAYALAM LETTER CHILLU RR # →൪→
12621268

1263-
# ဂာ രာ က
1269+
# രာ ဂာ က
12641270
(‎ രာ ‎) 0D30 102C MALAYALAM LETTER RA, MYANMAR VOWEL SIGN AA
12651271
← (‎ ဂာ ‎) 1002 102C MYANMAR LETTER GA, MYANMAR VOWEL SIGN AA
12661272
← (‎ က ‎) 1000 MYANMAR LETTER KA # →ဂာ→
@@ -1271,7 +1277,7 @@
12711277
← (‎ က္က ‎) 1000 1039 1000 MYANMAR LETTER KA, MYANMAR SIGN VIRAMA, MYANMAR LETTER KA
12721278
← (‎ ဣ ‎) 1023 MYANMAR LETTER I # →က္က→
12731279

1274-
# ဂှ രှ
1280+
# രှ ဂှ
12751281
(‎ രှ ‎) 0D30 103E MALAYALAM LETTER RA, MYANMAR CONSONANT SIGN MEDIAL HA
12761282
← (‎ ဂှ ‎) 1002 103E MYANMAR LETTER GA, MYANMAR CONSONANT SIGN MEDIAL HA
12771283
← (‎ ႁ ‎) 1081 MYANMAR LETTER SHAN HA # →ဂှ→
@@ -1446,7 +1452,7 @@
14461452
(‎ သြ ‎) 101E 103C MYANMAR LETTER SA, MYANMAR CONSONANT SIGN MEDIAL RA
14471453
← (‎ ဩ ‎) 1029 MYANMAR LETTER O
14481454

1449-
# သြော် သြେာ် ဩော် ဪ
1455+
# သြେာ် သြော် ဩော် ဪ
14501456
(‎ သြେာ် ‎) 101E 103C 0B47 102C 103A MYANMAR LETTER SA, MYANMAR CONSONANT SIGN MEDIAL RA, ORIYA VOWEL SIGN E, MYANMAR VOWEL SIGN AA, MYANMAR SIGN ASAT
14511457
← (‎ သြော် ‎) 101E 103C 1031 102C 103A MYANMAR LETTER SA, MYANMAR CONSONANT SIGN MEDIAL RA, MYANMAR VOWEL SIGN E, MYANMAR VOWEL SIGN AA, MYANMAR SIGN ASAT # →ဩော်→
14521458
← (‎ ဩော် ‎) 1029 1031 102C 103A MYANMAR LETTER O, MYANMAR VOWEL SIGN E, MYANMAR VOWEL SIGN AA, MYANMAR SIGN ASAT
@@ -1464,5 +1470,5 @@
14641470
(‎ へ ‎) 3078 HIRAGANA LETTER HE
14651471
← (‎ ヘ ‎) 30D8 KATAKANA LETTER HE
14661472

1467-
# total : 518
1473+
# total : 527
14681474

0 commit comments

Comments
 (0)