Character encoding is scrambled and truncated

[Mirabel-le-Grand]

Would it be possible to fix the scrambled and deficient character encoding of wpunix? All other WordPerfect versions that I use, including versions 6.0 and 6.2 for DOS, 8 for Windows, and 8.1 for Linux use an identical character encoding scheme (without any difference, as far as I have discerned, other than the change of 4,72 to the Euro sign). While the common Latin letters and basic punctuation characters in wpunix mostly follow the standard encoding, the portability of documents with any amount of extended punctuation, technical symbols, or other alphabets is severely crippled. Opening documents in wpunix that I created in other versions of WordPerfect and vice versa results in mojibake. Amazingly, this faulty character encoding made it into the WordPerfect for UNIX User's Guide (Appendix P, starting on page 721) without any of the developers noticing.

Methods

Since I use technical symbols in my documents, I made a table comparing the mappings from /opt/wp80/wplib/charactrc.tst and CHARACTR.DOC (included with version 6.2 for DOS) and tested them in wpunix. I have corrected it to reflect the aforementioned versions of WordPerfect and updated it with the correct Unicode characters that have been encoded since the 1990s.

Observations

• Character sets 0 (ASCII), 3 (Box Drawing), and 7 (Math/Scientific Extension) are complete.
• Character set 1 (Multinational): The last 8 characters from 1,234 to 1,241 are missing and cannot be entered at all, even by Control+v or Font -> Characters.
• Character set 2 (Phonetic): This set has 28, randomly selected, characters of the 144 characters of the standard character set. None of the character codes match the standard set. Without the missing characters, these are useless for any representation of IPA.
• Character set 4 (Typographic Symbols): The last 17 characters are missing.
• Character set 5 (Iconic Symbols): Only 35 of the 254 standard characters are included. The character codes of four of them do not match the standard set.
• Character set 6 (Math/Scientific): The last three characters are missing.
• Character set 8 (Greek): Only the first 52 character codes match. The order of all the extended characters, including letters with the tonos and dialytika diacritics for monotonic Greek, and all letters with polytonic diacritics, is scrambled. Key characters needed for both orthographies are missing or have character names that do not match either the coded character or the standard encoding. This character set is unsuitable for either monotonic or polytonic Greek.
• Character set 9 (Hebrew): Only the first 44 characters match the standard character set. Only 12 of the cantillation marks are included, but their names do not match the encoded characters, and they are useless without the missing cantillation marks. The extended Hebrew characters are also missing.
• Character set 10 (Cyrillic): Only the letters of the basic Russian alphabet match those of the standard character set. All the other characters, required for Old Church Slavonic, Slavic, Turkic, Uralic, and other languages, are in a scrambled order and do not match the standard codes. Many key characters are missing. This character set is not suited for all Cyrillic alphabets other than basic Russian.
• Character set 11 (Japanese Kana): This is the only character set to include more characters than the standard set (e.g.: adding hiragana and precomposed dakuten and handakuten). Not one of the characters in this set, however, matches the standard encoding.
• Character sets 13 (Arabic 1) and 14 (Arabic Script) are absent.

Conclusions

I think that the idiosyncratic character encoding of wpunix was botched rather than a result of technical limitation or intention. This is evidenced by its departure from the standard followed by other version of WordPerfect, haphazard omission of key characters, illogical re-orderings, and internally contradicting character names.

1. I think that the character encoding of wpunix should be modified to follow the standard used across other versions of WordPerfect. Symbol insertion (via system keyboard entry, Control+v, and Font -> Characters) and importation of Unicode text and its proper conversion to the standard WordPerfect encoding should be assured.
2. WPunix should now display more WordPerfect characters in the terminal. Many extended characters are substituted in the terminal display by unaccented letters or squares. More sophisticated terminals, expansions to Unicode, and better terminal display fonts since the 1990s means that more characters should display in the interface, including those represented by Unicode as base letters with combining diacritics.
3. I would happily share my conversion table to assist such an effort. It would also be useful to those using Unicode fonts in WordPerfect, editing printer drivers, or converting documents. I am new to Github: perhaps a collaboratively edited version could be hosted here?

It would be awesome to use WordPerfect in a character-based terminal like that of DOS, but free from the constraints of 8-bit code pages and utilising my custom Linux input methods.

[emendelson]

The character set mapping was not botched. All wpunix versions starting with 5.1 use the WP 5.1 character set, which is different from the 6.x character set in the ways that you describe. You can find that character set in the WPDOS AND WPWin 5.X printed manuals. And you can see the unicode equivalents of the 5.1 and 6.x sets in this file from the libwpd project, which has already done much of the work that you propose:

https://www.wpuniverse.com/vb/filedata/fetch?id=447032&d=1701004235

The original file is supposed to be on SourceForge, but it times out when I try to access it:

https://sourceforge.net/p/libwpd/code/ci/master/tree/src/lib/libwpd_internal.cpp

Wpunix will save files in the WP 6.x format (though without the characters that the 5.1 character set does not support) if you use Ctrl-F5 and Save As WordPerfect 6x-7.

Whether it is possible to update wpunix to use the 6.x character set natively is something only Tavis can tell us.

EDIT: To avoid the problem that you describe when opening files in wpunix, save the files from other WP versions in WP 5.1 format, and they will open correctly in wpunix. If you routinely open files in wpunix, you may want to set the default save format in other WP versions as WP 5.1.

[Mirabel-le-Grand]

Thank you very much, emendelson, for your most helpful explanation. (If you are the one behind the https://mendelson.org/wpdos/ website, I would like to thank you for that work also.) I have been using WordPerfect since version 6.0 and was unaware of the 5.1 way. Thankfully, WordPerfect addressed many of the deficiencies of the 5.1 character sets in 6.x. Comparing the printed manuals, however, wpunix still leaves out some characters included in 5.1.

The libwpd project looks fantastic! I downloaded the code from the Sorceforge site. On my preliminary viewing of libwpd_internal.cpp, it seems that it inherited the incorrect substitutions and lacunae of CHARACTR.DOC reflective of a much more primitive state of the Unicode standard. I would like to join the libwpd project and help overhaul the conversion tables to bring them in line with correct Unicode usage.

I still would like to make the request that I made in number 2 above. WPunix substitutes unaccented letters or squares in the terminal display for many of the characters. Would it be possible to make it display the correct characters?

[emendelson]

Anyone can post a "ticket" at the SourceForge site:

https://sourceforge.net/p/libwpd/tickets/

I don't think there is a lot of activity there, but it can't hurt to ask for fixes!

[Mirabel-le-Grand]

Sourceforge is telling me, "To create a new ticket, you must be authorized by the project admin.". It looks like I have to join the libwpd project to submit my list of corrections.

[emendelson]

Any errors in libwpd will also be in LibreOffice. You can also file a bug report there, referrring to LibreOffice itself. LibreOffice uses libwpd, so any correction to LibreOffice will be made by changing libwpd.

[taviso]

I still would like to make the request that I made in number 2 above. WPunix substitutes unaccented letters or squares in the terminal display for many of the characters. Would it be possible to make it display the correct characters?

Do you have a way to reproduce the problem? I don't fully understand, but there may be errors from when I converted the libwpd tables - I didn't verify every single entry was correct.

Regarding adding new characters, you can add them to set 12, but it's sort of a complicated process... I guess we can improve that, but there'll be a limit of 256 new characters... I dunno, is that good enough? It's probably not easy to add full utf-8 support, although it might be possible with some hacking.

[Mirabel-le-Grand]

After looking at the libwpd files on the Sourceforge site, I think that libwpd has deficiencies rather than there being a mistake in the conversion of libwpd. Character codes can be entered into a document in wpunix using control+v. For each WordPerfect character entered, the terminal display may do one of the following:

1. display the correct Unicode character that matches the WordPerfect character. Example: 1,40 (E Acute) is displayed as U+00C9 LATIN CAPITAL LETTER E WITH ACUTE.
2. display an incorrect substitute. Example: 1,25 (Dotless j) is displayed as U+006A LATIN SMALL LETTER J when it should be displayed as U+0237 LATIN SMALL LETTER DOTLESS J.
3. display an unaccented base letter when it should display as a combination of two or more Unicode characters. Example 1,212 (D Macron) is displayed as U+0044 LATIN CAPITAL LETTER D when it should display as U+0044 U+0304 LATIN CAPITAL LETTER D, COMBINING MACRON. These WordPerfect characters do not have precomposed Unicode counterparts and must be represented in Unicode as a base letter plus one or more combining diacritics.
4. display a square. Example: 8,208 UPSILON Diaeresis displays as a square when it should display as U+03AB GREEK CAPITAL LETTER UPSILON WITH DIALYTIKA.

In cases 2, 3, and 4, the Unicode character that is displayed in the terminal should be changed. Also, the correct translation of characters entered via the keyboard to the correct WordPerfect characters sould be assured. If you give me time, I'll test the characters in wpunix and submit a list of WP character codes with their correct Unicode representations.

[taviso]

Hmm, I see what you mean, that does look like an error inherited from libwpd. Okay, thanks, if you're willing to compile a list of errors I see no reason we can't import them!

[emendelson]

The dotless-j error is also in LibreOffice, but (I think) the upsilon-diaresis is correct in LibreOffice. Both seem to be correct in vDosWP, which has its own routines (based on modified libwpd) for displaying the WP character set.

[Mirabel-le-Grand]

WP5x_Characters_to_Unicode.txt

WP6x_Characters_to_Unicode.txt

WP5x_to_WP6x_Character_Conversion.txt

Please find attached three files:

1. WordPerfect 5.x Characters to Unicode Conversion Table
2. WordPerfect 6.x Characters to Unicode Conversion Table
3. WordPerfect 5.x to WordPerfect 6.x Conversion Table

These are very much works in progress, still having many lacunae and, undoubtedly, mistakes. Nevertheless, I have addressed many of the glaring shortcomings of the LIBWPD files. I designed them to be machine-readable to facilitate their implementation.

If WPUNIX could be modified to use the WP6.x character encoding natively, that would be fantastic! As far as I can tell, the Latin letters, punctuation, and box drawing elements used for the available WPUNIX interface languages (Dutch, English, French, German, Italian, and Spanish) are the same in WP5.x and WP6.x encodings.

[Mirabel-le-Grand]

I would gladly contribute to WPUNIX by making the modifications to the character code files myself. It would be a pleasure to thus share the knowledge that I have gained from the book Working with Foreign Languages and Characters in WordPerfect 5.1 by Peter Kahrel, Unicode technical documentation, and WordPerfect's graphic fonts. My coding skills are rudimentary, however, so I would need a modicum of guidance that I hope will not be too burdensome. My questions are the following:

Character display in the terminal:

1. Where does the table live that tells the terminal interface what Unicode characters to display? Is it charset.c?
2. Some WordPerfect characters need to be displayed in the terminal as a combination of more than one Unicode character: often a base letter followed by one or more combining (nonspacing) diacritics. This Unicode combination representing one WordPerfect character needs to be displayed in one character width on the display grid. Can WPUNIX do this?

Keyboard entry when typing characters into a document:

3. Where does the table live that converts Unicode text typed from the system keyboard into a WordPerfect document? Is it charmap.ini?
4. In some cases, two or more Unicode alternatives converge on one WordPerfect character. An example of this is Unicode's distinction between Greek vowels with a tonos diacritic versus vowels with an acute accent (oxia). No matter if the typist is using a monotonic Greek keyboard entering vowels with a tonos, or a polytonic keyboard entering vowels with an oxia, both should converge on the same WordPerfect Greek characters with "acute accent". Thus, the relevant file will need two or more input entries for some WordPerfect characters. Can WPUNIX do that?
5. There are instances of single Unicode characters that, when typed from the keyboard, need to be converted to a pair of WordPerfect characters. For example, the Old Cyrillic iotified ligatures are encoded by Unicode as single codepoints, but would need to be converted to two WordPerfect characters, the i ligature followed by the base vowel, when typed into a document. Can WPUNIX do that?
6. Would it be possible to convert a typed sequence of two Unicode characters into one WordPerfect character? (This is the keyboard entry application of question 2 and the inverse of question 5.) Examples are Unicode pairs of a Cyrillic letter followed by a combining (nonspacing) acute or grave diacritic. When typed from the keyboard, these would need to be converted to a single WordPerfect character representing the precomposed Cyrillic letter with diacritic.

Thank you so much for your hard work on this awesome character-based version of WordPerfect! Sorting out the character input and display issues will add much value.

[taviso]

I had never heard of that book, interesting!

Character display in the terminal:

1. Where does the table live that tells the terminal interface what Unicode characters to display? Is it charset.c?

It's the terminal definition files e.g. xterm.trs. I generated these with code from the libwpd tables, but you can actually modify them manually without code -- but honestly, the editor is not great and takes some patience to use! If you want to try it, I would do this:

$ cd
$ cp /opt/wp80/shlib10/xterm.trs testing.trs
$ /opt/wp80/shbin10/wpterm

Make whatever edits you want to testing, then copy the testing.trs file back to /opt/wp80/shlib10, then to use it do something like WPTERM=testing wp. It sounds like you might enjoy exploring the options, if you do make some improvements we can just integrate a new trs file.

2. Some WordPerfect characters need to be displayed in the terminal as a combination of more than one Unicode character: often a base letter followed by one or more combining (nonspacing) diacritics. This Unicode combination representing one WordPerfect character needs to be displayed in one character width on the display grid. Can WPUNIX do this?

Yes, you can do this in wpterm - it can be configured to output an arbitrary byte sequence, which can just be a sequence of UTF-8 codepoints. There are already some characters I do this for, so you can browse through the existing definitions for examples. If you need unprintable bytes you enter things like [194][123].

Keyboard entry when typing characters into a document:

3. Where does the table live that converts Unicode text typed from the system keyboard into a WordPerfect document? Is it charmap.ini?

No, charmap is for printing. It's the terminal definitions that handle that.

5. In some cases, two or more Unicode alternatives converge on one WordPerfect character. An example of this is Unicode's distinction between Greek vowels with a tonos diacritic versus vowels with an acute accent (oxia). No matter if the typist is using a monotonic Greek keyboard entering vowels with a tonos, or a polytonic keyboard entering vowels with an oxia, both should converge on the same WordPerfect Greek characters with "acute accent". Thus, the relevant file will need two or more input entries for some WordPerfect characters. Can WPUNIX do that?

Hmm, I suppose that depends on what you want to happen if someone enters a partial sequence. I think it is probably possible.

7. There are instances of single Unicode characters that, when typed from the keyboard, need to be converted to a pair of WordPerfect characters. For example, the Old Cyrillic iotified ligatures are encoded by Unicode as single codepoints, but would need to be converted to two WordPerfect characters, the i ligature followed by the base vowel, when typed into a document. Can WPUNIX do that?
8. Would it be possible to convert a typed sequence of two Unicode characters into one WordPerfect character? (This is the keyboard entry application of question 2 and the inverse of question 5.) Examples are Unicode pairs of a Cyrillic letter followed by a combining (nonspacing) acute or grave diacritic. When typed from the keyboard, these would need to be converted to a single WordPerfect character representing the precomposed Cyrillic letter with diacritic.>

I think there are not directly possible with the terminal definition... but we can do it with a macro for additional language support (for example, take a look at emdash.mac, which converts the sequence -- into a typographical em-dash).

11. Thank you so much for your hard work on this awesome character-based version of WordPerfect! Sorting out the character input and display issues will add much value.

Thanks for your interest!