View Issue Details

IDProjectCategoryView StatusLast Update
0006543The Dark ModGUIpublic30.08.2024 20:52
ReporterGeep Assigned To 
PrioritynormalSeveritynormalReproducibilityN/A
Status newResolutionopen 
Product VersionTDM 2.12 
Summary0006543: Change TDM codemap to add 2 new characters
DescriptionThe current TDM char set has a redundant treatment of 2 characters:
    Ô appears at both 0x88 and 0xD4
    ô appears at both 0x98 and 0xF4

It is proposed to end this redundancy, allowing 2 new characters, namely Ğ/ğ, with breve (cup), where
    Ğ uses TDM codepoint 0x88
    ğ uses TDM codepoint 0x98

The Stone 24pt revision underway for TDM 2.13 now includes these glyphs and DAT changes. Additional steps needed for TDM 2.13 are:
1) Revise the wiki Charset page
2) Alter the DAT files of all existing fonts/sizes (except Stone 24pt) to handle missing breve accent, as follows:
        codepoint 0x88 in DAT --> glyph for ASCII G
        codepoint 0x98 in DAT --> glyph for ASCII g
    Since this is just a copy/paste operation, it should not take long per DAT file.
3) Alter 4 language-specific codepoint maps (tdm_base01\strings\czech.map, hungarian.map, polish.map, and slovak.map) to remove these transformations:
        source target
        ===== =====
        0x88 0xD4 // Ô
        0x98 0xF4 // ô
    This will allow utilization of Ğ/ğ in Stone 24pt (and any future font upgrade that includes it.)
4) Include (2) and (3) in the 2.13 betas.
Steps To ReproduceN/A
Additional InformationYou can assign this work to me, except for (4).

For more, including role of Ğ/ğ and alternatives in 8859-x standards and TDM-supported languages, see:
https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/&do=findComment&comment=494855

Additional concerns -
A) It is possible that some translators of the 4 languages used the 0x88/0x98 codepoints instead of 0xD4/0xF4 directly. The symptom would be - after change (3) above - "G" now appearing where a "Ô" was expected (and likewise lower case). This could affect:
-- strings in the main menu system
-- strings in a particular i18n FM (This is probably a low-level post-2.13 concern)
Translator help may be needed. The fix presumably involves revision of all.lang and regeneration of the czech.lang file (etc), ideally by someone with a 8859-2 codepage PC.

B) Existing main menu fonts don't extend beyond "top 50 European characters". Most FM fonts don't extend beyond ASCII. This doesn't address that larger problem. Short-term, it mainly is for the benefit of Stone 24 pt and its potential uses.
TagsNo tags attached.

Activities

Geep

Geep

22.06.2024 00:37

reporter   ~0016765

Item (1) has now been addressed.
Geep

Geep

26.06.2024 21:00

reporter   ~0016779

Item (2) is queued up, but it uses refont, and I need to further improve warning messages in refont first.
Geep

Geep

04.07.2024 19:03

reporter   ~0016788

Here's a status report about work on part (2) of planned font conversion to use G-Breve.

At this stage, refont was used to generate first-draft REF files from the full corpus of TDM font DAT files. As part of this effort, refont's problem detection was iteratively improved, to give better minor and major warnings of problems. An overview of findings follow, with categorization into 3 buckets, as to how ripe the REF files are for continuation to the next stage. As for refont itself, improvement of v 1.2 is on-going.

Particularly problematic fonts are mason and mason_glow. These 48-pt-only fonts use per-character scaling that none of the other fonts do. Refont currently doesn't much support this, and won't for v 1.2; maybe later.

Early candidates
----------------
These fonts gave no minor or major warnings as their DAT files were read by refont. They can be converted now by hand (e.g., hand-edit REF, then --> DAT). [Maybe do some by hand, then, once understanding of cases is reached, automate it.]

12 pt size: all except mac_humaine

24 pt size: lotharus [and presumably current 2.13 alpha of stone]

48 pt size: bamberg, carolingia, chrishand, ellianerelle, everett, gothica2, jd_hand, lotharus, medusa, nancy, popsies, rapscallionpirate, shoppinglist, summertime

Midterm candidates
------------------
These gave only minor warnings. A minor warning indicates imprecision-from-integer that is less than 0.001. Extension of refont to optionally auto-correct these imprecise values (of which there are typically lots) is desirable. Then do G-Breve conversion as with early candidates.

12 pt size: mac_humaine

24 pt size: andrew_script, bamberg, carolingia, chrishand, ellianerelle, everett, gothica2, jd_hand, mac_humaine, medusa, nancy, popsies, shoppinglist, summertime

48 pt size: andrew_script, mac_humaine

Last candidates
---------------
These gave major warnings. One needs to proceed carefully.

24 pt size: camberic, carleton, carleton_bold, carleton_condesned, carleton_glow, rapscallionpirate, treasure_map

48 pt size: camberic, carleton, carleton_bold, carleton_glow, carleton_condensed, mason, mason_glow, stone, treasure_map
Geep

Geep

20.07.2024 21:12

reporter   ~0016801

Last edited: 20.07.2024 21:16

July 20 Status Report about (2)

Refont 2.1 Released; 2.2 in Beta
==========================

2.1 allows suppression of useless warnings due to scaling.
2.2 corrects a problem with buffer overflow of shaderName, encountered with rapscallionpirate font.

Revised Workplan for 2.13 Font Improvements
======================================

Further analysis of all TDM fonts/sizes was done, to triage plausible improvements for 2.13, looking at two issues:

A) Removal of minor and major warnings detected by refont. Minor warnings are due to inadequate precision, and can be easily auto-corrected by what will be referred to here as “touching” the DAT. (By this, I mean a round-trip in refont: DAT ==> REF ==> DAT.) Major warnings are best addressed by using datBounds and looking at the bitmap, fixing the REF file, then compiling the revised DAT. BTW, the TDM engine is robust in the face of these problems, but major warnings due to out-of-range metrics can cause odd glyph renderings.

B) Substitution of redundant O-circumflex/o-circumflex codepoints by G-breve/g-breve. (Analysis here was aided by creation of "gfix", a hacked-up version of refont -stats focused just on this issue.)

Broadly, the results suggest different treatments for two categories: System Fonts and FM-only Fonts.

System Fonts
-------------------

These are the carleton family (4 types x 3 sizes), mason family (2 types x 1 size: 48pt), and stone (2 sizes; but 24pt is already done, leaving 48pt).

Generally, these fonts will need individual inspection, both to address major warnings (copious and nasty in some cases) and to understand how to implement G-breve/g-breve. A minimum REF-edit-only implementation of the latter would map to G/g. A fuller implementation would involve bitmap editing to provide the breves.

FM-only Fonts
--------------------

These 18 fonts are mostly ASCII-only. Other European characters are generally displayed as a hollow box or a “zero box”. This includes the existing 4 codepoints for O-circumflex/o-circumflex.

Given that, the two codepoints to be considered G-breve/g-breve can continue to be displayed as a hollow box or a zero box at this time. That is, no change to the DAT file is needed for reason (B).

For reason (A), almost all these fonts generate no warnings about any character font metrics, and so require no change to their 12pt & 48pt size DATs. Conversely, most generate minor warnings for 24pt size DATs, and should be “touched”.

Special cases for (A): Major warnings are seen with camberic (all sizes), medusa (all sizes), rapscallionpirate (24pt), and treasure_map (all sizes). Most of these are about a negative value for a “top” metric, which should be a quick fix (unless it’s not).

Special cases for (B): treasure_map in 24pt & 48pt has some support for O-circumflex/o-circumflex, so warrants similar support for G-breve/g-breve. It is more like a system font in that regard.

In summary, across all the FM-only fonts in 3 sizes, there are around:
- 28 DATs that are ok as is
- 16 that need to be “touched”
- 10 that need individual inspection and correction.

Only treasure_map is a near-term candidate among FM-only fonts for bitmap editing to add G-breve/g-breve.
Geep

Geep

24.07.2024 15:10

reporter   ~0016802

For the FM-only/ASCII-only fonts, the 16 DATs that needed to be touched are done. I'll wait until later to release them, probably along with less-automated repair work on camberic, medusa, and rapscallionpirate.
Geep

Geep

05.08.2024 16:27

reporter   ~0016808

Pausing any further work on 12pt for all fonts.

Camberic 24 & 48 pt is done. Took a bit longer than expected, because needed a few poor-glyph touchups, plus moving a bunch of punctuation marks higher vis a vis baseline. Also prototyped a better test FM to review ASCII font chars within readables.

Medusa next.
Geep

Geep

18.08.2024 21:03

reporter   ~0016813

Medusa 24 & 48pt are "done". Only a few glyphs moved. There are many that could be made more readable, but not a priority at this time. On to rapscallionpirate, and then planned release of this font trache.

Related: New font-review FM "readablesASCII" further along. Has readables so far with revised camberic and medusa. (Also, as reported in forum, datBounds upgraded too.)
Geep

Geep

28.08.2024 19:12

reporter   ~0016817

Rapscallionpirate 24 & 48pt done, again in a limited sense... a few highly-unreadable chars improved.

So next, I'll be better organizing and summarizing this font tranche, for release no later than mid-Sept.
Geep

Geep

30.08.2024 20:52

reporter   ~0016818

Now available is the first update in a series of font file improvements. This covers 14 fonts, and simply corrects minor internal rounding errors. It is thus a trivial update, but clears the deck for more substantive updates of remaining fonts.

See the .docx file within the .zip for details.

2024_August_TDM_Font_Update.zip
https://drive.google.com/file/d/1RNIDLmDkV_WT56kjtxSsXzqUxgYn5wh8/view?usp=sharing

Issue History

Date Modified Username Field Change
10.06.2024 18:06 Geep New Issue
22.06.2024 00:37 Geep Note Added: 0016765
26.06.2024 21:00 Geep Note Added: 0016779
04.07.2024 19:03 Geep Note Added: 0016788
20.07.2024 21:12 Geep Note Added: 0016801
20.07.2024 21:13 Geep Note Edited: 0016801
20.07.2024 21:14 Geep Note Edited: 0016801
20.07.2024 21:16 Geep Note Edited: 0016801
24.07.2024 15:10 Geep Note Added: 0016802
05.08.2024 16:27 Geep Note Added: 0016808
18.08.2024 21:03 Geep Note Added: 0016813
28.08.2024 19:12 Geep Note Added: 0016817
30.08.2024 20:52 Geep Note Added: 0016818