View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006736 | The Dark Mod | Design/Coding | public | 19.08.2026 22:59 | 31.10.2026 20:50 |
| Reporter | stgatilov | Assigned To | stgatilov | ||
| Priority | normal | Severity | feature | Reproducibility | N/A |
| Status | assigned | Resolution | open | ||
| Product Version | TDM 2.14 | ||||
| Target Version | TDM 2.15 | ||||
| Summary | 0006736: Strings framework: consume all.lang directly in UTF-8 | ||||
| Description | It would be great to get rid of Perl scripts. The only thing which holds them today in core assets is processing all.lang into individual .lang files. The idea is to consume all.lang directly, probably having it renamed to something like all.utf8lang. And the charmaps should contain mappings of UTF-8 characters into engine-internal codes (should have extension like .utf8map) The engine does not need to fully support UTF-8, it simply converts the byte sequences according to the map file. | ||||
| Additional Information | Also described here: * https://forums.thedarkmod.com/index.php?/topic/23204-proposed-i18n-improvements-for-tdm-215/#findComment-509456 | ||||
| Tags | No tags attached. | ||||
|
The new system works like this (when g_langScripts = 1): * all.lang is UTF-8 encoded and contains strings for all languages (same as before). * {language}.utf8map files defines how a few extra UTF-8 characters are mapped into 8-bit internal chars (ASCII 0-127 is mapped as is implicitly). * {language}.lang files are no longer loaded. Mission strings are exception: old-style strings are still loaded if new ones are not available. As a major plot twist, we later decided to support gettext-style strings instead (g_langScripts = 2): * english.pot contains strings for English language: msgctxt is #str_01234 and msgid is English text. * {language}.po contains strings for every non-English language: msgctxt is #str_01234 and msgstr is local text. * Everything is UTF-8 encoded of course, and the same {language}.utf8map are used to translate into internal 8-bit characters. The main benefit is that this format is rather standard, so we can use available tools. Like msgmerge can be used to copy new strings from POT to PO-s, poedit can be used by translators to actually translate. Of course, old-style 8-bit {language}.lang files are still loaded for missions for backwards compatibility reasons, in case gettext-style files are missing. Some local gettext extensions: * msgctxt "##" entries are ignored, they are TDM-special comments. * msgid/msgstr "#empty" means effective empty string as value, because: * msgstr "" is often present in PO file and it means "no translation available", in which case English string is used. ==== The list of commits so far ==== Preliminary/general cleaning: r11360 Deleted localizeGuis, localizeMaps console commands + cleanup. r17588 Deleted #str_02518 duplicate definition. r17589 Deleted some unreferenced strings. r17590 Regenerated lang files. r17591 Deleted unused #str_09000 from Italian. r17603 Fixed \n escape sequences in italian #str_10085 and #str_10086. Move to UTF-8 lang files: r11362 Now strings in localization framework are loaded from .utf8lang files (using .utf8map helper files). r17597 Added script that deduces per-language UTF-8->8-bit mapping from current .lang files. r17598 Now the utf8map-generating script takes old .map files into account. r17604 Added new .utf8lang and .utf8map files for localization. Move to gettest format: r11363 Supported loading localization strings from gettext-style files. r11364 Forgotten changes for previous commit. r11365 Replaced // with ##, supported #empty. r11366 g_langScripts now defaults to 2 (gettext). r17611 Added gettext-formatted core strings. r17612 Deleted english.po. r17613 Added manually three missing strings to POT. r17614 Corrected #str_02481 (slovencina). The current state of discussion is more or less here: * https://forums.thedarkmod.com/index.php?/topic/23204-proposed-i18n-improvements-for-tdm-215/page/4/#findComment-510765 The reorganising of strings files and conversion to gettext were done by @geep: https://wiki.thedarkmod.com/index.php?title=Converting_Translations_with_all_lang_to_po#Long_Text_and_Chunking |
|