View Issue Details

IDProjectCategoryView StatusLast Update
0002454The Dark ModGUIpublic30.01.2022 21:13
Reportertels Assigned Toduzenko  
PrioritynormalSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
Product VersionTDM 1.06 
Target VersionTDM 2.10Fixed in VersionTDM 2.10 
Summary0002454: Implement CC (closed captions, subtitles)
Descriptionndk already created a proof-of-concept, we just need to integrate it in our code.
TagsNo tags attached.

Relationships

related to 0002779 confirmed Translate the menu and HUD into different languages 
has duplicate 0001858 closedtels ConversationSystem needs support for subtitles 

Activities

nbohr1more

nbohr1more

06.08.2018 14:24

developer   ~0010739

http://forums.thedarkmod.com/topic/11752-subtitles/
duzenko

duzenko

03.07.2021 18:34

developer   ~0014146

Last edited: 03.07.2021 18:40

At revision: 9434 (code)
At revision: 16317 (assets)
Pending the review only the briefing video is affected
Known issues: .srt file must be outside of .pk4, line breaks probably broken on Linux
And I'm not sure if I broke the CMake build too?
stgatilov

stgatilov

14.07.2021 11:26

administrator   ~0014153

Discussion on dev forums:
  https://forums.thedarkmod.com/index.php?/topic/20992-subtitles-support/
Basically, more work is needed =)
stgatilov

stgatilov

25.07.2021 13:56

administrator   ~0014181

Picked up the torch =)

Here are some generic changes, which I used in subsequent subtitles-related code:
  r9501. idStr: Added Split with list of string delimiters, and SplitLines like in Python.
  r9502. idBinSearch: support custom comparator, support heterogeneous lookup.
  r9503. idCinematic can now return filename/path of the video.

The library for parsing SRT was pretty awful, like that guy tried to add as much files/classes as he could, even though they were empty =)
So I reimplemented SRT parsing (it is a trivial format):
  r9504. Replaced SubtitleParser library with hand-written function, moved subtitles from idSoundChannel to idSoundSample.

The major chunk of architectural changes comes next:
  r9505. Expanded support of subtitles: in-game sounds, several subtitles at once, fromVideo sounds.
  r16327. Fixed subtitles in briefing videos, added subtitles overlay GUI for in-game.
At this point, subtitles should work both in-game and in briefing video.

Then it turned out that we'd better require mentioning subtitles in sound shader, so here is one more architectural change:
  r9506. Now sound shader must explicitly mention subtitles in order to load them.

Finally, some minor things:
  r9507. Sync subtitles to video if sound is played from it.
  r16328. Added subtitles setting into main menu GUI.
stgatilov

stgatilov

25.07.2021 14:16

administrator   ~0014182

In order to enable subtitles, one has to:
  1) Write "subtitle story" or "subtitle speech" or "subtitle effect" in sound shader.
  2) For every x/y/z.ogg sound file mentioned in the shader, add x/y/z.srt file with subtitles (same for ,wav).
If sound is loaded from video x/y/z.mp4 ("fromVideo" keyword), then put subtitles into file x/y/z.srt.
Note that "subtitle" keyword must be BEFORE referencing sound files or video.

The second word in "subtitle" line means verbosity level.
  "story" is for FM-specific story-related text, which player should be able to understand.
  "speech" is for regular speech, e.g. guards saying something generic on patrol, or crying words in combat.
  "effect" is for non-speech sounds, I'm not even sure yet that we need this level...
Depending on cvar tdm_subtitles, non-important subtitles can be filtered out.
For instance, by default only "story" subtitles are displayed, but player can raise it to "speech" level in the menu.
stgatilov

stgatilov

25.07.2021 14:25

administrator   ~0014183

Last edited: 25.07.2021 14:26

The general architecture is like this.

When sound shader is parsed, we look for "subtitle" keyword.
If it is present, then we try to load .srt file for every sound sample loaded (idSoundSample::LoadSubtitles).
Note that the situation when .srt is missing is silently ignored.
The subtitles data is stored in idSoundSample.

When sound thread mixes sounds, it calls idSoundWorldLocal::AddChannelContribution for every active idSoundChannel.
In this method we fetch active subtitles with idSoundChannel::GatherSubtitles, and store them into active idSoundWorld object.
The sound world has two sets of subtitles data: one currently being filled from sound thread, another one returned to everyone who wants to know the currently active subtitles (switching between sets is guarded by mutex).

The GUI object has method idUserInterfaceLocal::UpdateSubtitles, which fetches active subtitles from the sound world and updates gui::subtitle0, gui::subtitle1, etc. variables.
We call it for:
  1) main menu when it is active
  2) subtitles overlay during game
As the result, these GUIs can easily display subtitles.
Note that we have several subtitle slots in GUI (for showing several parallel conversations at once), and UpdateSubtitles method distributes active subtitles into slots.
duzenko

duzenko

25.07.2021 20:05

developer   ~0014184

Wow, guys, it sure feels good to see an 11-year old ticket closed just like this
Thank you @stgatilov, @nbohr1more
It's a pity the 1.x era programmers have left us after going standalone
stgatilov

stgatilov

26.07.2021 04:00

administrator   ~0014185

Fixed Linux build in svn rev 9508.

I'm afraid I closed it too early.
It seems that the current way of specifying subtitles is too inconvenient for thousands of short sounds that we have.
stgatilov

stgatilov

08.08.2021 13:21

administrator   ~0014257

Some TODOs for future:
1) Set subtitles in decl files of new type (support both inline and .srt reference).
2) Revise names of verbosity levels (story / speech / effect is probably not the best choice).
3) Add subtitles section into briefing GUI.
4) Require name prefix define before including tdm_subtitles_common.gui (to resolve name collision).

By the way, I checked that idDecl parsing does not break on localized characters.
I took strings/all.lang, removed all "[spanish]" sections, added type + name where braced section starts.
The code for extracting decls from a file works perfectly.
stgatilov

stgatilov

21.08.2021 18:28

administrator   ~0014310

New wave of development.

First of all, reverting previous architecture of setting subtitles in sound shader:
  r9571. Mostly reverted commit 9506: sound shader no longer specifies subtitles.

A general fix independent of subtitles:
  r9572. Various fixes in "fromVideo" sounds.
Most importantly, sound sample which plays from cin.mp4 is now named "fromVideo cin.mp4".
That's important because this name can be used to assign subtitles to video-embedded sound.

And the main pack of changes:
  r9573. Now subtitles are globally defined in the new "subtitles" decl files.
  r9574. Better subtitles reloading, use "root" subtitle instead of "root/english".

So, now we have .subs files inside "subtitles" directory, which contain "subtitles" declarations.
They looks approximately like this:

subtitles root {
    verbosity speech
    inline "sound/voices/wench/tdm_ai_wench_alertdown_to_idle_seen_evidence11.ogg" "He'd better not come back if he knows what's good for him."
    inline "sound/voices/wench/tdm_ai_wench_enemy_out_of_reach01.ogg" "Coward! Hiding from a girl!"
    inline "sound/voices/wench/tdm_ai_wench_find_light_off12.ogg" "Why are the lights out?"
    srt "sound/voices/wench/idle_10bb.ogg" "sound/voices/wench/idle_10bb.srt"

    verbosity effect
    inline "sound/voices/wench/000_clear1.ogg" "(clears throat)"

    include "tdm_ai_cynic"
}
  
subtitles tdm_ai_cynic {
    verbosity speech

    inline "sound/voices/cynic/tdm_ai_berny_combat_hit_player_company01.ogg" "Look! I got'em!"
    inline "sound/voices/cynic/tdm_ai_berny_combat_hit_player_company02.ogg" "Look at him bleed!"
    inline "sound/voices/cynic/tdm_ai_berny_combat_hit_player01.ogg" "Is that all you've got?!"
    inline "sound/voices/cynic/tdm_ai_berny_combat_hit_player02.ogg" "How'd you like that?!"

    inline "sound/voices/cynic/tdm_ai_berny_combat_throw01.ogg" "Take this!"

    inline "sound/voices/cynic/tdm_ai_berny_combat_melee01.ogg" "Heyyah!"
    inline "sound/voices/cynic/tdm_ai_berny_combat_melee02.ogg" "Heyyah!"
    inline "sound/voices/cynic/tdm_ai_berny_combat_melee03.ogg" "Heyyah!"
    inline "sound/voices/cynic/tdm_ai_berny_combat_melee05.ogg" "Take this!"

    inline "sound/voices/cynic/tdm_ai_berny_enemy_out_of_reach01.ogg" "You think you are pretty smart, ha?"
    inline "sound/voices/cynic/tdm_ai_berny_enemy_out_of_reach02_2.ogg" "I see you there!"
    inline "sound/voices/cynic/tdm_ai_berny_enemy_out_of_reach03.ogg" "Hey! I got him cornered here!"
}

The engine uses hardcoded "root" decl, but any decl can in turn "include" more decls into itself.
This allows to build include hierarchy like in scripts.
It is possible to reload subtitles by executing reloadDecls.

Not yet done:
1) languages: ideally, all languages should be defined right here.
2) checking for duplicate mappings.
Old TODOs:
3) Revise names of verbosity levels.
4) Add subtitles into briefing.
5) Resolve name collisiong with subtitles GUI in main menu.
stgatilov

stgatilov

30.08.2021 16:44

administrator   ~0014326

initial infrastructure is in SVN:
  r9576. Warn about missing SRT file, load from "tdm_root" instead of "root".
  r16351. Added initial infrastructure for subtitle decls.

Fixed points 4 and 5 the previous post:
  r16352. Fixing subtitles GUI.
stgatilov

stgatilov

22.12.2021 16:37

administrator   ~0014606

Cleanup:
  r16405. Removed garbage from subtitles decls.

Finally found a way to show subtitles with good contrast regardless of what colors are behind:
  r9735. Set "subtitleN_nonempty" boolean variable in addition to "subtitleN" string.
  r16412. Added semi-transparent black backgrounds to subtitle text.
stgatilov

stgatilov

06.01.2022 12:50

administrator   ~0014648

More generic commits on subtitles:
  r9759. Save/restore handle of in-game subtitles overlay.
  r16417. Slightly reduced font size of subtitles.

And added story subtitles for Saint Lucia:
  r16422. Added secondary subtitles for Saint Lucia mission.
  r16418. [stlucia] Added subtitles.

Issue History

Date Modified Username Field Change
04.12.2010 09:50 tels New Issue
04.12.2010 09:50 tels Status new => assigned
04.12.2010 09:50 tels Assigned To => tels
04.03.2011 17:12 tels Assigned To tels =>
04.03.2011 17:12 tels Status assigned => new
19.07.2011 19:45 tels Relationship added related to 0002779
19.07.2011 19:45 tels Assigned To => tels
19.07.2011 19:45 tels Status new => assigned
19.07.2011 19:45 tels Product Version => TDM 1.06
19.07.2011 20:27 tels Relationship added has duplicate 0001858
19.06.2012 21:36 tels Assigned To tels =>
03.01.2015 16:14 grayman Status assigned => new
06.08.2018 14:24 nbohr1more Note Added: 0010739
15.05.2020 04:00 nbohr1more Assigned To => duzenko
15.05.2020 04:00 nbohr1more Status new => assigned
03.07.2021 18:34 duzenko Note Added: 0014146
03.07.2021 18:40 duzenko Note Edited: 0014146
14.07.2021 11:26 stgatilov Note Added: 0014153
25.07.2021 13:35 stgatilov Target Version => TDM 2.10
25.07.2021 13:56 stgatilov Note Added: 0014181
25.07.2021 14:16 stgatilov Note Added: 0014182
25.07.2021 14:25 stgatilov Note Added: 0014183
25.07.2021 14:26 stgatilov Note Edited: 0014183
25.07.2021 14:27 stgatilov Status assigned => resolved
25.07.2021 14:27 stgatilov Resolution open => fixed
25.07.2021 14:27 stgatilov Fixed in Version => TDM 2.10
25.07.2021 20:05 duzenko Note Added: 0014184
26.07.2021 04:00 stgatilov Note Added: 0014185
26.07.2021 04:00 stgatilov Status resolved => feedback
26.07.2021 04:00 stgatilov Resolution fixed => reopened
26.07.2021 04:00 stgatilov Status feedback => assigned
08.08.2021 13:21 stgatilov Note Added: 0014257
21.08.2021 18:28 stgatilov Note Added: 0014310
30.08.2021 16:44 stgatilov Note Added: 0014326
22.12.2021 16:37 stgatilov Note Added: 0014606
06.01.2022 12:50 stgatilov Note Added: 0014648
06.01.2022 12:51 stgatilov Status assigned => feedback
30.01.2022 21:13 nbohr1more Status feedback => resolved
30.01.2022 21:13 nbohr1more Resolution reopened => fixed