View Issue Details

IDProjectCategoryView StatusLast Update
0004167The Dark ModTDM Installerpublic14.05.2016 20:03
Reportertaaaki Assigned Totaaaki  
PrioritynormalSeveritynormalReproducibilityalways
Status resolvedResolutionfixed 
Product VersionTDM 2.03 
Target VersionTDM 2.04Fixed in VersionTDM 2.04 
Summary0004167: Fix bad pk4 file dates in release pk4s
DescriptionEnsure that at least 1 file is modified or removed from the following PK4s to ensure that the bad dates get fixed:

tdm_ai_humanoid_builders01.pk4
tdm_ai_humanoid_guards01.pk4
tdm_gui01.pk4
tdm_models01.pk4
tdm_models02.pk4
tdm_models_decls01.pk4
tdm_prefabs01.pk4
tdm_sound_ambient01.pk4
tdm_sound_ambient02.pk4
tdm_sound_sfx01.pk4
tdm_sound_vocals02.pk4
tdm_sound_vocals03.pk4
tdm_sound_vocals04.pk4
tdm_sound_vocals06.pk4
tdm_sound_vocals07.pk4
tdm_textures_base01.pk4
tdm_textures_carpet01.pk4
tdm_textures_decals01.pk4
tdm_textures_door01.pk4
tdm_textures_glass01.pk4
tdm_textures_metal01.pk4
tdm_textures_nature01.pk4
tdm_textures_plaster01.pk4
tdm_textures_stone_brick01.pk4
tdm_textures_stone_cobblestones01.pk4
tdm_textures_stone_flat01.pk4
tdm_textures_stone_natural01.pk4
tdm_textures_stone_sculpted01.pk4
tdm_textures_window01.pk4
tdm_textures_wood01.pk4
TagsNo tags attached.
Attached Files
touch_pk4s_for_204.py (1,789 bytes)   
"""We need to touch 1 file from each of 30 pk4s.

Find the first file in each of the pk4s."""

PK4LIST = [
    "tdm_ai_humanoid_builders01.pk4",
    "tdm_ai_humanoid_guards01.pk4",
    "tdm_gui01.pk4",
    "tdm_models01.pk4",
    "tdm_models02.pk4",
    "tdm_models_decls01.pk4",
    "tdm_prefabs01.pk4",
    "tdm_sound_ambient01.pk4",
    "tdm_sound_ambient02.pk4",
    "tdm_sound_sfx01.pk4",
    "tdm_sound_vocals02.pk4",
    "tdm_sound_vocals03.pk4",
    "tdm_sound_vocals04.pk4",
    "tdm_sound_vocals06.pk4",
    "tdm_sound_vocals07.pk4",
    "tdm_textures_base01.pk4",
    "tdm_textures_carpet01.pk4",
    "tdm_textures_decals01.pk4",
    "tdm_textures_door01.pk4",
    "tdm_textures_glass01.pk4",
    "tdm_textures_metal01.pk4",
    "tdm_textures_nature01.pk4",
    "tdm_textures_plaster01.pk4",
    "tdm_textures_stone_brick01.pk4",
    "tdm_textures_stone_cobblestones01.pk4",
    "tdm_textures_stone_flat01.pk4",
    "tdm_textures_stone_natural01.pk4",
    "tdm_textures_stone_sculpted01.pk4",
    "tdm_textures_window01.pk4",
    "tdm_textures_wood01.pk4"
]
# The folder where we'll look in pk4s to find a file to touch
TDM_RELEASE_DIR = r"C:\darkmod"
# The SVN folder where we'll touch the files
TDM_SVN_DIR = r"E:\dm-dev\darkmod"

import os, zipfile

for pk4 in PK4LIST:
    fname = os.path.join(TDM_RELEASE_DIR, pk4)
    f = zipfile.ZipFile(fname)
    firstfile = f.infolist()[0].filename.replace('/', '\\')
    touchfile = os.path.join(TDM_SVN_DIR, firstfile)
    # print touchfile # List of files for touching
    # print pk4, ' --> ', firstfile # List of pk4s and files for tracker issue
    # generate batch commands for windows touches. Need to switch to E:\ first
    dir, fil = os.path.split(touchfile)
    print 'cd', dir
    print 'copy "' + touchfile + '" /B +,,'

touch_pk4s_for_204.py (1,789 bytes)   
date_check.py (1,005 bytes)   
"""Check the modified dates of files in the TDM pk4s. Look out for 1980 dates and 2088 dates."""
import os, fnmatch, zipfile

PATH = r"G:\darkmod_beta"

results = { 'ok' : 0 }
pk4counter = 0
filecounter = 0

for root, dirnames, filenames in os.walk(PATH):
    for filename in fnmatch.filter(filenames, '*.pk4'):
        pk4counter += 1
        fpath = os.path.join(root, filename)
        f = zipfile.ZipFile(fpath, 'r')
        for member in f.infolist():
            filecounter += 1
            year = member.date_time[0]
            if year < 2000 or year > 2016:
                results[fpath + ' --> ' + member.filename] = member.date_time
            else:
                results['ok'] += 1

print ("Results: Checked %d pk4s, found %d files" % ( pk4counter, filecounter ))
print ("Found %d ok files, %d problem dates\n" % ( results['ok'], len(results)-1 ))
results.pop('ok')
for k in sorted(results.keys()):
    print (k, "%04d-%02d-%02d %02d:%02d:%02d" % results[k])

date_check.py (1,005 bytes)   

Relationships

related to 0004110 resolvedtaaaki textures/darkmod/stone/brick/blocks_sepia_grey broken 

Activities

grayman

grayman

03.01.2016 18:17

viewer   ~0007955

Any chance this can get worked on the next 2 months, for 2.04?
taaaki

taaaki

03.01.2016 20:41

administrator   ~0007957

Last edited: 03.01.2016 20:44

This is something that can get done during the release testing. Once we have something close to release, we can check which of the files in the above list have already been modified and then make a plan to force changes in the remaining ones.

From what I recall of the issue, this was the easiest way to correct existing TDM installations and ensure that new installations aren't affected.

grayman

grayman

03.01.2016 21:17

viewer   ~0007962

Thx.
SteveL

SteveL

06.04.2016 18:03

reporter   ~0008065

Committed at rev 14473
SteveL

SteveL

06.04.2016 18:07

reporter   ~0008066

I had to touch the files in 2 ways:
• update file modified date
• add an svn property to each file "comment_204" = "Touched but not modified in 2.04 for 0004167"

The svn property is needed to make svn update a file with no differences.

I used a python script to find the files to change and generate the windows batch commands (attached). I did the svn properties manually because I didn't fancy installing the command line client.

1 file done for each pk4:
tdm_ai_humanoid_builders01.pk4 --> dds\models\md5\chars\builders\acolyte_d.dds
tdm_ai_humanoid_guards01.pk4 --> dds\models\md5\chars\chainmail.dds
tdm_gui01.pk4 --> dds\guis\assets\blanklevelshot.dds
tdm_models01.pk4 --> dds\models\darkmod\architecture\house_04.dds
tdm_models02.pk4 --> dds\models\darkmod\props\textures\airpotion_a.dds
tdm_models_decls01.pk4 --> materials\tdm_models_animals.mtr
tdm_prefabs01.pk4 --> prefabs\Kitchen\wine_bottle01.pfb
tdm_sound_ambient01.pk4 --> sound\ambient\environmental\animal_bats01.ogg
tdm_sound_ambient02.pk4 --> sound\ambient\ambience\alien01_loop.ogg
tdm_sound_sfx01.pk4 --> sound\machinery\loop_machinery_03.ogg
tdm_sound_vocals02.pk4 --> sound\voices\builders\builder1\conversations\builder1_conv_1.ogg
tdm_sound_vocals03.pk4 --> sound\voices\lord\fool.ogg
tdm_sound_vocals04.pk4 --> sound\voices\animal\raven\caw01.ogg
tdm_sound_vocals06.pk4 --> sound\voices\critic\alarm01.ogg
tdm_sound_vocals07.pk4 --> sound\voices\lady\alarm.ogg
tdm_textures_base01.pk4 --> dds\textures\particles\arcturus_fire.tga
tdm_textures_carpet01.pk4 --> dds\textures\darkmod\carpet\rugs\ornate_black_gold01.dds
tdm_textures_decals01.pk4 --> dds\textures\darkmod\decals\building_facades\city_buildings01.dds
tdm_textures_door01.pk4 --> dds\textures\darkmod\door\frame\arched_heavy01_frame.dds
tdm_textures_glass01.pk4 --> dds\textures\darkmod\glass\crystal_milky.dds
tdm_textures_metal01.pk4 --> dds\textures\darkmod\metal\detailed\bars_rivets_iron.dds
tdm_textures_nature01.pk4 --> dds\textures\darkmod\nature\bark\bark1.dds
tdm_textures_plaster01.pk4 --> dds\textures\darkmod\plaster\adobe_rough.dds
tdm_textures_stone_brick01.pk4 --> dds\textures\darkmod\stone\brick\blocks_brown.dds
tdm_textures_stone_cobblestones01.pk4 --> dds\textures\darkmod\stone\cobblestones\beatoldpavment_s.dds
tdm_textures_stone_flat01.pk4 --> dds\textures\darkmod\stone\flat\cement_002.dds
tdm_textures_stone_natural01.pk4 --> dds\textures\darkmod\stone\natural\coal_heap.dds
tdm_textures_stone_sculpted01.pk4 --> dds\textures\darkmod\stone\sculpted\arch01.dds
tdm_textures_window01.pk4 --> dds\textures\darkmod\window\diamond_pattern01_dark.dds
tdm_textures_wood01.pk4 --> dds\textures\darkmod\wood\boards\board_red_finegrain.dds
SteveL

SteveL

06.04.2016 18:09

reporter   ~0008067

Last edited: 06.04.2016 18:12

taaaki, do you have a suggestion for verifying the fix? You and gnartsch did the checks the first time round.

NB testing will have to wait for grayman's next release build

taaaki

taaaki

06.04.2016 18:13

administrator   ~0008068

There's a date check python file attached to issue 4110 (linked in Relationships section above) that can check for bad dates in the pk4s. I've got some other work to do tonight and tomorrow, but I'll download the update to the 2.04 beta and check files on Friday.
taaaki

taaaki

09.04.2016 18:43

administrator   ~0008069

I've updated the date_check.py code to cater for 2016, but some of the files still have issues. However, this is because I forgot to rebuild the packager on the update server. I'll have to rebuild the packager and then redo the test package build.
taaaki

taaaki

10.04.2016 19:32

administrator   ~0008079

I had another look at the TDM packager source code and it would seem that a date change is not sufficient to trigger the required updates. We'll have to find some other method to update the files that doesn't compromise integrity of the modified files.
taaaki

taaaki

11.04.2016 20:42

administrator   ~0008091

I will add a "scrub pk4" feature to the updater rather than trying to modify files to trigger the date scrubbing automatically.
taaaki

taaaki

14.05.2016 20:03

administrator   ~0008146

I have implemented an additional step in the updater process to check for and correct PK4s with bad dates. The changes are in darkmod_src/trunk revision 6588.

The check is implemented as part of the clean-up phase. I have not implemented this as a configurable/toggled feature - if there are no bad dates, this process only adds around 350ms (on SSD) to the update process.

If bad dates are detected, the extra processing time will depend on the the number of bad PK4s and the user's system specs. There is a progress indicator to keep track of PK4 regeneration process.

I have currently only tested this on the Windows MFC-based updater. We'll just need to test it on the console (Windows and Linux) updater to be sure that the progress logs make sense.

I verified the fix by running the date_check.py script before and after the updater was run.

Issue History

Date Modified Username Field Change
21.06.2015 17:46 taaaki New Issue
21.06.2015 17:46 taaaki Status new => assigned
21.06.2015 17:46 taaaki Assigned To => taaaki
21.06.2015 17:47 taaaki Relationship added related to 0004110
03.01.2016 18:17 grayman Note Added: 0007955
03.01.2016 20:41 taaaki Note Added: 0007957
03.01.2016 20:44 taaaki Note Edited: 0007957
03.01.2016 21:17 grayman Note Added: 0007962
06.04.2016 18:03 SteveL Note Added: 0008065
06.04.2016 18:07 SteveL Note Added: 0008066
06.04.2016 18:09 SteveL Note Added: 0008067
06.04.2016 18:11 SteveL File Added: touch_pk4s_for_204.py
06.04.2016 18:11 SteveL Status assigned => feedback
06.04.2016 18:12 SteveL Note Edited: 0008067
06.04.2016 18:13 taaaki Note Added: 0008068
06.04.2016 18:13 taaaki Status feedback => assigned
09.04.2016 18:43 taaaki Note Added: 0008069
09.04.2016 18:45 taaaki File Added: date_check.py
10.04.2016 19:32 taaaki Note Added: 0008079
11.04.2016 20:42 taaaki Note Added: 0008091
14.05.2016 20:03 taaaki Note Added: 0008146
14.05.2016 20:03 taaaki Status assigned => resolved
14.05.2016 20:03 taaaki Fixed in Version => TDM 2.04
14.05.2016 20:03 taaaki Resolution open => fixed
19.05.2021 05:34 stgatilov Category TDM Updater => TDM Installer