View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0004167 | The Dark Mod | TDM Installer | public | 21.06.2015 17:46 | 14.05.2016 20:03 |
Reporter | taaaki | Assigned To | taaaki | ||
Priority | normal | Severity | normal | Reproducibility | always |
Status | resolved | Resolution | fixed | ||
Product Version | TDM 2.03 | ||||
Target Version | TDM 2.04 | Fixed in Version | TDM 2.04 | ||
Summary | 0004167: Fix bad pk4 file dates in release pk4s | ||||
Description | Ensure that at least 1 file is modified or removed from the following PK4s to ensure that the bad dates get fixed: tdm_ai_humanoid_builders01.pk4 tdm_ai_humanoid_guards01.pk4 tdm_gui01.pk4 tdm_models01.pk4 tdm_models02.pk4 tdm_models_decls01.pk4 tdm_prefabs01.pk4 tdm_sound_ambient01.pk4 tdm_sound_ambient02.pk4 tdm_sound_sfx01.pk4 tdm_sound_vocals02.pk4 tdm_sound_vocals03.pk4 tdm_sound_vocals04.pk4 tdm_sound_vocals06.pk4 tdm_sound_vocals07.pk4 tdm_textures_base01.pk4 tdm_textures_carpet01.pk4 tdm_textures_decals01.pk4 tdm_textures_door01.pk4 tdm_textures_glass01.pk4 tdm_textures_metal01.pk4 tdm_textures_nature01.pk4 tdm_textures_plaster01.pk4 tdm_textures_stone_brick01.pk4 tdm_textures_stone_cobblestones01.pk4 tdm_textures_stone_flat01.pk4 tdm_textures_stone_natural01.pk4 tdm_textures_stone_sculpted01.pk4 tdm_textures_window01.pk4 tdm_textures_wood01.pk4 | ||||
Tags | No tags attached. | ||||
Attached Files | touch_pk4s_for_204.py (1,789 bytes)
"""We need to touch 1 file from each of 30 pk4s. Find the first file in each of the pk4s.""" PK4LIST = [ "tdm_ai_humanoid_builders01.pk4", "tdm_ai_humanoid_guards01.pk4", "tdm_gui01.pk4", "tdm_models01.pk4", "tdm_models02.pk4", "tdm_models_decls01.pk4", "tdm_prefabs01.pk4", "tdm_sound_ambient01.pk4", "tdm_sound_ambient02.pk4", "tdm_sound_sfx01.pk4", "tdm_sound_vocals02.pk4", "tdm_sound_vocals03.pk4", "tdm_sound_vocals04.pk4", "tdm_sound_vocals06.pk4", "tdm_sound_vocals07.pk4", "tdm_textures_base01.pk4", "tdm_textures_carpet01.pk4", "tdm_textures_decals01.pk4", "tdm_textures_door01.pk4", "tdm_textures_glass01.pk4", "tdm_textures_metal01.pk4", "tdm_textures_nature01.pk4", "tdm_textures_plaster01.pk4", "tdm_textures_stone_brick01.pk4", "tdm_textures_stone_cobblestones01.pk4", "tdm_textures_stone_flat01.pk4", "tdm_textures_stone_natural01.pk4", "tdm_textures_stone_sculpted01.pk4", "tdm_textures_window01.pk4", "tdm_textures_wood01.pk4" ] # The folder where we'll look in pk4s to find a file to touch TDM_RELEASE_DIR = r"C:\darkmod" # The SVN folder where we'll touch the files TDM_SVN_DIR = r"E:\dm-dev\darkmod" import os, zipfile for pk4 in PK4LIST: fname = os.path.join(TDM_RELEASE_DIR, pk4) f = zipfile.ZipFile(fname) firstfile = f.infolist()[0].filename.replace('/', '\\') touchfile = os.path.join(TDM_SVN_DIR, firstfile) # print touchfile # List of files for touching # print pk4, ' --> ', firstfile # List of pk4s and files for tracker issue # generate batch commands for windows touches. Need to switch to E:\ first dir, fil = os.path.split(touchfile) print 'cd', dir print 'copy "' + touchfile + '" /B +,,' date_check.py (1,005 bytes)
"""Check the modified dates of files in the TDM pk4s. Look out for 1980 dates and 2088 dates.""" import os, fnmatch, zipfile PATH = r"G:\darkmod_beta" results = { 'ok' : 0 } pk4counter = 0 filecounter = 0 for root, dirnames, filenames in os.walk(PATH): for filename in fnmatch.filter(filenames, '*.pk4'): pk4counter += 1 fpath = os.path.join(root, filename) f = zipfile.ZipFile(fpath, 'r') for member in f.infolist(): filecounter += 1 year = member.date_time[0] if year < 2000 or year > 2016: results[fpath + ' --> ' + member.filename] = member.date_time else: results['ok'] += 1 print ("Results: Checked %d pk4s, found %d files" % ( pk4counter, filecounter )) print ("Found %d ok files, %d problem dates\n" % ( results['ok'], len(results)-1 )) results.pop('ok') for k in sorted(results.keys()): print (k, "%04d-%02d-%02d %02d:%02d:%02d" % results[k]) | ||||
Any chance this can get worked on the next 2 months, for 2.04? | |
This is something that can get done during the release testing. Once we have something close to release, we can check which of the files in the above list have already been modified and then make a plan to force changes in the remaining ones. From what I recall of the issue, this was the easiest way to correct existing TDM installations and ensure that new installations aren't affected. |
|
Thx. | |
Committed at rev 14473 | |
I had to touch the files in 2 ways: • update file modified date • add an svn property to each file "comment_204" = "Touched but not modified in 2.04 for 0004167" The svn property is needed to make svn update a file with no differences. I used a python script to find the files to change and generate the windows batch commands (attached). I did the svn properties manually because I didn't fancy installing the command line client. 1 file done for each pk4: tdm_ai_humanoid_builders01.pk4 --> dds\models\md5\chars\builders\acolyte_d.dds tdm_ai_humanoid_guards01.pk4 --> dds\models\md5\chars\chainmail.dds tdm_gui01.pk4 --> dds\guis\assets\blanklevelshot.dds tdm_models01.pk4 --> dds\models\darkmod\architecture\house_04.dds tdm_models02.pk4 --> dds\models\darkmod\props\textures\airpotion_a.dds tdm_models_decls01.pk4 --> materials\tdm_models_animals.mtr tdm_prefabs01.pk4 --> prefabs\Kitchen\wine_bottle01.pfb tdm_sound_ambient01.pk4 --> sound\ambient\environmental\animal_bats01.ogg tdm_sound_ambient02.pk4 --> sound\ambient\ambience\alien01_loop.ogg tdm_sound_sfx01.pk4 --> sound\machinery\loop_machinery_03.ogg tdm_sound_vocals02.pk4 --> sound\voices\builders\builder1\conversations\builder1_conv_1.ogg tdm_sound_vocals03.pk4 --> sound\voices\lord\fool.ogg tdm_sound_vocals04.pk4 --> sound\voices\animal\raven\caw01.ogg tdm_sound_vocals06.pk4 --> sound\voices\critic\alarm01.ogg tdm_sound_vocals07.pk4 --> sound\voices\lady\alarm.ogg tdm_textures_base01.pk4 --> dds\textures\particles\arcturus_fire.tga tdm_textures_carpet01.pk4 --> dds\textures\darkmod\carpet\rugs\ornate_black_gold01.dds tdm_textures_decals01.pk4 --> dds\textures\darkmod\decals\building_facades\city_buildings01.dds tdm_textures_door01.pk4 --> dds\textures\darkmod\door\frame\arched_heavy01_frame.dds tdm_textures_glass01.pk4 --> dds\textures\darkmod\glass\crystal_milky.dds tdm_textures_metal01.pk4 --> dds\textures\darkmod\metal\detailed\bars_rivets_iron.dds tdm_textures_nature01.pk4 --> dds\textures\darkmod\nature\bark\bark1.dds tdm_textures_plaster01.pk4 --> dds\textures\darkmod\plaster\adobe_rough.dds tdm_textures_stone_brick01.pk4 --> dds\textures\darkmod\stone\brick\blocks_brown.dds tdm_textures_stone_cobblestones01.pk4 --> dds\textures\darkmod\stone\cobblestones\beatoldpavment_s.dds tdm_textures_stone_flat01.pk4 --> dds\textures\darkmod\stone\flat\cement_002.dds tdm_textures_stone_natural01.pk4 --> dds\textures\darkmod\stone\natural\coal_heap.dds tdm_textures_stone_sculpted01.pk4 --> dds\textures\darkmod\stone\sculpted\arch01.dds tdm_textures_window01.pk4 --> dds\textures\darkmod\window\diamond_pattern01_dark.dds tdm_textures_wood01.pk4 --> dds\textures\darkmod\wood\boards\board_red_finegrain.dds |
|
taaaki, do you have a suggestion for verifying the fix? You and gnartsch did the checks the first time round. NB testing will have to wait for grayman's next release build |
|
There's a date check python file attached to issue 4110 (linked in Relationships section above) that can check for bad dates in the pk4s. I've got some other work to do tonight and tomorrow, but I'll download the update to the 2.04 beta and check files on Friday. | |
I've updated the date_check.py code to cater for 2016, but some of the files still have issues. However, this is because I forgot to rebuild the packager on the update server. I'll have to rebuild the packager and then redo the test package build. | |
I had another look at the TDM packager source code and it would seem that a date change is not sufficient to trigger the required updates. We'll have to find some other method to update the files that doesn't compromise integrity of the modified files. | |
I will add a "scrub pk4" feature to the updater rather than trying to modify files to trigger the date scrubbing automatically. | |
I have implemented an additional step in the updater process to check for and correct PK4s with bad dates. The changes are in darkmod_src/trunk revision 6588. The check is implemented as part of the clean-up phase. I have not implemented this as a configurable/toggled feature - if there are no bad dates, this process only adds around 350ms (on SSD) to the update process. If bad dates are detected, the extra processing time will depend on the the number of bad PK4s and the user's system specs. There is a progress indicator to keep track of PK4 regeneration process. I have currently only tested this on the Windows MFC-based updater. We'll just need to test it on the console (Windows and Linux) updater to be sure that the progress logs make sense. I verified the fix by running the date_check.py script before and after the updater was run. |
|
Date Modified | Username | Field | Change |
---|---|---|---|
21.06.2015 17:46 | taaaki | New Issue | |
21.06.2015 17:46 | taaaki | Status | new => assigned |
21.06.2015 17:46 | taaaki | Assigned To | => taaaki |
21.06.2015 17:47 | taaaki | Relationship added | related to 0004110 |
03.01.2016 18:17 | grayman | Note Added: 0007955 | |
03.01.2016 20:41 | taaaki | Note Added: 0007957 | |
03.01.2016 20:44 | taaaki | Note Edited: 0007957 | |
03.01.2016 21:17 | grayman | Note Added: 0007962 | |
06.04.2016 18:03 | SteveL | Note Added: 0008065 | |
06.04.2016 18:07 | SteveL | Note Added: 0008066 | |
06.04.2016 18:09 | SteveL | Note Added: 0008067 | |
06.04.2016 18:11 | SteveL | File Added: touch_pk4s_for_204.py | |
06.04.2016 18:11 | SteveL | Status | assigned => feedback |
06.04.2016 18:12 | SteveL | Note Edited: 0008067 | |
06.04.2016 18:13 | taaaki | Note Added: 0008068 | |
06.04.2016 18:13 | taaaki | Status | feedback => assigned |
09.04.2016 18:43 | taaaki | Note Added: 0008069 | |
09.04.2016 18:45 | taaaki | File Added: date_check.py | |
10.04.2016 19:32 | taaaki | Note Added: 0008079 | |
11.04.2016 20:42 | taaaki | Note Added: 0008091 | |
14.05.2016 20:03 | taaaki | Note Added: 0008146 | |
14.05.2016 20:03 | taaaki | Status | assigned => resolved |
14.05.2016 20:03 | taaaki | Fixed in Version | => TDM 2.04 |
14.05.2016 20:03 | taaaki | Resolution | open => fixed |
19.05.2021 05:34 | stgatilov | Category | TDM Updater => TDM Installer |