 |
In order to get data, I added logging of all ::Read and ::Seek calls of IdFile_InZip (temporarily). For each call, I saved number of compressed bytes read&decompressed during the call. Here are cumulative numbers for all Read and all Seek calls: {'read': 871032184, 'seek': 1705039195} If we look at the numbers per file extension, only several types of files are actually seeked: 6888556 | .lwo:read 138472732 | .lwo:seek 473887194 | .ogg:read 1558358316 | .ogg:seek 2536785 | .wav:read 859881 | .wav:seek So ogg files cause the most of unnecessary reading&decompressing. Ironically, ogg files are already compressed, so there is no need to compress them inside pk4 files again. Here is the sequence of read+seek calls for a single ogg file (number tells how many bytes were read&decompressed during the call): tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 65536 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 65536 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 seek tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 0 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 65536 read tdm_gui01.pk4/sound/meta/menu/amb_nocturne_menu.ogg 3566560 read |
|
|
One of the possible improvements for this issue is to store ogg files in uncompressed pk4 archives. However, simply repacking the pk4 files is not enough, because currently both idFile_InZip and underlying unzip implementation (which is some beta version of modern minizip library) do not support seeking directly through uncompressed data. So I copied the adapted version of minizip from dhewm3 repository and replaced the old unzip code with it. Then I copied unzseek function from the latest minizip release (version 1.1), and integrated it into idFile_InZip::Seek method. Lastly, I repacked all tdm_sound_*.pk4 files into pk4 archives without compression. As the result, the total number of bytes read has reduced in process explorer (like from 2.7 GB to 1.5 GB), and the total loading time has reduced from 1:25 to 1:00 on the investigated save. The total size of pk4 files has increased from 592 MB to 610 MB. |
|
|
I have committed the repacking code. It checks any pk4 file being opened in idFileSystemLocal::LoadZipFile, and if it contains a critical file compressed, then the whole archive is repacked. During repacking, a temporary file is created. Only the files which have wrong compression level are actually uncompressed, all the other files are copied directly without recompression. At the end, the original files is deleted and the temporary files is renamed. |
|
|
Some exact data about resulting improvement in loading times can be found in this forum post: http://forums.thedarkmod.com/topic/18758-inefficient-ogg-loading-from-pk4-and-minizip-update/?p=404957 |
 |
Should we do the same with .jpg's? And I think .dds's are compressed already too? |
|
|
Jpg and dds files are read on one pass: there is no excessive seeking, like there is for ogg and video files. If you decompress all jpg files, you'll only save on their decompression time. which is likely not much. Here I would prefer approach "do not fix it until it's broken". As for DDS files, they are not compressed (just look inside). The same applies to tga. Removing compression from them would significantly increase size of packages, and probably even slow down loading -- how knows =) |
- Anonymous
