View Revisions: Issue #5042

Summary 0005042: Timestamps fiasco: Unix vs DOS, local vs UTC
Revision 15.08.2019 16:25 by stgatilov
Description The engine code heavily relies on file timestamps. They influence incremental reloading of assets, choosing how to resolve conflict when same asset is present in many places, etc. Moreover, if file timestamp cannot be determined properly, then the engine may decide that the file does not exist (but still read it at the same time). All of this causes terrible issues sometimes.

One problem about it was noticed in 2016 at TDM 2.05:
  http://forums.thedarkmod.com/index.php?/topic/18582-bogus-timestamps-in-tdm-update-files-cause-tdm-crashes-under-linux/&tab=comments#comment-399317
The next problem happened in 2019 at TDM 2.07 (confirmed):
  http://forums.thedarkmod.com/index.php?/topic/20050-proccpuinfo-cpu-frequency-200007-mhz-signal-caught-segmentation-fault-si_code-1/

The code stores timestamps as ID_TIME_T, which is a typedef for time_t from C. This type stores a Unix Timestamp, i.e. the number of seconds passed since Epoch, where Epoch is specified in UTC. Most importantly, Unix timestamp for an event is logically the same on every machine in every timezone. It is NOT a local time.

A major convention used in engine (e.g. by image loading code) is:
  If timestamp is -1, then the file was not loaded.
The same convention is used by mktime function, which computes Unit Timestamp from local time:
  If local time cannot be converted, then timestamp -1 is returned.
Adding these two conventions together: if some error happens when handling file timestamp, then the file is considered as "not loaded" (sort of... partially)

Now back to why we need local times at all.
The engine uses zip file format for all assets on the player side. Zip format is very old, and it stores last modification time of each file in DOS format. Which is just "year, month, day, hour, minute, second", without any timezone/DST info. By common convention, everyone treats DOS time as local time.
Since zip modification times are considered local, the engine converts them to Unix Timestamps on reading. This is done in function Sys_DosToUnixTime, and it calls aforementioned mktime standard function to convert from local time to Unit Timestamp.

Here are some problems:
1) mktime function works differently depending on locale/timezone/DST/whatever. It may fail due to some DST weirdness or due to random value stored in zip datetimes.
2) Sys_DosToUnixTime function does not initialize the field "dostm.tm_isdst" before calling mktime. Depending on the sign of this value, mktime considers input to be at non-DST/DST/uncertain moment, which may result in different output or even fail (depending on locale).
3) The pk4 files have somewhat imprecise timestamps. Because TDM server packages everything in one timezone, while players read these zips in another timezone.
Revision 15.08.2019 16:25 by stgatilov
Description The engine code heavily relies on file timestamps. They influence incremental reloading of assets, choosing how to resolve conflict when same asset is present in many places, etc. Moreover, if file timestamp cannot be determined properly, then the engine may decide that the file does not exist (but still read it at the same time). All of this causes terrible issues sometimes.

One problem about it was noticed in 2016 at TDM 2.05:
  http://forums.thedarkmod.com/index.php?/topic/18582-bogus-timestamps-in-tdm-update-files-cause-tdm-crashes-under-linux/&tab=comments#comment-399317
The next problem PROBABLY happened in 2019 at TDM 2.07 (confirmed):
  http://forums.thedarkmod.com/index.php?/topic/20050-proccpuinfo-cpu-frequency-200007-mhz-signal-caught-segmentation-fault-si_code-1/

The code stores timestamps as ID_TIME_T, which is a typedef for time_t from C. This type stores a Unix Timestamp, i.e. the number of seconds passed since Epoch, where Epoch is specified in UTC. Most importantly, Unix timestamp for an event is logically the same on every machine in every timezone. It is NOT a local time.

A major convention used in engine (e.g. by image loading code) is:
  If timestamp is -1, then the file was not loaded.
The same convention is used by mktime function, which computes Unit Timestamp from local time:
  If local time cannot be converted, then timestamp -1 is returned.
Adding these two conventions together: if some error happens when handling file timestamp, then the file is considered as "not loaded" (sort of... partially)

Now back to why we need local times at all.
The engine uses zip file format for all assets on the player side. Zip format is very old, and it stores last modification time of each file in DOS format. Which is just "year, month, day, hour, minute, second", without any timezone/DST info. By common convention, everyone treats DOS time as local time.
Since zip modification times are considered local, the engine converts them to Unix Timestamps on reading. This is done in function Sys_DosToUnixTime, and it calls aforementioned mktime standard function to convert from local time to Unit Timestamp.

Here are some problems:
1) mktime function works differently depending on locale/timezone/DST/whatever. It may fail due to some DST weirdness or due to random value stored in zip datetimes.
2) Sys_DosToUnixTime function does not initialize the field "dostm.tm_isdst" before calling mktime. Depending on the sign of this value, mktime considers input to be at non-DST/DST/uncertain moment, which may result in different output or even fail (depending on locale).
3) The pk4 files have somewhat imprecise timestamps. Because TDM server packages everything in one timezone, while players read these zips in another timezone.
Revision 12.08.2019 17:46 by stgatilov
Description The engine code heavily relies on file timestamps. They influence incremental reloading of assets, choosing how to resolve conflict when same asset is present in many places, etc. Moreover, if file timestamp cannot be determined properly, then the engine may decide that the file does not exist (but still read it at the same time). All of this causes terrible issues sometimes.

One problem about it was noticed in 2016 at TDM 2.05:
  http://forums.thedarkmod.com/index.php?/topic/18582-bogus-timestamps-in-tdm-update-files-cause-tdm-crashes-under-linux/&tab=comments#comment-399317
The next problem PROBABLY happened in 2019 at TDM 2.07 ( !!!NOT CONFIRMED YET!!! ):
  http://forums.thedarkmod.com/index.php?/topic/20050-proccpuinfo-cpu-frequency-200007-mhz-signal-caught-segmentation-fault-si_code-1/

The code stores timestamps as ID_TIME_T, which is a typedef for time_t from C. This type stores a Unix Timestamp, i.e. the number of seconds passed since Epoch, where Epoch is specified in UTC. Most importantly, Unix timestamp for an event is logically the same on every machine in every timezone. It is NOT a local time.

A major convention used in engine (e.g. by image loading code) is:
  If timestamp is -1, then the file was not loaded.
The same convention is used by mktime function, which computes Unit Timestamp from local time:
  If local time cannot be converted, then timestamp -1 is returned.
Adding these two conventions together: if some error happens when handling file timestamp, then the file is considered as "not loaded" (sort of... partially)

Now back to why we need local times at all.
The engine uses zip file format for all assets on the player side. Zip format is very old, and it stores last modification time of each file in DOS format. Which is just "year, month, day, hour, minute, second", without any timezone/DST info. By common convention, everyone treats DOS time as local time.
Since zip modification times are considered local, the engine converts them to Unix Timestamps on reading. This is done in function Sys_DosToUnixTime, and it calls aforementioned mktime standard function to convert from local time to Unit Timestamp.

Here are some problems:
1) mktime function works differently depending on locale/timezone/DST/whatever. It may fail due to some DST weirdness or due to random value stored in zip datetimes.
2) Sys_DosToUnixTime function does not initialize the field "dostm.tm_isdst" before calling mktime. Depending on the sign of this value, mktime considers input to be at non-DST/DST/uncertain moment, which may result in different output or even fail (depending on locale).
3) The pk4 files have somewhat imprecise timestamps. Because TDM server packages everything in one timezone, while players read these zips in another timezone.