View Issue Details

IDProjectCategoryView StatusLast Update
0005250The Dark ModDistributionpublic19.01.2021 15:29
Reporterstgatilov Assigned Tostgatilov  
PriorityhighSeveritymajorReproducibilityN/A
Status resolvedResolutionfixed 
Product VersionTDM 2.08 
Target VersionTDM 2.09Fixed in VersionTDM 2.09 
Summary0005250: New differential update and weekly builds
DescriptionFull description is provided here:
  https://forums.thedarkmod.com/index.php?/topic/19837-208-new-tdm_update/

The rough idea is to:
1) Add new type of manifests, listing all files inside zips with secure BLAKE2 hashes.
2) Differential update over HTTP, downloading chunks directly from zips (strongly ties updater to zip file format).
3) Allow using several manifest/sources to build single version of TDM.
TagsNo tags attached.

Relationships

related to 0004833 resolvedstgatilov zsync-like differential updates (unofficial prototype) 
related to 0005076 resolvedstgatilov Switch updater/packager Linux build from SCons to CMake. 
related to 0005495 resolvedstgatilov Terminate tdm_update support 

Activities

cabalistic

cabalistic

11.05.2020 10:20

developer   ~0012486

Purely out of curiosity: why BLAKE2, specifically? :)
stgatilov

stgatilov

11.05.2020 10:28

administrator   ~0012487

It is quite recent and secure. And among such hashes, this one is very fast.

Since the updater will update from arbitrary current state, it will have to compute hashes of the whole TDM installation every launch.
This is the biggest downside of the new system, so the speed of hashing matters a lot.
stgatilov

stgatilov

11.05.2020 10:29

administrator   ~0012488

Moreover, it is 32-bit BLAKE2 hash, since our Windows updater is still 32-bit =)
Although it we would stop providing 32-bit build of TDM out of the box, it would probably make sense to make updater 64-bit too.
cabalistic

cabalistic

11.05.2020 10:40

developer   ~0012489

Ok, fair enough :) Btw. the BLAKE2 website states that BLAKE3 is available and even faster :D

What is not clear to me, though, is if the speed comparisons against other hash algorithms take hardware acceleration into account? For example, SHA1 and SHA256 have specific SSE instructions supported by Intel and Ryzen CPUs.
stgatilov

stgatilov

11.05.2020 11:10

administrator   ~0012490

I think SHA instructions are not taken into account, because only the most recent CPUs have them. I wouldn't rely on them either.
BLAKE2 uses something like SSE2 which is taken into account and is in fact required by most software.
Besides, are there instructions to accelerate SHA2 and SHA3 computation?

BLAKE3 did not even come out when I searched for a hash function. And I don't see any security analysis of it yet.
cabalistic

cabalistic

11.05.2020 11:17

developer   ~0012491

SHA-256 and SHA-224 are supported as part of the SHA-2 family, but nothing above that. Also you're right; it appears that Intel has chosen to implement those instructions only on a subset of specialized CPUs, and for standard desktop CPUs the instructions are only available from Cannon Lake upwards. So actually, AMD with Ryzen now has probably more widespread support for it than Intel :D
stgatilov

stgatilov

02.07.2020 17:30

administrator   ~0012644

I decided to settle on FLTK for GUI.
Perhaps one day we can use it for tools --- although we should decide if it would be better or worse than ImGui.

Added third-party libs: svn rev 8784 and 8786.

First version of installer committed in svn rev 8790 and 8791.
Of course, nothing works properly there =) I have just started.

The zipsync subdirectory will be kept in sync with my GitHub repo:
  https://github.com/stgatilov/zipsync
But updating that repo is purely my own responsibility, of course.
stgatilov

stgatilov

09.07.2020 16:49

administrator   ~0012647

After a lot more fixes, the installer has been published for preliminary testing:
  https://forums.thedarkmod.com/index.php?/topic/20460-new-tdm_installer-and-dev-builds/
stgatilov

stgatilov

19.07.2020 12:53

administrator   ~0012661

Last edited: 19.07.2020 12:53

Something like 7 updates to tdm_installer has already been published, and it is fairly usable now.

I have also written a detailed wiki article about it:
  https://wiki.thedarkmod.com/index.php?title=Tdm_installer_and_zipsync
stgatilov

stgatilov

08.11.2020 04:01

administrator   ~0012865

Added two new features:

1) Better mirrors handling.
First of all, mirrors can now be weighted in config file --- for better load balancing.
Second, if one mirror goes down, it does not prevent installation now.
Except for the core files (installer, config, target manifest), which must always come from the main TDM server.

2) Added "unattended" mode.
In theory, it allows installing TDM as part of some script.
stgatilov

stgatilov

17.12.2020 03:56

administrator   ~0013154

The installer is being used in 2.09 beta, and the feedback has been good so far.
I think I'll consider it "done" and create new issues if some problems happens.

Issue History

Date Modified Username Field Change
11.05.2020 09:45 stgatilov New Issue
11.05.2020 09:45 stgatilov Status new => assigned
11.05.2020 09:45 stgatilov Assigned To => stgatilov
11.05.2020 09:45 stgatilov Relationship added related to 0004833
11.05.2020 10:20 cabalistic Note Added: 0012486
11.05.2020 10:28 stgatilov Note Added: 0012487
11.05.2020 10:29 stgatilov Note Added: 0012488
11.05.2020 10:40 cabalistic Note Added: 0012489
11.05.2020 11:10 stgatilov Note Added: 0012490
11.05.2020 11:17 cabalistic Note Added: 0012491
02.07.2020 17:30 stgatilov Note Added: 0012644
09.07.2020 16:49 stgatilov Note Added: 0012647
19.07.2020 12:53 stgatilov Note Added: 0012661
19.07.2020 12:53 stgatilov Note Edited: 0012661
08.11.2020 04:01 stgatilov Note Added: 0012865
17.12.2020 03:56 stgatilov Note Added: 0013154
17.12.2020 03:56 stgatilov Status assigned => resolved
17.12.2020 03:56 stgatilov Resolution open => fixed
17.12.2020 03:56 stgatilov Fixed in Version => TDM 2.09
19.01.2021 15:28 stgatilov Relationship added related to 0005076
19.01.2021 15:29 stgatilov Relationship added related to 0005495