View Issue Details

IDProjectCategoryView StatusLast Update
0005349The Dark ModCodingpublic01.12.2020 17:44
Reporterstgatilov Assigned Tostgatilov  
PrioritynormalSeveritytweakReproducibilityN/A
Status resolvedResolutionfixed 
Product VersionTDM 2.08 
Target VersionTDM 2.09Fixed in VersionTDM 2.09 
Summary0005349: Implement priorities and checksums for FM download mirrors.
DescriptionSome mirror owners can't handle too much traffic.
It's better to choose random mirror according to weights specified in downloaded XML.

Another thing is that downloader should verify checksum if XML file contains it.

BTW, probably makes sense implement weighting in tdm_installer too.
TagsNo tags attached.

Relationships

related to 0003419 resolvedtels in-game downloader does not randomize the server URLs 

Activities

stgatilov

stgatilov

08.11.2020 04:12

administrator   ~0012866

Here is how the XML looks now:

<tdm>
<availableMissions>
<mission id="146" title="A Good Neighbor" releaseDate="2020-03-08" size="81.3" version="2" internalName="good" type="single" author="Amadeus">
<downloadLocation language="English" url="http://www.southquarter.com/tdm/fms/good.pk4" />
<downloadLocation language="English" url="http://bloodgate.com/mirrors/tdm/fms/good.pk4" />
<downloadLocation language="English" url="http://darkmod.taaaki.za.net/fms/good.pk4" />
<downloadLocation language="English" url="http://www.fidcal.com/darkuser/missions/good.pk4" />
</mission>
<mission id="101" title="A House of Locked Secrets" releaseDate="2015-05-29" size="195.6" version="1" internalName="ahouseoflockedsecrets" type="multi" author="Moonbo">
<downloadLocation language="English" url="http://www.southquarter.com/tdm/fms/ahouseoflockedsecrets.pk4" />
<downloadLocation language="English" url="http://darkmod.taaaki.za.net/fms/ahouseoflockedsecrets.pk4" />
<downloadLocation language="English" url="http://www.fidcal.com/darkuser/missions/ahouseoflockedsecrets.pk4" />
<downloadLocation language="English" url="http://bloodgate.com/mirrors/tdm/fms/ahouseoflockedsecrets.pk4" />
</mission>
<mission id="117" title="A Matter of Hours" releaseDate="2017-04-28" size="1.3" version="1" internalName="matterofhours" type="single" author="Springheel">
<downloadLocation language="English" url="http://www.fidcal.com/darkuser/missions/matterofhours.pk4" />
<downloadLocation language="English" url="http://darkmod.taaaki.za.net/fms/matterofhours.pk4" />
<downloadLocation language="English" url="http://www.southquarter.com/tdm/fms/matterofhours.pk4" />
<downloadLocation language="English" url="http://www.mindplaces.com/save/matterofhours.pk4" />
<downloadLocation language="English" url="http://bloodgate.com/mirrors/tdm/fms/matterofhours.pk4" />
</mission>
<mission id="147" title="A Night Of Loot: One Man's Treasure" releaseDate="2020-06-13" size="3.9" version="1" internalName="anol" type="single" author="OGDA">
<downloadLocation language="English" url="http://darkmod.taaaki.za.net/fms/anol.pk4" />
<downloadLocation language="English" url="http://www.fidcal.com/darkuser/missions/anol.pk4" />
<downloadLocation language="English" url="http://www.southquarter.com/tdm/fms/anol.pk4" />
<downloadLocation language="English" url="http://bloodgate.com/mirrors/tdm/pub/pk4/fms/anol.pk4" />
</mission>
<mission id="52" title="A Night to Remember" releaseDate="2011-10-31" size="33" version="2" internalName="antr" type="single" author="Fieldmedic">
<downloadLocation language="English" url="http://darkmod.taaaki.za.net/fms/antr.pk4" />
<downloadLocation language="English" url="http://www.fidcal.com/darkuser/missions/antr.pk4" />
<downloadLocation language="English" url="http://www.southquarter.com/tdm/fms/antr.pk4" />
<downloadLocation language="English" url="http://bloodgate.com/mirrors/tdm/fms/antr.pk4" />
<localisationPack url="http://www.fidcal.com/darkuser/missions/antr_l10n.pk4" />
<localisationPack url="http://bloodgate.com/mirrors/tdm/fms/antr_l10n.pk4" />
<localisationPack url="http://darkmod.taaaki.za.net/fms/antr_l10n.pk4" />
</mission>
...
stgatilov

stgatilov

08.11.2020 04:13

administrator   ~0012867

The most appropriate fix would be to add "weight" attribute to "downloadLocation: and "localizationPack". Then:
1) The old TDM versions would ignore it if it is present.
2) The new TDM versions can have a default weight of "1.0" if it is not set.
So the change would be fully compatible both ways.

‚Äč
stgatilov

stgatilov

14.11.2020 04:12

administrator   ~0012919

Committed in svn rev 8977.

Among several provided URLs, the probability to choose a particular one is proportional to its weight.
The syntax for adding weights is like this:
  <downloadLocation language="English" weight="0.1" url="http://bloodgate.com/mirrors/tdm/fms/good.pk4" />
If weight is not specified, then it gets default unit value (i.e. 1.0).

Note that it won't have effect until someone updates missionlist.xml on server.
See also this discussion:
  https://forums.thedarkmod.com/index.php?/topic/20624-store-missions-archive-in-svn/
stgatilov

stgatilov

01.12.2020 17:44

administrator   ~0013066

Implemented reading and verifying "sha256" attribute of download links (if it is present).
In svn commits 9022 & 16023.

If checksum does not match, then custom error message is shown (downloaded file is deleted immediately).
If attribute is not set, then it is not checked.
Normally the whole process is silent.

I use unoptimized implementation of SHA256 and compute it on-the-fly.
I believe that since download speed is anyway much slower than sha256 computation, computing hash in curl write callback should not increase wall time of download.

Issue History

Date Modified Username Field Change
01.10.2020 05:03 stgatilov New Issue
01.10.2020 05:03 stgatilov Status new => assigned
01.10.2020 05:03 stgatilov Assigned To => stgatilov
01.10.2020 05:04 stgatilov Description Updated View Revisions
08.11.2020 04:12 stgatilov Note Added: 0012866
08.11.2020 04:13 stgatilov Note Added: 0012867
14.11.2020 04:07 stgatilov Relationship added related to 0003419
14.11.2020 04:12 stgatilov Note Added: 0012919
14.11.2020 04:12 stgatilov Status assigned => resolved
14.11.2020 04:12 stgatilov Resolution open => fixed
14.11.2020 04:12 stgatilov Fixed in Version => TDM 2.09
01.12.2020 17:19 stgatilov Summary Implement priorities for FM download mirrors. => Implement priorities and checksums for FM download mirrors.
01.12.2020 17:19 stgatilov Description Updated View Revisions
01.12.2020 17:44 stgatilov Note Added: 0013066