View Issue Details

IDProjectCategoryView StatusLast Update
0006253The Dark ModCodingpublic16.06.2024 00:13
Reporterstgatilov Assigned Tostgatilov  
PrioritynormalSeveritynormalReproducibilityN/A
Status resolvedResolutionfixed 
Product VersionTDM 2.11 
Target VersionTDM 2.13Fixed in VersionTDM 2.13 
Summary0006253: Third-party libraries update and conan 2
DescriptionTime to update dependencies =)

Also, since conan 2 is long in beta and will become main version soon, I think I should start using it.
That could force me to fix/update/rewrite some custom recipes.
TagsNo tags attached.

Relationships

related to 0006314 resolvedstgatilov fails to build: ffmpeg 

Activities

es20490446e

es20490446e

31.03.2023 18:24

reporter   ~0015985

It's now the main version ;)

To use the previous one:
pip --quiet install conan==1.59.0
stgatilov

stgatilov

22.05.2024 11:05

administrator   ~0016703

Finally did the transition to conan 2:
  r10738 Upgraded third-party libs build scripts to conan 2.
  r10739 Renamed BLAKE2 recipe directory to blake2_temp.
  r10740 Renamed blake2_temp recipe directory to blake2.
  r10742 Regenerated all artefacts from scratch.
  r10743 Renamed BLAKE2 artefacts directory to blake2_test.
  r10744 Renamed blake2_test artefacts directory to blake2.
  r10745 Fixed case of blake2 artefacts directory in CMake configs.

Right now everything is as close as possible to the previous state.
All the same version of all the libs are used, the scheme is the same.

I think I'll publish a dev build with this, then proceed to upgrade the libs.
During upgrade, it would be worthwhile to look at issues:
  * libjpeg is build with "link-time code generation" on Windows, which produces larger .lib files and slows down TDM linking (with warning).
  * maybe newer ffmpeg can be built trash-less without additional options? =)
  * don't forget to upgrade mbedtls (dependency of libcurl)

Another question to investigate is: can we make the connection between build systems and conan thinner?
New conan has "deploy" concept which puts all artefacts in some place, maybe we can use it.
Maybe we can also use the conan-generated configs for MSVC and CMake... although this seems more fragile.
es20490446e

es20490446e

22.05.2024 13:19

reporter   ~0016704

That said... the Bincrafters repo seems kind of abandoned:
https://github.com/bincrafters/community
stgatilov

stgatilov

06.06.2024 05:16

administrator   ~0016723

Tightened conan integration:
  r10757 Third-party libs integration with conan has been greatly reworked.
  r10758 Updated third-parth artefacts with the new layout.
  r10759 Fixed vcxproj integration of conan: added bitness as suffix to artefacts directory.

Previously conan was use only to generate artefacts (headers + libs) into special directory, and only for the main platforms we have.
All the glue for builds systems (CMake, vcxproj) was written manually then.
There was a way to do something custom, but it worked under obscure THIRDPARTY_PLATFORM_OVERRIDE and forced everyone to deal with the same 2_build_all.py script.

Now conan generates all the glue automatically, and it can be used more or less like in a typical conan project.
You are not forced to generate artefacts into local directory in order to build TDM/installer/whatever.
You can do ordinary conan install into conan cache and then use the libs directly in TDM build.

Of course, the old system with putting all the artefacts into "artefacts" subdirectory so that people could use it without conan is still used for main platforms.
The main difference in this workflow is that a custom "deployer" is injected into conan install --- that's the new conan 2 feature.

Ideally, now you should be able to build TDM locally with autodetected settings like this:
  conan build . -of temp -o build_game=True -o build_installer=True -o build_packager=True -b missing
This would also build installer and packager.
It might not work depending on setting though --- perhaps you'll need to customize something.

The price for this is that conan-generated glue is 0000827:0001300 text file of size 4 MB in total.
That's because conan generates separate set of files for every MSVC configuration (we have 5 of them) for every bitness (2 cases), and separate set of CMake configs for every platform (2 cases),every arch (2 cases), and every config (2 cases). That's 18 sets of glue files and 18 libraries in each set --- 324 glue instances. And conan also generates lib glue in several files =)

Now I want to configure some CI to build all the stuff using all workflows:
  1) Build stock vcxproj and CMake with prebuilt artefacts.
  2) Build directly with conan build.
  3) Delete prebuilt artefacts, regenerate them and build stock vcxproj and CMake after that.
stgatilov

stgatilov

14.06.2024 21:34

administrator   ~0016738

The new stuff did not build, here are the fixes:
  r10760 Fixed mbedtls case in our CMakeLists.txt to fix build on Linux filesystem.
  r10761 Added relativization of paths in .props generated by conan as postprocessing.
  r10762 Rerun artefacts generator to get relative paths in props.

And then I worked on restoring AppVeyor CI:
  r10763 Added a bunch of script to test builds extensively on CI.
  r10765 Major rewrite of AppVeyor CI configuration.
  (this attempts continued until rev 10796)

As the result, here we have 15 builds:
  https://ci.appveyor.com/project/thedarkmod/trunk
Takes 5.5 hours to build them all =)
Still need to put this onto daily/weekly schedule.

The individual builds are parametrized:
OS:
  Windows / Linux
BITNESS:
  64 / 32
THIRDPARTY:
  artefacts --- use prebuilt artefacts from SVN
  rebuild --- use script to force conan rebuild artefacts, then kill everything conan-related and build from fresh artefacts
  conan --- use conan build command, which builds/prepares everything in local conan cache and generates projects directly
CONFIGURATION:
  msvc_main --- builds game using MSVC solution directly, only Release + Debug configurations
  msvc_extra --- builds game using MSVC solution directly, other 3 configurations
  cmake --- builds game, installer, and packager using CMake

Obviously, "msvc_*" configurations are only use on Windows.
"conan" thirdparty mode is only used with "cmake" configuration.
Only bitness = 64 is tested on Linux, since I did not manage to set up 32-bit builds on AppVeyor.
Finally, "msvc_extra" is only built with "artefacts" configuration (don't try to rebuild stuff for them).
es20490446e

es20490446e

16.06.2024 00:13

reporter   ~0016743

fails to build: ThirdParty/custom/openal

https://bugs.thedarkmod.com/view.php?id=6544

Issue History

Date Modified Username Field Change
07.02.2023 19:46 stgatilov New Issue
07.02.2023 19:46 stgatilov Status new => assigned
07.02.2023 19:46 stgatilov Assigned To => stgatilov
31.03.2023 18:24 es20490446e Note Added: 0015985
03.01.2024 01:42 nbohr1more Target Version TDM 2.12 => TDM 2.13
22.05.2024 11:05 stgatilov Note Added: 0016703
22.05.2024 13:19 es20490446e Note Added: 0016704
25.05.2024 18:28 stgatilov Relationship added related to 0006314
06.06.2024 05:16 stgatilov Note Added: 0016723
14.06.2024 21:34 stgatilov Note Added: 0016738
14.06.2024 21:34 stgatilov Status assigned => resolved
14.06.2024 21:34 stgatilov Resolution open => fixed
14.06.2024 21:34 stgatilov Fixed in Version => TDM 2.13
16.06.2024 00:13 es20490446e Note Added: 0016743