View Issue Details

IDProjectCategoryView StatusLast Update
0006118DarkRadiantSaving and loadingpublic06.10.2022 18:55
ReporterLDAsh Assigned To 
PrioritylowSeverityfeatureReproducibilityhave not tried
Status newResolutionopen 
Product Version3.3.0 
Summary0006118: Rotation matrix floats to transform matrix radians (greater exports)
DescriptionI'd like to propose a (possibly) simple solution to unlocking greater export functionality to DarkRadiant, specifically for projected lights. I'm working on a combination GAWK/Python/CMD toolset for translating MAP files to DAE files so lights can be imported into Blender/3DSMAX. This can work great, location/scale/length/cone/colour/intensity/etc, everything else, but after literally weeks of bashing my head against brick walls, I just can not handle the rotation matrix.

The solution I'm currently going for is a Python script that will match the closest-nearest between 2 lists of arrays, which compares from a list of about (a globe of) 200K lights to quantise with. Once that's done, the (selected parts of the) MAP file can be translated with Regex with a matching (globe of lights) DAE file, so although a lot of precision is lost, the lights can be finally imported, at least.

This hacky solution is already really tough for me, but after weeks it's seriously the best I can do, and it's still a struggle.

Hoping by posting this, some solution may be possible from DarkRadiant itself. If rotation matrices can be recalculated back and forth, then not only lights could be handled easily, but even paths. I would definitely make everything I'm doing publicly available, and the workflow I'm creating can even do materials and allow not just fully textured imports into Blender/3DSMAX, but with fleshed-out materials, to a degree. I'd love to share all of that when it's all ready for showtime.

It would be great to discuss if this proposal is easy or not, or if there's another means worth attempting.
Tagslighting

Activities

greebo

greebo

04.10.2022 16:44

administrator   ~0015289

To be honest, I don't have a clue what this is about :)... I gather you want to import something from DarkRadiant to Blender? But I don't understand what problem the rotations have, and why you need 200k lights? I'm rather lost.
orbweaver

orbweaver

04.10.2022 19:25

developer   ~0015291

I think this should be more of a forum discussion than a bug ticket. From what I can understand you want to import certain aspects of MAP files into Blender, for which Python is a reasonable solution especially since Python can be used to create Blender import scripts (like the ASE/LWO scripts which I maintain). However like Greebo I am not clear on the full requirements, and this doesn't seem like something that should be done with hacky AWK/CMD scripts and regexes, but rather with a properly developed Python script which runs inside Blender. It doesn't sound like this is a process which should involve DarkRadiant at all.
LDAsh

LDAsh

05.10.2022 01:00

reporter   ~0015294

I'm not a mathematician and not a proper programmer, but I can do some "tricks" in terms of data management, so I apologise for not being clear.

Blender has about 3 file formats that it can use to import lights. Only Collada can use a matrix, the others unwind euler angles, but the Collada matrix used is a 4X4 transform matrix using radians as its values. Radiant is using a 3X3 rotation matrix. I know these are compatible and possible to translate, but I spent weeks trying to figure out how the matrices work, how the winding works, how all angles have angles of their own but all share liminal proportions too, and also how to use Python to do this. Failure at each attempt.

I finally settled on a stoner hack using Regex, that will encode all values in the matrix to letters, spacing apart character entries further down the decimal places (to keep lower decimal values from overpowering) and find a nearest match by comparing against a premade file of about 200K lights, then matching indexes from a Collada file of about 200K lights. That means the rotation values are already there to translate to, it's just finding a nearest match using Regex which should result in a snap to the closest matching angle, in an awkwardly quantised way. Not pretty, could take a long time to process, but the best I can do and better than nothing. I will release this in the forum anyway. I don't mind people laughing at me.

I wanted to propose a Radiant feature, and honestly only because I think you guys could possibly tweak the core maths in the code (at least optionally) to also calculate using radians as values and in the same winding (this info is online, but doesn't help my brain too much) as the Collada format. Maybe the libraries used for DarkRadiant core maths already have that support waiting to be used? I just assumed that. I thought there's a chance it could be a trivial task to implement, since elaborate code to do it may already be there.

Example = changing something like "value: float" to "value: radian", or changing "winding: left/right, up/down" to "winding: up/down, left/right". An incite into what kind of programmer I am, but maybe something not too different from that.

So I popped in here to discuss that possibility - Radiant having an export function that calculates all rotation matrices into transform radians, so spotlights can be imported elsewhere.
It absolutely would be a better solution to have it done purely in Python and as a plugin rather than in the source, but I have no idea who would do that, because unfortunately I can't.
Regex hack-pack should be ready to share in a few weeks, depending on personal life.
greebo

greebo

05.10.2022 03:05

administrator   ~0015295

On a general note, I think a Collada exporter might be a feature that could be useful.

With regards to the transformation matrices: a matrix itself doesn't *use* radians or degrees, it's just a bunch of numbers that are used to transform coordinates or vectors. There are formulas which can be used to produce these matrices with degrees and radians respectively, but the result would be the same set of 4x4 or 3x3 numbers.

If something is using Euler angles on the other hand, there indeed are differences and conventions about the order in which the pitch/yaw/roll values are applied, which is influencing the result. But you said that Collada isn't using those.
LDAsh

LDAsh

05.10.2022 05:35

reporter   ~0015296

It does have a "TransRotLoc" method that uses some kind of other matrix, 3 arrays of 4 elements. I didn't assume it was exactly "euler angle", I think that uses quaternions, and then my damaged brain would finally explode and melt, for real. I did have a look at that but was like "nope, damn, that seems even harder!"

From my research, these conversions are really too complex for most folks. I found formulas, a LOT of formulas, but nothing I can possibly use for (my level of) programming, and no Python scripts or commandline utilities I could ever find. Euler-to-transform, yes, but not the reverse. I think it's something that needs a deeper fundamental solution from the outset, not tweaks after already exported, but that's where I'm stuck. That's why I assumed (and hoped) the math libraries DarkRadiant uses may have that built in? Maybe there are some flags to toggle to use that? Just as an option? Maybe just wishful thinking...

Really appreciate all your time, regardless. :)
greebo

greebo

06.10.2022 14:38

administrator   ~0015297

3 arrays of 4 elements is probably a memory-saving version of a 4x4 matrix with a constant 0,0,0,1 piece left out, so I'm pretty confident that this is a regular transformation matrix.

It's possible to convert the rotation part of this matrix to euler angles (which can in turn yield multiple sets of angles as valid solutions). But I really don't know if any of this is helping you on your project, since I'd have to see concrete examples to suggest anything.
LDAsh

LDAsh

06.10.2022 18:03

reporter   ~0015307

The "project" in this sense is a DarkRadiant light-exporter for every mapper to use. Since DR supports lights in a more useful way now, a perfect complement to the ASE exporters and such.

If you feel confident, here is a snippet of a Collada DAE file:-

      <node id="Spot" name="Spot" type="NODE">
        <translate sid="location">0 0 0</translate>
        <rotate sid="rotationZ">0 0 1 1.512392</rotate>
        <rotate sid="rotationY">0 1 0 32.39429</rotate>
        <rotate sid="rotationX">1 0 0 -48.94376</rotate>
        <scale sid="scale">2.188854 3.106285 2.568329</scale>
        <instance_light url="#Spot-light"/>
      </node>

What really scares me off is the "scale", makes me think it's not so simple. The fact that it's separate lines makes my whole data-management approach futile, simply because (at my skill level) I'd need to split those up too much. Light exporters sound great but not if they take 10 hours.

I can do a lot more examples if this is too vague and we can try to figure out what values are changing in-script when which axes are rotated in-software. It's just the correct values I want, not a whole Collada file. Everything but the rotations are trivial. That can be done data-management style and still be efficient, although just not elegant. Could just use Python to relay and still integrate it into DR, if that's important.

One thing I didn't mention is how the cubic volumes are handled currently. The highest value from both "top" and "right" determines the "cone". A mapper can't expect a long thin light, only symmetrical cones. Blender and 3DSMAX can do it, but, not straight-forward compared to the inherent nature of idTech lights. It exports spotlights as very general actual spotlights.
greebo

greebo

06.10.2022 18:31

administrator   ~0015308

I see. I'm interested in the Collada format in general, but I won't have the time to take on a full .dae exporter project right now. The light definition looks ok to me, obviously the translation part is mixed into the rotation matrix. This appears to be an unrotated light, if I'm not mistaken.

I'd definitely need more time to read into the format, and to create comparison scenes. If it's possible in principle to create Collada light definitions that have the same properties as idTech lights, then an exporter is possible too.
LDAsh

LDAsh

06.10.2022 18:55

reporter   ~0015311

Your word is definitely taken. :)

A full solution isn't necessary. The Collada file itself may support the same matrices but the software won't use them all, at least not as intended. I think Blender skews them but that offsets the angle more, and 3DSMAX just resets the proportions. The cubic volumes in either software are entirely different light objects altogether and I don't even think they can be skewed at all, only rotated and scaled.

I think most mappers will be fine with these limitations, and 99.9% of spotlights will be symmetrical and circular anyway. Skewing and scaling spotlights is something I can live without, because it's an obscure thing to do, point lights can still do that (even with my GAWK hacks) and that's likely a far more common practice among all mappers.

Issue History

Date Modified Username Field Change
04.10.2022 03:35 LDAsh New Issue
04.10.2022 03:35 LDAsh Tag Attached: lighting
04.10.2022 16:44 greebo Note Added: 0015289
04.10.2022 19:25 orbweaver Note Added: 0015291
05.10.2022 01:00 LDAsh Note Added: 0015294
05.10.2022 03:05 greebo Note Added: 0015295
05.10.2022 05:35 LDAsh Note Added: 0015296
06.10.2022 14:38 greebo Note Added: 0015297
06.10.2022 18:03 LDAsh Note Added: 0015307
06.10.2022 18:31 greebo Note Added: 0015308
06.10.2022 18:55 LDAsh Note Added: 0015311