| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002247 | The Dark Mod | Animation | public | 08.06.2010 14:23 | 11.02.2024 21:14 |
| Reporter | Springheel | Assigned To | |||
| Priority | normal | Severity | normal | Reproducibility | have not tried |
| Status | new | Resolution | open | ||
| Product Version | SVN | ||||
| Summary | 0002247: Facial Animations | ||||
| Description | Try making some head animations for non-idle states, like combat or pain. | ||||
| Tags | animation, realism | ||||
 |
models_src/proguard_mb has the following which might be for new head rig: head_idle.mb head_blink.mb head_death.mb |
 |
I am sure this is probably too much work but I thought I would post it for your consideration. Since all vertex animations require significant CPU overhead, I was wondering if it would be cheaper to use an animated normal map to change facial expressions. This would likely be light enough to allow very detailed animation changes akin to Source engine facial tech. You would create several template facial positions for categories of expressions then create scrolling normal maps for the actual animation. You'd probably need to bake the whole head for each animation frame unless you could break off just the "face portion". (As I said, probably too much hassle.) |
|
|
Setting aside the fact that you'd have to do that individually for every head (since they don't share normalmaps), it also would require code support, and I can't imagine how it would look convincing if the actual geography of the face didn't move. |
|
|
A note for later: I suspect that the lipsynching code might be overriding the head animations. |
|
|
The only thing that would look wrong is the silhouette. You could mitigate that a little by animating BOTH the low poly mesh and the normal map but you'd loose most of the performance benefit. Not sure you'd need code support? The animation editor allows you to apply different materials at different frames, no? Or different "Skins" ? "All" you would need to do is specify animated normal map A at frames 8 through 20 (etc) but it would also require careful timing between the two animation types. (and a hell of a lot of normal map baking... ) If not for the bulk of the animation, this could possibly be used for small animations like an eyebrow lift or small smirk when a conversation perks up to something interesting (etc) |
|
|
I don't know how you would presume to make the mouth or brows look like they're moving when none of the geography around the mouth moves. As for code support, what happens when the animation is interrupted before the frame command that restores the skin is called? It just doesn't seem practical on any level, and since the head mesh is constantly animated anyway, I can't see why there would be any performance saving. You might be able to do very tiny modifications, but it hardly seems worth all the effort for that. |
|
|
Ah, so the head mesh is always regenerated per frame even when no features change. Oh well... I suppose you could lower the frame rate to save a little... The normal map would be a bake of a full head animation frame so you wouldn't just be moving the mouth or eye per se. Of course, adding to the impracticality is that you would need to turn each MD5 anim frame into a high-poly static model for the normal. Yes, state interruption would be a nightmare to coordinate. Edit: Nope, when the vertex positions do not change between frames, a number of CPU intensive operations are conserved per this Intel document on Id Tech 4: http://software.intel.com/en-us/articles/optimizing-the-rendering-pipeline-of-animated-models-using-the-intel-streaming-simd-extensions/ So animated normal maps would improve performance (just for the record). |
|
|
Someone else is welcome to explore animated maps if they wish. All the other complications aside, I can't believe it would have any noticeable performance impact. A single AI has thousands of animated verts. Facial animations might use two dozen verts at a time...you'd need hundreds of AI on screen to equal the impact of a single AI. |
|
|
I was imagining giving the facial animation the apparent appearance of hundreds of animated bones verses the current numbers. Nonetheless, your point is valid. Without significantly boosting the facial animation quality, animated normalmaps would not offer any noticeable optimization. Yes, the verdict is not looking good on the animated normalmap concept. If videomap could be used for diffuse and normals then it would be a more realistic concept to approach. Until then, (if ever) it would require too much video RAM ??? or wouldn't offer much of a quality boost ??? After reading more about all the technical pitfalls of LA Noire's version, I've come to the conclusion that it's also not a good approach in general... Not sure how that translates to artist generated frames compared to "captured" frames though... |
|
|
Here's something from tdm_ai_base.script, don't know if it's relevant: void ai_darkmod_base::Head_Idle() { if (hasAnim(ANIMCHANNEL_HEAD, "idle")) { // play the animation playCycle(ANIMCHANNEL_HEAD, "idle"); eachFrame { if (getMoveType() == MOVETYPE_SLEEP) { animState(ANIMCHANNEL_HEAD, "Head_Idle_Sleep", 4); } } } } |
|
|
This would be useful: tdm_ai_showanimstate Shows the current anim states of the anim channels torso, legs and head, as well as their wait states. At least I could find out what animation the head is actually playing. |
|
|
Hmm, testing with that on gave confusing results. The head channel is listed as playing head_idle constantly, even when the AI is talking (even though it is actually playing head_talk.md5anim at this point). It must also be playing head_blink regularly. When the AI is killed, the head channel shows head_death. Also noticed that there is a channel eyelids on the head mesh. Here are the defined animations: anim idle models/md5/chars/heads/npcs/head_idle.md5anim anim blink models/md5/chars/heads/npcs/head_blink.md5anim anim talk1 models/md5/chars/heads/npcs/head_talk.md5anim anim death models/md5/chars/heads/npcs/head_death.md5anim { prevent_idle_override } anim sleep Trying to create a "walk" animation had no effect; the Ai still played head_idle while walking. |
|
|
Going back to this: void ai_darkmod_base::Head_Idle() { if (hasAnim(ANIMCHANNEL_HEAD, "idle")) { // play the animation playCycle(ANIMCHANNEL_HEAD, "idle"); eachFrame { if (getMoveType() == MOVETYPE_SLEEP) { animState(ANIMCHANNEL_HEAD, "Head_Idle_Sleep", 4); } } } } it sounds like the head is supposed to play head_idle constantly unless "movetype_sleep" is true. Does that mean every animation would need it's own "if" statement in order to play a different head animation? |
|
|
In the same script at the beginning, there is a list of animations, but only one has a Head channel listed: void Torso_Idle_Sleep(); void Legs_Idle_Sleep(); void Head_Idle_Sleep(); Also, it seems like each animation lists actions for the torso, and legs separately. Sleeping also includes one for the head: void ai_darkmod_base::Head_Idle_Sleep() { if (hasAnim(ANIMCHANNEL_HEAD, "sleep")) { // play the animation playCycle(ANIMCHANNEL_HEAD, "sleep"); eachFrame { if (getMoveType() != MOVETYPE_SLEEP && !AI_KNOCKEDOUT && !AI_DEAD) { animState(ANIMCHANNEL_HEAD, "Head_Idle", 4); } } } } So does death: void ai_darkmod_base::Head_Death() { if (hasAnim(ANIMCHANNEL_HEAD, "death")) { // play the animation playAnim(ANIMCHANNEL_HEAD, "death"); while( !animDone(ANIMCHANNEL_HEAD, 0)) { waitFrame(); } } // finish up and disable this animchannel finishChannelAction(ANIMCHANNEL_HEAD, "death"); finishAction("death"); wait(1); disableAnimchannel(ANIMCHANNEL_HEAD); } and KOing |
 |
The reason those scripts are hard to interpret is that under the hood each AI is 2 animated entities: a body, which is the AI entity, and an attached head, which is an AF Attachment. I suspect you couldn't see blinking and talking using tdm_ai_showanimstate because it was showing you the head channel of the AI entity, instead of an animchannel on the head entity. Every animated entity, even something as simple as a waving flag, has the same 5 anim channels: ANIMCHANNEL_ALL ANIMCHANNEL_TORSO ANIMCHANNEL_LEGS ANIMCHANNEL_HEAD ANIMCHANNEL_EYELIDS Entities that have legs and torsos etc apparently map a set of joints to each channel. Not found that mapping yet but I read it on the wiki and it makes sense. Entities that don't have arms and legs, like flags, simply use ANIMCHANNEL_ALL for their animations, which affects all joints. When the scripts request an animation on ANIMCHANNEL_HEAD, *some* of the game functions treat that as a special case and they find the head that's attached to the AI and play the animation on that entity instead. But not necessarily all. Some might simply play it on the head channel of the body mesh, which would affect either nothing (if no joints are mapped to HEAD), or only the angle and position of the head. Either way they wouldn't affect the joints *in* the head. How did you want to start head animations? By implementing a rule that says, "if the head has a matching animation to what gets played on the torso, then play it on the head too?" Or something more flexible? in the meantime, I'll carry on mapping out the current set-up of any script that specifies ANIMCHANNEL_HEAD. |
|
|
Thinking in ideal terms, There are four different kinds of animations we might want to play on the head: 1. Random blinking (limited to eyelid channel) 2. Lip-syncing when talking (already covered in a primitive way) 3. Animations linked to body animations 4. Facial animations to indicate mood 0000001 and 0000002 already work. 0000003 would be covered by a "matching animation" system. 0000004 might not be necessary, but I could imagine mappers wanting to call certain facial animations when using the conversation system--for example, an AI plays an "angry" animation on the head while hearing bad news. //Not found that mapping yet// That's done in the modelDef--you can see it in any ai .def file. |
|
|
I've taken a read through the game code, and this all *ought* to work already, but the scripts might not be set up to make use of it. I've not read through those thoroughly yet, but you quoted the relevant stuff above. 3) Should work by adding a call to SyncAnimChannels to sync the head after starting an animation on the torso. SyncAnimChannels is already set up to look for an anim on the head that has the same name as the torso anim, and to match up the frame (by number). I'm not sure whether any of the scripts do that though, so we'll need to think about when we want it to happen. 4) Is supportable through the existing script events PlayAnim and PlayCycle, which already forward the play request to the head entity when called with ANIMCHANNEL_HEAD. But it'll need a new custom AnimState script creating for the head, so that the custom anims don't get interrupted by idle anims. The game scripts currently provide custom animstates only for torso and legs. Those are used by the path tasks and conversations as well as the AI state system. Right now the game only has a very few AnimStates for the head: those are the scripts you quoted above: sleeping and death and idle. No reason why there can't be a custom animstate for the head too, that allows a mapper-specified animation to play without interruption. We'll need to tweak the scripts you quoted above too so that the head can drop out of its idle state without having to be asleep. To test it all I'll need to find (1) a head animation to play in isolation, and (2) two compatible anims for head and body with the same name and number of frames. Any ideas? |
|
|
Hmm, 2 is something I hadn't considered before. When I was running tests I was just taking an existing head animation, like "death" and renaming it. If it didn't have the same number of frames as the body animation, would that cause it not to work? |
|
|
It should still work. It just won't be synchronised in any meaningful sense. I'll check but I'm pretty sure that if the sync code requests frame 45 on the head, and its anim has only 30 frames, it'll wrap and start playing at frame 15. |
|
|
Heads only have 2 effective anim channels: ALL and EYELIDS. No others are defined. Lip sync plays on animchannel_all, and it does something unique: it plays the frames of the animation out of sequence, by selecting a frame based on the sound. So we'll need to avoid mixing it with custom anims. Then again custom anims will play on animchannel_all too, so lip syncs can be allowed to override custom anims if a bark kicks in. That's almost certainly what's causing a clash with blinking 0003605, even if I haven't been able to observe it yet. The lip sync will be interpolating its own positions for the eyelids with the blink animation. I reckon we should consider mapping all non-eyelid joints to ANIMCHANNEL_HEAD, so that lip sync and custom anims can be independent of blinking. |
|
|
"lip syncs can be allowed to override custom anims if a bark kicks in." Hmm. That's going to be limiting, but I'm not sure we can do anything about it. For example, it would be nice if while in combat, AI had an "angry" expression. But they also make barks while attacking. Does that mean the head would snap from angry to neutral to angry? Not sure how that would look. How difficult is it to establish new animation channels? "I reckon we should consider mapping all non-eyelid joints to ANIMCHANNEL_HEAD" How would this affect the KO and sleep animations, that control the eyelids? |
|
|
// How difficult is it to establish new animation channels? // It should be straightforward, as long as you don't want to exceed the standard 4 sets of custom mappings per animated entity. They can re-use the existing names, with whatever joints mapped, or even use new names for the existing channel slots. // How would this affect the KO and sleep animations, that control the eyelids? // I guess those would still play on animchannel_all, so they'd still affect all joints. Forgot to mention, animchannel_head on the head is unused right now. Those scripts above that play those anims on animchannel_head: those scripts are on the AI not the head, and the code passes them on to animchannel_ALL on the head. Sleep and death already have their own animstate scripts, so it's easy to make special cases of them. Lip syncing and custom anims could play on the non-eyelid joints. I guess we could split up the head further to try to isolate lip movement, but I'm not sure how it would look if the face frowns but the lips don't. Maybe not bad, and we can experiment with blending. Lip syncing is already another special case, so its settings can be tweaked. |
|
|
My thought was, if it's not too difficult to make new channels, we could create a separate channel for the mouth. Then lipsynching would run only on that channel. If I have this right, that means an angry combat animation could run on the _all channel, but animations on the "mouth" channel would override them--this would result in the AI keeping his angry face when yelling a combat bark--only his mouth would move. |
|
|
That's a good alternative. I'm looking through the existing setup so I can work out what will affect what. |
|
|
Random notes An Actor's "headAnim" AnimState controller is hard-wired to animchannel_all on the head. Can be changed but a lot of stuff uses it, and nothing has to specify a channel. Opened a new tracker for some animation-related script events that look broken for heads. TODO: Check whether any of these are actually used. |
|
|
Attached a diagram laying out the various paths by which anims get played or controlled on the head. Most stuff goes through the Actor's headAnim controller which currently always specifies animchannel_all (i.e. all joints) on the head. Lip syncing is a special case so could be unhitched from that method and played on a new lip channel without disrupting the rest of the code. That would fix the other issue (eyelids jittering when speaking) but it probably won't play well with potential new head anims like frowning, if those play on animchannel_all. Making all anims default to a new channel that misses out lips and eyelids wouldn't work too well either when the lips *aren't* talking. And some of those anims need to play on the eyelids as you said: sleep, death etc. I'll continue to mull over my diagram. |
|
|
I won't pretend to understand the diagram....but do we have control over which channels take precedence if there's a conflict? |
|
|
We do, there are a few tools for controlling the precedence: syncing one channel to another; starting an overriding anim later so it takes over; and tweaking animation weights manually if necessary. The tricky bit is how to get it to work without making the picture above even more complicated, or duplicating the more complex setup used for the main Actor's anim channels which would be overkill. Heads have one anim channel and one anim state right now. I can't fiddle with that anim state to build in special rules because it's the same anim state code that's used for the main Actor's anim channels. The same goes for over half the code in the diagram. But there's usually a way. How do I find out exactly which joints are used by the talk anims? Looking at models\md5\chars\heads\npcs\head_talk.md5anim, the only joints with flags are "LipUpperMiddle", "MouthRight", and "jaw". Not mouth left. Does that sound right? |
|
|
At the moment, as far as I know, "lip" synching only uses a single joint--the jaw. Although the head skeleton has several bones around the mouth, none of them are currently used by existing animations. It would be cool to have more accurate lip-synching that used more bones, but I imagine that's a complex task. edit: I went back and looked at the animation, and it looks like there are two other bones being moved--- mouth3 and mouth9. I'm not sure exactly what those bones do, however, as the vast majority of the heads (if not all) have no verts weighted to those bones. |
 |
I just came across this, and thought it might be worth posting the link: http://www.lunaran.com/page.php?id=181 |
