Automate lip sync in Cheetah3D (Sound Loader.js script + Morph tag)

[Filip]

I was working on some basic lip sync animation in Cheetah3D and was looking for a way to automate the process. That's when I found the Sound Loader.js script (https://www.cheetah3d.com/forum/index.php?threads/12343/). I downloaded it from the creators website (http://tres-graficos.jp/blog/files/category.php?k=Script) and installed it successfully but I don't understand how to get it to work? I want to import an audio file, and convert the audio amplitude to keyframes for a slider in a Morph tag. Any ideas?

 

[Lydia]

You won't get a real LipSync with it, just a very basic one. But maybe that's sufficient for you. You will additionally need the F-Curve Controller Script - also by Hiroto.
This thread describes how it works. In the file "talking_face.jas" you can see how the soundloader is connected to the morph tag:

Sound Loader.js + F-Curve Controller.js demo

Sound Loader.js + F-Curve Controller.js demo Hello, I made some demo movie with my latest scripts. demo movie: https://youtu.be/54B9-l6lc-k Best tg_jp, Hiroto

www.cheetah3d.com

 

[Lydia]

I took another look at Hiroto's file - maybe a few comments:
If your character has another tag like Material, the morph tag must be to the left of it, otherwise it won't work.

Hiroto has created the morph slider in his file the other way around than you would probably normally do: open on the left, closed on the right. If you don't have it this way, it should work if you simply enter "Math.abs( v *2 )" instead of "1 - Math.abs( v *2 )" for "formula". (The higher the number after v *, the more extreme the movement)

The lip movements often become very hectic, jerky and choppy. You will get it smoother if you enter a much lower value than fps in your file for "resolution" in the soundloader before loading. For example, if you have 30 fps, then load with a resolution of 10 or even lower. Then you could also set the F-curve to Smooth. There is a risk that it won't fit exactly, but it will look better:

lower resolution:

 

[Filip]

I took another look at Hiroto's file - maybe a few comments:
If your character has another tag like Material, the morph tag must be to the left of it, otherwise it won't work.

Hiroto has created the morph slider in his file the other way around than you would probably normally do: open on the left, closed on the right. If you don't have it this way, it should work if you simply enter "Math.abs( v *2 )" instead of "1 - Math.abs( v *2 )" for "formula". (The higher the number after v *, the more extreme the movement)

The lip movements often become very hectic, jerky and choppy. You will get it smoother if you enter a much lower value than fps in your file for "resolution" in the soundloader before loading. For example, if you have 30 fps, then load with a resolution of 10 or even lower. Then you could also set the F-curve to Smooth. There is a risk that it won't fit exactly, but it will look better:

View attachment 39859

lower resolution:

View attachment 39858

I really appreciate your help @Lydia . But for me it's impossible to get it work. I have everything set up exactly the same way as Hiroto's file, and just like you explain, I might just be stupid, but it just doesn't work. Here's a example screen recording: https://drive.google.com/file/d/1rlI7GFX57Ai_s3VlQPfBqjnwFI_L5qqM/view?usp=sharing

 

[Helmut]

* Any possibility that the Java script assumes a quad based mesh?
* Try InstantMeshes to recreate the topology.

 

[Lydia]

That actually all looks right at first.

Have you checked what the F-curve of the sound looks like? Does it have any sound on the left? Have a look here, if there is only deflection at sound right, you must of course enter "sound right" under "other".

Have you tried a different sound file?

I think the first thing we should look at is your F-curve, because that's what it's all about. The mouth of your character reacts when you change the value manually, so the connection works. But then the value remains when you press play, so there is no connection to an F-curve or there is no F-curve.

And two more notes:
If you change the resolution in the Soundloader - e.g. from the 120 default to 10, you have to reload the sound afterwards. In principle, the Soundloader does nothing other than convert the sound curve into keyframes. These are then set and remain so until you reload.

And perhaps once again for the general explanation of my previous post, why the morph tag must be on the left - actually unimportant for you, because you have set it correctly.
target tag index: the number means which tag is used, as simply 1 = first tag from the left (morph), 2 would then be the second, so in this case material..., -1 means the object itself. The morph tag could also be to the right of the material, in which case you would have to enter 2.
(It's probably clear anyway, I just wanted to say it).

 

[Lydia]

Ha, wait....
It could also be due to the file name of the sound file. Remove the ö and the space.

 

[Lydia]

Oh, and Helmut is basically right of course, triangles are not good for animations. I had a look at the model on Sketchfab (but didn't download it). It should actually have quads, right? That would be much better and you could make the mouth movement look nicer.

 

[Filip]

Ha, wait....
It could also be due to the file name of the sound file. Remove the ö and the space.

THANK YOU LYDIA! That solved it. By removing the Swedish letter Ö and the space from the file name it works perfectly. What a euphoria, I've been trying to figure this out for 10+ hours. Thank you!

 

[Filip]

* Any possibility that the Java script assumes a quad based mesh?
* Try InstantMeshes to recreate the topology.

Oh, and Helmut is basically right of course, triangles are not good for animations. I had a look at the model on Sketchfab (but didn't download it). It should actually have quads, right? That would be much better and you could make the mouth movement look nicer.

Yes I just downloaded and used that model as a basic example since I didn't have access to my work computer with the project files at the time. But yes, doing it the way I did it in the screen recording would look horrible.

 

[Helmut]

See also Pöpcørn, as enunciated and published by the Swedish Chef in the Muppets!!!

 

[Lydia]

THANK YOU LYDIA! That solved it. By removing the Swedish letter Ö and the space from the file name it works perfectly. What a euphoria, I've been trying to figure this out for 10+ hours. Thank you!

Yippee!

Yes I just downloaded and used that model as a basic example since I didn't have access to my work computer with the project files at the time. But yes, doing it the way I did it in the screen recording would look horrible.

Oh, I see.