So when compositing 3D models into 2D footage, how do I get it so that it will look accurate? For example, how do I get it to do a light wrap sort of effect, or color correct it more to my scene.
@HeySiri Not sure if the Light Wrap effect is available in Express but I use it a lot in Pro under Matte Enhancement in Keying. If not, then a way to get sort of get a light wrap would be to use a still of your footage where the model will appear as an Environment Layer and let it cast lighting on the model.
@tddavis I have Pro, but I thought you couldn't apply effects on 3D models?
You can add any effects to a 3D model as long as the model is not unrolled. Now I'm going to muddy the waters...you can also make a comp holding just the 3D model on a 2D layer and add effects to it there then you can bring that comp into your main comp and then unroll the model. All the effects you applied to the model in the 2D layer of the model comp will carry over. Confused yet???
Future reference - There are literally dozens of how-to videos for Hitfilm and 3D models. It might pay to educate yourself more with Hitfilm if you watch those tutorial. In this case YouTube is your friend.https://www.youtube.com/results?search_query=3d+models+hitfilm
Set the 3D object on the timeline to 2D, you can then add effects to it. The camera will still see the model as 3D.
@spydurhank @GrayMotion I think I get it, but does that mean you could still use the Foundry camera tracker to make the model move with the footage? I assume that would still work the same.
Yes. You've got it.
The best tutorial for integrating 3D models into footage is still this Hitfilm 2 tutorial from Simon Jones:
Here, Simon covers the basics of setting up a model, including animation groups, the advanced models tab and the Materials settings for the PHONG Illumination Model in Hitfilm (Cook-Torrance was added after this video was made). He'll discuss mocha tracking (mocha/foundry, either way you're solving a camera and setting up a floor planes), how to set up a shadow catcher, a good trick for added ambient occlusion, and a good starter range of effects for making your model look realistic. This tutorial is still the best foundation for models in Hitfilm.
The next tutorial to watch is this one:
The main takeaway in this video is how the Diffuse Reflectivity in the Cook-Torrance shader allows you to use an Environment Wrap in a fashion similar to HDRI lighting. Personally I think Simon over-lit this particular scene (His Hitfilm 2 composite is actually better, IMHO), but this is a fundamental technique to get for integrating models in footage.
Here's another good tutorial in a more recent version of Hitfilm:
Now, I mentioned that the Simon Video covered almost everything in the PHONG renderer, but there are two map types that video didn't cover.
A BUMP MAP is similar to how the parallax effect or Surface Studio. A grayscale image is used to create fake "divots" in the surface of the model. The main advantage of bump maps is you can make your own quickly, but it's a older technique and it's inferior to the next thing we'll discuss. There will be an image of two statues embedded below. The one on the right will use the stone texture file as a bump map. You'll see what it's doing
A NORMAL MAP is a bump map on steroids. this is a color image that records facing of a higher resolution model to map onto a lower resolution model. Each axis, X-Y-Z is mapped to a color channel R-G-B. Like a bump map this will create illusion of depth and detail. Normal maps are usually generated in a 3D app, but there are websites or software to create normal maps from photos.
Let's discuss COOK-TORRANCE a bit:
Identical controls in Cook-Torrance do the same thing in Phong. BTW - I recommend grabbing something like a plain sphere model with a Diffuse color of 50% gray to use as a "Test Object" if you decide to play around with material settings to learn what they do. A simple gray sphere will make it easy to see color-type changes, like if you use a strongly colored specular highlight color, or the next setting we discuss in the next paragraph:
FRESNEL is a control for how the color of a light can shift based on the angle of the light, object and camera. there are some presets here for various materials, but the three numbers are RGB values on a scale of 0 to 1. So, if as noted above, you loaded up a gray sphere, gave it a Fresnel value of 0, 0, 1, Drop the model in your project, create a white light and move the white light left, right, behind, in front of the sphere you'll see how as you move the light around you get more blue on different areas of the sphere.
ROUGHNESS is an important value, and possibly the biggest difference between Phong and Cook-Torrance. Phong treats every polygon as a perfect smooth surface. Great for polished stone or plastics, not as good for organic materials. Roughness controls the depth of a virtual texture - thing something similar to the divots on a golf ball. The more Roughness, the more this virtual texture softens and spreads the light. This gives a wider range of realistic materials. THe image below will show you what Roughness can do. The left statue will be Phong - which is also the same as "Roughness 0" while the right will be Cook-Torrance... You'll see how the roughness spreads out the highlights to turn the smooth polished stone with the rough pitted stone.
DIFFUSE REFLECTIVITY, etc is covered in the second tutorial I linked, so the last thing to explain in Cook -Torrance is...
REFRACTION: Note that this is not creating a "Volume refraction" where light bends as is passes through, say, a glass of water. This is creating "Surface refraction" where light bends around the surface. Really that's all you need to know to see what it will do.
Otherwise you can change the size of the maps used for a model's textures, shadow maps, reflections and refractions in the File>Options menu. If things look blocky, turn up the resolution. If things lag, turn down the resolution to edit and turn it back up to render.
Again, note both statues are the same model with the same lighting setup. All I did was change the material settings between rendering each statue as a PNG and slapping them on the background.
Finally, let me throw up the VFX breakdown of my Y-Wing shot and we'll discuss a few more point.
Ok, lighting a 3D model. The three basic lights in Hitfilm are point lights (light bulbs that radiate in a sphere) Spotlights (cones) and directional lights (lights of infinite size that radiate parallel beams). Directional lights are your most realistic "sun" light.... For the sun is a giant ball 93ish million miles away and any light that hits us is basically parallel. Directional lights are also good for "Bounce" lights, like the light bouncing off the ground onto an airplane... And yes, this matters. Remember, the light of the full moon is us seeing the light of the sun bouncing off moon rock and falling to earth.
The main thing to remember is you almost NEVER use a pure white light (255, 255, 255) "White light" isn't. Also, in Hitfilm you'll RARELY want a light at 100%. Pure white lights at 100% will look terrible and fake. Generally a good starting point is to make the combined value of ALL lights in a scene 100%. From there you can start adjusting individual lights as needed, but, if you light your scene at "400%" it's really easy to overexpose. Note that a total lighting value between 100 and 200% can look good, but, for me it's better to underlight at first and turn things up to adjust than to overlight and figure out what I need to dial back.
Conversely, you also NEVER want to use a light that's got a value of 0 in ANY color channel. You also don't want to use a texture, diffuse or specular color with a value of 0 in any color channel. Long story shot is, as your lights interact and you start pushing for strong highlights, if you have a color or texture with a channel with 0 in it, the color can never blow out to white as you brighten it. This can look very fake. @tddavis I think you recently shared a web page that discussed this that had some good pictures to illustrate. If that was you and you still have the link, can you share that? Thanks, Terry.
So, for my Y-Wing shot I've got three lights. My "SUN" light I had to finagle around to find the right angle. It's a directional light set to cast shadows. It's COLOR is sampled from the highlight of one of the "Grey" rocks in the video, then the color is adjusted to a luma value of 235. Use the eyedropper in the color picker to grab a value from your video. If you've shot your own footage it's a good idea to shoot a 25% gray card in direct light. That will show you the light's color. My Sun was set to about 80% intensity
The NEXT tip and one i didn't mention in the breakdown, was I changed the SHADOW COLOR of my light. Again, pure black shadows aren't actually realistic. There's almost always some color cast from bounce and ambient lights. Sampling a shadow color from a shadow color in the video will up the realism.
My second light was a "bounce light" representing the ground. It's another directional light, down near the "ground" and pointed straight up. Based on the first (video) shot, I sampled a green from the trees. No Cast Shadows, intensity of about 20%.
My third light was an ambient light and represented light scattered by all the particulates in the atmosphere. Basically I sampled this from the sky near the horizon. I set this to about 10%.
The Diffuse reflectivity of the model was set at 50% for all metal on the Y-wing, but to 70% for glass. The video layer was assigned as the Environment Wrap.
So with 80% Sun, 20% trees, 10% ambient and 50% Environment Wrap we have a "total" light value of 160%. That's fine, we want some highlights and blooming. Importantly the way the sun and bounce lights interact, there's not too much blend between them. The brightest parts of the model are getting 80% from the sun, 50% from the environment wrap and 10% from the ambient, for a total lighting value of 140% enough to get some nice specular highlights where we want them, but not enough to just blow out details.
So the "Power tip" in these paragraphs is simple. For more realistic integration of models against video plates, sample the light and shadow colors from the video plate itself. This lights and shades the model using colors from the video, so it blends better.
And, as we see in the Y-Wing breakdown the model is really matching the video before I grade. The grade is literally just a Hue Colorize effect set to tint at 15% and a little contrast added with color curves, then a Cinestyle to handle bars, and vignette.
There's another discussion we need to have on occlusion and the differences between 2D, 3D and 3D Unrolled as well as the oh so important Depth Layer and Depth Matte/Mask effects, but I'll save that for tomorrow. I just threw a lot of lighting at you. Over the years the Hitfilm team has had to deal with how to get things to properly occlude in 3D space while still being able to add effects to the models. 3D Unrolled, mentioned in a comment above, is a quick and dirty way to get everything to interact correctly in 3D space as the cost of not being able to add Effects to a 3D Unrolled model or particle layer. Later versions of Hitfilm add the Depth Layer/Mask/Matte functions which give you a lot more control over the final render but need a little thought to set up correctly. Like I said, I'll save this for tomorrow.
@FilmSensei @GrayMotion I'm tagging you because no official tutorials exist talking about Depth Layer, and my quick 1 minute breakdown on Youtube for some reason got flagged by a copyright bot and removed. BUT, tomorrow I'll be talking about the Depth Map/Matte effects which also have no video tutorials, and I've never talked about them in the forum before... You guys might have figured these out already (Greg, especially if you've been "studying under" Tony), but, if not, you're going to like what you're about to learn.
@Triem23 Copied and archived! Great info. Turns out I was screwing up and leaving my model 2D which is why I made my suggestion.
@Triem23 I have also copied and archived this. I am really looking forward to your next post. Thank you!
@Triem23 Yes, I recall the article I linked but having trouble running it down I know it's somewhere here on the board but taking a minute to find it.
@tddavis says "Turns out I was screwing up and leaving my model 2D"
How so?? As Mike mentioned above later versions added the ability to control depth/source in 2D layers ie; Depth Matte Layers and Depth Source Layers. Like he says..it take a bit of thought setting it up but once you figure it out you're in like Flynn.
@HeySiri - I hope you copy and tucked away what Mike laboriously typed out above. He sir is the real deal and it's rare that anyone would take the time to layout all the things he did above. I hope you are paying attention because if you are your 3D models in your scene will R O C K!!Nice to see Master @Triem23 back in all his glorious form. Plethora of information tucked away in that head of his. Mike - you know I'll be listening/watching.
"He sir is the real deal and it's rare that anyone would take the time to layout all the things he did above. "
AMEN to that! What a great thread this is. Learning a lot. Thanks to everyone contributing their expertise.
Me... Sitting at the feet of the master waiting for the next lesson.
@GrayMotion I was scrwing up because I have never used depth/source layer with my 2D models (that I know about )
@Triem23 I have gone back 48 pages of my comments to December 2017 and haven't spotted that elusive link I posted. I did find a reference you made to it this past March so I started there. One thing it has taught me is I post far too often for my limited knowledge I should shut up and read more....
@GrayMotion oh yes I am saving all of this! Haha
@tddavis what exactly is the depth/source layer thing?
@HeySiri No clue. GrayMotion mentioned it above my post and I have to look into it cause I feel like I'm missing a trick now.
Never fear...Mike will explain in detail...@HeySiri @tddavis"But working in 2D provides the advantage of being able to apply effects directly to models or particle systems. Using depth maps, your layers can remain in 2D, so effects can easily be applied, but they can still be rendered accurately into the depth of the 3D scene.
Depth maps are greyscale images that represent the distance from the camera to each object in the scene. Once generated, the depth map can be used as a matte, and applied to to other layers to control their visibility."source: https://fxhome.com/reference/hitfilm-pro-2017/index.html?working_with_depth_maps.htm
andhttps://www.manula.com/manuals/fxhome/hitfilm-pro/13/en/topic/depth-maskNow I'll wait for the expert on this matter to enlighten us all. This Padawan knows his place 😉
@Triem23 I searched and searched the Forum by every term I can think of and just my own posts and I'll be darned if I can find it again. Apparently, I didn't keep a hard drive copy either. Guess I thought I'd just refer back to it in the post : Silly me
Ok, let's talk about occlusion in Hitfilm, or "Putting things in front of other things in 3D Space."
This section applies to Express and Pro
Hitfilm was designed from the ground up to do 3D work - FXHOME talks about it as "3D Unified Space." It works pretty well, but, sometimes you need to jump through a couple of extra steps to really leverage all the power under the hood.
First thing we'll hit is the concept of "Render Batches" in Hitfilm.
When Hitfilm's 3D space is rendered for a frame objects are calculated in 3D space from the bottom of the layer stack to the top. However, when Hitfilm hits a 2D Photo, Video, Plane or Grade layer all layers underneath are forced to render. This is an important concept to grasp because it affects everything else we'll discuss here. It's also important because, if you don't know this about Hitfilm's render order, you can "disconnect" 3D lights from 3D models. Quick example. In this layer stack:
The 3D model isn't lit by the light because the grade layer has forced the model to render before the light is calculated. Note that it's sometimes desirable to pop a grade layer in to separate different 3D models anyway.
While the FXHOME term is "Render Batch," it's been helpful to me to think of them as "Render Rooms." where a 2D layer is a "wall between rooms." "Bedroom" light can't pass through the wall into the "Bathroom," so to speak.
To make matters even MORE confusing, the psuedo-3D effects like Parallax, Caustics and Surface Studio are not affected by the light separation of a Grade Layer. Those effects see ALL lights in a scene. This is just something to be aware of, and a good reason to learn the "Illumination" group in those effects - where you can choose which lights are "seen."
Next thing we need to talk about is the difference between 2D Plane, 3D Plane and 3D Unrolled, specifically as they apply to model and particle layers. (In earlier versions of Hitfilm, these were called 2D Compositing, 3D Compositing and 3D Unrolled. I'm not certain when they changed the terminology, but 2D/3D Plane is way better.)
2D Plane: In this case objects composite in Layer Order. Upper layer always renders over the lower layer. This, btw is also how things like Element 3D and Trapcode Particular render in AE. Both Element 3D and Particular have their own versions of the Hitfilm techniques we'll be discussing here, but Element can't send it's depth data directly to Particular and vice/versa.
3D Plane: The model or particle system is calculated in 3D space, then immediately rendered to a 3D Plane in 3D space, then passed to the effects engine. Now, the Model/Sim itself will render correctly inside this plane, but the plane itself is a 2D object. Rotating the camera around a model/sim in 3D mode will result in sections of the model/sim "vanishing" as the edges of the clipping plane pass through where the model/sim should be. Effects can be added to this layer, but they will cut off at the edges of the clipping plane. The Clipping Plane itself CAN be moved in 3D space (in fact, when you make a model/sim 3D a "Clipping Window" group will appear in the layer controls. This is where you'd move the clipping plane). It's a mode of limited use.
Where 3D Plane mode with models and particles IS cool if you've say, camera tracked a room and you align the clipping plane to match up with a door or window. You could make, say, a long particle tunnel stretching off into the distance, set the sim layer to 3D Plane mode and use the Clipping Window controls to stick the clipping window in the doorway. If the camera moved and arced around the room while facing the door the clipping plane would show the whole tunnel through the door, but cut off the tunnel outside the doorway. So, keep 3D Plane compositing in mind when you're comping something outside a window or door or creating a "Magic Door" like Al's hologram on Quantum Leap or the Time Portals on Legends of Tomorrow (one reference each for we old farts and you young punks!). Bear in mind the Clipping Window itself cannot be parented to a point, and the clipping plane itself can't auto orient. If you wanted to move a clipping plane for some reason it's all direct animation with the number sliders in the controls. No fun. But other than the single scenario of placing a 3D Plane model/sim's clipping window in the window or doorway of a video plate (or 2D plane 3D model), I've never found a good use for the function. For that one specific use case it's pretty killer.
3D Unrolled: The models and particles are all calculated in 3D space with all other 3D models/particles and layers. The relevant layers do not actually RENDER until a "Render Batch" is forced. This means that everything correctly occludes, but effects cannot be added to the layer because the "layer" is "never actually rendered." Instead the summed 3D Unrolled layers are all rendered as one (usually at a Grade Layer).
Note that an embedded COMPOSITE SHOT can be set to 3D Unrolled mode. This will preserve the 3D relationships of 3D layers within the embedded comp and effectively turn the entire embedded comp into a single 3D object. another nice thing here is that all the stuff we're going to discuss below? Those settings will correctly propagate from the embedded shot...
So, for a long time in Hitfilm the choice was if one wanted to be able to add effects to 3D models/particles, etc, or have everything occlude properly without having to do masking or generate some kind of depth matte or depth mask. Now, back in Hitfilm 2 I did develop a couple of tricks that made it somewhat possible to generate depth masks and mattes, but they were pretty clunky, pretty brute force and required embedding Composite Shots. The methods introduced in Hitfilm Pro 2017 are a lot easier. That said, those methods are Pro-Only, so, maybe, one day I'll go into the workarounds for Express owners who have purchased the models and/or particles packs...
You can put multiple models into a single layer.: If you have set up animation groups on a model before you've already seen how each animation group actually comes up under a "Models" tab. You can just... Add more models to the same layer. Wait, each individual model/animation group ends up with it's own transform controls and can be parented to it's own point! Just be careful about combining parenting a Layer with parenting an animation group/submodel to it's own point. The other limitation is, of course, all models in a single layer share the same Effects. Still, knowing this is often useful, just to clean up a timeline! Maybe you're setting up something with lots of buildings and streets etc as 3D models. Like a big city. Moving everything into one layer makes it easier to turn the city on and off and makes the layer stack a lot smaller.
The Particle Sim uses 3D Models as particles: Just a reminder. Also, you don't always have to use a lot of particles. Perhaps I'm making a ringed Planet. One emitter drops one single particle of my planet model, the other emitter(s) can make the rings of of 3D rock models. Other particle emitters can handle ships, weapons, explosions, and, eventually, you build an entire scene in a few particles layers, with a couple of 3D models on single layers in 3D Unrolled.
This shot pretty much uses all of the above tricks in Hitfilm Pro 3 to create a complex scene with a lot of things interacting.
Welcome to the Pro-Only Section...
Express users, the things we're about to discuss aren't available as add-ons for Express.
Starting with Hitfilm Pro 2017 we have some other tools to help us composite models and particles while being able to leave the individual layers in the 2D Plane mode so effects can be added.
Depth Source: This is the easiest to use. When a 3D Model/Particle layer is set to 2D Plane these controls will be in the Layer Properties. Choose a Depth Source layer for any layer (Model/Particle) BELOW. The layer will calculate a depth mask for the lower layer and apply it to itself, then pass to the Effects tab. Using Depth Source you can only select one other layer to occlude, but, combined with the other tips above and below I've never really been limited by this.
If you were to go back to the VFX breakdown for my Y-Wing shot: In the segment where showing the blown off nacelle just before it flies past camera.... The engine glow is a white sphere with some glow effects on it parented to the engine. It's on top of the Y-Wing using the Y-Wing as it's Depth Source.
Depth Mask (Effect): Yup, this is an effect. What does it do? Well, it's the same thing as Depth Source, but in a handy effect. You would use this an an embedded 3D Composite Shot if it needed to interact with something else, or it works really nicely with 3D Plane layers and the Quick 3D Effects in Hitfilm. Again, in the Y-Wing Sequence, the 3D Bonfire effects and the Missile Smoke effects both have Depth Mask applied.
What Depth Mask and Depth source are effectively doing are looking at the source layer, calculating any z-space occlusion, then drawing a pixel mask on the target layer. It's a lot easier than roto! Depth Mask also has a feather (Smoothness) slider and a Depth Shift offset to read the mask from farther forward or backwards in Z-space to fine tune things. Note that the Depth Shift SLIDER only goes to +/-100 But you can set much wider values.
Depth Matte (Effect): Ok, this should probably be applied to a plane or a grade layer, and, usually, you'll need to stack another grade layer on top of the layer with this effect.
Depth Matte draws the occlusion masks you get from a Depth Source or Depth Mask. Same controls as Depth Matte, but, what you get is a black and white image. What would you do with this? I dunno, use it like you'd use any other matte! Although if you're do the person/thing coming through wall/portal type look, once again, this can save you some roto. ;-)
I think that covers it for now. I don't think I missed any of the ways Hitfilm deals with occlusion. Other than roto. Roto always works.
@tddavis no worries. I just thought that particular article explained things very clearly. I can never find my own posts either.
@FilmSensei now how freaking insane a space battle could one create by, say, hand animating ships A, B, and, C maybe linking via the Follow Behavior ships X, Y and Z where all six models are in the same layer (so parents to points in their groups), then using the Particle Sim to duplicate the animated model rig... Set points on the model's guns correctly and your particle sim can include weapon fire. Do the same for engines and get glow and thrusters going. It's doable - I've already shared the video that handles proof of concept.
@Triem23 I could not let it go...it's that tenacious bulldog blood in me...I went through again today opening virtually every post I've made and finally FOUND it! From March 27th in this thread.
I have printed it as a PDF and put it next to your 14 pages of wonderful text from above (Well, I also put @GrayMotion 's little bit in there too. Keep all my ducks in a row.
@Triem23 noted "Welcome to the Pro-Only Section..."
@Triem23 I apologize for not commenting on this sooner. I am working through your well written information, and I was afraid to make a comment because I think I am in shell shock!
This is really amazing stuff, and I am slowly working to digest what you have written. There is no doubt that it will definitely revolutionize how I view models and their composition in HitFilm. I am rewatching the Simon's original tutorials with a new understanding of what he is talking about. Thank you so much for these notes. They are wonderful.
With regard to the "Insane Space Battle," the proof of concept is absolutely solid. Pulling it all off, on the other hand, will not be easy for me personally, let alone my computer! Good thing I have a new beast PC at my fingertips. Adding in the new Foundry Tracker to help include ship landings on planets, along with Light Saber Effects, Blasters, stock footage from Action VFX and Production Crate, music from Eric Matyas, and the incredibly solid green screen keying in HitFilm, just imagine what kind of production can be created.
@FilmSensei "Adding in the new Foundry Tracker to help include ship landings on planets, along with Light Saber Effects, Blasters, stock footage from Action VFX and Production Crate, music from Eric Matyas, and the incredibly solid green screen keying in HitFilm, just imagine what kind of production can be created."
Maybe a new Disney+ Streaming series?
@FilmSensei Adding in the new Foundry Tracker to help include ship landings on planets
I like my spaceship landings like I like my whiskey: hard and smokey Can't wait to check out your two new postings from today btw. Now that my poor old computer (who just got a boost back to 32 GB RAM) is freed up.
@Triem23 @tddavis this is turning out to be so much more information than I thought I would get! But I will use this all when working with 3D models in Hitfilm!
I would imagine Blender works a bit differently but that some of the basics may be similar, so hopefully some knowledge will transfer over... I wanted to use Blender for the entirely digital scenes because I find it a bit faster than Hitfilm on my computer.
@HeySiri As far as rendering goes, I have the same experience for the most part except for some of these physics heavy ones. as far as being able to get a grasp on applying effects Hitfilm wins there.
HeySiri - Your true intentions were hidden in your very first post...I should have caught on that you wanted to use another app (Blender) and then composite in Hitfilm. Apologies.
This is my thoughts -As far as matching camera and animation so it lines up with the Hitfilm 3D world and Camera you're going to need something like Filmer. I imagine you will have to track your footage in Blender and then export Blenders empties (from the tracking), camera, animation and the model itself over to Hitfilm and then polish your scene (your media you used in Blender) there. What you think @spydurhank ?What Mike outlined above is GOLD! ... actually mind blowing to tell you the truth. I'll be dissecting everything Master typed you can be sure! What he lined out can be used to super charge your scenes. So definitely retain that when working with your Blender work inside Hitfilm!
You have successfully subscribed to the newsletter.
You can unsubscribe from newsletters at any time.
© 2020 FXhome Limited. All rights reserved. "FXhome", "PhotoKey", "Imerge" and "HitFilm" are trademarks of FXhome Limited.