Can't use Mask on Grade Layer over Projection?

Background: I have a projection on a plane that's been tracked with mocha in a comp (#2) which I Quad Warp to fill the screen. This is then used on another comp (#1) as the basis for another plane (using same tracking data) to clone over something in that scene at a different location, but following the same camera that produced the original Projected section.

Think: using something like Clone Stamp, but from any plane in a scene to any other plane in a scene, with a moving camera.

To feather the edges, I want to use a Mask on that projected plane in comp (#2), but it appears Masks are done before Effects, so I can't mask the Quad Warped image. :(  I can mask the original projection before the Quad Warp, but it's a lot smaller and fiddlier (it's a section of distant ground in this instance) but the mask that then appears on the Quad Warped version is egg shaped, rather than round, due to perspective; and I certainly can't do any free form masking accurately.

So, to get the Mask I wanted on everything after it had been flattened, I dropped a Grade Layer on top of everything and applied the Mask to that. But nothing happened. At all.

Just to check, I put a Brightness and Contrast Effect on the Grade Layer and turned up the brightness. There was a hole in the brightness where the mask had cut it out - but the original image was still there in the hole.

Why isn't the Grade Layer flattening everything below it and why can't I mask the image itself, but can mask the Effects?

I even dropped the Clone Stamp Effect on the Grade Layer and managed to get that to work within the Quad Warped Image, just as if it was a normal flat plane (which it effectively is), but not the Mask.

I then tried just grabbing the whole lot and making it a composite (even though it already is) so I'd just be left with the single Quad Warped flat image as a one layer composite and try applying the mask to that, but it wouldn't let me! When I selected everything and Right Clicked, the Make Composite Shot option was not available, so I'm a bit stumped. :(

If Grade Layers don't flatten everything below them(!) and I can't make everything into a Composite to really(?) flatten it, what are my options?


  • edited March 2016

    @SimonKJones, @Triem23 any ideas about the masking issue and how to get around it?

    Anyway, I had a play with a different texture that was mostly front facing - rather than the very distorted ground texture I was playing with before - so I could apply a mask to it in the main comp that's using the embedded version without it being too deformed and hard to do.

    Quick first attempt at the door enlarging in this brick structure as the camera moves past, as well as using a couple of versions of it for a fake 'well' in the grass beside it.

    In a final version I'd also add a blank wall behind this overlay so that the original top of the doorway isn't revealed as it moves up and do some fixups on any obviously duplicated brick textures (inside the doorway) , but to change the geometry of something on a live video - even if not moving during the shot - is pretty useful. I could just make it fatter, or shorter and it'll track pretty convincingly.

  • OK, I'm talking to myself, but now I've found another problem with Projector and/or Layers. ;)

    I thought it might just be to do with the order of operations, but it looks like a bit more than that.

    Details and video here: Bugs Thread

  • A mask on a grade layer isn't going to remove the layers under it, it is going to mask the grade layer itself, before the effects are applied.

    To apply a mask after effects, you want to use an embedded composite shot. Right-click the comp that you want to mask, and convert it to a new comp again, then apply the mask to the new one.

  • @AxelWilkinson Thanks, but I tried to do that and it would not let me do it. I selected everything, Right Clicked, but Make Composite Shot was not an available option.

    And is there a list anywhere of what works on what layer and in what order?

  • Triem23Triem23 Moderator

    What works on each layer is inherent in the display of the layers.

    Layers are processed in the timeline in order of bottom-to-top.

    The processing within a layer works from top-to-bottom as displayed in the layer controls. So, if you twirl open an asset layer (Plane, video, image) you'll see things calculate in the following order: MotionTracking>Masks>Effects>Transformations(>Material--3D layers only).

    For 3D models the order is: World Transform(Transformations that effect every model in this layer)>Materials>Models (This is where submodel/animation group transformations happen)>Masks>Effects

    For Grade Layers the order is Transform>Masks>Effects.

    The change of processing order between Grade and Asset layers is why Grades aren't doing what you're wanting the to do, but a Grade Layer HAS TO process in that order.

  • edited March 2016

    @Triem23 OK, well that's interestingly weird. But how do I do what I want to do: simply make a Mask of everything in the composite if dropping the Grade Layer on there doesn't do it?

    I cannot make a New Composite because everything in there is already a Composite shot and it simply wont let me do it.

    See my other video in the bugs thread where I got something weird happening when I applied two Grade Layers. I want to put a Mask on that Quad Warped image. Can't find a way to do it.

    Sharpen sort of works as expected with the Single Grade Layer, although it goes way out of range when you apply two of them and then does other freaky stuff, but I can Right Click all day long on what's in there to try and roll it all up into a single embedded Composite to apply the Mask, and it just won't let me.

  • @Palacono - can you clarify what you mean when you "selected everything"? You should just be selecting one layer (the comp you added the grade layer in), then right-click it to embed it again. You can repeat this process as many times as you like. If you don't already have that entire comp on another timeline, then embed it by creating a new Comp, and dragging the original one onto it. then add your mask directly to it.

  • Triem23Triem23 Moderator

    @AxelWilkinson sorry, I need clarification on that. In a nutshell I should be able to right-click an embedded composite shot and directly embed it on step deeper?

    @Palacono there's nothing weird about the order of Layer operations. That's the way asset vs adjustment layers calculate in AE, also. 

  • @Triem - yes, that is correct. I've just tested it, to make sure I wasn't confused myself, but yeah. You can continue right-clicking and embedding a comp as many times as you want.

  • edited March 2016

    @AxelWilkinson ; I have the comp, it has about 6 layers. A plane, tracking camera, the default camera, the video to get the Projection from, another plane I use to align the Quad Warped texture with and the point to track.

    It started by me making the tracked plane into its own comp, then adding everything to that to get the projection on it, then Quad Warping it to be flat so I could slap it on the tracked plane (with some scaling) to fit right on the video with no distortion. That works fine, as long as I don't want to mask it too much.

    So that plane is already embedded in another comp, but in there it's in 3D, and tracking the same point, so I can't add the mask to that because it's just as difficult as trying to add it to the pre-Quad Warped version of the Projected plane in the comp.

    This one I did actually manage to mask, because it's mostly face on, so not too much distortion, but for one on the ground - covering up a blemish (actually most likely to be me, staring up at the aircraft taking the video, can't always stay out of shot, but can stand on some easily cloned grass) - then that's much more distorted and it's just too hard to blend that in with a mask that I can't accurately create on a tiny sliver of a plane. Hence me trying to mask it in the comp where it is still flat by adding the Grade Layer at the top...and failing.

    The wall version looks like this:

    Stretched onto a plane with Quad Warp, so it's flat when used back in the main comp. I can then slap it anywhere.

    Can't make it any more of a comp than it already is.

    Painless remodelling video...

    And right here was where I ran into that darn Proxy + Transitions Bug as in my Bug Thread , as that render with the multiple doorways is pretty slow, but if I proxy it to line up the transitions correctly, it shows as black at the overlaps. :(  Can we please get a bug fix for that for HF3P and HF3E? It's a lot more important than whether Twixtor works or not.

  • edited March 2016

    @AxelWilkinson OK, sorry, I reread what you were saying. I went to the embedded comp in the main comp and did Make Composite Shot to that single layer. That just broke everything.

    Yes, it was flat and created in a  new comp, but it threw away all the scaling and size and the fact it was 3D and tracking a point. I tried with it both leaving everything here and taking it with it. Neither worked. One was worse than the other, neither did what I wanted. I had to recreate everything again by noting down the position, size, scale etc. from the transform, re-parenting it and making it 3D again.

    If I wanted to do that I could reset everything for the plane, add the mask, then make it 3D, tracked and scaled in the first place without creating a comp-comp.

    But... by playing with the various ways of doing it, duplicating the layer before I comp the comp, and then copying all the values back that get thrown away, then deleting the duplicate, I can get to where I need to, It just seems like a lot of effort to replace not being able to mask a comp using a Grade Layer, when using multiple Grade Layers clearly does have effect - even a weird one - as shown in bug #10 with Sharpen.

  • Aladdin4dAladdin4d Moderator
    edited March 2016

    @Palacono I'm not going to talk about everything because I'm still taking it all in but I'm going to try to describe something about grade layers that will either help or confuse you even more. Consider yourself warned ;)

    Triem23 already laid out the processing order but there's more to it than that there's also a different type of processing involved with grade layers. When you do anything to a regular asset layer the processing is destructive. Going with the each layer is a piece of paper analogy it's the equivalent of physically changing the paper in some way.

    Grade layers are a form of non-destructive processing. Adobe calls them adjustment layers but Corel Draw Suite (Specifically Photo-Paint) has the best name for this type of layer for wrapping your head around what's going on with one and that's lens layer because looking at other layers though this type of layer is just like looking at them through a lens. 

    Back to the paper analogy again let's say the lens is a clear piece of glass. You look at the stack of papers through it and everything looks the same. Now stain the glass with some red pigment and look at the stack of papers through it. Everything will suddenly look like it's been dyed red but it hasn't. The papers are completely unchanged it's just an optical illusion created with a piece of glass and some pigment. Hopefully you can see how such a thing would be useful to grade a shot without damaging it. (see what I did there?)

    At this point I'm sure you're thinking something along the lines of this is all fine and dandy but what the heck has that got to do with a mask on a grade layer? Good question and I told you all of that stuff just so I could tell you this next bit with some hope of it making sense to you......

    Masking is an inherently destructive operation like physically cutting a hole in a piece of paper. Grade layers are always completely non-destructive so when you cut mask on one you're not actually cutting anything. Instead what you're really doing is laying out all the parameters necessary to create the illusion of having cut a mask in the layer. This brings us back to processing order and baking things in with embedded comps. This specific illusion can't be created until EVERYTHING is finalized and that usually means embedding.

  • edited March 2016

    @Aladdin4d Thanks, I appreciate you taking the time to write that and yes, it does make a bit more sense. :)

    The Grade Layer+Grade Layer bug (#10) doesn't though. ;)

    So, I guess then, if I have to embed the composite layer and comp-comp one that's already a comp, then I'm most disappointed with the two options available when you do that....

    Neither of them does the job properly, they both ruin the set up and I have to correct the damage. It can't be beyond the wit of man to have a composite shot created that has all the parameters that were just there in that single layer that you re-comped reapplied to it - because I'm able to copy it afterwards from a duplicate - so that nothing needs to be done afterwards. Or, for that matter: several layers.

Sign in to comment