Corner Pin unstretch (vs. AE Power Pin)

Last weekend I worked with the mocha tutorial "Master Mocha Vol. 2". They did a set extension there a way I haven't seen before:

An areal scene (fly over)  of houses where a front, top and side of a building was extented to a bigger house. They used corner Pin in AE and then something called Power Pin to unstretch the corner pin. Doing that for three planes they build a house of it in 3D and inserted it. 

Since they didn't do it with stills the scene is "alive". Now I tried this in Hitfilm with different techniques and it did not work out.

Does anybody here know a technique do first corner pin something (using quad warp) and then stretching the pinned area to full resolution?

This distored video is later put back in a similar place so the distortion is eleminated.

I thought of using the projector but what I tried so far did not work.

The result can be seen in short here at 0:31:

https://www.youtube.com/watch?v=jeibWCltl2s

 

Comments

  • I tried a couple of ideas, and, well, they didn't work so well.

    I assume you're referring to the "UnWarp" function in CC Power Pin. Theoretically this could be replicated with a Quad Warp or Bezier Warp by pulling the perspective shape to fill the screen. In practice (and I've been messing with this for about 30 minutes), it's really damn fiddly. I did a test pulling a photo of a chessboard to fill the screen and kept getting it close, but not quite right. I figure if it's taking more that 5-10 minutes to manually do the undistort, no one's going to bother to do it.

    So, yes, you could MANUALLY use another Quad or Bezier Warp to undistort the image, but you'd have to be really committed to take the time to tweak it correctly. Probably would need to do it in multiple stages--first rotating the source to try and get two edges to line up with a screen boundary, then pumping a Quad or Bezier Warp onto a Grade Layer and trying to pull the shape out.

    I was trying with this image:

    https://cdn6.bigcommerce.com/s-5p6k1/products/2192/images/8303/mission_craft_chess_board_padauk_high_view_1100x790__41213.1443123482.1280.1280.jpg?c=2

    Which probably was making life harder on me, since that's a lot more perspective to correct than, say, the building in the mocha demo.

    Projection---I think I see where you're going with that, but I'm not going to test this myself tonight.

  • Triem23Triem23 Moderator
    edited February 2018

    Ok, Normally I'd edit the above post, but I'll leave it.

    Typing the above I started thinking, and, yeah, the image I was using made my life a lot harder. Stand by--I'm going to record a quick video.

    Long story short, I'm correct--you can do this with Quad or Bezier Warp.

  • @Triem23: Thanks Mike. I tried it with a second quad warp strecthcing that image opposite to fit full screen. But that is a lot of manual work especially when there are a lot of frames. For that scene with three planes and so three corner pins there are 12 points for each frame. That can take a huge amount of time.

    The last approach I started in the evening was using the projector in a 3D solve with the solved cam as projector and a cam fixed towards a layer. But I was too tired then so I will give it a try this evening to see if that can be used.

    I know the projector ahs lots of power. I use it to convert a soccer scene to 3D for the player positions in lectures of soccer tactics. It can be used in weird ways and is powerful so I think that maybe that is a solution.

    A simple 2D replacement by moving the house front up does not work due to the parallax change since the camera is flying over the houses. That's why they did it with that power pin. I am pretty sure there must be a way with a setup of in 3D reconstruction maybe with the projector but I didn't find it so far.

  • edited February 2018

    You can certainly do it with Quad Warp, I used it to re-project a plane that I'd done some work on - to use as a clean plate - back onto a wall.

    The thing to remember is turn off the 'Perspective' option (which is broken anyway) so it's only doing a Linear/Affine stretch, otherwise you'll be fiddling with it forever. Zoom out and expect to have to move the corner points way outside the screen boundaries.

    But first: Export the image and crop it to tightly fit into a rectangle, or you'll be stretching a lot of blank space, which takes ages. Then put that in a separate comp and flatten it out to fill the rectangular comp there. You then have a 'normal' flat texture.

    Triem's example of the chessboard just tightly fitting into the rectangle is what you want to be manipulating. Then use the resulting flat texture on a plane that you 3D project back into the scene.

    Don't be tempted to use Quad Warp to re-manipulate that flat texture - using corner points, for example - for anything that's moving: it sucks at that and will jiggle and fizz all over the place and look rubbish.

  • Triem23Triem23 Moderator
    edited February 2018

     

    Ok, attached below is a no-frills video of what I did. I'll type up these notes as my video renders and uploads.

    Now, I did this in Express, because, um, reasons. You'll want to adjust this workflow a bit since you're using mocha, but you should be able to figure it out from this.

    • I imported my source image--because I think I'm funny, I took a screengrab of the mocha video you posted above.
    • I used the masking tool to grab a section of the building wall--with a mocha track consider corner pinning a white plane to the mocha plane over a black background, stick this under a grade layer and use it as the source for a Set Matte to punch your wall from your original video.
    • I created a second composite shot and dragged in my masked frame.
    • I added a Grade Layer an put a grid effect on it. I change point 1 to 0,0 and point 2 to 100,100. This grid is just an eyeball reference.
    • I adjusted the anchor point of my masked bit to more-or-less center the sliced image in frame.
    • I scaled up the sliced image to fill as much of the frame as possible.
    • I deleted the Grid effect from the Grade Layer and added Quad Warp.
    • I pulled the Quad Warp points to undistort the image--uncheck "Perspective!". In retrospect I should have left the grid on the grade layer and done the Quad Warp on a Grade UNDER the grid--so I could keep the grid as a reference to make certain my edges were straight.
    • I created a third composite shot and dragged in my second (undistorted) comp, then converted it to a 3D layer, positioned it, duplicated it, positioned it to make two walls meeting at a 90 degree angle.
    • I dragged in the original image and dragged my 3D layers into position to match the frame--with a mocha track you'd use the tracking data to align.

    Now, the key question you were asking is the "Undistort." This video is in real time--you'll see it took me under three minutes to mask the image, center it, scale it and undistort it with the Quad Warp. The final positioning of the 3D layers matches pretty well with the source image. And that's in three minutes, total, just eyeballing everything. Again, the actual mocha track will give you more accurate position data.

    Depending on what you're doing you might be doing this to multiple walls, but, again, under two minutes to undistort ain't bad. It would have been faster, but, as you'll see, I forgot to uncheck "perspective" when I started.

    It would probably be smarter to export the undistorted image as a PNG or something, but... Whatcha want in a quick example? The bottom line is it took far longer for my screen recording software to write the MP4 and for me to upload to Youtube than it did to do the actual test. :-)

    I think you're good to go.

    https://www.youtube.com/watch?v=jcTq_rWYw4s&feature=youtu.be

  • @Palacono: Exporting the image and cropping it brings it to the point: It's not about a still insertion, it's about a part of the scene duplicated and fit in with regard of parallax changes.

    With a still image it's like you said: Stretch it with another quad warp and that's it. But doing it for a movie insert is to much work doing that for every image of the video.

  • @Triem23: Thanks again so much, Mike. But keep in mind: This way it works for a single image. But what I try to reproduce is the way they did it with a "copy" of the video. It was not a still extension which I totally agree would be done the way you did it. And BTW: Man, you are fast in editing. :)

    Since the camera is moving a corner pin moves also so this way it has to be done with every frame. Even being as fast as you it would take hours for 130 frames.

    I started with the same approach you did but then I got lost then. What they inserted above the house is the same scene of the video running, not a still. And for that I can't find a solution isolating the corner pin into full screen automatically instead of frame by frame manually.

  • edited February 2018

     @Juda1 You don't do it for every frame. You capture/Screengrab one frame, do it once, then reproject that single frame back into the video on a 3D plane.

  • Add: I think it could be done even more easier than in AE if the quad warp had an option "unstretch" which stretches the warped image to full screen. That's what they did in AE using the power pin connected to the bcc corner pin using "show corner region only". But with a quad warp using unstretch it would be more easier. And so I got lost finding a solution to add points to do that work for me by using inverted exports from mocha and so on ... none of it worked so far.

  • edited February 2018

    @Palacono: But that way it is a still insertion! I am talking about a video insertion...

    EDIT: With a video insertion the corner pin is different in every frame so you do have to it for every frame since the source quad warp is different and so the unstretching quad warp also has to be adjusted.

  • edited February 2018

    We overlapped posts:  Works for clips or images, as here where the door is clearly changing shape, plus it's projected onto the ground as a 'fake well':

    https://www.youtube.com/watch?v=Rs1Qi5jocGg

  • @Palacono: I don't want to bother or stretch your nerves too far :)

    But the way you described it as well as what Triem23 did is isolating one image. That's what you wrote. So that is what I call a set extension with a still image. Projected what has to be done into your castle example would be: Imagine someone is standing inside that door and is waving. Now you want do replicate that scene above that door NOT masking the person etc. but cloning the scene with all content. Since your scene is a fly by as my scene the usual clone of hitfilm will not work as far as I know because of the parallax change. That's what they showed at the course with an insertion in mocha: It did not work because of the change in parallax. That's why they did it that way. 

    I guess in your example you inserted one image of the sceen as door and changed it scale, bit not cloning the video part of the door itself (I truly hope I tell this right to be understood). My example with a person standing there is more accurate: A still inside the video would be immediately recognized by the viewer. So it's about cloning that part of the video into another area WITH regard of the different parallax in that area. 

    @Triem23 approach is what they did. But again: The did not isolate a single image and inserted it. They isolated the video sequence and inserted that one. It's not very good to see in that promo I linked, but in the course there is some movement at the windows and balconies so a simple cloning of one image and inserting that would be soon recognized by the viewer because of the missing "action" inside the set extension...

  • Add: I am still afraid I did not tell the source of the problem right, now I found an example of what to do:

    A crowd replication of only one scene with a moving camera is sort of the same problem to solve. A simple cloning will not work because of the parallax change. 

    Maybe that leads better to what I am trying so solve here.

    BTW: Thank's guys for being patient and supporting :)

  • edited February 2018

    Look again at the door. It's a moving clip. The original doorway was tracked by mocha, then simply reprojected in a different place above it, just like the walls growing on the building example you showed. I don't know what else you want it to do.

    Yes, you could also mask out a person standing in the doorway, because the door arch was masked out to be a curve (notice it fits into the bricks as it slides up) and at the bottom to blend into the original doorway. Also in the 'fake well' on the ground, which is two versions of the archway with one of them upside down and the view 'down the well' changes as the camera moves. But this masking was only done once, on the flattened version of the image that was stretched flat by Quad Warp.

    But yes, it would be useful to have an "Unstretch to fit frame" Effect in Hitfilm instead of having to fight with Quad Warp to do the same job. :)

    You can also add multiple versions to change the architecture of something. ;)

    https://www.youtube.com/watch?v=LPSjoGX0rnM

  • Okay. Let's think this through step by step and please point out where or when I am missing something:: I did that you suggested and triem demonstrated in that video yesterday:

    I took the corner pin with "show corner region only" which looks like the image Mike took but beeing a video.

    Next I zoomed it up, applied a quad warp to move the edges back to full screen like Mike did.

    But when I step one frame forward the corner pin moves, so my warp is not matching anymore. So when I did what you described doing it once the data is only valid for the one frame where I did that. Not for the whole sequence. 

    So how do you apply your unwarp to the rest of the sequence? 

    You wrote you "reprojected" it. So did you use the projector in 3D space to reproduce it like I thought that could lead to a solution? Am I missing something?

  • @Palacono: Both videos are a good demonstration. But in the second you can see a bit of the parallax problem I am talking about:

    The upper is very thick as you can see at the top of it. At 0:05 short bevor the wall is coming back you can see that this is not reflected in the windows. The first window looks like being part of the wall - the third one looks like having a much more thinner wall.

    This could be avoided by doing that insertion in 3D with two planes: One for the front and one for the inner side. That's what they also wanted to avoid in that course separating the video in 3 video sequences: Top, left, front and then assembling that video (!) sequences  in 3D space.

  • The third 'window' looks narrower than the first at 0:05 because of perspective. It was a very wide angle lens; a GoPro that had the image flattened out and it is also looking downwards.  Note that the windows do stay attached to the bricks on the wall which also have perspective applied to them. The slight jiggles of the reprojected doorway as 'windows' are due to an imperfect 3D solve by mocha, but as it was just a test of the method, I wasn't that bothered. Yes, it uses the Projector effect and the doorway/"windows" are all 3D planes in 3D space.

  • @Palacono: Ah, so the projector is a solution. I have to try it when I am back home this evening since I do not have the footage here. It just came to my eyes that an extension of that castle would be exact the same challenge they did. Can you provide a link to the source footage you used?

  • @Triem23 @Palacono: Just to let you know I THINK (only tried it with Palaconos clip as source) I found a possible solution:

    First of all: I wanted to clone the lower front with the door. and expand that building.

    I took the cam solve as source. I inserted a plane 3D parented to the center of the front, projector on it with original camera as cam.

    Next I duplicated that plane, called it front plane, moved it to where the cloe should be and resized it. Next I duplicated the camera, called it front camera, put it below the main camera and parented it to the main camera. Now with the projector at the front plane I set the camera to front camera. Now I moved the front camera up and so on that front plane there comes up the extension I wanted.

    When I play the scene all parallax looks right so far with a very small drift at the beginning, but I only got a 58% cam solve doing it very quick and inaccurate. But it looks like that is a way. 

    Maybe Palacono did it that way with the projector?

    I will give it a try later with the original footage.

  • Probably close to what he did. 

    Have you seen this tutorial? While it's using projection for object removal the principle should work for object insertion as well, since you're matching a location. 

    https://youtu.be/zvlAS9mAyGE

  • @Triem23: I saw that tut before and it helped me to understand why that couldn't be done by simply adding a clean plate to the ground without the projector: because then the clean plate has to be made of a top view 90° of the road and there is now top view of 90° in that shot. That's parallax....

    Now first of all: Working through your post reminded me on the movie matrix where Morpheus said: It's a difference between knowing the path and walking it. :)

    I KNOW that a grade layer separates 3D and also forces everything below to be rendered and baked together but I never had in mind using a grade for the matte of the track like you did at step two. I had that matte in a comp including that in another comp and applying the matte there. Your way is really smart and tips like those are helpful in "walking" the hitfilm path instead of knowing something about it.

    Now to keep you up to date: Up to step 8 I did the same before as I already wrote but then it failed since the scene is moving and so the corner pin is. So I worked on the question: How to stick that moving scene in one place and this is the solution I found:

    I followed your steps 1+2, then I
    - took the cam solve, named it front solve, removed everything but the camera, the front center point and the plane.
    - made the plane 4096*4096, parented it to the front center and reset transform. Then I added the projector using the main camera and the corner pin comp.
    - added a cam, called it front cam, parented it to the center point and made it follow the center point so that it is aligned 90° to the front
    - zoomed the cam so that it "films" the the front with as much of the scene as needed
    - played through the scene: Now it stays in one place, changes parallax of course, and behaves like the still you took
    - continued with sour step 3 with the front comp instead of a still image

    I didn't have enough time to do that for side and top last evening. But inserted that one front into the cam solve to see if it nearly looks like what I expect. And it did. So I guess doing that procedure for side and top will work.

    So in short: I used the projector to project that moving part and "film" it with a static camera to get the static view. That's what they did in the tutorial with the power pin and all the points and null linking. I think the hitfilm way is not slower or more work once I know what to do.

    What I learned from you yesterday (and I already learned a lot from your posts and your youtube channel, Mike), is: Grade layers can also be used in a lot situations where a render is needed, e.g. more layers and effects needed as matte, and where I used to use nested comps to achieve that.

    Thanks a lot for pointing that our so clear with your example!

  • Juda, glad to help. Maybe once you get this down you could do your own tutorial on it, because I'm not 100% certain I followed everything in text--and I haven't seen the mocha tutorial you're adapting. Heck, maybe you could even do it as a guest tutorial for Hitfilm! A good technique for matchmoving 3D set extensions would be useful to a lot of people. 

    I must confess--dropping a Grade Layer to "bake in" effects for source layering was something that had escaped my notice for YEARS until @JoshDaviesCEO did it in a sky replacement tutorial. The sound of my "D'oh!" startled two cats and my wife... There's always more to learn with a powerful and complex software package like Hitfilm. :) We all learn together, and soon enough you'll be one of the people passing your wisdom on to newer users. :D

  • @Triem23 @Palacono: This is the resulting render. I am not 100% happy with it but I think it works if someone doesn't zoom in trying to find the edges. If someone doesn't know it was an extension he probably will not find out - what do you think?

    For comparison the original sequence is first playing revers, then the extended one forward. All is done using the cam solve of mocha and projector with planes where needed. No color grading is done.

    Let me know what you think of it.

    https://youtu.be/gyXA3FQOC_M

     

  • Looks really clean to me! Nice work!

  • I'll admit I didn't zoom in to try and find the edges, but it looks real to me 

  • @Juda1 ; That looks really great.  I would never have known if not for the discussion.  I will have to thoroughly go over your posts to see if I can pull it off myself and hope for a tutorial. :)  Hint, hint...

  • Looks good to me.  I'm inspired to have a go with a local parking garage.

    Nice job!

  • @ttdavis: Here's the hint :) 
    I didn't do it like described above since that did not work out. The principle is as follow (You need a goo d cam solve of course) for every plane. 

    You need a combination of a camera, a base point of the camera (cams own 3D orbit center) and a target point to which the clone focus goes in 3D space and of course a plane to take the projection.

    Taking the right front (it is made of two planes) as an example here is what I did:

    - before: The source media is at the bottom of the layer stack and visible. I added the color invert effect. Since the projector effect takes the footage without effects it will not affect it. I did that so during setup I won't be fooled between whats projected and what simply is there because of the main footage. Now I

    - copied the main cam, called if ExtFrontCam

    - created a point in 3D at 0-0-0, called it ExtFrontRig and parented the ExtFrontCam to it.

    - created a point and move it in 3D to the plane/source I wanted to clone (the center of the front track). If there is a tracking point you can set it by parenting it to the track and resetting transform. But then (very important) you must unparent it and reset the orientation - this base point must have 0-0-0 orientation! I called it ExtFrontFocus

    - then I parented the ExtFrontRig to the ExtFrontFocus.

    That is the base rig. Now move the first two down in the layer stack so you never touch them :)

    At last the clone is made. I 

    - set a plane, made it 3D and called it ExtFrontProjection. I set it to the front tracking point and then I moved it up to where the extension later will be. At this point it can be much bigger than the place it should go. Remember: It only holds the projection. It is not the content of the clone! This is weird when you start with that solution, it got me mad a few times. It went better after I decided to use the color invert.

    - I moved the ExtFrontFocus on the plane. That also can be done by parenting it to the target plane and reset the position (Remember: Do not touch the orientation!)

    - I added the projector effect to ExtFrontProjection with the ExtFrontCam as camera and the footage as source.

    - Now I adjust the the focus point by moving it in 3D. Best usage then: Unparent the focus point because later you will adujst the size of the plane and due to the parenting it will mess up everything. Do not forget that part before you go on.

    - I did that for all other planes: front left, side and top.

    At final I needed a mask because some part of the left building was affected. Masks are simple:

    - take a plane in 3D space and move it to the place where you want to mask, means: where you want to see the original. It can be bigger, size doesn't matter, only the border to the masking area. 

    - add the projector effect and take the main camera as camera.

    That way in the area the mask plane covers you always see the original footage since it is projected there. Like a mask but much faster. Copy it if you need another one, move it in 3D space, turn it, finished :)

    Most important of this solution is to build that rig: Every static camera is a copy of the main camera which reflects the cam solve. It needs its own zero point which I called xxxxxRig. That point is mapped to the point of interes to clone in 3D space. Thats what I called xxxxxFocus.

    The rig may not be modified nor may it change zoom or orientation. Be aware of that and it works find once you get used to it. Always move that complete rig by moving the focus point to the plane which should receive the projection. That's it. This way the footage as is will be cloned. Could be interesting when crowd replication is needed and the source video has cam movement and no clean 90° shot which you can replicate in 3D space.

    I am working on a complete different way now to achieve this. If that also works I will also make a description of it when someone is interested in.

  • @Juda1 ; Thanks for the detailed description.  When I get back to my computer and off my phone I'll be sure and print that out.  It will help tons.

Sign In or Register to comment.