Visual glitches when applying sharpen after a key

edited February 2016 in Express Support

@Ady , @SimonKJones

I signed up for the VFX Guerrilla course, of course, and thought I'd take the green screen image in that post and mess about with it.


As the driver's hair is a little wispy at the top, it's actually quite a challenge for a good key, so I experimented with the Color Difference Key and Matte Cleaner and also added some Blur and Sharpen and found this bug which causes speckly halos around the edge of the mask.

I've reduced the problem to the fewest number of steps to duplicate it in the video below. It only needs Blur and Sharpen to appear.

It's present in Hitfilm 3 Express, (awkward for the course) and also Hitfilms 3 and 4 Pro - worse in 16bit float - although I didn't bother to put HF3P in the video

It's quite similar to a couple of related previous issues with (apparently) GoPro footage, since fixed in a later update (I think, I never checked.)

(MOD edit: Have edited the title of this post to make it more specific)


    edited February 2016


    To me, that's not a big deal. I signed up for the course, and as long as I will be able to replace the background, it won't be an issue for me personally.


  • edited February 2016

    Yeah, it's not a biggy at all, but as it applies across the range, it might as well get fixed, especially as a beginner could just as easily mess about in the same way I did and get a downer on HF because of it.

    Of course, everything has bugs - except anything I write; they're 'features' - but forewarned is forearmed.

  • I'm fairly sure that's not a bug, it's oversharpening. Sharpening works by adding microcontrast to both sides of any edges,  which will create bits of light and dark along those edges. Particularly in a case where you are created fine edge detail with a key, this is likely to create visible artifacting. Your key settings are leaving bits of the image around the edges, these are then being softened and feathred by your Matte cleaner effect, then further blurred by your blur effect, at which point there is very little info left. Then if you add a sharpen to try correct for the blur you just added, the microcontrast addd to those little points is going to appear as tiny bits of black and white along their edges. Its how sharpening works.

    If you want to fine-tune the edge detail, I would suggest you start by adjusting the key settings, rather than adding sharpening over the top of several layers of blur.  I could be wrong, but this is pretty much what I'd expect to see from that combination of effects.

  • edited February 2016

    To correctly remove the green screen where fine hair is concerned, use spill supression in the keyer.  Green screen removal has nothing to do with sharpness and contrast.  Look up garbage matte as well. Green screen removal is an art form and it took me a long time to get good at it in my tools. It's a balance of many things.


  • @AxelWilkinson watch the video again. Oversharpening black should not produce white. Some pixels that are white (and other colours) are surrounded by black themselves, so where did they get their colour from? It's just a simple bounds checking omission. Numbers become negative and instead of being capped at 0 they instead wrap around and become large positive numbers. Pretty easy fix.

    @Karma Spill removal is there, as is the Matte cleaner. Hitfilm provide these three things together for Green Screen removal as a single Effect to save you having to combine them yourself. But that's not really the issue. I might be able to replicate the bug doing something else. Blur combined with Sharpen has done something similar previously; this was just the latest example.

    And it's also very, very hard to get a good key off that image without losing most of his hair, or getting a hard edge which looks unnatural. Give it a go yourself. 

  • edited February 2016

    Now that I actually looked at your video (I was out and internet was sketchy at best), what you are seeing is jpg compression artifacts.  In a video, you would not see this unless severely compressed.  In a jpg image, the compression is obvious.


  • Kinda curious why you're blurring and sharpening the whole image instead of just the keyed edges.  This same problem would occur even if you just used an edgeblur, but the edgeblur would probably give you better results on the image overall, and you wouldn't need sharpening, unless you wanted it on your finished composite overall

    Anyway, the problem I see here is doing the blur and sharpen after the key.  I haven't taken the course, so I don't know if this is how they talk about it, but that won't work.  The problem is you're blurring and sharpening a premultiplied image.  The key creates an alpha channel which is then multiplied on the image.  To do adjustments like this, you need to unpremultiply the keyed image, do your adjustments such as blurring and sharpening, then premultiply it again.  Not the simplest concept to understand, but one that is essential for good compositing.

    Hair is notoriously difficult to key.  However, the way to get around this is to pull multiple keys.  For example, have one key for the wispy hair then another for everything else.  It's much better that way because then you don't waste all your time trying to get a perfect single key.  Instead you can finesse individual keys to get the best results for different parts of the image then blend them together.

    It's been forever since I've tried to do any of this in a layer based compositor, so someone else will have to step in to say how this is done in Hitfilm.  But those are the basic ideas for what you're trying to do.

  • Triem23Triem23 Moderator
    edited February 2016

    For highly compressed images, such as the JPEG from the blog, or low-bitrate mp4, there are a couple of other things that can help get a cleaner key.

    First, put a Chroma Blur filter before the key--this will blur out the color channels but leave luma intact, which helps to smooth the greens (key edges) without killing shot detail.

    Second, it can sometimes work better to key your shot in an embedded composite then actually bring this in as an embedded composite shot and use the baked in alpha of the pre-keyed layer to Sett-Matte key another copy of the footage--especially if you've had to use a lot of Chroma Blur.

    Also, using a copy of the footage, pre-keyed to generate an alpha matte means you don't have to worry about things like blur/sharpen order on your final key.

  • edited February 2016

    Sharpening after blurring seems an odd thing to try. Sharpening is after all creating artifacts/halos by its actions. The real thing I wonder is what the heck is sharpening mean around alpha non existent pixels. Sharpening is looking at surrounding pixels. The white pixels at alpha edge might be some artifact of a situation where there is no surrounding pixel. 

    Really I think you sharpen after the green screen is replaced so sharpening can take into account the new backdrop pixels. I'm not sure you want a razor sharp edge on a green screen mask anyway.

    I got a decent "Express" key on that image with just a few settings. Chroma UV blur softens the hair mask edges. The 2D preset is kinda aggressive.

  • One thing about sharpening after the key is the sharpening is going to hit pixels that are partially transparent from any antialiasing or blur at the matte edge. This can cause artifacts.

    As others have said, keying is an art. There are entire books--long books--covering nothing but lighting and keying greenscreen. Hitfilm's keyer is easier to use than many, and very powerful, but it still takes practice to get things right.

    The tutorials can be misleading--not that any information is incorrect, but when you're watching Simon or Axel keying a shot, both of these men have years of experience. Also, they probably did the shot BEFORE the tutorials, so, rebuilding the shot during recording they already kind of know what settings need to be used. I'll bet video of the first effort yields a minute or two of keying, not 5-10 seconds. ;-)

  • SimonKJonesSimonKJones Moderator
    edited February 2016

    As others have pointed out, trying to key a highly compressed JPEG is never going to work especially well. 

    I can't think of a situation when I'd want to combine blur and sharpen together like you're doing here, but some of the visual glitching does seem a bit weird. I'll look into that.

  • edited February 2016

    Hmmm... people are clearly distracted by why I was (or whether I even should be) doing what I was doing and not what the result is.

    It's not a .jpg artifact, it's not a low resolution image, it's a high resolution, high quality 200kb .jpg and there isn't a single visible compression artifact in the whole thing.

    For those that didn't watch it until the end, where it showed how much worse it was in 16bit float mode, here's a screengrab. 16bit error.jpg

    @SimonKJones, thanks for looking into it. The reason why I was experimenting was that the edge was pretty sharp with the default GreenScreen key settings and experimenting with the 3 effects, Color Difference, Matte Cleaner and Spill Removal, I couldn't get a good key that didn't lose a lot of his hair. So, I tried to soften the best mask I could create with some light blur, which gave me a softer edge, with nearly all of his hair still present, but then I wanted to sharpen it up to get back some of the hair detail, as Blur is global and Sharpen deals with edges/lines, i.e hair.

    The final result would have actually been combined with a second layer on top, with no blur applied, but with the Matte Cleaner set to 'eat into' more of the image. This would have 99% of the image centre nice and sharp and only the outer few pixels blurred into the Matte. That way, I get all the hair, a good matte and a sharp main image, with only the hair wisps blurry, as they should be, which sounds something like what @Mandalorian was suggesting.

    But, on the way to that, I discovered the bug: so the video only shows the steps required to reproduce it.

    Maybe there are other ways, I've not found Chroma Key to work too well in the past TBH (didn't try it here), but I'm sure the course will cover alternatives.

    Best of luck fixing it. I've also got a humdinger with Set Matte to make a video of,  but as it locks up the PC, grabs 200% of available RAM and 100% of the CPU, I'm having a bit of a problem recording a video of it in action.

  • The issue isn't related to matte cleaner, spill removal or blur. All you need is to create an alpha channel in the layer (eg using a keyer), then apply some sharpening. I've logged it for the devs to look at. Thanks for reporting it!

    This is tangential to the main issue of the sharpen glitching, but regarding the JPEG compression: a 1920x1080 image that's only 196KB is highly compressed. The same image at 100% quality as a JPEG is 1.68MB. The same image as a high quality PNG is 2.66MB. That's a lot of data disappearing, which looks fine to the eye but isn't friendly to keying or grading.

  • As a side note, that JPEG in an uncompressed format would be about 6MB (about 2 million pixels times 24 bits, divided by 8 to get bytes. Or, 2 million times 3 if we collapse terms), so, yes, a 200k JPEG is compressed at roughly 30:1. This is, indeed, massive compression. 

    JPEG (and MPEG) use a "macroblock" compression where 8x8 matrices are compressed. If you push some kind of heavy adjustment you should be able to bring out the macroblock edges. 

    This is why I recommended Chroma Blur. 

  • edited February 2016

    @SimonKJones so you discovered the root cause. Result!

    @Triem23 as you're here... How do I 'greenscreen' something with a Black background?

    Chroma Key doesn't work, I guess because Black has no Chroma, and Colour Difference Key only has Green, Blue and Red options. Threshold is way too sharp, so...what's next?

    I tried using Set Matte on a composite which had Threshold and some Blur on it (works OK for sky removal behind trees) but the results were...not good.

  • Luma Key would be a good tool for removing a black background. That uses brightness values to generate the key.

    Premult would be better, since Premult is specifically designed to remove a black background from stock footage. :-)

    As suggested above, depending on the footage, multi-pass keying might help. Embedding the original footage and using a Crush Blacks/Whites, then doing the Premult on that embed in another comp, finally using the premulted embed to drive a Set Matte/Alpha. But try the straight Luma Key or Premult first.

  • @Triem23 Of course. Thanks! Half the battle is knowing what the names are before you search for them.

    Actually, two Luminance Keys on top of each other worked pretty well.

    I couldn't quite get the first to give the soft edges I wanted without having a touch of grey around it which showed over the layer underneath, no matter how I juggled the Threshold and Tolerance. But the second one on top, tweaked it a tiny bit more and cleaned it up nicely.

    I can use the second one's Threshold to play around with the overall Opacity (probably could just use Opacity, and maybe that's all I'm really doing...) but it seems to be working. I'll look into those other suggestions if/when I need to do this in anger. Just messing about with some stock footage.

  • Good. Glad it's working!

  • It's called Demult in HitFilm, FYI.

  • @SimonKJones grrr... Demult, of course! Thanks! Sorry, my comp is locked up doing proxies.... I'm waiting for the last shot to finish so I can sleep. It's 7am for me....

  • AND Thank Goodness for proxies.  I just discovered how to use them and it has really changed my workflow on comps!  Video scrubs just fine, but any heavy comp requires a proxy render to scrub properly.  Awesome feature.


  • Aladdin4dAladdin4d Moderator
    edited February 2016

    @Palacono Sharpening is a convolution kernel matrix and can't be used on edge boundaries i.e top, bottom, left and right edges because the values necessary to calculate the matrix don't exist beyond those boundaries. To avoid problems during a sharpening pass edge pixels are just copied so sharpening only takes effect on an area  -1 pixel smaller on every edge.

    By keying you're creating an artificial edge boundary where one shouldn't exist and the convolution kernel doesn't know or care that there's a new edge where there shouldn't be one. Sharpen only works on RGB values and when it reaches the edge of your key it isn't "seeing" black, that's an RGB value but your key is actually composed of Alpha channel values (premultiplied like @Mandalorian mentioned) that exist outside of the RGB values the kernel can process properly. 

    Normally this isn't a big problem until the multiplier of the kernel (Strength value) is pushed beyond a limit. When that happens you'll end up one of two results, absolute white or absolute black. For the record I've seen the same behavior in apps other than HitFilm so it's not uncommon but it is hard to wrap your head around the math and understand why.

  • @Alladin4D Thanks, I know hard edges of the layer are a limit, but this wasn't the usual line around the surface you get when you sharpen it. As Simon's apparently sussed the actual issue: maybe update #3 will be even more amazing than promised.

    But you did remind me of another Set Matte bug I'd forgotten about, where it creates a surface that isn't flat in 3D and extends past the outer edges of a layer. Another video to produce... 

Sign in to comment

Leave a Comment