Thank you for putting in the effort to create this PR!

I expect this will require extensive testing on Windows and Linux as they have sources that behave differently from macOS.

Downloads:

• macOS
• Windows x64
• Windows x86

After some basic testing on Windows, this works mostly as expected. There are some visual differences.

Left: OBS, Right: Photoshop CS6. Windows 10.

Layers: Subtract (Window Capture), Normal (Browser Source), Normal (Color Source, white)

Layers: Lighten (Browser Source), Normal (Window Capture), Normal (Color Source, white)

I've also verified that the Screenshot (source) function does not include any strange blending artifacts, which is awesome to see.

Thanks for testing @WizardCM ! I'm glad to see it's working on windows/d3d11.

And yes, I had noticed some visual differences when comparing the results of the blending modes I've added, to those in Krita (Photoshop alternative) and Natron (Nuke alternative). I think the visual differences were down to either:

1. An additional viewer colour-space transform happening in the other apps, that isn't happening in OBS (most likely a result of my colour management settings being incorrect in either app).
2. The named blending modes not being equivalent between apps. For example, I think "Linear Dodge" in photoshop is implemented as a component-wise addition of colours, so would be equivalent to "Additive" in the blending modes I've added. For Nuke/Natron this is just named "Plus" I believe. The "Subtractive" mode should just be a component-wise subtraction of colours (The source's output colour subtracted from the backgound's current colour), so I'm not sure why it looks different from Photoshop in your testing...

Instead of matching the blending mode names/behaviour to other apps, I aimed to match the results numerically to those described here, according to which ones I was able to implement using the programmable blending pipeline for both OpenGL and d3d11. I think the best way to ensure that the modes I've implemented are numerically correct, is to devise some kind of "test image" (similar to the rainbow + rubber duck image from the above docs, and my original forum post), and to compare the results of blending on the test image in OBS to results calculated mathematically (I'll try to think of something).

Chiming in to say thank you for this work!

OK, I've devised two very non-scientific images to use for testing these blending modes across different platforms.
Foreground test image:

Background test image:

Test scene in OBS:

...and here are the visual results of each blending mode on macOS:
Additive:

Subtractive:

Screen:

Multiply:

Darken:

Lighten:

If any users on Windows/Linux can post the results of the same scene setup, that'd be much appreciated!

Comparing these to blending modes in Krita, I get the same visual result for each blending mode in areas of solid colour. In areas of gradient, there is a slight difference in the mathematical modes (add, subtract, screen, multiply). From some experimentation in Natron, I've concluded that this is because Krita (like most image editors) performs all of it's colour/blending operations in linear colour space, converting to sRGB only for it's render view and file export. For OBS, it looks like the test images are loaded in as sRGB, so we'll perform the blending operations in sRGB space. I'm no colour scientist, so this may be incorrect - it's just my conclusion from experimenting with various combinations of Natron's viewer and file input colour space options.

If it's critical that these blending modes replicate the behaviour of other software exactly, I'll consider re-writing this PR to use a shader based approach. That would allow for an 'un-sRGB' step prior to blending, and potentially more blending modes that couldn't otherwise be expressed using programmable blend states. I think this approach may be slightly less performant, as it would require an extra texture copy & draw call for each source using non-default blending, but I'm not sure how much of a concern that is in practise. I'd imagine that the current approach in this PR is sufficient for most use cases, considering that leveraging the programmable blending pipeline makes this effectively free, from a performance standpoint.

thank you for this work! I was just looking for this feature, and it's cool that you've started work on this so recently.

on Windows, the Additive, Subtractive, Darken, and Lighten modes work, but when I use Screen or Multiplicative, the image becomes invisible. here are the Windows screenshots you requested for your test:

Additive:

Subtractive:

Screen:

Multiply:

Lighten:

Darken:

As you can see, "Screen" and "Multiply" show only the background image. if I drag the foreground image around, it just appears as a fully transparent box.

poking around in the code, it seems like the issue seems to be with any blend modes that use COLOR (GS_BLEND_DSTCOLOR, GS_BLEND_INVSRCCOLOR, etc). I will poke at this a bit more, but it's a bit outside my expertise, so I don't promise that I will make progress. but perhaps this information is enough for you to identify the problem!

oh! once the issues are resolved, I also might suggest adding "Soft Additive" and "2x Multiply". I believe these would be defined as:

//Soft Additive
{
   GS_BLEND_INVDSTCOLOR,
   GS_BLEND_ONE,
   GS_BLEND_OP_ADD,
}

and

//2x Multiply
{
   GS_BLEND_DSTCOLOR,
   GS_BLEND_SRCCOLOR,
   GS_BLEND_OP_ADD,
}

(these are adapted from the "Common blend types" section on this page: https://docs.unity3d.com/Manual/SL-Blend.html)

Hi @awwbees, thanks for testing this! I'm not too sure why the Screen and Multiply blending modes are not working on Windows - all of the blending modes use the same functions for setting up the blending state, so there's very little that should go wrong between the different blending modes, but I'll try and reproduce it next time I'm able to access a Windows machine.

Would you be able to post a log file from your OBS session? I presume the problem is limited to the d3d11 implementation, but I believe OBS can run on either d3d11 or OpenGL under Windows(?), so it'd be good to check. It may also be worth testing these modes with different images, or different source types.

Here's some screenshots of the two additional blending modes you suggested, Soft Additive:

and Multiply 2x:

Soft additive seems to produce the same result as my implementation of Screen, though I believe they should be different (Screen is implemented as src + (dst/src), Soft Additive is implemented as (src/dst) + dst). Multiply 2x looks good, but results in black in areas of transparency on image sources, because presumably OBS is using premultiplied alpha:

I was able to workaround this in the regular Multiply mode by setting the destination blend factor to 1-srcAlpha, though it's worth noting that this issue is also present in the Darken blending mode (as I've noted here, but have forgotten to mention in this thread 😄). I don't believe this is a huge issue as it only effects image sources with transparency, which could be recomposited onto a white background as a workaround.

I'll leave it up to someone else to decide whether we should add these two additional modes - I've intentionally left out a few modes that could be implemented (divide, inverse-divide, reverse-subtract) as I didn't want to introduce too many new options, and some modes have limited practical use.

Would you be able to post a log file from your OBS session?

sure thing! I'm new to the world of OBS dev... is there a doc on how to do this? or maybe you just want me to post stdout?

@awwbees I'm not sure how to access it on windows, but it should be available from the toolbar/menubar:

aha! yes, the log certainly looks relevant:
2021-11-13 22-43-57.txt
(note these lines at the end: 22:44:42.591: device_draw (D3D11): Failed to create blend state (80070057))

and I switched over to opengl and thing appear to function correctly, save for some weirdness at the edges in Multiply:

I tried to build in Debug mode to see if I could learn more about the "Failed to create blend state" errors, but I hit some linker errors, looks like I'd need to futz with my config to get Debug working. but, in my case, switching over to opengl appears to be a sufficient solution to get what I want.

although... "Screen" and "Additive" appear to look identical in all scenarios I've seen, so maybe something is still not right.

aha! yes, the log certainly looks relevant: 2021-11-13 22-43-57.txt (note these lines at the end: 22:44:42.591: device_draw (D3D11): Failed to create blend state (80070057))

Ah, that's useful - If it's hitting the codepath I expect it to, then error code 0x80070057 suggests invalid arguments are being passed to CreateBlendState. Fortunately there aren't too many parameters to this function, so it'll be a case of checking whether what's being set in libobs-d3d11/d3d11-subsystem.cpp:gs_device::AddBlendState makes sense - I'll try and get OBS building on a windows machine and take a look.

Thanks again for your help!

Ok, I think I've got a lead on what the issue is (which thankfully won't require me to set up an OBS build on windows to test):

This github issue hits a suspiciously similar problem, and thankfully the solution is explained in the comments - we're setting an invalid blend factor for the srcAlpha and dstAlpha as according to section 17.2 of the D3D11 spec. The reason why this issue isn't hit on OpenGL, is because we're setting the colour and alpha blend factors simultaneously using glBlendFunc rather than glBlendFuncSeparate, which seems to set a sensible alpha factor for blend factors using SRC_COLOR or DST_COLOR.

I should be able to solve this by specifying the colour and alpha blend factors seperately in libobs/obs-scene.c:obs_blend_mode_params, which thankfully I'll be able to test on OpenGL 😄 .

awesome, yeah, that appears to work! here is D3D11 on Windows. OpenGL looked the same.

Additive:

Subtract:

Screen:

Multiply:

Lighten:

Darken:

Thanks again for testing @awwbees - I'll mark the testing task as complete now that we know that this feature functions identically between OpenGL and d3d11 backends.

Latest binaries:
linux64
macOS
win32
win64

one very minor bug I noticed: duplicating a layer doesn't duplicate its blend mode settings, it just defaults to "Normal". I'm not sure if this is indicative of other serialization issues.