Creating ID/mask passes using python based on selection sets

aov_mask_id_arnold_maya

Creating masks/id passes it’s really simple, but when working on a scene last week I needed to make id passes based on selections and didn’t want to mess with render layers. So every set of objects would have a different ID pass. Also, just wanted to have full red, green and blue colors, because that makes more easy to mask by channel in photoshop or nuke, instead of having arbitrary colors that you might get with default id passes/shaders. So in this post I am going to show you how I used python to automate the process.

The main idea

maya selection sets

So I started by creating selections sets. Every group of objects I wanted to have a mask like the front windows, I created a set/quick selection set. I had already a lot of sets from the uv stage, when saved out the edge cuts, so to later filter the sets used for the aov’s/passes I gave it a prefix of “aiMask_”.

The main idea is to use the “tripleShadingSwitch” utility node with 3 colors (red,green,blue) nodes and assign it to a custom aov. So for every 3 selection sets, create a tripleShadingSwitch, assign a color to each one, create a flat shader, a custom aov and make all the necessary connections. Doing this mannually would be time consuming, so with python we can automate the process. I should mention that I will be using arnold renderer, but you can recreate this in any render inside maya.

Python script

import maya.cmds as cmds

def aiMaskSet():

    allSets = cmds.ls(set=1)
    aiMasksSets = []

    for set in allSets:
        if set[0:6] == 'aiMask':
            aiMasksSets.append(set)

    if aiMasksSets == []:
        return cmds.warning('No aiMask sets!')

    if cmds.objExists('defaultArnoldDriver') is False:
        import mtoa.core as core
        core.createOptions()

    aiColors = [
    cmds.shadingNode('aiUserDataColor', asShader=1),
    cmds.shadingNode('aiUserDataColor', asShader=1),
    cmds.shadingNode('aiUserDataColor', asShader=1)
    ]

    cmds.setAttr(aiColors[0]+'.defaultValue',1,0,0,typ='double3')
    cmds.setAttr(aiColors[1]+'.defaultValue',0,1,0,typ='double3')
    cmds.setAttr(aiColors[2]+'.defaultValue',0,0,1,typ='double3')

After importing maya cmds and defining your function, create a variable called “allSets” to store all the selection sets in the scene, using the ls command with the flag “set”.

Then create an empty list where we will append only the sets with the prefix ‘aiMask’. That’s what the first for loop is doing, for every set in allSets if the first 6 letters are “aiMask”, append it to the list.

After that make 2 simple tests. If the aiMasksSets return an empty list we should stop the function and return with a warning, just like a selection based script, we should always make a test if we have the data needed to continue. The second test is an arnold “thing”, we need to make sure that we have the aov’s default driver in order to later connect our custom aov’s.

The aiColors list will create and store the red, green and blue colors to use in the masks, you can use any maya color node like a flat ramp, I used an aiUserDataColor, where you can set a basic rgb color. Finally just set the colors for each node using the “defaultValue” attribute.

    for aiSet in xrange(0, len(aiMasksSets), 3):

        tSwitch = cmds.shadingNode('tripleShadingSwitch',au=1)
        cmds.setAttr(tSwitch+'.default',0,0,0,typ='double3')

        aiUshader = cmds.shadingNode('aiUtility', asShader=1)
        cmds.setAttr(aiUshader+'.shadeMode',2)
        cmds.connectAttr(tSwitch+'.output', aiUshader+'.color',f=1)

The next step is to create a new for loop that for every 3 selection sets that founds (in this scene I have 7), create a “tripleShadingSwitch”, set it’s default color to black, create an aiShader, set it to flat mode like a surfaceShader and connect the tripleShadingSwitch to it.

So every aov will have only 3 colors, because that’s what I wanted, only full red, green and blue. That’s why we create a shader and a switch for each group of 3 sets. And then every set in those groups will have a different color.

        for n, i in enumerate( aiMasksSets[aiSet:aiSet+3] ):
            aiColor = aiColors[n % len(aiColors)]

            for obj in cmds.listRelatives(cmds.sets(i,q=1),pa=1):
                inpt = cmds.getAttr(tSwitch+'.input',s=1)

                cmds.connectAttr(obj+'.instObjGroups[0]', tSwitch+'.input['+str(inpt)+'].inShape',f=1)
                cmds.connectAttr(aiColor+'.outColor', tSwitch+'.input['+str(inpt)+'].inTriple',f=1)

Now, inside of the last for loop create a new one that will store the color that every set will have. Then a new nested for loop that will go through every object of the ‘current’ selection set, store the input size of the current tripleSwtich, and connect the object instanceGroups[0] to the tripleShadinSwitch input shape. The last line will connect the stored color to the tripleShadingSwitch.

To know how to make the right connects do it manually in the node editor and see the result in the script editor, that’s how I found the instanceGroups[0] maya thing.

Creating the aov’s

We’re almost done with the script, just missing the aov’s setup. The next block of code is arnold renderer specific. Instead of this you can create your aov’s mannually and give the name you want, I just wanted to automate the all thing.

        # AOV'S
        aovListSize = cmds.getAttr('defaultArnoldRenderOptions.aovList',s=1)

        customAOV = cmds.createNode('aiAOV',n='aiAOV_rgbMask_'+str(aiSet//3+1), skipSelect=True)
        cmds.setAttr(customAOV+'.name','rgbMask_'+str(aiSet//3+1),type='string')
        cmds.connectAttr(customAOV+'.message','defaultArnoldRenderOptions.aovList['+ str(aovListSize)+']',f=1)

        cmds.connectAttr('defaultArnoldDriver.message',customAOV+'.outputs[0].driver', f=1)
        cmds.connectAttr('defaultArnoldFilter.message',customAOV+'.outputs[0].filter', f=1)

        # connect to default shader
        cmds.connectAttr(aiUshader+'.outColor',customAOV+'.defaultValue',f=1)

Store the ‘aovListSize’ so you can know the right connection to to later to the aov’s render settings list. Then just create a custom aov with your naming convention, connect it to the ‘defaultArnoldRenderOptions’, the default driver and filter. This is just an arnold thing where all the aov’s need to have a driver and a filter.

Finally we connect the shader containing the color information to the custom aov that we create in each iteration, connecting to the ‘defaultValue’ attribute.

Final script

def aiMaskSet():

    allSets = cmds.ls(set=1)
    aiMasksSets = []

    for set in allSets:
        if set[0:6] == 'aiMask':
            aiMasksSets.append(set)

    if aiMasksSets == []:
        return cmds.warning('No aiMask sets!')

    if cmds.objExists('defaultArnoldDriver') is False:
        import mtoa.core as core
        core.createOptions()

    aiColors = [
    cmds.shadingNode('aiUserDataColor', asShader=1),
    cmds.shadingNode('aiUserDataColor', asShader=1),
    cmds.shadingNode('aiUserDataColor', asShader=1)
    ]

    cmds.setAttr(aiColors[0]+'.defaultValue',1,0,0,typ='double3')
    cmds.setAttr(aiColors[1]+'.defaultValue',0,1,0,typ='double3')
    cmds.setAttr(aiColors[2]+'.defaultValue',0,0,1,typ='double3')

    for aiSet in xrange(0, len(aiMasksSets), 3):

        tSwitch = cmds.shadingNode('tripleShadingSwitch',au=1)
        cmds.setAttr(tSwitch+'.default',0,0,0,typ='double3')

        aiUshader = cmds.shadingNode('aiUtility', asShader=1)
        cmds.setAttr(aiUshader+'.shadeMode',2)
        cmds.connectAttr(tSwitch+'.output', aiUshader+'.color',f=1)

        for n, i in enumerate( aiMasksSets[aiSet:aiSet+3] ):
            aiColor = aiColors[n % len(aiColors)]

            for obj in cmds.listRelatives(cmds.sets(i,q=1),pa=1):
                inpt = cmds.getAttr(tSwitch+'.input',s=1)

                cmds.connectAttr(obj+'.instObjGroups[0]', tSwitch+'.input['+str(inpt)+'].inShape',f=1)
                cmds.connectAttr(aiColor+'.outColor', tSwitch+'.input['+str(inpt)+'].inTriple',f=1)

        # AOV'S

        aovListSize = cmds.getAttr('defaultArnoldRenderOptions.aovList',s=1)

        customAOV = cmds.createNode('aiAOV',n='aiAOV_rgbMask_'+str(aiSet//3+1), skipSelect=True)
        cmds.setAttr(customAOV+'.name','rgbMask_'+str(aiSet//3+1), type='string')
        cmds.connectAttr(customAOV+'.message', 'defaultArnoldRenderOptions.aovList[' + str(aovListSize)+']',f=1)

        cmds.connectAttr('defaultArnoldDriver.message', customAOV+'.outputs[0].driver', f=1)
        cmds.connectAttr('defaultArnoldFilter.message', customAOV+'.outputs[0].filter', f=1)

        # connect to default shader
        cmds.connectAttr(aiUshader+'.outColor',customAOV+'.defaultValue',f=1)

Final result

maya aovs node editor

I had 7 selection sets in the scene, so the script created 3 swtiches/aov’s, the last aov will have only one set and one color.

custom aov arnold maya

Ok that’s it, hopefully you learned something with this like I did, you can try to make your own version of it, instead of selection sets, make for individual objects, based of shader, etc. Let me know if this post was usefull.

  • perrington

    Thanks! I have limited programing knowledge and this helps a lot, exactly what I was looking for!

  • Macbeth R.

    Hi, just an opinion, it is actually exactly the same thing, it just look cleaner and saves you time, but I think is better to use the arnold already implemented python functions. For example for create AOVs and list AOVs:

    import mtoa.aovs as aovs
    # Create AOV
    aovs.AOVInterface().addAOV(‘cputime’, aovType=’float’)
    # List all AOVs with their names
    print(aovs.AOVInterface().getAOVNodes(names=True))

    Maybe is not the best example, but using arnold python utilities can save you time and scripting, cause you don’t have to make all the connections by yourself, or figure out, witch other things arnold does in the background when it does a specific function. For example when you create a StandIn or stuff like that…
    There is no api reference for python arnold so you will have to take a look at the code, or check de c++ api reference.

    Good Work!!

    • therenderblog

      Thanks for that Macbeth. Since there is no API reference we have to search and experiment, but it’s good that we can always find multiple ways to do something. Also the goal here was also to show how to use some python commands and how easy is to create our own scripts, even if it’s not the most efficient way. :)

  • Andrea

    Hi, i’m trying to figure it out how it works your script, i just copy it and paste in my maya but it’s not doing anything. I’m a little noob about scripting: i created some quick selection sets with aiMask_ prefix then run your script but nothing happens.

    any advice? thanks

    • therenderblog

      I think the problem is “whitespace”. Get familiar with the concept and when you copy and paste from here, clean the whitespace (tabs and blank spaces). It could be that, but what error do you have? Check the script editor and tell me which error you’r getting.

      Download the script from here and see if you still getting any error:

      http://therenderblog.com/wp-content/uploads/2014/05/aov_mask_sets.py

      • Andrea

        Hi, i tried with the file, but as well i can’t lunch it.
        what i have to wriite in the script editor to make it works?
        i tried
        import aov_mask_sets
        but nothing happens, even if i copy all the script in the script editor and execute it nothing happens :(

        Thanks

        • therenderblog

          You can copy the code or import. But you don’t get any error? No extra passes in the AOV manager?

          • Andrea

            nop! i don’t get any type of error, all blank e no aov created. it is maybe the version of maya? or mtoa?
            what precicslu i have to write to execute the script?

          • therenderblog

            Sorry, my bad, you have to execute the command after declaring the function. It was so obvious that I forgot to mention.

            Copy the content of the script to the script editor and add this line to execute the command.

            aiMaskSet()

          • Andrea

            Well, it works like a charm, many thanks to you i really appriciate your help.
            thanks again ;)

      • Andrea

        Hi, i tried with your file but still nothing happens, if i execute it in the script editor or if i put it into the scripts folder.
        i have to use a command to run it or what?

        thanks

  • -SeB-

    Hi,

    It’s a very nice script you have done.
    I have tested it and there is a bug for me.

    For a reason I don’t understand your script break my lighting.
    Before the script my directionnal light works and when I run the scripts the light doesn’t work anymore.

    I have found that the connection between the object shape and the switch is the problem.
    So, instead of using the “.instObjGroups[0]” of the object, I use the “.inMesh” and it works fine.

    Do you have an idea why it change the lighting ?
    I use it of course with arnold.

  • Manuel Reyes

    This was extremely useful! Thanks a lot :)

  • Santosh Das

    Hi this is working fine with the solid objects but when there is a transparent object/shader transperency is not working the cuttouts are coming with solid cuttout/color