Although Group Masks are covered relatively well in the modo
documentation, often users are still confused as to what they are, and
to what the differences are with hitting "m" on polygons, vs creating a
mask by itself. I feel this is because of how they are explained.
Generally they are just explained by showing the power of what they can
do, and how to use them, but often users have trouble differentiating
them from Layer Groups, or in the end, just understanding what they
are. So, in this article i will try and explain Group Masks as how i
see them, and focus on some general basics of them.
As said, the modo documentation covers it quite well in my opinion:
Group layers can be thought of as "containers" which hold materials,
textures and or shaders. When used as containers they are merely
serving an organizational function and do not impact the result of your
project. However, Groups can also act as Masks to limit the effect of
their contents to specific areas of your project. Groups can mask by
Mesh Item or Polygon Tag. Polygon Tags can come from Material Tags,
Part Tags or Selection Sets. Additionally, Groups can be set to use
apply textures to entire sets of sub-groups. This provides an extensive
method for applying textures and material attributes to the content of
your project.
Their explanation of Containers holds very true. Group Masks (Known as GMasks
from here on) can either be an open or closed container, where
contents from the inside are alone, or where contents from the inside
and outside effect each other. Figure 1.1 Illustrates this.
Figure 1.1: The below image shows two objects, one has no GMask assigned to it and one doesn't. However they both have materials, one having the red material that is inside the GMask, and one blue material outside the GMask. What this is doing, is making the Red Material, only effect the contents of the GMask, ie, the Toroid.

To fully understand GMasks you have to partially understand the shader
tree. You must understand that things from the bottom, go up and effect
the top. If you have 3 global materials, bottom red, middle blue, and
the top white, the top will be the one that has 100% effect. Now you
can do more with those,ie, mixing types, alpha, or using various types of masks. All a GMask
is, is a Mask. That is obvious, it has that name after all. More
specifically, it is a mask that can effect multiple things, and be used
in many different ways. Figure 1.1 shows the mask effecting one object,
the Toroid . Despite both those objects being in one layer. The reason
it is doing that is because it was told to only effect a specific
"Polygon Tag". A Polygon Tag is just that, a Polygon, that has been
tagged with a name. Think of a Polygon with a name, his name is Bob (hi
Bob!). Now to make a GMask only effect Bob, you need to tell it to use Bob. You do this by simply assigning the "Polygon Tag" dropdown to the Polygon Tag, Bob. Figure 2.1 Shows this.
Figure 2.1: A GMask that is only effecting polygon(s) with the Polygon Tag "Bob".

You may be wondering how Bob got his name in the first place, well thats
where the "m" (material assign) key comes into play. When you select
polygons, whether its 1 polygon or a full room of objects, you are
simply assigning it a polygon tag. GMasks also have two other methods by
which they can modify an objects material properties. As you can see in
Figure 2.1, there is also "Item", and a "Polygon Tag Type".
Here it gets a bit more involved. A Polygon Tag Type is well.. a type of polygon tag. modo
is built to be very open with possibilities. What that means is in the
end, there are not too many differences between different types of
things. Many things, such as groups, are generally the same just with
slightly different properties and different uses. What all this means
is that, all though they serve different uses, and are used
differently, modo has no problem overlapping uses for a "Selection Set",
or a "Material Selection". This is where the polygon tag type comes in.
You can use both Material Tags, or Selection Set Tags to tell a GMask what polygons it is modifying. The GMask
does not care one bit, because a Material Selection, and a Selection
Set, are still both selections of polygons. Bob is a Material
Selection, and Fred is a Selection Set, but GMasks can happily use a material to color Bob Green, or Fred green, it does not care one way or the other.
The 3rd type, is the ever loving Item. Items, are again, modo not caring what things are. An item to modo is a Camera, a Mesh Layer, or a Light. They are all Items to modo..
with slightly different properties. However in this instance, we can
only use one type, the Mesh Item. What this means is you can for
example, make a new GMask , add a Purple Material to it, and assign it
to any specific Mesh Item (Layer) you want, and it will only effect the
objects in that Mesh Item (Layer).
Now we can start mixing and matching. You can of course, combine the effect parameters of the GMask so that it has a polygon tag to effect, and it has an item to effect. In figure 3.1 you can see the Bob GMask, which was told to effect all polygons with the tag (name) "Bob" and turn them the color of the GMask's
material (green in this case). Now i gave some of the polygons on the
Cone the tag Bob, and i also gave the whole capsule the tag Bob.
Figure 3.1: You can see the two sets of polygons with the Tag Bob, which are being turned green because of it.

Now
here is where it may get a bit confusing. In figure 3.2 You can see
that the layer "A Bunch Random Objects" has two objects in it, the
capsule and the sphere. Now the GMask "(Bob)" is effecting all the tags with the name Bob, but it has no Item assigned to it. If i were to give the GMask "(Bob)" an item type as well, it would only effect objects inside the specified Item, with the
specified Polygon tag, which is "Bob". Figure 3.3 & 3.4 shows the
Item parameter using two different groups. Although the polygons with
the tag "Bob" were never changed, the GMask was told to only effect polygon tags with the name "Bob" who happen to also be inside the specified Mesh Item (Layer).
Figure 3.2: You can see that the layer (or Mesh Item if you prefer) "A Bunch Random Objects" has two objects inside of it.

Figure 3.3: Shows the GMask
"(Bob)" effecting the polygons with the tag "Bob" which reside only
inside the layer "Cone". This also leaves the capsule and the sphere
purple because they have a GMask assigned to their Layer "A Bunch Random Objects".

Figure 3.4: Shows the GMask
"(Bob)" effecting the polygons with the tag "Bob" which reside only
inside the layer "A Bunch Random Objects". You can see that the Cone
loses its section of green, and that the capsule turns green, because
of what item the GMask "(Bob)" is effecting. Also note that the Sphere is still purple, this is because that although the GMask
"(Bob)" is effecting the item the sphere happens to be in, it is only
effecting the polygons with the tag "Bob", which the sphere has none of.

Welp this is basically it, I could go deeper and explain some points of the shader
tree, and why if i switched the group masks orders that a lot of this
would work the other way, but that is for another tutorial. I wanted to
explain as thoroughly as i could, how group masks fundamentally work, and how they look at the objects they are effecting. Hopefully i did that, and didn't just confuse you.
If you have questions feel free to post on the forums here, and i will do my best to update this article on any weak points.
Thanks for reading!
Lee O