can tab icons have different color when tabs are docked to caption bar?
i searched and found out about something called icon masks but there is no explanation of how they work
Yes and no. You can use an icon mask element, but that is only for inactive (lowlight) tabs. Icon masks have a transparency value in addition to a color value, for example; "0,0,0,0".