You could create an additional field and use your expression as a default for this field. Then display the color based on this field. This way, the calculation isn’t performed every time the map is drawn, but only when an object is newly created or changed.
Please forget my suggestion. I just noticed that the change in the relation table doesn’t trigger the calculation of the default value in the main table.