Thank you Wolfgang, your hack is perfect. Hans, what do you think about adding a style and color key in \setupreferencestructureprefix to avoid having to use this defineprocessor hack?

And by the way, the name \setupreferencestructureprefix does not sound really natural to me because we are talking about a reference NUMBER and there is not always a prefix. What about changing it to something like \setupreference or \setupreferencenumber (like \setuppagenumber)?