The first part is correct, it is not used to describe a normal skin colour; the second is wrong, it is not used to refer to a divergence from a common standard but instead to refer to the temporarily abnormal colour of one single person, the reasons for such a change to a lighter hue being various, for instance, fear, sickness, unusual diet. It wouldn't be used for people with dark coloured skin as such variations are probably not often detectable.
It is moreover very often used in speaking of someone's face, (forehead, cheeks) and hands and not so often about the other parts of the skin.
(TLFi) 1. [En parlant du teint, de la peau, d'une partie du corps et en partic. du visage] Qui a perdu sa couleur ou qui, naturellement, n'a pas ou a peu de couleur. Anton. coloré, vif, animé, rouge, rosé.Face, front pâle; mains pâles.