I think Mat is the problem, if you can even call it a problem. In the first book, it's mentioned that Rand is paler than the usual Two Rivers folk. He's somewhat tanned from being outside all the time, but someone pulls his sleeve up to show his untanned skin colour which is taken as a sign that he's different as he should be darker skinned (i.e. The natives of the area are naturally darker skinned even without a tan). On that note, the darker skinned actors are no darker than a white person could get from being very tanned, so it seems in the photos I've seen anyway. Problem is, Mat is as pale as Rand here.
As for the diversity, some people are really overstating how isolated the village is. Not only has it had thousands of years to have travellers and traders slowly but surely move into town, it's also said that the blood of Manetheren has weakened and strengthened over time as different people bred with different people. If the only people in the Two Rivers were all the direct descendants of Manetheren, there'd be no waxing and waning of the old blood at all.