I think Mat is the problem, if you can even call it a problem. In the first book, it's mentioned that Rand is paler than the usual Two Rivers folk. He's somewhat tanned from being outside all the time, but someone pulls his sleeve up to show his untanned skin colour which is taken as a sign that he's different as he should be darker skinned (i.e. The natives of the area are naturally darker skinned even without a tan). On that note, the darker skinned actors are no darker than a white person could get from being very tanned, so it seems in the photos I've seen anyway. Problem is, Mat is as pale as Rand here.

As for the diversity, some people are really overstating how isolated the village is. Not only has it had thousands of years to have travellers and traders slowly but surely move into town, it's also said that the blood of Manetheren has weakened and strengthened over time as different people bred with different people. If the only people in the Two Rivers were all the direct descendants of Manetheren, there'd be no waxing and waning of the old blood at all. 

The same thing seems to be happening to me that happened when I got into ASoIaF. At the time I was playing the first Witcher game and asked people online if there were any books (other than the Witcher ones themselves) that had a similar tone. ASoIaF was recommended repeatedly. Not long after that, the GoT TV series started. Now I'm reading the WoT series, totally coincidentally, its own TV is starting fairly soon.

I actually stopped watching GoT early in S2 as I felt it was already diverging too much from the source material. Plus, I didn't want two versions of events in my head that would confuse me when the rest of the books came out (still waiting, George!) Fortunately with the WoT series, it's all already out so I can read it all without getting my memories muddied by the TV version.