well, I recall that all especially lucky males were targeted during the Vileness - not only channelers : The Vileness | A Wheel of Time Wiki | Fandom : Immediately [Jarna] started a great hunt, an even more intensive search than usual for men who could channel - for every male of any age who showed any sign that they might be able to channel or to be a ta'veren. Even a rumour, a hint was enough. And since being lucky or having a sudden rise to prominence were among the signs to look for, it was a very bad time for men with a bit of fortune. For that reason hundreds of men were killed secretly by "accidents" engendered by Black sisters. When that was not possible, they were accused of being channelers and killed with the excuse that they were "trying to escape" or the like.

You already had the Red Ajah hunting male channelers for millennia in Randland, while Seanchan had a systematic culling, Aiel went to the Blight to die, and in Shara you had the male Ayyad as breeding stock. Another point not discussed for Randland is 20 years before with the Vileness : Red Ajah overhunted potential male channelers, manipulated by the Black Ajah. So I would say a lot of potential male channelers went into hiding then - and they had no reason to go back afterwards, until Rand's proclamation.