I saw someone say that there weren't any other clues except when Sylvie said Baalzamon is a fool.   Doesn't anyone else pay attention to little things?   To use the term 'Great Lord' kind of means you're a darkfriend or Forsaken.  It could not have been Birgitte.  I think anyone other than Lanfear would have used compulsion or killed Egwene.  Plus she said Chosen I think.  Honestly I think it was a mistake on her part to say that, but no one says Lanfear was smart. Haha!

Also you're not taking into consideration the fact that just because Mat doesn't have memories from any point after the War of 100 Years, it doesn't mean that he hadn't had any other lives.  There's evidence suggesting that the memories that were installed in him weren't all his past life memories... some of them could have been great generals that were not him.  Being as those bound to the Wheel are reborn for mundane lives more often than momentous ones, I'd say it's pretty even odds that this theory could be valid.  He just doesn't have the unnecessary memories of a gambler, even if he might have inclinations cultivated by the lives of many before him that were gamblers.