A couple of points:
The yellow, grays, whites would have been terrible fighters since their whole reason for existing was anathema to violence.
The browns and blues had people who went into the world and might have needed to protect themselves, so individual sisters would have been proficient in self-defense, but again there was no call for projective attack weaves for either ajah.
Bottom line, it is not unreasonable that the sitters from these ajahs were not accomplished fighters able to handle the pressure of a surprise firefight.
I find the time to get off weave question focusing on the wrong issue. Several times in the books, it was noted that the AS were trained to unnecessarily wave their hands during weaving. While they could have done the spell much quicker, they waste time making physical gestures. So, by book logic, it would take noticeable time to cast the weaves. Someone trained not to use the hand gestures will get the spell off much faster.