Why does it matter whether “you have to be one to know one”?

What is at stake in the claim (thesis) of insider epistemology that “you have be one to know one,” either in the individual sense or in the collective/social sense? Consider this assertion: If insider epistemology is really true, it follows then that, “Only African-Americans can really know what it is like to be an African-American, and so only African-Americans can tell what it is to be an African-American. Similarly for other groups: the working class can only be studied fruitfully by those who are themselves members of the working class; only English historians can write a good history of England; only women can describe and explain the actions, feelings, and relations of women. Every group must be its own social scientist.” (Fay 1996: 12)

Is there something wrong with this view? If so, why? If not, why not?  What is your opinion about this? Pay attention to and think about which precise sense of the verb ‘know’ is in play when we are trying to answer this question?