Muslim women in Britain who wear full veils make better, positive relations between communities more difficult, Commons leader Jack Straw has said
Failing to show the mouth and nose was "a visible statement of separation and of difference", the MP wrote in the Lancashire Evening Telegraph.
Now he asks women who meet him if they would consider taking off their veils.
But the Islamic Human Rights Commission claimed the Blackburn MP's request would cause selective discrimination.
"It is astonishing that someone as experienced and senior as Jack Straw does not realise that the job of an elected representative is to represent the interests of the constituency, not to selectively discriminate on the basis of religion," said commission chairman Massoud Shadjareh.
And Halima Hussain from Muslim Public Affairs Committee asked: "Who is Jack Straw to comment on negative symbols within a religion that is not his own?
"The point is these women have chosen to wear the veil and it's their own decision. It's not something that has been forced upon them," she told BBC News 24.
"These are not oppressed women. I don't think he's right to say this at all."
Mr Straw, who is the MP for a Lancashire town where Muslims make up about a quarter of the population, said he was seeking true "face-to-face" conversations with constituents.
"The value of a meeting, as opposed to a letter or phone call, is that you can - almost literally - see what the other person means, and not just hear what they say."
Later he told BBC Radio Lancashire that this was "an issue that needs to be discussed because in our society, we are able to relate, particularly to strangers, by being able to read their faces, and if you can't read people's faces, that does provide some separation".
He said he understood why some women wanted to be covered, citing a recent meeting with a constituent who "said she felt more comfortable when she was outside wearing the veil and she was less troubled by people".
"What I'm saying on the other side is, would those people who do wear the veil think about the implications for community relations," he said.
Mr Straw was home secretary from 1997 to 2001, and then foreign secretary until 2005, a period which included the build-up to, and invasion of, Iraq.
Jack Straw calls on Muslim women to lift their veils
Muslims outraged by Jack Straw's veil veto
UK: Politician Says Muslim Veils Hinder Community Relations