The survey, by anti-racism campaigners Hope Not Hate, also found that nearly two-thirds of Tory activists believe the religion is "a threat to western civilisation".
Labour said the findings showed that "Islamophobia and racism are ingrained" among some Conservative members.
Some 60% think that "Islam is generally a threat to Western civilisation", with fewer than one-in-five (17%) taking the opposite view.
The YouGov study follows polling results released last month, which found that almost half of Conservative members would not accept a Muslim as the country's Prime Minister.
They were released as Tory activists vote on whether Boris Johnson or Jeremy Huny should be their next leader and Prime Minister.
Shadow equalities minister, Dawn Butler said: "This poll shows that Islamophobia and racism are ingrained among some Conservative Party members. It is worrying that the people choosing our next Prime Minister hold these views.
"Boris Johnson has a lot to answer for - both because of his own record of racism and Islamophobia, including just last year comparing Muslim women to 'bank robbers' and 'letterboxes', and because of the likelihood that these Tory Party members will vote for him as leader.
"He should be held to account for it and have the decency to apologise. [PoliticsHome] Read more
