Most respondents think plans to build the center near the site known as Ground Zero are not appropriate, said the survey conducted among likely New York state voters by the Quinnipiac University Polling Institute.
Overall, the poll found 80 percent of respondents believe the Muslim group has the right to build the center near the site, while 15 percent do not think so. But 57 percent think the location is inappropriate, while 32 percent say it is appropriate.
Sixty-seven percent think the Muslim group should voluntarily build the center and mosque somewhere else, while 21 percent do not think so, it found.
The plans for a proposed Islamic center in downtown Manhattan have prompted widespread public debate.
Critics say the proposal is insensitive to the families of the nearly 3,000 people killed in the attacks, while supporters defend the right to build it.
The poll showed public opinion shifting in support of the right to build the center. A poll released in late August showed 54 percent of voters thought Muslims had the right to build it, while 40 percent did not think so.
That poll, taken by Quinnipiac among likely New York state voters Aug. 23 to 29, found 71 percent thought the Muslim group should voluntarily build the center somewhere else, while 21 percent supported its planned location.
The controversy has become a campaign issue ahead of elections in November, when Republicans hope to take control of the U.S. Congress.
Opinion on the center was split along party lines, according to the latest poll. Among Republicans, 80 percent think the plans are wrong while 8 percent say the plans are appropriate. Among Democrats, 50 percent think the plans are appropriate and 34 do not, it found.
Quinnipiac conducted interviews with 751 New York state likely voters from Sept. 16 to 20. The poll had a margin of error of plus or minus 3.6 percentage points