WASHINGTON—President Barack Obama defended the plan to build a mosque near the site of the 2001 terror attacks in New York, telling Muslim guests at a Ramadan dinner at the White House that the nation's commitment to freedom of religion "must be unshakable.''

Mr. Obama's remarks came after weeks of the White House sidestepping the debate that has roiled New York and the nation since developers announced plans to build a $100 million, 13-story mosque and Islamic cultural center just two blocks from the World Trade Center site.

Opponents, particularly conservatives, have called the proposed mosque an affront to those who died in the attack by Islamist terrorists.

The president, in his prepared remarks for the Friday evening event, said, "I understand the emotions that this issue engenders. Ground Zero is, indeed, hallowed ground. "But let me be clear: as a citizen, and as president, I believe that Muslims have the same right to practice their religion as anyone else in this country. That includes the right to build a place of worship and a community center on private property in lower Manhattan,'' he said.

A Marist College poll found that 53% of New York City voters oppose constructing the mosque near the former site of the World Trade Center. Just 34% favored the plan.

Debra Burlingame, whose brother Charles Burlingame was the pilot of the plane the terrorists hijacked and flew into the Pentagon, said she was furious over the president's remarks. "I'm so angry. I believe this president has abandoned the American people,'' she said. "This isn't a fight about religious freedom for Muslims. No one has argued they don't have the property rights. This is about a project led by someone who says he's trying to build bridges and bring the community together and he's chosen probably the worst place in America and the worst way to do it.''