The US move fulfilled a pledge by US President Donald Trump, who has recognised the holy city as the Israeli capital, but it has fired Palestinian anger and drawn criticism from many foreign governments as a set back to peace efforts.
"Today we open the United States Embassy in Jerusalem, Israel," US Ambassador to Israel David Friedman said at the inaugural ceremony, attended by a U.S. delegation from Washington and Israeli leaders.
Trump, in a tweet, called it "A great day for Israel".
On the Gaza border, at least 41 Palestinians were killed by Israeli gunfire in the latest in a round of protests dubbed the "Great March of Return", health officials said.
It was the highest Palestinian death toll in a single day since a series of protests began at the border with Israel on March 30, and since the 2014 Gaza war.
The health officials said 900 Palestinians were wounded, about 450 of them by live bullets.
Tens of thousands had streamed to the coastal enclave's land border, some approaching the Israeli fence - a line Israeli leaders said Palestinians would not be allowed to breach. Clouds of black smoke from tyres set alight by demonstrators rose in the air.
Demonstrators, some armed with slingshots, hurled stones at the Israeli security forces, who fired volleys of tear gas and intense rounds of gunfire.
"Today is the big day when we will cross the fence and tell Israel and the world we will not accept being occupied forever," said Gaza science teacher Ali, who declined to give his last name.