Mr Abbas told an Arab League summit he would not resume negotiations as long as Israel maintained the "status quo" in East Jerusalem and the West Bank.
He was seeking support after Israel appeared to refuse to back down in a row with the US over East Jerusalem.
Israeli tanks have withdrawn from Gaza after an overnight incursion.
It came after the killing of two Israeli soldiers and two Palestinian militants in the worst fighting in the territory for more than a year.
Hamas said its fighters had been involved in the initial border clash, but insisted their actions were defensive. Israel said it began when its troops spotted militants planting explosives along the border.
Reports said one Palestinian was killed during the subsequent Israeli incursion.
In Gaza City, Islamic Jihad militants rallied on Saturday to urge the summit to reject talks with Israel.
In a speech to the summit in Sirte, Libya, President Abbas demanded an immediate end to Israel's building on occupied territory, particularly East Jerusalem.
"We cannot resume indirect negotiations as long as Israel maintains its settlement policy and the status quo," he said.
"Negotiations on the borders [of a future Palestinian state] would be absurd if Israel decides on the ground the border.
"We have always said that Jerusalem is the jewel in the crown and the gate to peace."
Nearly half a million Jews live in more than 100 settlements built since Israel's 1967 occupation of the West Bank and East Jerusalem. They are held to be illegal under international law, though Israel disputes this.
Turkish Prime Minister Recep Tayyip Erdogan, a guest of Libyan leader Muammar Gaddafi, described the Israeli position as "madness" and warned Israel could find itself isolated.
The Arab League's Secretary-General, Amr Moussa, said member states should prepare for the possibility of the peace process's "complete failure".
"It's time to face Israel," he added. "We have to have alternative plans because the situation has reached a turning point."
Mr Moussa also attacked US and Israeli efforts to forge a regional front against Tehran over its controversial nuclear programme.
Israel, he said, was exploiting the question of Iran to further its own interests at the expense of the Arabs.
"I know there is a worry among Arabs regarding Iran but this situation confirms the necessity of a dialogue with Iran," he added.
US officials recently indicated that failure to revive peace talks between Israel and the Palestinians was making it harder to rally Washington's Arab allies to isolate Iran.
'Capital of two states'
UN chief Ban Ki-moon attended the summit and urged Arab leaders to continue supporting US efforts to revive the peace talks.
He said Jerusalem's significance should be respected and the city "should emerge from negotiations as the capital of two states".
The BBC's Rana Jawad, in Tripoli, says this is the first time the UN has specified what it would like to result from the talks about Jerusalem.
The chief Palestinian negotiator, Saeb Erekat, welcomed the UN statement, telling the BBC it was "the right course for a solution in accordance with international law".
Israel's approval two weeks ago of plans for 1,600 new homes in the East Jerusalem settlement of Ramat Shlomo prompted the Palestinians to pull out of the proximity talks mediated by the US special envoy, George Mitchell, which both sides had only just agreed to attend.
Unveiled at the start of a visit to the Middle East by US Vice-President Joe Biden, the decision caused one of the worst crises in US-Israeli ties for decades.
The Arab League summit is expected to end on Sunday with a resolution to include a plan to establish a commission of legal advisors to pursue cases in international courts regarding East Jerusalem.
Correspondents note this summit has seen fewer Arab heads of state attending than usual, with Egypt, Saudi Arabia and Lebanon sending other representatives.