Aryeh Lippo, an Israeli rabbi, was slapped with a two-week ban from the compound last month after praying there, but a Jerusalem court on Tuesday struck down the move, saying Lippo's whispered prayer did "not violate police instructions".
Jews are allowed to visit the site but may not overtly pray or engage in rituals there.
Israeli police appealed the decision, and Jerusalem District Court judge Aryeh Romanoff on Friday upheld the ban, saying officers had acted "within reason".
"The fact that there was someone who observed [Lippo] pray is evidence that his prayer was overt," Romanoff wrote. "I restore the decision of the police commander."
Palestinians, as well as officials in Jordan, Egypt and Saudi Arabia had condemned the lower court's decision.
Sacred to Muslims as the third-holiest site in Islam and revered by Jews as the location of two ancient temples, the mosque and its surrounding plaza has long been a flashpoint in the Israeli-Palestinian conflict.
Israel captured east Jerusalem including the mosque in 1967, but Jordan is custodian of Islamic sites in the city.
No Israeli law prohibits Jewish prayer at the Al-Aqsa compound, but since 1967, Israeli authorities have enforced a ban to prevent tensions.
In a statement in favour of the police ban on Friday, Israel's Public Security Minister Omer Bar-Lev had warned that a change in the status quo would "endanger the public peace".
