Babar finished with 5-74 in only his third Test, while debutant Shah took 4-50 to dismiss a resolute Australia for 216 in their second innings and give Pakistan a 1-0 lead in the two-Test series.
Pakistan, who set Australia a 438-run target, were left frustrated as Mitchell Johnson (61) and Steven Smith (55) fought hard before they succumbed in a tantalising finish with just 21.5 overs left in the day.
Pakistan will now hope they either win or avoid defeat in the second Test in Abu Dhabi, starting October 30, to clinch their first series win over Australia since 1994.
This was Australia's ninth defeat in Asia in the last 14 Tests, having lost eight to India since 2008. Their only win came in Sri Lanka in 2011 with four draws as they fell to south Asia's specialist spinners.
Babar dismissed number ten Peter Siddle, caught bat-pad in the 92nd over, to seal the victory as Pakistani fielders removed the stumps even before the Australian review was turned down.
The defeat looked inevitable with Australia reeling at 117-7 at lunch but Smith and Johnson defied the spinners during their 65-run stand for the eighth wicket.
Babar threatened to dismiss Smith twice but Sarfarza missed a stumping with the batsman on 37 and Misbah-ul Haq failed to pick a difficult chance at 44.
Babar also saw Ahmed Shehzad drop Johnson at deep square leg on 23 and two runs later by Yasir Shah as both the batsmen frustrated the Pakistani attack.
Smith hit Shah for his third four to reach his seventh half-century but fell five runs later, caught at short-leg.
Smith resisted for 175 balls.
It was Babar who struck twice before lunch to bring Pakistan closer to victory.
Australia had resumed on 59-4 but Chris Rogers (43) and Smith added 32 in the first hour before paceman Imran Khan made the breakthrough, bowling Rogers who played over the delivery.
Babar then came into his own, removing Mitchell Marsh (three) with a turning delivery. The edge was well taken at short cover by Azhar Ali.
Four overs later, Babar spun one across Brad Haddin's forward push to hit the stumps and dismiss the wicketkeeper for a duck as Australia lost their seventh wicket for 105.
Johnson smashed Babar for his fifth boundary to reach ninth half-century off 104 balls. In all he hit six fours and one six during his 163-minute fight.
Babar's previous best figures were 3-89 against South Africa in his debut Test in Abu Dhabi last year.
Australia's top order, including captain Michael Clarke, had flopped on the fourth evening, with Babar and leg-spinner Yasir Shah causing the damage.
Pakistan dominated the game throughout, piling up 454 and 286-2 declared while Australia scored 303 in their first innings.
This becomes Pakistan's second straight win over Australia after beating them at Leeds in the teams' last Test in 2010, but their biggest in terms of the winning margin.
Pakistan's previous best win over Australia in terms of runs was their 74-run win in Sydney in 1995.