Following on from his 11-wicket heroics at Centurion when SA took a 1-0 lead in the three-match series, the in-form paceman took 4/48 as the hosts bowled their guests out for 177 in the first innings.
On a green pitch on Thursday morning, Proteas skipper Faf du Plessis sent in the Asians and soon the new-ball pair of Dale Steyn and Vernon Philander had reduced the visitors to 13/2 with a wicket each.
Once the first four wickets were down for 51 runs, Kagiso Rabada and Olivier had struck too.
Olivier had his second in the same spell adding Babar Azam to the scalp of Azhar Ali to leave Pakistan in trouble at 54/5.
Pakistan No 3 Shan Masood stood strong with his 44 until he was lured into edging a Rabada delivery to wicketkeeper Quinton de Kock.
The visitors’ skipper Sarfraz Ahmed then got stuck with a defiant 55, but it was that man Olivier again who got his wicket when his uppercut was taken by De Kock as Pakistan crumbled to 156/7.
Olivier sent Yasir Shah packing for this fourth before Steyn (3/48) cleaned the tail up.
SA finished well in control of the Test on 123/2, trailing by 54 runs after a dominant Aiden Markram 78 before he was bowled by Shan Masood with the final ball off the day.