McIlroy shared the lead with Viktor Hovland prior to the final lap, but Smith, who was in the lead after the second lap and was in all battle all weekend, took his first Major title on the Old Course.
It wasn’t to be for home favorite McIlroy, who came home third and 18-under after Cameron Young posted a great eagle on 18th to take second place and temporarily threaten a triple playoff.
Smith not only secured the trophy and £2.1 million prize money as well as the course record, beating the 19-under scored in 2000 by Tiger Woods – a player who probably last toured the field as a Major player this weekend.
McIlroy and Ryder Cup teammate Hovland started the day four shots away from the chasing peloton after matching the third round of 66, with Hovland tripping first when he three-putted the fourth from long distance.
A faulty drive then meant Hovland was unable to reach par-five fifth in two, but McIlroy easily found the putting surface with a towering iron shot and two-putted for his first birdie of the day to put two ahead .
Smith, who had completed the front nine in 34, birdied the 10th and 11th to close inside one of McIlroy, with the Northern Irishman reacting immediately.
Though he had ridden on the green on 10th, McIlroy was closer to the flag on eighth on the huge shared green, but cool two-putt from 125 feet.
Smith made three birdies in a row after hitting the green on the 12th par-four and when he also holed from 18 feet on the 13th, McIlroy had company again at the top of the standings.
An inspired Smith then took the outright lead by extending his birdie streak to five by two-putting from the back of the green on the par-five 14th, slightly more apparent than McIlroy in the group behind.
Smith appeared to fall victim to the infamous 17th when his approach failed and left his path to the flag blocked by the Road Hole bunker, but the world’s No. 6 calmly exhausted to 12 feet, making a hole for a vital par.
McIlroy was unable to convert his long birdie putt on the 17th, as Smith made a putt from shortly before the 18th green to complete a tap-in birdie, before playing against partner Young who completed a 15-foot hole for an eagle.
Smith calmly converted his birdie putt and watched as McIlroy was unable to refuse it and claim an overdue fifth Major.
McIlroy won the Open at Royal Liverpool in 2014, but an ankle injury sustained while playing football meant he couldn’t try to keep the Claret Jug at St Andrews the following year.
The 33-year-old admitted this week that’s why he felt the title defense he never had and enjoyed an ideal start with an opening 66, the same score he shot to kick off his wire-to-wire win over Hoylake.
It continued a welcome trend of good starts for the Northern Irishman, whose opening 73 in the April Masters meant he was a combined 35 over-par in the first round of Majors since his last win in the 2014 US PGA.
Since then, he started with a 65 in the US PGA and a 67 in the US Open, but was all too aware that he couldn’t convert those into overdue wins, a third round of 74 wreaking havoc in Southern Hills and a 73 simultaneously podium at Brookline proves equally expensive.
This time the battle was no less for McIlroy, but he simply had no response to an incredible performance from Smith, who added a Major title to his Players Championship win earlier this year.
Additional reporting by PA.