A 90th-minute header by half-time substitute Aleksandar Mitrovic lifted Serbia into the group's top spot on 20 points, three more than second-placed Portugal, which only needed a draw to finish on top on goal difference and qualify.
Instead, Ronaldo & Co. will head into the March playoffs with the other group-stage runners-up to try to reach Qatar 2022.
Mitrovic beat home goalkeeper Rui Patricio with a downward header at the far post to spark wild celebrations on Serbia's bench after visiting captain Dusan Tadic had cancelled out an early Renato Sanches opener for Portugal.
Jubilant fans thronged central Belgrade, bringing traffic at the main Republic Square to a halt as they waved Serbian flags and hooted their car horns.
With Portugal's talismanic captain Ronaldo stifled, they produced little up front and midfielder Joao Palhinha conceded the home side were below par.
"We have to take responsibility because we had a poor game," he told Portugal's RTP television. "We just defended in the second half. Credit to Serbia, they played very well but we had the capacity to be on top.
"Everyone in the dressing room is disappointed but we have to look ahead to the playoffs now because we are more than good enough to be in the World Cup.
"You can't win all the time. Difficulties are part of life and this feeling should give us more strength."
The home side made a dream start in the second minute after Bernardo Silva robbed Nemanja Gudelj of the ball in the danger zone and fed Sanches, who drove it past goalkeeper Predrag Rajkovic from 15 metres.
Unfazed by falling behind, the visitors came close when Dusan Vlahovic, preferred to Mitrovic as the lone striker, hit the post before Patricio gifted Serbia the equaliser.
A seemingly innocuous raid ended with Tadic taking a deflected shot from 20 metres which glided over Patricio's fingertips over the line.
More poor goalkeeping presented Portugal with a gilt-edged chance to retake the lead on the stroke of halftime as Rajkovic spilled the ball, but Joao Moutinho's shot was blocked on the line by Nikola Milenkovic.
Serbia coach Dragan Stojkovic threw Mitrovic on for Gudelj and his gamble to deploy an extra forward for a defensive midfielder paid off as the Fulham striker, his country's all-time top scorer with 44 goals in 69 appearances, delivered.
It seemed Portugal would hang on after Mitrovic headed inches wide in the 85th minute but the striker had the last laugh as he pulled away from his markers at the death to head home after Tadic's cross from the right.
Mitrovic said Serbia's confidence ahead of the game was vindicated by a bold performance.
"We came here to play football and having been the better side, we earned it all," he told Serbia's Nova television.
"We are reaping the rewards of hard work and the belief that we would be able to defy the odds and beat Portugal. We are overjoyed, for our country and the people first and foremost."
Stojkovic added: "We deserved to win tonight as the lads showed so much character. This is a massive accomplishment for Serbian football.
"The team showed maturity and composure under the most intense pressure and I would have said the same thing if it had stayed 1-1 and we got confined to the playoffs."