Radical leader Tomislav Nikolic called on the government to resign
The Centre for Free Elections and Democracy, a monitoring body, said the radicals had taken 28.5% of the vote.
The SRS candidate for prime minister, Tomislav Nikolic, urged the government to resign - but it is unclear whether he can form a governing coalition.
The pro-reform Democratic Party (DS) and Democratic Party of Serbia (DSS) were credited with 22% and 17%.
After the first projections were announced, Mr Nikolic said: "We have won as we had expected."
"Despite running against the parties led by the prime minister and the president (Boris Tadic) and their vicious campaigns against us, we proved our strength," Mr Nikolic said.
He ruled out a coalition with the governing DS and DSS but on the subject of coalitions with other parties he said: "Let's wait and see the final results. Things are clearer after a night's sleep. We will see what will happen."
The SRS ruled the country under late leader Slobodan Milosevic during the 1990s.
Prime Minister Vojislav Kostunica, who heads the conservative DSS, had urged citizens to vote - it was, he said, the best way to keep the country on a safe and stable path.
President, Boris Tadic, whose Democratic Party is challenging the prime minister, said he hoped the outcome would produce a government capable of major reforms
The poll was overshadowed by the issued of Kosovo, with the UN expected to rule on the future of the province this year. BBC