Mr Nikolic becomes the second most powerful leader in Serbia
Tomislav Nikolic was elected with support from outgoing Prime Minister Vojislav Kostunica's conservatives.
The conservatives and a pro-Western party had earlier failed to agree on a coalition that excluded the Radicals, which won most seats in recent polls.
The Radicals reject closer ties with the EU and Nato and co-operation with the war crimes tribunal in The Hague.
Correspondents say Mr Nikolic's election could mark a return to the isolationist nationalism of former Serb leader Slobodan Milosevic, who spearheaded the Balkan wars of the 1990s.
Under the Serb constitution, the speaker is the second most powerful official after the president.
'You are history'
Mr Nikolic won 142 votes from the 244 lawmakers present during a marathon 15-hour debate to nominate a speaker.
A rival candidate from the pro-Western Democratic Party of President Boris Tadic won 99 votes.
Talks between Mr Tadic and Mr Kostunica's parties over forming a coalition government broke down over the weekend.
Both sides blamed each other for failing to reach an agreement that would keep the Radicals out of power.
During coalition talks, Mr Kostunica was keen to remain the leader of any new government.
His party came third in an election in January which saw the Radicals emerge as the single strongest party, with Mr Tadic's party in second place.
"The future belongs to us, and you are history," Radical Party lawmaker Aleksander Vucic said in parliament, apparently addressing the Democratic Party.
A pro-Western lawmaker said in turn that Serbia had taken a step back "to the dark days of Milosevic's reign".
Serbian leaders must reach agreement on a new government by 14 May to avoid another election. Fair use from BBC.