The independence referendum is asking Montenegrins whether they want to end their union with Serbia set up in 2002.
Enthusiastic voters quickly passed the 50% benchmark required to validate the vote, which has deeply divided the republic.
The pro-independence bloc needs 55% of the votes cast to ensure victory.
Serbian officials and church leaders, as well as anti-independence Montenegrins, have urged voters to reject independence, invoking the strong cultural, economic and family ties between the two republics.
Queues formed early at polling stations, which opened at 0800 local time (0600 GMT).
There was a steady stream of voters. One man said: "This is the most important day for Montenegro in 100 years."
The polls are due to close at 2100 (1900 GMT) and first official results are expected on Monday, though election monitors are hoping to have an accurate prediction by the end of Sunday.
Voters, some of them dressed in their Sunday best clothes, pondered the referendum question: "Do you want Montenegro to be an independent state with full international and legal legitimacy?"
The build-up to the vote had reached a climax on Saturday night, with independence supporters setting off fireworks, blaring car horns and draping the national flag from their balconies.
The campaign for independence has been led by Prime Minister Milo Djukanovic, who argues it will bring faster European integration and a stronger economy.
In the days running up to the vote, thousands of Montenegrins living abroad have travelled home in order to cast their ballot.
With a population of fewer than 700,000, it is these votes from the diaspora that could ultimately decide the fate of the republic, says the BBC's Nick Hawton in the capital Podgorica.
The country seems to be divided largely along ethnic lines, with ethnic Montenegrins and Albanians in the coastal west favouring independence, while the more Serbian areas near the eastern border prefer the status quo.
The last time Montenegro was independent was nearly 90 years ago at the end of World War I, when it was absorbed into the newly formed Yugoslavia.
Under a European Union-brokered deal, the independence bloc needs 55% of the vote to be successful.
One of the key questions is what happens if a majority do vote for independence but the 55% threshold is not reached, our correspondent says.
Despite the peaceful run-up to the vote, some observers have expressed fears that the result of the referendum - whatever it is - could trigger a spasm of violence.
There is a precedent in the Balkans, with the Bosnian war beginning on the day the country voted for independence in 1992.
But these fears were played down by Prime Minister Djukanovic, who said: "The security forces are ready, but I'm sure there won't be trouble. We have learned our history lesson." BBC