The BN victory has handed the ruling Pakatan Harapan (PH) alliance three embarrassing by-election losses in a row.
Mr Mohamad, who is acting president of Umno, has received 10,397 of the votes.
His main rival, anaesthetist Dr S. Streram from PH won 5,887.
The other two candidates, who are independents, received fewer than 100 votes each.
Malays form 53 per cent of Rantau's nearly 21,000 voters. With Umno and former rival Parti Islam SeMalaysia now working together, Malay voters had been expected to flock to Mr Mohamad.
Indians make up 27 per cent of the voters in the ward, Chinese voters another 19 per cent, and the remainder from other ethnic groups.
Saturday's by-election was called after an election court annulled the walkover win by Mr Mohamad in the general election last year.
Mr Mohamad, 62, had been favoured to hold onto the seat that he has held for three terms. He was also the former menteri besar of Negeri Sembilan.
A loss for BN would have been disastrous as Mr Mohamad is the man tasked to lead Umno, BN's lynchpin party, after its loss in the general election.
For PH, the defeat was the third consecutive one in a by-election, and will feed the perception that Malaysia's governing alliance has lost the support of Malays and other voters.
PH had in recent months been beaten by BN in the Semenyih (Selangor) and Cameron Highlands (Pahang) by-elections.
PH had won the first four by-elections called after last year's May general election.
Dr Streram is from PKR, and party president Anwar Ibrahim - who is called the prime-minister-in-waiting by his supporters - campaigned hard for its candidate in Rantau, but to no avail.