Australia's Prime Minister Malcolm Turnbull is clinging to power after narrowly winning a leadership challenge that left his ruling Liberal-Nation coalition in tatters.

In a bid to stave off a challenge from Home Affairs Minister Peter Dutton, Mr Turnbull yesterday morning invited a leadership ballot at a meeting of Liberal Party MPs.

After Mr Dutton challenged, Mr Turnbull won by 48-35 in a secret ballot - a margin that most analysts think is too slim to secure his future or prevent a further contest.

Mr Turnbull said he had cut taxes and produced strong growth in jobs despite holding just a one-seat majority in Parliament.

"We know that disunity undermines the ability of any government to get its job done," he said. "Unity is absolutely critical."

Hardliner Mr Dutton challenged amid concern over the coalition's poor standing in opinion polls.

He was backed by a group of conservative MPs who have expressed concerns about Mr Turnbull's announcement of an energy policy that included a target for cutting carbon emissions by 26 per cent.

To save his leadership, Mr Turnbull softened the policy, then abandoned it, but that did not prevent a challenge. After the ballot, Mr Dutton resigned from the Cabinet despite Mr Turnbull inviting him to stay. Mr Dutton did not rule out a further challenge and is now well-positioned to canvass backbench support and criticise Mr Turnbull.

"I have gone through what my job is now and that is to make sure that I can help the coalition win the next election," Mr Dutton said.

Australia has suffered years of leadership chaos, including coups against sitting prime ministers.

The last leader to serve a complete term was Mr John Howard after the 2004 election.

Mr Turnbull has faced questions about his leadership since narrowly winning the 2016 election.

To try to maintain control of the party's right wing, Mr Turnbull abandoned many of his progressive positions on issues such as climate change and same-sex marriage.

This damaged his standing in the electorate and failed to win the support of the party conservatives.

Most analysts believe Mr Turnbull is likely to face a further leadership challenge in the near future.

Political commentator Laura Tingle wrote on the ABC website: "There is blood in the water and the entire political apparatus, notably the media, is on the scent."

Mr Dutton, as leader, would be likely to curb immigration, invest in infrastructure and take a tough approach on crime and national security. But his challenge was largely about politics, rather than policy.

It followed the Liberals' dismal result at a by-election in the state of Queensland three weeks ago.

Mr Dutton holds the neighbouring seat by just 1.6 per cent. The by-election appears to have persuaded him and several other MPs that their political future was at risk under Mr Turnbull, who is due to hold a general election by May next year.

But it is not clear whether Mr Dutton would improve the coalition's political fortunes. Opinion surveys show he is favoured as Liberal leader by only 3 to 11 per cent of voters. It is still possible that other leadership contenders could emerge if another challenge is mounted.

Mr Dutton's resignation from the Cabinet will force Mr Turnbull to rearrange his ministers, potentially adding to the instability. Several of Mr Dutton's backers also offered to resign from the Cabinet last night.

Labor leader Bill Shorten, who is ahead in the polls, said: "This is a narcissistic government consumed with their own jobs and their own struggles and they have forgotten the people of Australia."