SYDNEY (AFP) - Australian Muslim leaders said on Wednesday (Nov 21) they would boycott a meeting with Prime Minister Scott Morrison after he said they needed to do more to halt terror attacks in the country.

Mr Morrison singled out Muslim community leaders as having a "special responsibility" to counter "the radical and dangerous ideology of extremist Islam" following a terror attack in Melbourne earlier this month that left two dead.

"They must be proactive, they must be alert and they must call this out," Mr Morrison said, adding that he would hold a roundtable meeting with Muslim leaders this week to discuss the problem.

But Australian Grand Mufti Ibrahim Abu Mohamed and a group of other senior Muslim figures rejected the invitation in an open letter to Mr Morrison on Wednesday.

The men said they were "deeply concerned and disappointed" with comments by Mr Morrison and other ministers "which infer that the community is collectively culpable for the criminal actions of individuals and should be doing more to prevent such acts of violence".

"These statements have achieved nothing to address underlying issues, but rather, have alienated large segments of the Muslim community," they said in the letter, which was published by Australian media.

The letter prompted a tweetstorm from Mr Morrison, who accused those behind the boycott of "continuing down a path of denial" and making their communities "less safe and more vulnerable".

"We all have responsibilities to make Australia safe, and that means making sure Muslim communities do not become infiltrated with this dangerous ideology," he tweeted.

In the Nov 9 attack in Melbourne, a Somali-born man who police said was inspired by the Islamic State in Iraq and Syria (ISIS) group, stabbed and killed one man and wounded two others before being fatally shot by police.

And this week, police arrested three more men in Melbourne for allegedly plotting another ISIS-inspired attack on the city - the 15th terror attack that the authorities say they have thwarted since 2014.

The terror threat has fuelled calls from some conservatives for a further toughening of Australia's already tight immigration laws, raising concerns among the Muslim community.

In their letter, the Islamic leaders, who head community groups in several Australian states, said they would attend a later meeting with Mr Morrison if their "views and concerns will be genuinely respected".