Australia has made a "huge" diplomatic error by ditching a multi-billion-dollar order for French submarines in favour of an alternative deal with the United States and Britain, France's envoy to Canberra said on Saturday.

Australia announced on Thursday it would scrap the 2016 deal with France's Naval Group to build a fleet of conventional submarines and instead build at least eight nuclear-powered ones with U.S. and British technology after striking a trilateral security partnership.

Australia's decision has also riled China, the major rising power in the Indo-Pacific region, and Malaysia expressed concern on Saturday that Canberra's decision to build atomic-powered submarines could trigger a regional nuclear arms race.

"It will provoke other powers to also act more aggressively in the region, especially in the South China Sea," the Malaysian prime minister's office said, without mentioning China.

Beijing's foreign policy in the region has become increasingly assertive, particularly its maritime claims in the resource-rich South China Sea, some of which conflict with Malaysia's own claims.

France, a NATO ally of the United States and Britain, has branded the cancellation of the deal - valued at $40 billion in 2016 and reckoned to be worth much more today - a stab in the back and recalled its ambassadors from Washington and Canberra.

"This has been a huge mistake, a very, very bad handling of the partnership - because it wasn't a contract, it was a partnership that was supposed to be based on trust, mutual understanding and sincerity," Ambassador Jean-Pierre Thebault told reporters in Canberra before returning to Paris.

"Deep Disappointment'

US State Department spokesperson Ned Price said France was a "vital ally" and that the United States would work in the coming days to resolve the differences.

Thebault said he was very sad to have to leave Australia but added there "needs to be some reassessment" of bilateral ties.

In separate comments made to SBS radio, Thebault said of the ditched agreement: "It was not about selling salads or potatoes, it was a relationship of trust at the highest level covering questions of the highest level of secrecy and sensitivity."

Australia said it regretted the recall of the French ambassador, and that it valued the relationship with France and would keep engaging with Paris on other issues.

"Australia understands France's deep disappointment with our decision, which was taken in accordance with our clear and communicated national security interests," a spokesperson for Foreign Affairs Minister Marise Payne said on Saturday.

The row between Paris and Canberra marks the lowest point in their relations since 1995, when Australia protested France's decision to resume nuclear testing in the South Pacific and recalled its ambassador for consultations.