Nick Nicholas’ answer to Do Americans agree that the Prime Minister of Australia should have been treated the way he was by President Trump during their first phone call? In retrospect, that was more an answer to this question than that one.
TL;DR: Trump did it, because he’s not remotely interested in being a statesman, or preserving the delicate balances of the American World Order. He’s interested in being an alpha male who delivers to his base. (Or, per Scott Adams, he’s a Master Manipulator and Paradigm Disruptor.) And the point wasn’t haranguing Turnbull; it was being seen by the American voting public to be haranguing Turnbull. Which is why Bannon (presumably) leaked it.
As to the question details:
why would an American president go out of his way to alienate an ally whose soldiers fought and died alongside Americans in every war of the past century—including Vietnam?
ANZUS was founded in “We fought together in the Coral Sea! We were brothers! We would die for each other.” Except (as I wrote in my answer above), the Coral Sea did not motivate the US to deploy a single GI in support of Australia in Malaysia. And conversely, the Coral Sea meant for Australia that it wanted America stuck in the morass of Vietnam, because that made it more obvious that America would defend Australia.
Alliances are about self-interest, not glorious shared history. It was ever thus with Athens. It was ever thus with the US. It should have been ever thus with Australia.