Telstra did not act on repeated warnings to fix an impending issue with a piece of equipment that ultimately triggered a devastating network outage, a parliamentary inquiry has heard.
US-based supplier Microchip Technology warned the telco multiple times that a Network Time Protocol server required a software update to prevent a "rollover problem" that would take down the system, Telstra officials told a hearing in Canberra.
The snap inquiry was set up in response to the ensuing outage, which left train networks in disarray and customers unable to dial triple zero.
In a submission the inquiry, the telco said a GPS card within the server did not operate as expected once power was restored to the device following scheduled maintenance.
The change had not been properly documented, meaning the maintenance team was not aware the issue would occur when the device was restarted.
A software update, which could have averted the outage, had also not been applied to the device.
Telstra chief executive Vicki Brady said she was "deeply sorry" for the failure.
"Telstra let Australians down. We let our customers down, we let the community down, and we fell short of what people rightly expect from us," she told the hearing on Friday.
"Triple zero sits at the heart of public trust in Australia's communication system. Telstra has a critical role in the system and we take that responsibility extremely seriously.
"No one should be left wondering whether a call for emergency help will get through."
Gerard Tracey, the executive responsible for end-to-end service performance, said the supplier first warned about the impending issue in the SSU 2000 server, a 15-year-old piece of tech which costs about $30,000, in November 2020.
The issue resulted in the timing server being reset to 2006, causing invalid authentification certificates to spread across the network and preventing customers from making calls and accessing data.
Inquiry chair Sarah Hanson-Young questioned why the repeated warnings hadn't been acted on.
"This doesn't seem like a hardware matter. This seems like incompetence," the Greens senator said.
Telstra received another reminder in January.
The reason why the warnings went unheeded would be probed in an external investigation Telstra has commissioned into the outage, Ms Brady said.
"This is not an excuse, but putting in context, there are many software updates and changes getting made across the network," she said.
"Any software update creates a risk when we put it into testing.
"In this case, they thought, given the design that was in place, it was not relevant because we were not using this equipment in a way that would trigger this risk."
The telco has been in damage control since the outage occurred on July 8.
An EFTPOS payment system used by about 80,000 businesses confirmed it had been impacted, with a number of customers taken offline and unable to process payments.
Train services had to be cancelled after severe network disruptions in both Victoria and NSW, triggering strong criticism from the federal government.
Telstra is accepting compensation claims from affected customers and small businesses who can provide evidence to support their case.
Telstra undertook more than 630 welfare checks on customers that were prevented from having triple-zero calls connected due to the outage.
Representatives from the Australian Communications and Media Authority and the communications department will also give evidence at the inquiry.