IBM lambasted by ABS for failing to handle Census DDoS

IBM failed to adequately address the risk posed to the Census systems it was under contract to provide, the Australian Bureau of Statistics (ABS) has said in its submission [PDF] to the Census Inquiry by the Senate Standing Committee on Economics, and the ABS has said it will manage risk better in the future.

"The online Census system was hosted by IBM under contract to the ABS, and the DDoS attack should not have been able to disrupt the system," the ABS said. "Despite extensive planning and preparation by the ABS for the 2016 Census, this risk was not adequately addressed by IBM and the ABS will be more comprehensive in its management of risk in the future."

The August 9 Census was pulled offline by the ABS after it experienced a series of denial-of-service attacks, suffered a hardware router failure, and baulked at a false positive report of data being exfiltrated.

During this year, the ABS said it had "received various assurances" from IBM on its ability to have distributed denial-of-service attacks (DDoS), but the bureau did not independently test the DDoS protections for the online Census.

"At no time was the ABS offered or advised of additional DDoS protections that could be put into place. Additionally, no suggestion was made to the ABS that the DDoS protections that were planned were inadequate," it said in the submission.

According to a timeline prepared by the ABS, the DDoS attacks began at 10.10am AEST on August 9. Following a second attack at 11.45am, non-Australian traffic to the Census servers was geoblocked.

The fourth attack, which resulted in the ABS pulling the Census site down, began at 7.28pm.

"The nature of the attack was different from the three previous attacks, and the online Census system degraded faster," the ABS said. "In addition to DNS reflection traffic, the online Census web services began experiencing resource exhaustion issues as all available HTTP worker threads were occupied.

"As a result of the DDoS attack, the ABS understands that IBM began to experience problems with its border routers. IBM first attempted to reboot its system at 7.43pm, but this was not successful."

It was during this attack that the ABS and IBM saw "an unusual spike in outbound traffic", which led to concerns the Census systems were compromised. At 7.56pm, the Australian Signals Directorate contacted and remained engaged throughout the night.

At 8.09pm, the ABS asked IBM to put the Census site into a state that prevented Australians from lodging further Census forms online.

The first phone call to the minister responsible for the Census, Minister for Small Business Michael McCormack, was made at 8.26pm.

IBM was able to reboot the Census systems at 10.26pm, but the decision was made to keep the systems closed to the public until the cause of the outbound traffic and failure to prevent the DDoS attack was identified.

"Investigations subsequently identified that IBM failed to properly implement geoblocking," the ABS said.

The statistics agency said its contract with IBM specified 98 percent uptime on the night of the Census during the four-hour peak in expected traffic, and fault resolution times were limited to a maximum of 30 minutes.

The bureau said it looked at using the Blaise system from Statistics Netherlands during 2012 and 2013, but in 2014 decided it was not a viable solution. The ABS subsequently used Capability Driven Acquisition (CapDA) to find a suitable vendor to handle the online Census.

"The CapDA assessment recommended that in consideration of the limited time frame and the inherent risks in working with any new organisation, the ABS should consider a limited tender to IBM given their existing experience of the application and Census program. IBM had successfully provided the online Census applications in the 2006 and 2011 Censuses," the ABS said.

"In July 2014, the ABS issued a request for tender (RFT) with a statement of requirements (SOR) to IBM for the 2016 online Census solution. Among other things, the SOR specified that the application must be built to best practice to prevent attack, and that the hosted environment must be protected from distributed denial-of-services attacks."

In response, the ABS said IBM put forward a range of solutions from fully cloud-based to fully dedicated hardware. Big Blue recommended using dedicated systems for Census data and cloud for non-sensitive components, a plan that the ABS agreed to.

"Apart from the requirement to store Census data on dedicated infrastructure, the ABS placed no restriction on IBM in the use of cloud-based services, as long as the use of these services was in line with Australian government IT security requirements. The IBM response also outlined measures to ensure that it would be 'highly resistant to web application security attacks', including DDoS attacks," the ABS said.

In September 2014, contracts were signed, as IBM was "determined as being value for money".

Days after the botched Census, Australian Treasurer Scott Morrison called out IBM, saying that if it is found responsible for the failure of the Census 2016 website, the federal government will pursue the global giant.

"You can expect the government to look so thoroughly into this to understand where the ultimate system failure occurred, and where that responsibility lay, and if there are issues that relate to the service provider in this case, you can expect us to pursue that to the nth degree," he said on August 12.

"The resources were there. The capability assessments and reviews were undertaken, the assurances were provided, and the events of 48 hours ago or thereabouts occurred."

In its own submission to the Census inquiry, the Community and Public Sector Union (CPSU) said financial pressures and the need for savings affected work on the Census.

"The decision to try to save money by trying to cancel the Census in 2016 stopped planning for six months at a critical juncture," the union quoted one member as saying. "It was then too late to ensure systems would be ready."

One member of the CPSU said that as far back as February 2014, the progress on the Census was disjointed due to IT funding uncertainties, while another said systems testing was six months behind schedule and that its systems were less than ideal because there was not enough time to build them properly.

"With all the delays, we had to descope and ... we were unable to run our dress rehearsal (an end-to-end test of systems and processes) in 2015," a member said. "Instead, we only ran tests on a couple of targeted systems and processes."

According to the ABS submission, as of September 20, 94.4 percent of households had completed the Census, with 59 percent completed online.

"The number of persons refusing to complete the Census form is low. At 20 September 2016, there had been 6,743 refusals. This compares with 13,194 refusals received in 2011," the ABS said.

Special Adviser to the Prime Minister on Cyber Security Alastair MacGibbon is due to hand down his own report into what went wrong on Census night.

On August 10, MacGibbon said handling denial-of-service attempts was a normal part of business for the government.

"The reason why the fourth [DDoS] incident was significant was because there were actually two failures. The first was a geoblocking service fell over ... that's one of the main defences used against denial of service," he said.

"The attack was no more significant than the types of attacks we would see all the time against Australian government systems. It's just that there was a confluence of events."

According to MacGibbon, the incident was not a defeat for the ABS.

"The more we talk about it, the more people decide to see if they are better than we are," he said.

"In this case what I'd say is, it almost ended up a draw.

"They managed to tip over some systems."