Big Data

Australia’s flagship government research agency – the CSIRO – has ramped-up its Big Data project work on behalf of both public and private sector clients, calling it the most revolutionary and potentially transformative IT technology to emerge in years.

The CSIRO increasingly offers its services on commercial terms, and is using its strength in high-end mathematics and computing to help Australian customers take advantage of Big Data capabilities.

Alan Dormer, the Government and Commercial Services theme leader within the CSIRO’s Digital and Productivity Services Flagship, says Australia is still a long way behind the US in extracting that potential that Big Data can offer. But it is a huge area of focus, and that Australia is following its usual path as an enthusiastic early adopter.

Mr Dormer will present at the CeBIT Global Conferences’ Big Data event being held in Sydney on October 31 – November 1. He will outline the Big Data impact on the economy and society, running through a host of project examples that the CSIRO has been engaged in.

Mr Dormer says sophisticated commercial and government users in Australia are well aware of the generic capabilities of Big Data projects. But until it is applied to a specific organisation, it is difficult to get a read on its value.

“It’s not the product that’s the issue. It’s the customer knowing what to do with it,” Mr Dormer said.

“We basically do research in this area (on behalf of clients.) So as people see what’s possible, they tend to get more ambitious in terms of (what they want to do with) Big Data,” he said.

“The appetite grows with the eating (for clients.) And we have the largest concentration of mathematicians in the country, so we have been pretty active in this space.”

The CSIRO projects are at the sharp end of research, rather than commercial implementations or consulting. These projects usually involve its officers writing applications specific to the project, and sometimes taking advantage of the supercomputing facilities at CSIRO.

Among the projects so the CSIRO has engaged with so far, the focus has been in government services (including looking how people interact with services,) financial services (including payment security and fraud) and at the disaster management area.

Customers have included the federal Department of Human Services , the United Nations and AusAid, and NSW Fire and Rescue.

Mr Dormer says the agency is now starting a focus on the retail sector, an area where he said Big Data will have an enormous impact. “It’s going to make a huge difference to the way that people deal with customers – and that includes clients, or citizens, depending on the organisation dealing with them.”

See Alan Dormer, Science Leader, Government and Commercial Services, CSIRO, at the Big Data Conference in Sydney on
31 October – 1 November 2012.

Australia’s corporate sector has taken to burgeoning Big Data market in the same fashion it has taken to other waves of new technology; as early innovative adopters, according to Daniel Miller, the local country manager of Nasdaq-listed big data veteran Splunk.

Big Data issues are not just about data volume. The variety of different kinds of data, and the speed at which it is growing are added complexities. It is unstructured and ballooning.

Consider that every 60 seconds Google serves more than 694,445 search queries, or that 600 videos are uploaded on YouTube videos, adding more than 25 hours of content, or that 168,000,000 emails are sent.

Or that every 60 seconds 695,000 status updates are published on Facebook: or 79,364 wall posts and 510,040 comments. And those numbers don’t include all the machine-generated data that results from each of those human creations.

Set up in Australia just two-and-a-half years ago with a single employee, Splunk has been one of the most active technology transfer drivers for Big Data in this country. It has quietly created a significant, fast-growing subsidiary with nine employees and a trajectory that will double head count again in the next 12 months.

“Australia has always been an early innovative adopter of new technology, and that has been no different with Big Data,” Miller said.

Splunk is bringing one of its senior commercial and technical experts to Australia to speak at the Big Data Conference in Sydney on October 31 to November 1, an event presented by the CeBIT Global Conferences group.

But where most of the Big Data vendors and services providers tend to be consultants on the Google-inspired Hadoop and Maproute systems, Splunk brings different products and different value propositions to its customers. It has its own distributed indexing architecture and its own search language.

Like the rest of the Big Data sector, Splunk focuses on unstructured data – but its real focus is in the massive volumes of machine-generated data. According to research group IDC, about 90 per cent of the data in today’s organisations is machine generated – by websites, applications, servers, networks, mobile devices and the like.

The whole value proposition of Big Data rests on the notion that these massive pools of unstructured data hold tremendous value, Mr Miller says. The trick to unlocking that value rests in being able to handle such massive data volumes, being able to handle lots of different types of data, and being able to handle the sheer speed at which new data is being generated.

Splunk’s Enterprise product collects, monitors, indexes and analyzes the machine data generated by IT applications and infrastructure–physical, virtual and in the cloud. This machine data is massive in scale and contains a definitive record of all transactions, systems, applications, user activities, security threats and fraudulent activity. This data is largely untapped; Splunk helps organisations unlock its value.

Mr Miller says the Splunk platforms differ from competitors in that they contain an in-built dashboard and tools, as well as specialist visualization features that simplify the trending and results information generated by vast pools of raw data.

As the largest information technology user in the country, the Federal Government is expected to become a big user of Big Data toots, and will be a focus for Splunk.

Mr Miller says local customers are often reluctant to talk about precisely what they are using Splunk tools for – sometimes because they are still in test “suck it and see” mode, and sometimes because they don’t want to reveal a competitive advantage.

But the company has made strong inroads in the University sector – especially as Big Data systems are well suited to development and research environments – and Mr Miller expects strong growth among public sectors customers in the coming year.