Big Data is no longer a buzzword. Instead, it’s a transformational tool that is propelling many of today’s industries.
Every day, we collect vast amounts of data from the complex network of devices we employ to run our businesses and do our jobs. This data requires sophisticated analysis in order to make it meaningful. The Merriam-Webster dictionary describes big data as “an accumulation of data that is too large and complex for processing by traditional database management tools”. Big data programs collect and analyse data to provide important, timely and relevant information that can provide industry with the tools to make decisions and adapt to market trends.
In 2008, a number of prominent American computer scientists popularized the term ‘Big Data’ predicting that it would “transform the activities of companies, scientific researchers, medical practitioners, and defence and intelligence operations.” Ten years since that statement and an IDC report predicted that the big data and the business analytics market would increase from $130.1 billion in 2017 to more than $203 billion in 2020, and by 2030, big data will add around $15 trillion to the value of global economy.
Never before have we had access to such a wealth of information and because of this, big data is having a dramatic effect on almost every area of our lives.
The IDC report also outlines where big data is having the most significant impact. Banking was cited as being the industry with the largest investment in big data and business analytics solutions at nearly $17.0 billion in 2016, although telecommunications, utilities, insurance, and transportation are forecast to have the largest compound annual growth rates (CAGR) by 2020. Other areas where big data is having the most impact are healthcare, retail and sports.
As the world’s population continues to grow and as people live longer, there’s an urgent need to address the rising costs of care and medication. In the healthcare sector, trends identified by the analysis of big data help reduce overhead costs and increase efficiency that could ease the strain on healthcare systems and providers and ultimately, provide better quality of care.
Analysis of patient data and populations trends identified by vast numbers of people passing through a country’s healthcare system can help predict epidemics, disease outbreaks, and occasionally forecast future ailments from mapping health patterns and symptom indicators. This allows a government’s health systems and health providers to plan ahead and manage resources when need spikes.
On a personal level, a trend for healthcare apps has emerged with individuals tracking their own health status by linking their smartphones or wearable devices to platforms that monitor everything from temperature to calories burned, heart rate and distance walked. This data enables an individual to tailor or change their lifestyle in order to remain healthy. In addition, the data stored within these apps could also enable a physician assessing the patient to provide better diagnosis, treatment plans and eventually medication plans, as personalised health begins to emerge in the mass market.
Increasingly, retailers are counting on social media and behavioral data for inventory management, planning and marketing. It can help predict consumer trends and allow companies to amend their product and service offering ahead of a trend becoming mainstream. Whilst these predictions cannot be fully relied upon, increasingly, the data collected from purchasing trends and socioeconomic data can make a significant difference to a company’s bottom line. Walmart, for example, is using big data to determine when prices should be dropped and when to increase prices. In the days before big data, most large retailers depended on local demand-supply dynamics for price reduction. But studies have shown that when prices are personalised for each consumer, increasing revenue and customer loyalty.
Big data can also be applied to supply chains and delivery routes. By using big data and analytics, transportation routes can be optimised by cutting down shipping time and costs. For example, in the near future, drones and other unmanned aircraft will be able to deliver goods using big data and analytics to predict where demand will be highest and move goods closer to the location. Amazon is pioneering this field by shipping goods to nearby facilities before they are ordered, significantly reducing delivery time and overheads.
The 2003 Michael Lewis book “Moneyball: The Art of Winning an Unfair Game” explains how Billy Beanem, Oakland Athletics general manager, turned a struggling team into a title contender by employing big data and analytics. This story of success inspired many sports teams to incorporate big data and analytics into their strategies. Since the publication of the book and premiere of the film “Moneyball”, the use of big data and analytics has become highly strategic both for sports teams and leagues.
Big data technology allows coaches to measure and analyse previous games and matches. Instead of relying on experience and intuition, sports participants and enthusiasts can examine data that tells the real story of every aspect of the game – from player recruitment to fan engagement. Many sports now have the players or participants wearing monitoring devices embedded in their kit to track their every move. For example, in the UK, Premier League football team Arsenal invested millions in developing an analytics team to manage the data it collects from player tracking. The team have installed eight cameras around their stadium to their player and interactions on the pitch.
In addition, Prozone, a sports analytics provider tracks 10 data points per second for every player on a pitch, resulting in 1.4 million data points per game. The system analyzes games using automated algorithms alongside manual coding of every interaction with the ball to increase accuracy and maximise the value of the data collected.
Being able to track each player and the overall success of each match on pure data allows for in depth post-match analysis that can make a tangible difference to the next performance, as well as improved training plans or game plans.
Whilst the benefits of big data seem to grow exponentially with every advancement in capability, the importance of safeguarding those that data is collected from becomes paramount. Unless stringent codes and frameworks are put in place, big data could suffer irreparable reputational issues that could mean we have to forfeit the potential of this ability.
As a result, data governance will profoundly affect the future of big data. To protect the technology, and those that it tracks, data governance must tag the data to associate where it came from, who is able to access it, whether it was modified, and where it was applied. This allows assurance and accountability in the form of traceability, provenance, context and privacy, and security. Whilst technology leaps ahead in the area of big data, in order to realise its true benefits, governance must evolve at the same pace.