BIG DATA: Challenges and Opportunities

Disculpa, pero esta entrada está disponible sólo en Inglés Estadounidense y Portugués De Brasil. For the sake of viewer convenience, the content is shown below in one of the available alternative languages. You may click one of the links to switch the site language to another available language.

Companies have never known more about their customers that at present, but their innovation processes have remained a hit-or- miss. Why is that?

It will demand more than piles of data for businesses to be in fact successful. For the efficient use of big data, it will take a lot of energy, technology, and expertise.

There is a joke going around the Internet about a man who tries to order pizza and discovers that the pizza place is far more knowledgeable of more than just ingredients and crispy dough. The joke is about a customer calling to order some pizza, but the attendant soon tried to talk him out of it because of its fatty toppings due to his high cholesterol rates. The man was absolutely appalled about how the attendant could possibly know of his condition and how the pizza place had had access to his medical records. The customer then tried to use his credit card to settle the expense, but the attendant said that would not be possible because, as far as he knew, the customer’s card was over the limit and he also owed the bank some money. Later, the customer asked about how long it would take to deliver the food and the attendant replied that it would take around 45 minutes, but suggested that the customer could use his own scooter to pick it up faster instead. Finally, when claiming the bottles of soda he would get for free, the attendant said that it would be a bad idea since the customer had diabetes. Being completely annoyed but the whole experience, the customer simply hung up the phone.

This joke presents an unsuccessful purchase since the customer rejected the way business was conducted, the overwhelming invasion of his privacy, and not getting what he wanted. This anecdote could be interpreted in various ways, but for the purpose of this paper, it will be examined in light of how businesses have gathered and stored greater amounts of information on its current and potential customers, suppliers, demographics, and whatever is of interest in order to use data strategically to boost revenues and discover new business opportunities.

The strategy of collecting data from customers and elsewhere is not actually a recent task. Businesses of all sizes and shapes have gathered data to help managers prior to their 2 decision making. Before the term big data was coined, this relevant task was referred to as data mining/processing and data warehousing. The difference is that recently, according to a study by TWDI, an organization in business education, large data volumes skyrocketed in the early 2000s, so that storage and CPU technologies became overwhelmed by terabytes of data. Since then, fast-pacing advances in modern IT technologies have led to lower costs in storage and analytics. Thus, today it is possible to store and process trillions of data at a fast pace. “Big data” differs from the data collected previously according to three characteristics: volume, velocity, and variety.

Nowadays, no one will claim that big data is unimportant for enterprises, but have they changed their processes to accommodate this powerful tool? How much are these companies still reliant on their traditional managerial structure which is incompatible with the use of such valuable information?

It will demand more than piles of data for businesses to be in fact successful. For the efficient use of big data, it will take a lot of energy, technology, and expertise.For some experts from the McKinsey Global Institute (2011), the value estimated from the adequate use of big data in industries like health care in the USA alone approaches 300 billion dollars. For retailers, profit margin could increase up to 60%. On the other hand, some studies also show that simply gathering and analyzing an uncontrollable amount of data without reorganizing corporate structures can slow down or disrupt internal processes. “Big data” requires a rearrangement of organizational structures and decision-making processes.

The potential gains are overwhelming but they will demand a lot of efforts and changes on the part of organizations. Traditional practices and the growing need for data analysts seem to be just the tip of the iceberg. Several authors have drawn attention to the necessity of changes in managerial culture. McAfee and Brynjolfsson (2012) have pinpointed the following five management challenges that come along any enterprise that decides to use big data: leadership, talent management, technology, decision making, and company culture.

Business leadership, which has been traditionally organized in a top-down fashion where the more experienced senior executives were the ones with the last word in decision 3 making, will have to face changes. Experience and intuitive thinking are still very important, but leaders will have to start balancing them with the tons of data collected and processed that have become available. This does not mean that people will play secondary roles, but the power of data analytics is as stronger as ever and will have definitely to be part of decision making. The participation of Chief Information Officers (CIOs) in the decision making process have come to a whole new level.

As for talent management, big data analytics will demand more IT experts than ever before. Companies will have to recruit more data specialists, statisticians, and many other professionals that will manage both the necessary hardware and the software to use increasingly sophisticated algorithms to process tons of data into more digestible information for business decision makers. The data revolution seems to have outpaced the market’s ability to develop skilled IT professionals on time for the jobs. Companies will simply have to cope with that, and they will have to go to war for talented people.

Technology as well is by far not a settled issue either. The velocity of change and the increase of social media and other channels of communication force the IT industry to produce solutions to process all that content. It is up to companies to choose the right analytical tools, which are adequate to their data and objectives, because errors in this process can cost money and time. In addition, companies need to face increasing risks in relation to data security and Internet security, which will require financial and professional resources.

Another challenge of adopting the “big data” is the process of strategic decision-making, which is closely related to changes in the corporate culture and hierarchical structure. On several occasions facts and data will be seen as more convincing than experience and previous corporate standards and will have to be seriously taken into consideration. For large and traditional companies this can be a major challenge to overcome, but the new information era demands a more information-related approach. Therefore, companies must adopt a new way of conducting business that, like with any change in company culture, can take time and distress.

Companies that have managed to successfully tackle the challenges posed by big data analytics in order to enjoy its advantages are likely to be ahead of their competition. In this sense, successful cases in the corporate world can be seen everywhere. However, it is 4 worth remembering that some experts in Consumer Behavior and Psychology warn, without disregarding the powerful implications of “big data” to companies, that there is a lot more involved in consumers’ buying decisions than just data can show us.

Some academic references to look further into this theme:

ALMQUIST, E., SENIOR, J., and BLOCH, N. The Elements of Value. Harvard Business Review. September 2016.

BARBIERI, Carlos. Business Intelligence – Modelagem e Tecnologia. Rio de Janeiro: Axcel Books do Brasil, 2001.

CANARY, V. P. A Tomada de Decisão no Contexto o Big Data: Estado Único de Caso. Trabalho de Conclusão de Curso. Escola de Administração. Universidade Federal do Rio Grande do Sul. Porto Alegre, 2013.

CRAWFORD, Liz, The Shopper Economy: The New Way to Achieve Marketplace Success by Turning Behavior into Currency, 2012.

DESFORGES, Toby, The Shopper Marketing Revolution: Consumer – Shopper – Retailer: How Marketing Must Reinvent Itself in the Age of the Shopper, 2013.

ISMAIL, S., MALONE, M. S., VAN GEEST, Y. and DIAMANDIS, P. H. Exponential Organizations: Why new organizations are ten times better, faster, and cheaper than yours. And what to do about it, 2014.

MANYIKA, J. et al. Big data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey Global Institute. May 2011.

MCAFEE, A and BRYNJOLFSSON, E. Big Data: The Management Revolution. Harvard Business Review. Oct. 2012.

PHAM, P. The Impacts of Big Data That You May not Have Heard of. Retrieved from

STAHLBERG, Markus, Shopper Marketing: How to Increase Purchase Decisions at the Point of Sale, 2012.

Rodrigo Pádua
Founding partner of Grão Inteligência and Professor at Fundação Getúlio Vargas.