Trusting big data

Big Data

Photo Credit: TheDigitalArtist via Pixabay

There are some issues that businesses have to consider before starting big data projects, writes Timo Elliott for

The completeness and accuracy of the data gathered are important because the recorded values may be approximations rather the exact values. For instance, a business scrutinizing customer sentiment from social media may find the task difficult because of the nuances of language. As such, it would be very difficult to determine the true motive behind a tweet or a blog. It has also been shown that people interpret the same emoji in various ways.

Businesses should also be concerned about the credibility of big data because these often come from external sources and may contain biases or outright false values, such social media bots.

Data consistency should also be a matter of concern for businesses because big data systems employ eventually consistent systems, where two people may get different answers to the same questions.

The generation of useful information entails extensive processing, interpretation and the use of algorithms. This may lead to bias and wrong conclusions than with traditional database systems.

For instance, self-driving cars use algorithms and image recognition to read traffic signs but may read the signs wrongly when these have been defaced or vandalized.

The validity of big data is still subject to human intervention to determine if it is an appropriate and useful source of information.

Businesses should also assess where big data could generate business value, determine if the data is reliable and how much investment might be required.

They should also install data orchestration systems that monitor the information lifecycle of big data and enable transfers to traditional database systems that fulfill the governance and security requirements of the enterprise.

Big data should also be transparent, and data inaccuracies and biases should be readily available to businesses.

Even though the analysis of big data is best left to data scientists, non-technical users should also be made to understand the consequences of using big data in their decisions.  

Big data and the Internet of Things may offer businesses unprecedented opportunities to exploit but if it is intrusive to employees, customers and to society, its best not to proceed with it. If customers find it unsettling when they discover how much businesses know about their activities, then it should not be done in the first place.