The Time and Tides of Big Data:

The world is exploding with information. Information, as we know it is changing by the hour. Traditionally, organizations the world over have broadly considered “Relevant Information” as data which is recorded by employees and clients, data relevant to the nature of the engagement such as contracts, statement of work(SOW), Work Orders, Project Requirements, Feasibility studies, project plans, project tracking documents etc.. Most of this data was and is being captured in Management Information Systems in order to help companies leverage this data for future engagements. However, we will be dealing with what is known as “Big Data”.

                Wikipedia defines Big Data as “Data sets with sizes beyond the ability of commonly-used software tools to capture, curate, manage, and process the data within a tolerable elapsed time”. Big data essential deals with a category of raw data that was traditionally not considered information.  Picture this:

1.       You switch on your laptop and connect to the internet using your Mozilla Firefox Browser.

2.       You then search google for that brand new smart phone you are planning to buy.

3.       Pages of search results then appear on your browser.

4.       You might browse through several of the listed websites, spend anything between a few minutes to an hour and finally make your purchase.

Let us go through the whole sequence again to understand a few aspects regarding the power of Big Data.

1.       The fact that you have connected to the internet via a “Mozilla Firefox Browser” becomes a piece of information.

2.       i. You may see search results regarding all the available categories of smart phones unless you are specific about the search criteria.

ii. You will also come across sponsored advertisements by select companies on the search results page.

iii. Somehow, these advertisements seem to understand how much money you are willing to shell out or your new phone.

iv. They even seem to understand the exact features you are looking for in a smart phone.

v. You then open individual websites and again to your convenience you see more advertisements which seem to have narrowed down to your selection set as offer you a variety of options you had not explored before.

vi. There are various other cues that you can observe; however, we will stop with you purchasing the smart phone of your choice.

Every piece of information mentioned above relates to Big Data in one way or another. Technology today has evolved to the extent that it is possible to identify an individual by reconstructing his/her profile by uncovering hidden patterns in the vast data trail left by the individual on the web. Currently, nearly 2.5 exabytes(10^18 bytes)  of data are being produced each day. This number is expected to double every 40 months.

The vast amount of data has important implications for today’s companies. In my next blog we will try to understand what these implications are and how companies can leverage the use of Big Data to their advantage.