A note on big data
Big data. That seems to be the talk of the town. Combine it with IoT, Cloud and Machine Intelligence, we have all the key tech world hype memes covered.
So essentially the idea boils down to this:
- Collect through various means, observations which might have a bearing on a particular situation
- Analyze patterns in those observations to arrive at a causality model
- Extrapolate the causality model to predict future responses for various stimuli
- Provide a certain stimuli to test the prediction or wait for such a stimuli to occur in the system and observe the response
- Depending on the observed response and predicted response, we can ascribe a confidence measure to the causality model and use the model further for predicting responses for other such stimuli
This is not a new thing. It is what we call as the practice of science and the scientific method. People such as Brahe, Kepler, Pasteur etc practiced this.
But then there are two key differences.
Change:
The context of science is based on ancient objects and ancient rules like the human body and gravitation while the context of big data which finds most application in business and business like scenarios, is based on rules which are in constant flux – rules which are based on three key things – human psychology, technology evolution and social structures. These are constantly changing and are not set in stone like the genetic code. Infact constantly we are faced with disruptions in these areas changing the underlying rule fabric dramatically, in turn changing the geodesics of the objects observed in ways we couldnt have foreseen.
That is the challenge. To be proficient enough in observing the fast changing undulations in the rule fabric and to be nimble enough to adapt to such changes. It is akin to surfing in turbulent waters! Unfortunately we currently move in such glacial pace in doing this, that before we understand a particular rule fabric through our various Big Data tools, the fabric has changed quite dramatically.
Complexity:
Science is mostly simple, modeled with a rather small set of simple variables. On the other hand the situations pertaining to business are far from simple and would need a much higher number of complex variables. Unfortunately we tend to use similar approaches in figuring out the rule fabric as we did previously in science like for instance statistical tools like regression analysis but I feel that such elementary tools arent good enough to bell this cat. It is akin to stock traders trying to predict the market or federal bankers trying to control monetary situation in each country through some rudimentary mechanisms which typically fail miserably again and again.
This is the second challenge.
But having said that, Big data will be useful in certain minimal ways to handle certain situations which typically are governed by less complex unchanging rule fabric. The trick is to find if the situation you are working on is such a easy golf course which can yield to pot bellied aging simple minded executives or a turbulent sea which yields, if at all it yields, only to fit, nimble and open minded athletes.
