Have you read Thinking Fast and Slow by Daniel Kahnemen?
He's very big on the side of data over intuition. Not sure I agree in all cases though; data, especially big data, can really lack resolution. I teach and am constantly bombarded with data; and interesting one came up recently - a study showing that "what" and "how" questions work better on students than "why" questions (less aggressive). I don't doubt the data, but I still trust my intuition to tell me when a student is ready for a "why" question, and when a more aggressive style is called for in order to push a student out of their comfort zone.
Data is great for policy, but freedom should be left in the hands of the practitioners to use their intuition or else we only serve those who sit in the middle of the bell curve and leave behind the outliers.
I would think that practitioners primarily using their intuition rather than data would be the ones in danger of sitting just in the middle of the bell curve, ignoring the outliers. It is unexpected data
that allows us to see the outliers, not our intuition. Intuition is, in a sense, the result of large amounts of potentially correlated data being processed for long periods of time that 'wire in' shortcut algorithms for our decision making process. Intuition has the effect of elevating
the weight of data that meets our expectations over the weight of data that do not
meet our expectations (in terms of how that data is incorporated into our decision-making process).
Intuition is great - it's the process of creating cognitive shortcuts for us so that we don't have to spend so much time rationally processing every little bit of data. It allows us to make connections (that are actually
correlated) that we otherwise wouldn't make if subject just to rationality. However, it also allows us to make connections that we otherwise wouldn't make if subject to just rationality that are not actually correlated
So how best to take advantage of both worlds? I propose that one should accept their intuition, but if possible, check it against the data
, and if the data disagrees with your intuition, go with the data
. If your intuition is strong enough to make the data less compelling than you rationally think it should be (i.e. the data says it's a straight line, but it really, really, really feels like a logarithmic function), bring other intuitions (i.e. other people) into the decision making process. Do what we can to eliminate intuitive gut feel that comes from correlating data that isn't actually correlated. It's not perfect, and will still lead to some missing of outliers, but this seems, to me, to be the most practical consideration.
But I think I totally need to read that book.