Pattern Discovery: Partnering With Data Scientists

Pattern Discovery: Partnering With Data Scientists

The challenge in the age of ‘big data’ is finding the meaning and insights in these large datasets. Companies and business leaders have been archiving terabytes and petabytes of information in somewhat of a cryogenic fashion waiting for the day when someone will finally unlock the patterns contained within the data and discover the insights hidden within.

Talented statisticians and data scientists have used many methods to gain insights, but often they are finding information based on prior knowledge – in some sense finding what they already know – and updating that information with the current results over the last year. What it is not allowing for, however, is for the data itself to tell its own story – what is the most significant data in the dataset, and how one set of features relate to other features in the data? What features within the datasets are the most responsible elements for a specific outcome, or set of outcomes?

When the data tells the story, it reveals the full extent of what is within the data, and how the data is related to that which is known, and unknown. It can confirm what the data scientist already knows and may also provide the hints that experienced data scientists and subject matter experts use to develop new insights into related factors which heretofore were unrecognized players in the outcome.

Read the paper here.