Solving Big Problems with Small Data

Big Data has no value unless it’s linked to a business metric. With all the hype and excitement surrounding Big Data, business leaders and teams have gotten lost as to where to start in order to achieve this critically important jump.

Business teams posses the skills and experience to know where value creation is needed - after all, they know their products, customers, and markets better than anyone.

Related Article: The Final Stretch in Big Data Management: Driving Business Value for Enterprises

The key skill gap that business teams face today is to take what are often well thought-out KPIs and set Big Data on the right path of discovery.

This is where Big Data meets small data.

Young businesswoman sitting on chair with big light bulb above.jpeg

What is small data? How is it different than big data?

As business leaders, we have been working with small data for decades - it’s this data that helps us achieve business goals every day. Teams periodically plan and assess long-term strategy, setting and resetting strategic goals - but what are the metrics, or small data, that matter day-to-day?

  • Grow e-commerce sales by 34% to achieve $142m
  • Blue Tree division to deliver $1.46bn in sales
  • EMEA to deliver $1.7bn in sales
  • Deliver $33m in Supply Chain savings
  • Deliver $429m in sales from Projects (growth of 10%)
  • Reduce unprofitable SKU´s by 58%
  • Efficiency Savings $12.5M

I have worked with numerous teams who monitor KPIs and metrics through a dashboard - looking at flashing red, amber or green lights that fluctuate against planned activity.

Teams that successfully take these metrics (small data) and ask Big Data a question as to how to move them will be successful. 

Big Data examples are everywhere - sentiment analysis from social media data, mobile phone GPS data, data from sensors in street lamps, bus stops, and cars - all amount to huge data sets that normal business leaders lack the skills to manage. Luckily, many business leaders and companies now have skilled data scientists accessible to their teams to wrestle with this challenge.

It’s here that when Big Data and small data work in harmony and are complementary, we start to really unlock the value of Big Data.  We can see the business value of the investments in large data infrastructure, we can start to see where data science talent has the biggest impact.  Fundamentally, we can start to capture value from Big Data.

Small Data Working Together with Big Data - A Military Example

To just offer a data scientist “here is the KPI - go solve it” is not good enough. Business leaders and analytics expertise need to go to big data using a familiar and simple language as a handrail. Scaling this language will enable teams across the organization to build confidence to do this and operate more effectively. 

As a former Military Intelligence Officer, we referred to such a process as the intelligence preparation of the environment or “IPE”. Front line commanders would clarify what was important to them (small data) and we would connect this to the larger ecosystem, leveraging the scale of the group. Getting this right meant that we could break data silos across the battlefield and ensure the right data was going to the right mission at the right time. 

For example, in combating IEDs (Improvised Explosive Devices), front line commanders like myself would look at tactical mission success on a foot patrol or convoy move and link to Big Data sets such as DNA databases, social media data, sensor data or human intelligence - such as tip offs from informants as to where and when strikes were most likely. Getting this linkage right was a life or death skill.

Big and small data worked in harmony through a simple language across teams.


Using Small Data to Measure ROI of Big Data

Small data must not be ignored. Through these metrics and indicators, organizations can identify, at scale, Big Data success throughout their teams across the business. Ignore this and you will fail to realize Big Data potential. If you are not clear on the dial you’re trying to move and by how much and by when, how can you measure it?

If teams start with Big Data or just say to a data scientist “go explore and see what you find,” measuring business value will be difficult.

Connecting big data and small data and aligning to what’s important is the starting point. Subsequently, going to data with better and more relevant questions is the thinking skill that organizations must build throughout their teams in order to lead in this information age.

Industrialize data-driven decisions with Connectworxs

The ADAPT Framework™ is a guide for teams to use everyday as they connect Big and small data with their teams.


Get new insights straight to your inbox