Enterprise Analytics Solutions Have Arrived at the Mid-Market

For over a decade, businesses have been told to "be more data-driven." They’ve also heard that "if you can't measure it, you can't improve it," and that "AI is reshaping business." Yet one thing left out of these conversations is the sizable investment needed to follow this advice. Plenty of data success stories exist about businesses like Kroger and Netflix, but we don’t hear much about mid-market organizations without a dedicated data engineering or data science team. Although these businesses sit on lots of customer data, they don't have the same access to money, technology, and talent as enterprise organizations.  

In the last five years, a series of technological shifts have changed all that. Mid-market businesses seeking to tap into their customer data now have access to cloud-based, turn-key solutions that dramatically lower their financial, technological, and personnel-based barriers to entry. These shifts have been gradual, but we’ll explain how they’ve accumulated over time to transform the world of customer data and analytics. 

 What Might Happen to Customer Data in 2021

Before we explore these technology changes, we’ll review why this matters. A 2020 survey from Seagate indicated that only 32% of data collected by businesses is used. Based on that statistic, organizations that can activate a greater share of their available data receive a competitive edge. Unfortunately, it’s far too common for mid-market businesses to either skip data collection or collect data into silos, causing them to miss out on the insights and actions they could take if they merged the data into a central analytics repository like a data warehouse. With such a warehouse in place, businesses have access to: 

• Better campaign attribution and targeting which results in improved advertising return on investment (ROI) 
• Sophisticated cross-channel personalization routines that bounce between offline, email, web, app, and SMS interactions 
• Better reporting that blends front-end and back-office systems 
• Real-time, unified customer insights that help drive product vision and test product changes 
• Standardized data formats and regulatory compliance tools 

With these advantages in mind, we’ll dive into recent changes that have allowed mid-market businesses to derive more value while spending less on data warehouses and analytics solutions.  

The Rise of MPP Data Warehouses 

Modern data warehouses can run transformations on their own, reducing their infrastructure and eliminating the need for specialized staff to run them. 

For many years, the terms “data pipeline,” “big data,” and “data warehouse” were often followed by another term: Hadoop. As an open-source standard for processing big data, Hadoop opened up a world of potential for working with enormous, real-time data sets. However, this came at the cost of skills and infrastructure because Hadoop is a specialized technology that requires dedicated servers and engineers.  

Enter the Massively Parallel Processing (MPP) data warehouse. MPP products like Redshift, BigQuery, and Snowflake have moved data processing into the data warehouse by extracting the infrastructure and code associated with Hadoop. This has had a massive impact on the analytics community since a whole swath of server infrastructure and specialized skills were eliminated for the most common analytics use cases. Additionally, the cloud-native approach of MPPs means that businesses can now scale from megabytes to petabytes with minimal IT involvement.  

The Introduction of Serverless 

Modern serverless warehouses reduce IT burden and provide limitless scalability.  

Plenty of articles have been written about how the cloud has revolutionized the tech world, but it's worth pointing out the enormous effect the cloud has had on modern data warehouses. 

The Availability of Turn-Key Integrations 

Vendors have stepped in to simplify what used to be a very complex challenge: getting marketing tools to talk to one another. 

When designing any moderately complex warehouse or data pipeline, it was often assumed that some duct tape, in the form of custom code, would be needed. However, mid-market businesses don't have the internal engineering staff to continually maintain and refactor this custom code, making this a common barrier to entry. Fortunately, a number of technological standards and vendors have simplified the space and made data integrations more turn-key.  

How to Analyze Audience Behavior on Association Websites 

Monolithic technology stacks offered by vendors like Salesforce and Adobe have often benefited from the selling point of "Our products can talk to one another.” This competitive advantage has slipped away, however, as standards like REST, JSON, and webhooks have taken hold and allowed unrelated platforms to speak the same language. What this means for mid-market businesses is that they can BYOS or "Bring Your own Stack" by picking from best-of-breed vendors rather than depending on a single, monolithic vendor.  

Also benefiting from these standards are vendors such as Fivetran, Stitcher, and Segment. These tools had early success winning over IT administrators hoping to save on a few lines of code. Recently, however, their products have matured to the point where CMOs and COOs are taking notice of their immediate value to the business. This is partly due to their ever-expanding catalog of connectors that give executives confidence that their unique marketing stack is covered. However, this is also partly due to how the vendors have positioned themselves as more than just integration tools. Segment famously re-branded as a Customer Data Platform that provides advanced identity resolution and audience segmentation tools while Fivetran now supports data transformation in addition to integration. Finally, these vendors have strategically anchored their costs to scale with the business value they provide—typically by charging based on number of rows or customers processed.

Velir Expands Data-Driven Marketing Services with Segment 

The Rise of the Analytics Engineer 

In-warehouse transformation means less dependency on engineers, greater agency for analysts, and quicker time to value. 

Take a look at any industry survey on the roll-out of analytics or customer data platforms and you'll see a consistent theme among the roadblocks to execution: talent and expertise. Data-oriented skillsets are in-demand and the high salaries plus the limited pool of applicants make it exceedingly difficult for mid-market businesses to enter this arena. Fortunately, recent technological shifts have reduced the need for specialized data engineers and moved certain tasks to analysts. This is primarily due to the shift towards in-warehouse processing (and away from tools like Hadoop) which has allowed tools like DBT to flourish and form a new role: the Analytics Engineer. 

Analytics Engineers are analysts capable not only of pulling reports, but creating the data tables necessary to source those reports. Previously, only Data Engineers could run these transformations. Now, this can be accomplished in SQL, the lingua franca of any data role. At first glance, Analytics Engineers may seem as hard to find as Data Engineers. If so, how does their existence improve the standing of mid-market businesses? The difference is that Analytics Engineers can be trained up from analysts.

In other words, it's much easier to train an analyst to apply their SQL knowledge towards engineering than to find a software engineer who can apply their Python/Java skills towards data challenges. When combined with the reduced burden on IT administrators mentioned previously, what emerges is a consolidation of roles that allows a single person to do more with less. Given the right scenario, a single Analytics Engineer can serve as your cloud IT administrator, data engineer, and analyst. 

Becoming More Data-Driven 

Mid-market companies have access to sophisticated data capabilities that were unattainable just five years ago. The creation of MPP data warehouses, the introduction of serverless warehouses, the availability of turn-key integrations, and the rise of the analytics engineer role have made these competencies possible. If you’re a mid-market business you should take advantage of the tools we’ve discussed becoming more-data driven as an organization. Because the businesses that jump on the opportunity to harness their data more effectively will have a competitive advantage over those who assume, incorrectly, that these solutions are out of reach.  

Learn more about how Velir can help break down data silos with our Data Integration and Activation services. Or, reach out to find out how we can maximize your data’s impact.