Author: Chris Brown
Author: Chris Brown
As part of a new role I’ve been looking at the composition of Data and Analytics teams within a broader organisational context. I’ve come across two ways that data teams have been formed and are operating but both ways are broken. By 'broken' I mean they are providing sub-optimal outcomes for the parent organisation.
The data team model for the company I’m working with is a set of business franchise-aligned analytics teams that draw on help from a shared pool of data engineers and architects. The analytics teams create models, conduct research and build visualisations and reports. They only do light data engineering, with anything too complex being passed onto the central data engineering team’s backlog.
Recently I encountered another organisational set up that doesn’t have any identifiable data teams in the traditional sense. Product feature teams are expected to take on all the aspects of data analytics, reporting, engineering and management. This philosophy extends all the way to the C-suite, that saw no need for senior representation of data on the Exec team.
It’s probably no surprise that neither organisational model is working too well.
The first company suffers because of the competition for the finite data engineering resources. The analytics teams are frustrated when they can’t get engineering resources and don’t understand the technicalities and complexities of the underpinning data systems. The Data Engineering department is too far removed from the business areas to understand the purpose of their work. To please everyone the engineers, are frequently switching priorities. The analytics teams resort to develop isolated, custom solutions. It’s easier to shift data into pseudo-production databases (called 'sandpits') than it is to wait for engineering to build pipelines into golden official sources. This is storing up all kinds of problems.
The second company uses a ‘feature team’ approach and are facing a problem because feature teams are incentivised to add or improve features to their product. When it comes to the crunch, no feature team will prioritise improvements to the underlying data ecosystem over adding another whizzy feature. They certainly won’t be putting effort into really important data governance aspects such as quality, regulation, privacy and ethics, content, and lineage. Couple this with a lack of representation at a senior level to communicate vision and purpose and you have the problem that any data concerns other than those relating to features will fall by the wayside.
With no identifiable data platform (there’s no dedicated team to build it), data becomes an operational concern rather than a research and analysis asset. This 'side-of-the-desk' approach to data feels like completely the wrong direction in a world where companies are becoming data-driven and are treating data and its people as a first class asset.
The solution lies somewhere in between the two models I’ve observed.
Take model one. Here the central engineering effort needs to be broken up and federated into the business-aligned analytics teams. This leaves a core engineering team who deal with the really gnarly problems of large scale data processing.
The second model’s effort to get the analytics people much closer to their feature counterparts makes sense. So here we need to build close ties between business stakeholders and the multi-disciplinary data teams but also ensure that a coherent effort is made to engineer the underlying data platform and its concerns. This will require some engineers to be earmarked as data engineers.
The result of fixing both models should be a blended team of data specialists covering the spectrum of data engineering, through analytics/science to quality and governance, closely align to the products the parent is developing.
Team members identify strongly with the business data. They care as much about the commonwealth of the data platform as they do about the models and features they are building for the business.
Oh, and also give a top job to someone who really gets data. If you want your company to truly become data-driven then you will need them.