Understanding data mesh in public sector: Pillars, architecture, and examples

Domain ownership

Instead of relying on a central IT team to manage all data, data ownership is distributed across government agencies and departments. Essentially, you’re building technical teams that mirror how the agency itself is composed. You want the people who are most intimately familiar with that data to own it. This can be applied to public health, defense, urban planning, and more — just about any public sector use case.

For example, the US Cybersecurity and Infrastructure Security Agency (CISA) uses a data mesh approach to gain visibility into security data from hundreds of federal agencies, while allowing each agency to retain control of its data.

Learn more about accelerating CISA Zero Trust with Elastic as a unified data layer.

This leads us to the second (and arguably most important) pillar — the one which the other three pillars are designed to support:

Data as a product

Each dataset is treated as a product with clear documentation and quality standards. The department that owns the data needs to make sure it’s easily accessible and organized for when other departments need it. In other words, they are accountable and responsible for sharing that data as a usable product.

From a government perspective, this could be census information, emergency response data, or intelligence reports, for example. It all depends on the structure of the project or government agency. What’s important is that this curated data will be ready to use when other teams come looking for it, and they won’t have to spend time cleaning or verifying it.

So, you may ask, isn’t this just another way to silo analytical data? What are the nuts and bolts of how other departments can access it? That leads us to our next pillar.

Self-service platforms

Departments are being asked to do a lot here, and they’ll need convenient platforms that make their data accessible to others. Searchable catalogs for easy data discovery, query tools for real-time analysis, and the ability for users to clean and integrate data themselves as well as share insights through dashboards and APIs are all tools that can be used.

They’ll also need built-in governance to enforce access controls, which leads us to our final pillar.

Federated computational governance

So, we’ve established that each department is in control of its own data. However, the data mesh still needs overarching governance protocols to keep it secure and prevent risk.

These security controls should be built into the system that retrieves the data, rather than applied separately by each department. The system should check user permissions as part of the search and make sure people only see the data they’re allowed to access right from the start.

In the public sector, this could be anything from privacy regulations in healthcare data to classified information in defense systems.

Data mesh vs. other approaches

How does data mesh compare to alternative forms of analytical data architecture and storage? Let’s look at two others that often draw comparisons: data fabric and data lakes.

Data mesh vs. data fabric

Data mesh and data fabric are similar approaches in that they both take a decentralized approach, collecting data at remote sites. However, a data fabric takes data collected at one site and copies it to another site. This data is shared as individual records and cannot be correlated with other records unless it gets consumed by something that makes sense of it. This approach can often lead to data silos.

A data mesh approach, on the other hand, does not rely on copying data and instead indexes data locally upon ingest into a distributed platform where users can search for data locally and across remote sites. In this model, data is unified at the search platform layer. Data is indexed once and then is available to any authorized user or use case through this unified layer.

Data mesh vs. data lake

You may have noticed that there are a lot of water-related metaphors in data: data streams, data pipelines, etc. Data, like water, can be collected, stored, filtered, and distributed — sometimes efficiently, sometimes chaotically.

In the same way that a lake collects water from multiple sources, a data lake collects data and holds it for future use. In other words, it’s a storage environment for any combination of structured, semi-structured, or unstructured data. 

Data lakes can sometimes be helpful to data mesh domain owners as they process and curate their data products. They can use a data lake for long-term storage of large, unstructured datasets (say, satellite imagery or public records) that don’t have a specific purpose yet. But if a data lake becomes disorganized and difficult to navigate, it turns into a data swamp — murky, cluttered, and hard to extract value from.

Data mesh and AI

Data mesh can offer a way to democratize AI and machine learning for public sector agencies. Traditionally, data science teams have operated as centralized hubs, pulling data from multiple sources to develop machine learning models. However, as noted earlier, this process can cause redundant work and inconsistencies, leading to challenges with model reproducibility.

By flipping that model around with data mesh and embedding AI development within domain teams, you can clean and refine data at its source and create an AI-driven data product other departments can utilize.

Take national disaster response as an example. AI models embedded in emergency response teams often analyze data like real-time satellite imagery, sensor data, and even social media reports to identify the hardest-hit areas. With data mesh, different agencies ranging from government agencies to first responders could access this information immediately without waiting for centralized processing and improve their response times as a result.

Data mesh also improves AI governance because it incorporates it right from the start, standardizing tasks like model validation, bias detection, explainability, and monitoring for model drift.

How to implement data mesh for public sector

Each public sector organization has a unique set of data needs, which is why one-size-fits-all data silos can be slow and stifling to internal and external users. Two out of three public sector leaders said that they’re unsatisfied with the data insights available to them. 

Data mesh can be customized to the unique needs of each public sector agency, from defense to national security or federal, state, and local government.

To get started with data mesh, public sector agencies will need to follow a few steps:

• Assign responsibility for data to specific departments.

• Treat datasets as well-documented, accessible assets designed for internal and external use and make sure they comply with regulatory requirements.

• Implement tools that let agencies, analysts, and policymakers easily access and analyze data without relying on centralized IT teams.

• Enforce governance across agencies, keeping in mind frameworks like FedRAMP, CMMC, and Zero Trust.

• And finally, encourage data sharing across organizations to make better decisions and improve public services while maintaining security controls.

Government and defense applications

Data mesh is a natural fit for government and defense sectors, where vast, distributed datasets have to be securely accessed and analyzed in real time.

In defense, it assists with faster intelligence gathering and asset management so operators in the field can act with the latest data. In public health, it can help rapidly integrate epidemiological data from hospitals or research labs to respond to outbreaks. Transportation departments can analyze traffic and weather data across cities. Education departments can view children’s test scores over the past decade and cross-reference them to other data, such as time spent learning remotely versus in-person.

Let’s take this example from the US Navy: Its push for digital modernization hinges on the ability to “securely move any information from anywhere to anywhere” to achieve information superiority. But traditional centralized data storage is too risky, especially in air-gapped and Denied, Degraded, Intermittent, and Limited (DDIL) environments. Here’s a case where a global data mesh can help, allowing data to remain at its source while still being searchable and accessible across the Navy’s vast operational landscape. This decentralized approach keeps ops resilient even if a server or data center fails and provides a unified view of mission-critical data without needing to move or duplicate it.

Data mesh in action with Elastic

As the Search AI Company, Elastic’s data analytics platform serves as a powerful global data mesh, offering machine learning, natural language processing, semantic search, alerting, and visualization in a unified system. In other words, Elastic serves a unifying function by giving agencies full visibility into their data as well as the ability to ingest, organize, access, and analyze it. 

Three key features set Elastic apart: 

• Cross-cluster search (CCS), which lets you run a single search request against one or more remote clusters

• Searchable snapshots, which provide a cost-effective way for you to access and query infrequently used, historical data

• Role-based access control, which provides integrated security 

Elastic’s data mesh approach also can serve as a foundation for modern security frameworks like Zero Trust and opens up new possibilities for data-driven operations.

Learn more about how Elastic helps government, healthcare, and education teams maximize data value with speed, scale, and relevance.