Data management in a digital world

By Sherrine Green-Thompson, SmarterData Area Lead at Obsidian Systems

Data management plans (DMPs) are evolving as the technology that enables organisations to better analyse their data become more powerful and innovative. But what impact does this have on operations and the efficiency of managing the complexities of data in a changing regulatory environment?

DataOps, an automated methodology to improve the quality and reduce the cycle time of data analytics, has emerged as a powerful enabler in this regard. This seeks to make the transition from idea to analytic visualisation or presentation faster and provide decision-makers with more lean data development process features.

But despite the automation it provides, DataOps still requires a combination of people, processes, and technology to enable data management solutions to become more agile. It is reliant on a collaborative approach between teams to define the data management strategy and ensure it is integrated across the business. Replacing the silo approach of old with a culture that encourages communication across these teams is vital.

This will also lead to various process changes and enhancements. As defined by the DMP, these ‘new’ processes will make it clear how data will be delivered with a faster time to value.

Making it good

Underpinning this is the need to ensure good data management especially in the hybrid environment adopted by many in a cloud world. Part of this entails understanding how the data is structured in the organisation and the use cases for data management across the business.

And just because data and workloads are transitioning to the cloud, it does not mean the best practices of data management will disappear. Instead, these become more important in guiding how the organisation will manage the data and use it across its various environments. To this end, the DMP should outline an environment-agnostic way in which data is collected, stored, used, and trusted. This must also be auditable, secure, governed, and fit for consumption by the various stakeholders.

Data federation and metadata help drive this management and is a requirement for aspects such as self-service data access, preparation, visualisation, and analytics. Without it, non-technical users will not be easily able to work with data independently from subject matter specialists. This will also mitigate the data redundancy issue where all teams and stakeholders are now able to leverage a single source of truth.

Data management future

Some of the future innovations to keep in mind when it comes to DMP include more advanced data management technologies that will see roles and jobs inside the organisation change as well. After all, data is no longer the exclusive domain of the IT department. It has become a shared collaboration between users across the organisation.

Things such as DaaS (data as a service) will deliver an improved and prioritised focus on privacy, security, and data governance. It will enable the data to be easily moved from one platform to another, reduce redundancy, and provide for easier administration. It then becomes a case of users simply subscribing to or pulling data streams as and when the need arises.

Another technology to keep in mind is that of augmented data management. This converts metadata from being used for audit, lineage, and reporting only, to powering dynamic systems. It will see metadata changing from being passive to active and playing a key role in driving artificial intelligence and machine learning adoption in the business.

The increased collaboration and use of DataOps will result in continuous intelligence for the organisation. This is a design pattern in which real-time analytics are integrated within a business operation, processing current and historical data to prescribe actions in response to events. It provides decision automation or decision support. Continuous intelligence leverages multiple technologies such as augmented analytics, event stream processing, optimisation, business rule management, and machine-learning.

DMP are here to stay. However, the nature of how the organisation is developing and approach them is changing, resulting in a more dynamic and agile environment.