The next phase of open data transformation in the public sector

The tools of digital transformation and service design, networking becoming mainstream, and continued incentives for organisational collaboration, are combining to create stronger public sector open data services and products.

Although there might be wider gloom about the state of open data in the UK, in some ways I think we’re at a new frontier. There is a depth of maturity in the best examples of the use of open data in the public sector - not necessarily in terms of the impacts (yet) - but in terms of the approach and working practices. In addition, the financial and organisational circumstances that the public sector currently finds itself in create underlying incentives for collaboration - which is uniquely predisposed to open data approaches. Combine all of this with growing peer networks of those with deep expertise both within and without the public sector, and these are exciting times to be working in this area.

I’ve recently finished a piece of work for the Open Data Institute to better understand the use of open data in the delivery of public services. We looked at those examples in the UK where open data had been incorporated into public service delivery - from bin collection to public transport and policy development. On the basis of these examples, we set out three high-level patterns for the use of open data in public services - where open data is released to help people engage with the service, help with efficient delivery chains and help develop policy. We also began to describe the characteristics of the ecosystems where open data is used in the public sector. 

Do check out the report as there are some particularly good illustrations of the details of the services (as well as some not-so-great early drawings from yours truly). The services we found are still few and far between but they are real examples of the growing maturity and nuance in the design and delivery of open data enabled public services.

Get it out!… and then what?

Up until recently, less effort has been focused on this craft of building a public sector open data service / product. Instead, the focus has been on more arcane aspects such as charters, data standards and top 10s. When it came to the practice of making open data available, the mantra was “just get it out” and public sector organisations were more or less left to their own devices, with only some high-level guidance to support them to publish data. And the question remained “How?”. Things got too technical too quickly for a lot of people and the well-rehearsed “disagreements” about publication format standards etc were alienating, or worse, could leave people with the impression that real issues weren’t being dealt with on these projects. 

There are good examples of the setting out the knowhow of open data - School of Data, Data Journalism Handbook - but these have mostly grown outside of the public sector. This is in contrast to the practices of digital transformation developed in the public sector (although of course heavily influenced by commercial practice).

Digital Transformation x Service Design x Open Data

Many of the best examples we found in the report of using open data in the public sector have benefitted from using working practices, approaches, roles and tools which are used as a matter of course in digital transformation and service design. By this I mean practices and approaches like agile project management, ‘design thinking’, self-organising teams and open working practices. When these are incorporated into an open data project, there is more likelihood that the result will meet the initial expectations and more clearly deliver to the needs of users and public sector workers.

Many attempts to use open data in the public sector start in the wrong place - with the data. Organising work as a digital service and borrowing techniques of human centred design / user centred design helps to define the problem, helps identify any data that’s missed and also de-risks the work by building a product or service rather than producing a more academic analysis of the data.

Specifically, practices which can be adapted from “digital transformation” and service design are:

Service and product thinking

Thinking about, and designing, the provision of open data as a service or product helps bake in sustainability for the work. Who is the service for? Who is going to maintain the data? These types of questions come to the fore when you start thinking about data as a service and step away from viewing open data work as analytical projects with a defined end and a one-time analysis product. 

Iterating through Discovery, Alpha and Beta phases

In the early stages of a piece of work where data is central there’s a risk of getting lost in the detail. Public sector data isn’t the prettiest stuff - it’s not always easy to get hold off, it’s incomplete and messy - and it’s a job of work to get anything usable out of it. Focusing too much on these issues in the early stages of the project can slow down progress and sometimes bring it to a halt. Organising the project so that there is a clearly defined ‘Discovery’ phase where issues around data availability and quality are bottomed out and then moving to Alpha with a focus on building an initial response, can help keep momentum and give people some sense of clarity about the process.

Multidisciplinary teams

Just because data is involved in a public sector project doesn’t mean that the core driving force of a team needs to be a data analyst or data scientist. There are a number of other roles which can contribute to a data informed public service and strengthen the outcome. Having a lead who is familiar with product management and also having a user researcher involved (see next) helps bring other perspectives and skills. Ideally also having developers intimately involved, rather than relying on outsourced capacity or agency handoff, can help to build a data product.

User research

Bringing a user researcher into the team early on helps to counterbalance any tendency in the team to overly focus the work on the data available. Getting insight into the needs of the end user of a product also helps identify whether all the right data has been gathered and, where it isn’t yet available, insights from user research give leverage to have conversations with colleagues to persuade them to make it available.


Visualising processes, journeys and ecosystems is the bread and butter of service design. This is being increasingly used in open data projects and was a key technique we used in the ODI research. We were able to surface knowledge about data in services without getting lost in overly technical discussions and could also identify opportunities and similarities between services.

This list isn’t exhaustive and borrows heavily from the working practices of the likes of GDS, service design agencies and others. The way I think about which approaches can be used where is that “digital transformation” techniques (which to me includes agile delivery approaches and ‘lean startup’) can help build a team, give guidance on how to know where to start and provide techniques to identify what to prioritise. Human Centred Design and service design approaches help to expand thinking about the service, provide tools to help teams generate ideas (e.g. workshop approaches) and help build empathy with users. 

Bringing these elements together was something we tried to incorporate in the Open Data Challenge Series and are also evident in the best public sector open data services. There are also a number of service design agencies and public sector digital consultants who are helping develop this approach and showing the way - uscreates, Snook, FutureGov being just some. There are no doubt other techniques and approaches and some further characteristics are listed in the report. ODI Devon are also continuing the work by bringing together learning resources for those working on these services.

Making services with open data makes them better

All of these techniques could be used in delivering a data service / product of any sort, not just an open data one. Is there an added value to delivering a public service which uses open data rather than closed data? I think there is something special about an open data service as opposed to a closed data one and that the current underlying incentives in the public sector, in the longer run, will make open data run services just as likely as closed ones. The reason for this is the need for collaboration. Collaboration is what parts of the public sector really need right now as they are faced with smaller budgets and stretched resources. As a public sector organisation, releasing open data is an invitation to collaborate. An open data approach supports cross-organisational collaboration and can bring new players into delivery ecosystems.

In many cases collaboration will be better achieved through an open data approach to building a public service compared to a close one. An instinctive response for much of the public sector is to try to collaborate with a select few internal organisations, setting up data sharing agreements and protocols. This can be a painful process and again derail a project whilst everyone gets distracted and frustrated by legal and privacy considerations, and the process can inevitably slows down. Why not share data with each other by opening it up? Instead of sharing personal data about users between services, perhaps start by making available data about these services themselves, so that users are better able to use them in the first place? 

Publishing open data can be the easiest way to share data organisationally and also has the added benefit of getting that data into the hands of others who might also be able to support the same aims you’re working to. Clearly, not all public sector data can be published openly due to its sensitive nature. But starting any investigation from the position of open data publication could get faster results and help reduce privacy concerns by reducing the amount of personal data shared between organisations. Organisational collaboration is a slow, cultural change process and an open data approach to the delivery of services can be one of the many ways to build the relationships and behaviours which are needed in a modern public service.

Cliques... in a good way

Having new tools and incentives to build open data services is one thing, but it is super-charged when people can learn from others about what has worked for them and then apply to their own circumstances. Another development which is helping to build the ground for more open data powered public services is the way in which cross-organisational / grade / role networking is becoming mainstream working practice. These networks are both international and online (Twitter, Medium and the many Slack teams) to UK-based and in person (OneTeamGov, UKGovCamp and LocalGovDigital and the like). It wasn't a shock in the ODI research that those services which seemed to have successfully incorporated open data, had one or two people at the middle who were part of these networks.

Now arguably there have been networks of individuals of this sort for some time, but in the last couple of years this behaviour has been incorporated into business as usual in the public sector. The greater number and specificity of networks has also helped in the cross-fertilisation of approaches and techniques and has been a catalyst for bringing the new tools and approaches mentioned above into the arena of open data in public services. The next phase of maturity of these networks is to see how they will support in the actual design and delivery of public services - how networks of colleagues within and without government can help build the business as usual of organisational collaboration which the public sector so desperately needs.


There are still many challenges in terms of engagement and data literacy within the public sector - and a long way until the best practice found in the report becomes business as usual. However, there’s a growing community of public sector workers, practitioners and agencies working with an expanding toolbox of techniques and approaches. Combined with the organisational benefits of openness and collaboration for public sector organisations, this gives me hope that we’ll see yet more public services looking to build open data powered public services.

If you work in a public sector organisation and you’re interested in using open data to deliver a service, please do get in touch -

DataEd ParkesComment