DEF Software underlines its Data Transfer capability with recent Northern Ireland Planning Solution Go live

On December 5th 2022 ten of the eleven Local Authorities in Northern Ireland went live with a new planning solution, not only did this replace their legacy system it also introduced the ability to apply online for planning and regional property certificates (Land Charges) across the country.

The Problem

As with many major projects of this size and complexity, the challenge of transferring the data out of the legacy system into the new solution is always a major hurdle to overcome. In this case the legacy system held almost 1,000,000 planning applications within its database spanning the previous 20 years of operation. To further complicate this problem, during 2015 Northern Ireland transformed it’s planning function moving responsibility for the planning function away from central government and putting it in the hands of the local authorities. This lead to differences in the way that the data needed to be held prior to 2015 and subsequently, all this needed to be catered for in the data migration.

The Approach

Early in the project, Data Migration was highlighted as one of the major risks to the project success and a Data Transfer Team was formed to quite simply map the data, field by field from the incumbent system into the new DEF Software MasterGov solution. Each field was examined in turn and data mapping produced which documented both the source for each date item and the target field including any transformation rules that were required on this data. Although this was a very time-consuming exercise it would prove to be a good use of time and lead to high success rates in terms of data quality.

Quality Assurance

When attempting to transfer approaching 1,000,000 records from one system to another, it would be impractical to check every record and be far to time consuming. The project team decided upon a sample check using “control records” which were extreme edge cases in the existing database as use those to form a benchmark for data quality. The number of Control Records used in transferring the data was representative of both the volume of data and the level of risk associated with each data set.

Iterative Improvements

Once DEF Software had configured their Data Migration Tool to take account of the data mappings created earlier in the process the initial data transfer process began. On each occasion that the data transfer was ran quality assurance checks were executed and any issues raised. Subsequently the data migration tool was incrementally improved to resolve the issues raised. In total this project required four iterations of data transfer to reach the point where the migration was successful and passed the quality assurance checks that had been put in place.

Speed of Execution

The preparation of the data mapping proved beneficial in that when it came to the live execution of the data transfer the textual data was copied, transformed and written to the DEF Software MasterGov solution in slightly under 3 hours. This time is even more impressive when you take into account that there were over 254 million records read and 41 million records written to the new solution during this process. In addition to the textual data an additional 5.5 million documents and over 1 million spatial features were also transferred into the MasterGov solution.

In Summary

The key lessons learned from this undertaking is that the preparation of the data mapping document, although time-consuming at the time, proved to have been a worthwhile exercise in achieving the accurate a speedy results. In addition DEF Software’s expertise in Data Transfer and their highly-performant Data Migration Tools turned data transfer from being the most problematic area of the project to one that ran smoothly.