Migrating Messy Data
Wednesday, February 12, 2014.
Posted by Carl Ware
Data is an increasingly critical resource for public safety agencies. And, while some agencies are working with contemporary data systems which allow them to access data quickly and efficiently, there are still many other agencies which are struggling with data which may be spread across several dissimilar systems and databases.
Let’s say that your agency has key public safety records data going back 20 years in four different systems. One system is a relatively new records system with a relational database behind it. The data in this system has a built-in search function and names, vehicles, and addresses are all in master indices and are properly related to other records throughout the system. Unfortunately, you only have five years’ worth of data in this records system.
In addition, the new records system didn’t have the forms which would allow you to capture everything you needed, so you have a bunch of other records in Excel files and an Access database which are loosely linked together. Records older than five years are stored in an AS400 system which you can’t turn off because you are always going back to that system to find information for reopened cases and to bring forward history for repeat offenders.
Finally, you have public safety information in yet another system which was never intended to manage such data. Since you had a license for it and someone knew how to set up new forms and documents, you went ahead and used it for public safety data that didn’t really fit in any of your other systems. The biggest drawback for this system is that absolutely nothing is related to anything else, making it little more than a file system which happens to live in a non-relational database.
Taking our scenario one step further, let’s say that the time has come to move to a new records system but you would rather not keep all of the legacy systems going indefinitely (like you did last time you upgraded to a new system). Unfortunately, you’ve done some checking and people are telling you that trying to convert data from all of those old systems is messy, costly, and far from headache free. They recommend that you just live with the old systems, do a cutover to a new system, and keep on going.
Before you go down that road, there is another option. After all, data is data, and even data which is stored in less-than-optimal systems (and lacks the data relationships which are found in contemporary systems) can be converted into a much more accessible system. Yes, the process is not always straightforward, and often requires as much art as science. Yes, there is a cost associated with data conversion, though we are able to perform conversions for less than you might imagine. And, yes, there are practical logistical challenges with taking loosely structured data and placing it into a highly structured context. Despite all this, we believe the benefits outweigh the drawbacks, particularly when it comes to the long-term savings in time and effort.
Zuercher Technologies has migrated data from a wide variety of systems, including all of the types mentioned in the above scenario. Other vendors who offer data conversion will often contract with a third-party instead of doing it in-house—but that is no how things work here at Zuercher Technologies. We have staff who are dedicated to performing such data conversions and who are well versed in a variety of relational and non-relational databases and systems. These staff members will oversee the data conversion process from the first review all the way through go-live and final acceptance. People in your agency who are very knowledgeable with regard to both how that data is stored in your current system(s) and how that data is used by your agency provide the rest of the knowledge necessary to make the data conversion process go smoothly.
With experts driving the data conversion process from both our end and your end, there is no reason that the data from several older, less-than-optimal systems cannot be brought forward into Zuercher Suite. Once you’ve done this, you can turn off those old systems and forget the several different menus, tools, and commands which you had to remember and use. All of your information is now in one application, stored in one database, and accessible via one user interface.