Fiona, our GIS Business Manager, shares the journey our GIS team has taken to move their jobs to fully open source systems.
It’s been around two years in the making, but we are no longer reliant on ESRI (ArcGIS), ORACLE or any other proprietary software! Aligning GIS with the organisation-wide shift to open source systems, I had the privilege of completing the journey very recently when I archived the documentation for the old procedures.
As you can imagine this was not an easy job (as you may have gathered by the timeframe). So, how exactly did we get there? I’m going to share the path, procedures and benefits to the move in this blog.
First, we undertook training on QGIS, PostGIS and Python
In the past 18 months, we’ve been trained on new software to give us the basic skills needed to build on to get this mammoth task finished. The biggest jobs were the re-writes of over 8 complex Python Scripts which utilised proprietary modules, mainly arcpy. The scripts had become bloated over time. Even understanding what each script was doing was extremely hard, never mind rewriting them in a new format.
The savings going forward are significant
Whilst moving to open source has been very resource intensive, the savings are significant going forward. Most of our recurring jobs now require limited or no human input once running. This immediately saves time and reduces human errors, meaning that we don’t have to manually seek out mistakes and re-run the process as was common before
The runtime reduction for each job is also massive. The largest job – house classification – used to take up to 7 days to run, and currently on the best database setup, now only takes 3 hours.
In monetary terms, we’ll also save a third of our yearly licensing costs
It’s worth noting that we have retained a couple of desktop ArcGIS licences so we can look back on archived material, and for some functions ArcGIS is still a useful addition to our GIS toolbox.
There have been some growing pains during the re-writing process – for example, the user documentation for open source tools can be sparse compared to that for proprietary software. Some algorithms are implemented differently in the open source vs the proprietary software, which requires careful thought when continuity in reporting outputs is required.
By going through every procedure with a fine-toothed comb we have gained a greater understanding of the procedure and data used while also improving the documentation of the methodology used for each report, which makes it easier to highlight the limitations and assumptions that apply to each.
Many others have been instrumental in getting us over the finish line
It wasn’t just the GIS team – Analysts Mark, Jason, Jenny myself and Team Leader Marguerite – who have worked hard to complete this move.
We would also like to thank Alistair and Chris and their teams in the Mapping domain in supporting us with new software, access to data, help with code, and for patiently waiting for us to be fully open source before decommissioning a number of internal proprietary systems.
Marian our Database Administrator, who got our new PostGIS environment working and reminded us all off the importance of indexing when two of the team (including myself) inadvertently ran impossibly large queries on un-indexed tables which killed the database for some time.
And thanks to everyone else who supported us on the journey, helped where they could and accepted many months of double reporting as we tested our new outputs.
I’m really proud of the team – we’ve completed a very large body of work and truly reached the end of a GIS era or should that be the start of something new? Our learning continues many of our ‘new’ procedures have already been tweaked, improved or re-written again as our skills continue to improve. We have more resource to try new and more exciting things which will hopefully bring more value to RoS from the data we hold along with the stable recurring jobs we have worked so hard to improve.
We look forward to sharing more very soon!