Fiona, our GIS Business Manager, shares the journey our GIS team has taken to move their jobs to fully open source systems.
It’s been around two years in the making, but we are no longer reliant on ESRI (ArcGIS), ORACLE or any other proprietary software! Aligning GIS with the organisation-wide shift to open source systems, I had the privilege of completing the journey very recently when I archived the documentation for the old procedures.
As you can imagine this was not an easy job (as you may have gathered by the timeframe). So, how exactly did we get there? I’m going to share the path, procedures and benefits to the move in this blog.
First, we undertook training on QGIS, PostGIS and Python
In the past 18 months, we’ve been trained on new software to give us the basic skills needed to build on to get this mammoth task finished. The biggest jobs were the re-writes of over 8 complex Python Scripts which utilised proprietary modules, mainly arcpy. The scripts had become bloated over time. Even understanding what each script was doing was extremely hard, never mind rewriting them in a new format.
The savings going forward are significant
Whilst moving to open source has been very resource intensive, the savings are significant going forward. Most of our recurring jobs now require limited or no human input once running. This immediately saves time and reduces human errors, meaning that we don’t have to manually seek out mistakes and re-run the process as was common before
The runtime reduction for each job is also massive. The largest job – house classification – used to take up to 7 days to run, and currently on the best database setup, now only takes 3 hours.
In monetary terms, we’ll also save a third of our yearly licensing costs
It’s worth noting that we have retained a couple of desktop ArcGIS licences so we can look back on archived material, and for some functions ArcGIS is still a useful addition to our GIS toolbox.
There have been some growing pains during the re-writing process – for example, the user documentation for open source tools can be sparse compared to that for proprietary software. Some algorithms are implemented differently in the open source vs the proprietary software, which requires careful thought when continuity in reporting outputs is required.
By going through every procedure with a fine-toothed comb we have gained a greater understanding of the procedure and data used while also improving the documentation of the methodology used for each report, which makes it easier to highlight the limitations and assumptions that apply to each.
Many others have been instrumental in getting us over the finish line
It wasn’t just the GIS team – Analysts Mark, Jason, Jenny myself and Team Leader Marguerite – who have worked hard to complete this move.
We would also like to thank Alistair and Chris and their teams in the Mapping domain in supporting us with new software, access to data, help with code, and for patiently waiting for us to be fully open source before decommissioning a number of internal proprietary systems.
Marian our Database Administrator, who got our new PostGIS environment working and reminded us all off the importance of indexing when two of the team (including myself) inadvertently ran impossibly large queries on un-indexed tables which killed the database for some time.
And thanks to everyone else who supported us on the journey, helped where they could and accepted many months of double reporting as we tested our new outputs.
I’m really proud of the team – we’ve completed a very large body of work and truly reached the end of a GIS era or should that be the start of something new? Our learning continues many of our ‘new’ procedures have already been tweaked, improved or re-written again as our skills continue to improve. We have more resource to try new and more exciting things which will hopefully bring more value to RoS from the data we hold along with the stable recurring jobs we have worked so hard to improve.
We look forward to sharing more very soon!
Stay in touch
Follow us on Twitter and on Facebook, LinkedIn and YouTube for more from our teams.
Have you got any details available about the actual process you went through and what it cost? You kinda magic step through the article.
Some details of the business cases, training, production loss, outlay, data and product archiving and personnel retention challenges and lessons learned?
Moving to AGOL and/or ArcPro will break most of my companies processes, and cost us a significant amount in licensing, training and lost production. I’m seriously looking at a business case to investigate Open Source based on the training and production losses alone.
Hi Bob! Thanks a lot for your query about the blog. We’ve passed it onto Fiona and team and will get back to you as soon as we can 🙂 -Laura
Hi Bob, and thank you for your comment! We’re really happy with our move to open source systems. Whilst this has been a steep learning curve and it has been time-consuming, the improvements we’ve made means that we’d recommend a similar move to others like yourself.
We had no official business case as such. However, some of the benefits that we wanted to achieve were to reduce our licensing costs, the time taken to produce outputs, reduction in human interaction and to align with our organisation-wide strategy of using open source whenever possible.
We undertook training on Python and PostGIS via courses provided by external trainers, as well as free online and book-based learning.
Everything was documented internally and archived. Some product outputs changed. These were discussed and agreed with the business and resulted in more complete views.
I think it’s fair to say that this was a bigger piece of work than we expected. The actual re-writes and upskilling were the easy parts – finding the time to do it was difficult on top of our everyday work! Understanding the previous code was also tricky too.
We also didn’t consider the degree of parallel running that was required – on occasion, for up to a year. We hope that helps answer your queries, and we’re thinking about sharing some of the FAQs we’ve had about this blog in the next few weeks
Hello Fiona, I read with great interest your experience of moving to QGIS, I have been swithering on the cliff edge for a while but am sticking to ArcGIS for the moment. Occasionally I have to purchase GIS data from ROS and consequently get very annoyed when you expect me to provide shapefiles for large complicated properties for free. IS there any intention to level the playing field in the near future?
Hi Miles, many thanks for your interest in our move to QGIS. RoS are currently undertaking a review of our non-statutory fees, of which GIS forms part. Any changes to our current fee structure will be published through the usual channels. Best wishes, Roman