OSM: Going Back in Time

I’ve been playing around with the full planet file to look at going back in time in OSM. Mainly, this is to look at how Ramani Huria’s data has evolved over time and is all part of extracting more value from Ramani Huria’s data.

I’ve been playing around with the full planet file to look at going back in time in OSM. Mainly, this is to look at how Ramani Huria’s data has evolved over time and is all part of extracting more value from Ramani Huria’s data. This process wasn’t as straightforward as I had hoped, but eventually got there – also, this isn’t to say that this is the only or best way. It’s the one that worked for me!

To do this, you’ll need a pretty hefty machine – I’ve used a Lenovo x230 Intel i5 quad core 2.6ghz, 16gb of ram with over 500gb of free space – This is to deal with the large size of the files that you’ll be downloading. This is all running on Ubuntu 16.04.

Firstly, download the OSM Full History file. I used the uGet download manager to deal with the 10 hour download of a 60gb+ file over 10meg UK broadband connection. Leaving it overnight, I had a full file downloaded and ready for use. Now to set up the machine environment.

The stack is a combination of OSMIUM and OSMconvert. On paper, the OSMIUM tool should be the only tool needed. However, for reasons that I’ll come to, it didn’t work, so I found a workaround.

OSMconvert is easily installed:

sudo apt-get install osmctools

This installs OSMconvert other useful OSM manipulation tools. Installing OSMIUM is slightly more complicated and needs to be done through compiling by source.

Firstly, install LibOSMIUM – I found not installing the header files meant that compilation of OSMIUM proper would fail. Then use the OSMIUM docs to install OSMIUM. While there is a package included in Ubuntu for OSMIUM, it’s of a previous version which doesn’t allow the splitting of data by a timeframe. Now things should be set up and ready for pulling data out.

Dar es Salaam being the city of interest, has the bounding box (38.9813,-7.2,39.65,-6.45) – you’d replace these with the South West, North West point coordinates of your place of interest, and use OSMconvert, in the form:

$ osmcovert history_filename bounding_box o=output_filename

osmconvert history-170206.osm.pbf -b=38.9813,-7.2,39.65,-6.45 -o=clipped_dar_history-170206.pbf

This clips the full history file to that bounding box. It will take a bit of time. Now we can use OSMIUM to pull out the data from a date of our choice in the form:

$ osmium time-filter clipped_history_filename timestamp -o output_filename

osmium time-filter clipped_dar_history-170206.pbf 2011-09-06T00:00:00Z -o clipped_dar_history-170206-06092011.pbf 

This gives a nicely formatted .pbf file that can be used in QGIS (drag and drop), POSTGIS or anything else. As the contrast below illuminates!

tandale_01082011
Tandale, Dar es Salaam, Tanzania – 1st August 2011
tandale_2017_lowres
Tandale, Dar es Salaam, Tanzania – 13th February 2017

Enjoy travelling back in time!

All map data © OpenStreetMap contributors.

FOSS4G 2018: Dar es Salaam

This will be a different FOSS4G. As the first in a developing country, our vision is for an accessible and inclusive FOSS4G, a FOSS4G for All.

In 2013 I was fortunate to be on the Local Organizing Committee (LOC) of OSGEO’s Free and Open Source For Geospatial Annual Conference – aka FOSS4G. Each year, the location of FOSS4G is decided on the basis of Europe, North America and “Rest of the World”.

As Dar es Salaam, Tanzania is in the “Rest of the World”, when the call for proposals was released, myself and a small group of co-conspirators figured that we’d submit a letter of intent for Dar to be the host city. Ultimately, Dar es Salaam was selected by the OSGeo Board to be host city for 2018, co-chaired by myself and Msilikale Msilanga.

This will be a different FOSS4G. As the first in a developing country, our vision is for an accessible and inclusive FOSS4G, a FOSS4G for All. 

In aligning FOSS4G 2018 within the Sustainable Development Agenda, we aim to complement the existing OSGeo stable while empowering the Open Data, Participatory Mapping and Internet of Things work currently underway within Tanzania, the region and worldwide. This will expand the scope of the conference from traditional geography and location tech, to applications, use and best practices of our tools.

Location and geography are at the heart of the Sustainable Development Goals: FOSS4G 2018 in Dar es Salaam would invigorate our existing projects, bringing them to new users and developers while supporting and nurturing the existing OSGeo community.

The clock has started for the LOC to organise the conference and the press release from OSGeo is out! It’s going to be a few months till FOSS4G 2017 in Boston till Msilikale, our LOC and I will pick up the keys to the FOSS4G car. In the meantime the Dar LOC will assemble and start laying the groundwork for 2018. It’s going to be a long and rewarding road ahead!