OSM: Going Back in Time

I’ve been playing around with the full planet file to look at going back in time in OSM. Mainly, this is to look at how Ramani Huria’s data has evolved over time and is all part of extracting more value from Ramani Huria’s data.

I’ve been playing around with the full planet file to look at going back in time in OSM. Mainly, this is to look at how Ramani Huria’s data has evolved over time and is all part of extracting more value from Ramani Huria’s data. This process wasn’t as straightforward as I had hoped, but eventually got there – also, this isn’t to say that this is the only or best way. It’s the one that worked for me!

To do this, you’ll need a pretty hefty machine – I’ve used a Lenovo x230 Intel i5 quad core 2.6ghz, 16gb of ram with over 500gb of free space – This is to deal with the large size of the files that you’ll be downloading. This is all running on Ubuntu 16.04.

Firstly, download the OSM Full History file. I used the uGet download manager to deal with the 10 hour download of a 60gb+ file over 10meg UK broadband connection. Leaving it overnight, I had a full file downloaded and ready for use. Now to set up the machine environment.

The stack is a combination of OSMIUM and OSMconvert. On paper, the OSMIUM tool should be the only tool needed. However, for reasons that I’ll come to, it didn’t work, so I found a workaround.

OSMconvert is easily installed:

sudo apt-get install osmctools

This installs OSMconvert other useful OSM manipulation tools. Installing OSMIUM is slightly more complicated and needs to be done through compiling by source.

Firstly, install LibOSMIUM – I found not installing the header files meant that compilation of OSMIUM proper would fail. Then use the OSMIUM docs to install OSMIUM. While there is a package included in Ubuntu for OSMIUM, it’s of a previous version which doesn’t allow the splitting of data by a timeframe. Now things should be set up and ready for pulling data out.

Dar es Salaam being the city of interest, has the bounding box (38.9813,-7.2,39.65,-6.45) – you’d replace these with the South West, North West point coordinates of your place of interest, and use OSMconvert, in the form:

$ osmcovert history_filename bounding_box o=output_filename

osmconvert history-170206.osm.pbf -b=38.9813,-7.2,39.65,-6.45 -o=clipped_dar_history-170206.pbf

This clips the full history file to that bounding box. It will take a bit of time. Now we can use OSMIUM to pull out the data from a date of our choice in the form:

$ osmium time-filter clipped_history_filename timestamp -o output_filename

osmium time-filter clipped_dar_history-170206.pbf 2011-09-06T00:00:00Z -o clipped_dar_history-170206-06092011.pbf 

This gives a nicely formatted .pbf file that can be used in QGIS (drag and drop), POSTGIS or anything else. As the contrast below illuminates!

Tandale, Dar es Salaam, Tanzania – 1st August 2011
Tandale, Dar es Salaam, Tanzania – 13th February 2017

Enjoy travelling back in time!

All map data © OpenStreetMap contributors.

Data Driven Governance

At the Africa Open Data Conference I was fortunate to have a chat with Juliana Letara, the Town Planner for Kinondoni Municipality and Osiligi Lossai, the Ward Executive Officer of Tandale. We discussed the recent community mapping, how they are beginning to use the maps and the data and some unexpected outcomes.

A longer blog will be incoming on how data is being used to support governance in Dar es Salaam. Personally, I’m still digesting some of the implications and potential of what they’re discussing, regarding the potential impact that Ramani Huria (also here http://ramanihuria.org) could have and is having in Dar es Salaam.

Teaching Git at Mobile Monday

Mobile Monday Presentation

After the Sanitation Hackathon one of the key lessons learnt was that the coders had scant knowledge of source code repositories. Learning how to checkout, commit and merge are skills that weren’t covered in detail during my own time at university, but were very valuable once hitting the world. In Dar code was shared through USB pens, version control was through separate folders, if at all. Kinu and TanzICT (two great technology incubators) invited me to stick on a developer hat and do a code repository Mobile Monday workshop – MoMo is about fostering cooperation and innovation between developers globally.

Github in my opinion is the best repository on the internet because of the awesome tools, cost for open source projects (free!) and using Git as the entry point. Also the documentation, is thorough with step by step guides taking you from being a novice to a git ninja.

The workshop was divided into two sections with a break in the middle. In the first section a straw poll was conducted to get an idea of the operating systems in the room. It was about 2/3 Windows to 1/3 Linux. This was followed by an introduction to source code management and a conceptual overview of the process of committing and pulling code. Everyone started to download the appropriate software to their computer and install the .NET framework! Break time.

Coming back after learning some new things about the attendees (like ice cream and favourite drinks) we started to go through the git lifecycle. We followed the help.github.com notes to setup and created a repository called “Kinutest” to get social and collaborate. After starting with the basics of committing and pulling, collaboratively code started to get generated. Issues were encountered with merging branches. This kicked off a discussion on merging using Nvie’s model for development. After 3 hours of frantic work and many learning experiences we ended up with some collaborately written README.md files and two PHP files!

Obviously this is only the start of the process. Going really deep into Git is something that can’t be covered in 3-4 hour session, it’s always a constantly evolving and learning process. The crowd were excellent, however at times they would just use their exuberance and charge on ahead. This in itself isn’t a bad thing, however can be challenging on trying to keep the entire group together on a task. But hopefully provides an introduction into code repositories. Did someone mention unit testing…?

Written and submitted from KINU Innovation Space, Dar Es Salaam (-6.77802,39.26721)

Enter The Sanitation Judges

Live Coding Demonstration for Judges
Live Coding Demonstration for Judges

The format for the judging was adjusted slightly from the traditional ‘present to everyone’ approach. We set up in away from the main hackspace, where presentation groups presented to the judging panel composed of luminaries from the technology and sanitation sectors. In no particular order;

Jon Gore is the managing director of E-Fulusi a boutique mobile and software development company working in Dar Es Salaam.

Elias Chinamo is an assistant director of the Ministry of Health and Social Welfare.

Nathan Were is a Business Development Manager of the DBTi technology incubator at the Tanzanian Commission of Science and Technology.

Jason Cardosi is the country lead for the Water and Sanitation Program.

Jones Mrusha is one of the founders of the KINU innovation hub.

Gary Gale is the Director of Places at Nokia.

More to follow on the presentation of the hacks and results!

Written and posted from the Sanitation Hackathon, COSTECH, Dar es Salaam, Tanzania (-6.77457, 39.24125)

A Framework and Kit List For Community Mapping


Getting the right equipment and software is essential to the success of any undertaking. It is no different for the Tandale project. So far we have…

  • 10 Garmin Legend HCx GPS’
  • 5 Acer netbooks and cases
  • 2 Nikon L24 point and shoot cameras + cases
  • 2 Flip HD Video cameras + cases
  • 3 battery chargers
  • 72 Batteries
  • 5 mice
  • 2 Hard briefcases
  • 1 Large suitcase
  • Pens and paper
For software we use…
We have 18 community mappers and upto 25 university students (who will also have to write a report on how the mapping occurs and its impact). These will be split into 6 groups (one for each subward) containing 6-7 group members.  Each team member will become a specialist in a chosen area. The areas for speciailisation are surveying, editing, satellite image tracing and story telling.

Surveyors are the groups of people who go into the field with GPS’ and map the roads, streets and POIs of Tandale. Using a combination of GPS’, pens and paper they will be responsible for gathering data and supplying it to the editors.

Editors take information from the surveyors and using GPS Babel import data from the GPS’. This will then be loaded into JOSM, transformed by adding metadata like names, amenity type etc. to the already georeferenced object. Finally the new changeset will be uploaded to the OSM database. For a greater understanding of the data that can be attached to the nodes and ways (OSM’s data structure) please look at the OSM’s Tags wiki page.

Satellite image tracers will trace buildings and other natural features like rivers and submit them to the OSM database. They will work in conjunction with surveyors and editors to add context to the tracings.

Storytellers will engage with the community and will provide reflection and reporting of current events through the blogs and Ushahidi. As well as communicating the issues faced by the community of Tandale. Because of the short amount of time that the project has (1 Month) any videos that are recorded will be edited offsite, however we hope that in time videos will be created in much the same way as Kibera News Network and Mathare In Motion. However photos and writing will be submitted to the Flickr account or OSM diaries where appropriate.

Our first week will be about getting the teams to gel. We hope that the students (most of whom have some experience in surveying) will act as the go-to guys for the community members whereas we act more as facilitators and tutors.

The media strategy will be dealt with carefully during the project. Small parts will be available for local press; community newspapers, radio etc. International and national press involvement needs to be carefully managed. Involving and advertising the project is important, especially on a political level for other areas within Tanzania; we want other city councils to emulate our model. The key message is that this is a community project not one administered/run/by and for outside parties. This message and reality could be lost in translation when projecting to a bigger audience.

All of this considered, our first week begins next Tuesday (09/08/2011). We will start with a community forum, where the main concerns and needs of the community will be identified. On the Wednesday the community and students join into their teams and be given a sub-ward with which they will start to learn the mapping process. This data will then be assimilated and the editing process will start to be shown on Thursday. While this is happening Simon will facilitate the storytelling process, aiding the community in telling their stories. On Friday it’ll be more of the same; mapping streets and points of interest.

In the days and weeks after the first week, while streets and points of interest are completed, building outlines will be traced using Bing imagery and landuse identified, with the team specialists taking control of their domains.

Written and submitted in the World Bank Offices, Dar Es Salaam, Tanzania (-6.81298, 39.29194)