OSM: Going Back in Time

I’ve been playing around with the full planet file to look at going back in time in OSM. Mainly, this is to look at how Ramani Huria’s data has evolved over time and is all part of extracting more value from Ramani Huria’s data.

I’ve been playing around with the full planet file to look at going back in time in OSM. Mainly, this is to look at how Ramani Huria’s data has evolved over time and is all part of extracting more value from Ramani Huria’s data. This process wasn’t as straightforward as I had hoped, but eventually got there – also, this isn’t to say that this is the only or best way. It’s the one that worked for me!

To do this, you’ll need a pretty hefty machine – I’ve used a Lenovo x230 Intel i5 quad core 2.6ghz, 16gb of ram with over 500gb of free space – This is to deal with the large size of the files that you’ll be downloading. This is all running on Ubuntu 16.04.

Firstly, download the OSM Full History file. I used the uGet download manager to deal with the 10 hour download of a 60gb+ file over 10meg UK broadband connection. Leaving it overnight, I had a full file downloaded and ready for use. Now to set up the machine environment.

The stack is a combination of OSMIUM and OSMconvert. On paper, the OSMIUM tool should be the only tool needed. However, for reasons that I’ll come to, it didn’t work, so I found a workaround.

OSMconvert is easily installed:

sudo apt-get install osmctools

This installs OSMconvert other useful OSM manipulation tools. Installing OSMIUM is slightly more complicated and needs to be done through compiling by source.

Firstly, install LibOSMIUM – I found not installing the header files meant that compilation of OSMIUM proper would fail. Then use the OSMIUM docs to install OSMIUM. While there is a package included in Ubuntu for OSMIUM, it’s of a previous version which doesn’t allow the splitting of data by a timeframe. Now things should be set up and ready for pulling data out.

Dar es Salaam being the city of interest, has the bounding box (38.9813,-7.2,39.65,-6.45) – you’d replace these with the South West, North West point coordinates of your place of interest, and use OSMconvert, in the form:

$ osmcovert history_filename bounding_box o=output_filename

osmconvert history-170206.osm.pbf -b=38.9813,-7.2,39.65,-6.45 -o=clipped_dar_history-170206.pbf

This clips the full history file to that bounding box. It will take a bit of time. Now we can use OSMIUM to pull out the data from a date of our choice in the form:

$ osmium time-filter clipped_history_filename timestamp -o output_filename

osmium time-filter clipped_dar_history-170206.pbf 2011-09-06T00:00:00Z -o clipped_dar_history-170206-06092011.pbf 

This gives a nicely formatted .pbf file that can be used in QGIS (drag and drop), POSTGIS or anything else. As the contrast below illuminates!

Tandale, Dar es Salaam, Tanzania – 1st August 2011
Tandale, Dar es Salaam, Tanzania – 13th February 2017

Enjoy travelling back in time!

All map data © OpenStreetMap contributors.

Starting Ramani Huria – Mapping The Flood Prone Areas In Dar es Salaam

Four years ago, in August 2011 I was fortunate to manage the community mapping of Tandale. It was an experience that irrevocably changed my professional direction and interests. Over a month I trained and worked alongside brilliant students and community members, who were all focused on getting an open map of Tandale, something that had never been accomplished previously. When it was done, the reception across civil society and government was positive and intentions on scaling the pilot to the city were mooted but for one reason or another it never quite made it. Then in December, floods hit the city. In dense informal urban environments such as Tandale these floods are fatal and dramatically change the landscape as well as causing mass damage to survivor’s livelihoods and assets. Mitigating these floods are hard – where do you start in the fastest growing city in Africa? The population as of the 2012 census currently stands of 5 million, with projections showing it could grow to 10 million by 2030.

This rapid and unplanned urbanisation is in part the cause of flooding: the infrastructure with which to cope with high rainfall, such as drains and culverts, were not built alongside residential dwellings. This is especially acute in the unplanned, informal urban settlements where a majority of Dar es Salaam’s residents reside. The theory here is quite simple: If that if you can identify where it floods, you can either install or upgrade infrastructure to ameliorate the situation for residents. Unpacking this, the crux of the issue falls to two main points, governance and data.

Ramani Huria – Swahili for “Open Mapping” – is a operationalization of this theory of change. In March 2015, a coalition from across Tanzanian society, composed of the City Council of Dar es Salaam, the Tanzanian Commission for Science and Technology (COSTECH – under the Ministry of Science, Communication and Technology), the University of Dar es Salaam, Ardhi University, Buni Innovation Hub supported by the Red Cross and World Bank supported the inception of Ramani Huria, with the goal of mapping flood prone areas in Dar es Salaam, making this data openly available and supporting the use of this data into government where decisions can be made to mitigate flooding.

Mapping Phases
Mapping Phases

It is a far cry from 2011 where just mapping the ward of Tandale was a large task. Ramani Huria consists of a pilot phase and four subsequent phases. To pilot, the wards Ndugumbi, Tandale and Mchikichini, with a combined population of over 100,000 residents were mapped in series. This process combined 15 students matched with community members, leading to maps of all features within that community. This information, focusing on drainage and water ways, is critically needed to help understand and locate flood prone areas; this is high priority in Dar es Salaam due to the damage that annual floods wreak upon the city and its residents. In this piloting phase, conducted from March to the end of June these three wards were mapped, in part to generate the data that will generate flood inundation models and exposure layers but also to pilot the data model and gel the team, prior to Phase One.

Scale Up Workshop
Scale Up Workshop – https://www.facebook.com/ramanihuria

Phase one on paper is quite simple. Take 150 students from the University of Dar es Salaam’s Department of Geography and Ardhi University’s School of Urban and Regional Planning on industrial training, hold an inception workshop, deploy this contingent across six wards and work with community members to replicate the pilots, but running in parallel. At the time of writing, mapping is ongoing in six communities: Msasani, Keko, Makumbusho, Mabibo, Makurumla and Mburahati. According to the 2012 NBS census, these wards have a combined population of over 280,000 residents. Phase one was kicked off on the 6th of July and will run until the 14th of August.

Field Survey - https://www.facebook.com/ramanihuria
Field Survey – https://www.facebook.com/ramanihuria

Phases Two and Three, will integrate community volunteers from the Red Cross, these volunteers are committed to creating community level resilience plans. These plans will use the data produced by the mapping to create resident evacuation routes and aid Ward Exective Officers with planning decisions among many other uses. Additionally, with embedded long term volunteers monitoring change in their wards, this will hopefully result in detailed up-to-date maps in rapidly changing urban areas.

InaSAFE Training - https://www.facebook.com/ramanihuria
InaSAFE Training – https://www.facebook.com/ramanihuria

Phase Four unfortunately sees the students depart from the project, due to their graduation. With a remaining contingent of around 30 mappers, mapping will continue until February 2016. These phases cover the data component, consequently alongside these phases are dedicated training events aimed at building capacity to use and deploy this data in real world situations. On the 20th July the first such workshop series took place, with representatives from the Prime Minister’s Office for Disaster Management Department being trained in spatial analysis in QGIS and risk modelling using the QGIS plugin InaSAFE. A series of these workshops will take place, placing the data into the hands of those responsible for the city.

While this is ongoing in Dar es Saalam, you could get involved wherever you are in the world, through the Missing Maps project. Missing Maps is a collaboration between the Red Cross, Doctors Without Borders and Humanitarian OpenStreetMap Team, aimed at digitising “the most vulnerable places in the developing world”, but primarily do so by crowdsourcing the digitisation of aerial imagery. At the moment, there are three tasks for Dar es Salaam:

By helping digitise the buildings and roads, using the recent drone and aerial imagery, the process of mapping is faster, allowing the community mappers to focus on the detail of flood data. Additionally, the data from Ramani Huria is all placed into OpenStreetMap, its code is on Github and content available from Flickr and Facebook, all with an open licence. Please get involved!


Written on a plane somewhere between Tanzania and the United Kingdom


On the 3rd to the  5th of April I attended GISRUK (Geospatial Information Research in the United Kingdom) to give a paper on Community Mapping as a Socio-Technical Work Domain. In keeping with Christoph Kinkeldey‘s love of 1990s pop stars Vanilla Ice made a second slide appearance, leveraging the fact it’s a very technical academic title. In short I’m using Cognitive Work Analysis (CWA) to create a structural framework to assess the quality (currently defined by ISO 19113:Geographic Quality Principles – well worth a read…) where there is no comparative dataset.

CWA is used to assess the design space in which a system exists, not the system itself. In taking a holistic view and not enforcing constraints on the system you can understand what components and physical objects you would need to achieve the values of the system and vice-versa. In future iterations I’m going to get past first base and look at decision trees and strategic trees to work out how to establish the quality of volunteered geographic data without a comparative dataset. Building quality analysis into day one, as opposed to being an after thought.

Written and submitted from Home (52.962339,-1.173566)


H4D2 April 12th – 14th

The HXL-Team
The HXL-Team

Last year I attended the H4D2 (Humanitarian for Disaster 2.o) organised by (and at) Aston University and Geeks Without Bounds. One of the outputs that I worked on was the HXL Extractor. Basically take data out of  GeoSPARQL, a geospatial semantic database and fire it into a GIS program. One of the team members had already been experimenting with and semantic databases and triplestores (this was most definitely a good thing, allowing us to move quickly) so our ‘mission’ was to create a middle layer to connect to a triplestore, then using the WFS-T standard to fire the extracted data into a GIS program of your choice. Interestingly the ‘project lead’ was communicating with us from Geneva via Skype, this and the prior work bellies the need for clear and concise problem statements prior to the hack. Because some of the team had been able to think about what they had to do we’d been able to work more effectively, even while learning technologies on the fly.

Going to the International Conference for Crisis Mapping Hackathon in Washington a few months later, HXL was still going strong and I got to meet the instigator of the project CJ Hendrix face to face. He’d amassed a team which went on to rightly take first prize at ICCM, now its being used by by UNOCHA with papers forthcoming. The project is growing, as evidenced by the amount of work going on in the team repository. Understandably our small team in Birmingham just did a little bit, but every little bit, helps.

Now H4D2 is coming around again on April 12th – 14th. This will then be followed up by SMERST (Social Media and Semantic Technologies in Emergency Response) a more academic focused conference on April 15th – 16th. Most importantly, you didn’t need to code to contribute, all are welcome from designers, videographers, bloggers, journalists and you! Registration for the H4D2 is open and is again at Aston University in Birmingham. Register here: http://h4d2.eu/registration. It’s going to rock.

Written and submitted from the Serena, Dar Es Salaam (6.810617, 39.288284)

WherecampEU Rome 2013 Musings

WhereCampEU this year, rather earlier than normal, was in the Eternal City of Rome, Italy. After the threatening of Snowmeggeddon in the UK, a jaunt to Italy was a welcome respite. An action packed unconference timetable started with a presentation on Taarifa by myself. This was a follow on presentation from W3G but focusing on the characteristics of developing technology; needing to know the users and how they’ll use the ‘solution’. Developing solutions to first world problems then applying in the developing world isn’t useful and is dangerous, however, is the method de jure in some organisations.

A presentation on how the World Food Program uses the OpenDataKit, for collecting information in South Sudan followed. It would have been interesting to have heard more about the rationale and why they were using what they were using. The use-case was a take picture, see what is about, the intelligence that they sought to gather. However, the presenter didn’t stay around, so if anyone in the geo-sphere knows, please get in touch!

CartoDB was given a live demonstration. We’re quickly moving past the desktop for GIS and spatial analysis and into the cloud. I’d like to know how these cloud based GIS services compare with ESRIs and MapBox’s offerings. It’s a brave new world!

Michael Gould‘s 37 things you didn’t know about ESRI was a passionate talk about ESRI from its inception to the present day. A leviathan in the GIS space, the culture is seemingly anything but corporate America. In the examples mentioned the social conscious dominates decisions; from the positing of boulders on the ESRI campus to the acquisition of new companies.

A Taarifa breakout design session occurred with a special guest appearance from a snow-bound London. But more on this in a later blog post.

The day ended with an OSM Q&A by myself and Shaun McDonald turned into a wide ranging discussion about the OSM project and the challenges within. Getting new contributors to keep contributing was one point of discussion as was the need for improved internationalisation and languages.

An evening of Pizza, Dolcé and Grappa followed. The night ended in a spectacular deli/bistro/bar known only to locals and lost where campers. Bottles of Chanti and Prosecco were enjoyed and toasts made.

Standing out the following day was Laurence Penny‘s updated 1-D Maps . It’s never the same things, constantly reinventing itself with from the acquisitions and collection held by Laurence. Going from Doom, the Mille Miglia to Roman Era Road Routing with a detour around the metros and undergrounds. It was 2 hours long. Words fail to describe the brilliance that emanates from the presentation. I really look forward to seeing it in an updated form.

A certain Henk Hoff of the OSM Foundation, brought proceedings to a close on a wide ranging discussion on the foundation, how it functions and operates. The day and conferenced ended over pizza, chianti and sambucca. Just the way things should end!

Written and submitted on the Rome to Milan Eurostar (having just gone through Bologna!)

Involvement of Community Organisations

Community Based Organisations Learning About Our Project
Community Based Organisations Learning About Our Project

The first community forum went well. It was deal breaker for the project in the sense that if we didn’t get the community to share our ideals and objectives, making them their own, then the project would fail. Now the map is basically complete for first draft. We have enough of a basemap so we can now support platforms like Ushahidi and enable blogs to be geolocated.

Collecting the data and producing the map, as I’ve previously mentioned is only a first step. The same can be said for involving the students and community members. By creating a small nucleus of highly engaged people, proficient in mapping and storytelling techniques understand the project, they can evangelise the project to others in their community. This ‘infects’ the community from the inside allowing for more people to interact and share the project without ‘outside’ involvement. This will in time hopefully reach a plateau where the entire process of updating the map, reporting and blogging becomes self-sustaining using only the initial equipment and investment.

With this in mind, in the build up to the final community forum of the project (where presentations will be made to the community as a whole ie. interested citizens, civil servants and politicians) we gave a ‘pre-release’ talk to ten community based organisations. The format for this was quite simple, the students introduced the map with it’s features and intended functionality and the community members introduced the storytelling elements.

Within this process I spent most of it being a photographer and an observer. It wasn’t quite seeing the monster you created evolve but when presenting both students and the community are owning the process. The community organisations engaged in a Q&A session then participated in reporting using Ushahidi.

During the Q&A many questions dropped out regarding the future of the project and how the map can be used further. Because we are still formulating the future strategy it is difficult to say what the next step will be, but it will be along the lines of franchising to other areas overseen by community, NGO/CBO and Ground Truth, constructing this framework will be taking place for much of the coming week.

We are also printing the map and distributing it. This is key; in using Open Street Map, the collected data is freely available for viewing and data analysis without restrictions like a prohibitive licence. However accessibility to computers and internet understandably is a problem in communities like Tandale. To enable the community to view their map we will be printing A2 maps for placement in the sub-ward offices and printing A4 sub-ward handouts.

By placing it in a communal areas for each of the communities, we aim to reduce the barrier of people using the map by making it accessible. This process has started with our small nucleus of students and community, expanded by involving community based organisations and will be expanded further by integrating the map into governmental offices at the sub-ward level. Having the map built into the fabric of the community from the beginning of the project should make further incremental additions easier.

Community organisations being fully involved in the project is the next step, the process has been started and the ball is rolling. However Eid is coming in the next week, so everything is going pole pole, Swahili for slowly slowly. However tangible results are starting to become very clear, on all levels with all stakeholders.

Written and submitted from Slipway, Dar Es Salaam (-6.75174,39.27117)

Water, Sanitation and Geography

I have been surprised with during the Tandale project with how community members are familiar with the geographic boundaries and extent of their community. On touring areas with community members, it was clear how they used geographic features to navigate. Also the administrative boundaries were formed through natural features like rivers, without being imposed upon by an outside force.

Previously when facilitating mapping (essentially “There isn’t anything here, go have a look”) the mappers would have a difficultly in collating the map and their own mental model. Here in Tandale reading of seemed to be a lot easier than when I have previously experienced.

Map reading is a difficult skill, essentially it starts with understanding that the map is an abstract representation of space. As a map is a representation of space; a visualisation of various elements is at the whim of the cartographer (in our case the esteemed people which write the map styles for OSM) and the data that the cartographer/surveyor collects.

The people of Tandale seem to have a spatial awareness down to a very precise art, using landmarks and features with which to demarcate areas. This also relates to official and unofficial landuse within Tandale. Because of the lack of formal solid waste collection, most of the waste is dumped in swampland or wasteland. Unfortunately these waste areas have no buffer with residential homes, further illustrating the potential for disease.

On a tour of the Sokoni sub-ward executive he spoke at length on sanitation and water security. The conversation turned to common diseases and illnesses within Tandale with Malaria and HIV unsurprising common. Also mentioned in the same breath was Typhoid and Cholera, which, according to the officer,  outbreaks are common. Looking at the state of sanitation and drainage this is very believable.

We believe that the first step to solving these problems is to have a map. Hopefully have enough data to give evidence of the problems, to both inside and outside the community. We have also mapped dumping grounds, formal and informal medical facilities, toilets, water points among many other things. Using the map as a basemap in Ushahidi instance allows for the community members to use the map that they have created. Now our focus turns to completing the feedback loop, so there is an interface for the reports. Funnily enough that’s where my PhD comes in…

Written and submitted from the City Style Hotel, Sinza, Dar Es Salaam (-6.47319,39.13199)