More news from the shed…

In the month of May I seem to find myself playing with maps and numbers.

To the uninvolved this may appear to be rather similar to my earlier “That’s nice dear”, however the technology involved here is quite different.

This post is about extracting the results from the local elections held on 5th May from the Cheshire West and Chester website and displaying them as a map. I could have manually transcribed the results from the website, this would probably be quicker, but where’s the fun in that?

The starting point for this exercise was noticing that the results pages have a little icon at the bottom saying “OpenElectionData”. This was part of an exercise to make local election results more easily machine-readable in order to build a database of results from across the country, somewhat surprisingly there is no public central record of local council election results. The technology used to provide machine access to the results is known as RDF (standing for Resource Description Framework), this is a way of providing “meaning” to web pages for machines to understand - this is related to the talk of the semantic web. The good folks at Southampton University have provided a browser which allows you to inspect the RDF contents of a webpage. I used this to get a human sight of the data I was trying to read.

RDF content ultimately amounts to triplets of information: “subject”,”predicate”,”object”. In the case of an election then one triplet has a subject of “specific ward identifier” the predicate is “a list of candidates” and the object is “candidate 1;candidate 2; candidate 3…”. Further triplets specify the whether a candidate was elected, how many votes they received and the party to which they belong.

I’ve taken to programming in Python recently, in particular using the Python(x,y) distribution which packages together an IDE with some libraries useful to scientists. This is the sort of thing I’d usually do with Matlab, but that costs (a lot) and I no longer have access to it at home.

There is a Python library for reading RDF data, called RDFlib, unfortunately most of the documentation is for version 2.4 and the working version which I downloaded is 3.0. Searching for documentation for the newer version normally leads to other sites where people are asking where the documentation is for version 3.0!

The base maps come from the Ordnance Survey, specifically the Boundary Line dataset which contains administrative boundary data for the UK in ESRI Shapefile format. This format is widely used for geographical information work, I found the PyShp library from GeospatialPython.com to be well-documented and straightforward way to read the format. The site also has some nice usage examples. I did look for a library to display the resulting maps but after a brief search I adapted the simple methods here for drawing maps using matlibplot.

The Ordnance Survey Open Data site is a treasure trove for programming cartophiles, along with maps of the UK of various types there’s a gazetteer of interesting places, topographic information and location data for UK postcode.

The map at the top of the page uses the traditional colour-coding of red for Labour and blue for Conservative, some wards elect multiple candidates and in those where the elected councillors are not all from the same party purple is used to show a Labour/Conservative combination and orange a Labour/Liberal Democrat combination.

In contrast to my earlier post on programming, the key elements here are the use of pre-existing libraries and data formats to achieve an end result. The RDF component of the exercise took quite a while, whilst the mapping part was the work of a couple of hours. This largely comes down to the quality of the documentation available. Python turns out to be a compact language to do this sort of work, it’s all done in 150 or so lines of code.

It would have been nice to have pointed my program to a single webpage and for it to find all the ward data from there, including the ward names, but I couldn’t work out how to do this – the program visits each ward in turn and I had to type in the ward names. The OpenElectionData site seemed to be a bit wobbly too, so I encoded party information into my program rather the pulling it from their site. Better fitting of the ward labels into the wards would have been nice too (although this is a hard problem). Obviously there’s a wide range of analysis that can be carried out on the underlying electoral data.

Footnotes

The python code to do this analysis is here. You will need to install the rdflib and PyShp libraries and download the OS Boundary Line data. I used the Python(x,y) distribution but I think it’s just the matlibplot library which is required. The CWac.py program extracts the results from the website and writes them to a CSV file, the Mapping.py program makes a map from them. You will need to adjust file paths to suit your installation.

Obsession

This is a short story about obsession: with a map, four books and some numbers.

My last blog post was on Ken Alder's book "The Measure of All Things" on the surveying of the meridian across France, through Paris, in order to provide a definition for a new unit of measure, the metre, during the period of the French Revolution. Reading this book I noticed lots of place names being mentioned, and indeed the core of the whole process of surveying is turning up at places and measuring the angles to other places in a process of triangulation.

To me places imply maps, and whilst I was reading I popped a few of the places into Google Maps but this was unsatisfactory to me. Delambre and Mechain, the surveyors of the meridian, had been to many places. I wanted to see where they all were. Ken Alder has gone a little way towards this in providing a map: you can see it on his website but it's an unsatisfying thing: very few of the places are named and you can't zoom into it.

In my investigations for the last blog post, I discovered the full text of the report of the surveying mission, "Base du système métrique décimal", was available online and flicking through it I found a table of all 115 triangles used in determining the meridian. So a plan is formed: enter the names of the stations forming the 115 triangles into a three column spreadsheet; determine the latitude and longitude of each of these stations using the Google Maps API; write these locations out into a KML file which can be viewed in Google Maps or Google Earth.

The problem is that place names are not unique and things have changed in the last 200 years. I have spent hours transcribing the tables and hunting down names of obscure places in rural France, hacking away with Python and loved every minute of it. Cassini's earlier map of France is available online but the navigation is rather clumsy so I didn't use it. Although now I come to writing this I see someone else has made a better job of it.

Beside three entries in the tables of triangles are the words: "Ce triangle est inutile" - "This triangle is useless". Instantly I have a direct bond with Delambre, who wrote those words 200 years ago -  I know that feeling: in my loft is a sequence of about 20 lab books I used through my academic career and I know that besides an (unfortunately large) number of results the word "Bollocks!" is scrawled for very similar reasons.

The scheme with the the Google Maps API is that your program provides a place name "Chester, UK", for example, and the API provides you with the latitude and longitude of the point requested. Sometimes this doesn't work, either because there are several places with the same name or the placename is not in the database.

I did have a genuine Eureka moment: after several hours trying to find missing places on the map I had a bath and whilst there I had an idea: Google Earth supports overlay images on its maps. At the back of the "Base du système métrique décimal" there is a set of images showing where the stations are as a set of simple line diagrams. Surely I could overlay the images from Base onto Google Earth and find the missing stations? I didn't leap straight from the bath, but I did stay up overlaying images onto maps deep into the night. It turns out the diagrams are not at all bad for finding missing stations. This manual fiddling to sort out errant stations is intellectually unsatisfying but some things it's just quicker to do by hand!

You can see the results of my fiddling by loading this KML file into Google Earth, if you're really keen this is a zip file containing the image overlays from "Base du système métrique décimal" - they match up pretty well given they are photocopies of diagrams subject to limitations in the original drawing and distortion by scanning.

What have I learned in this process?

• I've learnt that although it's possible to make dictionaries of dictionaries in Python it is not straightforward to pickle them.
• I've enjoyed exploring the quiet corners of France on Google Maps
• I've had a bit more practice using OneNote, Paint .Net, Python and Google Earth so when the next interesting thing comes along I'll have a head start.
• Handling French accents in Python is a bit beyond my wrangling skills.

You've hopefully learnt something of the immutable mind of a scientist!


View

Book review: The Measure of All Things

“The Measure of All Things“ by Ken Alder tells the story of Pierre Méchain and Jean Baptiste Joseph Delambre’s efforts to survey the line of constant longitude (or meridian) between Dunkerque and Barcelona through Paris, starting amidst the French Revolution in 1792.

The survey of the meridian was part of a scheme to introduce a new, unified system of measures. The idea was to fix the length of the new unit, the metre, as 1/10,000,000th of the distance between the North Pole and the equator on a meridian passing through Paris.

At the time France used an estimated 250,000 different measures across the country with each parish having it’s own (uncalibrated) weights and measures with different measures for different types of material i.e. a “yard” of cotton was different from a “yard” of silk, and different if you were buying wholesale or selling to end users. These measures had evolved over time to suit local needs, but acted to supress trade between communities. Most nations found themselves in a similar situation.

Although the process of measuring the meridian started under the ancien regime, it continued in revolutionary France as a scheme that united the country. The names associated with the scheme: Laplace, Legrendre, Lavoisier, Cassini, Condorcet, leading lights of the Academie des Sciences, are still well known to scientists today.

Such surveying measurements are made by triangulation, a strip of triangles is surveyed along the line of interest. This involves precisely measuring the angles between each each vertex of the triangles in succession: given the three angles of a triangle and the length of one side of the triangle the lengths of the other two sides can be calculated. It’s actually only necessary to measure the length of one side on one triangle on the ground. Once you’ve done that you can use the previously determined lengths for successive triangles. All of France had been surveyed under the direction of César-François Cassini in 1740-80, the meridian survey used a subset of these sites measured at higher precision thanks to the newly invented Borda repeating circle. As well as this triangulation survey a measure of latitude was made at points along the meridian by examining the stars.
The book captures well the feeling of experimental measurement: the obsession with getting things to match up via different routes; the sick feeling when you realise you’ve made a mistake perhaps never to be reversed; the frustration at staring at pages of scribbles trying to find the mistake; the pleasure in things adding up.

Méchain and Delambre split up to measure the meridian in two sections: Delambre taking the northern section from Dunkerque to Rodez and Méchain the section from Rodez to Barcelona. Méchain delayed endlessly throughout the project, trusting little measurement to his accompanying team. Early on in the process, at Barcelona, he believed he had made a terrible error in measurement, but was unable to check whilst Spain and France were at war. He was wracked by doubt for the following years, only handing over doctored notes with great reluctance at the very end of the project. He was to die not long after the initial measurements were completed, leaving his original notes for Delambre to sift through.

At the time the measurements were originally made the understanding of experimental uncertainty, precision and accuracy were poorly developed. Driven in part by the meridian project and similar survey work by Gauss in Germany, statistical methods for handling experimental error more rigorously were developed not long afterwards. I wrote a little about this back here. Satellite surveying methods show that the error in the measurement by Méchain and Delambre is equivalent to 0.2 millimetres in a metre or 0.02%.

In the end the Earth turns out not to be a great object on which to base a measurement system: although it’s pretty uniform it isn’t really uniform and this limits the accuracy of your units. The alternative proposed at the time was to base the metre on a pendulum: it was to have the length necessary to produce a pendulum of period 2 seconds. This is also ultimately based on properties of the Earth since the second was defined as a certain fraction of the day (the time the Earth takes to rotate on its axis) and the local gravity which varies slightly from place to place, as Maskelyne demonstrated. 

Following the Revolution, France adopted, for a short time, a decimal system of time as well as metric units but these soon lapsed. However, the new metric units were taken up across the world over the following years - often this was during unification following war and upheaval.

The definition of the basic units used in science is still an active area. The definition of the metre has not relied on a unique physical object since 1960, rather it is defined by a process: the distance light travels in a small moment of time. However, the kilogram is still defined by a physical object but this may end soon with some exquisitely crafted silicon spheres.

I must admit to being a bit wary of this book in the first instance, how interesting can it be to measure the length of a line? However, it turns out I like to read history through the medium of science and the book provides an insight into France at the Revolution. Furthermore measuring the length of a line is interesting, or it is to a physicist like me.

Thanks to @beckyfh for recommending it!

Footnotes
1. The full-text of the three volume “Base du système métrique décimal" written by Delambre is available online. The back of the second volume contains summary tables of all the triangles and a diagram showing their locations.
2. The author’s website.
3. Some locations in Google Maps.

Book review: The World of Gerard Mercator

Once again I have been reading, this time "The World of Gerard Mercator" by Andrew Taylor. As before this blog post could be viewed as a review or, alternatively, as some notes to remind of what I have read. Overall I enjoyed the book, it provides the right amount of background information and doesn't bang on interminably about minutiae. I would have liked to have seen some better illustrations, but I suspect good illustrations of maps of this period are hard to come by and a full description of Mercator's projection was probably not appropriate.

The book starts off with some scene setting: at the beginning of the 16th century the Catholic church were still keen on Ptolemy's interpretation of world geography in fact to defy this interpretation was a heresy and could be severely punished. Ptolemy had put down his thoughts in Geographia produced around 150AD, which combined a discussion of the methods of cartography with a map of the known world. As a precedent Ptolemy's work was excellent, however by the time of the 16th century it was beginning to show it's antiquity. Geographical data, in Ptolemy's time, from beyond the Roman Empire was a little fanciful, and since the known world was a relatively small fraction of the surface of the globe the problems associated with showing the surface of a 3D object on a 2D map were not pressing. Ptolemy was well aware of the spherical nature of the world, Eratothenes had calculated the size of the earth in around 240BC, he stated that a globe would be the best way of displaying a map of the world. However, a globe large enough to display the whole world at sufficient detail would have to be very large, and thus difficult to construct and transport.

Truly global expeditions were starting to occur in the years before Mercator's birth: Columbus had "discovered"  the West Indies in 1492, John Cabot made landfall on the North American landmass in 1497. Bartolomeu Dias had sailed around the Southern tip of Africa in 1488, Vasco da Gama had continued on to India in 1497, around the Cape of Good Hope. The state of the art in geography could be found in Waldseemüller's map of 1507, showing a recognisable view of most of our world. Magellan's expedition would make the first circumnavigation of the globe in the early years of Mercator's life (1519-1522).

Mercator was born in Rupelmonde in Flanders on 5 March 1512, he died 2 December 1594 in Duisburg in what is now Germany at the age of 82. This was a pretty turbulent time in the Netherlands, the country was ruled by Charles V (of Spain) and there appears to have been significant repression of the somewhat rebellious and potentially Protestant population. Mercator was imprisoned for heresy in Rupelmonde in February 1543, remaining in custody until September, many in similar circumstances were executed, however Mercator seems to have avoided this by a combination of moderately powerful friends and a lack of any evidence of heresy.

Mercator's skill was in the collation and interpretation of geographical data from a wide range of sources including his own surveys. In addition he was clearly a very skilled craftsman in the preparation of copperplate engravings. He was commercially successful, manufacturing his globe throughout his life, as well as many maps and scientific instruments for cartographers. He also had a clear insight into the power of patronage.

His early work was in the preparation of maps of the Holy Land (in 1537) and Europe (in 1554), along with a globe produced in 1541. The globe seems to be popular amongst reproducers of antiquities, you can see details of it on the Harvard Map Collection Website.

Mercator is best known for his "projection", in this context a projection is a way of converting the world - which is found on the surface of a 3D sphere into a flat, 2D map. Mercator introduced his eponymous projection for his 1569 map of the world, illustrated at the top of this post. The particular feature of this projection is that if you follow a fixed compass bearing you find yourself following a straight line on the Mercator projected map. This is good news for navigators! The price you pay for this property is that, although all regions are in the correct places relative to each other, their areas are distorted so those regions near the poles appear much larger than those near the equator. Mercator seems to have made little of this discovery, nor described the method by which the projection is constructed - this was done some time later, in 1599, by Edward Wright. Prior to maps following Mercator's projection navigation was a bit hit and miss, basically you headed up to a convenient latitude and then followed it back to your destination - an inefficient way to plan your course. If you're interested in the maths behind the projection see here.

In terms of it's content the 1569 map shows Europe, Africa and a large fraction of Asia much as we would see it today, certainly in terms of outline. The Eastern coast of North and South America is fairly recognisable. The map fails in it's representation of the West coast of America - although to give credit where it is due, it at least has a west coast. The landmasses indicated at the northern and southern poles are close to pure fantasy. The Southern continent had been proposed by Ptolemy as a counterbalance to the known Northern continents - with no supporting evidence. Exploration of the far North was starting to occur during Mercator's life, with expedition such as that of Frobisher.

Mercator is also responsible for the word "atlas" to describe a book containing a set of maps, in this instance he coined the term to describe the volumes of maps he was preparing towards the end of his life, the last of which was published published posthumously by his son, Rumold, in 1595.

Following my efforts on Joseph Banks, I thought I'd make a map of significant locations in Mercator's life. You can find them here in Google Maps, zoom out and you will see the world in Mercator projection - a legacy from a man that lived nearly 500 years ago.