Canvassing on a shoestring (for techies)
A very technical discussion, including snippets of source code, of the major components used to create our interactive Canvass planning demo.
The article shows how to use a combination of Leaflet and OpenStreetMap to create a sophisticated and yet intuitive interactive geospatial search solution without using Google maps or Bing maps.
A very simple canvassing solution is used to showcase the technologies involved.
If you have not already done so have play with the canvass planning demo we have created at Canvass planning as much of this blog will refer to how we put this together using open source/free components and services.
The requirement
From our experience of building canvassing systems in the past, when you boil the requirements down to the essentials what both national and local canvass organisers alike really want to know is where have they already canvassed and where they plan on canvassing next.
Sure there are lots of extra features that can be added to capture lots of additional information which can, with sufficient budget, add value from a business intelligence perspective. However, in our experience these “advanced features” can often get in the way of the primary function of the canvassing application - namely canvassing opinion either through door stepping or call campaigns. In many cases it has proven far more efficient to have canvass responses captured on paper and transcribed using professional data entry staff than using sophisticated mobile solutions to capture data directly on the doorstep.
For the purposes of this demonstration we wanted to create a “bare bones” web application that would allow users to create and save a “walk list” by interacting with a map rather than simply working from a list of street names.
The “walk list” would normally then be assigned to one or more canvassers who would use this as their own personal map of streets they had to canvass that day.
We have stopped short of what would be done with any canvass responses captured as this heavily dependent on what is being asked and for what purpose.
The building blocks
The key components needed for this solution, the building blocks so to speak, can be summarised as follows:
- An interactive map that allows pan and zoom
- A geospatial search to find addresses
- A means of choosing streets to add to the walk list
- Some form of rudimentary walk list management
- A means of gluing these components together into a reactive web application
The Map
The first component we need is a mapping solution. For our demo application we are really only interesting in a mapping solution which covers Scotland as this is our target geographical area.
For this we chose the combination of Leaflet and OpenStreetMap. Leaflet is a lesser known open source mapping solution that leverages mapping data provided by OpenStreetMap. OpenStreetMap is a community driven source of geospatial data and is managed by the OpenStreetMap Foundation whose primary mission is to provide free geographic data, including map data, to its users for any purpose, commercial or otherwise.
We could have chosen Google Maps, Bing Maps or even the Ordinance Survey mapping solutions as each has powerful apis that would let us do what was needed but each of these has some fairly restrictive licensing terms which could result in unnecessary complications and hidden costs in a real world project.
One of the things we really like about OpenStreetMap is that it is fairly easy for users to add/update the map data themselves. Say for example that there was a recent housing development near you that does not appear in Google Maps, Bing Maps or OpenStreetMap but you wanted included in your canvassing campaign. With OpenStreetMap you can register as a contributor fairly easily and add the missing data yourself.
Having used all the mapping solutions mentioned over the years we have recently come to favour the combination of Leaflet and OpenStreetMap to that of their more commercial brethren both from a licensing perspective and from a technical perspective.
The Leaflet team provide some great tutorials and exhaustive documentation on their website leafletjs.comranging from simple mapping applications to advanced geospatial applications. We won’t delve deep into the numerous features that Leaflet offers but will instead only discuss those that are essential to our application.
First, as is the norm in this kind of web application, we declare a placeholder for our map to live in somewhere in our page html. The more astute of you will notice the Bootstrap 3 class “col-md-6”. We use Bootstrap to provide much of the layout of our site as it has proven to be a very flexible grid system.
<div id="geomap" class="col-md-6"></div>
Once the page has loaded we initialise the map. In our case we set the zoom level and initial latitude and longitude to centre our map on Scotland.
Note that the “L” object refers to the global Leaflet object. This is defined by leafletjs on instantiation so we don’t have to declare it as such. The global namespaced variable “$.canvas.map” is simply so we can reference the map object at a later time.
var map = L.map('geomap').setView([57.58, -4.42], 6);
$.canvas.map = map;
At this stage we have a placeholder and have instantiated a map object. However, there is more to drawing maps than just this. If we were to pause the code at this stage all we are likely to see is a grey box with some controls added to it. This is because we have not yet fetched any map tiles. In the majority of cases web and mobile app maps are drawn using map tiles. Map tiles are a clever way for map data providers to send you only the map data you need for the area you are viewing at the level of detail needed as dictated by the map zoom factor.
This is where OpenStreetMap comes in, it is OpenStreetMap that has all the geospatial data Leaflet is just a very clever wrapper for this that provides a great api to access and consume this geospatial data. Our application uses OpenStreetMap as it’s map tile provider so we must tell Leaflet where to find this.
Note the inclusion of the map data attribution which is about the only constraint that the OpenStreetMap licence places on you.
var osmUrl='https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png';
var osmAttrib='Map data © <a href="https://openstreetmap.org">OpenStreetMap</a> contributors';
var osm = new L.TileLayer(osmUrl, {minZoom: 3, maxZoom: 15, attribution: osmAttrib});
map.addLayer(osm);
Now we have a map and thanks to Leaflet in a handful of lines of code we have all the zoom and pan functionality we could want.
One thing to watch out for when following through the tutorials on the Leaflet site is that they suggest using a “cdn” url to reference their javascript and css files. In many instances using a CDN is a good idea as it minimises the load on your server. However, in this case it seems the SSL certificate for this CDN may have expired, thus if your site is an HTTPS site many of the newer browsers will refuse to fetch the Leaflet code and obviously this means your mapping application will not work as expected. Currently we recommend maintaining a local copy of Leaflet instead of relying on the CDN copy.
The search engine
It may not be obvious at first but there are two fundamentally different types of search being offered by the demo application. The first, and most obvious, is the address search carried out when a user enters a partial address in the address field and clicks the “Go” button. The second type of search occurs when a user selects a target street using the map itself.
To carry out a search based on the text entered in the search text field we use another of OpenStreetMap’s services. Nominatum is a geospatial search engine api provided by the OpenStreetMap Foundation expressly for the purposes of querying OpenStreetMap data. This is a complex api with a great many options and features but as before we will only discuss those directly relevant to our application.
The following code snippet shows a simple event handler that responds to “Go” button clicks. Before passing the search query to Nominatum we first give the value entered in the search box a quick sanity check using a regular expression comprising a list of “way” types.
For our purposes we envisage a usage scenario where the user is browsing the map or perhaps knows the street name they are considering including in their walk list prior to performing the search. Obviously this list is not exhaustive but it can be easily extended to add other “way” types e.g. crescent is an obvious omission. Similarly a regex pattern can be used which tests for valid postcode values should you wish to add this.
Assuming the search text entered meets rudimentary criteria we then encode the query to make it web safe using the standard Javascript encodeURIComponent function.
'click #findaddress': function(evt){
var query = $('#searchtext').val();
var query_exp = /( drive| road| way| place| street| lane| avenue)/;
if(!query_exp.test(query)){
alert("Invalid search text!");
return;
}
query = encodeURIComponent(query);
var jqxhr = $.get("https://nominatim.openstreetmap.org/search?format=json&polygon_geojson=1&addressdetails=1&q="+query,function(data){
drawFromAddressList(data, false)
})
.fail(function(){
alert("Error searching for street!");
});
},
We use a straight forward jQuery.get Ajax call to make the actual search request to Nominatum but notice that we have prepended our query with a number of query parameters that tell the Nominatum search engine what we want returned in its response. First we tell it we want the response format to be JSON then as we will be drawing the data on a map we set polygon_geojson=1 which tells the search engine to return the geometry of each of our search results in a geojson format and finally we tell the search engine the level of address detail we want returned by setting addressdetails=1.
Once we have the response we simply pass it to a function, drawFromAddressList, which parses the response and updates the UI accordingly.
Our implementation of this function, and subsequent functions, includes some lines of code which are particular to way of managing UI updates e.g. Session.set() but hopefully these will not detract from your understanding of the relevant parts of the code.
function drawFromAddressList(list, isLoaded){
var maxLat = 0;
var minLat = 180;
var maxLng = -180;
var minLng = 180
for (var i = 0; i < list.length; i++) {
var way = list[i];
var road = "";
if(way.osm_type=="way"){
road = getWayName(way.address);
if(road == ""){
continue;
}
var lat1 = +way.lat - 0.01;
var lng1 = +way.lon - 0.01;
var lat2 = +way.lat + 0.01;
var lng2 = +way.lon + 0.01;
console.log(way.geojson);
//drawPolyLineFromGeoJSON(way.geojson);
if(addOSMAddress(way) === true){
Session.set("map-status", "Fetching street data");
drawWay(way.osm_id, road, lat1, lng1, lat2, lng2, isLoaded);
}
if(+way.lat > maxLat){
maxLat = +way.lat;
}
if(+way.lat < minLat){
minLat = +way.lat;
}
if(+way.lon > maxLng){
maxLng = +way.lon;
}
if(+way.lon < minLng){
minLng = +way.lon;
}
}
}
if(maxLat > 0){
var southWest = L.latLng(minLat,minLng);
var northEast = L.latLng(maxLat,maxLng);
var bounds = L.latLngBounds(southWest, northEast);
$.canvas.map.fitBounds(bounds);
}
}
We start by iterating over the result set and act on only those items that have an osm_type == “way” as these represent “street” type objects. The term “way” is used by OpenStreetMap to refer to any street or road type object. In an ideal world we would simply pass the geojson geometry data associated with each “way” to function that draws lines which overlay the streets on the map as shown in the commented out line drawPolyLineFromGeoJSON(way.geojson);
Unfortunately, we have found through experimentation that frequently the geojson geometry returned by Nominatum is incomplete despite the fact that we have verified it actually exists in the OpenStreetMap repository. We have left the commented lines in to illustrate how this should have worked.
To overcome this issue we were forced to make a call out to yet another api called Overpass which is what the drawWay() function does. Before calling the drawWay function we first define a bounding box for our Overpass query - lat1, lon1, lat2, lon2 which encompasses the object we want to find the geometry for. Defining a bounding box this way narrows the area in which Overpass will look for results so improves the performance of the search. Once we get the results back from the drawWay() method we simply calculate the new bounds for the map and tell the map to fit itself to these which in turn sets the centre and zoom level of the map.
The drawWay() function we use is shown below:
// This does the actual drawing of the polyline. First though it fetches the correct data
function drawWay(osm_id, road, lat1, lng1, lat2, lng2, isLoaded){
var sourcelist = Session.get("source-list");
if(isLoaded === true){
sourcelist = Session.get("chosen-list");
}
sourcelist.push(osm_id);
// This is where we make the call to Overpass
var wayParameters = {};
wayParameters.data = '[out:json];way["name"="' + road + '"](' + lat1 + "," + lng1 + "," + lat2 + "," + lng2 + ");(._;>;);out;";
$.getJSON("https://overpass-api.de/api/interpreter",wayParameters, function(waydata){
console.log(waydata);
var nodes = [];
var ways = [];
Session.set("map-status", "Drawing street");
// Iterate over all the Overpass results
for (var i = 0; i < waydata.elements.length; i++) {
if(waydata.elements[i].type == "node") {
var node = {};
nodes.push(waydata.elements[i]);
}
else {
if(waydata.elements[i].type == "way"){
ways.push(waydata.elements[i]);
}
}
}
// We have to iterate over every way to get the nodes in the correct order to draw the road or street or whatever
var polylineArray = [];
for (var i = 0; i < ways.length; i++) {
var wayNodes = ways[i].nodes;
var path = [];
for (var j = 0; j < wayNodes.length; j++) {
for (var k = 0; k < nodes.length; k++) {
if(nodes[k].id == wayNodes[j]){
var latlng = L.latLng(nodes[k].lat, nodes[k].lon);
path.push(latlng);
break;
}
}
}
console.log(path);
// Now add the polyline to the map using Leaflet
var color = 'red';
if(isLoaded === true){
color = 'blue';
}
var polyline = L.polyline(path, {color: color}).addTo($.canvas.map);
polylineArray.push(polyline);
}
// We have an array of arrays of polylines
var addrPoly = {osm_id: osm_id, polylineArray: polylineArray};
if(ways.length > 0){
$.canvas.polylineList.push(addrPoly);
if(isLoaded === true){
Session.set("chosen-list", sourcelist);
}
else {
Session.set("source-list", sourcelist);
}
}
else {
// Remove from master address list as no valid ways present
removeOSMAddress(osm_id);
}
Session.set("map-status", "Ready for next search");
});
}
Again this code snippet is made slightly more complex by some of our UI and list management code but hopefully not overly so.
One line to pay particular attention to is the horrendously complex looking line which defines wayParameters.data. Despite how it looks this is really just building a complex string which will one used as the query parameter sent to Overpass.
Another thing to keep in mind studying this code is that in most cases the geometry of a road or street cannot be defined using a single line but instead requires a set of lines connected nose to tail. This is what we mean by the term “polyline”. Occasionally you find streets or roads with gaps in them, perhaps because the adjoin a town square for example, thus we can have an array of polylines which are in turn an array of lines associated with a single street or road.
All these lines and polylines must be ordered and translated into path that Leaflet can understand before they can be overplayed on the map. This is what this function does.
Now we turn our attention to the other type of search users can do i.e. selecting a street on the map.
For this we start with an event handler which intercepts any clicks on the map and translates these into Nominatum search queries. This time however we are doing what is called a reverse geocoding search. A reverse geocoding search essentially takes a longitude and latitude coordinate and returns what is near that location. In most apis you can specify what kind of thing you are looking for and in our case we stipulate that the osm_type we are looking for is “W” which means we want to find “way” objects as we are only interested in streets, roads etc.
Here we can see how easy Leaflet makes the task of responding to map clicks by doing all the heavy lifting for us. Once we have the latlng coordinates we simply give them a sense check to verify they are within the bounding box defined by minLat, minLon, maxLat and maxLon which constrains them to our target area - Scotland. Obviously if you were running a local campaign you could easily constrain this to a town, city, suburb etc.
Most of this code snippet is self explanatory but note as before we call the drawWay() function to manage the missing geometry issue.
Hopefully Nominatum will resolve this issue and may already have done so by the time you read this. It may also be that the issue may be limited to a particular geographic region and it may work for your target area.
function onMapClick(evt){
var parameters = {};
var maxLat = 60.9;
var minLat = 54.8;
var maxLng = -0.590;
var minLng = -8.99;
parameters.format = "json";
parameters.lat = evt.latlng.lat;
parameters.lon = evt.latlng.lng;
if(parameters.lat < minLat || parameters.lat > maxLat || parameters.lon < minLng || parameters.lon > maxLng){
alert("Invalid map target location. Demonstration limited to Scotland only.");
return;
}
parameters.addressdetails = 1;
parameters.osm_type = "W";
Session.set("map-status", "Looking up location");
$.getJSON("https://nominatim.openstreetmap.org/reverse",parameters, function(data){
Session.set("map-status", "Found location, scaling map");
$.canvas.map.setView([evt.latlng.lat, evt.latlng.lng], 15);
console.log(data);
var wayParameters = {};
var lat1 = evt.latlng.lat - 0.01;
var lng1 = evt.latlng.lng - 0.01;
var lat2 = evt.latlng.lat + 0.01;
var lng2 = evt.latlng.lng + 0.01;
var road = "";
if(!data.hasOwnProperty("address")){
return;
}
road = getWayName(data.address);
if(road == ""){
alert("No suitable target found in this area!");
Session.set("map-status", "No streets found!");
return;
}
// Attempt to add the street - returns false if it already exists
if(addOSMAddress(data) === true){
Session.set("map-status", "Fetching street data");
drawWay(data.osm_id, road, lat1, lng1, lat2, lng2, false);
}
else {
alert("Street already in the address list!");
Session.set("map-status", "Ready to search again");
}
});
}
Choosing streets for the walk list
Compared to searching for addresses the task of adding streets to our walk list is trivial but their are still one or two “gotchas” we found while putting this together that we thought worth mentioning.
To facilitate the selection process and add some eye candy we have implemented an HTML 5 based drag and drop UI which allows users to drag streets from the “Choices” list box to the “Walk list”. There is very detailed, albeit dated, tutorial on this on the HTML5 Rocks website at html5rocks.com and a much easier to follow tutorial on the w3schools site at w3schools.com.
When we were testing our implementation of the HTML5 drag and drop functionality we found that mobile devices did not support the native HTML5 drag and drop functionality at all. However, when reviewing the responsive layout of the page we realised that even if they had that would have been a messy way of doing things on a mobile device. In general when dealing with mobile devices experience has taught us to stick to the KISS principle. Mobile users will never thank you for a sophisticated UI that causes them pain to use but they may occasionally thank you for something that “just works”.
For mobile users we simply attached “touch end” event listeners to those controls which act as drag and drop data sources/destinations and then added some control logic to handle the transfer of an address from one list to another. The code snippet below illustrates how we do this:
// This is a work around for devices that don't support drag & drop
'touchend': function(evt){
var target = evt.currentTarget;
var event = evt.originalEvent;
if (event.preventDefault) {
event.preventDefault();
}
if (event.stopPropagation) {
event.stopPropagation();
}
target.style.opacity = '0.3';
$.canvas.selectedIndex = target.dataset.addressindex;
console.log("started");
}
// Later in the code ...
// This is a work around for devices that don't support drag & drop
'touchend #destinationlist': function(evt){
handleDestinationListDrop(evt);
},
'touchend #trashcan': function(evt){
handleTrashcanDrop(evt);
},
Managing the list
We have already illustrated how we manage the Create(C) and Update(U) aspects of the CRUD lifecycle for our simple demo application and now we have to consider how we manage the Delete(D) and Read(R) aspects.
The Delete action follows the same pattern as we used for the Create action, namely HTML5 drag and drop with a workaround for mobile devices only this time the destination is the trashcan and so we won’t detail this here as we would be repeating what has gone before, more or less.
Of more interest is the Read action which in our case corresponds to what happens when a user clicks the “Load walk list” button.
Clearly to be able to load a walk list the user must first have persisted the contents of their walk list using the “Save walk list” button. The save functionality has not been discussed previously as it pertains to all aspects of the CRUD lifecycle as it is only via the save action that any changes can be persisted thus it seemed reasonable to wait until now before discussing this functionality.
To keep the demo application simple while introducing an another very powerful HTML5 feature we have chosen not to build the persistence of our walk list on the conventional backend database pattern, though this is probably what you would do if this was part of a larger multiuser system. Instead we have opted to leverage HTML5’s local storagefunctionality.
In modern web browsers every HTML5 web app has essentially its own local storage just waiting for the application to store and read information from it. Generally a minimum of 5MB+ storage space is automatically maintained by the browser for your application.
This makes local storage very useful for persisting data locally and also in the case of mobile devices can be accessed when the device is disconnected from the web thus can act as a offline data store for mobile web apps. We have used this to great effect in a number of mobile web projects where internet access was problematic.
Using HTML5 local storage could not be simpler as the following code snippet shows:
function saveToLocalStorage() {
// Check if loacl storage is supported
if(testLocalStorage()) {
var chosenList = Session.get("chosen-list");
var walkList = [];
if(chosenList.length > 0) {
for (var i = 0; i < chosenList.length; i++) {
var address = getAddressObject(chosenList[i]);
walkList.push(address);
};
}
// Serialise our walk list data using JSON and store it
localStorage.setItem("walklist", JSON.stringify(walkList));
}
}
As HTML5 local storage only supports string data we must first serialise our walk list data and convert this into a string format. Fortunately the JSON.stringify() function makes this very simple.
Similarly reading that data back is every bit as easy:
function loadFromLocalStorage() {
if(testLocalStorage()) {
var walkList = JSON.parse(localStorage.getItem("walklist"));
drawFromAddressList(walkList, true);
}
}
We simply de-serialise the data stored in the HTML5 local storage repository and then pass this to the drawFromAddressList() function discussed previously.
Gluing everything together
While we could have implemented the demo application using just HTML5 and the various Javascript libraries mentioned previously e.g. Bootstrap 3, Leaflet, jQuery etc. we thought this would be a good opportunity to mention a web application platform that we have enjoyed using over the last couple of years - Meteor.
Meteor is a framework for developing serious Javascript applications that exposes all the power that Javascript and NodeJS have to offer but with very little of the pain. Out the box Meteor gives you a great deal of functionality not only to create powerful interactive web apps but also to create native mobile device applications.
In addition to what Meteor offers “out the box” there is a very vibrant developer community that actively develop and maintain Meteor packages which makes it exceptionally easy to add extra functionality. Almost every major Javascript library will have one ore more wrapper packages already created by the community thus adding major functionality to your application can be as simple as typing a single line to add a package. This blog and Bootstrap 3 were both added as Meteor packages.
If you need your web app solutions to respond very quickly to user interactions, pseudo realtime, while leveraging some of the most powerful Javascript libraries available we would strongly suggest you check Meteor out!
Conclusion
We hope this blog has shown that OpenStreetMap and associated apis represents a viable alternative mapping solution to that of Google Maps and Bing Maps.
Hopefully you found our simple choice of Use Case as interesting as we did when coding the demo and that the code snippets provided act as a springboard for your own mapping ideas.