Building a stand-alone off-Wiki layered map using Wikidata & SPARQL

This Jupyter notebook shows you step by step how to make a Wikidata-driven layered map that can be used off-Wiki.

It is part of Module 3 of the map making workshop From Wikidata to interactive off-Wiki maps in three steps

Generic workshop opening slide

For more context to this workshop you can check the

This notebook is publically available at

For questions about this notebook, contact Olaf Janssen on Wikidata // olaf.janssen@kb.nl // https://twitter.com/ookgezellig

Latest update of this notebook: 22 October 2019


Start of the notebook

To make the map we will use

  1. PAWS (Jupyter Notebooks as a cloud service) - https://www.mediawiki.org/wiki/PAWS
  2. SPARQL queries from Wikidata - https://www.wikidata.org + https://query.wikidata.org/
  3. ipyleaflet (or Jupyter-Leaflet) : Interactive maps in the Jupyter notebook - https://github.com/jupyter-widgets/ipyleaflet + https://ipyleaflet.readthedocs.io/en/latest/
  4. ipywidgets (or Jupyter-widgets) - https://github.com/jupyter-widgets/ipywidgets + https://ipywidgets.readthedocs.io/en/latest

I assume that you have started your PAWS server (https://paws.wmflabs.org/paws/hub) and that you have uploaded the raw json file WikidataMapMakingWorkshop.ipynb to it. We will use this as a starting point.

Check that the URL of this notebook is https://paws.wmflabs.org/paws/user/YOURUSERNAME/notebooks/WikidataMapMakingWorkshop.ipynb (unless you renamed the notebook of course)

OK, now run (the 5 commands in) the following cell.

If you are running this notebook on a local Jupyter Notebooks installation, such as Anaconda, you need to skip this step. See below for the alternative instructions.

Notes:

Sidestep - running this notebook locally on Anaconda

Instead of running this notebook cloud-based on PAWS, you can also run it on a local Jupyter Notebooks installation, such as Anaconda

If that case (thus using the Conda package manager), instead of using the pip commands above, run the following 3 commands in your Anaconda Prompt

Anaconda Promt

Also make sure all the libraries mentioned in the cell below are available in your Anaconda environment (most of them will be included by default)

End of sidestep

Next, run the cell below to import the necessary Python libraries

Next we want to import a dataset containing geo coordinates (P625) that is the result of a SPARQL query in Wikidata

For this, we are using the query https://w.wiki/6gJ (public libraries in the Netherlands, an (as per 22-10-2019) incomplete data set) Public libraries in the Netherlands, layered by province

Run this query, and under </>Code --> Python you will see the Python code that has been pasted in the cell below Python code for public libraries in the Netherlands, layered by province

What about the red 'circle' in the screenshot above? To be able to run any SPARQL queries we must first set an explicit User-Agent to avoid getting blocked from the Wikidata Query Service - see

You do this by replacing in the Python code the line

sparql = SPARQLWrapper(endpoint_url)

by

sparql = SPARQLWrapper(endpoint_url, agent='User:YourUserName from PAWS notebook NotebookName.ipynb')

(and replace YourUserName with your real user name and NotebookName with the name of your notebook)

In order to make maps later on, we first need to do some transformation on the coordinates in the result of the cell above:

Convert coordinates string tuple like 'Point(6.558683 52.9946471)' into coordinates float tuple (52.9946471, 6.558683)

Note that the lat long order has been switched

Now we have two data objects available to make a map with

  1. libdict (= a dictionary)
  2. lib_coor_list (= a list)

We can use them to start building a layered map step by step

First, we start with a simple basemap (see https://ipyleaflet.readthedocs.io/en/latest/api_reference/map.html)

Overview of basemaps:

Note: not all basemaps seems to work equally well

Next we add a dark layer to the map above

Next we add a simple heatmap of public libraries in The Netherlands - https://ipyleaflet.readthedocs.io/en/latest/api_reference/heatmap.html

You will notice that in this simple heatmap the indivudual libraries are not clickable. Let's fix this. We want to add an (onClick) popup for every library in the heatmap, containing the municipality and province in which the library is situated.

As we can't directly add popus to a heatmap, we first need to make a circular marker for every library, to which we then connect a popup.

Now we want to combine the three layers we created in the above cells

  1. simple_heatmap layer
  2. dark background layer
  3. circles popup layer

into one single layer - as per https://ipyleaflet.readthedocs.io/en/latest/api_reference/layer_group.html

Your resulting map should look something like this

Heatmap Dutch public libraries with popup
(Note to self: you can use HTML in Markdown - https://stackoverflow.com/questions/14675913/changing-image-size-in-markdown)

As a next step we want to make a layer displaying the boundaries of all municipalities in The Netherlands that have one or more public libraries, using GeoShapes, as explained in Module 2 of the workshop.

To make this layer, we first need to request GeoJson geoshapes (= municipal boundaries) from Wikidata Q-values. See

We do this via P131, Located in the administrative territorial entity = municipality where the library is located

In the cell below, the SPARQL query after nl_muni_url = "https://maps.wikimedia.org/geoshape?getgeojson=1&query= is (the encoded version) of

SELECT DISTINCT ?id WHERE {
      ?item p:P31/ps:P31 wd:Q28564,wd:Q11396180;wdt:P17 wd:Q55. #Public library branch in the Netherlands
      ?item wdt:P131 ?id. 
      ?id p:P31 [ps:P31 ?idIsA].
      VALUES ?idIsA {wd:Q2039348}. #filter only the values 'municipality of the Netherlands' Q2039348 
    }

Note that ?id must have this name!

Now that we have the GeoJson object (muni_geoshapes), we can build a map layer with it

https://ipyleaflet.readthedocs.io/en/latest/api_reference/geo_json.html

The resulting map should look like this (zoomed in, focus on the red municipal boundaries, no popup showing)

Heatmap Dutch public libraries with red municipal boundaries

in the previous steps we have added a number of layers to the map. Next we can add a control in the top right corner of the map to switch layers on and off.

We also add a Fullscreen control

Your resulting heatmap should look something like this, with the red municipal geoshapes, a popup, a layers control and a fullscreen control

Dutch public library heatmap with the red municipal geoshapes, popup, layers control and fullscreen control

As the final step we want to export the map to a (stand-alone, off-Wiki) HTML page that can be viewed in a browser.

https://ipywidgets.readthedocs.io/en/latest/embedding.html#python-interface

The map above is available at http://ookgezellig.github.io/WikidataMapMakingWorkshop/NetherlandsPublicLibrariesHeatmap.html

This is the end of the workshop

Reusing this notebook

You are free to reuse this notebook (and all the other workshop materials), as it is available under the license Creative Commons Attribution-ShareAlike 4.0 International

CC-BY-SA logo

The cells below contain additional code snippets that can be used as inspiration for building additional map features.

There is no guarantee that all the (perhaps sometimes messy or outdated) code will run smoothly, you might need to do some tweaking