Kartta Labs and re.city was a really cool Google Research project that I was involved in a few years ago. It enabled users to view, interact and contribute to historical streetscapes with a timeline. Working from historical maps, geo-rectifying them, tracing and digitizing buildings, adding historical photographs of buildings and with a catalogue of three dimensional models it produced a fascinating and engaging time enabled view into our past. The project was much bigger than I write about here so feel free to explore the videos at the end and linked blog posts and papers produced.
Three dimensional city streetscapes were created with a timeslider control. The buildings were modelled using photographs of the frontages of the buildings, and the footprints from historical maps.
I work with museums, galleries and archives on a number of historical crowdsourced mapping projects. Notably the New York Public Library’s Map Warper platform. Today this lives on in https://mapwarper.net and on Wikimedia’s Map Warper https://warper.wmflabs.org. I’m also familiar with the OpenStreetMap stack of applications, vector and raster tiles, cartography and a bit of server infrastructure – all these came into play during this project.
The components and applications of the project all ran on Google Cloud Project and with Kubernetes via the Google Kubernetes Engine (GKE). It used Google single sign on so users need only log into one place and authenticate for the different applications.
The front page showed an interactive map with pan and zoom control, a time slider at the top, links to the other applications, and help and about pages etc. The main applications are the Warper, Editor and Reservoir. This front page is where users can explore the map, select buildings to see or add historic photos of buildings and see the building footprints, and enter the interactive 3D view.
The maps layer was a custom style – to look vintage – using the colours inspired by 19th century street maps and with a subtle paper texture background.
Noter – application to allow a user to connect a building footprint and a face of a building based on geometry of a historic photo of a building
Reservoir – a repository of 3D models of buildings.
Warper was enhanced
Look for some of these improvements coming to the main mapwarper application soon. If there’s some you think are better than others let me know.
Quick Place
A new interface was added to help users quickly and easily place map images geographically. Users would see the collection of maps and could choose one to “quick place”.
There’s a simple scale and drag rotate the image on a map. You can either hit one “looks good” button, skip to do another or manually find a map to work on.
OCR for Geocoding
Using image processing + Google Cloud Vision for Optical Character Recognition, the map images are processed to extract any road names and any place names. Together with any map metadata, this can be help automatically locate the map, and could potentially be used for editing the features. This also helps users to rectify the maps, giving a good starting location.
I imagine that today’s more advanced AI image reading tools might give much better results.
Versioning
Maps had versions, enabling maps to be rolled back to other versions after a change in adding control points.
Social improvements
The users who worked on the map were stored and linked to so that a user could see who helped on the map, or see a call to action if no users had started on it.
Vector Tiles
Within the warper, vector tiles were added for rectification, and with a time slider, so users could see and rectify historic maps while looking at this map.
Other updates
The warper was updated to work via Docker and with Kubernetes. The Rails version was updated and OpenLayers was updated. As before, the vector tiles and time slider was integrated. (I note: One thing users of mapwarper.net suggest is that control points could be snapped to vector features). The Trace tab was changed so that you could load the rectified map into the Editor’s ID editor as a background image with the time tags so the user can quickly get started tracing over the historical map directly into the database. Mapwarper’s code was also updated to run with a later version of OpenLayers and the whole system with Docker.
Editor website
The editor website was a fork of the OpenStreetMap website and stack with the Id Editor and was Dockerized to work with Kubernetes and a GCP Cloud SQL Postgres server. This serves as the API where geometries of the buildings are kept. One edit for the website was to pass through some query parameters to the iD editor, so for example to load a rectified historic map or default historic dates to the editor. The id editor was customised to have a better interface for start and end times for features.
Vector Tiles
Tegola vector tile server was used and also run in Docker and in Kubernetes. The data was exported from the API as a planet and replication files created, and added into another GCP Cloud SQL Postgres server for vector tiles. Postgres was also uses to make the replication and planet service. The vector features had time properties within them so the maps are able to filter it.
AI / Machine Learning / Neural Networks
The output of the crowdsourcing mapping process fed into a workflow that heavily used AI to help create the 3D rendered cityscapes. We also considered using AI to assist with the georectification process.
Starting with footprints on maps and façade regions in historical images (both are annotated by crowdsourcing or detected by automatic algorithms), the footprint of one input building is extruded upwards to generate its coarse 3D structure. The height of this extrusion is set to the number of floors from the corresponding metadata in the maps database.
In parallel, instead of directly inferring the detailed 3D structures of each façade as one entity, the 3D reconstruction pipeline recognizes all individual constituent components (e.g., windows, entries, stairs, etc.) and reconstructs their 3D structures separately based on their categories. Then these detailed 3D structures are merged with the coarse one for the final 3D mesh. The results are stored in a 3D repository and ready for 3D rendering.
The key technology powering this feature is a number of state-of-art deep learning models:
Faster region-based convolutional neural networks (RCNN) were trained using the façade component annotations for each target semantic class (e.g., windows, entries, stairs, etc), which are used to localize bounding-box level instances in historical images.
DeepLab, a semantic segmentation model, was trained to provide pixel-level labels for each semantic class.
A specifically designed neural network was trained to enforce high-level regularities within the same semantic class. This ensured that windows generated on a façade were equally spaced and consistent in shape with each other. This also facilitated consistency across different semantic classes such as stairs to ensure they are placed at reasonable positions and have consistent dimensions relative to the associated entry ways.
Videos
The project is over now but a bunch of videos covering all the components can be found on the YouTube channel: https://www.youtube.com/@re-city8789
Want to do the same kind of project today? I’d suggest to start using OpenHistoricalMap. I was involved in development and hosting of the OpenHistoricalMap project from the beginning and evangelize for the project at a number of conferences and whenever I can!
In some ways this project and OHM’s redesign happened at the same time, but for different purposes. re.city’s data is Open Data License (ODBL) which enables the use of OpenStreetMap’s data which is useful for existing city streets and buildings. Users can then work back from today’s map into the past. This project was focused on cities. OpenHistoricalMap data is CC0 / Public Domain which potentially limits it’s ability to use OSM data but which which focuses on much wider historical timeframes and lots of existing historical data is public domain.
thinkwhere