{"id":327,"date":"2014-11-29T20:18:53","date_gmt":"2014-11-30T01:18:53","guid":{"rendered":"http:\/\/brian.digitalmaddox.com\/blog\/?p=327"},"modified":"2014-11-29T20:18:53","modified_gmt":"2014-11-30T01:18:53","slug":"more-fun-with-old-maps","status":"publish","type":"post","link":"https:\/\/brian.digitalmaddox.com\/blog\/?p=327","title":{"rendered":"More Fun with Old Maps"},"content":{"rendered":"<p>I&#8217;ll admit it, I really like old maps.\u00a0 I especially like old topographic maps.\u00a0 I think it started when I used to work for the US Geological Survey.\u00a0 To me, it&#8217;s interesting to see how things change over time.\u00a0 From my old urban grown prediction days, I like to see where and when populations change.<\/p>\n<p>Since the USGS put out their <a title=\"USGS Historic Topographic Map Collection\" href=\"http:\/\/nationalmap.gov\/historical\/\" target=\"_blank\">Historical Topographic Map Collection<\/a> (HTMC), I&#8217;ve been playing with the maps off and on for about a year now.\u00a0 I finally decided to start merging available maps of the same scale and vintage to study and possibly do feature extraction down the road.\u00a0 I&#8217;ll be placing them for download here in case anyone is interested as I process them.<\/p>\n<p>I thought I&#8217;d share how I put them together in case anyone else is interested.\u00a0 The first step is to go to the USGS website and pick the files you want to use.\u00a0 The files there are available in <a title=\"Wikipedia GeoPDF Article\" href=\"http:\/\/en.wikipedia.org\/wiki\/GeoPDF\" target=\"_blank\">GeoPDF<\/a> format.\u00a0 First thing you need to understand is that you may not find a map covering your area of interest at your scale of interest and vintage.\u00a0 Not everything managed to survive to the current day.\u00a0 For example, I made a merged 125K map of Virginia and most of southern VA is missing at that resolution.<\/p>\n<p>Once I download the GeoPDFs I want to work on, I use a modified version of the geopdf2gtiff.pl script from the <a title=\"The Xastir Project\" href=\"http:\/\/xastir.org\/index.php\/Main_Page\" target=\"_blank\">Xastir<\/a> project.\u00a0 The link to my modifications can be found <a title=\"My mod of the geopdf2gtiff script\" href=\"http:\/\/brian.digitalmaddox.com\/files\/Scripts\/geopdf2gtiff.pl\" target=\"_blank\">here<\/a>.\u00a0 I use LZW compression for my GeoTIFFs as it&#8217;s lossless and keeps the quality from the GeoPDFs.\u00a0 It is a Perl script and requires that you have GDAL and the GDAL API installed.\u00a0 What it does is calculate the neat-lines of the GeoPDF and then clips it to the neat-line while converting it to a GeoTIFF.\u00a0 Running it as as simple as:<\/p>\n<pre>geopdf2gtiff.pl inputfile.pdf<\/pre>\n<p>Once you have all of your GeoPDF files download and converted, the next step is to merge them.\u00a0 The fastest way I&#8217;ve found to merge involves using <strong>gdalbuildvrt<\/strong> and <strong>gdal_translate<\/strong>, also from GDAL.\u00a0 The first step is to create a virtual dataset of all of your files by running something like:<\/p>\n<pre>gdalbuildvrt -resolution highest -a_srs EPSG:4326 merged.vrt parts\/*.tif<\/pre>\n<p>The options I chose here are to pick the highest pixel resolution (<strong>-resolution<\/strong>) based on the input files.\u00a0 For this case the resolutions should be the same, but this way I don&#8217;t have to go through and verify that.\u00a0 Next I change the projection of the output file to WGS84 (<strong>-a_srs<\/strong>).\u00a0 Next is the file name of the virtual dataset and then the input files.<\/p>\n<p>Now that the virtual dataset is done, it&#8217;s time to actually merge all of the files together.\u00a0 The virtual dataset contains the calculated bounding box that will contain all of the input files.\u00a0 Now we use <strong>gdal_translate<\/strong> to actually create the merged GeoTIFF file:<\/p>\n<pre>gdal_translate -of GTiff -co COMPRESS=LZW -co PREDICTOR=2 merged.vrt ~\/merged.tif<\/pre>\n<p>Here again I use LZW compression to losslessly compress the output data.\u00a0 Note that <strong>gdal_translate<\/strong> will automatically add an Alpha channel as Band 4 in the image to denote areas that had no input data.\u00a0 That&#8217;s why we do NOT add the -addalpha flag to gdalbuildvrt.\u00a0 For performance tips, I&#8217;d suggest keeping the source data and output file on separate drives unless you&#8217;re running something like a solid state drive.\u00a0 To give you an idea of the output file sizes, Virginia merged (which did have a lot of areas missing), was around 500 megabytes.<\/p>\n<p>Next you&#8217;ll need a Shapefile to use as a cut file to clip the data.\u00a0 Since I have the Census Tiger 2013 data in a local PostGIS database (see previous posts to this blog), I used QGIS to select just the VA state outline and then saved it as a Shapefile.<\/p>\n<p>Finally, we will use <strong>gdalwarp<\/strong> to clip the merged GeoTIFF against the state outline to produce the clipped GeoTIFF that is just the state itself.\u00a0 This operation can take a bit of time depending on how powerful a machine you&#8217;re running it on.\u00a0 The command you will use is similar to this:<\/p>\n<pre>gdalwarp --config GDAL_CACHEMAX 1024 -wm 1024 -cutline va_outline.shp -crop_to_cutline -multi -t_srs EPSG:4326 -co COMPRESS=LZW -co PREDICTOR=2 -co BIGTIFF=YES -co TILED=YES ~\/merged.tif clipped.tif<\/pre>\n<p>Some of the command line parameters I used are optional, I just tend to leave them in since I do a lot of copying and pasting \ud83d\ude09\u00a0 First we tell GDAL to increase the size of its caches using the <strong>&#8211;config GDAL_CACHEMAX<\/strong> and <strong>-wm<\/strong> options.\u00a0 Next we specify the file to clip against with the <strong>-cutline<\/strong> and <strong>-crop_to_cutline<\/strong> options.\u00a0 The -multi option tells GDAL to process using multiple threads.\u00a0 I again specify the output projection and the LZW compression parameters.\u00a0 Here I also specify the BIGTIFF option just in case the output file goes over four gigabytes.\u00a0 Finally, I tell <strong>gdalwarp<\/strong> to tile the output TIFF so it will load faster in a GIS by separating it into tiles.<\/p>\n<p>The output will look something like the below figure.\u00a0 I&#8217;ll start posting files as I get time.\u00a0 Hope everyone is having a great holiday!<\/p>\n<div id=\"attachment_328\" style=\"width: 310px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/brian.digitalmaddox.com\/blog\/wp-content\/uploads\/2014\/11\/virginia_clipped.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-328\" class=\"size-medium wp-image-328\" src=\"https:\/\/brian.digitalmaddox.com\/blog\/wp-content\/uploads\/2014\/11\/virginia_clipped-300x211.png\" alt=\"Clipped Virginia 125K Old Maps\" width=\"300\" height=\"211\" srcset=\"https:\/\/brian.digitalmaddox.com\/blog\/wp-content\/uploads\/2014\/11\/virginia_clipped-300x211.png 300w, https:\/\/brian.digitalmaddox.com\/blog\/wp-content\/uploads\/2014\/11\/virginia_clipped-1024x723.png 1024w, https:\/\/brian.digitalmaddox.com\/blog\/wp-content\/uploads\/2014\/11\/virginia_clipped.png 1251w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-328\" class=\"wp-caption-text\">Clipped Virginia 125K Old Maps<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ll admit it, I really like old maps.\u00a0 I especially like old topographic maps.\u00a0 I think it started when I used to work for the US Geological Survey.\u00a0 To me, it&#8217;s interesting to see how things change over time.\u00a0 From &hellip; <a href=\"https:\/\/brian.digitalmaddox.com\/blog\/?p=327\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-327","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/327","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=327"}],"version-history":[{"count":1,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/327\/revisions"}],"predecessor-version":[{"id":329,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/327\/revisions\/329"}],"wp:attachment":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=327"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=327"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=327"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}