{"id":520,"date":"2018-09-23T13:29:49","date_gmt":"2018-09-23T17:29:49","guid":{"rendered":"http:\/\/brian.digitalmaddox.com\/blog\/?p=520"},"modified":"2018-09-23T13:29:49","modified_gmt":"2018-09-23T17:29:49","slug":"revisiting-historic-topographic-maps-part-1","status":"publish","type":"post","link":"https:\/\/brian.digitalmaddox.com\/blog\/?p=520","title":{"rendered":"Revisiting Historic Topographic Maps Part 1"},"content":{"rendered":"

My first professional job during and after college was working at the US Geological Survey<\/a> as a software engineer and researcher. My job required me to learn about GIS<\/a> and cartography<\/a>, as I would do things from writing production systems to researching distributed processing. It gave me an appreciation of cartography and of geospatial data. I especially liked topographic maps<\/a> as they showed features such as caves and other interesting items on the landscape.<\/p>\n

Recently, I had a reason to go back and recreate my mosaics of some Historic USGS Topomaps<\/a>. I had originally put them into a PostGIS<\/a> raster database, but over time realized that tools like QGIS<\/a> and PostGIS raster can be extremely touchy when used together. Even after multiple iterations of trying out various overview levels and constraints, I still had issues with QGIS crashing or performing very slowly. I thought I would share my workflow in taking these maps, mosaicing them, and finally optimizing them for loading into a GIS application such as QGIS.\u00a0 Note that I use Linux and leave how to install the prerequisite software as an exercise for the reader.<\/p>\n

As a refresher, the USGS has been scanning in old topographic maps and has made them freely available in GeoPDF<\/a> format here<\/a>. These maps are available at various scales and go back to the late 1800s. Looking at them shows the progression of the early days of USGS map making to the more modern maps that served as the basis of the USGS DRG<\/a> program. As some of these maps are over one-hundred years old, the quality of the maps in the GeoPDF files can vary widely. Some can be hard to make out due to the yellowing of the paper, while others have tears and pieces missing.<\/p>\n

Historically, the topographic maps were printed using multiple techniques from offset lithographic printing to Mylar separates. People used to etch these separates over light tables back in the map factory days. Each separate would represent certain parts of the map, such as the black features, green features, and so on. While at the USGS, many of my coworkers still had their old tool kits they used before moving to digital. You can find a PDF here<\/a> that talks about the separates and how they were printed. This method of printing will actually be important later on in this series when I describe why some maps look a certain way.<\/p>\n

Process<\/b><\/p>\n

There are a few different ways to start out downloading USGS historic maps. My preferred method is to start at the USGS Historic Topomaps<\/a> site.<\/p>\n

\"USGS<\/a>

USGS Historic Maps Search<\/p><\/div>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

It is not quite as fancy a web interface as the others, but it makes it easier to load the search results into Pandas<\/a> later to filter and download. For my case, I was working on the state of Virginia, so I selected Virginia<\/i> with a scale<\/a> of 250,000<\/i> and Historical<\/i> in the Map Type<\/b> option. I purposely left Map Name<\/b> empty and will demonstrate why later.<\/p>\n

\"Topo<\/a>

Topo Map Search<\/p><\/div>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

Once you click submit, you will see your list of results. They are presented in a grid view with metadata about each map that fits the search criteria. In this example case, there are eighty-nine results for 250K scale historic maps. The reason I selected this version of the search is that you can download the search results in a CSV<\/a> format by clicking in the upper-right corner of the grid.<\/p>\n

\"Topo<\/a>

Topo Map Search Results<\/p><\/div>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

After clicking Download to Excel (csv) File<\/b>, your browser will download a file called topomaps.csv<\/em>. You can open it and see that there is quite a bit of metadata about each map.<\/p>\n

\"Topo<\/a>

Topo Map CSV Results<\/p><\/div>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

If you scroll to the right, you will find the column we are interested in called Download GeoPDF<\/b>. This column contains the download URL for each file in the search results.<\/p>\n

\"Highlighted<\/a>

Highlighted CSV Column<\/p><\/div>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

For the next step, I rely on Pandas<\/a>. If you have not heard of it, Pandas is an awesome Python data-analysis library that, among a long list of features, lets you load and manipulate a CSV easily. I usually load it using ipython<\/a> using the commands in bold below.<\/p>\n

bmaddox@sdf1:\/mnt\/filestore\/temp\/blog$ ipython3<\/strong>\r\nPython 3.6.6 (default, Sep 12 2018, 18:26:19) \r\nType \"copyright\", \"credits\" or \"license\" for more information.\r\n\r\nIPython 5.5.0 -- An enhanced Interactive Python.\r\n?         -> Introduction and overview of IPython's features.\r\n%quickref -> Quick reference.\r\nhelp      -> Python's own help system.\r\nobject?   -> Details about 'object', use 'object??' for extra details.\r\n\r\nIn [1]: import pandas as pd<\/b>\r\n\r\nIn [2]: csv = pd.read_csv(\"topomaps.csv\")<\/b>\r\n\r\nIn [3]: csv<\/b>\r\nOut[3]: \r\n   Series     Version  Cell ID     ...      Scan ID GDA Item ID  Create Date\r\n0    HTMC  Historical    69087     ...       255916     5389860   08\/31\/2011\r\n1    HTMC  Historical    69087     ...       257785     5389864   08\/31\/2011\r\n2    HTMC  Historical    69087     ...       257786     5389866   08\/31\/2011\r\n3    HTMC  Historical    69087     ...       707671     5389876   08\/31\/2011\r\n4    HTMC  Historical    69087     ...       257791     5389874   08\/31\/2011\r\n5    HTMC  Historical    69087     ...       257790     5389872   08\/31\/2011\r\n6    HTMC  Historical    69087     ...       257789     5389870   08\/31\/2011\r\n7    HTMC  Historical    69087     ...       257787     5389868   08\/31\/2011\r\n..    ...         ...      ...     ...          ...         ...          ...\r\n81   HTMC  Historical    74983     ...       189262     5304224   08\/08\/2011\r\n82   HTMC  Historical    74983     ...       189260     5304222   08\/08\/2011\r\n83   HTMC  Historical    74983     ...       707552     5638435   04\/23\/2012\r\n84   HTMC  Historical    74983     ...       707551     5638433   04\/23\/2012\r\n85   HTMC  Historical    68682     ...       254032     5416182   09\/06\/2011\r\n86   HTMC  Historical    68682     ...       254033     5416184   09\/06\/2011\r\n87   HTMC  Historical    68682     ...       701712     5416186   09\/06\/2011\r\n88   HTMC  Historical    68682     ...       701713     5416188   09\/06\/2011\r\n\r\n[89 rows x 56 columns]\r\n\r\nIn [4]:<\/pre>\n
As you can see from the above, Pandas loads the CSV in memory along with the column names from the CSV header.<\/p>\n
In [6]: csv.columns<\/b>\r\nOut[6]: \r\nIndex(['Series', 'Version', 'Cell ID', 'Map Name', 'Primary State', 'Scale',\r\n       'Date On Map', 'Imprint Year', 'Woodland Tint', 'Visual Version Number',\r\n       'Photo Inspection Year', 'Photo Revision Year', 'Aerial Photo Year',\r\n       'Edit Year', 'Field Check Year', 'Survey Year', 'Datum', 'Projection',\r\n       'Advance', 'Preliminary', 'Provisional', 'Interim', 'Planimetric',\r\n       'Special Printing', 'Special Map', 'Shaded Relief', 'Orthophoto',\r\n       'Pub USGS', 'Pub Army Corps Eng', 'Pub Army Map', 'Pub Forest Serv',\r\n       'Pub Military Other', 'Pub Reclamation', 'Pub War Dept',\r\n       'Pub Bur Land Mgmt', 'Pub Natl Park Serv', 'Pub Indian Affairs',\r\n       'Pub EPA', 'Pub Tenn Valley Auth', 'Pub US Commerce', 'Keywords',\r\n       'Map Language', 'Scanner Resolution', 'Cell Name', 'Primary State Name',\r\n       'N Lat', 'W Long', 'S Lat', 'E Long', 'Link to HTMC Metadata',\r\n       'Download GeoPDF', 'View FGDC Metadata XML', 'View Thumbnail Image',\r\n       'Scan ID', 'GDA Item ID', 'Create Date'],\r\n      dtype='object')<\/pre>\n
The column we are interested in is named Download GeoPDF<\/i> as it contains the URLs to download the files.<\/p>\n
In [7]: csv[\"Download GeoPDF\"]<\/strong>\r\nOut[7]: \r\n0     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n1     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n2     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n3     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n4     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n5     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n6     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n7     https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n                            ...                        \r\n78    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n79    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n80    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n81    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n82    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n83    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n84    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n85    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n86    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n87    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\n88    https:\/\/prd-tnm.s3.amazonaws.com\/StagedProduct...\r\nName: Download GeoPDF, Length: 89, dtype: object<\/pre>\n
The reason I use Pandas for this step is that it gives me a simple and easy way to extract the URL column to a text file.<\/p>\n
In [9]: csv[\"Download GeoPDF\"].to_csv('urls.txt', header=None, index=None)<\/strong><\/pre>\n
This gives me a simple text file that has all of the URLs in it.<\/p>\n
https:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/DC\/250000\/DC_Washington_255916_1989_250000_geo.pdf\r\nhttps:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/DC\/250000\/DC_Washington_257785_1961_250000_geo.pdf\r\nhttps:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/DC\/250000\/DC_Washington_257786_1961_250000_geo.pdf\r\n\u2026\r\nhttps:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/WV\/250000\/WV_Bluefield_254032_1961_250000_geo.pdf\r\nhttps:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/WV\/250000\/WV_Bluefield_254033_1957_250000_geo.pdf\r\nhttps:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/WV\/250000\/WV_Bluefield_701712_1957_250000_geo.pdf\r\nhttps:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/WV\/250000\/WV_Bluefield_701713_1955_250000_geo.pdf\r\n<\/pre>\n
Finally, as there are usually multiple GeoPDF files that cover the same area, I download all of them so that I can go through and pick the best ones for my purposes. I try to find maps that are around the same data, are easily viewable, are not missing sections, and so on. To do this, I use the wget<\/a> command and use the text file I created as input like so.<\/p>\n
bmaddox@sdf1:\/mnt\/filestore\/temp\/blog$ wget -i urls.txt<\/strong> \r\n--2018-09-23 13:00:41--  https:\/\/prd-tnm.s3.amazonaws.com\/StagedProducts\/Maps\/HistoricalTopo\/PDF\/DC\/250000\/DC_Washington_255916_1989_250000_geo.pdf\r\nResolving prd-tnm.s3.amazonaws.com (prd-tnm.s3.amazonaws.com)... 52.218.194.10\r\nConnecting to prd-tnm.s3.amazonaws.com (prd-tnm.s3.amazonaws.com)|52.218.194.10|:443... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 32062085 (31M) [application\/pdf]\r\nSaving to: \u2018DC_Washington_255916_1989_250000_geo.pdf\u2019\r\n\u2026\r\n\u2026<\/pre>\n
Eventually wget will download all the files to the same directory as the text file. In the next installment, I will continue my workflow as I produce mosaic state maps using the historic topographic GeoPDFs.<\/p>\n","protected":false},"excerpt":{"rendered":"
My first professional job during and after college was working at the US Geological Survey as a software engineer and researcher. My job required me to learn about GIS and cartography, as I would do things from writing production systems … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"class_list":["post-520","post","type-post","status-publish","format-standard","hentry","category-gis"],"_links":{"self":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/520","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=520"}],"version-history":[{"count":8,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/520\/revisions"}],"predecessor-version":[{"id":533,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/520\/revisions\/533"}],"wp:attachment":[{"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=520"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=520"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/brian.digitalmaddox.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=520"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}