Some Thoughts on Creating Machine Learning Data Sets

My whole professional career (26 years now!) has been in slinging data around. Be it writing production systems for the government or making deep learning pipelines, I have made a living out of manipulating data. I have always taken pains to make sure what I do is correct and am proud of how obsessive I am about putting things together.

Lately I have been back to doing deep learning and computer vision work and have been looking at some of the open datasets that are out there for satellite imagery. Part of this work has involved trying to combine datasets to train models for object classification from various satellite imagery providers.

One thing I have noticed is that it is still hard to make a good deep learning dataset. We are only human, and we miss things sometimes. It is easy to misclassify things or accidentally include images that might not be a good candidate. Even ImageNet, one of the biggest computer vision and deep learning datasets out there, has been found to contain errors in the training data.

This got me thinking about putting together datasets and what “rules of thumb” I would use for doing so. As there do not seem to be as many articles out there about making datasets versus using them, I thought I would add my two cents on making a good machine learning dataset. This is by no means a criticism about the freely-available datasets out there now. In fact, we should all be thankful so many people are putting them out there, and I hope to add to these soon! So, in no particular order, here we go.

Limit the Number of Object Classes

One thing I have noticed is many datasets try to break out their object classes into too fine of detail. This looks to be especially true for satellite datasets. For example, one popular dataset breaks aircraft down into multiple classes ranging from propeller to trainer to jet aircraft. One problem with this approach is that it becomes easy to pick the wrong category while classifying them. Jet trainers are out there. Should that go into the jet category, or the trainer category? There are commercial propeller aircraft out there. What do we do with them?

It is one thing if you are purposely trying to differentiate different types of aircraft from satellite imagery. Even then, however, I would guess that a neural network would learn mostly the same features for each object class, and you would end up getting the aircraft category correct but have numerous misclassifications for the type. It will also be a lot easier to accidentally mislabel things from imagery while you are building the dataset. Now this is different if you are working with imagery to which only certain types of three letter agencies have access. But, most of us have to make due with the types of imagery that are freely available.

Verify Every Dataset Before Use

We all get into a hurry. We have deadlines, our projects are usually underfunded, and we spend large amounts of time putting out fires. It is still vitally important to not blindly use a dataset to train your model! Consider the below image. In one public domain dataset out there, around eight of these images are all labeled as boats.

Image of a car labeled as a boat.

This is a classical example of “we’re only human.” I would wager that a lot of datasets contain errors such as this. It is not that anyone is trying to purposely mislead you. It just happens sometimes. This is why it is essential to go through your dataset no matter how boring it might be. Labeling is a tough job, and there is probably a room in Hell that tortures people by making them label things all day and night. Again, everyone makes mistakes.

Clean up Your Datasets

Some datasets out there are derived from Google Earth. It is a source of high quality imagery and for now the terms seem to not require your firstborn or an oath of loyalty. The problem comes when you include things like the image below in your training set.

Here you can see that someone used an aircraft that had the Google Earth watermark superimposed on top of it. If you only have one or two, it likely will not be an issue. However, if you have a lot of things like this, then your network could learn features from the text and start to expect imagery to have this in it. This is an example of where you should practice due diligence in cleaning up your data before putting it into a machine learning dataset.

Avoid Clusters (Unless Looking for Them)

When extracting objects from Google Earth, you might come across something you consider a gold mine. Say you are extracting and labeling cars from imagery, and you come across a full parking lot. This seems like an easy way to suddenly add a lot of training data, and you might end up with a lot of images such as the one below.

In a dataset I recently worked with, there were several object classes that had a lot of images of the same types of objects right next to each other. If you do this, keep in mind that there could be some side effects from having a lot of clusters in your dataset.

  • It could (possibly) be helpful because your object class can present different views if they are present in a cluster. In the above, you can see the top of one bus and the back of another right beside it. This can aid in learning different features for that class.
  • A negative is that if you have a lot of clusters of objects in your training data, your classifier might lean towards detecting objects in groups instead of individual objects. If you are specifically looking for clusters then this is OK. If you want to search for individual ones, then it could hurt your accuracy.
  • The complexity of your training data could increase and lead to slow-downs during the training process by having more features present for each example object than would be present ordinarily.

In general, it is OK to have some clusters of objects in your data. Just be mindful that if you are looking for individual objects, you should try not to have a lot of clusters in your training data.

Avoid Duplicate Objects

This is true for objects in the same class or between object classes. If you have a lot of duplicates, you can run the risk of over-fitting during training. Problems can also arise if you, say, have a car in both a car object class and in a truck class. In this case you can end up with false detections because you accidentally matched the same features in multiple classes.

Pick Representative Training Data

If you are trying to train a model to detect objects from overhead imagery, you would not want to use a training set of pictures people have taken from their cellular phones. Similarly, if you want to predict someone’s mood from a high resolution camera, you would not want to train using fuzzy webcam images. This is stressed in numerous deep learning texts out there, but it is important to repeat. Check if the data you are using for training matches the data you intend to be running your model on. Fitness for use is important.

Invariance

The last thing I want to discuss here is what I feel is one of the most important aspects of deep learning: invariance. Convolutional neural networks are NOT invariant to scale and rotation. They can detect translations, but you do not get scale or rotational invariance by default. You have to provide this using data augmentation during your training phase.

Consider a dataset of numbers that you are using to train a model. This dataset will likely have the numbers written as you would expect them to be: straight up and down. If you train a model on this, it will detect numbers that match a vertical orientation, but accuracy will go down for anything that is off the vertical axis.

Frameworks like Keras make this easy by providing classes or functions that can randomly rotate, shear, or scale images during training before they are input into the network. This helps the classifier learn features in different orientations, something that is important for classifying overhead imagery.

Conclusion

In summary, these are just some general guidelines I use when I am putting together a dataset. I do not always get everything right, and neither will you. Get multiple eyes looking at your dataset and take your time. Labeling is a laborious, and if your attention drifts it is easy to get wrong. The better quality your training data, the better your model will perform.

Using Free Geospatial Tools and Data Part 2: Last of the History Lesson I Promise

As GPS became more popular, another problem came about. Map updates for these units were expensive. They were in proprietary formats and could only be used from a specific vendor for their units. Many units could not be updated at all. Even better, in some cases individual units could not share the same update. The updates were slow to incorporate all new areas. At one point, a popular GPS maker only employed a hand full of cartographers who were responsible for the entire world. As you can imagine, they could only do so much at any given time.

Some enterprising souls decided they wanted to try to convert their expensive data from their proprietary GIS packages and put them on their GPSs. After some reverse engineering they actually managed to get data into a format so they could put maps on their GPS units. Even then the GIS data sets were not current for everywhere as the vendors focused on popular areas.

This approach had a few problems. First, there were not that many people who had the skills and software available to do the GPS reverse engineering and convert the vendor data sets to use with them. The commercial GIS data were not necessarily any better than the GPS data in terms of being up to date. Plus, everything was proprietary. The GPS and GIS data were owned by vendors who would go after people if they even thought about making data available for free to anyone who had not bought a license. So even if you had access to the GIS data and tools, you could only update your own GPS and not post the files online for people to download. And you were likely violating some license agreement even if you only used it for yourself.

If fact, the vendors were very aggressive about protecting their data and took action against anyone who violated their copyright. Early web map companies would introduce errors into their data sets in an attempt to watermark them so they would know when someone was illegally using them. This was before the days of widespread in-car GPS units and Google or NAVTEQ cars driving around recording roads with GPS precision and accuracy. With closed software and data, no one really had to worry about accuracy for the casual user.

Open Source Comes to the Table

The frustrations with proprietary vendors and data sets started a small cottage industry of developers who wanted to give everyone access to the same types of tools that the commercial vendors had. In the commercial GIS space there were only one or two real sellers of GIS software. This monopoly led to stagnant development and large monolithic software programs. These Open Source developers wanted to write new tools that everyone could use to manipulate what free data was out there. They wrote libraries such as GDAL, libgeotiff, and others to provide access to the various file formats. The tools followed to allow users to do simple manipulations of geospatial data.

Now that people had tools, they wanted data to work on. Data at this time was scarce, mainly reposted USGS Digital Raster Graphics (scanned paper maps) and Digital Orthophotos (aerial photographs) in the raster data space. The US Census made their TIGER vector map data (think roads) available for download, but early on it had issues with spatial accuracy and was hard to work with unless the user spent time converting it to work in their GIS. The USGS also had some vector data for hydrography and transportation data but was also somewhat difficult to use due to the formats in which it was distributed.

People at this time had been doing various things with GPS units and early GIS tools to make data available. Some people posted GPS tracks of trails for others to go hiking on. Geocaching had caught on in a big way and introduced a lot of people to the convenience of a GPS. Moving map display GPS units allowed people to navigate roads without need for a paper map. More and more people began wanting up to date data for their devices, and they did not want to pay the expensive prices the commercial vendors wanted.

The explosion in GPS use and availability of Open Source tools to use it led to outcries of people who wanted more data so they could keep their GPS units up to date or just play with photos of their neighborhoods. Technology had evolved to the point where computers could more easily manipulate the large raster data sets that were out there. Eventually governments began to make more data available to the tax payers who felt they paid for it once so should not pay again to download it. The USGS made DRGs and DOQs available for free. The TIGER vector data got a lot more accurate and was updated on a much more regular basis. But still, a lot of the data was not current or up to date since there was only so much money spent by governments on mapping programs.

Along comes the OpenStreetMap (OSM) project in 2004 with the goal of creating a free base map of the world. OSM came out in the early years of the social media craze and provided a collaborative platform so people could add mapping data through either GPS traces or by volunteering their time to vectorize satellite photos. Suddenly, people all over the world could contribute to creating free maps of their areas and use the data however they wanted to use them. Combined with more and more governments providing their data free to download, we came to the modern era where we have more GIS data available than ever before.

Where are we Now?

To be superfluous, we’re now in a golden age of Open Source GIS tools and open data. The very capable QGIS application has recently hit version 2.0. OpenStreetMap continues to grow and is up to a compressed 30 gigabyte file with high-resolution user-contributed data under an open license. Toolkits and libraries such as GDAL power many Open Source and even some commercial applications. Many cars now come with GPS units built-into the dash. Cell phones with 3G+ data connections and mapping apps from Google, Apple, and others have caused traditional GPS companies such as Garmin to scramble to determine their future relevance. Anyone can now take Open Source tools and convert open data to update maps in their GPSs. Web map services using open standards make even more data available to web browsers and other applications. Times have gone from a scarcity of geospatial data to so much that management and discovery of it has become difficult due to the volume and number of providers.

Next time we’ll take a look at the Open Source tools of the trade that anyone can download and use.

Using Free Geospatial Tools and Data Part 1: Introduction and a History Lesson

This kicks off a series of posts about GIS tools and data. I’ve wanted to do something like this for a while now, but am finally forcing myself to do more writing to get back into the habit. So to start off, I’d like to go through a brief history lesson to discuss where GIS was and where it is today. I’ve had a long history in the GIS field, spending most of my professional career working for a government mapping agency where I wrote production systems and then transitioned into a research and development mode. So on to Part 1.

In days of old (OK, only around a decade or three ago), getting into the geospatial field could be a costly endeavor. Most data were locked up in proprietary vaults and you had to pay for access to them, if you could get access at all. The world had just started to transition from paper maps and traditional cartography to applying technology to mapping tasks. With the less advanced technology of the day, transcribing paper maps into a digital representation was a labor-intensive task and required many hours of work. In many cases, creating vector data involved someone drawing vectors on top of an on-screen image of a paper map (which sadly, is still one of the main methods in use today!).

There was also a lot of confusion in the early days about how to do digital mapping in the first place. Most of the people involved came from paper mapping and had spent decades learning how things worked in a physical world. The move to digital changed things. Accuracy in the paper world was no longer sufficient in digital, as what looked like it lined up on paper actually didn’t line up in digital. You could zoom in with a computer and see how far things actually were from lining up. There was a lot of resistance in the early years as mappers did not think using computers was “real” cartography.

As an example, consider how paper maps were printed. They used multiple Mylar plates that contained various parts of the map. When the map was printed, the plates would be inked and roll against the paper to lay down the layers (colors) of the map. These plates were aligned by the use of stud holes that were cut into the Mylar so they could be positioned correctly during the printing process. When these plates were scanned in and referenced, they did not line up. In the analog world, a gap of a millimeter or so would not really show up on the paper map. In digital, however, that gap will stand out like a sore thumb when the digital map separates are combined in a GIS.

 

Cartographer scribing a map separate Credit: U.S. Geological Survey Department of the Interior/USGS U.S. Geological Survey.

Cartographer scribing a map separate
Credit: U.S. Geological Survey
Department of the Interior

GIS tools themselves were expensive and were mainly only available from two vendors. Additionally, these tools were really only available on Windows and proprietary UNIX platforms such as Solaris and IRIX. Using them was difficult and required users to have an in-depth understanding of geospatial data and how to use the tools. Many people had to take week long vendor classes before they could use them. And of course, most of the tools had to be run from the command line.

Then things began to change. GPS was opened up to the public and for the first time mapping became dynamic for the non-GIS user. As GPS advanced, they became equipped with moving map displays so people could actually see where they were on a map and could see what was around them.

Early GPS, however, also suffered the same issues as early GIS systems. In the beginning they were bulky and only gave you your latitude and longitude on a digital display. This was OK for some people, as they could look at a paper map to reference where they were. They were expensive and only came from a handful of vendors. These early units had accuracy issues, the major being the intentional inaccuracy mandated by the US government who were afraid they could be used against America. They were very dependent on line of sight and took a while to get a general lock (which then meant they were accurate to the tens of feet instead of a foot or two).

 

Various GPS Receivers Credit: Wikipedia Photographer: Stefan Kühn

Various GPS Receivers
Credit: Wikipedia
Photographer: Stefan Kühn

 

GPS and GIS technologies began to evolve. GIS software got better and became more user friendly, making use of GUI technologies of the day. GPS technology evolved and became faster and smaller so that people could use hand held units to venture into the outdoors. GIS professionals began to try to hook up their software systems to a GPS so they could use their data with real-time measurements. Companies then got the bright idea to sell software so that you could hook a GPS to your laptop and then watch your location in real time on the screen. It was clunky and hard to use, but was the direct ancestor of every system we take for granted today.

Next time I’ll discuss the rise of Open Source GIS tools and data.