In the last session, I described what ML is and how ML models aren't programmed with traditional logic, like if, this, then, that. Instead, you program them by giving them examples and letting them come up with the logic. In this lesson, I describe what this means for the data we collect and use to train our ML models. Remember, training the ML model simply means showing it examples labeled with what you want to predict. The main thing to know is that for machine learning, your models will only be as good as your data is, and more often than not, you need a lot of data. For our example, you'll need a large data set of historical examples of rejected parts and parts in good condition, in order to train a model to categorize parts as defective or not. The basic reason why ML models need high-quality data is because they don't have human general knowledge. Data is the only thing they have access to. That's why data is part of my definition, it is one of the most important ingredients in our recipe. To understand why data is important to ML, it's helpful to compare ML to traditional software development. In traditional software development, a bug is a mistake in the code that causes unexpected or undesired behavior. In ML, while there can be bugs in the implementation of the standard algorithm that you're using, it is much more common that the bugs are in the data. You often fix errors in an ML model by showing it more examples to increase coverage or by correcting the labels on mislabeled data to improve data quality. Like humans, ML algorithms make mistakes too. Consider this example. A few years ago, some Googlers wanted to use ML to help diagnose diabetic retinopathy, which is the fastest growing cause of blindness, potentially affecting more than 415 million diabetic patients worldwide. They trained an ML model to diagnose diabetic retinopathy based on images of the back of eyes. Working closely with doctors in the US and India, they created an ML model that would diagnose diabetic retinopathy with performance equal to that of ophthalmologists. In the graph on the right, the model's performance is depicted in the black line while the professional ophthalmologists' performance is depicted with colored dots. What's important to note is that firstly, neither humans nor the models exhibit perfect performance. These data show values that are less than 100 percent. These are exciting results, they are close to 100 percent, but there is still a lot of work to improve the diagnosis. So data is important, but that doesn't mean that any old data will do. The best data has three qualities, it has coverage, it's clean, and it's complete. You can apply these three ideas to determine whether your data is good enough for ML. When I say coverage I mean coverage of all the possible scenarios or all the possible inputs and outputs. Better coverage results in a better ML model. So think back to our manufacturing use case. The data you use to train the model is from only red parts. The model may not realize that the same visual signs of a defect apply to blue parts also. Calculating the coverage of your data is somewhat complex. So instead, most people use data quantity as a proxy. When you have more data and broader coverage, your ML model would be more accurate. Another important factor related to how much data you need is cleanliness. Quite often, the data you have will have errors in it. What kind of errors? Perhaps you are training on images of manufacturing parts, but some of your images have shadows in them, maybe things are mislabeled in your data. You have parts that are marked as being fractured, but actually they were discarded because their size was wrong. Remember, the retinal images, there are lots of images where different doctors could not agree. That's probably because the data had issues, maybe the image was blurry, or maybe there was a speck of dirt on the lens when the picture was taken. The wrong labels on such images will cause data cleanliness issues. Dirty data can make machine learning hard. The more wrong data you have, the more correct data you will need to provide to counterbalance, so that the ML model doesn't learn the wrong thing. Dirty data could also happen because of human error. Maybe when sales data was entered, it was entered wrongly. Dirty data can happen because of software bugs, maybe the transaction was recorded incorrectly every time the register ran out of paper. Can you think of dirty data that exists in your business? Don't worry, unclean data is actually the norm. Most businesses have issues with data cleanliness. It's important that you understand and correct issues with data cleanliness. I often tell people that if someone tells you there are no issues with data cleanliness, you probably aren't talking to the right person about your data. Another factor to consider related to your data is completeness. Incomplete data is data that doesn't contain enough information about the world to replace human knowledge. Incomplete data can be the result of lack of availability of better data, mistaken expectations about how ML works and what it is capable of, or poor execution of program design and implementation. Let's think back to our manufacturing example. Imagine that one of the major sources of defects is overheating, but you aren't collecting temperature data. That's an example of incomplete data. Even if you start collecting data now for temperature, you might not have enough historical examples of temperature data. What you have to remember is data is the tunnel through which your model views the world. Anything the model can't see, it assumes does not exist. For example, a model that looked at this picture but only saw the left side might think that the road was open and traffic free, when in reality it's just closed. Conversely, if you only had examples from customers in a particular region, that would be a coverage issue with the data. Coverage versus completeness can be confusing. If you think of a table of data where each row is an example and each column is some information about that example, coverage is about your rows not covering all scenarios. Completeness is about not having all of the columns you need. The good news is that most of these problems can be solved simply by getting more data, but you have to be purposeful in collecting that data. Do you need to improve coverage, improve quality, improve completeness? Only have examples for a few types of parts in your manufacturing business? Get more data to increase coverage. Have data that is somewhat unreliable or unstructured? Get more data and the model will tease out the signal from the noise. Maybe you realize that the location is crucial for determining pricing? Get location data to make your data complete. So as you start working on ML projects, be sure to review the data you have and check for any issues. Pay special attention to coverage, quality, and completeness. Remember, data is a key ingredient in our recipe for ML.
