‫Modeling Part four.
‫This is the final part of modeling and its comparison.
‫Wanting to know here is how will our model perform in the real world.
‫So after you've churned and improved your model's performance through hyper parameter tuning, it's
‫time to see how it performs on the test set.
‫The test set is like the final exam for machine learning models.
‫If you've created your data splits correctly, it should give you an indication on how your model will
‫perform once deployed in production.
‫That means customer facing rather than just being on your local computer.
‫Since your model has never seen data in the test set, evaluating your model on it is a good way to
‫see how it generalizes.
‫And remember, by generalizing, I mean adapts to data it hasn't seen before, such as how our Heart
‫Disease Prediction Machine Learning model would perform at classifying whether a patient has heart disease
‫or not on a patient who wasn't in our original data set.
‫A good model will yield similar results on the training, validation and test sets, and it's not uncommon
‫to see a slight decline in performance from the model on the training and validation set to the test
‫set.
‫For example, your model might achieve 98% accuracy on the training data set and 96% accuracy on the
‫test set.
‫What you should be worried about is if the training set performance is dramatically higher than the
‫test set, also known as unbefitting.
‫And if the test set performance is higher than the training set performance also known as overfitting.
‫Overfitting and unbefitting of both examples of a model not being able to generalize well, which is
‫what we don't want.
‫The ideal model shows up in the Goldilocks zone.
‫It fits just right.
‫Not too well, but not too poorly.
‫You see here this this machine learning model, if this was your data, these green data points, this
‫line here, it kind of fits the shape.
‫But it's it's this would be classified as as under fitting.
‫This is not what we want our model to do.
‫And this one over here, well, it's doing a good job of fitting all the data points, but it's going
‫far too close.
‫It's almost a too perfect model.
‫It's just snaking between them.
‫So this example would mean that the the model has learned the patterns too well in this data set.
‫It would be like seeing the final exam before actually taking the final exam.
‫This one here is the Goldilocks zone.
‫Right.
‫This is an iterative process where exactly this Goldilocks zone is like a balanced model.
‫We really depend on your data and the problem you're trying to solve.
‫That's why, again, it's it's an iterative process, finding this this balanced zone.
‫And now after some experience and practice working on different machine learning problems, you'll be
‫able to start to tell whether your model is overfitting or under fitting.
‫Now there are several reasons why unbefitting and overfitting can happen, but the main ones are data
‫leakage and data mismatch.
‫Data leakage happens when some of your test data leaks into your training data, and this often results
‫in overfitting or a model doing better on the test set than on the training data set.
‫It's like, remember, it was like if you were to have a look at the final exam or everyone had to look
‫at the final exam as the practice exam, your machine learning model has just learned what it's about
‫to be test on.
‫So when it comes time to modelling, it's just learned it way too well and it starts to fit the data
‫like this.
‫And this is why it's important to do your splits correctly and ensure that machine learning model training
‫happens only on the training data set.
‫Validation and model tuning happens only on the validation or training data set.
‫And that testing and model comparison happens on the test data set.
‫And I remember some different approaches.
‫Use only a training and testing set and do model tuning on the training set.
‫But testing always stays the same.
‫It's like when you when you go to university, you're doing a course.
‫You want to make sure that the final exam is kind of an indication of how well you understand things.
‫Same with the test data set for machine learning.
‫The test data set is used as an indication of how well your model will generalize in the real world.
‫So you want to avoid data leakage.
‫Data mismatch happens when the data you're testing on is different to the data you're training on,
‫such as having different features in the training data to the test data.
‫Having this kind of mismatch can lead to models performing poorly on test data compared to training
‫data.
‫This is why it's important to ensure that training is done on the same kind of data as you'll be testing
‫on and as close as possible to what you'll be using in your future applications.
‫Other ways to combat unbefitting include using a more advanced model.
‫This could mean a totally different model or increasing the number of hyper parameters on your current
‫model.
‫Remember when we're cooking our chicken dish, we might alter one of the hyper parameters of our oven
‫by turning it up.
‫That might be something that you might do on a machine learning model instead of only using two layers
‫in a neural network you might use for.
‫We'll see more of this in a future project.
‫You could also reduce the number of features you're trying to model.
‫Maybe your data has too many features, and the model you're using is struggling to find patterns in
‫them.
‫Finally, you could train your model for longer.
‫Sometimes models take longer to train or longer to learn than you'd expect.
‫One of your experiments may involve a longer training phase.
‫To reduce overfitting.
‫Useful solutions are to collect more data.
‫More data will provide more potential patterns for a model to find and thus lower the potential for
‫it to find them all.
‫Or you could try use a less advanced model.
‫This is uncommon, but it's possibility.
‫The model you're using is too good at learning and it models your data too well.
‫Be cautious of models performing too well as they might lead to incorrect predictions.
‫Remember, no model is perfect, so be sure to check your good results as much as you check your poor
‫results.
‫Finally, when comparing two different models to each other, it's important to ensure you're comparing
‫apples with apples and oranges with oranges, for example.
‫Model two Trained on data.
‫Set one versus Model three trained on data, set one.
‫And during comparison, you want to make sure you take into account not only the final result, but
‫what it took to get there.
‫If Model two takes 1/2 to make a prediction at 93.1% accuracy.
‫Model three takes 4 seconds to make a prediction at 94.7 accuracy.
‫Is that extra 3% accuracy worth that extra 3 seconds of prediction time?
‫Now, this will depend on what the goal is, but if you are optimizing for prediction time, you want
‫to make predictions as fast as possible.
‫You might choose model two because it makes predictions four times faster than Model three, but at
‫a higher accuracy level.
‫Again, this will be different depending on on what kind of application or production use case you want
‫to use.
‫But just some things to keep in mind is more than just how a model performs that goes into choosing
‫which one you should use.
‫A couple of things you want to remember from this lesson Avoid overfitting and unbefitting.
‫You want a model that heads towards generality.
‫It's like when you do your practice exam, if you saw the final exam, you might just become an expert
‫memorization machine rather than someone who could use their knowledge in the real world.
‫Keep the test set separate at all costs.
‫When you split your data, you want to have a training set and then throw away the test data set and
‫lock it up.
‫And once your model has been trained, then you can open up the test data set, you can unlock it,
‫take it out of the safe and see how your model performs when you're comparing models.
‫Compare apples to apples, have model one and model two on data, set one and data set one.
‫You want to make sure the two models you're comparing have been created in the same sort of environment
‫so that you can ensure that what you're comparing is is legitimate comparisons.
‫Finally, one best performance metric does not equal the best model.
‫Remember, in our example, you may be optimizing for prediction time.
‫So although a model that makes a faster prediction doesn't get as high accuracy as is another model
‫that takes a little bit longer.
‫It might not matter to you because you need something that can predict as fast as possible.
‫Phew, that was a lot, but we'll see plenty more of this in action throughout the course.
‫You'll also be using it throughout your entire machine learning career, so it's important to remember
‫these concepts.
‫Let's push on to the next step and see how we can put all of this previous steps together in step six
‫experimentation.