‫Alright.
‫Now we've done a little bit of EDA or exploratory data analysis.
‫We're going to do one more step and that is building a correlation matrix.
‫You might be thinking, what is a correlation matrix?
‫In layman's terms, it's a big table of numbers telling us how each independent variable is related
‫to each other.
‫Okay, these columns here, how do they interact with each other?
‫Are they a positive correlation or are they a negative correlation?
‫Now, let's not talk about it for much longer because we want to see it.
‫We want to see what is happening.
‫So let's go make a correlation matrix.
‫And to do so we can use DF score.
‫We hit shift tab here compute pairwise correlation of columns excluding any slash null values.
‫Well, pairwise correlation that's just basically saying compare each of these columns to each other.
‫Well, let's see it in action.
‫Hmm.
‫If you're looking at this for the first time, quite confusing because there's a lot of numbers going
‫on here.
‫Okay.
‫You can see the columns are on the rows as well as the columns and the diagonal.
‫Here is one one, one, one.
‫There's a lot of decimals and some are negative and some are positive.
‫Hmm.
‫Well, we might do, though.
‫Instead of staring at this and exploring it for a little bit, we might make this a little bit more
‫visual.
‫So let's make our correlation matrix a little prettier.
‫And we're going to do that using C Bourne's heat map.
‫So we'll set it up in a variable.
‫So call matrix equals dfki.
‫Wonderful.
‫And then we'll set up a figure and an X using what is it, plt dot subplots.
‫That's it.
‫Fig size.
‫I kind of know the fig size because I've seen this plot before, so I want it to be 15 width and height.
‫And then we're going to go x equals SAS.
‫Heat map.
‫Would help if our types map and then we're going to pass it core matrix and we want to annotate it.
‫So we'll set a not to equal true and we also want a line width line widths.
‫Yeah, I believe that's it of 0.5.
‫And then what else do we need.
‫There was a couple more things.
‫Format equals we only want two decimal places.
‫So see here, these are like six.
‫So two two is going to be enough for us and then see map to give it some nice colors.
‫We'll go yellow.
‫Green.
‫Blue.
‫Wonderful.
‫Let's.
‫Let's check it out.
‫Invalid syntax.
‫Oh.
‫Forgot the comma here.
‫Classic.
‫Wonderful.
‫Okay, so what is this showing us?
‫We've got some cut off things here.
‫That's that pesky map plot.
‫Lib 3.1.1 error.
‫But that's alright, we can fix it so we might go bottom.
‫You might not see that if you're using a later version of map plot lib than what I'm using at the moment.
‫We need to adjust the y axis limitations so we're going to get y limb wonderful and then we're going
‫to go x dot set.
‫William That's that's it.
‫That's how we change it.
‫We want to move the bottom up and we want to move the top down.
‫Does that make sense?
‫So bottom plus 0.5 and top -0.5, does this work?
‫Wonderful.
‫That's looking a little bit better.
‫It's not cut off.
‫And this little adjustment here, this is from the research that I found by just just searching for
‫it.
‫So map plot lib seaborne first.
‫I think that my history might have had it there in that plot.
‫Lib There we go first and last row cut off and it's an example of searching for your problem when you
‫have it right, when you run into an issue, it's just about trying to figure it out.
‫So they've got a similar problem to what we just had cut off.
‫And then come down here you can find this is the lines of code that I'm using here.
‫And in fact, they're making a correlation matrix as well.
‫Beautiful.
‫Yeah.
‫Unfortunately.
‫Map Plot lib 3.10.1 broke seaborne heat maps.
‫But that's alright.
‫We've got our heat map, we've got our correlation matrix again.
‫This is just the same version of this.
‫A more visual format.
‫I look at this and I understand this one.
‫A lot better than I do this one.
‫So what is actually going on here?
‫Well, we can see on the diagonal we've got a lot of wine values.
‫But what this means, the diagonal is we might zoom out one tick so we can see the diagonal.
‫This is giving a perfect correlation, which means the age column is exactly equal to the Age column,
‫which makes a whole bunch of sense.
‫Right.
‫And the sex column is completely equal to the sex column and so on and so on and so on, all the way
‫down the diagonal here.
‫But then we have some other.
‫A lot of other actually little cells here.
‫We can see on the side we've got zero up to one.
‫And some of them are positive, like this one here, and some of them are negative.
‫We just compared chest pain tight, which was CP to our target.
‫Let's investigate that.
‫What's the correlation here?
‫They've got CP it's got a positive correlation.
‫So 0.43, remember the maximum value is one.
‫But this is telling us how chest pain is related to the Target column.
‫Hmm.
‫Well, if we're looking at this and we did some research on a correlation matrix, I'm going to tell
‫you, because I've done this research myself.
‫A higher positive value means a potential positive correlation, and a higher negative value means a
‫potential negative correlation or a decrease.
‫What this is saying is as CP goes up.
‫The target value also increases.
‫Now.
‫Does that make sense?
‫If we come up here.
‫So if we look at this plot that we did before.
‫So as CP goes up, the target value also increases.
‫So remember, our target value is 010 for no disease, one for disease.
‫Now we're most interested here in the correlation, because if you looked at this, no disease is highest
‫at zero.
‫So that means no disease.
‫The target is zero.
‫And chest pain type is also zero.
‫Okay.
‫So according to our correlation matrix as CP goes up, because it's a positive correlation.
‫The target value should also go up.
‫So let's investigate.
‫Disease.
‫This is the one value and this is zero.
‫So blue is one.
‫Salmon is zero salmon.
‫What a beautiful color.
‫As our chest pain type goes up 012, three.
‫Does the blue column for one of target value go up?
‫If we look here, it does go up slightly and this drops off dramatically and then it goes up here.
‫It goes down here.
‫But the ratio of disease to no disease is still much higher.
‫So that's where the positive correlation comes from.
‫It comes from chest pain type increasing or LCP column as that increases our target column, meaning
‫disease or no disease also increases.
‫So that's a positive correlation.
‫That's why it's positive here.
‫Now let's have a look at one of the negative columns.
‫So you've got a negative correlation here.
‫It's about the same ratio, but just in a negative way.
‫That means if we look here x ng, we'll check our data dictionary for that in a second as x ng goes
‫down.
‫So should the target value.
‫Let's not take this correlation matrix word for it.
‫Let's reproduce this plot except using the x and column.
‫But before we do that, we might jump up here.
‫What is x ng?
‫Exercise induced angina.
‫Now, if you're wondering what angina is, we checked out that before.
‫A condition marked by severe pain in the chest.
‫So this means that if you exercise, do you get heart pain?
‫And let's have a look.
‫Exercise induced angina, one equals yes, zero equals no.
‫So do you get heart pain when you exercise?
‫If you do, that's a one.
‫And if you don't, that's a zero.
‫There's a negative correlation according to our correlation matrix.
‫I'm saying correlation a lot.
‫I know.
‫So x ng compared to the target value.
‫Do they have disease or not?
‫Is negatively correlated.
‫That means as Zion goes down, the target should also go down.
‫Now, there's only two values here for each of these.
‫So this is actually a pretty straightforward example.
‫So if we think about this based off what you've heard about in terms of heart disease.
‫If someone was telling you that when they exercise, they get chest pain so they would have one.
‫Would you predict closer to someone having heart disease or not?
‫If they had exercised induced chest pain, what would you be more inclined?
‫Do they have heart disease or they don't?
‫Well, for me, I would be more inclined to say, well, based off the rest of the data, if they're
‫getting exercised induced chest pain, there could be something going on with their heart.
‫Right?
‫There could be heart disease.
‫So this is where there's a negative correlation.
‫This is where the negative correlation comes into play.
‫Let's go down here.
‫Back to our correlation matrix.
‫There's a negative correlation of negative point 44.
‫So that means as X and goes down, the target value goes down based off what the core function is seeing
‫in our data.
‫Exercised induced angina.
‫When I compare that column to the Target column, I'm noticing that when X is lower, so is the Target
‫column.
‫Based off what we're looking here.
‫It's a remember it's only a 0.44 correlation.
‫It's not a perfect one, like negative one correlation, but just it's trending towards if someone didn't
‫have chest pain so it didn't have exercise to induce angina.
‫The trend is negative, so it's trending towards them not having heart disease.
‫Does that make sense?
‫And you could go through all of these values.
‫So this is pretty flat, like this is almost basically zero.
‫There's no correlation between FBS and the target.
‫What's our data dictionary telling us?
‫FBS.
‫So fasting blood sugar.
‫Hmm.
‫So this is saying is the fasting blood sugar above 120 milligrams per deciliter.
‫I believe that might be deciliter one equals two zero equals false.
‫So that's basically saying they're perfectly even.
‫There's no real positive or negative correlation.
‫It's just it is what it is.
‫And so that's the kind of insight that you can start to form with this correlation matrix.
‫And that's the kind of insight that we want to sort of figure out on our own if we want medical experts
‫and we could only work with the data that we have.
‫This is the purpose of EDA.
‫This is the purpose of what we've gone through here.
‫Comparing different variables, looking at the distribution.
‫Comparing more different variables here and this process.
‫We could keep going with this, right?
‫We could keep going until we really, really understand the data.
‫But.
‫What we want to do next is model driven EDA.
‫So what that means is building a machine learning model to derive insights of how.
‫These independent variables here contribute to the target variable.
‫Now correlation matrix is one of them, but we're more interested in trying to predict for the future.
‫The correlation matrix only deals with the data that we have.
‫We are more interested in building a machine learning model that has predictive power based on the data
‫that we have to predict on things in the future.
‫So that's what we're going to look at next.
‫If you're a little bit stuck here of what the correlation matrix is, don't worry.
‫It's a pretty in-depth concept to to understand.
‫But where you can probably go next is by just searching what is a correlation matrix?
‫So go here.
‫What is a correlation matrix?
‫You could go to this first link.
‫A correlation matrix is a table showing correlation coefficients between variables.
‫Well, that sounds like a good resource.
‫Start to read up on a little bit more.
‫Hey, that looks a little bit like the graph that we made, but we've covered most of the crux of a
‫correlation matrix.
‫It's really just going okay, looking at two variables and going, Is the relationship positive?
‫A.K.A as one increases, so does the other, or is the relationship negative?
‫As one decreases, so does the other.
‫Let's have a look at our workflow, see where we're up to.
‫We've been through some data analysis.
‫Now's time to get onto the machine learning in the modelling section.
‫Let's see where we're up to here.
‫If we look at our new project workflow, we've covered this, we've looked at some exploratory data
‫analysis using NumPy as a map plot lib.
‫Now it's time to get into the machine learning part of things.
‫Using Psychic Learn, we'll start to test our little theories about our what we've found out in the
‫exploratory data analysis.
‫We'll start to see does our machine learning model agree with what we found here?
‫Just exploring the data on our own.