4 min read programming

Statistical learning Week 2

Share

Regression models Introduction -

During the second week of the ISLR course, we are introduced to an example of statistical learning, where the sales of a product is modeled by the tv, newspaper and radio advertisements. Each variable is shown as a scatter plot of variable vs. sales, and the trend of each plot is summarized using a regression line. But ideally, we would like to know what the combined relationship is between the sales and tv + newspaper + radio. Sales is modeled as a function of tv, radio and newspaper. Sales here is y. TV, radio and newspaper is x1, x2 and x3 = vector X.

Our model can be represented as follows -

Y = f(X) + ε (y is a function of vector X plus error)

Y encompasses all samples in the data, while y denotes individual samples. The error term (ε) captures the measurement errors in y or other discrepancies, as our function of X can never model y perfectly.

Properties of f(X) -

  • with a good f, we can make good predictions of Y at newer points X=x
  • we can understand which features of X are important in explaning y
  • depending on the complexity of f, we can tell how each feature in X affects Y

If we generate a scatter plot of a single x and y, at one value x=4, there may be many values of y. But a function can take on only 1 value for y. So one good value would be the average values of those ys who have x=4. This can be written as follows -

f(4) = E(Y|X=4)

i.e., the function at value 4 is the expected value of y given x=4. Expected value is also known as conditional average.

Once we repeat this procedure for every value of x, we would be able to trace out a red curve over the scatter plot. This curve represents the regression function. This function gives us the conditional expectation of y given a value of x.

Nearest neighbor averaging -

If we consider a scatter plot of a single x and y, we may not have data corresponding to our desired x value all the time, which makes it difficult to compute E(Y|X = x). Therefore, we can relax the idea 'at the point X' to 'in the neighbhorhood of point X'. We can use a sliding window on either side of X=x, and take the average of the points falling within the window. We can use this approach to trace out a regression line. This is the idea of nearest neighbor algorithm. Nearest neighbhor algorithm works well for classification problems as well.

This algorithm -

  • is best for small no. of variables and large sample size N
  • does not do well in high dimensions because of 'curse of dimensionality'. Nearest neighbors tend to be far away in high dimensions.

We should get a good fraction of the N values of yi to bring the variance down: eg. 10% of the entire dataset. We should average the points in each neighbhorhood so that our estimate has a small variance. It is really hard to find near neighbhors in high dimensions and stay local. And this is the reason why we do not use nearest neighbhor algorithm for solving all problems.

To deal with this, we introduce structure to our models. Linear model for instance. We estimate the parameters of the model by fitting it to our data. A quadratic model would fit the data slightly better than the linear model.

Following are some of the tradeoffs while building models -

  1. Prediction accuracy vs. interpretability - linear models are easier to interpret, but more complex models like the thin plate splines are not
  2. We can have a good fit vs. over-fit or under-fit. We need to be able to select models which are of the right fit.

Bias variance trade-off -

Consider f̂(x), a model which is fit to the training data. Let's say x0 and y0 is a test observation drawn from the population, and we evaluate the model at this single test observation. True model is given by f(X). What is the expected difference b/w the true value of y0 and the value you get after fitting f̂(x) on x0?

The expected difference depends on 3 terms -

  1. Variability of the model. Depending on the training set used, model parameters may change
  2. Bias, and
  3. The random variation that comes in the new test point x0 (irreducible error).

The inability of the model to capture the true relationship of the data is called bias. Bias is the difference b/w the average prediction at x0 and the true f(x0). It refers to the error that is introduced by trying to approximate a complicated real-life problem using a simpler model.

As the flexibility of the f̂ increases, it's variance increases and its bias incresaes. So choosing the flexibility based on average test error amounts to a b-v-tradeoff. Both bias and variance shold be low. Flexible is better when n is large. When n is small, less flexible models are preferred.

Classification problems -

Goals of classification problems -

  • build a classifier that accurately predicts the class of a sample
  • assess the performance of the model (using misclassification rate)
  • understand the roles of different features among X

*k nearest neighbors in 2D*

If we pick a point on a graph and draw a circle of 1mm radius around it. The proportion of blue dots and yellow dots within the circle can be estimated. We can then assign probability of the picked point being blue/yellow based on the surrounding dots. If we repeat this for every dot on the graph, we obtain a decision bondary. The decision boundar is decided by each point on the graph. If k=1, you will get a very jagged line. But if k=100, boundary will be smoother.