Polynomial Regression Computations

This discussion covers the most crucial computational background for polynomial regression. You will see that this is only an extension of the simple- and multiple linear regression modeling covered in Module 2, Linear Regression, and Module 3, Multiple Linear Regression.

The objective of this section is to give you necessary formulae and tools to develop polynomial regression models. This includes the steps for testing the model parameter significance, as well as overall model significance and model assumptions.

Note: Please keep in mind that statements made here with respect to polynomial regression are also valid in other regression modeling.

The steps used in simple- and multiple linear regression are also used in polynomial regression modeling. In general, we assume in polynomial regression that the behavior of a dependent variable y can be explained by a polynomial, additive relationship between the dependent variable and a set of k independent variables (xj, j=1 to k). It is also assumed that the independent variables are independent of each other. In addition, it is assumed that the Ordinary Least Squares (OLS) assumptions hold.

The general form of a complete second degree polynomial regression model in two variables x1, x2 may be expressed as

The complete second degree polynomial model includes the linear terms x1 and x2, second degree terms x12 and x22, and the interaction term x1x2.

The estimated model may be written as

Note: Again, this model type is simply an assumption. This assumption is based, for example, on an analysis of residual plots of a multiple linear regression model. Recall the case in the previous module. Here we assume that there are two independent variables, and that there may also be some curvilinear relationships as well as some interaction present. We then proceed to develop a complete second degree polynomial model. We eliminate terms based on parameter tests like we did in the case in Module 3. We then rerun the model without non-significant parameters. The final model should contain only significant parameters.

Note: You can easily expand this model to a complete second degree polynomial model in three (or more) variables. For example, such a three variable model would include the linear terms x1, x2 and x3, the second degree terms x12, x22 and x32, as well as all two-way- and three-way interaction terms x1x2, x1x3 and x2x3 as well as x1x2x3. Similarly you can continue to expand the model to higher degree polynomial models.

Like before the model parameter estimators are found by minimizing the sum of squares of errors, SSE

In order to minimize this quadratic function SSE we take partial derivatives of the function with respect to each of the unknowns bj, for j=0,1,2,3,4,5. Then we set the partial derivatives equal to zero and solve the resulting system of linear equations simultaneously. Please notice that here the unknowns are the b's and not the x's.

The system of linear equations in (k+1=6) unknowns (b0,...,b5) becomes

Note: Don't give up just yet!! Please analyze the above system of linear equations. You should notice that there is a pattern with respect to the known elements (the sums relating to x) ---- the first element of the second column is the same as the first element of the second row; the second element of the third column is the same as the second element of the third row, and so on. The left side of the system of equations forms a symmetrical square matrix with respect to the elements, which can be determined from the data. You may also notice that the simple linear regression case is also embedded in the system ---- the square matrix consisting of the first two columns and rows, and the first two elements of the right-hand-side. You can also see that a two variable multiple linear regression model is embedded in the system of linear equations ---- the square matrix consisting of the first three columns and rows, and the first three elements of the right-hand-side. Once you know that this type of a pattern exists, you don't need to deal with the partial derivatives etc., for any multiple- or polynomial regression models, but just recognize the pattern and use it.

Note: Please recall again, that here bj's are the unknowns, and the x and y values come from the data.

The system can be represented using matrix notation with three matrices A, b and c as follows:

Note: Please notice the symmetry and the pattern.

The matrix (or vector) of unknowns b

The right-hand-side (RHS) matrix (or vector) c

Note: Please notice the pattern.

In the above matrices all elements of the matrices A and c are determined from the data. The system of linear equations in matrix form is solved for the vector b by the following matrix operations:

Teaser: Can you tell how many different regression models are embedded in the above complete second degree polynomial model?! The answer may surprise you. There are 31 different models. Please try to list all the models. (There are 5 one variable models, 10 two variable models, 10 three variable models, 5 four variable models and one five variable model. Models containing a significant 0 only do not count as regression models). With more variables the models become very rapidly very large and complex. I would recommend to favor smaller and simpler models over large models with many variables. Larger models always contain more unwanted interaction and multicollinearity.