On Friday, we derived the normal equations for continuous least squares regression. \[X^*X b = X^* f\] We used the standard polynomial basis \(\{1, x, x^2, \ldots, x^n\}\) to define the linear transformation \(X:\mathbb{R}^{n+1} \rightarrow L^2[a,b]\). This is not the best basis to use for continuous least squares regression. It would be better if we had a basis of polynomials \(\{\phi_0(x), \phi_1(x), \ldots, \phi_n(x)\}\) that are orthogonal to each other, that is such that \[\langle {\phi_i(x), \phi_j(x)} \rangle = \int_a^b \phi_i(x) \phi_j(x) \, dx = 0 ~~~ \text{ when } i \ne j.\] One such orthogonal basis is the Legendre polynomials on the interval \([-1,1]\).
The first three Legendre polynomials on \([-1,1]\) are \[L_0(x) = 1, ~~~ L_1(x) = x, ~~~ L_2(x) = \tfrac{3}{2}x^2 - \tfrac{1}{2}.\] The Legendre polynomials are constructed by applying the Gram-Schmidt algorithm to the standard basis.
With an orthogonal basis, the normal equations are much easier to solve. We want to find the best fit polynomial \[p(x) = b_0\phi_0(x) + b_1 \phi(x) + \ldots + b_n \phi_n(x).\] So \(p\) is a linear transformation of a vector of coefficients \(b \in \mathbb{R}^{n+1}\). Let \(X: \mathbb{R}^{n+1} \rightarrow L^2[a,b]\) denote this transformation.
For \(f \in L^2[a,b]\), the coefficients of the continuous least squares polynomial approximation \(p(x) = b_0 \phi_0(x) + b_1 \phi_1(x) + \ldots + b_n \phi_n(x)\)can be found by solving the normal equation \[X^*X b = X^* f\] where \(X^*X\) is an \((n+1)\)-by-\((n+1)\) matrix with entries \[(X^*X)_{ij} = \langle {\phi_i,\phi_j} \rangle = \int_a^b \phi_i(x) \phi_j(x) \, dx,\] and \(X^*f\) is a vector in \(\mathbb{R}^{n+1}\) with entries \[(X^*f)_i = \langle {\phi_i,f} \rangle = \int_a^b \phi_i(x) f(x) \, dx.\]
If we have an orthogonal basis, then \(X^*X\) has zero entries except on the main diagonal: \[X^* X = \begin{bmatrix} \|\phi_0\|^2 & & & \\ & \|\phi_1\|^2 & & \\ & & \ddots & \\ & & & \|\phi_n\|^2 \end{bmatrix}.\]
Therefore the solution of the normal equations is just \[b_i = \frac{\int_{a}^b \phi_i(x) f(x) \, dx}{\|\phi_i\|^2} \text{ for each } 0 \le i \le n.\]
We computed the 2nd degree continuous least squares regression polynomial on the interval \([-1,1]\) using the Legendre basis polynomials for the following functions.
\(f(x) = x^2\).
\(f(x) = e^x\).
The Legendre polynomials on the interval \([-1,1]\) aren’t the only example of an orthogonal set of functions. Probably the most important example of an orthogonal set of functions is the set \[\{\sin(x), \sin(2x), \sin(3x), \ldots \} \cup \{1,\cos(x), \cos(2x), \cos(3x),\ldots \}\] on the interval \([-\pi,\pi]\). Any function in \(L^2[-\pi,\pi]\) can be approximated by using continuous least squares with these trig functions. Since there are an infinite number of functions in this orthogonal set, we usually stop the approximation when we reach a high enough frequency \(\sin(nx)\) and \(\cos(nx)\).
Use the trig product formulas \[\begin{align*} \cos(\alpha) \cos(\beta) &= \tfrac{1}{2}[\cos(\alpha+\beta)+\cos(\alpha - \beta)] \\ \cos(\alpha) \sin(\beta) &= \tfrac{1}{2}[\sin(\alpha+\beta)+\sin(\beta - \alpha)] \\ \sin(\alpha) \cos(\beta) &= \tfrac{1}{2}[\sin(\alpha+\beta)+\sin(\alpha - \beta)] \\ \sin(\alpha) \sin(\beta) &= \tfrac{1}{2}[\cos(\alpha-\beta)-\cos(\alpha + \beta)] \end{align*}\] to prove that the functions \(\sin(kx)\) and \(\cos(mx)\) are orthogonal in \(L^2[-\pi,\pi]\) for any \(k, m \in \mathbb{Z}\).
Use the trig product formulas to calculate \(\|\sin(kx)\|^2\) for every \(k \in \mathbb{N}\).
Use the trig product formulas to calculate \(\|\cos(mx)\|^2\) for every \(m \in \mathbb{N}\).
The same formula we derived yesterday applies to any orthogonal family of functions. Thus any \(f \in L^2[-\pi,\pi]\) can be approximated by \[f(x) \approx \sum_{k = 1}^n \frac{\int_{-\pi}^\pi f(x) \sin(kx) \, dx}{\|\sin(kx)\|^2} \sin(kx) + \sum_{m = 0}^n \frac{\int_{-\pi}^\pi f(x) \cos(mx) \, dx}{\|\cos(mx)\|^2} \cos(mx).\]
This formula is the Fourier series for \(f(x)\).