EE 345 | Learning Outcomes

Formulate linear least squares from Maximum Likelihood and Bayesian perspectives
Understand Bayesian linear least squares and ridge regression
Connect Ridge Regression with conditional means of multivariate Gaussians
Appreciate the distinction between geometry of regression and generalization
Some idea of significance of features ((t-)statistics)
Feature selection using LASSO

Referencing modules: Linear Regression

You are comfortable with the formal definition of vectors in \({\mathbb R}^n\)
You understand addition, scaling and linear combinations of vectors
You recall the parallelogram law of addition/subtractions
You can visualize simple sets of linear combinations geometrically (in 2 and 3 dimensions)
You understand how to compute the dot product and lengths of vectors
You understand how the dot product between vectors correlates with angle between them (in 2 and 3 dimensions)
You know how to write a system of linear equations that checks if a vector can be written as a linear combination of other vectors.
You can write code in python+numpy to do all the above

You can multiply matrices the inner product or outer product way
You understand matrix multiplication is the means to track linear combinations
You understand row/column operations as matrix multiplication
You understand the simple graph algorithms used to illustrate matrix multiplication

Referencing modules: Matrix Multiplication

You understand linear combinations of (column) vectors can be represented as a matrix times the vector of coefficients
You can set up a system of equations to check if a vector is a linear combination of other vectors
You understand the Elimination algorithm to get to the Reduced Row Echelon Form
You understand that you find a pivot in a column during Elimination iff the column is not a linear combination of prior columns
You understand how Elimination can solve a set of equations
You can figure out how many solutions a system of linear equations has
You can implement Elimination using code

Referencing modules: Gaussian Elimination

You understand the special case of elimination on square matrices with a full set of pivots
You understand what an inverse is
You understand square matrices with a full set of pivots have an inverse
You can find via elimination when inverses do not exists
You can find the inverse via elimination
You understand how to solve certain equations with an inverse
You can compute inverses of products of matrices

Referencing modules: Inverses

Linear independence highlights the properties pivot columns have
\(\{ {\bf v}_1, \ldots, {\bf v}_n \}\) are linearly independent if \(V= \begin{bmatrix} {\bf v}_1 & \cdots & {\bf v}_n \end{bmatrix}\) has a pivot in every column
The set \(\{ {\bf v}_1, \ldots, {\bf v}_n \}\) of linearly independent vectors in a linear space \({\mathcal L}\) is maximal for \({\mathcal L}\) if \(\{ {\bf v}_1, \ldots, {\bf v}_n, {\bf x}\}\) becomes linearly dependent for any other \(\bf x\) from the space \(\mathcal L\)
Every vector of a linear space can be written uniquely as a linear combination of vectors from a maximally independent set of the linear space
Maximal linearly independent sets of a linear space \(=\) basis
All bases/maximal linearly independent sets in a linear space have the same size
The size of a basis/maximal linearly independent set in a linear space is a fundamental property of the linear space, called its dimension

Referencing modules: Linear Independence

You understand what “closed under linear combinations” means
You know what a linear space is
You know the 4 fundamental spaces of a matrix: the row, column, left and right null spaces
\(\textrm{col}(A) = \{ A{\bf x} : \textrm{ all values of }{\bf x} \}\) (column)
\(\textrm{null}(A) = \{ {\bf x} : A{\bf x} = {\bf 0} \}\) (right null)
row space of \(A\) \(=\) col space of \(A^T\)
left null space of \(A\) \(=\) right null space of \(A^T\)
If \(A\) is \(m\times n\), number of coordinates of vectors in each of the spaces

Referencing modules: Linear Spaces

Basis = Maximal linearly independent set
Natural coordinate basis for \({\mathbb R}^n\): columns of \(I_n\)
Dimension of a linear space = Size of the basis
Identify basis for fundamental spaces of \(A\) using (pivot) rows/columns of \(A\), as well as non-zero rows/cols of rref(\(A\)) and rref(\(A^T\))
Dimension of all fundamental spaces of \(A\) in terms of rank and shape of \(A\)

Referencing modules: Bases and dimension

Look for functions from square matrices to numbers satisfying three axioms (linear function of any row, if all others are held fixed; changes sign if rows exchanged; identity matrix assigned 1)
Only one function satisfies all—that function is the determinant
If rows are linearly dependent, determinant is 0 (from axioms)
Determinant of lower/upper triangular or diagonal matrices: product of diagonal entries (from axioms)
determinant = product of pivots
Square \(A\) invertible iff determinant(\(A\)) \(=0\) (follows from previous)
determinant\((AB)\) = determinant\((A)\) determinant\((B)\) (from axioms)
determinant\((A)\) = determinant\((A^T)\) (from elimination written in matrix form)
Recursive formula for determinants (from axioms)
Cramer’s rule

Referencing modules: Determinants

You know how to compute eigenvalues and eigenvectors of a matrix
The set of all eigenvectors with the same eigenvalue (plus the all-0 vector) is a linear space, the eigenspace
Algebraic multiplicity (AM) and Geometric multiplicities (GM) of eigenvalues * Similar matrices
Specialization to symmetric matrices: real eigenvalues, AM=GM for all eigenvalues, distinct eigenvalues have orthogonal eigenspaces
Symmetric matrices: Orthonormal basis using eigenvectors

Referencing modules: Spectral Decomposition

Learning Outcomes What you will know