Linear Algebra( Common for EEE and ME)

Linear Algebra: Matrix algebra.

Introduction to Matrices

Matrix algebra is used quite a bit in advanced statistics, largely because it provides two benefits.

• Compact notation for describing sets of data and sets of equations.
• Efficient methods for manipulating sets of data and solving sets of equations.

This lesson introduces the matrix, the rectangular array at the heart of matrix algebra.

Matrix Definition

A matrix is a rectangular array of numbers arranged in rows and columns. The array of numbers below is an example of a matrix.
                                                    

	21	62	33	93
	44	95	66	13
	77	38	79	33

The number of rows and columns that a matrix has is called its dimension or its order. By convention, rows are listed first; and columns, second. Thus, we would say that the dimension (or order) of the above matrix is 3 x 4, meaning that it has 3 rows and 4 columns.
Numbers that appear in the rows and columns of a matrix are called elements of the matrix. In the above matrix, the element in the first column of the first row is 21; the element in the second column of the first row is 62; and so on.

Matrix Notation

Statisticians use symbols to identify matrix elements and matrices.

• Matrix elements. Consider the matrix below, in which matrix elements are represented entirely by symbols.
  

  	A11	A12	A13	A14
  	A21	A22	A23	A24

  By convention, first subscript refers to the row number; and the second subscript, to the column number. Thus, the first element in the first row is represented by A₁₁. The second element in the first row is represented by A₁₂. And so on, until we reach the fourth element in the second row, which is represented by A₂₄.
• Matrices. There are several ways to represent a matrix symbolically. The simplest is to use a boldface letter, such as A, B, or C. Thus, A might represent a 2 x 4 matrix, as illustrated below.
  

  A =	11 62 33 93 44 95 66 13

  Another approach for representing matrix A is:
  
  A = [ A_ij ] where i = 1, 2 and j = 1, 2, 3, 4
  This notation indicates that A is a matrix with 2 rows and 4 columns. The actual elements of the array are not displayed; they are represented by the symbol A_ij.

Other matrix notation will be introduced as needed. For a description of all the matrix notation used in this tutorial, see the Matrix Notation Appendix.

Matrix Equality

To understand matrix algebra, we need to understand matrix equality. Two matrices are equal if all three of the following conditions are met:

• Each matrix has the same number of rows.
• Each matrix has the same number of columns.
• Corresponding elements within each matrix are equal.

Consider the three matrices shown below.

A =     111   x y 444

B =     111 222 333 444

C =     l   m   n o   p   q

If A = B, we know that x = 222 and y = 333; since corresponding elements of equal matrices are also equal. And we know that matrix C is not equal to A or B, because C has more columns than A or B.

Test Your Understanding of This Lesson

Problem 1
The notation below describes two matrices - A and B

A = [ Aij ] where i = 1, 2, 3 and j = 1, 2

B =     111 222 333 444 555 666 777 888

Which of the following statements about A and B are true?

I. Matrix A has 5 elements.
II. The dimension of matrix B is 4 x 2.
III. In matrix B, element B₂₁ is equal to 222.

(A) I only
(B) II only
(C) III only
(D) All of the above
(E) None of the above

Solution
The correct answer is (E).

• Matrix A has 3 rows and 2 columns; that is, 3 rows, each with 2 elements. This adds up to 6 elements, altogether - not 5.
• The dimension of matrix B is 2 x 4 - not 4 x 2. That is, matrix B has 2 rows and 4 columns - not 4 rows and 2 columns.
• And, finally, element B₂₁ refers to the first element in the second row of matrix B, which is equal to 555 - not 222.

Types of Matrices

This lesson describes a few of the more important types of matrices: transpose matrices, vectors, and different kinds of square matrices.

Transpose Matrix

The transpose of one matrix is another matrix that is obtained by using by using rows from the first matrix as columns in the second matrix.
For example, it is easy to see that the transpose of matrix A is A'. Row 1 of matrix A becomes column 1 of A'; row 2 of A becomes column 2 of A'; and row 3 of A becomes column 3 of A'.

A =     111 222 333 444 555 666

A' =     111 333 555 222 444 666

Note that the order of a matrix is reversed after it has been transposed. Matrix A is a 3 x 2 matrix, but matrix A' is a 2 x 3 matrix.
With respect to notation, this web site uses a prime to indicate a transpose. Thus, the transpose of matrix B would be written as B'.

Vectors

Vectors are a type of matrix having only one column or one row.
Vectors come in two flavors: column vectors and row vectors. For example, matrix a is a column vector, and matrix a' is a row vector.

a =     11 12 33

a' =     11 22 33

We use lower-case, boldface letters to represent column vectors. And since the transpose of a column vector is a row vector, we use lower-case, boldface letters plus a prime to represent row vectors. Thus, vector b would be a column vector, and vector b' would be a row vector.

Square Matrices

A square matrix is an n x n matrix; that is, a matrix with the same number of rows as columns. In this section, we describe several special kinds of square matrix.

• Symmetric matrix. If the transpose of a matrix is equal to itself, that matrix is said to be symmetric. Two examples of symmetric matrices appear below.
  

  A = A' =

  1 2 2 3

  B = B' =

  5 6 7 6 3 2 7 2 1

  Note that each of these matrices satisfy the defining requirement of a symmetric matrix: A = A' and B = B'.
• Diagonal matrix. A diagonal matrix is a special kind of symmetric matrix. It is a symmetric matrix with zeros in the off-diagonal elements. Two diagonal matrices are shown below.
  

  A =

  1 0 0 3

  B =

  5 0 0 0 3 0 0 0 1

  Note that the diagonal of a matrix refers to the elements that run from the upper left corner to the lower right corner.
  
• Scalar matrix. A scalar matrix is a special kind of diagonal matrix. It is a diagonal matrix with equal-valued elements along the diagonal. Two examples of a scalar matrix appear below.
  

  A =

  3 0 0 3

  B =

  5 0 0 0 5 0 0 0 5

These square matrices play a prominent role in the application of matrix algebra to real-world problems. For example, a scalar matrix called the identity matrix is critical to the solution of simultaneous linear equations. (We cover the identity matrix later in the tutorial.)

Test Your Understanding of This Lesson

Problem 1
Consider the matrices shown below - a, A, B, and C

a =     1 2

A =     3 5 4 6

B =     3 4 5 6

C =     -1 0 0 6

Which of the following statements are true?

I.   a is a row matrix
II.  A = B'
III. C is a symmetric matrix

(A) I and II
(B) I and III
(C) II and III
(D) None of the above
(E) All of the above

Solution
The correct answer is (C), as explained below.

• Matrix a is a column vector, not a row matrix
• The transpose of a matrix is created by interchanging corresponding rows and columns. When this is done to matrix B, we see that A = B'.
  
  

  A =

  3 5 4 6

  B =

  3 4 5 6

  B' =

  3 5 4 6

  so A = B'

  
  
• The transpose of C is equal to C; that is C = C'. Therefore, C is a symmetric matrix.

Note that the off-diagonal elements of matrix C are equal to zero; so matrix C is a diagonal matrix, which is a special kind of symmetric matrix.

Matrix Addition and Matrix Subtraction

Just like ordinary algebra, matrix algebra has operations like addition and subtraction.

How to Add and Subtract Matrices

Two matrices may be added or subtracted only if they have the same dimension; that is, they must have the same number of rows and columns.
Addition or subtraction is accomplished by adding or subtracting corresponding elements. For example, consider matrix A and matrix B.

A =

1 2 3 7 8 9

B =

5 6 7 3 4 5

Both matrices have the same number of rows and columns (2 rows and 3 columns), so they can be added and subtracted. Thus,

A + B =

1 + 5 2 + 6 3 + 7 7 + 3 8 + 4 9 + 5

=

6 8 10 10 12 14

And,

A - B =

1 - 5 2 - 6 3 - 7 7 - 3 8 - 4 9 - 5

=

-4 -4 -4 4 4 4

And finally, note that the order in which matrices are added is not important; thus, A + B = B + A.

Test Your Understanding of This Lesson

Problem 1
Consider the matrices shown below - A, B, C, and D

A =     1 2

B =     3 5 4 6

C =     4 5 6 6

D =     -1 0 -2 0

Which of the following statements are true?

I.   A + B = C
II.  B + C = D
III. B - C = D

(A) I only
(B) II only
(C) III only
(D) I and II
(E) I and III

Solution
The correct answer is (C), as shown below.

B - C   =     [ 3 - 4 5 - 5 ] 4 - 6 6 - 6     =     [ -1 0 ] -2 0     =    D

Note that Matrices A and B cannot be added, because B has more columns than A. Matrices may be added or subtracted only if they have the same number of rows and the same number of columns.

Matrix Multiplication

In matrix algebra, there are two kinds of matrix multiplication: multiplication of a matrix by a number and multiplication of a matrix by another matrix.

How to Multiply a Matrix by a Number

When you multiply a matrix by a number, you multiply every element in the matrix by the same number. This operation produces a new matrix, which is called a scalar multiple.
For example, if x is 5, and the matrix A is:

A =	100 200 300 400

Then,

xA   =   5A   =    5

100 200 300 400

=

5 * 100 5 * 200 5 * 300 5 * 400

=

500 1000 1500 2000

=    B

In the example above, every element of A is multiplied by 5 to produce the scalar multiple, B.
Note: Some texts refer to this operation as multiplying a matrix by a scalar. (A scalar is a real number or a symbol representing a real number.)

How to Multiply a Matrix by a Matrix

The matrix product AB is defined only when the number of columns in A is equal to the number of rows in B. Similarly, the matrix product BA is defined only when the number of columns in B is equal to the number of rows in A.
Suppose that A is an i x j matrix, and B is a j x k matrix. Then, the matrix product AB results in a matrix C, which has i rows and k columns; and each element in C can be computed according to the following formula.

C_ik = Σ_j A_ijB_jk

where

C_ik = the element in row i and column k from matrix C
A_ij = the element in row i and column j from matrix A
B_jk = the element in row j and column k from matrix B
Σ_j = summation sign, which indicates that the a_ijb_jk terms should be summed over j

Let's work through an example to show how the above formula works. Suppose we want to compute AB, given the matrices below.

A =

0 1 2 3 4 5

B =

6 7 8 9 10 11

Let AB = C. Because A has 2 rows, we know that C will have two rows; and because B has 2 columns, we know that C will have 2 columns. To compute the value of every element in the 2 x 2 matrix C, we use the formula C_ik =  Σ_j A_ijB_jk, as shown below.

• C₁₁ = Σ A_1jB_j1 = 0*6 + 1*8 +2*10 = 0 + 8 + 20 = 28
• C₁₂ = Σ A_1jB_j2 = 0*7 + 1*9 +2*11 = 0 + 9 + 22 = 31
• C₂₁ = Σ A_2jB_j1 = 3*6 + 4*8 +5*10 = = 18 + 32 + 50 = 100
• C₂₂ = Σ A_2jB_j2 = 3*7 + 4*9 +5*11 = 21 + 36 +55 = 112

Based on the above calculations, we can say

AB   =   C =

28 31 100 112

What we did to compute Matrix C was not complicated. All we did was to multiply row elements in Matrix A by corresponding column elements in Matrix B.

Multiplication Order

As we have already mentioned, in some cases, matrix multiplication is defined for AB, but not for BA; and vice versa. However, even when matrix multiplication is possible in both directions, results may be different. That is, AB is not always equal to BA.
Because order is important, matrix algebra jargon has evolved to clearly indicate the order in which matrices are multiplied.

• To describe the matrix product AB, we can say A is postmultiplied by B; or we can say that B is premultiplied by A.
  
• Similarly, to describe the matrix product BA, we can say B is postmultiplied by A; or we can say that A is premultiplied by B.

The bottom line: when you multiply two matrices, order matters.

Identity Matrix

The identity matrix is an n x n diagonal matrix with 1's in the diagonal and zeros everywhere else. The identity matrix is denoted by I or I_n. Two identity matrices appear below.

I2 =     1 0 0 1

I3 =     1 0 0 0 1 0 0 0 1

The identity matrix has a unique talent. Any matrix that can be premultiplied or postmultiplied by I remains the same; that is:

AI = IA = A

Test Your Understanding of This Lesson

Problem 1
Consider the matrices shown below - A, B, and C

A =     a b c d

B =     e f g h

C =     w x y z

Assume that AB = C. Which of the following statements are true?

(A) w = a*e + b*h
(B) x = a*f + b*h
(C) y = c*g + d*h
(D) All of the above
(E) None of the above

Solution
The correct answer is (B). To compute the value of any element in matrix C, we use the formula C_ik = Σ_j A_ijB_jk.
In matrix C, x is the element in row 1 and column 2, which is represented in the formula by C₁₂. Therefore, to find x, we use the formula to calculate C₁₂, as shown below.

x = C₁₂ = Σ_j A_1jB_j2 = A₁₁B₁₂ + A₁₂B₂₂ = a*f + b*h

All of the other answers are incorrect.

Vector Multiplication

The multiplication of a vector by a vector produces some interesting results, known as the vector inner product and as the vector outer product.
Prerequisite: This material assumes familiarity with matrix multiplication.

Vector Inner Product

Assume that a and b are vectors, each with the same number of elements. Then, the inner product of a and b is s.

a'b = b'a = s

where
      a and b are column vectors, each having n elements,
      a' is the transpose of a, which makes a' a row vector,
      b' is the transpose of b, which makes b' a row vector, and
      s is a scalar; that is, s is a real number - not a matrix.
Note this interesting result. The product of two matrices is usually another matrix. However, the inner product of two vectors is different. It results in a real number - not a matrix. This is illustrated below.

a =

1 2 3

b =

4 5 6

Then,

a'b = 1*4 + 2*5 + 3*6 = 4 + 10 + 18 = 32

Thus, the inner product of a'b is equal to 32.
Note: The inner product is also known as the dot product or as the scalar product.

Vector Outer Product

Assume that a and b are vectors. Then, the outer product of a and b is C.

ab'= C

where
      a is a column vector, having m elements,
      b is a column vector, having n elements,
      b' is the transpose of b, which makes b' a row vector, and
      C is a rectangular m x n matrix
Unlike the inner product, the outer product of two vectors produces a rectangular matrix, not a scalar. This is illustrated below.

a =

v w

b =

x y z

Then,

C    =    ab'    =

v * x v * y v * z w * x w * y w * z

Notice that the elements of Matrix C consist of the product of elements from Vector A crossed with elements from Vector B. Thus, Matrix C winds up being a matrix of cross products from the two vectors.

Test Your Understanding of This Lesson

Problems
Consider the matrices shown below - a, b, and c

a =     0 1

b =     2 3

c =     4 5 6

Using a, b, and c, answer the questions below.

1. Find a'b, the inner product of a and b.
2. Find bc', the outer product of b and c.
3. True or false: bc' = cb'

Solutions

1. The term a'b is an inner product, which is equal to 3. The solution appears below.
   

   a'b   =     0 1     *     2 3     =     0*2 + 1*3    =    3

2. The term bc' is an outer product, which results in the 2 x 3 matrix D.
   

   bc'   =     2 3     *     4 5 6     =     2*4 2*5 2*6 3*4 3*5 3*6     =     8 10 12 12 15 18     =    D

3. The statement bc' = cb' is false.
   Note that b is a 2 x 1 vector and c is a 3 x 1 vector. Therefore, bc' is a 2 x 3 matrix, and cb' is a 3 x 2 matrix. Because bc' and cb' have different dimensions, they cannot be equal.
   
   
   

   Elementary Matrix Operations

   Elementary matrix operations play an important role in many matrix algebra applications, such as finding the inverse of a matrix and solving simultaneous linear equations.
   

   Elementary Operations

   There are three kinds of elementary matrix operations.
   
4. Interchange two rows (or columns).
5. Multiply each element in a row (or column) by a non-zero number.
6. Multiply a row (or column) by a non-zero number and add the result to another row (or column).

When these operations are performed on rows, they are called elementary row operations; and when they are performed on columns, they are called elementary column operations.

Elementary Operation Notation

In many references, you will encounter a compact notation to describe elementary operations. That notation is shown below.

	Operation description	Notation
Row operations	1. Interchange rows i and j	Ri <--> Rj
	2. Multiply row i by s, where s ≠ 0	sRi --> Ri
	3. Add s times row i to row j	sRi + Rj --> Rj
Column operations	1. Interchange columns i and j	Ci <--> Cj
	2. Multiply column i by s, where s ≠ 0	sCi --> Ci
	3. Add s times column i to column j	sCi + Cj --> Cj

Elementary Operators

Each type of elementary operation may be performed by matrix multiplication, using square matrices called elementary operators.
For example, suppose you want to interchange rows 1 and 2 of Matrix A. To accomplish this, you could premultiply A by E to produce B, as shown below.

R1 <--> R2    =	0 1 1 0		1 3 5 2 4 6	=	0 + 2 0 + 4 0 + 6 0 + 1 0 + 3 0 + 5	=	2 4 6 1 3 5
	E		A				B

Here, E is an elementary operator. It operates on A to produce the desired interchanged rows in B. What we would like to know, of course, is how to find E. Read on.

How to Perform Elementary Row Operations

To perform an elementary row operation on a A, an r x c matrix, take the following steps.

1. To find E, the elementary row operator, apply the operation to an r x r identity matrix.
2. To carry out the elementary row operation, premultiply A by E.

We illustrate this process below for each of the three types of elementary row operations.

• Interchange two rows. Suppose we want to interchange the second and third rows of A, a 3 x 2 matrix. To create the elementary row operator E, we interchange the second and third rows of the identity matrix I₃.
  

  1 0 0 0 1 0 0 0 1	⇒	1 0 0 0 0 1 0 1 0
  I3		E

  Then, to interchange the second and third rows of A, we premultiply A by E, as shown below.
  

  R2 <--> R3  =	1 0 0 0 0 1 0 1 0		0 1 2 3 4 5	=	1*0 + 0*2 + 0*4 1*1 + 0*3 + 0*5 0*0 + 0*2 + 1*4 0*1 + 0*3 + 1*5 0*0 + 1*2 + 0*4 0*1 + 1*3 + 0*5	=	0 1 4 5 2 3
  	E		A				B

• Multiply a row by a number. Suppose we want to multiply each element in the second row of Matrix A by 7. Assume A is a 2 x 3 matrix. To create the elementary row operator E, we multiply each element in the second row of the identity matrix I₂ by 7.
  

  1 0 0 1	⇒	1 0 0 7
  I2		E

  Then, to multiply each element in the second row of A by 7, we premultiply A by E.
  

  7R2 --> R2  =	1 0 0 7		0 1 2 3 4 5	=	1*0 + 0*3 1*1 + 0*4 1*2 + 0*5 0*0 + 7*3 0*1 + 7*4 0*2 + 7*5	=	0 1 2 21 28 35
  	E		A				B

• Multiply a row and add it to another row. Assume A is a 2 x 2 matrix. Suppose we want to multiply each element in the first row of A by 3; and we want to add that result to the second row of A. For this operation, creating the elementary row operator is a two-step process. First, we multiply each element in the first row of the identity matrix I₂ by 3. Next, we add the result of that multiplication to the second row of I₂ to produce E.
  

  1 0 0 1	⇒	1 0 0 + 3*1 1 + 3*0	⇒	1 0 3 1
  I2				E

  Then, to multiply each element in the first row of A by 3 and add that result to the second row, we premultiply A by E₂.
  

  3R1  +  R2 -->  R2  =	1 0 3 1		0 1 2 3	=	1*0 + 0*2 1*1 + 0*3 3*0 + 1*2 3*1 + 1*3	=	0 1 2 6
  	E		A				B

How to Perform Elementary Column Operations

To perform an elementary column operation on A, an r x c matrix, take the following steps.

1. To find E, the elementary column operator, apply the operation to an c x c identity matrix.
2. To carry out the elementary column operation, postmultiply A by E.

Let's work through an elementary column operation to illustrate the process. For example, suppose we want to interchange the first and second columns of A, a 3 x 2 matrix. To create the elementary column operator E, we interchange the first and second columns of the identity matrix I₂.

1 0 0 1	⇒	0 1 1 0
I2		E

Then, to interchange the first and second columns of A, we postmultiply A by E, as shown below.

C1 <--> C2    =	0 1 2 3 4 5		0 1 1 0	=	0*0 + 1*1 0*1 + 1*0 2*0 + 3*1 2*1 + 3*0 4*0 + 5*1 4*1 + 5*0	=	1 0 3 2 5 4
	A		E				B

Note that the process for performing an elementary column operation on an r x c matrix is very similar to the process for performing an elementary row operation. The main differences are:

• To operate on the r x c matrix A, the row operator E is created from an r x r identity matrix; whereas the column operator E is created from an c x c identity matrix.
  
• To perform a row operation, A is premultiplied by E; whereas to perform a column operation, A is postmultiplied by E.

Test Your Understanding of This Lesson

Problem 1
Assume that A is a 4 x 3 matrix. Suppose you want to multiply each element in the second column of matrix A by 9. Find the elementary column operator E.
Solution
To find the elementary column operator E, we multiply each element in the second column of the identity matrix I₃ by 9.

1 0 0 0 1 0 0 0 1	⇒	1 0 0 0 9 0 0 0 1
I3		E