Matrix Algebra Continued

Echelon Form of a Matrix

This lesson introduces the concept of an echelon matrix. Echelon matrices come in two forms: the row echelon form (ref) and the reduced row echelon form (rref).

Row Echelon Form

A matrix is in row echelon form (ref) when it satisfies the following conditions.

• The first non-zero element in each row, called the leading entry, is 1.
• Each leading entry is in a column to the right of the leading entry in the previous row.
• Rows with all zero elements, if any, are below rows having a non-zero element.

Each of the matrices shown below are examples of matrices in row echelon form.

1 2 3 4     0 0 1 3     0 0 0 1		1 2 3 4 0 0 1 3 0 0 0 1 0 0 0 0		1 2 0 1 0 0
Aref		Bref		Cref

Reduced Row Echelon Form

A matrix is in reduced row echelon form (rref) when it satisfies the following conditions.

• The matrix satisfies conditions for a row echelon form.
• The leading entry in each row is the only non-zero entry in its column.

Each of the matrices shown below are examples of matrices in reduced row echelon form.

1 2 0 0 0 0 1 0 0 0 0 1		1 2 0 0 0 0 1 0 0 0 0 1 0 0 0 0		1 0 0 1 0 0
Arref		Brref		Crref

Test Your Understanding of This Lesson

Problem 1
Which of the following matrices is in row echelon form?

0 1 1 0 0 0	1 2 0 1 0 0	1 2 0 1 0 1	1 0 0 0 0 1
A	B	C	D

(A) Matrix A
(B) Matrix B
(C) Matrix C
(D) Matrix D
(E) None of the above

Solution
The correct answer is (B), since it satisfies all of the requirements for a row echelon matrix. The other matrices fall short.

• The leading entry in Row 1 of matrix A is to the right of the leading entry in Row 2, which is inconsistent with definition of a row echelon matrix.
• In matrix C, the leading entries in Rows 2 and 3 are in the same column, which is not allowed.
• In matrix D, the row with all zeros (Row 2) comes before a row with a non-zero entry. This is a no-no.

Problem 2
Which of the following matrices are in reduced row echelon form?

1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0	1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1	0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0
A	B	C

(A) Only matrix A
(B) Only matrix B
(C) Only matrix C
(D) All of the above
(E) None of the above

Solution
The correct answer is (D), since each matrix satisfies all of the requirements for a reduced row echelon matrix.

• The first non-zero element in each row, called the leading entry, is 1.
• Each leading entry is in a column to the right of the leading entry in the previous row.
• Rows with all zero elements, if any, are below rows having a non-zero element.
• The leading entry in each row is the only non-zero entry in its column.

Changing a Matrix Into Echelon Form

This lesson shows how to convert a matrix to its row echelon form and to its reduced row echelon form.

Echelon Forms

A matrix is in row echelon form (ref) when it satisfies the following conditions.

• The first non-zero element in each row, called the leading entry, is 1.
• Each leading entry is in a column to the right of the leading entry in the previous row.
• Rows with all zero elements, if any, are below rows having a non-zero element.

A matrix is in reduced row echelon form (rref) when it satisfies the following conditions.

• The matrix is in row echelon form (i.e., it satisfies the three conditions listed above).
• The leading entry in each row is the only non-zero entry in its column.

A matrix in echelon form is called an echelon matrix. Matrix A and matrix B are examples of echelon matrices.

1 2 3 4 0 0 1 3 0 0 0 1 0 0 0 0		1 2 0 0 0 0 1 0 0 0 0 1 0 0 0 0
A		B

Matrix A is in row echelon form, and matrix B is in reduced row echelon form.

How to Transform a Matrix Into Its Echelon Forms

Any matrix can be transformed into its echelon forms, using a series of elementary row operations. Here's how.

1. Pivot the matrix
   1. Find the pivot, the first non-zero entry in the first column of the matrix.
   2. Interchange rows, moving the pivot row to the first row.
   3. Multiply each element in the pivot row by the inverse of the pivot, so the pivot equals 1.
   4. Add multiples of the pivot row to each of the lower rows, so every element in the pivot column of the lower rows equals 0.
      
2. To get the matrix in row echelon form, repeat the pivot
   1. Repeat the procedure from Step 1 above, ignoring previous pivot rows.
   2. Continue until there are no more pivots to be processed.
      
3. To get the matrix in reduced row echelon form, process non-zero entries above each pivot.
   1. Identify the last row having a pivot equal to 1, and let this be the pivot row.
   2. Add multiples of the pivot row to each of the upper rows, until every element above the pivot equals 0.
   3. Moving up the matrix, repeat this process for each row.

Transforming a Matrix Into Its Echelon Forms: An Example

To illustrate the transformation process, let's transform Matrix A to a row echelon form and to a reduced row echelon form.

0 1 2 1 2 1 2 7 8	⇒	1 2 1 0 1 2 2 7 8	⇒	1 2 1 0 1 2 0 3 6	⇒	1 2 1 0 1 2 0 0 0	⇒	1 0 -3 0 1 2 0 0 0
A		A1		A2		Aref		Arref

To transform matrix A into its echelon forms, we implemented the following series of elementary row operations.

1. We found the first non-zero entry in the first column of the matrix in row 2; so we interchanged Rows 1 and 2, resulting in matrix A₁.
2. Working with matrix A₁, we multiplied each element of Row 1 by -2 and added the result to Row 3. This produced A₂.
3. Working with matrix A₂, we multiplied each element of Row 2 by -3 and added the result to Row 3. This produced A_ref. Notice that A_ref is in row echelon form, because it meets the following requirements: (a) the first non-zero entry of each row is 1, (b) the first non-zero entry is to the right of the first non-zero entry in the previous row, and (c) rows made up entirely of zeros are at the bottom of the matrix.
4. And finally, working with matrix A_ref, we multiplied the second row by -2 and added it to the first row. This produced A_rref. Notice that A_rref is in reduced row echelon form, because it satisfies the requirements for row echelon form plus each leading non-zero entry is the only non-zero entry in its column.

Note: The row echelon matrix that results from a series of elementary row operations is not necessarily unique. A different set of row operations could result in a different row echelon matrix. However, the reduced row echelon matrix is unique; each matrix has only one reduced row echelon matrix.

Test Your Understanding of This Lesson

Problem 1
Consider the matrix X, shown below.

X  =	0 1 1 2 0 5

Which of the following matrices is the reduced row echelon form of matrix X ?

0 1 1 0 0 0	1 0 0 1 0 0	1 0 0 1 0 1	1 0 0 0 0 1
A	B	C	D

(A) Matrix A
(B) Matrix B
(C) Matrix C
(D) Matrix D
(E) None of the above

Solution
The correct answer is (B). The elementary row operations used to change Matrix X into its reduced row echelon form are shown below.

0 1 1 2 0 5	⇒	1 2 0 1 0 5	⇒	1 2 0 1 0 0	⇒	1 0 0 1 0 0
X		X1		X2		Xrref

To change X to its reduced row echelon form, we take the following steps:

1. Interchange Rows 1 and 2, producing X₁.
2. In X₁, multiply Row 2 by -5 and add it to Row 3, producing X₂.
3. In X₂, multiply Row 2 by -2 and add it to Row 1, producing X_rref.

Note: Matrix A is not in reduced row echelon form, because the leading entry in Row 2 is to the left of the leading entry in Row 3; it should be to the right. Matrix C is not in reduced row echelon form, because column 2 has more than one non-zero entry. And finally, matrix D is not in reduced row echelon form, because Row 2 with all zeros is followed by a row with a non-zero element; all-zero rows must follow non-zero rows.

Vector Dependence

This lesson introduces the topic of vector dependence. One vector is dependent on other vectors, if it is a linear combination of the other vectors.

Linear Combination of Vectors

If one vector is equal to the sum of scalar multiples of other vectors, it is said to be a linear combination of the other vectors.
For example, suppose a = 2b + 3c, as shown below.

11 16	=	2	1 2	+	3	3 4	=	2*1 + 3*3 2*2 + 3*4
a			b			c

Note that 2b is a scalar multiple and 3c is a scalar multiple. Thus, a is a linear combination of b and c.

Linear Dependence of Vectors

A set of vectors is linearly independent if no vector in the set is (a) a scalar multiple of another vector in the set or (b) a linear combination of other vectors in the set; conversely, a set of vectors is linearly dependent if any vector in the set is (a) a scalar multiple of another vector in the set or (b) a linear combination of other vectors in the set.
Consider the row vectors below.

a =	1 2 3		d =	2 4 6
b =	4 5 6		e =	0 1 0
c =	5 7 9		f =	0 0 1

Note the following:

• Vectors a and b are linearly independent, because neither vector is a scalar multiple of the other.
• Vectors a and d are linearly dependent, because d is a scalar multiple of a; i.e., d = 2a.
• Vector c is a linear combination of vectors a and b, because c = a + b. Therefore, the set of vectors a, b, and c is linearly dependent.
• Vectors d, e, and f are linearly independent, since no vector in the set can be derived as a scalar multiple or a linear combination of any other vectors in the set.

Test Your Understanding of This Lesson

Problem 1
Consider the row vectors shown below.

0 1 2	3 2 1	3 3 3	3 4 5
a	b	c	d

Which of the following statements are true?

(A) Vectors a, b, and c are linearly dependent.
(B) Vectors a, b, and d are linearly dependent.
(C) Vectors b, c, and d are linearly dependent.
(D) All of the above.
(E) None of the above.

Solution
The correct answer is (D), as shown below.

• Vectors a, b, and c are linearly dependent, since a + b = c.
• Vectors a, b, and d are linearly dependent, since 2a + b = d.
• Vectors b, c, and d are linearly dependent, since 2c - b = d.
  
  
  

  Matrix Rank

  This lesson introduces the concept of matrix rank and explains how the rank of a matrix is revealed by its echelon form.
  

  The Rank of a Matrix

  You can think of an r x c matrix as a set of r row vectors, each having c elements; or you can think of it as a set of c column vectors, each having r elements.
  The rank of a matrix is defined as (a) the maximum number of linearly independent column vectors in the matrix or (b) the maximum number of linearly independent row vectors in the matrix. Both definitions are equivalent.
  For an r x c matrix,
  
• If r is less than c, then the maximum rank of the matrix is r.
• If r is greater than c, then the maximum rank of the matrix is c.

The rank of a matrix would be zero only if the matrix had no elements. If a matrix had even one element, its minimum rank would be one.

How to Find Matrix Rank

In this section, we describe a method for finding the rank of any matrix. This method assumes familiarity with echelon matrices and echelon transformations.
The maximum number of linearly independent vectors in a matrix is equal to the number of non-zero rows in its row echelon matrix. Therefore, to find the rank of a matrix, we simply transform the matrix to its row echelon form and count the number of non-zero rows.
Consider matrix A and its row echelon matrix, A_ref. Previously, we showed how to find the row echelon form for matrix A.

0 1 2 1 2 1 2 7 8	⇒	1 2 1 0 1 2 0 0 0
A		Aref

Because the row echelon form A_ref has two non-zero rows, we know that matrix A has two independent row vectors; and we know that the rank of matrix A is 2.
You can verify that this is correct. Row 1 and Row 2 of matrix A are linearly independent. However, Row 3 is a linear combination of Rows 1 and 2. Specifically, Row 3 = 3*( Row 1 ) + 2*( Row 2). Therefore, matrix A has only two independent row vectors.

Full Rank Matrices

When all of the vectors in a matrix are linearly independent, the matrix is said to be full rank. Consider the matrices A and B below.

A =

1 2 3 2 4 6

B =

1 0 2 2 1 0 3 2 1

Notice that row 2 of matrix A is a scalar multiple of row 1; that is, row 2 is equal to twice row 1. Therefore, rows 1 and 2 are linearly dependent. Matrix A has only one linearly independent row, so its rank is 1. Hence, matrix A is not full rank.
Now, look at matrix B. All of its rows are linearly independent, so the rank of matrix B is 3. Matrix B is full rank.

Test Your Understanding of This Lesson

Problem 1
Consider the matrix X, shown below.

X =	1 2 4 4 3 4 8 0

What is its rank?

(A) 0
(B) 1
(C) 2
(D) 3
(E) 4

Solution
The correct answer is (C). Since the matrix has more than zero elements, its rank must be greater than zero. And since it has fewer rows than columns, its maximum rank is equal to the maximum number of linearly independent rows. And because neither row is linearly dependent on the other row, the matrix has 2 linearly independent rows; so its rank is 2.
Problem 2
Consider the matrix Y, shown below.

Y =	1 2 3 2 3 5 3 4 7 4 5 9

What is its rank?

(A) 0
(B) 1
(C) 2
(D) 3
(E) 4

Solution
The correct answer is (C). Since the matrix has more than zero elements, its rank must be greater than zero. And since it has fewer columns than rows, its maximum rank is equal to the maximum number of linearly independent columns.
Columns 1 and 2 are independent, because neither can be derived as a scalar multiple of the other. However, column 3 is linearly dependent on columns 1 and 2, because column 3 is equal to column 1 plus column 2. That leaves the matrix with a maximum of two linearly independent columns; e.g., column 1 and column 2. So the matrix rank is 2.

Matrix Determinants

The determinant is a unique number associated with a square matrix. In this lesson, we introduce notation for a determinant and we show how to compute the determinant for any square matrix.

Notation for a Determinant

There are at least three ways to denote the determinant of a square matrix.

• Denote the determinant by vertical lines around the matrix name; thus, the determinant of matrix A would be indicated by |A|.
• Another approach is to enclose matrix elements within vertical straight lines, as shown below.

|A|  =

A11 A12 A13 A21 A22 A23 A31 A32 A33

• And finally, some references refer to the deteriminant of A as Det A. Thus, |A| = Det A.

On this web site, we will use the first option; that is, we will refer to the determinant of A as |A|.

How to Compute the Determinant of a 2 x 2 Matrix

Suppose A is a 2 x 2 matrix with elements A_ij, as shown below.

A  =	A11 A12 A21 A22

We compute the determinant of A according to the following formula.

|A| = ( A₁₁ * A₂₂ ) - ( A₁₂ * A₂₁ )

How to Compute the Determinant of an n x n Matrix

The formula for computing the determinant of a 2 x 2 matrix (shown above) is actually a special case of the general algorithm for computing the determinant of any square matrix.

|A| = Σ ( + ) A_1qA_2rA_3s . . . A_nz

This algorithm requires some explanation. Here are the key points.

• The determinant is the sum of product terms made up of elements from the matrix.
• Each product term consists of n elements from the matrix.
• Each product term includes one element from each row and one element from each column.
• The number of product terms is equal to n! (where n! refers to n factorial).
• By convention, the elements of each product term are arranged in ascending order of the left-hand (or row-designating) subscript.
• To find the sign of each product term, we count the number of inversions needed to put the right-hand (or column-designating) subscripts in numerical order. If the number of inversions is even, the sign is positive; if odd, the sign is negative.

Unless you are a computer, this explanation is probably still confusing, so let's work through an example. Suppose A is a 3 x 3 matrix with elements A_ij, as shown below.

A  =	A11 A12 A13 A21 A22 A23 A31 A32 A33

To begin, let's list each product term. In constructing this list, we will arrange elements of each product term in ascending order of their row-designating subscript. Our list of product terms appears below.

|A| = Σ ( + ) A_1qA_2rA_3s . . . A_nz
|A| =  +  A₁₁A₂₂A₃₃  +  A₁₂A₂₃A₃₁  +  A₁₃A₂₁A₃₂  +  A₁₃A₂₂A₃₁  +  A₁₂A₂₁A₃₃  +  A₁₁A₂₃A₃₂

Note that we have 3! or 6 product terms, each consisting of one element from each row and one element from each column. The task remaining is to find the sign for each product term. To do this, we count the number of inversions needed to put elements in ascending order of their column-designating subscript.
To demonstrate how to count inversions, let's look at two of the product terms.

• Consider the second product term in the list: A₁₂A₂₃A₃₁. To put all of the column-designating subscripts in ascending order, we need to move A₃₁ from the end of the term to the front of the term, which results in: A₃₁A₁₂A₂₃. This movement counts as two inversions, since we moved A₃₁ two positions to the left. Since two is an even number, the sign of that product term should be positive.
• Consider the last product term in the list: A₁₁A₂₃A₃₂. To put all of the column-designating subscripts in ascending order, we need to interchange the second and third elements, which results in: A₁₁A₃₂A₂₃. This counts as one inversion, since we moved A₃₂ one position to the left. Since one is an odd number, the sign of that product term should be negative.

If we repeat this process for each of the other product terms, we get the following formula for the determinant of a 3 x 3 matrix.

|A| = Σ ( + ) A_1qA_2rA_3s . . . A_nz
|A| =  +  A₁₁A₂₂A₃₃  +  A₁₂A₂₃A₃₁  +  A₁₃A₂₁A₃₂  -  A₁₃A₂₂A₃₁  -  A₁₂A₂₁A₃₃  -  A₁₁A₂₃A₃₂

We can employ the same process to compute the determinant for any size matrix. However, as the matrix gets larger, the number of product terms increases very quickly. For example, a 4 x 4 matrix would have 4! or 24 terms; a 5 x 5 matrix, 120 terms; a 6 x 6 matrix, 720 terms, and so on. A 10 x 10 matrix would have 3,628,800 terms. You would not want to calculate the determinant of a large matrix by hand.

Test Your Understanding of This Lesson

Problem 1
What is the determinant of matrix A?

A =

5 1 2 6

(A) -7
(B) -28
(C) 7
(D) 28
(E) None of the above

Solution
The correct answer is (D), based on the matrix determinant algorithm shown below.

|A| = Σ ( + ) A_1qA_2rA_3s . . . A_nz

Because A is a 2 x 2 matrix, we know that the determinant algorithm has 2! or 2 product terms. And each product term includes one element from each row and one element from each column. We list the product terms below, with the elements of each product term arranged in ascending order of the left-hand (or row-designating) subscript.

|A| =  +  A₁₁A₂₂  +  A₁₂A₂₁

To determine whether each product term is preceded by a plus or minus sign, we count the inversions needed to put all of the column-designating subscripts in ascending order.

• The column-designating subscripts for the first term, A₁₁A₂₂, are already in ascending order; so the first term needs zero inversions. Since zero is an even number, the sign of the first term is positive.
• For the second term, A₁₂a₂₁, we must move the second element A₂₁ one position to the left; that is, we need one inversion to put the column-designating subscripts in ascending order. Since one is an odd number, the sign of the second term is negative.

The formula for the determinant of a 2 x 2 matrix is thus:

|A| =  +  A₁₁A₂₂  -  A₁₂A₂₁

So the determinant of matrix A is:

|A| = ( 5 * 6 ) - ( 1 * 2 ) = 30 - 2 = 28

Linear Algebra( Common for EEE and ME)

Linear Algebra: Matrix algebra.

Introduction to Matrices

Matrix algebra is used quite a bit in advanced statistics, largely because it provides two benefits.

• Compact notation for describing sets of data and sets of equations.
• Efficient methods for manipulating sets of data and solving sets of equations.

This lesson introduces the matrix, the rectangular array at the heart of matrix algebra.

Matrix Definition

A matrix is a rectangular array of numbers arranged in rows and columns. The array of numbers below is an example of a matrix.
                                                    

	21	62	33	93
	44	95	66	13
	77	38	79	33

The number of rows and columns that a matrix has is called its dimension or its order. By convention, rows are listed first; and columns, second. Thus, we would say that the dimension (or order) of the above matrix is 3 x 4, meaning that it has 3 rows and 4 columns.
Numbers that appear in the rows and columns of a matrix are called elements of the matrix. In the above matrix, the element in the first column of the first row is 21; the element in the second column of the first row is 62; and so on.

Matrix Notation

Statisticians use symbols to identify matrix elements and matrices.

• Matrix elements. Consider the matrix below, in which matrix elements are represented entirely by symbols.
  

  	A11	A12	A13	A14
  	A21	A22	A23	A24

  By convention, first subscript refers to the row number; and the second subscript, to the column number. Thus, the first element in the first row is represented by A₁₁. The second element in the first row is represented by A₁₂. And so on, until we reach the fourth element in the second row, which is represented by A₂₄.
• Matrices. There are several ways to represent a matrix symbolically. The simplest is to use a boldface letter, such as A, B, or C. Thus, A might represent a 2 x 4 matrix, as illustrated below.
  

  A =	11 62 33 93 44 95 66 13

  Another approach for representing matrix A is:
  
  A = [ A_ij ] where i = 1, 2 and j = 1, 2, 3, 4
  This notation indicates that A is a matrix with 2 rows and 4 columns. The actual elements of the array are not displayed; they are represented by the symbol A_ij.

Other matrix notation will be introduced as needed. For a description of all the matrix notation used in this tutorial, see the Matrix Notation Appendix.

Matrix Equality

To understand matrix algebra, we need to understand matrix equality. Two matrices are equal if all three of the following conditions are met:

• Each matrix has the same number of rows.
• Each matrix has the same number of columns.
• Corresponding elements within each matrix are equal.

Consider the three matrices shown below.

A =     111   x y 444

B =     111 222 333 444

C =     l   m   n o   p   q

If A = B, we know that x = 222 and y = 333; since corresponding elements of equal matrices are also equal. And we know that matrix C is not equal to A or B, because C has more columns than A or B.

Test Your Understanding of This Lesson

Problem 1
The notation below describes two matrices - A and B

A = [ Aij ] where i = 1, 2, 3 and j = 1, 2

B =     111 222 333 444 555 666 777 888

Which of the following statements about A and B are true?

I. Matrix A has 5 elements.
II. The dimension of matrix B is 4 x 2.
III. In matrix B, element B₂₁ is equal to 222.

(A) I only
(B) II only
(C) III only
(D) All of the above
(E) None of the above

Solution
The correct answer is (E).

• Matrix A has 3 rows and 2 columns; that is, 3 rows, each with 2 elements. This adds up to 6 elements, altogether - not 5.
• The dimension of matrix B is 2 x 4 - not 4 x 2. That is, matrix B has 2 rows and 4 columns - not 4 rows and 2 columns.
• And, finally, element B₂₁ refers to the first element in the second row of matrix B, which is equal to 555 - not 222.

Types of Matrices

This lesson describes a few of the more important types of matrices: transpose matrices, vectors, and different kinds of square matrices.

Transpose Matrix

The transpose of one matrix is another matrix that is obtained by using by using rows from the first matrix as columns in the second matrix.
For example, it is easy to see that the transpose of matrix A is A'. Row 1 of matrix A becomes column 1 of A'; row 2 of A becomes column 2 of A'; and row 3 of A becomes column 3 of A'.

A =     111 222 333 444 555 666

A' =     111 333 555 222 444 666

Note that the order of a matrix is reversed after it has been transposed. Matrix A is a 3 x 2 matrix, but matrix A' is a 2 x 3 matrix.
With respect to notation, this web site uses a prime to indicate a transpose. Thus, the transpose of matrix B would be written as B'.

Vectors

Vectors are a type of matrix having only one column or one row.
Vectors come in two flavors: column vectors and row vectors. For example, matrix a is a column vector, and matrix a' is a row vector.

a =     11 12 33

a' =     11 22 33

We use lower-case, boldface letters to represent column vectors. And since the transpose of a column vector is a row vector, we use lower-case, boldface letters plus a prime to represent row vectors. Thus, vector b would be a column vector, and vector b' would be a row vector.

Square Matrices

A square matrix is an n x n matrix; that is, a matrix with the same number of rows as columns. In this section, we describe several special kinds of square matrix.

• Symmetric matrix. If the transpose of a matrix is equal to itself, that matrix is said to be symmetric. Two examples of symmetric matrices appear below.
  

  A = A' =

  1 2 2 3

  B = B' =

  5 6 7 6 3 2 7 2 1

  Note that each of these matrices satisfy the defining requirement of a symmetric matrix: A = A' and B = B'.
• Diagonal matrix. A diagonal matrix is a special kind of symmetric matrix. It is a symmetric matrix with zeros in the off-diagonal elements. Two diagonal matrices are shown below.
  

  A =

  1 0 0 3

  B =

  5 0 0 0 3 0 0 0 1

  Note that the diagonal of a matrix refers to the elements that run from the upper left corner to the lower right corner.
  
• Scalar matrix. A scalar matrix is a special kind of diagonal matrix. It is a diagonal matrix with equal-valued elements along the diagonal. Two examples of a scalar matrix appear below.
  

  A =

  3 0 0 3

  B =

  5 0 0 0 5 0 0 0 5

These square matrices play a prominent role in the application of matrix algebra to real-world problems. For example, a scalar matrix called the identity matrix is critical to the solution of simultaneous linear equations. (We cover the identity matrix later in the tutorial.)

Test Your Understanding of This Lesson

Problem 1
Consider the matrices shown below - a, A, B, and C

a =     1 2

A =     3 5 4 6

B =     3 4 5 6

C =     -1 0 0 6

Which of the following statements are true?

I.   a is a row matrix
II.  A = B'
III. C is a symmetric matrix

(A) I and II
(B) I and III
(C) II and III
(D) None of the above
(E) All of the above

Solution
The correct answer is (C), as explained below.

• Matrix a is a column vector, not a row matrix
• The transpose of a matrix is created by interchanging corresponding rows and columns. When this is done to matrix B, we see that A = B'.
  
  

  A =

  3 5 4 6

  B =

  3 4 5 6

  B' =

  3 5 4 6

  so A = B'

  
  
• The transpose of C is equal to C; that is C = C'. Therefore, C is a symmetric matrix.

Note that the off-diagonal elements of matrix C are equal to zero; so matrix C is a diagonal matrix, which is a special kind of symmetric matrix.

Matrix Addition and Matrix Subtraction

Just like ordinary algebra, matrix algebra has operations like addition and subtraction.

How to Add and Subtract Matrices

Two matrices may be added or subtracted only if they have the same dimension; that is, they must have the same number of rows and columns.
Addition or subtraction is accomplished by adding or subtracting corresponding elements. For example, consider matrix A and matrix B.

A =

1 2 3 7 8 9

B =

5 6 7 3 4 5

Both matrices have the same number of rows and columns (2 rows and 3 columns), so they can be added and subtracted. Thus,

A + B =

1 + 5 2 + 6 3 + 7 7 + 3 8 + 4 9 + 5

=

6 8 10 10 12 14

And,

A - B =

1 - 5 2 - 6 3 - 7 7 - 3 8 - 4 9 - 5

=

-4 -4 -4 4 4 4

And finally, note that the order in which matrices are added is not important; thus, A + B = B + A.

Test Your Understanding of This Lesson

Problem 1
Consider the matrices shown below - A, B, C, and D

A =     1 2

B =     3 5 4 6

C =     4 5 6 6

D =     -1 0 -2 0

Which of the following statements are true?

I.   A + B = C
II.  B + C = D
III. B - C = D

(A) I only
(B) II only
(C) III only
(D) I and II
(E) I and III

Solution
The correct answer is (C), as shown below.

B - C   =     [ 3 - 4 5 - 5 ] 4 - 6 6 - 6     =     [ -1 0 ] -2 0     =    D

Note that Matrices A and B cannot be added, because B has more columns than A. Matrices may be added or subtracted only if they have the same number of rows and the same number of columns.

Matrix Multiplication

In matrix algebra, there are two kinds of matrix multiplication: multiplication of a matrix by a number and multiplication of a matrix by another matrix.

How to Multiply a Matrix by a Number

When you multiply a matrix by a number, you multiply every element in the matrix by the same number. This operation produces a new matrix, which is called a scalar multiple.
For example, if x is 5, and the matrix A is:

A =	100 200 300 400

Then,

xA   =   5A   =    5

100 200 300 400

=

5 * 100 5 * 200 5 * 300 5 * 400

=

500 1000 1500 2000

=    B

In the example above, every element of A is multiplied by 5 to produce the scalar multiple, B.
Note: Some texts refer to this operation as multiplying a matrix by a scalar. (A scalar is a real number or a symbol representing a real number.)

How to Multiply a Matrix by a Matrix

The matrix product AB is defined only when the number of columns in A is equal to the number of rows in B. Similarly, the matrix product BA is defined only when the number of columns in B is equal to the number of rows in A.
Suppose that A is an i x j matrix, and B is a j x k matrix. Then, the matrix product AB results in a matrix C, which has i rows and k columns; and each element in C can be computed according to the following formula.

C_ik = Σ_j A_ijB_jk

where

C_ik = the element in row i and column k from matrix C
A_ij = the element in row i and column j from matrix A
B_jk = the element in row j and column k from matrix B
Σ_j = summation sign, which indicates that the a_ijb_jk terms should be summed over j

Let's work through an example to show how the above formula works. Suppose we want to compute AB, given the matrices below.

A =

0 1 2 3 4 5

B =

6 7 8 9 10 11

Let AB = C. Because A has 2 rows, we know that C will have two rows; and because B has 2 columns, we know that C will have 2 columns. To compute the value of every element in the 2 x 2 matrix C, we use the formula C_ik =  Σ_j A_ijB_jk, as shown below.

• C₁₁ = Σ A_1jB_j1 = 0*6 + 1*8 +2*10 = 0 + 8 + 20 = 28
• C₁₂ = Σ A_1jB_j2 = 0*7 + 1*9 +2*11 = 0 + 9 + 22 = 31
• C₂₁ = Σ A_2jB_j1 = 3*6 + 4*8 +5*10 = = 18 + 32 + 50 = 100
• C₂₂ = Σ A_2jB_j2 = 3*7 + 4*9 +5*11 = 21 + 36 +55 = 112

Based on the above calculations, we can say

AB   =   C =

28 31 100 112

What we did to compute Matrix C was not complicated. All we did was to multiply row elements in Matrix A by corresponding column elements in Matrix B.

Multiplication Order

As we have already mentioned, in some cases, matrix multiplication is defined for AB, but not for BA; and vice versa. However, even when matrix multiplication is possible in both directions, results may be different. That is, AB is not always equal to BA.
Because order is important, matrix algebra jargon has evolved to clearly indicate the order in which matrices are multiplied.

• To describe the matrix product AB, we can say A is postmultiplied by B; or we can say that B is premultiplied by A.
  
• Similarly, to describe the matrix product BA, we can say B is postmultiplied by A; or we can say that A is premultiplied by B.

The bottom line: when you multiply two matrices, order matters.

Identity Matrix

The identity matrix is an n x n diagonal matrix with 1's in the diagonal and zeros everywhere else. The identity matrix is denoted by I or I_n. Two identity matrices appear below.

I2 =     1 0 0 1

I3 =     1 0 0 0 1 0 0 0 1

The identity matrix has a unique talent. Any matrix that can be premultiplied or postmultiplied by I remains the same; that is:

AI = IA = A

Test Your Understanding of This Lesson

Problem 1
Consider the matrices shown below - A, B, and C

A =     a b c d

B =     e f g h

C =     w x y z

Assume that AB = C. Which of the following statements are true?

(A) w = a*e + b*h
(B) x = a*f + b*h
(C) y = c*g + d*h
(D) All of the above
(E) None of the above

Solution
The correct answer is (B). To compute the value of any element in matrix C, we use the formula C_ik = Σ_j A_ijB_jk.
In matrix C, x is the element in row 1 and column 2, which is represented in the formula by C₁₂. Therefore, to find x, we use the formula to calculate C₁₂, as shown below.

x = C₁₂ = Σ_j A_1jB_j2 = A₁₁B₁₂ + A₁₂B₂₂ = a*f + b*h

All of the other answers are incorrect.

Vector Multiplication

The multiplication of a vector by a vector produces some interesting results, known as the vector inner product and as the vector outer product.
Prerequisite: This material assumes familiarity with matrix multiplication.

Vector Inner Product

Assume that a and b are vectors, each with the same number of elements. Then, the inner product of a and b is s.

a'b = b'a = s

where
      a and b are column vectors, each having n elements,
      a' is the transpose of a, which makes a' a row vector,
      b' is the transpose of b, which makes b' a row vector, and
      s is a scalar; that is, s is a real number - not a matrix.
Note this interesting result. The product of two matrices is usually another matrix. However, the inner product of two vectors is different. It results in a real number - not a matrix. This is illustrated below.

a =

1 2 3

b =

4 5 6

Then,

a'b = 1*4 + 2*5 + 3*6 = 4 + 10 + 18 = 32

Thus, the inner product of a'b is equal to 32.
Note: The inner product is also known as the dot product or as the scalar product.

Vector Outer Product

Assume that a and b are vectors. Then, the outer product of a and b is C.

ab'= C

where
      a is a column vector, having m elements,
      b is a column vector, having n elements,
      b' is the transpose of b, which makes b' a row vector, and
      C is a rectangular m x n matrix
Unlike the inner product, the outer product of two vectors produces a rectangular matrix, not a scalar. This is illustrated below.

a =

v w

b =

x y z

Then,

C    =    ab'    =

v * x v * y v * z w * x w * y w * z

Notice that the elements of Matrix C consist of the product of elements from Vector A crossed with elements from Vector B. Thus, Matrix C winds up being a matrix of cross products from the two vectors.

Test Your Understanding of This Lesson

Problems
Consider the matrices shown below - a, b, and c

a =     0 1

b =     2 3

c =     4 5 6

Using a, b, and c, answer the questions below.

1. Find a'b, the inner product of a and b.
2. Find bc', the outer product of b and c.
3. True or false: bc' = cb'

Solutions

1. The term a'b is an inner product, which is equal to 3. The solution appears below.
   

   a'b   =     0 1     *     2 3     =     0*2 + 1*3    =    3

2. The term bc' is an outer product, which results in the 2 x 3 matrix D.
   

   bc'   =     2 3     *     4 5 6     =     2*4 2*5 2*6 3*4 3*5 3*6     =     8 10 12 12 15 18     =    D

3. The statement bc' = cb' is false.
   Note that b is a 2 x 1 vector and c is a 3 x 1 vector. Therefore, bc' is a 2 x 3 matrix, and cb' is a 3 x 2 matrix. Because bc' and cb' have different dimensions, they cannot be equal.
   
   
   

   Elementary Matrix Operations

   Elementary matrix operations play an important role in many matrix algebra applications, such as finding the inverse of a matrix and solving simultaneous linear equations.
   

   Elementary Operations

   There are three kinds of elementary matrix operations.
   
4. Interchange two rows (or columns).
5. Multiply each element in a row (or column) by a non-zero number.
6. Multiply a row (or column) by a non-zero number and add the result to another row (or column).

When these operations are performed on rows, they are called elementary row operations; and when they are performed on columns, they are called elementary column operations.

Elementary Operation Notation

In many references, you will encounter a compact notation to describe elementary operations. That notation is shown below.

	Operation description	Notation
Row operations	1. Interchange rows i and j	Ri <--> Rj
	2. Multiply row i by s, where s ≠ 0	sRi --> Ri
	3. Add s times row i to row j	sRi + Rj --> Rj
Column operations	1. Interchange columns i and j	Ci <--> Cj
	2. Multiply column i by s, where s ≠ 0	sCi --> Ci
	3. Add s times column i to column j	sCi + Cj --> Cj

Elementary Operators

Each type of elementary operation may be performed by matrix multiplication, using square matrices called elementary operators.
For example, suppose you want to interchange rows 1 and 2 of Matrix A. To accomplish this, you could premultiply A by E to produce B, as shown below.

R1 <--> R2    =	0 1 1 0		1 3 5 2 4 6	=	0 + 2 0 + 4 0 + 6 0 + 1 0 + 3 0 + 5	=	2 4 6 1 3 5
	E		A				B

Here, E is an elementary operator. It operates on A to produce the desired interchanged rows in B. What we would like to know, of course, is how to find E. Read on.

How to Perform Elementary Row Operations

To perform an elementary row operation on a A, an r x c matrix, take the following steps.

1. To find E, the elementary row operator, apply the operation to an r x r identity matrix.
2. To carry out the elementary row operation, premultiply A by E.

We illustrate this process below for each of the three types of elementary row operations.

• Interchange two rows. Suppose we want to interchange the second and third rows of A, a 3 x 2 matrix. To create the elementary row operator E, we interchange the second and third rows of the identity matrix I₃.
  

  1 0 0 0 1 0 0 0 1	⇒	1 0 0 0 0 1 0 1 0
  I3		E

  Then, to interchange the second and third rows of A, we premultiply A by E, as shown below.
  

  R2 <--> R3  =	1 0 0 0 0 1 0 1 0		0 1 2 3 4 5	=	1*0 + 0*2 + 0*4 1*1 + 0*3 + 0*5 0*0 + 0*2 + 1*4 0*1 + 0*3 + 1*5 0*0 + 1*2 + 0*4 0*1 + 1*3 + 0*5	=	0 1 4 5 2 3
  	E		A				B

• Multiply a row by a number. Suppose we want to multiply each element in the second row of Matrix A by 7. Assume A is a 2 x 3 matrix. To create the elementary row operator E, we multiply each element in the second row of the identity matrix I₂ by 7.
  

  1 0 0 1	⇒	1 0 0 7
  I2		E

  Then, to multiply each element in the second row of A by 7, we premultiply A by E.
  

  7R2 --> R2  =	1 0 0 7		0 1 2 3 4 5	=	1*0 + 0*3 1*1 + 0*4 1*2 + 0*5 0*0 + 7*3 0*1 + 7*4 0*2 + 7*5	=	0 1 2 21 28 35
  	E		A				B

• Multiply a row and add it to another row. Assume A is a 2 x 2 matrix. Suppose we want to multiply each element in the first row of A by 3; and we want to add that result to the second row of A. For this operation, creating the elementary row operator is a two-step process. First, we multiply each element in the first row of the identity matrix I₂ by 3. Next, we add the result of that multiplication to the second row of I₂ to produce E.
  

  1 0 0 1	⇒	1 0 0 + 3*1 1 + 3*0	⇒	1 0 3 1
  I2				E

  Then, to multiply each element in the first row of A by 3 and add that result to the second row, we premultiply A by E₂.
  

  3R1  +  R2 -->  R2  =	1 0 3 1		0 1 2 3	=	1*0 + 0*2 1*1 + 0*3 3*0 + 1*2 3*1 + 1*3	=	0 1 2 6
  	E		A				B

How to Perform Elementary Column Operations

To perform an elementary column operation on A, an r x c matrix, take the following steps.

1. To find E, the elementary column operator, apply the operation to an c x c identity matrix.
2. To carry out the elementary column operation, postmultiply A by E.

Let's work through an elementary column operation to illustrate the process. For example, suppose we want to interchange the first and second columns of A, a 3 x 2 matrix. To create the elementary column operator E, we interchange the first and second columns of the identity matrix I₂.

1 0 0 1	⇒	0 1 1 0
I2		E

Then, to interchange the first and second columns of A, we postmultiply A by E, as shown below.

C1 <--> C2    =	0 1 2 3 4 5		0 1 1 0	=	0*0 + 1*1 0*1 + 1*0 2*0 + 3*1 2*1 + 3*0 4*0 + 5*1 4*1 + 5*0	=	1 0 3 2 5 4
	A		E				B

Note that the process for performing an elementary column operation on an r x c matrix is very similar to the process for performing an elementary row operation. The main differences are:

• To operate on the r x c matrix A, the row operator E is created from an r x r identity matrix; whereas the column operator E is created from an c x c identity matrix.
  
• To perform a row operation, A is premultiplied by E; whereas to perform a column operation, A is postmultiplied by E.

Test Your Understanding of This Lesson

Problem 1
Assume that A is a 4 x 3 matrix. Suppose you want to multiply each element in the second column of matrix A by 9. Find the elementary column operator E.
Solution
To find the elementary column operator E, we multiply each element in the second column of the identity matrix I₃ by 9.

1 0 0 0 1 0 0 0 1	⇒	1 0 0 0 9 0 0 0 1
I3		E