If we view $A$ as a transformation, the question that we have to ask of $A^{-1}$ is what transformation gets us back to $\mathbf x$? In the case of elimination matrices, that's actually not too hard to answer.
Consider the elimination matrix $E_{21} = \begin{bmatrix}1 & 0 & 0 \\ -4 & 1 & 0 \\0 & 0 & 1 \end{bmatrix}$ which subtracts 4 of the first row from the second row. The clear inverse operation to restore the matrix would be to just add 4 of the first row back to the second row. We can express this as a matrix: $E_{21}^{-1} = \begin{bmatrix}1 & 0 & 0 \\ 4 & 1 & 0 \\0 & 0 & 1 \end{bmatrix}$.
Strang goes through a number of different properties of inverses in (2.2). The important question is: when does a matrix $A$ have an inverse? These properties basically tell us that a matrix has an inverse if and only if its columns are linearly independent.
Let's look at a few of these in a bit more detail.
If $A$ has an inverse, it is unique.
Suppose that $A$ has two inverses, $B$ and $C$. Since $AA^{-1} = A^{-1} A$, we can write $BA = I$ and $AC = I$. We can multiply these matrices together. By associativity, we have $(BA)C = B(AC)$. This gives us $IC = BI$, which means $B = C$.
This is a common approach for showing that a mathematical object is unique: suppose that you have two of them and then work your way to showing that the two things you thought you had are actually the same thing.
If $A \mathbf v = \mathbf 0$ for some $\mathbf v \neq \mathbf 0$, then $A$ has dependent columns and $A$ has no inverse.
We note that if $A \mathbf v = \mathbf 0$ and $\mathbf v = \mathbf 0$, this means that there is a linear combination of the columns of $A$ that produces $\mathbf 0$ that doesn't zero out all the vectors. This is one of the conditions that tells us that our set of vectors is linearly dependent.
So now we have $A \mathbf v = A \mathbf 0 = \mathbf 0$. This means that there are at least two vectors that produce $\mathbf 0$. In other words, there is more than one solution to $A \mathbf x = \mathbf 0$, so $A^{-1}$ cannot exist.
Since we're interested in the inverses of elimination matrices, we should think about how to consider the inverses of products of matrices.
Let $A$ and $B$ be matrices of the same size. If $A$ and $B$ are invertible, then $AB$ is also invertible. Furthermore, the inverse of $AB$, $(AB)^{-1}$ is $B^{-1} A^{-1}$.
This is important—because the order in which the matrices are multiplied matters, the order in which their inverses are multiplied also matters—we must reverse the order.
If we view matrices as a function that transforms vectors, then we can consider the transformation on a given $\mathbf x$. The product $AB\mathbf x$ can be thought of as a transformation that first creates $B \mathbf x = \mathbf y$ and then computes $A \mathbf y = \mathbf z$. \[\mathbf x \xrightarrow{B} \mathbf y \xrightarrow{A} \mathbf z\]
The procedure to reverse this process becomes clear if we have the inverses $A^{-1}$ and $B^{-1}$: we first need to compute $A^{-1} \mathbf z = \mathbf y$ and then take this vector to get $B^{-1} \mathbf y = \mathbf x$: \[\mathbf z \xrightarrow{A^{-1}} \mathbf y \xrightarrow{B^{-1}} \mathbf x\] This is exactly $(AB)^{-1} \mathbf z = B^{-1} A^{-1} \mathbf z$.
Recall that $E_{21} = \begin{bmatrix}1 & 0 & 0 \\ -4 & 1 & 0 \\0 & 0 & 1 \end{bmatrix}$ subtracts 4 of the first row from the second row and the inverse operation adds 4 of the first row back to the second row: $E_{21}^{-1} = \begin{bmatrix}1 & 0 & 0 \\ 4 & 1 & 0 \\0 & 0 & 1 \end{bmatrix}$.
Now, since we handled the second row, suppose we want to do elimination on the third row by using the second row. We use a matrix like $E_{32} = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\0 & -2 & 1 \end{bmatrix}$. As above, the inverse of this matrix is $E_{32}^{-1} = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\0 & 2 & 1 \end{bmatrix}$.
So suppose we accomplish $E_{32} E_{21} A = U$. What is $E_{32} E_{21}$? We can compute this: \[E_{32} E_{21} = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\0 & -2 & 1 \end{bmatrix}\begin{bmatrix}1 & 0 & 0 \\ -4 & 1 & 0 \\0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 \\ -4 & 1 & 0 \\ 8 & -2 & 1 \end{bmatrix}.\]
Now, we went through the trouble of thinking about inverses, so let's consider them. We have \[(E_{32} E_{21})^{-1} = E_{21}^{-1} E_{32}^{-1} = \begin{bmatrix}1 & 0 & 0 \\ 4 & 1 & 0 \\0 & 0 & 1 \end{bmatrix}\begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\0 & 2 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 \\ 4 & 1 & 0 \\ 0 & 2 & 1 \end{bmatrix}.\]
There's something in the "niceness" of the matrices that we get. The inverse of $E$ looks "nicer" or "cleaner" than $E$—the multiples go in the right spot, no fuss, no mess. But in $E$, we have the negatives, yes, but for some reason there's an extra entry. What's going on?
Remember that these are elimination matrices, which means that we have been zeroing out entries of $A$. If we do this process in sequence, then this means that we must have zeroed out the $2,1$ entry before moving on to think about the elimination matrices in the third row. What this means is that each elimination matrix is really describing how to combine rows of $U$, not $A$. So if we want a matrix $E$ that describes combinations of rows of $A$, there is going to be some "mess" in some of the entries. But going in reverse, we don't have that problem, so $E^{-1}$ contains only the constants we expect.
This leads to the following observation. We have been using elimination on a matrix $A$ to transform it into an upper triangular matrix $U$. Elimination can be expressed as a series of elimination matrices. Because elimination changes the bottom rows of a matrix based on the top rows, our elimination matrices are lower triangular matrices, with $1$s along the diagonal.
So if we take all of our elimination matrices, we can multiply them together to get a single matrix $E$. We then have the relationship $E_k E_{k-1} \cdots E_2 E_1 A = EA = U$, where $E_i$ is a single elimination matrix. This is a great result, but as we saw in the example above, the entries are not quite so satisfying. If $\ell_{ij}$ is the corresponding "multiple" from elimination matrix $E_{ij}$, then for a $3 \times 3$ matrix, we have \[E = \begin{bmatrix} 1 & 0 & 0 \\ -\ell_{21} & 1 & 0 \\ \ell_{32} \ell_{21} - \ell_{31} & -\ell_{32} & 1 \end{bmatrix}.\]
However, $E$ is invertible and to get $E^{-1}$, all we need to do is flip the signs for our elimination matrices and multiply them in reverse order. When we do, we get a very interesting looking matrix: \[E^{-1} = \begin{bmatrix} 1 & 0 & 0 \\ \ell_{21} & 1 & 0 \\ \ell_{31} & \ell_{32} & 1 \end{bmatrix}.\]
What's more is notice that we get \[E^{-1} E A = A = E^{-1} U.\]
Notice that like $E$, $E^{-1}$ is also a lower triangular matrix. Elimination matrices are lower triangular and their inverses are lower triangular and their products are lower triangular. Since $E^{-1}$ is lower triangular, we let $L = E^{-1}$. Then we get the factorization \[A = LU.\]
This is the LU-factorization.