Quant Out of Water

Constraint Programming, Puzzles, ILP, and z3

Sun, 24 Sep 2023 00:00:00 -0700

Solving puzzles with computers.

Introduction

Earlier in the year I stumbled upon a puzzle posted by Jane Street called the 4-in-1 24-7 puzzle . The puzzle is in the front matter of this post. On a first look, and on first reading through the rules for the puzzle, it appears cartoonishly complicated. But, if you’d like to take your own crack at the problem, there is a simpler 24-7 puzzle which can be solved first (as I did), before tackling the 4-in-1 version.

I’m not sure if the creators of the puzzle imagined people would print the puzzle onto paper and solve it by hand, but I immediately thought to myself: I am way too lazy for that. So instead I naturally wanted to get a computer to do all the work for me…

Throughout my short academic career, I generally took much more interest in continuous optimization than I did in discrete optimization. Almost everything I learned about discrete optimization I learned ages ago from this Coursera course , and the application of semidefinite programming to approximating MAXCUT . That being said, I knew that constraint programming was a natural approach to solving puzzles, the 24-7 puzzles being no exception.

Interestingly, I’m no stranger to algorithmic puzzle-solving. Many years ago, my girlfriend showed me an optical illusion puzzle that involved fitting a 4x4 grid of tiles with matching patterns on adjacent edges – you had to figure out how to place the tiles so that the patterns on the edges all matched.

I wrote a simple C++ program to basically brute force this puzzle – I remember running it for about a day on my ancient laptop. Armed with the tools described in this blog post, it could probably be solved in under a second, and the modeling and implementation could be done in a couple hours (rather than the whole weekend, as the C++ program probably took me).

Constraint Programming

Constraint programming is an algorithmic methodology that targets problems that have a discrete solution space, such as arranging puzzle pieces, rather than continuous problems like optimizing the shape of an aluminum can. Moreover, in contrast to optimization, the focus is upon simply finding a feasible solution that satisfies a given set of constraints, rather than optimizing an objective function subject to those constraints.

The approach involves systematically enumerating constraints that a solution must meet and which, if met, constitute a solution. A search algorithm then explores the solution space and either successfully returns a solution, or proves that one does not exist.

An example of how constraint programming might be used is in solving Sudoku puzzles. For instance, a common technique involves initially considering all numbers from 1 to 9 as candidates for each square. You then iteratively narrow down these options based on existing numbers in the grid, filling in a square when only one candidate remains and backtracking when no candidates are left. The process of checking how a decision affects the constraints is called constraint propagation and it is one particular algorithmic building block for trying to tackle constraint programming problems.

I followed a strategy like this to hack together a Sudoku solver , similarly inspired when I saw my girlfriend working on a book of Sudoku puzzles… Perhaps this is a theme 🤔 … At the time I was rather satisfied to have been able to solve Sudokus in a couple hundred milliseconds (so slow largely because of a rather inefficient representation, and writing it in Python), though there are some rather fast programs that can solve Sudoku on the order of microseconds.

These examples are ad-hoc, but I am more interested in general approaches. Similarly to how CVX makes it incredibly easy to experiment with convex optimization problems, there are general modeling tools that can be used to model and solve whole classes of discrete optimization problems. Indeed, if the constraints can be appropriately described, general purpose algorithms can be applied to search for feasible solutions.

For the rest of this post, I will describe two such general tools: Integer Linear Programming, and Satisfiability Modulo Theories, and how they can be used to solve constraint programming problems.

Integer Linear Programming

A concrete and practical means of describing constraint programs is by means of integer linear programming (ILP). Specifically, a feasibility ILP:

\begin{equation} \begin{aligned} \text{find}&\ x\\ \text{such that} &\ Ax \le b\\ &\ x_i \in \mathbb{N}_0. \end{aligned}\notag \end{equation}

This looks a lot like an ordinary linear program $\text{min}\{c^{\mathsf{T}} x\ |\ Ax \le b, x \in \mathbb{R}^n\}$ except there is no objective (practically speaking, just set $c = 0$) and there is an integer constraint $x_i \in \mathbb{N}_0$. That is, each element of the solution vector $x$ must be an integer (including zero): $0, 1, 2, \ldots$

While linear programming problems are convex optimization problems, and thus entirely tractable, ILPs are NP-complete. Theoretically, there is no known “fast” algorithm for solving ILPs in general, but practically speaking, we can usually still find optimal solutions (or good suboptimal approximations) fast enough for it to be useful.

Remark (Totally Unimodular Matrices): If the matrix $A$ in the ILP constraints is “totally unimodular”, then the LP solution and the ILP solution coincide. This is an important and tractable special case .

Another special case of ILP is what one might call Boolean Linear Programming (BLP), but is usually actually referred to as $0-1$ linear programming. In this case, we have the constraint $x_i \in \{0, 1\}$ (equivalent to just including the constraint $x_i \le 1$ in the ILP). To give a sense of how powerful ILP is, the reader is encouraged to think about why the following constraints encode logical operators as BLP constraints:

logical-and: $z = x \wedge y$ is encoded as $z \ge x + y - 1, z \le x, z \le y$.
logical-not: $z = \neg x$ is encoded as $z = 1 - x$

Given that we can encode AND and NOT, we can encode any Boolean expression – these operators together are universal. Here’s a couple more examples:

logical-enforcement (if x then y): $x \implies y$ is encoded as $x \le y$
logical-implication: $z = (x \implies y) = y \vee \neg x$: is encoded as $z \le (1 - x) + y, z \ge 1 - x, z \ge y$
- This looks complicated, but is nothing but the combination of logical-or and logical-not.

The reader is encouraged to try to encode some other logical expressions, e.g., exclusive-or, logical-or, etc.

I’ll shortly use this modeling framework to encode and solve the (vanilla) 24-7 puzzle.

Satisfiability Modulo Theories

Similarly to ILP, Satisfiability modulo theories (SMT) provide a means of implementing constraint programming. SMT is a powerful methodology that generalizes Boolean Satisfiability (SAT) to allow the inclusion of arithmetic (among other things) in the constraints. For example, an instance of a SAT problem is to find a Boolean assignment to the variables $x,y,z$ such that

\begin{equation} \begin{aligned} (x \vee \neg y \vee z) \wedge (y \vee \neg z) \end{aligned}\notag \end{equation}

evaluates to True, or prove that such an assignment does not exist.

An example of an SMT problem is to solve the (nonlinear!) system of inequalities

\begin{equation} \begin{aligned} x^2 + y^2 &> 3\\ x^3 + y &< 5 \end{aligned}\notag \end{equation}

over the integers. The inclusion of arithmetic makes SMT more general than SAT.

SAT and SMT both have rich mathematical theory surrounding them. However, I am primarily interested in SMT as a practical engineering tool, so I admit to knowing almost nothing about the theory. Indeed, I hardly know anything at all about theoretical computer science, and everything I’ve learned about SMT so far comes from this very practical book . Computationally, a library for doing SMT programming is the z3 theorem prover , which has convenient Python bindings . That same page has plenty of examples, as does this guide .

Here’s a quick example that solves the above nonlinear system:

import z3
x = z3.Int("x")
y = z3.Int("y")
z3.solve(z3.And(
    x**2 + y**2 > 3,
    x**3 + y < 5))

`1`	`[x = -2, y = 0]`

I’ll shortly use z3 to solve the complicated 4-in-1 24-7 puzzle.

Solving the Vanilla 24-7 Puzzle

Now the fun begins. Here I’m going to apply integer linear programming to solve the “vanilla” 24-7 puzzle, which looks vaguely similar to a Sudoku:

The rules are described thus:

The grid is incomplete. Place numbers in some of the empty cells below so that in total the grid contains one 1, two 2’s, etc., up to seven 7’s. Furthermore, each row and column must contain exactly 4 numbers which sum to 20. Finally, the numbered cells must form a connected region, but every 2-by-2 subsquare in the completed grid must contain at least one empty cell.

Read this over a few times – the first couple of constraints sound straightforward, but the “connected region” constraint is difficult to even wrap one’s head around. It means that the cells which are filled in need to form a connected set – there are no “islands” of numbers. We’ll see these in detail as I show how they can be encoded as constraints in an ILP.

First, I am using ortools , so let’s do some setup:

from ortools.sat.python import cp_model
from itertools import product

N = range(7)  # Grid size
K = range(1, 8)  # Values

model = cp_model.CpModel()
solver = cp_model.CpSolver()

This sets up a Constraint Programming model called model and a solver. We now just need to add our constraints!

Remark (performance): I generally first write code that is readable and easy to debug and understand, only trying to optimize when it is necessary. I’m quite directly describing the constraints of the problem, and not trying to do anything clever. I’ve introduced a number of variables which I believe add clarity, but which may, technically, be redundant. Fortunately, it is quite likely that the ILP solver is fairly capable of recognizing and eliminating obvious redundancies. That being said, the way in which a program is encoded (in any modeling language) can potentially have a significant impact on performance, so it can be worth it to revise and refactor if one is not getting the throughput they need.

Remark (modeling tip): While it may be counter intuitive, it is actually possible that increasing the number of constraints in the problem can actually improve performance. The reason being that the search space is more constrained.

The first variable I’m introducing is what I’ve called a panel. This is a $7 \times 7 \times 7$ grid of $\{0, 1\}$ indicators where panel[k][i][j] = 1 if and only if the cell at position (i, j) is equal to k. Let’s first encode all the static assignments using this variable.

# Declare variables which encode the values on the board as 7 sets of Boolean indicators
panel = {k: [[model.NewIntVar(0, 1, f"z_{k}{i}{j}") for j in N] for i in N] for k in K}

# Ensure that only one of the panel indicators is active
for i, j in product(N, N):
    model.Add(sum(panel[k][i][j] for k in K) <= 1)

# Manually encode the given values
model.Add(panel[4][0][1] == 1)
model.Add(panel[6][1][2] == 1)
model.Add(panel[3][1][3] == 1)
model.Add(panel[6][1][6] == 1)
model.Add(panel[5][2][5] == 1)
model.Add(panel[5][2][6] == 1)
model.Add(panel[4][3][3] == 1)
model.Add(panel[4][4][0] == 1)
model.Add(panel[7][4][1] == 1)
model.Add(panel[2][5][0] == 1)
model.Add(panel[7][5][3] == 1)
model.Add(panel[4][5][4] == 1)
model.Add(panel[1][6][5] == 1)

The function model.Add adds a constraint to the ILP.

Notice that the constraint sum(panel[k][i][j] for k in K) <= 1, along with the fact that panel[k][i][j] is either $0$ or $1$, ensures that just one of these indicators can be active at a time.

Another obvious variable to introduce is the grid, which is a $7 \times 7$ array of variables equal to the value of that cell.

# Variables that Store the value of the grid at position (i, j)
grid = [[model.NewIntVar(0, 7, f"g_{i}{j}") for j in N] for i in N]

# Encode the values of the grid from the zero-one variables
for i, j in product(N, N):
    model.Add(grid[i][j] == sum(k * panel[k][i][j] for k in K))

Since only one panel indicator can be active, sum(k * panel[k][i][j] for k in K) must be equal to the integer value that is being encoded.

This grid variable makes it easier to encode the row and column sum constraints.

# Each row and column must sum to 20
for row in N:
    model.Add(sum(grid[row]) == 20)
for col in N:
    model.Add(sum(grid[i][col] for i in N) == 20)

And, the reason I introduced the panel variable in the first place is because it makes is easy to encode the counting constraint:

1
2
3

# The grid must contain one 1, two 2s, three 3s, etc.
for k in K:
    model.Add(sum(sum(panel[k][i][j] for j in N) for i in N) == k)

The constraint that each row and column contain exactly four non-zero numbers, and that each $2 \times 2$ subsquare contain an empty cell is easy to encode if we introduce another Boolean nz variable where nz[i][j] = 1 if and only if grid position $(i, j)$ is non-zero.

This variable feels redundant, as we should be able to just do things like if grid[i][j] > 0 right? Technically yes, but encoding if-else statements directly in an ILP is cumbersome and confusing – the nz variable makes reasoning easier. We’ll see later how z3 addresses this with higher level programming constructs.

# An indicator variable for whether position (i, j) is non-zero
nz = [[model.NewIntVar(0, 1, f"nz_{i}{j}") for j in N] for i in N]

# Encode the non-zero indicator
for i, j in product(N, N):
    model.Add(grid[i][j] <= len(N) * nz[i][j])
    model.Add(nz[i][j] <= grid[i][j])

# Each row and column must contain exactly four non-zero numbers
for row in N:
    model.Add(sum(nz[row]) == 4)
for col in N:
    model.Add(sum(nz[i][col] for i in N) == 4)

# Every 2x2 subsquare must contain at least one empty cell
for i in N[:-1]:
    for j in N[:-1]:
        model.Add(nz[i][j] + nz[i + 1][j] + nz[i][j + 1] + nz[i + 1][j + 1] <= 3)

Finally, the entries need to form a connected region. This is the most difficult constraint to encode. If you are confused about what constitutes a “connected region”, take a look at the solution – you can trace a path between any two filled in digits: there are no “islands”.

One idea is to try encode this is to do some kind of connected component labeling , but it seems difficult (to me?) to encode this in an ILP such that it is both necessary and sufficient. That is, it is fairly easy to construct a set of ILP constraints which will be satisfied if there is just one connected component, but going in the other direction (that there is necessarily a single connected component when the constraints are satisfied) seems much more difficult.

An alternative method is to imagine that the cells and their neighbours form a network of pipes, and then model some kind of flow within that network. We can let each non-zero part of the grid consume a single unit of flow, and then designate one of the given filled-in cells as a source with exactly enough units of flow to cover the whole grid. Since there is one 1, two 2s, etc. up to seven 7s, we know there must be exactly 28 non-zero cells.

It is well known that flow problems can be modeled with linear programs, so this approach seems promising for our ILP. The following code does exactly this – the comments should make the code fairly clear. However, it can be subtle.

# The maximum amount of flow
NEEDED_FLOW = 28

def _neighbours(i, j):
    if i + 1 < len(N):
        yield (i + 1, j)
    if i - 1 >= 0:
        yield (i - 1, j)
    if j + 1 < len(N):
        yield (i, j + 1)
    if j - 1 >= 0:
        yield (i, j - 1)


# Outgoing flow from cell (i, j) to neighbouring cell (p, q)
flow = [
    [
        [
            (p, q, model.NewIntVar(0, M, f"flow_{i}{j}{p}{q}"))
            for (p, q) in _neighbours(i, j)
        ]
        for j in N
    ]
    for i in N
]

# Sum up all the outgoing flow
outgoing_flow = [[0 for j in N] for i in N]
for i, j in product(N, N):
    for _, _, f in flow[i][j]:
        outgoing_flow[i][j] += f

# And separately keep track of the incoming flow
incoming_flow = [[0 for j in N] for i in N]
for i, j in product(N, N):
    for p, q, f in flow[i][j]:
        incoming_flow[p][q] += f

# Cells which are zero must have zero flow through them
for i, j in product(N, N):
    model.Add(outgoing_flow[i][j] <= M * nz[i][j])
    model.Add(incoming_flow[i][j] <= M * nz[i][j])

# Pick an arbitrary fixed node as the source
i_source, j_source = 0, 1

# Give it enough flow to fill all non-zero cells (minus itself)
model.Add(
    outgoing_flow[i_source][j_source] == NEEDED_FLOW - 1
)

# The source should have no incoming flow
incoming_flow[i_source][j_source] == 0

# Conserve flow in the network
for i, j in product(N, N):
    if i == i_source and j == j_source:
        # Except the source is unconstrained
        continue

    # non-zero cells must have positive incoming flow
    # which ensures they're connected to the source
    model.Add(incoming_flow[i][j] >= nz[i][j])

    # The outgoing flow must be one less than the incoming flow
    # i.e., each node sinks 1 unit of flow.
    model.Add(outgoing_flow[i][j] == incoming_flow[i][j] - nz[i][j])

This code looks fairly simple, but writing it was quite a challenge. It was particularly difficult to write down the right constraints that ensured only the one designated node could act as a source. In many earlier iterations I failed to properly enforce this and wound up with islands of non-zero cells. I also had a subtle bug wherein I wrote model.Add(outgoing_flow[i][j] == incoming_flow[i][j] - 1) instead of model.Add(outgoing_flow[i][j] == incoming_flow[i][j] - nz[i][j]).

Remark: When I originally solved the 24-7 puzzle (and the 4-in-1 puzzle below) I actually didn’t encode the path constraint condition. Instead, I just encoded a simpler sufficient condition and printed out all of the possible solutions (there were only 3) and selected the correct one manually – an expedient method.

In any case, we can finally print out a solution. I’ve hidden the implementation of the panel_to_str function as it is uninteresting. It just relies on getting the value of variables in the program with the method solver.Value, for example: solver.Value(panel[k][i][j]), etc.

if solver.Solve(model) == cp_model.OPTIMAL:
    print("Optimal solution found:")
    print(panel_to_str(panel, solver))
else:
    print("FAILED TO FIND A SOLUTION")

Optimal solution found:
7, 4, 3, 0, 6, 0, 0
0, 0, 6, 3, 5, 0, 6
0, 0, 5, 0, 5, 5, 5
0, 3, 6, 4, 0, 0, 7
4, 7, 0, 0, 0, 7, 2
2, 0, 0, 7, 4, 7, 0
7, 6, 0, 6, 0, 1, 0

And there we have it 😀

Solving the 4-in-1 Puzzle

The previous example with the 24-7 puzzle shows that ILP can be a rather cumbersome approach to encoding constraint programming problems, particularly if you endeavor to directly construct all of the linear inequalities needed to encode your program. In this section, I’m going to use the z3 theorem prover for SMT, which I think is considerably more expressive.

I’ll also be tackling a more difficult puzzle – the 4-in-1 24-7 puzzle. Indeed, the previous section was merely a warm up. The specific rules for the 4-in-1 puzzle is to:

Place numbers in some of the empty cells so that in total each of the four 7-by-7 outlined grids is a legal “Twenty Four Seven” grid. Namely: each 7-by-7 grid’s interior should contain one 1, two 2’s, etc., up to seven 7’s. Furthermore, each row and column within the 7-by-7’s must contain exactly 4 numbers which sum to 20. Finally, the numbered cells must form a connected region, but every 2-by-2 subsquare in the completed grid must contain at least one empty cell.

Some numbers have been placed inside the grid. Additionally, some blue numbers have been placed outside of the grids. A number outside the grid represents either the sum of the row or column it is facing, or the value of the first number it sees in that row or column.

Let’s begin!

import z3
from itertools import product

MAX_VAL = 7
OFFSET = 5
N = range(7)  # Subgrid size
K = range(1, 8)  # Available values
full_N = range(12)  # The whole grid

# The full grid
grid = [[z3.Int(f"x_{i}_{j}") for j in full_N] for i in full_N]

# And each of the subgrids
grids = [
    [[[grid[i + di][j + dj] for j in N] for i in N] for dj in (0, OFFSET)]
    for di in (0, OFFSET)
]

# We'll append all our constraints here
constraints = []

As before, I’m keeping track of the grid values, with a separate convenience array for the subgrids. As opposed to ILP, z3 is expressive enough that I don’t have to directly introduce multiple separate variables for different conditions like nz, panel, etc.

Constraints are set similarly, and can also be tagged. The constraints list will contain tuples of the form (tag, constraint), as the tag can be useful for debugging. You can ask z3 to print out the tags of some set of unsatisfiable constraints (whenever it finds one), so you can track down bugs more easily. Ironically, I didn’t actually make proper use of this feature, since I noticed the bug I was searching for during the process of meticulously tagging each constraint.

# Manually encode the grid
constraints.extend(
    [
        ("g[1][3] = 1", grid[1][3] == 1),
        ("g[1][9] = 6", grid[1][9] == 6),
        ("g[1][10] = 5", grid[1][10] == 5),
        ("g[2][5] = 3", grid[2][5] == 3),
        ("g[2][10] = 6", grid[2][10] == 6),
        ("g[3][1] = 4", grid[3][1] == 4),
        ("g[3][8] = 7", grid[3][8] == 7),
        ("g[4][4] = 2", grid[4][4] == 2),
        ("g[4][11] = 7", grid[4][11] == 7),
        ("g[5][2] = 6", grid[5][2] == 6),
        ("g[5][8] = 3", grid[5][8] == 3),
        ("g[5][9] = 7", grid[5][9] == 7),
        ("g[8][3] = 7", grid[8][3] == 7),
        ("g[8][5] = 5", grid[8][5] == 5),
        ("g[9][1] = 5", grid[9][1] == 5),
        ("g[9][5] = 7", grid[9][5] == 7),
        ("g[10][1] = 6", grid[10][1] == 6),
        ("g[10][2] = 7", grid[10][2] == 7),
        ("g[11][4] = 6", grid[11][4] == 6),
    ]
)

I didn’t set bounds for the variables when I constructed them, so lets do that here. Notice that z3 directly supports convenient constructs like z3.And. There is also z3.Or, z3.Implies, z3.If, and many others – pay attention for these throughout. These functions make it much easier to encode the constraints than having to reason through encoding them with ILP.

# Bounds for all the integers
constraints.extend(
    [
        (f"bounds_[{i}][{j}]", z3.And(grid[i][j] >= 0, grid[i][j] <= MAX_VAL))
        for i, j in product(full_N, full_N)
    ]
)

The following lengthy code block implements again the 24-7 puzzle logic, but this time in z3. Since I’ve already explained it, I don’t do so again, except for the comments in the code itself. Consider comparing this code with the ILP implementation.

def encode_247_puzzle(subgrid_ix, flow_source: tuple):
    """
    The flow_source is an integer tuple for where flow originates from.
    """

    i_sg, j_sg = subgrid_ix
    subgrid = grids[i_sg][j_sg]

    def _neighbours(i, j):
        if i + 1 < len(N):
            yield (i + 1, j)
        if i - 1 >= 0:
            yield (i - 1, j)
        if j + 1 < len(N):
            yield (i, j + 1)
        if j - 1 >= 0:
            yield (i, j - 1)

    constraints = []

    # For every row and column
    for row in N:
        # They must contain exactly four non-zero numbers
        constraints.append(
            (
                f"row_count4_{i_sg}{j_sg}[{row}]",
                sum(z3.If(subgrid[row][j] > 0, 1, 0) for j in N) == 4,
            )
        )

        # And sum up to exactly 20
        constraints.append((f"row_sum20_{i_sg}{j_sg}[{row}]", sum(subgrid[row]) == 20))
    for col in N:
        constraints.append(
            (
                f"col_count4_{i_sg}{j_sg}[{col}]",
                sum(z3.If(subgrid[i][col] > 0, 1, 0) for i in N) == 4,
            )
        )
        constraints.append(
            (f"col_sum20_{i_sg}{j_sg}[{col}]", sum(subgrid[i][col] for i in N) == 20)
        )

    # Must contain one 1, two 2s, ..., seven 7s.
    for k in K:
        constraints.append(
            (
                f"counts_{i_sg}{j_sg}[{k}]",
                sum(z3.If(subgrid[i][j] == k, 1, 0) for i, j in product(N, N)) == k,
            )
        )

    # Every 2x2 subsquare must contain at least one zero
    for i in N[:-1]:
        for j in N[:-1]:
            constraints.append(
                (
                    f"2x2_{i_sg}{j_sg}[{i}][{j}]",
                    (
                        z3.If(subgrid[i][j] > 0, 1, 0)
                        + z3.If(subgrid[i + 1][j] > 0, 1, 0)
                        + z3.If(subgrid[i][j + 1] > 0, 1, 0)
                        + z3.If(subgrid[i + 1][j + 1] > 0, 1, 0)
                    )
                    <= 3,
                )
            )

    # Again implementing path connectedness by using the flow model
    # Outgoing flow from cell (i, j) to neighbouring cell (p, q)
    flow = [
        [
            [
                (p, q, z3.Int(f"flow_{i_sg}{j_sg}[{i}][{j}][{p}][{q}]"))
                for (p, q) in _neighbours(i, j)
            ]
            for j in N
        ]
        for i in N
    ]
    for i, j in product(N, N):
        for p, q, f in flow[i][j]:
            constraints.append((f"f_{i_sg}{j_sg}[{i}][{j}][{p}][{q}] >= 0", f >= 0))
            constraints.append(
                (f"f_{i_sg}{j_sg}[{i}][{j}][{p}][{q}] <= M", f <= len(N) ** 2)
            )

    # Sum up all the outgoing flow
    outgoing_flow = [[0 for j in N] for i in N]
    for i, j in product(N, N):
        for _, _, f in flow[i][j]:
            outgoing_flow[i][j] += f

    # And separately keep track of the incoming flow
    incoming_flow = [[0 for j in N] for i in N]
    for i, j in product(N, N):
        for p, q, f in flow[i][j]:
            incoming_flow[p][q] += f

    # Cells which are zero must have zero flow through them
    for i, j in product(N, N):
        constraints.append(
            (
                f"flowbal_{i_sg}{j_sg}[{i}][{j}]",
                z3.Implies(
                    subgrid[i][j] == 0,
                    z3.And(outgoing_flow[i][j] == 0, incoming_flow[i][j] == 0),
                ),
            )
        )

    # The source has exactly enough outgoing flow to fill the network
    # This time, I directly summed up the count of non-zero entries.
    i_source, j_source = flow_source
    constraints.append(
        (f"sourceinc_{i_sg}{j_sg}", incoming_flow[i_source][j_source] == 0)
    )
    constraints.append(
        (
            f"sourceout_{i_sg}{j_sg}",
            outgoing_flow[i_source][j_source]
            == sum(z3.If(subgrid[i][j] > 0, 1, 0) for i, j in product(N, N)) - 1,
        )
    )

    # Conserve flow in the network
    for i, j in product(N, N):
        if i == i_source and j == j_source:
            # Except the source is unconstrained
            continue

        # non-zero cells must have positive incoming flow
        # which ensures they're connected to the source
        constraints.append(
            (
                f"nzflow+_{i_sg}{j_sg}[{i}][{j}]",
                z3.Implies(subgrid[i][j] > 0, incoming_flow[i][j] > 0),
            )
        )

        # The outgoing flow must be one less than the incoming flow,
        # i.e., each node sinks 1 unit of flow.
        constraints.append(
            (
                f"sink+_{i_sg}{j_sg}[{i}][{j}]",
                outgoing_flow[i][j]
                == incoming_flow[i][j] - z3.If(subgrid[i][j] > 0, 1, 0),
            )
        )

    return constraints

This function is now being used as a subroutine – each subgrid needs to be a valid 24-7 puzzle. I specify the flow sources for each subgrid manually and arbitrarily from the fixed given data.

# Pick some arbitrary fixed nodes for the zero groups.
constraints.extend(encode_247_puzzle((0, 0), (1, 3)))
constraints.extend(encode_247_puzzle((1, 0), (0, 2)))
constraints.extend(encode_247_puzzle((0, 1), (2, 0)))
constraints.extend(encode_247_puzzle((1, 1), (0, 3)))

Now, there are some additional rules described for the 4-in-1 puzzle. Specifically, recall:

Some numbers have been placed inside the grid. Additionally, some blue numbers have been placed outside the grid. A number outside the grid represents either the sum of the row or column it is facing, or the value of the first number it sees in that row or column.

This constraint is a bit confusing, but ultimately we can just encode it with a sequence of nested if/else statements. I’ve constructed this with a recursive function as follows, but one could just as easily write down the constraints “manually” with a keyboard macro .

def blue_num_constraint(i: int, j: int, di: int, dj: int, num):
    """
    Starting at position i, j and moving in the direction di, dj,
    either the sum of the whole line is num OR the first non-zero
    value encountered is equal to num.
    """

    def nest_if_else(m):
        if m >= len(full_N):
            # Terminal condition: we reached the end of the row or column
            return True
        else:
            g = grid[i + m * di][j + m * dj]
            # If g > 0 then it must equal num.  Otherwise, check the next cell.
            return z3.If(g > 0, g == num, nest_if_else(m + 1))

    return (
        f"blue_[{i}][{j}][{di}][{dj}][{num}]",
        # Either the sum of the row or column is num, or we go into the nested if else
        # statements to constraint the value of the first non-zero entry.
        z3.Or(
            sum(grid[i + m * di][j + m * dj] for m in full_N) == num, nest_if_else(0)
        ),
    )

We now need to manually write down all the constraints from the figure. I enumerate them in counter-clockwise order, starting from the top-left corner.

# Down the left side
constraints.append(blue_num_constraint(0, 0, 0, 1, 5))
constraints.append(blue_num_constraint(1, 0, 0, 1, 7))
constraints.append(blue_num_constraint(2, 0, 0, 1, 7))
constraints.append(blue_num_constraint(3, 0, 0, 1, 33))
constraints.append(blue_num_constraint(4, 0, 0, 1, 29))
constraints.append(blue_num_constraint(5, 0, 0, 1, 2))
constraints.append(blue_num_constraint(6, 0, 0, 1, 40))
constraints.append(blue_num_constraint(7, 0, 0, 1, 28))
constraints.append(blue_num_constraint(10, 0, 0, 1, 36))

# left-to-right along the bottom
constraints.append(blue_num_constraint(11, 0, -1, 0, 6))
constraints.append(blue_num_constraint(11, 3, -1, 0, 4))
constraints.append(blue_num_constraint(11, 11, -1, 0, 5))

# Up the right side
constraints.append(blue_num_constraint(11, 11, 0, -1, 7))
constraints.append(blue_num_constraint(3, 11, 0, -1, 1))
constraints.append(blue_num_constraint(0, 11, 0, -1, 4))

# right-to-left along the top
constraints.append(blue_num_constraint(0, 10, 1, 0, 7))
constraints.append(blue_num_constraint(0, 7, 1, 0, 27))
constraints.append(blue_num_constraint(0, 6, 1, 0, 40))
constraints.append(blue_num_constraint(0, 5, 1, 0, 3))
constraints.append(blue_num_constraint(0, 4, 1, 0, 27))
constraints.append(blue_num_constraint(0, 3, 1, 0, 34))
constraints.append(blue_num_constraint(0, 2, 1, 0, 30))
constraints.append(blue_num_constraint(0, 1, 1, 0, 36))
constraints.append(blue_num_constraint(0, 0, 1, 0, 6))

Finally, we instantiate and call the solver.

def solve_constraints(constrs):
    solver = z3.Solver()

    # Add all the tagged constraints
    for tag, constr in constrs:
        solver.assert_and_track(constr, tag)

    # This will enable the solver to tell you about a subset of unsat constraints.
    solver.set(unsat_core=True)

    if solver.check() == z3.sat:
        solution = solver.model()
        print_full_grid(solution, grid)
        return solution
    else:
        print("INFEASIBLE")

        # The unsat core is some subset of unsat constraints.
        # It need not be minimal in any sense, but may be useful for
        # debugging.
        print("Unsat subset: ", solver.unsat_core())

solution = solve_constraints(constraints)

0, 5, 0, 6, 0 | 3, 6 | 0, 0, 0, 7, 4
0, 7, 7, 1, 0 | 0, 5 | 0, 4, 6, 5, 0
0, 0, 0, 7, 5 | 3, 5 | 0, 6, 0, 6, 0
6, 4, 3, 0, 0 | 7, 0 | 5, 7, 1, 0, 0
7, 0, 0, 0, 2 | 7, 4 | 2, 0, 0, 0, 7
------------------------------------
2, 0, 6, 6, 6 | 0, 0 | 6, 3, 7, 0, 4
5, 4, 4, 0, 7 | 0, 0 | 7, 0, 6, 2, 5
------------------------------------
7, 0, 0, 0, 1 | 5, 7 | 1, 0, 0, 7, 0
0, 5, 3, 7, 0 | 5, 0 | 0, 5, 0, 4, 6
6, 5, 0, 0, 0 | 7, 2 | 0, 6, 0, 0, 5
0, 6, 7, 3, 0 | 0, 4 | 6, 6, 4, 0, 0
0, 0, 0, 4, 6 | 3, 7 | 0, 0, 3, 7, 0

Which is a correct solution! We can also check that the solution is unique by adding the negation of the solution back as another constraint and then re-solving.

constraints.append(
    (
        "soln1",
        z3.Or([grid[i][j] != solution[grid[i][j]] for i, j in product(full_N, full_N)]),
    )
)

solve_constraints(constraints)

INFEASIBLE
Unsat subset:  [g[1][3] = 1,
 g[1][9] = 6,
 g[1][10] = 5,
 g[2][5] = 3,
 g[2][10] = 6,
 g[3][1] = 4,
 g[3][8] = 7,
 g[4][4] = 2,
 g[4][11] = 7,
 g[5][2] = 6,
 g[5][8] = 3,
 g[5][9] = 7,
 g[8][3] = 7,
 g[8][5] = 5,
 g[9][1] = 5,
 g[9][5] = 7,
 g[10][1] = 6,
 g[10][2] = 7,
 g[11][4] = 6,
 bounds_[0][0],
 bounds_[0][1],
 bounds_[0][2],
 bounds_[0][3],
 bounds_[0][4],
 bounds_[0][6],
 bounds_[0][7],
 bounds_[0][8],
 bounds_[0][9],
 bounds_[0][10],
 bounds_[1][0],
 bounds_[1][1],
 bounds_[1][2],
 bounds_[1][3],
 bounds_[1][4],
 bounds_[1][5],
 bounds_[1][6],
 bounds_[1][7],
 bounds_[1][8],
 bounds_[1][11],
 bounds_[2][0],
 bounds_[2][1],
 bounds_[2][2],
 bounds_[2][3],
 bounds_[2][4],
 bounds_[2][6],
 bounds_[2][7],
 bounds_[2][8],
 bounds_[2][9],
 bounds_[2][11],
 bounds_[3][0],
 bounds_[3][1],
 bounds_[3][2],
 bounds_[3][3],
 bounds_[3][4],
 bounds_[3][5],
 bounds_[3][6],
 bounds_[3][7],
 bounds_[3][8],
 bounds_[3][9],
 bounds_[3][10],
 bounds_[3][11],
 bounds_[4][0],
 bounds_[4][1],
 bounds_[4][2],
 bounds_[4][3],
 bounds_[4][5],
 bounds_[4][6],
 bounds_[4][7],
 bounds_[4][8],
 bounds_[4][9],
 bounds_[4][10],
 bounds_[4][11],
 bounds_[5][1],
 bounds_[5][4],
 bounds_[5][5],
 bounds_[5][6],
 bounds_[5][7],
 bounds_[5][9],
 bounds_[5][10],
 bounds_[5][11],
 bounds_[6][0],
 bounds_[6][1],
 bounds_[6][3],
 bounds_[6][4],
 bounds_[6][5],
 bounds_[6][6],
 bounds_[6][7],
 bounds_[6][8],
 bounds_[6][9],
 bounds_[6][10],
 bounds_[6][11],
 bounds_[7][0],
 bounds_[7][1],
 bounds_[7][2],
 bounds_[7][3],
 bounds_[7][4],
 bounds_[7][5],
 bounds_[7][6],
 bounds_[7][7],
 bounds_[7][8],
 bounds_[7][9],
 bounds_[7][10],
 bounds_[7][11],
 bounds_[8][0],
 bounds_[8][1],
 bounds_[8][2],
 bounds_[8][3],
 bounds_[8][4],
 bounds_[8][6],
 bounds_[8][7],
 bounds_[8][8],
 bounds_[8][9],
 bounds_[8][10],
 bounds_[8][11],
 bounds_[9][0],
 bounds_[9][2],
 bounds_[9][3],
 bounds_[9][4],
 bounds_[9][5],
 bounds_[9][6],
 bounds_[9][7],
 bounds_[9][8],
 bounds_[9][9],
 bounds_[9][10],
 bounds_[9][11],
 bounds_[10][0],
 bounds_[10][1],
 bounds_[10][2],
 bounds_[10][3],
 ...]

This proves that the solution I found is unique, and also serves to illustrate the “unsat subset” functionality.

Remark: You can find my name on the solution page of this puzzle, so you know I’m no fraud. But similarly as the vanilla 24-7 puzzle, I didn’t fully encode the entire problem as I’ve done now. Instead, since I only cared about getting to a final result, I manually narrowed down some of the constraints with my own direct reasoning, or through sufficient conditions, and then printed out all the candidate solutions (there were 8) with ILP. I finally found the only correct one ad-hoc, and computed the product of the areas of white space by hand.

Conclusion

We’ve seen how both Integer Linear Programming (ILP) and Satisfiability Modulo Theories (SMT), a generalization of SAT, can be used to solve constraint programming problems. These tools are incredibly powerful, and it seems likely that most practical “day to day” discrete optimization problems I might care to solve can be easily implemented and solved with ortools or z3. I find z3 to be considerably more expressive than ILP, and it is probably a good go-to tool for constraint programming. That being said, there are plenty of mathematical optimization problems which can be encoded as ILPs, so if optimization is the goal, rather than simple constraint satisfaction, ILP might be the right choice.

All About Quadratic Forms

Tue, 18 Jul 2023 00:00:00 -0700

Quadratic forms are functions defined through symmetric matrices and represent a ubiquitous class of functions for which there is an enormous amount of useful theoretical and computational results. Indeed, in “linear-quadratic” models, it is possible to provide analytic solutions to more-or-less any question you would like to ask. This post is a tour of some foundations and results relating to quadratic forms.

Introduction

Quadratic forms are functions $f: \R^n \rightarrow \mathbb{R}$ defined by a constant $\kappa \in \mathbb{R}$, a vector $q \in \mathbb{R}^n$ and a matrix $Q \in \mathbb{S}^n$ ($\mathbb{S}^n$ denoting the set of symmetric $n \times n$ matrices), namely, $f(x) = \frac{1}{2}x^{\mathsf{T}} Q x + q^{\mathsf{T}} x + \frac{1}{2}\kappa$. That the matrix $Q$ is assumed symmetric is without loss of generality since if $Q$ were not symmetric, we could instead consider $\frac{1}{2}Q + \frac{1}{2}Q^{\mathsf{T}}$ without affecting the value of the function. The $\frac{1}{2}$’s in the definition of $f$ will be seen to be a convenient convention.

Firstly, we can recognize that it is possible to expand this out explicitly as \[f(x) = \frac{1}{2}\sum_{i = 1}^n \sum_{j = 1}^n Q_{ij}x_i x_j + \sum_{i = 1}^n q_i x_i + \frac{1}{2}\kappa.\] Thus, the function $f$ is defined through a linear combination of all pairs of products in $x$, but not higher order powers. Another convenient way to write QFs is through a single matrix, augmenting $x$ with a $1$

\begin{equation} f(x) = \frac{1}{2}\begin{bmatrix}x\\ 1\\ \end{bmatrix}^{\mathsf{T}}\begin{bmatrix}Q & q\\ q^{\mathsf{T}} & \kappa \end{bmatrix}\begin{bmatrix}x\\ 1 \end{bmatrix}, \notag \end{equation}

so for the most part, techniques that work for the simpler case with $q = 0$ and $\kappa = 0$, also apply to the more general case.

Assumption: For the purposes of this post, I will usually assume that $Q$ is invertible. This assumption is not essential, but if this does not hold, then the quadratic form is degenerate and breaks down into a lower-dimensional full-rank quadratic form (restricted to the span of the columns of $Q$) and a linear form restricted to the perpendicular complement thereof. It is analogous to the assumption that $a \ne 0$ in the classical quadratic polynomial $ax^2 + bx + c$.

From Whence?

Where do quadratic forms arise from? At the most superficial level, they are nothing but a natural multivariate generalization of the quadratic polynomial $ax^2 + bx + c$. However, they also arise in extremely natural ways in numerous applications. They are truly ubiquitous.

Least Squares: The least-squares cost function $f(x) = \frac{1}{2}||y - Ax||_2^2$ is a quadratic form $f(x) = \frac{1}{2}x^{\mathsf{T}} A^{\mathsf{T}} A x - (A^{\mathsf{T}} y)^{\mathsf{T}} x + \frac{1}{2}||y||_2^2$ where $Q = A^{\mathsf{T}} A, q = A^{\mathsf{T}} y$ and $\kappa = ||y||_2^2$. The least-squares cost function arises in statistical regression, estimation, as a design objective, etc.

Variance of a Sum: If we have some vector random variable $Z$ with variance matrix $\Sigma \in \mathbb{S}^n$, then the variance of the linear combination $\text{Var}[Z^{\mathsf{T}} x] = x^{\mathsf{T}} \Sigma x$ is a quadratic form when viewed as a function of $x$.

Local Approximation: Generally, differentiable functions $f(x)$ can be expanded into a Taylor series about some point $x_0$ as in $f(x) = f(x_0) + \mathsf{D}f(x_0) (x - x_0) + \frac{1}{2} (x - x_0)^{\mathsf{T}} \mathsf{D}^2 f(x_0)(x - x_0) + R(x)$ where the remainder term $R(x)$ is a $3^{\text{rd}}$ order polynomial of the components of $(x - x_0)$. Truncating the $R(x)$ term results in a quadratic form that gives a local approximation of the function $f$ near the point $x_0$. These approximations are used in (second order) iterative optimization algorithms – particularly Newton’s method, sequential least squares, and interior point algorithms.

Gaussian Distributions: The Gaussian distribution is one of the most important distribution functions in probability. It is characterized by two parameters $\mu \in \mathbb{R}^n$ (the mean) and $\Sigma \in \mathbb{S_{+}^n}$ (the positive semidefinite variance matrix). When $\Sigma$ is positive definite (hence invertible) Gaussian random variables have the density function $f(x) \propto \text{exp}\bigl(\frac{1}{2}(x - \mu)^{\mathsf{T}} \Sigma^{-1} (x - \mu) \bigr)$, which involves a quadratic form. Moreover, the characteristic function $\psi(t) = \text{exp}\bigl(j\mu^{\mathsf{T}} t - \frac{1}{2}t^{\mathsf{T}} \Sigma t\bigr)$ is also a quadratic form in $t$. (Remark: The definition of the Gaussian should most properly be given in terms of $\psi$, instead of the density, since the expression for $\psi$ does not require that $\Sigma$ be invertible!)

Use in Optimization: A huge number of general optimization problems, each of which have their own space of applications, involve quadratic forms in the objective function. Examples are given by Quadratic Programming (including Support Vector Machines ), the Quadratic Assignment Problem , the Linear Quadratic Regulator , and many more.

Completing the Square

For the classical quadratic polynomial $ax^2 + bx + c$, to “complete the square” is an elementary algebraic operation wherein we eliminate the linear term $bx$ by writing $ax^2 + bx + c = a(x + b / (2a))^2 + c - b^2 / (4a^2)$. This procedure enables one to derive the quadratic formula, identify minimizers and maximizers, etc. There is an analogous procedure for general quadratic forms:

\begin{equation} \begin{aligned} f(x) &= \frac{1}{2}x^{\mathsf{T}} Q x + q^{\mathsf{T}} x + \frac{1}{2}\kappa\\ &= \frac{1}{2}(x + Q^{-1} q)^{\mathsf{T}} Q (x + Q^{-1} q) + \frac{1}{2}\kappa - q^{\mathsf{T}} Q^{-1} q, \end{aligned}\notag \end{equation}

where the $Q^{-1}$ can be replaced by a pseudo-inverse $Q^\dagger$ if $Q$ is not invertible but merely $q \in \mathcal{R}(Q)$. Completing the square is a useful technique for derivations involving quadratic forms – particularly, it will help us to recognize minimizers of $f$, and it is commonly used for manipulating expressions relating to Gaussian distributions. I solve a few optimization problems in this post, and nowhere do I resort to differentiation.

Optimization

Quadratic forms are extremely important in optimization. Since they are so tractable, many algorithms for solving general optimization problems proceed by solving quadratic optimization problems in sequence, iteratively refining the approximation as they proceed.

Convexity

There is a particular class of quadratic forms which are of particular interest in optimization – these are convex quadratic forms. We will see later that, more generally, quadratic forms can be decomposed into convex, concave, and linear subspaces.

Recall that a function $f$ is convex if \[\forall t \in (0, 1), x, y \in \mathsf{dom}\ f: f(tx + (1 - t)y) \le tf(x) + (1 - t)f(y).\] This means that the line joining two points on the graph of $f$ is above the graph itself. Intuitively, convex functions “look like bowls” and their importance stems, at least in part, from the fact that any local minimizer is also a global minimizer. Thus, for example, gradient descent applied to convex functions is guaranteed to converge to a global minimizer of the function. A function is concave if its negative is convex – these functions look like upside down bowls, and gradient ascent will converge to a global maximizer.

How do we determine when a quadratic form is convex? Without loss of generality, we consider simply the function $f(x) = \frac{1}{2}x^{\mathsf{T}} Q x$ and calculate (notice that we need not resort to computing second order derivatives!):

\begin{equation} \begin{aligned} &\ f(tx + (1 - t)y) - tf(x) - (1 - t)f(y)\\ &= \frac{1}{2}t^2 x^{\mathsf{T}} Q x + t(1 - t) x^{\mathsf{T}} Q y + \frac{1}{2}(1 - t)^2 y^{\mathsf{T}} Q y - \frac{1}{2}tx^{\mathsf{T}}Q x - \frac{1}{2}(1 - t) y^{\mathsf{T}} Q y\\ &= \frac{1}{2}(t^2 - t) x^{\mathsf{T}} Q x + t(1 - t) x^{\mathsf{T}} Q y + \frac{1}{2}\bigl((1 - t)^2 - (1 - t)\bigr)y^{\mathsf{T}} Q y\\ &= -\frac{1}{2}t(1 - t) x^{\mathsf{T}} Q x + t(1 - t) x^{\mathsf{T}} Q y - \frac{1}{2}t (1 - t) y^{\mathsf{T}} Q y \\ &= -\frac{1}{2}t(1 - t) \bigl[x^{\mathsf{T}} Q x - 2x^{\mathsf{T}} Q y + y^{\mathsf{T}} Q y\bigr] \\ &= -\frac{1}{2}t(1 - t) (x - y)^{\mathsf{T}} Q (x - y)\\ &\overset{(a)}{\le} 0, \end{aligned} \notag \end{equation}

where if the final inequality $(a)$ holds over all $x, y, t$, then we will have verified the definition of convexity for the function $f$. Since $t \in (0, 1)$ it must be that $t(1 - t) \in (0, 1)$ as well, and because $x, y$ appear only as the difference $(x - y)$, we can consider simply the condition where $\forall z:\ z^{\mathsf{T}} Q z \ge 0$. This is exactly what it means for the matrix $Q$ to be positive semi-definite, denoted $Q \succeq 0$.

Thus (also using the fact that a convex function plus the linear function $x \mapsto q^{\mathsf{T}} x$ is still convex), a quadratic form $f$ is convex if and only if $Q \succeq 0$. If the matrix $Q$ is both non-singular and positive semi-definite, then it is positive-definite written as $Q \succ 0$, which also means that the inequality in the definition of convexity holds strictly, and we call the function strictly convex.

Remark: It is worth noting here that every quadratic form associated with a Gaussian random variable is a convex quadratic form (since variance matrices are positive semidefinite). Moreover, for smooth functions, they are locally convex in the region of minimizers, and the local quadratic approximations of convex functions are themselves always convex. As well, the quadratic form $\frac{1}{2}||Ax||_2^2 = x^{\mathsf{T}} A^{\mathsf{T}} A x$ will always be convex as well since $A^{\mathsf{T}} A \succeq 0$. Convex quadratic forms arise almost as often as quadratic forms themselves!

Minimizing Quadratic Forms

From the representation of the quadratic form $f$ after completing the square \[f(x) = \frac{1}{2}(x + Q^{-1} q)^{\mathsf{T}} Q (x + Q^{-1} q) + \kappa - q^{\mathsf{T}} Q^{-1} q,\] it appears rather immediate to identify a minimizer: \[x^\star = - Q^{-1} q.\] However, we must be careful – in making this claim we have already assumed that a minimizer exists, and the existence of minimizers should not be taken lightly. If the matrix $Q$ is indefinite (i.e., $Q$ nor $-Q$ are positive semi-definite), then the point $-Q^{-1} q$ may be only a saddle point and in fact no minimizer exists: $\underset{x \in \mathbb{R}^n}{\text{inf}}\ f(x) = -\infty$. The question of existence in infinite dimensions is incredibly interesting , but it is straightforward in the present finite-dimensional case: we need only eliminate the possibility that the infimum is $-\infty$ by the strict convexity assumption $Q \succ 0$ (more generally, we can have $Q \succeq 0$ and $q \in \mathcal{R}(Q)$). If $Q \prec 0$, then $x^\star$ will be a maximizer.

The actual value of the function $f$ at the minimizer can also be easily identified by inspection after completing the square:

\[f^\star := f(x^\star) = -q^{\mathsf{T}} Q^{-1} q + \kappa.\]

It is an interesting observation that this is a concave quadratic form in $q$.

The Shape of a Quadratic Form

We can get some intuition about convex and non-convex quadratic forms from the contour plots below. In the convex case, the function is a prototypical “bowl-shaped” function with a clearly defined minimizer, and the contour lines are concentric ellipses. In the non-convex case, the function is hyperbolic and the only minimizing sequences diverge. In this hyperbolic case, the point $-Q^{-1} q$ is a saddle point.

The prototypical hyperbolic function is $f(x, y) = xy$. However, it is clearly possible for functions containing $xy$ terms to still be convex, e.g., $f(x, y) = \frac{1}{2}(x^2 - xy + y^2)$ (the function on the left in the figure), but not $f(x, y) = \frac{1}{2}(x^2 - 2xy + y^2)$ (the function on the right). Whether or not the function contains the pernicious hyperbolic behaviour comes down to the question of whether or not $Q \succeq 0$.

Subspace Constraints

There are many important cases, particularly in optimization, where we want to restrict the domain of the function $f$ to some subspace (actually, an affine space, which may include a constant offset from $0$) $\mathbb{V} \subseteq \R^n$, of some dimension strictly less than $n$, i.e., $f: \mathbb{V} \rightarrow \R$. In practice, this typically arises through linear constraints $Ax = b$ on the variable $x$, where $A \in \R^{n \times m}$. The representation $\mathbb{V} = \{x\ |\ Ax = b\}$ however is not analytically insightful.

To obtain a more friendly representation of the space $\mathbb{V}$, take a (compact) Singular Value Decomposition of the matrix: $A = U\Sigma V^{\mathsf{T}}$ wherein $U \in \R^{n \times r}$, $\Sigma \in \R^{r \times r}$ and $V \in \R^{m \times r}$; I use $r \le \text{min}(m, n)$ to indicate the rank of the matrix $A$. It is essential to assume that $b \in \mathcal{R}(A)$, i.e., that $b$ is in the range of $A$ – if this is not the case then the subspace $\mathbb{V}$ is ill-defined. Now, recall that the columns of $U$ constitute an orthonormal basis of $\mathcal{R}(A)$ of dimension $r$ and that the columns of $V$ constitute an orthonormal basis for $\mathcal{N}(A)^\perp$, the orthogonal complement of the nullspace of $A$, which is of dimension $n - r$. Operationally, this means that $U^{\mathsf{T}} U = I_r$ and $V^{\mathsf{T}} V = I_r$ (but definitely not, at least in general, that $UU^{\mathsf{T}} = I_n$ or $VV^{\mathsf{T}} = I_m$). Thus, we can simplify the representation of the subspace $\mathbb{V}$ as in:

\begin{equation} \begin{aligned} \mathbb{V} &= \{x\ |\ Ax = b\}\\ &= \{x\ |\ U\Sigma V^{\mathsf{T}} x = b\}\\ &\overset{(a)}{=} \{x\ |\ V^{\mathsf{T}} x = \Sigma^{-1} U^{\mathsf{T}} b\}\\ &\overset{(b)}{=} \{V \Sigma^{-1} U^{\mathsf{T}} b + \overline{V}u\ |\ u \in \R^{n - r}\}\\ &\overset{( c)}{=} \{A^{\dagger} b + \overline{V}u\ |\ u \in \R^{n - r}\}, \end{aligned}\notag \end{equation}

where in $(a)$ it is always the case that $\Sigma$ is invertible (and also diagonal) for a compact SVD and in $(b)$ the matrix $\overline{V} \in \R^{n \times (n - r)}$ is a matrix whose columns form a basis for the nullspace of $A$ (which can be obtained, for example, by running a full SVD on $A$), and in $( c)$ we use the definition of the Moore-Penrose inverse of $A$, denoted by $A^{\dagger} = V\Sigma^{-1} U^{\mathsf{T}}$. The vector $\bar{x} := V \Sigma^{-1} U^{\mathsf{T}} b$ is usually taken to be the nominal solution of $Ax = b$ (it is easy to verify that it is a solution) and the remainder of the degrees of freedom are provided by the span of the columns of $\overline{V}$. This makes more clear that $\mathbb{V}$ is an affine space: the subspace $\{\bar{V}u\ |\ u \in \R^{n - r}\}$ is offset from $0$ by $A^\dagger b$.

From here, a quadratic form $f$ subject to the constraint $x \in \mathbb{V}$ can be represented simply by modifying the parameters $Q, q, \kappa$ and using the variable $u$ from which the original $x$ variable can be recovered by $x(u) = A^{\dagger} b + \overline{V}u$:

\begin{equation} \begin{aligned} f(x) &= \frac{1}{2}x^{\mathsf{T}} Q x + q^{\mathsf{T}} x + \frac{1}{2}\kappa; \quad Ax = b\\ \rightarrow f(x(u)) &= \frac{1}{2}(A^{\dagger} b + \overline{V}u)^{\mathsf{T}} Q (A^{\dagger} b + \overline{V}u) + q^{\mathsf{T}} (A^{\dagger} b + \overline{V}u) + \frac{1}{2}\kappa\\ \rightarrow \tilde{f}(u) &= \frac{1}{2}u^{\mathsf{T}} \overline{V}^{\mathsf{T}} Q \overline{V} u + \bigl((\overline{V}^{\mathsf{T}} q + \overline{V}^{\mathsf{T}} QA^\dagger b \bigr)^{\mathsf{T}} u + q^{\mathsf{T}} A^\dagger b + \frac{1}{2}\kappa + \frac{1}{2} (A^\dagger b)^{\mathsf{T}} Q (A^{\dagger} b)\\ &= \frac{1}{2}u^{\mathsf{T}} \tilde{Q} u + \tilde{q}^{\mathsf{T}} u + \tilde{\kappa}\\ \end{aligned}\notag \end{equation}

where $\tilde{Q} = \overline{V}^{\mathsf{T}} Q \overline{V}$, $\tilde{q} = \overline{V}^{\mathsf{T}} (q + QA^\dagger b)$ and $\tilde{\kappa} = \kappa + (A^\dagger b)^{\mathsf{T}} Q (A^{\dagger} b)$. Thus, the quadratic form restricted to a subspace $\mathbb{V}$ is yet again just another quadratic form. Moreover, there are computationally tractable means of explicitly representing the subspace $\mathbb{V}$ when it is described through the linear system $Ax = b$.

Convex and Concave Subspaces

To get some further geometric sense of the function $f$, one will usually appeal to the eigendecomposition of the matrix $Q$. Since it is symmetric, it is guaranteed to be orthogonally diagonalizable, and admits of real eigenvalues. This is The Spectral Theorem . Specifically, we have \[Q = V\Lambda V^{\mathsf{T}} = \sum_{i = 1}^N \lambda_i v_i v_i^{\mathsf{T}} \] where $V \in \mathbb{R}^{n \times n}$ is an orthogonal matrix of eigenvectors, and $\Lambda = \mathsf{Dg}(\lambda_1, \ldots, \lambda_n)$ is a diagonal matrix containing the associated eigenvalues. The space can be split into three subspaces, call them

\begin{equation} \begin{aligned} S_+ &= \mathsf{span}\bigl\{v_i\ |\ \lambda_i > 0\}\\ S_- &= \mathsf{span}\bigl\{v_i\ |\ \lambda_i < 0\}\\ S_0 &= \mathsf{span}\bigl\{v_i\ |\ \lambda_i = 0\}. \end{aligned} \end{equation}

The subspace $S_+$ might be called the “convex subspace” since the function $x \mapsto \frac{1}{2} x^{\mathsf{T}} Q x$ is convex when restricted to $x \in S_+$, and $S_-$ might be called the “concave subspace” since $x \mapsto \frac{1}{2} x^{\mathsf{T}} Q x$ is concave when restricted to $x \in S_-$. The subspace $S_0$ is the linear subspace since $\forall x \in S_0:\ \frac{1}{2} x^{\mathsf{T}} Q x = 0$ – when a quadratic form is restricted to $S_0$ it degenerates into a linear function. The subspace $S_0 = \{0\}$ if $Q$ is non-singular, and $S_- = \{0\}$ if $Q \succeq 0$.

Remark: There are practical reasons why these subspace distinctions are interesting. In some natural problems, we may encounter a matrix $Q$ which is indefinite, but where certain problem constraints guarantee that we are fully confined to the convex subspace $S_+$. Therefore, given the constraints of the problem, the quadratic form associated with $Q$ is “effectively” convex.

Using ideas from the previous section, we can animate 2-d slices of a quadratic form in $\R^4$ where the 2-d slices are constructed to smoothly vary across convex and concave subspaces. The transition between the two, where the subspace passes through $S_+$ and $S_-$ could be termed a hyperbolic subspace of the quadratic form.

I created this animation by constructing the matrix $Q = 2q_1 q_1^{\mathsf{T}} + q_1 q_1^{\mathsf{T}} - p_1 p_1^{\mathsf{T}} - 2 p_2 p_2^{\mathsf{T}}$ where $q_1, p_1, q_2, p_2$ form an orthogonal basis of $\mathbb{R}^4$, and then using a matrix \[H(t) = [\frac{3}{2}(\frac{2}{3} - t)_+ q_1 + 3(t - \frac{2}{3})_+ p_1, \frac{3}{2}(t - \frac{1}{3})_+ p_2 + 3(\frac{1}{3} - t)_+ q_2]\] to project $x = H(t) u$ onto convex, hyperbolic, and concave subspaces depending on the value of $t$. The plot is over $u \in \mathbb{R}^2$ and animated over $t \in [0, 1]$.

Linear Algebra

The linear algebra related to quadratic forms largely comes down to the linear algebra associated with the symmetric matrix $Q$. The main aspects are the Cholesky factorization and Schur complements. The Cholesky decomposition is one of the most important (and fast!) matrix factorizations available, and the Schur complement arises in many different ways. Moreover, these two aspects are intimately linked.

Schur Complements

Quadratic forms often arise as bivariate quadratic forms (or more generally, multivariate quadratic forms with natural distinctions between different blocks of variables). For now, consider the function of $x \in \mathbb{R}^{n_x}$ and $y \in \mathbb{R}^{n_y}$ given by

\begin{equation} \begin{aligned} f(x, y) &= \frac{1}{2}\begin{bmatrix}x\\ y\\ 1\end{bmatrix}^{\mathsf{T}}\begin{bmatrix}P & C & p\\ C^{\mathsf{T}} & Q & q\\ p^{\mathsf{T}} & q^{\mathsf{T}} & \kappa \end{bmatrix}\begin{bmatrix}x\\ y\\ 1\end{bmatrix}\\ &= \frac{1}{2}x^{\mathsf{T}}Px + \frac{1}{2}y^{\mathsf{T}}Qy + x^{\mathsf{T}} C y + p^{\mathsf{T}} x + q^{\mathsf{T}} y + \kappa \end{aligned} \notag \end{equation}

where I’ll assume $P, Q$ are both symmetric, that $P$ is non-singular, and that $C$ is such that the overall block matrix is symmetric positive definite. These assumptions mean that the function $f(x, y)$ is jointly convex in the variables $(x, y)$.

There are two other interesting functions associated with $f$, namely, \[y \mapsto \underset{x}{\text{inf}}\ f(x, y),\] and \[x \mapsto \underset{y}{\text{inf}}\ f(x, y).\]

The computation of these functions can be done through a mechanical procedure called Schur-complementation. The idea of the Schur complement can serve as a natural “chunk” in one’s cognition and enables easy manipulation of complicated expressions.

Partial Minimization

The above functions are those formed from partial minimization of the function $f$, and for concreteness I’ll focus on the function $g(y) := \underset{x}{\text{inf}}\ f(x, y).$ We know from the previous section that we can guarantee that a minimizer exists if $P \succ 0$, so we will also make this assumption here to avoid degeneracy.

To identify the minimizer of $f$ with respect to $x$, we need only make the identifications $q \leftarrow Cy + p$ and $\kappa \leftarrow y^{\mathsf{T}} Q y + q^{\mathsf{T}} y + \kappa$ in the original general definition of a quadratic form to recognize that $x^\star(y) = -P^{-1} (Cy + p)$. An explicit formula for the function $g$ is then obtained through the calculation

\begin{equation} \begin{aligned} g(y) &= \frac{1}{2}y^{\mathsf{T}} Q y - y^{\mathsf{T}} C^{\mathsf{T}} P^{-1} (C y + p) + q^{\mathsf{T}} y + \kappa \\ &= \frac{1}{2}y^{\mathsf{T}} (Q - C^{\mathsf{T}} P^{-1} C) y + (q - C^{\mathsf{T}}P^{-1} p)^{\mathsf{T}} y + \kappa\\ &= \frac{1}{2}\begin{bmatrix}y\\ 1 \end{bmatrix}^{\mathsf{T}} \begin{bmatrix}Q - C^{\mathsf{T}} P^{-1} C & (q - C^{\mathsf{T}}P^{-1} p)\\ (q - C^{\mathsf{T}}P^{-1} p)^{\mathsf{T}} & \kappa \end{bmatrix} \begin{bmatrix}y\\ 1\end{bmatrix}. \end{aligned} \notag \end{equation}

The beauty of this partial minimization is that $g$ is still a quadratic form. If it is still convex (we’ll check this next!) then we could say that the set of positive definite quadratic forms is closed under the operation of partial minimization.

The formulas above seem rather complicated at first. However, they follow a completely mechanical pattern and you can write down analytic expressions for cartoonishly complicated block quadratic forms with simple notation, and understand at a glance exactly what is going on (even if you’re not reading every single equation in detail).

Definiteness Conditions

What we have discovered through exploring this is called a Schur Complement . Suppose for simplicity that $p = q = 0$ and $\kappa = 0$ and for sake of notation that \[\mathcal{Q} := \begin{bmatrix}P & C\\ C^{\mathsf{T}} & Q\end{bmatrix}.\] Then, $g(y) = y^{\mathsf{T}}(Q - C^{\mathsf{T}} P^{-1} C)y$ and the matrix $Q - C^{\mathsf{T}} P^{-1} C$ is called The Schur Complement of $P$ in $\mathcal{Q}$ and it is usually denoted $\mathcal{Q} / P$. There is also a Schur complement of $Q$ in $\mathcal{Q}$ given by $\mathcal{Q} / Q := P - CQ^{-1}C^{\mathsf{T}}$. The way to remember which is which is to think of the “$/P$” as a form of division, analogous to $P^{-1}$ (now you remember which matrix is being inverted) and then to just match up the shapes of $C$ and $C^{\mathsf{T}}$ with $P$ or $Q$ to figure out if you should write $C^{\mathsf{T}} P^{-1} C$ or $CP^{-1} C^{\mathsf{T}}$.

Forming the matrix $\mathcal{Q} / P$ requires at least that $P$ be non-singular. If, in addition, it is positive definite $P \succ 0$, then $g(y) = y^{\mathsf{T}} (\mathcal{Q} / P) y$ is the partial minimizer of the whole quadratic form with respect to $x$. In order for $g(y)$ to have a minimizer (i.e., the infimum is not $-\infty$) then the Schur complement must be positive-definite as well: $\mathcal{Q} / P \succ 0$. This, combined with the the fact that \[\underset{x}{\text{inf}}\ \underset{y}{\text{inf}}\ f(x, y) = \underset{y}{\text{inf}}\ \underset{x}{\text{inf}}\ f(x, y) = \underset{x, y}{\text{inf}}\ f(x, y),\] implies the following methods of checking definiteness:

If $P \succ 0$ and $\mathcal{Q} / P \succ 0$, then $\mathcal{Q} \succ 0$.
If $\mathcal{Q} \succ 0$ and $P$ is non-singular, then $P \succ 0$ and $\mathcal{Q} / P \succ 0$.

These relations have important theoretical use.

Square Roots and Cholesky Factorization

There is a natural way of discovering the Cholesky Decomposition by continuing the minimization of the block quadratic form $f(x, y)$. Indeed, computing the minimizer $(x, y)$ of this function involves solving the linear system

\begin{equation} \begin{aligned} \begin{bmatrix}P & C\\ C^{\mathsf{T}} & Q\end{bmatrix} \begin{bmatrix}x\\ y \end{bmatrix} = \begin{bmatrix}p\\ q \end{bmatrix}, \end{aligned} \notag \end{equation}

involving the matrix $\mathcal{Q}$ discussed above. Following the idea of (block) Gauss-Jordan elimination on this matrix, the Schur complement makes another appearance:

\begin{equation} \begin{aligned} \begin{bmatrix}P & C\\ C^{\mathsf{T}} & Q\end{bmatrix} &= \begin{bmatrix}I & 0\\ C^{\mathsf{T}}P^{-1} & I\end{bmatrix}\begin{bmatrix}P & C\\ 0 & Q - C^{\mathsf{T}} P^{-1} C\end{bmatrix}, \end{aligned} \notag \end{equation}

which is a block LU decomposition of $\mathcal{Q}$, having a (dense) Schur-complement in the lower-right entry of the block upper triangular matrix. Now, rather than a block triangular decomposition, what we would really like to have is a bona-fide triangular decomposition $\mathcal{Q} = LU$ where $L$ is lower-triangular and $U$ is upper triangular. Moreover, in analogy with non-negative numbers, which have square-roots, we would like to ask for even more of a positive definite matrix $\mathcal{Q} \succ 0$ (the positive semi-definite case is possible, but more involved) – we would like an LU-decomposition $\mathcal{Q} = LL^{\mathsf{T}}$, i.e., where $U = L^{\mathsf{T}}$, which is a matrix square root.

Operating under the standing assumption that $\mathcal{Q} \succ 0$, and that $P$ is non-singular (hence $P \succ 0$ by the above), we will obtain this factorization by induction. Make the induction hypothesis that any positive definite matrix admits of a lower-triangular square root and let $P = L_{11}L_{11}^{\mathsf{T}}$. As well, using the induction hypothesis again, along with the definiteness theorem described in the last section, we know that the Schur complement $\mathcal{Q} / P = Q - C^{\mathsf{T}} P^{-1} C \succ 0$ is also positive definite and thus admits another decomposition $\mathcal{Q} / P = L_{22}L_{22}^{\mathsf{T}}$. We can now write

\begin{equation} \begin{aligned} \mathcal{Q} &= \begin{bmatrix}I & 0\\ C^{\mathsf{T}}P^{-1} & I\end{bmatrix}\begin{bmatrix}P & C\\ 0 & Q - C^{\mathsf{T}} P^{-1} C\end{bmatrix}\\ &= \begin{bmatrix}I & 0\\ C^{\mathsf{T}} (L_{11} L_{11}^{\mathsf{T}})^{-1} & I\end{bmatrix}\begin{bmatrix}L_{11} L_{11} & C\\ 0 & L_{22}L_{22}^{\mathsf{T}} \end{bmatrix}\\ &= \begin{bmatrix}L_{11} & 0\\ C^{\mathsf{T}} L_{11}^{-\mathsf{T}} & L_{22} \end{bmatrix}\begin{bmatrix}L_{11} & 0\\ C^{\mathsf{T}} L_{11}^{-\mathsf{T}} & L_{22} \end{bmatrix}^{\mathsf{T}}, \end{aligned} \notag \end{equation}

which is a Cholesky decomposition obtained recursively from the decomposition of $P$ and the Schur complement $\mathcal{Q} / P$. Beginning the recursion with $P = \mathcal{Q}_{11}$ (the upper left entry) and $L_{11} = \sqrt{P}$ leads to a practical algorithm for computing the decomposition. The definiteness condition of $\mathcal{Q} \succ 0$ is essential to continuing this process since otherwise we would not be able to recurse on the Schur complement $\mathcal{Q} / P$. What would happen in practice is that we would try to divide by zero or to take the square root of a negative number.

import numpy as np
from math import sqrt

FLOAT = np.float64

class LinAlgError:
    pass

def cholesky(A):
    """An extremely terrible Cholesky implementation."""
    A = np.array(A, dtype=FLOAT)
    n = A.shape[0]  # A = [[P; C]; [C'; Q]]
    if len(A.shape) != 2 or A.shape[1] != n:
        raise ValueError(f"A must be a square matrix but got {A.shape=}")
    if not np.all(A == A.T):
        raise ValueError("A must be a symmetric matrix!")

    L = np.zeros_like(A)
    Q = np.copy(A)
    for i in range(n):
        P = float(Q[0, 0])
        C = Q[0, 1:][None, :]  # a row vector
        Q = Q[1:, 1:]

        try:
            L[i, i] = sqrt(P)  # This will detect negative eigenvalues
        except ValueError:
            raise LinAlgError("Matrix A is not positive definite")

        L[i + 1 :, i] = (1 / L[i, i]) * C
        Q = Q - C.T @ (C / P)  # Division by P will fail on a *semi*-definite matrix
    return L

for n in np.random.randint(3, 30, size=1000):
    A = np.random.normal(size=(n, n))
    A = A @ A.T
    L = cholesky(A)
    assert np.allclose(L @ L.T, A)

return "Passed :)"  # org-mode recognizes this odd 'return' as the output

`1`	`Passed :)`

Remark (Testing Definiteness): The Cholesky decomposition is not only guaranteed to succeed if $\mathcal{Q} \succ 0$, it is also guaranteed to fail if $\mathcal{Q} \succ 0$ does not hold. Thus, computationally, the Cholesky decomposition serves as a reliable means of testing the definiteness of a matrix: a matrix $A$ is positive definite if and only if a Cholesky decomposition algorithm successfully terminates in a factorization $A = LL^{\mathsf{T}}$.

Remark (Computing Schur Complements): Even though we’ve derived the Cholesky decomposition from the Schur complement, we can also go the other way. That is, the Cholesky decomposition algorithm can be used to compute Schur complements. Moreover, this results directly in the Cholesky decomposition of the Schur complement itself, and therefore in a representation of $\mathcal{Q} / P$ which provides a numerical guarantee on the positive definiteness of the Schur complement. To see this, consider the calculations below:

\begin{equation} \begin{aligned} \mathcal{Q} &= \begin{bmatrix}P & C\\ C^{\mathsf{T}} & Q\end{bmatrix}\\ &= \begin{bmatrix}L_{11} & 0\\ L_{12} & L_{22} \end{bmatrix}\begin{bmatrix}L_{11}^{\mathsf{T}} & L_{12}^{\mathsf{T}} \\ 0 & L_{22}^{\mathsf{T}} \end{bmatrix}\\ &= \begin{bmatrix}L_{11}L_{11}^{\mathsf{T}} & L_{11}L_{12}^{\mathsf{T}} \\ L_{12}L_{11}^{\mathsf{T}} & L_{12}L_{12}^{\mathsf{T}} + L_{22}L_{22}^{\mathsf{T}} \end{bmatrix} \end{aligned} \notag \end{equation}

whereby we can calculate the Schur complement $$\mathcal{Q} / P = L_{12}L_{12}^{\mathsf{T}} + L_{22}L_{22}^{\mathsf{T}} - L_{12}L_{11}^{\mathsf{T}} (L_{11}L_{11}^{\mathsf{T}})^{-1} L_{11}L_{12} = L_{22}L_{22}^{\mathsf{T}}.$$ Thus, the Cholesky decomposition of $\mathcal{Q} / P$ is nothing but the bottom-right block of the Cholesky decomposition of the block matrix $\mathcal{Q}$. The advantage of this method is in numerical stability: the subtraction $Q - C^{\mathsf{T}} P^{-1} C$ not only involves a pesky inversion of the matrix $P$, but it also involves the difference of two positive definite matrices, which, due to numerical roundoff, can result in a matrix which is in actuality indefinite. Using this method, if the Cholesky algorithm succeeds, then we are in the clear.

Probability and Statistics

Quadratic forms appear in many places in statistic, largely as a result of computing second order statistics (the mean and the variance). Remarkably, though I derived them from examples in optimization, both the Schur complement and the Cholesky factorization play important roles in this theory as well.

Computations with the Mean and the Variance

Two of the most important statistics of an $\mathbb{R}$-valued random variable, call it $X$, are the mean and the variance $\mu = \mathbb{E}X$ and $\sigma^2 = \mathbb{E}(X - \mu)^2$. Taken together, $(\mu, \sigma^2)$ are the second-order statistics of $X$. The mean value measures the central tendency of $X$, serving as one estimate of its “typical” value, and the variance $\sigma^2$ is the average squared distance of $X$ from its mean value. When $\sigma^2$ is small, then the probability that $X$ is close to $\mu$ is high, and when $\sigma^2$ is large, that probability is small.

For vectors of random variables $\mathbf{X} = (X_1, \ldots, X_n) \in \mathbb{R}^n$ the mean is simply defined element-wise \[\mathbb{E}\mathbf{X} = \begin{bmatrix} \mathbb{E}X_1\\ \vdots\\ \mathbb{E}X_n\\ \end{bmatrix} = \begin{bmatrix}\mu_1\\ \vdots \\ \mu_n \end{bmatrix} = \bm{\mu}.\] The variance of the vector random variable becomes somewhat more complicated, as the best definition is now a matrix. This variance matrix, usually denoted $\Sigma \in \mathbb{R}^{n \times n}$, is defined through an outer-product of the centered random variables \[\Sigma = \mathbb{E}(\mathbf{X} - \mathbf{\mu})(\mathbf{X} - \mathbf{\mu})^{\mathsf{T}},\] and includes all of the individual (i.e., marginal) variance $\Sigma_{ii} = \mathbb{E}(X_i - \mu_i)^2$, but also all of the co-variances $\Sigma_{ij} = \mathbb{E}(X_i - \mu_i)(X_j - \mu_j)$. These covariance terms provide a measure of how the random variables in the vector tend to move together – i.e., they provide a quantitative measure of dependence. The reason for this interpretation may become clear when we look at estimating one random variable given another.

Sampling

It is easy to see that the covariance matrix is symmetric – indeed, $\Sigma_{ij} = \mathbb{E}(X_i - \mu_i)(X_j - \mu_j) = \mathbb{E}(X_j - \mu_j)(X_i - \mu_i) = \Sigma_{ji}$. What is somewhat less obvious is that it is also positive semi-definite $\Sigma \succeq 0$. To see this:

\begin{equation} \begin{aligned} a^{\mathsf{T}} \Sigma a^{\mathsf{T}} &= a^{\mathsf{T} }\bigl[\mathbb{E}(\mathbf{X} - \bm{\mu})(\mathbf{X} - \bm{\mu})^{\mathsf{T}} \bigr] a\\ &\overset{(a)}{=} \mathbb{E} a^{\mathsf{T}} (\mathbf{X} - \bm{\mu})(\mathbf{X} - \bm{\mu})^{\mathsf{T}} a\\ &\overset{(b)}{=} \mathbb{E} ||a^{\mathsf{T}} (\mathbf{X} - \bm{\mu})||_2^2 \ge 0, \end{aligned}\notag \end{equation}

where $(a)$ uses the linearity of expectation $a\mathbb{E}X = \mathbb{E}[aX]$ and $(b)$ uses $a^{\mathsf{T}} a = ||a||_2^2 := \sum_{i = 1}^n a_i^2$. For simplicity, I will assume that $\Sigma$ is positive definite $\Sigma \succ 0$, which is generally the case – if $\Sigma$ is not positive definite, then it can be shown that (at least) two components of the vector $\mathbf{X}$ must be perfectly linearly related, e.g., $X_1 = X_2 - X_3$, a case we simply exclude.

Since $\Sigma \succ 0$, it must admit a Cholesky factorization $\Sigma = LL^{\mathsf{T}}$. This factorization can be used computationally to sample random variables with a prescribed variance matrix. As long as it is possible to sample independent random variables with variance $1$, then we can construct a random vector with variance equal to the $n \times n$ identity matrix $I_n$ (how?) – call this vector $\mathbf{Z}$. Supposing that $\mathbf{Z}$ has zero mean $\mathbb{E}\mathbf{Z} = 0$ (adding back a constant mean vector $\bm{\mu}$ afterwards is easy) we can now transform $\mathbf{Z}$ so that it has variance $\Sigma$: let $\mathbf{X} = L\mathbf{Z}$ and calculate:

\begin{equation} \begin{aligned} \mathbb{E}\mathbf{XX}^{\mathsf{T}} &= \mathbb{E}(L\mathbf{Z})(L\mathbf{Z})^{\mathsf{T}}\\ &= \mathbb{E}L\mathbf{Z}\mathbf{Z}^{\mathsf{T}} L^{\mathsf{T}}\\ &= L\bigl[\mathbb{E}\mathbf{Z}\mathbf{Z}^{\mathsf{T}} \bigr]L^{\mathsf{T}}\\ &= LI_n L^{\mathsf{T}}\\ &= \Sigma. \end{aligned}\notag \end{equation}

Incidentally, a slight modification to the calculation above can show us that if $\mathbf{X}$ has variance matrix $\Sigma$, then for any matrix $A$, the variance of $A\mathbf{X}$ is given by $A\Sigma A^{\mathsf{T}}$.

Linear Minimum Mean Squared Error Estimation

To understand more about the meaning of the variance matrix, let’s see how it appears in a particularly important quadratic optimization problem: the linear minimum mean-squared error estimator (LMMSE estimator). The setup for this problem is that we have a joint random variable $(X, Y)$ where $X$ has dimension $m$ and $Y$ has dimension $n$, and where $X, Y$ have a known joint variance matrix:

\begin{equation} \text{Var}\begin{bmatrix}X\\ Y\end{bmatrix} = \begin{bmatrix}\Sigma_X & R\\ R^{\mathsf{T}} & \Sigma_Y \end{bmatrix} := \Sigma \notag \end{equation}

$\Sigma \in \mathbb{S}_{++}^{n + m}$, and mean values $\mu_X \in \mathbb{R}^m \mu_Y \in \mathbb{R}^n$. We want to find a “gain matrix” $K \in \mathbb{R}^{m \times n}$ and a “bias vector” $b \in \mathbb{R}^{n}$ to solve the following problem:

\begin{equation} \begin{aligned} \underset{K, b}{\text{minimize}}\ \frac{1}{2}\mathbb{E}||X - KY - b||_2^2. \end{aligned}\notag \end{equation}

The interpretation of this problem is that we are making an actual observation of $Y$, and want to use that to make some inference about what the value of $X$ is, given the joint second order statistics (mean and variance) of $(X, Y)$. An example application is that $X$ is some “state” we are interested in (e.g., temperature, position, velocity, etc.) and $Y$ is the measurement of some sensor system.

To solve this problem, first start by minimizing over $b$. If we expand the objective we have

\begin{equation} \begin{aligned} \frac{1}{2}\mathbb{E} ||X - KY - b||_2^2 &= \frac{1}{2}\mathbb{E}||X - KY||_2^2 + \frac{1}{2}||b||_2^2 - \mathbb{E}(X - KY)^{\mathsf{T}} b\\ &= \frac{1}{2}\bigl((\mu_X - K\mu_Y) - b\bigr)^{\mathsf{T}} \bigl((\mu_X - K\mu_Y) - b\bigr) + \frac{1}{2}\mathbb{E}||X - KY||_2^2 - (\mu_X - K\mu_Y)^{\mathsf{T}}(\mu_X - K\mu_Y), \end{aligned}\notag \end{equation}

from which it is clear the optimal value is $b = \mu_X - K\mu_Y$. It is quite convenient that the optimal value of $b$ makes $X - KY - b$ have zero mean. Because of this we can simply assume, without loss of generality, that $\mathbb{E}X = 0$ and $\mathbb{E}Y = 0$; indeed, we could make the substitutions $X_0 = X - \mu_X$ etc., but this is unnecessarily cumbersome. So, making this zero mean assumption, we again expand out the objective function

\begin{equation} \begin{aligned} \mathbb{E}||X - KY||_2^2 &= \mathbb{E}(X - KY)^{\mathsf{T}}(X - KY)\\ &= \mathbb{E}\bigl( X^{\mathsf{T}} X - X^{\mathsf{T}}KY - Y^{\mathsf{T}}K^{\mathsf{T}}X - Y^{\mathsf{T}}K^{\mathsf{T}}KY \bigr)\\ &= \mathsf{tr}\bigl(\Sigma_X - KR^{\mathsf{T}} - R^{}K^{\mathsf{T}} + K\Sigma_Y K^{\mathsf{T}} \bigr)\\ &= \mathsf{tr}(K - R\Sigma_Y^{-1})\Sigma_Y(K - R\Sigma_Y^{-1})^{\mathsf{T}} + \mathsf{tr}(\Sigma_X - R \Sigma_Y^{-1} R^{\mathsf{T}}), \end{aligned}\notag \end{equation}

where in the last step we are again completing the square. The second trace term does not depend upon $K$, and the first term is minimized by the choice $K = R\Sigma_Y^{-1}$ (why? Obtain a generalization for quadratic forms and use $\Sigma_Y \succ 0$). Remarkably, the value of the objective at the optimum is given exactly by

\begin{equation} \underset{K}{\text{min}}\ \mathbb{E}||X - KY||_2^2 = \mathsf{tr}(\Sigma_X - R \Sigma_Y^{-1} R^{\mathsf{T}}) = \mathsf{tr}(\Sigma / \Sigma_Y), \notag \end{equation}

which is another Schur complement! More generally, at $K^\star = R\Sigma_Y^{-1}$ we have that $\text{Var}(X - K^\star Y) = \Sigma / \Sigma_Y$ so that the variance matrix of the error vector $E = X - KY$ is given exactly by a Schur complement in the original joint variance matrix. This Schur complement is measuring the amount of variance which is removed from $\Sigma_X$ by conditioning on $Y$. Another worthwhile observation is that $\mathsf{tr}E^{\mathsf{T}} K^\star Y = \mathsf{tr}(X - R\Sigma_Y^{-1} Y)^{\mathsf{T}} R\Sigma_Y^{-1} Y = 0$, which is to say that the estimate is orthogonal to the error. This is not a coincidence and is a consequence of the Projection Theorem .

We can construct an illustrative example of these calculations, along with the sampling scheme of the previous section in two dimensions. For this illustration I’m using the variance matrix $\Sigma = \begin{bmatrix}\sigma_X^2 & \rho\sigma_X \sigma_Y \\ \rho \sigma_X \sigma_Y & \sigma_Y^2 \end{bmatrix}$ and thus the optimal gain is $K = \rho \sigma_X \sigma_Y \times \frac{1}{\sigma_y^2} = \rho \frac{\sigma_X}{\sigma_Y}$ which leads to $\hat{X}|Y = \rho \frac{\sigma_X}{\sigma_Y} Y$, and $\sigma^2_{X|Y} = (1 - \rho)\sigma_X^2$ which are all worthwhile formulas to commit to memory. The particular values used in the below example are $\sigma_X^2 = 4, \sigma_Y^2 = 1, \rho = 0.8$. The way that the estimator changes in response to changes in parameters can be visualized by inspecting the equations provided earlier – in particular, the gain $K$ and the conditional variance $\sigma^2_{X|Y}$ both change linearly in response to changes in $\rho$ or $\sigma_X^2$.

It may at first look a bit odd that the regression line is not aligned with the principle axis of the contour plot. This is not a mistake. The regression line which is aligned with that “natural looking” direction is obtained from Principle Component Regression , and is sub-optimal in this setting where we know exact statistics. If the statistics $\Sigma$ have to be estimated from available data, then it is possible that the regularization effect got from looking only at some top $k$ principle components gives us overall lower squared error on a test set.

Remark (MMSE Estimator): As opposed to the LMMSE estimator, the MMSE estimator is the optimal estimator which is not constrained to be linear. The MMSE in general depends upon all of the higher order statistics, and cannot be computed without full knowledge of the joint distribution of $X, Y$.

Remark (The Gaussian Distribution): These formulas also arise in the conditioning formulas for the Gaussian distribution . This is not exactly a coincidence – Gaussian’s are fully characterized by their second order statistics and, as a family of distributions, are closed under linear combinations. In the Gaussian case, the MMSE coincides with the LMMSE.

Linear Regression

Let’s look at yet another example: linear regression. This is a problem which is closely related to Linear Minimum Mean Squared Error Estimation , but instead of using the observations $Y$ of a sensor to estimate some state variable $X$, we consider a finite dataset $\mathcal{D}_N = \{(x_i, y_i)\}_{i = 1}^N$ of $N$ examples. The goal of linear regression is to find some model function $f$ such that the average squared error of the model \[y_i \approx f(x_i)\] is minimized. Formally, we would write

\begin{equation} \underset{f \in \mathcal{F}}{\text{minimize}}\ \frac{1}{2N}\sum_{i = 1}^N \big(y_i - f(x_i)\big)^2 \tag{$R_{\mathcal{F}}$} \end{equation}

where $\mathcal{F}$ is some “admissible class” of functions.

The simplest useful class of functions $\mathcal{F}$ are linear functions $f(x) = x^{\mathsf{T}} \beta$, parameterized by some vector $\beta$. Specializing the problem to this case is the problem of (ordinary) linear regression. By expanding the objective function we can easily pluck out the optimizer:

\begin{equation} \begin{aligned} \frac{1}{2N}\sum_{i = 1}^N \big(y_i - f(x_i)\big)^2 &= \frac{1}{2N}\sum_{i = 1}^N \big(y_i - x_i^{\mathsf{T}} \beta \big)^2\\ &= \frac{1}{2N}\sum_{i = 1}^N y_i^2 - \frac{1}{N}\sum_{i = 1}^N y_i (x_i^{\mathsf{T}} \beta) + \frac{1}{2N}\sum_{i = 1}^N (x_i^{\mathsf{T}} \beta)^2\\ &= \kappa - \big(\frac{1}{N}\sum_{i = 1}^N y_i x_i \big)^{\mathsf{T}} \beta + \frac{1}{2N}\sum_{i = 1}^N \mathsf{tr}\ (x_i^{\mathsf{T}} \beta)(x_i^{\mathsf{T}} \beta) \\ &= \kappa - \big(\frac{1}{N}\sum_{i = 1}^N y_i x_i \big)^{\mathsf{T}} \beta + \frac{1}{2} \beta^{\mathsf{T}} \big(\frac{1}{N}\sum_{i = 1}^N x_i x_i^{\mathsf{T}} \big) \beta, \end{aligned}\notag \end{equation}

and hence \[\beta^\star_{lr} = \big( \frac{1}{N}\sum_{i = 1}^N x_i x_i^{\mathsf{T}} \big)^{-1} \big( \frac{1}{N}\sum_{i = 1}^N y_i x_i \big).\]

The whole problem can be made considerably simpler by means of matrix notation

\begin{equation} \mathbf{y} = \begin{bmatrix}y_1\\ \vdots\\ y_N \end{bmatrix}, \mathbf{X} = \begin{bmatrix}x_1^{\mathsf{T}} \\ \vdots \\ x_N^{\mathsf{T}} \end{bmatrix},\notag \end{equation}

so that

\begin{equation} \begin{aligned} \frac{1}{2N} \sum_{i = 1}^N \bigl( y_i - x_i^{\mathsf{T}} \beta \bigr)^2 &= \frac{1}{2N}||\mathbf{y} - \mathbf{X}\beta||_2^2\\ \implies \beta^\star_{lr} = \bigl(\mathbf{X}^{\mathsf{T}} \mathbf{X}\bigr)^{-1} \mathbf{X}^{\mathsf{T}} \mathbf{y}, \end{aligned}\notag \end{equation}

which is a rather famous expression.

Remark (Admissible Functions): Contrary to initial impressions, regression problem $R_\mathcal{F}$ can actually be solved, in closed form, at least up to an $\epsilon$ of approximation error, for an extremely large class of functions $\mathcal{F}$. Indeed, for just one example when each $x$ is a single real number (to lighten notation) $x_i \in \R$ (as opposed to a vector) and $\mathcal{F}$ is the class of all bounded and continuous functions, then any $f \in \F$ can be (uniformly) approximated to arbitrary accuracy using a polynomial $f_a(x) \approx \sum_{k = 0}^K a_k x^k$ parameterized by the coefficients $a$ (this is the Stone-Weierstrass Theorem ). Each sample $x_i$ can then be transformed into a vector $(1, x_i, x_i^2, \ldots, x_i^K)$, and we can perform linear regression with the parameters $\beta = (1, a_1, \ldots, a_K)$.

Remark: The way I’ve written the optimal solution $\beta_{lr}^\star$ is naive. Computationally, the problem will usually be solved by means of a QR or Singular Value decomposition of the data matrix $X$ or by an iterative descent algorithm. Analytically, it is also possible to write the optimal solution using the pseudo-inverse as in $\beta_{lr}^\star = X^\dagger y$, which is much more notationally compact, and doesn’t assert the existence of an inverse matrix. Indeed, writing out $\bigl(\mathbf{X}^{\mathsf{T}} \mathbf{X}\bigr)^{-1} \mathbf{X}^{\mathsf{T}} \mathbf{y}$ is a plain waste of ink.

Conclusion

This post is a bit of a whirlwind tour of fundamental facts about quadratic forms and positive definite matrices. All of these facts are truly elementary pieces of the computational linear algebra toolkit, and serve as building blocks for more sophisticated problems. Indeed, a substantial motivation for writing all of this down is that now I can fearlessly use all these facts in future posts without feeling like I’m skipping over important details!

I hope you, dear reader (I trust there is at least one 🙏), have enjoyed this tour and can now comfortably apply this knowledge to your own problems :)

Generalized Eigenvalue Problems and Trace Optimization

Tue, 18 Jul 2023 00:00:00 -0700

Most people are familiar with the concept of matrix eigenvalues. Less well known is that this concept can be fruitfully expanded to the generalized eigenvalues of pairs of matrices. Closely related are matrix trace optimization problems, which extremizes the trace of certain matrix products. Trace optimization constites a large class of practically solvable non-convex optimization problems commonly useful for dimensionality reduction and which includes the unsupervised weighted principle component analysis and the supervised method of Fisher’s Linear Discriminant. The purpose of this post is to explore some of these problems, their intuition, and their applications.

Introduction

The eigenvalues of a matrix are profoundly important quantities and are usually encountered early on in one’s mathematical career. The concept of the eigenvalues of a single matrix $A$ can be expanded to the generalized eigenvalues of a pair of matrices $(A, B)$. My purpose here is to summarize and distill some of what I’ve learned about the theory and application of generalized eigenvalues.

Spectral Theory

Before discussing eigenvalue problems, we need to set up some basic notation and review critical facts about matrices.

Consider an $n \times n$ matrix $A \in \R^{n \times n}$. The eigenvalues $\lambda_i \in \mathbb{C}$ and eigenvectors $v_i \in \mathbb{C}^n$ of the matrix $A$ are those (complex!) scalars and vectors such that \[Av_i = \lambda_i v_i.\] That is, the eigenvectors are the “directions” in which the operation of the matrix $A$ are “simple”, i.e., nothing but multiplication by a scalar. It is one of the hallmarks of linear algebra that the eigenvectors of an $n \times n$ matrix will often (not always) happen to provide us with a basis for $\mathbb{R}^n$, i.e., we can use the eigenvectors as coordinate axes. This is incredibly useful when working with the matrix $A$, since the matrix does nothing but scale the axes in this basis, a rather simple operation. When this is possible, we say that the matrix is diagonalized by the change of basis. Conditions of matrix diagonalizability are captured by the Spectral Theorem, so called because the collection of eigenvalues of a matrix are often referred to as “the spectrum” (of the matrix).

The Spectral Theorem

To diagonalize the matrix $A$ means that we can re-write it as $A = V \Lambda V^{-1}$ where $\Lambda = \mathsf{Dg}(\mathbf{\lambda})$ is a diagonal matrix of the eigenvalues of $A$, and $V = \begin{bmatrix}v_1 & v_2 & \cdots & v_n \end{bmatrix} \in \mathbb{C}^{n \times n}$ are the eigenvectors. The matrix $V^{-1}$ represents a transformation into an eigenbasis and $V$ is a transformation out of it. Unfortunately, diagonalizing a matrix is not always possible, and matrix diagonalizability is quite a slippery issue. Fortunately though, there is a simple criteria, occuring frequently for matrices of interest in practice, that guarantees that $A$ is diagonalizable. The condition is that $A$ be real-valued and symmetric: $A = A^{\mathsf{T}}$ (more generally, $A$ can be normal: $AA^{\mathsf{T}} = A^{\mathsf{T}} A$).

Spectral Theorem: Let $A \in \R^{n \times n}$ be a symmetric matrix. Then, there exist $n$ real eigenvalues $\lambda_i$ of $A$ and $n$ eigenvectors $v_i \in \R^n$ such that $A_i v_i = \lambda_i v_i$. Moreover, the vectors $v_i$ constitute an orthonormal basis of $\R^n$.

If we express $x$ in the eigenbasis as $x = \sum_{i = 1}^n \langle v_i, x \rangle v_i$ then the matrix-vector product is $Ax = \sum_{i = 1}^n \lambda_i \langle v_i, x \rangle v_i$, and in matrix notation this is simply $Ax = V\Lambda V^{\mathsf{T}}x$ where $V = \begin{bmatrix}v_1\ \cdots v_n \end{bmatrix}$. In fact, this shows that $A = V\Lambda V^{\mathsf{T}}$, and since $V$ is an orthonormal basis we have $V^{\mathsf{T}}V = I_n = VV^{\mathsf{T}}$ meaning that $V^{-1} = V^{\mathsf{T}}$. So the eigenvectors of $A$ diagonalize $A$ itself.

The spectral theorem is one of the most important results in mathematics and generalizes in various ways to certain sorts of infinite dimensional operators.

Remark: Conventionally, the eigenvalues are sorted from low to high as in $\lambda_1(A) \le \cdots \le \lambda_n(A)$ (the opposite sorting convention is also common).

The Courant-Fischer Variational Characterization

The Courant-Fischer theorem provides a variational characterization of the eigenvalues of a symmetric matrix. Basically, a variational characterization is a way to calculate some quantity in terms of an optimization problem, the term “variational” presumably arising from the Calculus of Variations , and the idea of an infinitestimal “variation” of a function. Specifically, the smallest eigenvalue $\lambda_1(A)$ of an $n \times n$ symmetric matrix can be characterized by the optimization problem

\begin{equation} \begin{aligned} \underset{x \in \R^n}{\text{maximize}}&\ x^{\mathsf{T}} A x\\ \text{subject to}&\ ||x||_2 = 1, \end{aligned}\tag{$\lambda_n(A)$} \end{equation}

where the maximum is obtained by an eigenvector corresponding to the smallest eigenvalue $\lambda_n$, which is also the value of the problem. Variational characterizations are incredibly useful in practice – they are often providing one or both of: (a) a concrete procedure for calculating some quantity by means of an optimization algorithm or (b) a way to solve seemingly difficult optimization problems by directly calculating the quantity known to be the solution. The optimization problem I called $(\lambda_n(A))$ above falls into the second category. Particularly since it is not a convex problem, directly applying an iterative optimization algorithm to this problem is not likely to pan out well, but there do exist reliable algorithms for calculating matrix eigenvalues.

To see why the problem $(\lambda_n(A))$ provides us with the smallest eigenvalue, we recognize that it is exactly equivalent to the following:

\begin{equation} \begin{aligned} \underset{u \in \R^n}{\text{maximize}}&\ u^{\mathsf{T}} \Lambda u\\ \text{subject to}&\ ||u||_2 = 1, \end{aligned}\notag \end{equation}

where $\Lambda$ is the diagonal matrix of eigenvalues from $A = V\Lambda V^{\mathsf{T}}$. This equivalence follows since $V$ is an orthonormal basis: for any $x$ with $||x||_2 = 1$ there exists a $v$ such that $x = Vu$ and $||u||_2 = 1$ since $||Vu||_2 = \sqrt{\lang Vu, Vu\rang} = \sqrt{\lang u, V^{\mathsf{T}} Vu\rang} = ||u||_2$. The optimizer of this latter problem is clearly $u^\star = (0, 0, \ldots, 1)$ (selecting out the largest eigenvalue) with the value $\lambda_n$ and the corresponding solution of the original problem is $x = Vu^\star = v_n$, an eigenvector associated with $\lambda_n$.

Remark: Incidentally, this also shows that $\lambda_n(A)$ is a convex function of $A$ since $A \mapsto x^{\mathsf{T}} A x$ is a linear function of the matrix $A,$ and the maximum over a family of linear functions is a convex function. Similarly, it can be shown that $\lambda_1(A)$ is a concave function of $A,$ and can be obtained by replacing “maximize” with “minimize” in the variational problem.

The Courant-Fischer theorem takes this one step further by providing a variational characterization of any of the eigenvalues. Specifically,

Theorem (Courant-Fischer): Let $A \in \R^{n \times n}$ be a real symmetric matrix. Then

\begin{equation} \lambda_k = \underset{U}{\text{min}}\ \Bigl\{\underset{x \in \R^n}{\text{max}}\ x^{\mathsf{T}} A x\ \big|\ ||x||_2 = 1, x \in U, \mathsf{dim}(U) = k \Bigr\},\notag \end{equation}

where the minimization is over subspaces of $\R^n$.

Intuitively, if $x$ can live in a $k$ dimensional space, then the maximization over $x$ can always attain at least the value of the $k^{\text{th}}$ smallest eigenvalue, no matter which $k$ dimensional space it is confined to. From the optimization problem described earlier, the maximizing $x$ is given by the eigenvector corresponding to the largest eigenvalue, subject to the constraint that that eigenvector be in the space $U$. The minimizing space $U$ is then of course given by the space spanned by the $k$ smallest eigenvectors $U = \mathsf{span}\{v_1, \ldots, v_k\}$.

To finally see the connection with trace optimization problems, it follows from the Courant-Fisher theorem that the value of the optimization problem (generalizing the first variational characterization above)

\begin{equation} \begin{aligned} \underset{U \in \R^{n \times k}}{\text{minimize}}&\ \mathsf{tr}\ U^{\mathsf{T}} A U\\ \text{subject to}&\ U^{\mathsf{T}} U = I \end{aligned}\tag{${\mathcal{E}_k}$} \end{equation}

is given by $\lambda_1 + \cdots + \lambda_k$, and that an optimizing $U = \begin{bmatrix}v_1\ \cdots\ v_k \end{bmatrix}$ is given by the $n \times k$ matrix formed from the first $k$ eigenvectors. When reading this problem, be careful to remember that $U^{\mathsf{T}} U = I_k$ definitely does not imply $UU^{\mathsf{T}} = I_n$, unless $n = k$.

This optimization problem can be obtained from the Courant-Fischer theorem more-or-less by simply taking the summation of the first $k$ eigenvalues and stacking separate “$x$” optimization variables together into a matrix, along with elementary facts about traces.

Main Point: The main point I am making here is that Problem $(\mathcal{E}_k),$ (parameterized by the number $k$ of vectors in $U$) which is a non-convex optimization problem, can be solved by computing an eigendecomposition of the symmetric matrix $A.$ This is remarkable, since there exist efficient algorithms for computing this factorization.

Generalized Eigenvalues

The eigenvalue problem of finding a pair $(\lambda, v)$ for a matrix $A$ such that $Av = \lambda v$ can be generalized in various ways. One particular direction is to consider a generalized eigenvalue problem (GEVP) associated to a pair of matrices $(A, B)$ wherein we seek to find $(\lambda, v)$ such that $Av = \lambda Bv$. Such pairs are now referred to as generalized eigenpairs associated with $(A, B)$.

The case of most interest in applications is where $A$ is symmetric, and $B \succ 0$ is positive definite. Recall that a positive definite matrix can be characterized by the fact that they admit a Cholesky factorization $B = LL^{\mathsf{T}}$ where $L$ is an invertible lower-triangular matrix – this is a particular type of matrix square root. Using this Cholesky decomposition, the GEVP can be reduced to an ordinary EVP:

\begin{equation} \begin{aligned} Av &= \lambda Bv\\ \iff A L^{-\mathsf{T}} L^{\mathsf{T}} v &= \lambda LL^{\mathsf{T}} v\\ \iff \bigl(L^{-1} A L^{-\mathsf{T}}) w &= \lambda w;\ w = L^{\mathsf{T}} v, \end{aligned}\notag \end{equation}

wherein $S \overset{\Delta}{=} L^{-1} A L^{-\mathsf{T}}$ is a symmetric matrix with eigenpairs $(\lambda, w)$ from which the generalized eigenvectors $v = L^{-\mathsf{T}} w$ can be recovered. (Remark: I avoid the computation $B^{-1} A$ since this need not be symmetric.)

To understand the intuition of this situation, we first apply the fact that the eigenvectors $W = \begin{bmatrix} w_1\ \cdots\ w_n \end{bmatrix}$ of $S$ will form an orthogonal basis for $\R^n$. Thus, since $W = L^{\mathsf{T}} V$ we must have

\begin{equation} \begin{aligned} W^{\mathsf{T}} W &= I\\ \implies V^{\mathsf{T}} LL^{\mathsf{T}} V &= I\\ \implies V^{\mathsf{T}} B V &= I. \end{aligned}\notag \end{equation}

Thus, it becomes natural to construct the associated trace optimization problems for the pair $(A, B)$

\begin{equation} \begin{aligned} \underset{U \in \R^{n \times k}}{\text{minimize}}&\ \mathsf{tr}\ U^{\mathsf{T}} A U\\ \text{subject to}&\ U^{\mathsf{T}} B U = I. \end{aligned}\tag{${\mathcal{G}_k}$} \end{equation}

To reiterate, the point here is that this is a class of non-convex optimization problems which can be efficiently solved by eigenvalue algorithms. Indeed, the value of this problem is given by \[\lambda_1(A, B) + \cdots + \lambda_k(A, B),\] the sum of the first $k$ generalized eigenvalues, and the optimizing matrix $U \in \R^{n \times k}$ constitutes the first $k$ generalized eigenvectors of $(A, B)$. Keep in mind that the problem is only well defined when $A$ is symmetric, and $B$ is positive-definite (though a reasonable solution for the semidefinite case can probably still be worked out). If these assumptions do not hold, the situation becomes substantially more complicated.

Interpretations and Intuition

The generalized eigenvectors $V = \begin{bmatrix}v_1\ \cdots\ v_n \end{bmatrix}$ are still a basis for $\R^n$ (since $L$ is invertible) but they are no longer orthogonal. Instead, they are orthogonal with respect to an inner product modified by the matrix $B.$ That is, if we write $x^{\mathsf{T}} y \overset{\Delta}{=} \lang x, y \rang$ then we might generalize this by writing \[\lang x, y \rang_{B^{-1}} = x^{\mathsf{T}} B y = \lang L^{\mathsf{T}} x, L^{\mathsf{T}} y \rang,\] where the inner product $\lang \cdot, \cdot \rang_{B^{-1}}$ (which, recall, describes the geometry of the space through the relationship to angles $||x|| ||y|| \mathsf{cos}\ \theta_{xy} = \lang x, y \rang$ and distances $||x|| = \sqrt{\lang x, x \rang}$) has modified the geometry of $\R^n$. Using a subscript $B^{-1}$, rather than just $B$ for this inner product is, I believe, the convention.

We’ve now equipped $\R^n$ with two different inner products – the ordinary one $\langle x, y \rangle = x^{\mathsf{T}}y$, and the new one $\langle x, y \rangle_{B^{-1}} = x^{\mathsf{T}} B y$. To gain some intuition for this new inner product, consider the ordinary euclidean ball $\mathbb{B} = \{x \in \R^n\ |\ ||x||_2 \le 1\}$ and the ball in the norm $||x||_{B^{-1}} = \sqrt{x^{\mathsf{T}} B^{} x}$, namely \[\mathbb{B}_B = \{x \in \R^n\ |\ ||x||_{B^{-1}} \le 1\},\] which is an ellipsoid. Since if $y = L^{-\mathsf{T}} x$ and $||x||_2 \le 1$ then $||L^{\mathsf{T}} y||_2 = ||y||_{B^{-1}} \le 1$ the mapping $x \mapsto L^{-\mathsf{T}}$ transforms the ordinary euclidean ball into $\mathbb{B}_B$ as in $L^{-\mathsf{T}} \mathbb{B} = \mathbb{B}_B$.

To visualize the ball $L^{-\mathsf{T}} \mathbb{B}$ recognize that if $u_i$ is an (ordinary) eigenvector of $B$, then $u_i^{\mathsf{T}} B^{} u_i = \lambda_i(B)$ and therefore any vector proportional to $u_i$ and with an euclidean length between $0$ and $1 / \sqrt{\lambda_i(B)}$ will be inside of the ball. Essentially what this means is that the eigenvectors $u_i$ of $B$ define the principle axes of the ellipse, each of which have a length $1 / \sqrt{\lambda_i(B)}$. After stretching the space through the mapping $x \mapsto L^{-\mathsf{T}}x$ the directions associated with large eigenvalues are shortened, and directions associated with small eigenvalues of $B$ are lengthened. Another way to think about it is through a topographic map – the terrain in the direction of small eigenvalues is very steep.

The figure above depicts the transformation of Euclidean space $\R^n$ by the mapping $x \mapsto L^{-\mathsf{T}}x$. The dotted lines are points in the original space $\R^n$, and the solid lines are the result of applying the transformation. The ellipse in particular includes eigenvectors of $B$ (as calculated by scipy.linalg.eigh) scaled by $1 / \sqrt{\lambda_i(B)}$ in white. The contours are drawn for the function $x \mapsto \frac{1}{2}x^{\mathsf{T}} B x$ and I’ve used the particular matrix \[B = \begin{bmatrix}5 & 1 \\ 1 & 1 \end{bmatrix},\] which has an eigendecomposition \[B \approx \begin{bmatrix}0.230 & -0.973 \\ -0.973 & -0.230 \end{bmatrix}\begin{bmatrix}0.764 & 0 \\ 0 & 5.236 \end{bmatrix}\begin{bmatrix}0.230 & -0.973 \\ -0.973 & -0.230 \end{bmatrix}^{\mathsf{T}}.\]

Applications and Computations

Practically speaking, a reliable means of solving generalized eigenvalue problems is by means of scipy.linalg.eig (which will try to solve $Av = \lambda B v$ for any square $A, B$) and scipy.linalg.eigh (which assumes $A$ is symmetric, $B$ is positive definite and always returns a real factorization). These functions also provide the option to (efficiently) compute and return only a subset of eigenvalues (along with corresponding eigenvectors). Moreover, this subset can be defined either by indices (e.g., return the third, fourth, and fifth) or by values (e.g., return eigenvalues between $0$ and $1$). These are incredibly useful featuers in practice.

Principle Component Analysis

The most famous application of trace optimization problems is the feature-extraction or dimensionality-reduction technique known as Principle Component Analysis (PCA). Suppose we have a random variable $X \in \R^n$ with mean zero $\mathbb{E}[X] = 0$ and variance $\mathbb{V}[X] = \Sigma \succ 0$. The goal of PCA is to find a matrix $V \in \R^{n \times k}$ having orthogonal columns (so that $V^{\mathsf{T}} V = I$) and such that the total variance of $Y = V^{\mathsf{T}} X$ is maximized. The number $k$ of columns of $V$ is the key here – we will usually choose $k \ll n$ so that the idea of maximizing the variance corresponds to searching for the $k$ “most explanatory” directions in $\R^n$. Another way to think about this is that we are finding a subspace of dimension $k$ which “explains” the greatest amount of variability in $X$.

It is easy to set this problem up as a trace optimization problem. The total variance of $Y$ (which is necessarily also zero mean) is given by

\begin{equation} \begin{aligned} \mathbb{E} Y^{\mathsf{T} }Y &= \mathbb{E} \mathsf{tr}\ YY^{\mathsf{T}}\\ &= \mathbb{E} \mathsf{tr}\ V^{\mathsf{T}} X X^{\mathsf{T}} V\\ &= \mathsf{tr}\ V^{\mathsf{T}} \Sigma V.\\ \end{aligned}\notag \end{equation}

Thus, the problem comes down to solving:

\begin{equation} \begin{aligned} \underset{V \in \R^{n \times k}}{\text{maximize}}&\ \mathsf{tr}\ V^{\mathsf{T}} \Sigma V\\ \text{subject to}&\ V^{\mathsf{T}} V = I, \end{aligned}\notag \end{equation}

which is a standard trace optimization problem. The solution of which is, famously, given by $k$ eigenvectors corresponding to the $k$ largest eigenvalues of $\Sigma$.

Remark: Given an actual data matrix $X \in \R^{N \times n}$, one might naturally think to just estimate a covariance matrix $\widehat{\Sigma} \approx \frac{1}{N} X^{\mathsf{T}} X$. However, this is a mistake. The eigendecomposition of $\widehat{\Sigma}$ can be computed directly from a Singular Value Decomposition of the matrix $X$ itself. Indeed, for most purposes I am aware of, the actual computation of $X^{\mathsf{T}} X$ is unnecessary and should not be performed in practice.

Weighted Principle Component Analysis

The choice in the last section of maximizing the total variance of $Y = V^{\mathsf{T}} X$ was arbitrary. There is a natural generalization of this formulation where we will still find a matrix $V$ to compute some projection $Y = V^{\mathsf{T}} X$ as usual, but now we will weight the “importance” of the various components of $X$ by some matrix $W$ (which is often likely to be diagonal, though this is not a requirement). A natural objective is then the quantity $\mathbb{E}||V^{\mathsf{T}} W^{1/2} X||_2^2 = \mathsf{tr}\ V^{\mathsf{T}} W^{1/2} \Sigma W^{1/2} V$. This objective can be plugged into another trace optimization problem with constraint $V^{\mathsf{T}} V = I$.

Interestingly, if we absorb the factor $W^{1/2}$ into $V$ as in $\tilde{V}_W = W^{1/2} V$ we can re-write the constraint in terms of $\tilde{V}_W$ as in $V^{\mathsf{T}} V = \tilde{V}_W W^{-1} \tilde{V}_W = I$. That is, the matrix $\tilde{V}_W$ will be orthogonal with respect to the inner product $\lang\cdot, \cdot\rang_W$.

The PCA weighting scheme will not necessarily make any significant difference in regards to the estimated or reconstructed vector $\widehat{X} = \sum_{k = 1}^K \lang v_i, X \rang_W W^{-1} v_i$, particularly if both the covariance matrix of $X$ and the weight matrix $W$ are diagonal. The differences arise when there is significant correlation structure in $X$, in which case the weighting scheme can substantively impact the correlation structure of the factors $v_i$.

Remark: When reconstructing an estimate of an image (or other data) from a weighted PCA, one must be careful to carry out the linear projections with respect to the weighted inner product, otherwise the results will be nonsensical.

Rayleigh Quotients and Fisher’s Linear Discriminant

Principle component analysis is a ubiquitous technique for dimensionality reduction. However, it suffers a major drawback in that it is completely unsupervised, whereas data often come attached with class labels. For example, we may have a random variable $X$ and, associated with $X$ is a random class label $Y \in [C] = \{1, \ldots, C\}$ indicating that $X$ belongs to one of $C$ separate classes. These classes may represent a data source, a target label, or be simply some discrete feature of the data.

For some typical concrete examples, ponder these possibilities:

The data $X$ are natural language embeddings and the labels $Y$ are the language (English, French, Persian, …)
Data $X$ represent stock returns, labels $Y$ indicate the stock’s industry
$X$ is real-valued features of a house, and the class label $Y$ is a categorical variable such as the neighbourhood.

In PCA, the projection matrix $V$ is constructed from analyzing $X$ alone, disregarding the class label $Y$. Ideally, we would use the information available in $Y$ to construct $V$, and Fisher’s Linear Discriminant (FLD) is one particular method to do so. The idea is to find the projection $V$ such that samples $X_i$ which share a class label $Y_i$ are clustered close to each other, and at the same time that clusters formed by samples with different class labels are far apart.

Quantitatively, first suppose that we’ve centered our data $\mu = \mathbb{E}[X] = 0$. Then, to establish more notation, let $\mu_c = \mathbb{E}[X\ |\ Y = c]$ and $\Sigma_c = \mathbb{E}[(X - \mu_c)(X - \mu_c)^{\mathsf{T}}\ |\ Y = c]$ be the class conditional mean and variance of $X$, respectively. Finally, $\mathbb{P}(Y = c) = \pi_c$ be the probability that each data sample is from class $c$.

Now, just as we did for principle component analysis, let’s consider the variance of the projection $V^{\mathsf{T}} X$. To bring out the class labels, we will condition on the value of $Y$:

\begin{equation} \begin{aligned} \mathbf{Var}[V^\mathsf{T} X] &= V^{\mathsf{T}} \mathbb{E}(X - \mathbb{E}[X])(X - \mathbb{E}[X])^{\mathsf{T}} V\\ &= V^{\mathsf{T}} \mathbb{E}[XX^{\mathsf{T}}] V\\ &= V^{\mathsf{T}} \mathbb{E}\mathbb{E}[XX^{\mathsf{T}} \ |\ Y]V\\ &= V^{\mathsf{T}} \sum_c \pi_c \mathbb{E}[XX^{\mathsf{T}} \ |\ Y = c]V\\ &\overset{(a)}{=} V^{\mathsf{T}} \sum_c \pi_c \bigl(\mathbf{Var}[X\ |\ Y = c] + \mathbb{E}[X\ |\ Y = c]\mathbb{E}[X\ |\ Y = c]^{\mathsf{T}}\bigr) V\\ &= V^{\mathsf{T}} \sum_c \pi_c \bigl(\Sigma_c + \mu_c \mu_c^{\mathsf{T}} \bigr) V, \end{aligned}\notag \end{equation}

where $(a)$ is rearranging the variance formula $\mathbf{Var}(Z) = \mathbb{E}[ZZ^{\mathsf{T}}] - \mathbb{E}[Z]\mathbb{E}[Z]^{\mathsf{T}}$. The same result could be got by applying the Law of Total Variance , which is what I did first before seeing that the above calculation is easier.

In the context of FLD, the term $\Sigma_w = \sum_c \pi_c \Sigma_c$ is a measure of the average within-class variance (i.e., the variance of the data conditioned on a class label) and $\Sigma_b = \sum_c \pi_c \mu_c \mu_c^{\mathsf{T}}$ is the between-class variance (i.e., a measure of the separation between the classes).

The key idea of FLD is to maximize the within class variance (of the projection) $V^{\mathsf{T}} \Sigma_w V$, and minimize the between class variance $V^{\mathsf{T}} \Sigma_b V$. Projections which achieve this goal are identifying the axes which simultaneously explain the differences and similarities between classes.

To operationalize what it means to minimize the within-class variance, and maximize the between-class variance we need two things: (1) we need to turn the matrix-valued variances into a scalar and (2) we need to define the tradeoff between maximizing and minimizing. There are surely many ways of doing this, but the choice of objective function made by FLD is to minimize the following scalar cost:

\begin{equation} J(V) = \mathsf{tr} (V^{\mathsf{T}} \Sigma_w V)^{-1} (V^{\mathsf{T}} \Sigma_b V). \end{equation}

The trace serves to turn the objective into a scalar, and inverting the within-class variance means that the minimization objective $J$ will lead us to attempt to maximize that wiithin-class variance.

While the function $J(V)$ obviously looks like it has a lot of interesting structure, it may not be immediately clear what to do with it or how to optimize it (certainly, it is not a convex function of $V$). However, we can recognize an analogy – suppose that $V$ is just a vector, and that $\Sigma_b = I$. In this case, $J(v) = \frac{v^{\mathsf{T}} \Sigma_w v}{v^{\mathsf{T}} v}$, which is a Rayleigh Quotient for the matrix $\Sigma_w$. The reader familiar with how to connect Rayleigh quotients to eigenvalues should be inspired – the Rayleigh Quotient is invariant to rescaling of the vector, so let’s consider a change of basis matrix $C \in \mathbb{R}^{k \times k}$ (i.e., an invertible matrix) and check the objective:

\begin{equation} \begin{aligned} J(VC) &= \mathsf{tr}\ (C^{\mathsf{T}} V^{\mathsf{T}} BV C)^{-1} (C^{\mathsf{T}} V^{\mathsf{T}} A V C)\\ &= \mathsf{tr}\ C^{-1} (V^{\mathsf{T}} BV)^{-1} C^{-\mathsf{T}} C^{\mathsf{T}} (V^{\mathsf{T}} A V) C\\ &= \mathsf{tr}\ C^{-1} (V^{\mathsf{T}} BV)^{-1} C^{-\mathsf{T}} C^{\mathsf{T}} (V^{\mathsf{T}} A V) C\\ &= \mathsf{tr}\ (V^{\mathsf{T}} BV)^{-1} (V^{\mathsf{T}} A V) \\ &= J(V). \end{aligned}\notag \end{equation}

Therefore, the objective is invariant to a change of basis (just like ordinary Rayleigh Quotients are invariant to rescalings) so we might as well impose the convenient constraint that $V^{\mathsf{T}} \Sigma_w V = I_k$ since if $U$ is some minimizer of $J$ which does not satisfy this condition, we can rescale $U$ so that it does. Thus, we are left with the GEVP:

\begin{equation} \begin{aligned} \underset{V \in \mathbb{R}^{n \times k}}{\text{minimize}}&\ \mathsf{tr}(V^{\mathsf{T}} \Sigma_b V)\\ \text{subject to}&\ V^{\mathsf{T}} \Sigma_w V = I_k, \end{aligned}\notag \end{equation}

the solutions of which provide a matrix $V \in \R^{n \times k}$ which can be used to reduce the dimensionality of the data $X$ such that the different classes are well separated in the lower dimensional space.

Remark: Notice that optimizing the Rayleigh quotient provides another variational characterization of the generalized eigenvalues!

There is however an important caveat. Notice that $\Sigma_w = \sum_c \pi_c \mu_c \mu_c^{\mathsf{T}}$ is a sum of $C$ rank-1 matrices. This matrix therefore has rank at most $C$, and then $V^{\mathsf{T}} \Sigma_w V$ must also have rank at most $C$, though it is of dimension $k \times k$ (where $V$ is $n \times k$ and $k \le n$). Thus, we need to be careful that $k \le C$ (smaller for degenerate cases), otherwise the inverse is not well defined – this places a restriction on the dimensionality of the space onto which we project.

Example

One of the most natural datasets on which to apply this technique is the classic MNIST handwritten digits data , for which I’ve just used the data shipped with scikit-learn . This dataset consists of $1797$ examples of $8 \times 8$ images of written digits between $0$ and $9$. Pixel intensities are between $0$ and $16$.

Simply flattening out the $8 \times 8$ images into a vector gives us a data matrix $X \in \R^{1797 \times 64}$, which I’ll project into a lower dimensional subspace using either FLD or PCA – the labels $y \in \{0, \ldots, 9\}^{1797}$ are natural class labels for FLD.

Calculating the projections onto a $K \ll 64$ dimensional subspace (after centering and standardizing the $X$ matrix) results in two new datasets $\widehat{X}_{\text{pca}} \in \R^{1797 \times K}$ and $\widehat{X}_{\text{fld}} \in \R^{1797 \times K}$, the former being the projection onto the axes which maximizes the variance, and the latter being constructed to try to separate the clumps of classes.

In order to make a nice visual comparison between these two projections, I’d like to try to “line up” the projected data as well as possible so that if the data were to fall into similar clumps for PCA and FLD, that these clumps would appear in similar locations in a visualization. To do so, I’ve rotated the matrix $\widehat{X}_{\text{fld}}$ with an orthogonal matrix $Q$ obtained from solving an Orthogonal Procrustes Problem .

Orthogonal Procrustes Problem

This is a bonus problem which is similar in spirit to a trace optimization problem, but takes on a different form and has a different solution than the problems I’ve considered so far. Specifically, we solve

\begin{equation} \begin{aligned} \underset{Q \in \R^{n \times n}}{\text{minimize}} &\quad \frac{1}{2}||Q\widehat{X}_{\text{fld}} - \widehat{X}_{\text{pca}}||_F^2\\ \text{subject to} &\quad Q^\mathsf{T} Q = I, \end{aligned}\notag \end{equation}

so that $\widehat{X}_{\text{fld}}$ is rotated (and possibly reflected) to better line up with $\widehat{X}_{\text{pca}}$. We can make this look like a trace optimization problem by expanding the objective as in

\begin{equation} \begin{aligned} \frac{1}{2}||Q\widehat{X}_{\text{fld}} - \widehat{X}_{\text{pca}}||_F^2 &= \frac{1}{2}\mathsf{tr}(\widehat{X}_{\text{fld}}^{\mathsf{T}}Q^{\mathsf{T}}Q\widehat{X}_{\text{fld}} + \widehat{X}_{\text{pca}}^{\mathsf{T}}\widehat{X}_{\text{pca}}) - 2\mathsf{tr}(Q^{\mathsf{T}} \widehat{X}_{\text{pca}}\widehat{X}_{\text{fld}}^{\mathsf{T}})\\ &\overset{(a)}{=} \frac{1}{2}\mathsf{tr}(\widehat{X}_{\text{fld}}^{\mathsf{T}}\widehat{X}_{\text{fld}} + \widehat{X}_{\text{pca}}^{\mathsf{T}}\widehat{X}_{\text{pca}}) - \mathsf{tr}(Q^{\mathsf{T}} \widehat{X}_{\text{pca}}\widehat{X}_{\text{fld}}^{\mathsf{T}})\\ \end{aligned}\notag \end{equation}

where $(a)$ uses the orthogonality constraint. This leaves us with a sort of “one-sided” trace optimization problem, distinct from the problems encountered so far:

\begin{equation} \begin{aligned} \underset{Q \in \R^{n \times n}}{\text{maximize}} &\quad \mathsf{tr}(Q^{\mathsf{T}} \widehat{X}_{\text{pca}}\widehat{X}_{\text{fld}}^{\mathsf{T}})\\ \text{subject to} &\quad Q^\mathsf{T} Q = I. \end{aligned}\notag \end{equation}

To actual see the solution of this problem, expand out the related term

\begin{equation} \frac{1}{2}||Q - \widehat{X}_{\text{pca} }\widehat{X}_{\text{fld}}^{\mathsf{T}}||_F^2 = \frac{1}{2}\mathsf{tr}(Q^{\mathsf{T}}Q + \widehat{X}_{\text{fld}}\widehat{X}_{\text{pca}}^{\mathsf{T}}\widehat{X}_{\text{pca}} \widehat{X}_{\text{fld}}^{\mathsf{T}}) - \mathsf{tr}(Q^{\mathsf{T}} \widehat{X}_{\text{pca}}\widehat{X}_{\text{fld}}^{\mathsf{T}}),\notag \end{equation}

which similarly involves constants and the above trace objective. We can therefore equivalently solve \[\underset{Q: Q^{\mathsf{T}}Q = I}{\text{minimize}}\ \frac{1}{2}||Q - \widehat{X}_{\text{pca} }\widehat{X}_{\text{fld}}^{\mathsf{T}}||_F^2.\] Apply a Singular Value Decomposition $\widehat{X}_{\text{pca} }\widehat{X}_{\text{fld}}^{\mathsf{T}} = U\Sigma V^{\mathsf{T}}$ to the product, and using the fact that $U, V$ are orthogonal, and therefore we can freely multiply by $U, V$ inside the norm without change its value, leads to the solution $Q = UV^{\mathsf{T}}$, with cost $\sum_{i = 1}^m (1 - \sigma_i)^2$ for singular values $\sigma_i$ of $\widehat{X}_{\text{pca} }\widehat{X}_{\text{fld}}^{\mathsf{T}}$. Precisely,

\begin{equation} \begin{aligned} \underset{Q: Q^{\mathsf{T}}Q = I}{\text{min}}\ \frac{1}{2}||Q - \widehat{X}_{\text{pca}}\widehat{X}_{\text{fld}}^{\mathsf{T}}||_F^2 &= \underset{Q: Q^{\mathsf{T}}Q = I}{\text{min}}\ \frac{1}{2}||Q - U\Sigma V^{\mathsf{T}}||_F^2\\ &\overset{(a)}{=} \underset{D: D^{\mathsf{T}}D = I}{\text{min}}\ \frac{1}{2}||UDV^{\mathsf{T}} - U\Sigma V^{\mathsf{T}}||_F^2\\ &\overset{(b)}{=} \underset{D: D^{\mathsf{T}}D = I}{\text{min}}\ \frac{1}{2}||D - \Sigma||_F^2\\ &\overset{( c)}{=} \frac{1}{2}\sum_{i = 1}^m (1 - \sigma_i)^2, \end{aligned}\notag \end{equation}

where $(a)$ expresses $Q$ in the same basis ($D$ is not yet necessarily diagonal! e.g., consider $D = U^{\mathsf{T}} Q V$), and $(b)$ uses the fact that $U, V$ are orthogonal. Minimizing the approximation error of the diagonal matrix $\Sigma$ after equality $(b)$ now requires that $D$ be diagonal, and meeting the equality (which is derived from the requirement that $Q^{\mathsf{T}}Q = I$) necessitates (equality $( c)$) that $D = I.$ $\ \square$

I’ve used this matrix $Q$ to better align the data in the animation below. Notice that each coloured clump (corresponding to different digits) is roughly aligned in space – there is nothing intrinsic about PCA and FLD that would encourage this alignment.

In addition to the figure, I also fit a simple sklearn.svm.LinearSVC to the projected data in order to verify that the clusters are in fact well informed by the class labels. The MCC value is annotated in the title of the each figure.

Optimizing Signal to Noise Ratio

As a final example, let’s examine a problem in signal processing. Let $X \in \R^{n}$ be an unobserved zero mean random vector with variance $\Sigma$ and $E \in \R^{n}$ be another zero-mean and unobserved random vector with variance $\Omega$. The observed quantity is given by $Y = X + E$, i.e., a noisy measurement of the signal of interest.

Similarly to PCA, we will find a matrix $V \in \R^{n \times k}$ to project $Y$ onto some lower dimensional space, but this time we will optimize the signal-to-noise ratio (SNR), where the “signal” is $V^{\mathsf{T}} X$ (containing information about the vector $X$ that we care about) and the “noise” is $V^{\mathsf{T}} E$ (which is just corrupted by $E$). This leads to the optimization problem:

\begin{equation} \begin{aligned} \underset{V \in \R^{n \times k}}{\text{maximize}}&\ \frac{\mathbb{E}||V^{\mathsf{T}} X||_2^2}{\mathbb{E}||V^{\mathsf{T}} E||_2^2}\\ \text{subject to}&\ V^{\mathsf{T}} V = I. \end{aligned}\tag{$\mathcal{P}$} \end{equation}

This is a generalization of PCA since if $E$ is isotropic, $\mathbb{E}||V^{\mathsf{T}} E||_2^2$ is a constant given the constraint, and can be dropped. Moreover, this problem is different from the GEVPs encountered earlier, because we enforce the constraint that $V^{\mathsf{T}}V = I$, as opposed to $V^{\mathsf{T}}\Omega V = I$.

While this problem doesn’t immediately take the form of an eigenvalue problem, we can relate it to one with some clever manipulations (essentially drawn from Kokiopoulou et. al. ). Supposing that $s^\star > 0$ is the optimal SNR, it must be the case that \[\frac{\mathbb{E}||V^{\mathsf{T}} X||_2^2}{\mathbb{E}||V^{\mathsf{T}} E||_2^2} \le s^\star\] for every orthogonal matrix $V.$ Therefore, denoting by $\mathcal{O}_k$ the set of $n \times k$ matrices with orthogonal columns, we have

\begin{equation} \begin{aligned} \forall V \in \mathcal{O}_k:\ \mathbb{E}||V^{\mathsf{T}} X||_2^2 - s^\star \mathbb{E}||V^{\mathsf{T}} E||_2^2 &\le 0\\ \implies \forall V \in \mathcal{O}_k:\ \mathbb{E} \mathsf{tr}\ V^{\mathsf{T}} XX^{\mathsf{T}} V - s^\star \mathbb{E} \mathsf{tr}\ V^{\mathsf{T}} EE^{\mathsf{T}} V & \le 0\\ \implies \forall V \in \mathcal{O}_k:\ \mathsf{tr}\ V^{\mathsf{T}} \Sigma V - s^\star \mathsf{tr}\ V^{\mathsf{T}} \Omega V &\le 0\\ \implies \forall V \in \mathcal{O}_k:\ \mathsf{tr}\ V^{\mathsf{T}} \bigl(\Sigma - s^\star \Omega\bigr) V &\le 0\\ \implies \underset{V \in \mathcal{O}_k}{\text{max}}\ \mathsf{tr}\ V^{\mathsf{T}} \bigl(\Sigma - s^\star \Omega\bigr) V &\le 0.\\ \end{aligned} \end{equation}

From what we know of eigenvalue problems, the maximizer $V$ in $\mathcal{O}_k$ of $\mathsf{tr}\ V^{\mathsf{T}} \bigl(\Sigma - s \Omega \bigr) V$ is given by the last $k$ eigenvectors of the matrix $\Sigma - s \Omega$. Lets call these $V_s$. We want to maximize the SNR, and thus find the largest $s$ which satisfies the above inequalities – i.e., the optimal value $s^\star$ should be a root of the function \[\mathcal{S}(s) = \mathsf{tr}\ V^{\mathsf{T}}_s \bigl( \Sigma - s \Omega \bigr) V_s,\] a function which can be evaluated through an eigendecomposition of $\Sigma - s \Omega$. (Remark: expressions like $\Sigma - s\Omega$ involving a matrix minus a scalar times another matrix are often referred to as matrix pencils. I am told they are named after the concept in geometry .)

To check that this can actually occur, notice that since $\Omega \succ 0$, the function $\mathcal{S}$ must be monotone decreasing on $\R_+$. To establish the existence of an $s^\star$ such that $\mathcal{S}(s^\star) = 0$, and to apply bisection search, we need to show that $\mathcal{S}$ does actually cross $0$. To this end, $\mathcal{S}(0) > 0$ (since $\Sigma \succ 0$). Since $\mathcal{S}$ is strictly monotone, this is enough, but we’d like to find a compact interval where we know $s^\star$ lies. I show in the detail below that $\mathcal{S}(s) < 0$ for any $s > \lambda_n(\Sigma, \Omega)$, the largest generalized eigenvalue.

Generalized Eigenvalue Decomposition

Supposing that $\Lambda = \mathsf{Dg}(\lambda_1, \ldots, \lambda_n)$ is a diagonal matrix of generalized eigenvalues and $V \in \mathbb{R}^{n \times n}$ contains the generalized eigenvectors in the columns, we can write the GEVP as \[\Sigma V = \Omega V \Lambda.\] Then, since $V^{\mathsf{T}} \Omega V = I$ we have $VV^{\mathsf{T}} \Omega V = V$ which, since $V$ is invertible (we know it is a basis for $\R^n$) implies that $VV^{\mathsf{T}} \Omega = I$. Multiplying the GEVP on the right by $V^{\mathsf{T}} \Omega$ then provides to us a generalized eigenvalue decomposition of $\Sigma$ with respect to $\Omega$: \[\Sigma = \Omega V \Lambda V^{\mathsf{T}} \Omega.\]

We can use these facts to analyze the function $\mathcal{S}$ as in

\begin{equation} \begin{aligned} \mathcal{S}(s) &= \mathsf{tr}\ (\Sigma - s\Omega)\\ &= \mathsf{tr}\ (\Omega V \Lambda V^{\mathsf{T}} \Omega - s\Omega)\\ &= \mathsf{tr}\ \Omega (V \Lambda V^{\mathsf{T}} \Omega - sI)\\ &= \mathsf{tr}\ \Omega (V \Lambda V^{\mathsf{T}} \Omega - sVV^{\mathsf{T}} \Omega)\\ &= \mathsf{tr}\ \Omega V (\Lambda - sI) V^{\mathsf{T}} \Omega \\ &= \mathsf{tr}\ (\Lambda - sI)V^{\mathsf{T}} \Omega^2 V, \\ \end{aligned}\notag \end{equation}

and since $V^{\mathsf{T}} \Omega^2 V \succ 0$ we must have $\mathcal{S}(s) < 0$ for any $s > \lambda_n$. $\ \square$

Now, it follows that $s^\star$ is the unique root of the monotone function $\mathcal{S}$ and it lies in the interval $[0, \lambda_n(\Sigma, \Omega)]$. Thus, we can find $s^\star$ by performing bisection on $\mathcal{S}$, and then recover the optimal matrix $V_{s^\star}$ by computing the last $k$ eigenvectors of $\Sigma - s^\star \Omega$. The value $s^\star$, remember, is also the value of the optimal SNR.

For the Reader: I’ve glossed over another property that should be verified for $\mathcal{S}$ before making these conclusions. What is it and why does it hold?

An example of this technique is given for the same faces dataset as the PCA example, where I’ve corrupted the images with noise that is locally correlated in space. Clearly the result is not particularly good, in an absolute sense, but the PCA reconstruction is completely corrupted by the anisotropic noise distribution.

Some code to implement this technique in Julia is below.

using LinearAlgebra;

function snr_pca(Σ, Ω, V)
    "The objective function."
    return tr(V' * Σ * V) / tr(V' * Ω * V)
end

function optimal_snr_pca(Σ, Ω, k, eps=1e-6)
    """Computes the optimal projection matrix."""
    n = size(Σ)[1]
    S = s -> (sum ∘ eigvals)(Σ - s * Ω, n - k + 1 : n)
    V = s -> eigvecs(Σ - s * Ω)[:, n - k + 1 : n]
    s_max = eigvals(Σ, Ω)[n]

    s_star = bisection(S, s_max, eps)
    V_star = V(s_star)
    return V_star, s_star
end

function bisection(S, s_max, eps=1e-6)
    """Finds the zero of a monotone decreasing function in [0, s_max]."""
    s_left = 0.0
    s_right = s_max
    while abs(s_right - s_left) > eps
        s = s_left + (s_right - s_left) / 2
        if S(s) > 0
            s_left = s
        else
            s_right = s
        end
    end
    return s_left
end

Conclusion

This blog examines the theory and applications of eigenvalue problems and generalized eigenvalue problems. We’ve seen through the Spectral Theorem that the symmetry of the matrix is essential for obtaining real-valued eigenvalue decompositions (and thus an ordering to the eigenvalues). These eigenvalues can be characterized by variational problems, and useful applied problems (particularly PCA and its weighted version, and FLD) can be formulated in the same optimization problems.

The key implication of this is that these optimization problems admit of efficient solutions: eigenvalue decompositions. These optimization problems are by no means convex, yet they still admit of efficient solutions through the use of scipy.linalg.eigh.

We’ve also seen some “bonus” relationships with the singular value decomposition in the form of the Orthogonal Procrustes Problem, and the fact that you can compute the eigenvalues needed for PCA directly from $X$, rather than having to form $\widehat{\Sigma} = \frac{1}{N}X^{\mathsf{T}}X$. As well, the detailed computations for signal to noise ratio optimization show us the generalized eigenvalue decomposition $\Sigma = \Omega V \Lambda V^{\mathsf{T}}\Omega$ of $\Sigma$ with respect to $\Omega$. This is another useful decomposition which generalizes the “vanilla” eigenvalue decomposition $\Sigma = U \Lambda U^{\mathsf{T}}$.

These problems present plenty of further interesting avenues for exploration, which I invite the reader to meditate upon :)

Nature's Dartboard: The Axioms of Probability

Sun, 23 Apr 2023 00:00:00 -0700

The basic axioms of probability and the idea of a random variable as a function from the sample space to $\mathbb{R}$ are quite abstract and rather confusing at first pass. However, the axioms can be well motivated from our intuition, and defining random variables simply as functions turns out to be a brilliant and intuitive abstraction. My goal in this post is to try to explain the ideas behind the axiomatization of probability theory, and hopefully make the study of measure-theoretic probability seem a bit less intimidating.

Mathematics and Axiomatization

Exactly what is Mathematics is not necessarily an easy thing to pin down, and to try to give some sort of clear formal definition would be senseless. However, it is easy to say what it is not: it is not simply a prescribed set of rules for calculating numbers, or “solving for $x$” – a prescribed set of calculating rules has another name: an algorithm.

In contrast to the mechanical process described by an algorithm, mathematics is a highly creative endeavour. Indeed, many mathematicians are motivated largely by the aesthetics and the “beauty” of their results. To many mathematicians, it is the abstraction and the concepts themselves that are of interest, and they may not even be entirely interested in whether or not the results and calculations are $100\%$ correct! A great book about the process of doing math (which can be got for rather low prices) is The Mathematical Experience 💁‍♂️

Still, many mathematicians are at the same time concerned with correctness and rigor. That is, they want to be able to say with supreme confidence that their results are true and correct, with no room for doubt. Essential to this is the idea of a mathematical proof – a sequence of logical deductions which begin from some specified premises $P$ and lead inexorably to some conclusion $Q$. If the logical deductions are correct, then the mathematician has rigorously obtained a proof of the implication $P \implies Q$. That is, given that the premises $P$ are true, it must be the case that the conclusions $Q$ are true. Mathematicians call this $P \implies Q$ implication a Theorem.

Once a mathematician (or a group of them, perhaps spanning continents and generations) has amassed a rather sizable number of theorems, and understands something about the various connections amongst them, the mathematician might attempt to lay down a fundamental set of premises from which all of their conclusions can be derived. This set of fundamental premises, which cannot be derived (as far as the mathematician can tell) from any other still more fundamental premises, are called axioms. The axioms are the basic starting point for a theory, that is, for all the further conclusions the mathematician has derived. Axioms are ultimately just premises, but the former term tends to imply that it is a foundational premise – i.e., something that is self-evident that can serve as a starting point.

The most famous set of axioms are Euclid’s Axioms laying the classical foundation for geometry. Another being those of ZF Set Theory , which lay the foundations for set theory, and therefore of more-or-less all of modern “mainstream” mathematics. The axioms I care about discussing in this post are the axioms of Probability Theory .

As opposed to the rigorous series of logical deductions by which proofs proceed, the way that the idea of a Theorem comes about (i.e., the thought that some implication might possibly be true) is much more creative and messy. Often, one begins with a conclusion $Q$ that “seems true”, and then proceeds to search for the set of premises $P$ that make it true, possibly going back and forth many times tweaking the premises and the conclusion – a process reminiscent of Rawl’s reflective equilibrium in philosphy. The mathematician is done with this inquiry when they have obtained a Theorem which they think is “good”. I have seen it claimed that a Theorem is good if it is surprising, deep, and can be used to derive other good theorems. My memory tells me that this should be attributed to Polya, but I cannot find a reference (please email me if you know where this is from!).

As well, we usually come up with axioms long after having figured out most of the conclusions (possibly using stronger or more numerous premises). Axiomatization is a process of cleaning up the theory and placing it upon a nice, beautiful, and simple foundation. Thus, I think it can be rather confusing, from a didactic standpoint, to write textbooks or teach classes which simply begin with the axioms and proceed to derive all of the conclusions – the student is left confused and wondering what the axioms mean, where did they come from, etc. To be comfortable in following this process straight from the axioms requires some amount of experience and mathematical maturity.

Indeed, the process of derivation from a set of axioms is not reflective of how the theory was developed, and it is hardly even reflective of how mathematicians think about the theory themselves. My hope is that the rest of this post can clarify some of the intuition behind the axioms of probability.

The Axioms of Probability

The purpose of probability theory is to provide a set of mathematical tools for reasoning about events which are random. Exactly what random means is on one hand left to intuition (e.g., we think of a dice roll as random, because we have no practical means of determining what the outcome will be), and on the other hand, is rigorously axiomatized by the theory of probability. My purpose here is to help to make the axiomatization itself more intuitive.

The rigorous theory of probability was axiomatized in 1933 by Russian mathematician Andrey Kolmogorov (1903-1987). This axiomatization begins with a set $\Omega$ called the sample space. What this set is exactly need not be fully specified – it is just some abstract set, and it may be a finite set like $\{\mathsf{H}, \mathsf{T}\}$ (to model flipping a coin), it may be a countably infinite set like $\{1, 2, \ldots\}$, it could be an uncountably infinite set like $\mathbb{R}$, or some even more abstract set.

To set the stage, the following sections will introduce the probability triple $(\Omega, \mathcal{F}, \mathbb{P})$ consisting of a sample space $\Omega$, a $σ$-algebra $\mathcal{F}$ thereof, and the probability measure $\mathbb{P}$. On this triple we axiomatize probability with the following three premises:

The domain of $\mathbb{P}$ is $\mathcal{F}$ and for any $S \in \mathcal{F}$ we have $P(S) \ge 0$ (non-negativity)
$\mathbb{P}(\Omega) = 1$ (Normalization)
If $S_1, S_2, \ldots \in \mathcal{F}$ are disjoint, then $\mathbb{P}\bigl(\bigcup_{i = 1}^\infty S_i \bigr) = \sum_{i = 1}^\infty \mathbb{P}(S_i)$ (Countable Additivity)

By the end of this section, I hope the reader will have some understanding of what all of this means, and why these axioms are so beautiful. It is remarkable that probability theory flows from these three simple statements!

Samples and Events

In probability theory, an element $\omega \in \Omega$ of the sample space is called a sample. We don’t know what this sample is, but it encodes the state of the universe. Subsets $S \subseteq \Omega$ are called events (actually, they must be elements $S \in \mathcal{F}$, which is a set of subsets of $\Omega$ – more on this later), and we imagine that an event has occurred if $\omega \in S$.

Intuitively, there are many states of the universe that might lead to the same event; e.g., the outcome of a coin toss ultimately depends upon, at least, the arrangement of molecules in the air through which the coin travels, and there are many more than two such arrangements. Thus, there are many $\omega$ (arrangements of molecules) which result in the “Heads” outcome of the coin toss.

When we do modelling in practice, we usually don’t work directly with $\Omega$. Instead, we work with random variables, which are functions from the sample space into $\mathbb{R}$. The sample space is kept as an abstract entity “in the background”. But, before we discuss this additional layer of abstraction, we need to understand $(\Omega, \mathcal{F}, \mathbb{P})$.

Nature’s Dartboard

On an intuitive level, I think of the set $\Omega$ as being Nature’s Dartboard – it is just a disc, and “Mother Nature” is throwing a dart at it. The exact location at which the dart lands selects for us a state of the universe $\omega \in \Omega$.

In addition to the set $\Omega$, we are given a function $\mathbb{P}$ called a probability measure. The function $\mathbb{P}$ is a bit more abstract than the sort of function (a curve) that one might usually encounter in calculus. Instead, $\mathbb{P}$ is a function which maps from subsets of $\Omega$ into some number between $[0, 1]$. That is, if we have some set $S \subset \Omega$, we would like $\mathbb{P}$ to assign some number to $S$ as in $\mathbb{P}(S)$. This number is called the probability of the event (subset), and is directly analogous to the volume of $S$.

Indeed, if we imagine that Mother Nature throws her dart at the dartboard $\Omega$ completely “at random” (following our intuition) then our intuitive idea of the probability of the dart landing within some region $S \subseteq \Omega$ of the dartboard is nothing but $\text{vol}(S) / \text{vol}(\Omega)$, where $\text{vol}(S)$ measures the volume (in this case, just an area) of the set $S$.

Now, if we imagine that $\mathbb{P}(S) = \text{vol}(S) / \text{vol}(\Omega)$, then the axiom $\mathbb{P}(S) \ge 0$ is natural since volumes cannot be negative, and $\mathbb{P}(\Omega) = 1$ is elementary algebra. This is the basic intuition of the axioms of probability – we measure the probability of events $S$, which are nothing but subsets of the sample space $\Omega$, by appealing to the volume of the dartboard that they occupy.

Continuing this line of reasoning, imagine we have some set $S \subseteq \Omega$ which has some probability $\mathbb{P}(S)$. If we break this set into two pieces (i.e., two disjoint sets $S_1, S_2$ with $S_1 \cup S_2 = S$ and $S_1 \cap S_2 = \emptyset$) then it stands to reason that we should have \[\mathbb{P}(S) = \mathbb{P}(S_1) + \mathbb{P}(S_2),\] since the volume of one set must be the sum of the volumes of any partition of that set. This is the idea behind Axiom 3, additivity.

That Axiom 3 is stated for countably infinite partitions of events (i.e., infinite sums) is a technical condition: countable additivity. Simple finite additivity turns out to not be strong enough to result in a rich enough theory. Making the axioms works rigorously for these infinite sums, and for infinite sample spaces, is the whole point of the formalism surrounding the $σ$-algebra $\mathcal{F}$. We turn to this next.

The Banach-Tarski Paradox: Volume and Measure

The object $\mathcal{F}$ is called a $σ$-algebra of subsets of $\Omega$, and it is in general a subset of the set of all subsets of $\Omega$. Understanding this $σ$-algebra is part of the difficult technical work that goes into learning rigorous probability theory, but the purpose of $\mathcal{F}$ is to eliminate certain pathological sets in $2^\Omega$ which are not measurable (see: Measure Theory ). That is, there are some sets (e.g., Vitali Sets ) for which it simply makes no sense to assign a notion of volume, and the issue arises directly as a result of dealing with infinite sets.

A famous example illustrating this issue is the Banach-Tarski paradox. This paradox shows that we can take a sphere, partition it into six pieces, and then, after rotating some of those pieces in a clever way, reassemble two copies of the original sphere. Here is a Video explanation of the procedure as well as a Quanta Magazine article . The fact that we can take a set, partition that set, and then reassemble two identical copies of that set is rather problematic when it comes to assigning volumes! It seems that we can derive $\mathbb{P}(S) = 2\mathbb{P}(S)$ where $S$ is a sphere! This is a pathology that needs to be eliminated.

Indeed, we would really like for it to be the case that $\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B)$ if $A \cap B = \emptyset$ and where $\mathbb{P}(S) = \text{vol}(S) / \text{vol}(\Omega)$ is consistent with our notion of volume. But, this can only be done if we simply exclude a-priori so-called non-measurable sets. This is more-or-less what the $σ$-algebra $\mathcal{F}$ is doing – it is a special collection of subsets of $\Omega$ which ensures that any $S \in \mathcal{F}$ is a measurable set to which we can assign a notion of volume that corresponds with our intuition. In the language of probability, it means that only certain subsets of $\Omega$ can be assigned a probability: the measurable subsets. There are some pathological subsets of Nature’s dartboard which we simply cannot measure.

Specifically, a collection of subsets $\mathcal{F}$ of $\Omega$ is a $σ$-algebra of subsets if the following conditions are satisfied:

$\Omega \in \mathcal{F}$ (The universe is one of the sets)
$A, B \in \mathcal{F} \implies A \setminus B \in \mathcal{F}$ (Closed under set differences)
If $A_1, A_2, \ldots \in \mathcal{F}$ and $A_i \cap A_j = \emptyset$ if $i \ne j$, then $\bigcup_{i = 1}^\infty A_i \in \mathcal{F}$ (Closed under countable disjoint union)

Notice that these are basically nothing but the conditions that make the axioms of probability “work”, particularly closure under countable unions. That is, the notion of a $σ$-algebra is cooked up exactly for the purpose of making sure that measure corresponds to our intuition of volume.

To actually construct a volume measure for the dartboard, one begins by constructing a measure on the real line $\mathbb{R}$ by defining $\mathbb{P}([a, b]) = b - a$, and then generating a $σ$-algebra out of intervals (by taking complements and countable unions etc.). This guarantees that the resulting $\sigma$ algebra is generated by “reasonable” operations on the sets, excluding those which are not measurable. Then, by putting together the rules of the $σ$-algebra above, and using the basic intuition of $\mathbb{P}([a, b]) = b - a$, we can eventually construct what is called Lebesgue Measure , a rigorous way of assigning volume to (measurable) subsets of $\mathbb{R}$. This is enough to get us to a suitable notion of volume, and therefore to a construction of a probability measure $\mathbb{P}$!

Remark (Axiom of Choice): The problem of measure is not entirely inescapable. If we did not accept the Axiom of Choice , which basically just says that it is possible to choose an arbitrary element from an infinite set, then non-measurable sets could not be constructed, and the problem would disappear. Some mathematicians thus view the existence of non-measurable sets, which are incredibly pathological, as a reason to refuse to accept the axiom of choice. However, we need to pick our poison; there are at the same time a vast number of perfectly intuitive and incredibly useful theorems which depend on the axiom of choice. My favourite example being the Hanh-Banach Theorem , the cornerstone of convex analysis (named after the same Banach as the Banach-Tarski Paradox). Instead of doing without more-or-less all of functional analysis, most instead choose to accept non-measurable sets as a fact of (mathematical) life, and carry around all the baggage associated with $\sigma$-algebras and measure theory.

Random Variables

All of the formalism laid out in understanding the probability triple $(\Omega, \mathcal{F}, \mathbb{P})$ is rather complicated. Fortunately, in most “practical” scenarios where we want to use probability for modelling, we just let this formalism sit back as an abstract foundation, more-or-less out of sight. We instead work more directly with random variables which are defined as nothing but functions from $\Omega$ into $\mathbb{R}$ as in: \[X: \Omega \rightarrow \mathbb{R},\] sometimes written as $X(\omega)$.

The “randomness” thus passes from Nature’s Dartboard $\Omega$, an abstract set that we don’t want to think about too much, to $\mathbb{R}$, a natural set that we like to work with. The way this works is as follows – suppose we want to compute the probability that $X(\omega)$ is less than some real number $x$, that is $F(x) = \mathbb{P}(X \le x)$. This is an important function called the cumulative distribution function of $X$ and it turns out to be enough to fully determine the random behaviour of $X$. The notation $\mathbb{P}(X \le x)$ is really nothing but a shorthand for \[F(x) = \mathbb{P}(\{\omega: X(\omega) \le x\}).\]

That is, the common notation $\mathbb{P}(X \le x)$, which is used to mean “the probability that the random variable $X$ is less than or equal to the real number $x$” is really a shorthand for $\mathbb{P}(\{\omega:\ X(\omega) \le x\})$ which is “the volume of the set of all events $\omega$ such that the function $X$ evaluated $X(\omega)$ is less than or equal to the real number $x$”. We can simplify this further:

\begin{equation} \begin{aligned} F(x) &= \mathbb{P}(\{\omega: X(\omega) \le x\})\\ &= \mathbb{P}(\{\omega: X(\omega) \in (-\infty, x]\})\\ &= \mathbb{P}(X^{-1} ((-\infty, x]))\\ &= \mathbb{P} \circ X^{-1} ((-\infty, x]). \end{aligned}\notag \end{equation}

These elementary manipulations show us that that the cumulative distribution function $F$ is nothing but the composition of the probability measure $\mathbb{P}$ with the inverse mapping of $X$ (the domain of which are all the half-open intervals like $(-\infty, x]$)! What does this mean? It allows us to define events on $\mathbb{R}$, like the probability of the event $X \le x$ by measuring the size of the set of all $\omega$ which satisfy $X(\omega) \le x$. So, what we do “in practice” is that we start by positing a distribution function $F$ (like a Gaussian distribution) which we want to work with on $\mathbb{R}$, and then we just trust that there is some appropriate probability triple $(\Omega, \mathcal{F}, \mathbb{P})$ which is “adequately expressive” and some function $X$ such that $F = \mathbb{P} \circ X^{-1}$. The function $X$ is now a random variable having c.d.f. $F$. This gives us a huge variety of possibilities for modelling, and the randomness on $\mathbb{R}$ is inherited from the simple and intuitive notion of randomness as volume on Nature’s dartboard.

Remark (Measurable Functions): Technically, the random variable $X$ must be a measurable function . This is another important technical condition, but irrelevant for developing an intuition.

Remark (Functions of Random Variables): Suppose we have some function $g: \mathbb{R} \rightarrow \mathbb{R}$. It is worthwhile to point out that a function of a random variable $g(X)$ defines a new random variable $Y(\omega) = g \circ X(\omega)$, since it is nothing but a another function from $\Omega$ to $\mathbb{R}$. The cumulative distribution function of this new random variable is given by $F_Y(y) = \mathbb{P} \circ X^{-1} \circ g^{-1}((-\infty, y])$.

Expectation

Another essential concept in probability theory is that of the expectation. Intuitively, this is like a generalization of an average. If the sample space $\Omega$ is finite, then we define \[\mathbb{E}X = \sum_{i = 1}^N X(\omega) \mathbb{P}(\omega),\] and if $\mathbb{P}(\omega) = \frac{1}{N}$ then it is exactly an average of the random variable over the sample space. Generally, finite sample spaces aren’t rich enough to be interesting, so we define the expectation by means of an integral \[\mathbb{E}X = \int_\Omega X(\omega) \mathsf{d}\mathbb{P}(\omega).\] This is called a Lebesgue Integral , and our desire to define and use this integral is another part of the motivation for studying probability in the context of measure theory. Regardless, it is usually possible to dispense with this integral “in practice” and to work directly with the distribution function $F$. This works as follows:

\begin{equation} \begin{aligned} \mathbb{E}X &= \int_\Omega X(\omega) \mathsf{d}\mathbb{P}(\omega)\\ &\overset{(a)}{=} \int_{X(\Omega)} x\ \mathsf{d} \mathbb{P} \circ X^{-1} (x)\\ &\overset{(b)}{=} \int_{\mathbb{R}} x\ \mathsf{d} F(x)\\ &\overset{( c)}{=} \int_{-\infty}^\infty xf(x)\ \mathsf{d}x, \end{aligned}\notag \end{equation}

where $(a)$ comes from making the substitution $x = X(\omega)$, $(b)$ just simplifies using $F = \mathbb{P} \circ X^{-1}$, and $( c)$ is a final simplification for continuous random variables many readers may be used to: if $F$ is a differentiable function, then the random variable admits a density function $f(x) = F^\prime(x)$ and $\mathsf{d}F(x)$ turns into $f(x)\mathsf{d}x$, resulting in an “ordinary” Riemann integral (as opposed to $\int_\mathbb{R} x \mathsf{d} F(x)$ which is a Lebesgue-Stieltjes integral ).

Remark (Expectations of Functions of Random Variables): Given a function $g: \mathbb{R} \rightarrow \mathbb{R}$, we may like to compute the expectation of the new random variable $g(X)$ as in $\mathbb{E}g(X) = \int_\Omega g(X(\omega)) \mathsf{d} \mathbb{P}(\omega).$ We could do so by determining the distribution function $G(x) = \mathbb{P}(g(X) \le x)$ of the new random variable $g(X)$ and then doing the calculation $\int_\mathbb{R} x \mathbb{d} G(x)$. But alternatively, we can see by repeating the same substitution $x = X(\omega)$ in the integral that we made earlier that $\mathbb{E}g(X) = \int_\mathbb{R} g(x) \mathsf{d} F(x)$ which is still just an integral over the distribution of $X$! This is an incredibly convenient fact called the Law of the Unconcious Statistician .

Data and Sampling – the Passage to Statistics

Many people are now drawn to the study of probability for the purposes of doing statistics, i.e., analyzing data. So, I wanted to add some more commentary about random variables and their connection to the modelling of data. That is, how do statistics and probability connect?

Statistics is often concerned with experiments. Suppose we plan an experiment where we are going to measure someone’s height. The outcome of this experiment is not known a-priori, so it makes sense to model it using probability theory – the outcome is random. As well, a person’s height is a real number, and we now know that we can model random real numbers using random variables. Thus, we might like to construct a model of the measurement of someone’s height as a random variable $H(\omega)$.

If we measure the heights of many people, we will wind up with a multitude of random variables $H_1, H_2, \ldots, H_N$, where $N$ is the number of measurements we make. These random variables all together define some joint distribution $F(h_1, \ldots, h_N) = \mathbb{P}(H_1 \le h_1, \ldots, H_N \le h_N)$, and statistics is often concerned with what can be learned about the distribution of these random variables through the actually observed outcomes of experiments.

Independent and Identically Distributed Random Variables

The multivariate distribution described above can be difficult to work with – some assumptions need to be made. The most common assumption is that the random variables are independent and identically distributed, that is, i.i.d. What this means is that we assume:

$\mathbb{P}(H_i \le h) = \mathbb{P}(H_j \le h)$ (identically distributed)
$\mathbb{P}(H_1 \le h_1, \ldots, H_N \le h_N) = \mathbb{P}(H_1 \le h_1) \times \cdots \times \mathbb{P}(H_N \le h_N)$ (independent).

The idea of the random variables having an identical distribution is that we are observing separate outcomes of the same experiment. Independence, where the joint distribution is nothing but the product of the individual (marginal) distributions, encodes the idea that the outcome of one experiment does not influence (and is not influenced by) any other experiment.

Thus, one of the most common settings in statistics is that we are given a sequence $X_1, X_2, \ldots, X_N$ of i.i.d. random variables, and our goal is to learn something about the statistics of their common distribution, call it $F_X$. The word statistic here just means “functionals of the distribution”, i.e., functions which take the distribution function $F_X$ as an input, and output some real number. The mean value $\mathbb{E}X = \int_{\mathbb{R}} x \mathsf{d} F_X(x)$ being an important example, quantiles being another: $\mathbb{P}(X \le q) = \int_{\mathbb{R}} \mathbf{1}[x \le q]\mathsf{d} F_X(x)$.

Given data, and statistical theorems about how collections of i.i.d. random variables behave, statisticians are able to answer questions like: “what is the probability that a randomly selected person will be over six feet tall?”, “in a room of $N$ people, what is the probability that there will be someone more than six feet five inches?”, etc. Going further, the combination of powerful computers with sophisticated statistical models can be used to produce striking results – the dart throwing woman that serves as the cover photo of this blog post was itself generated from a statistical model. That is, the image itself is essentially nothing but a particular realization of a random variable!

Remark (The LLN): The most natural estimator of $\mathbb{E}X$ given the sample is the average $\overline{X}_N = \frac{1}{N}\sum_{i = 1}^N X_i$ and indeed, many of the most famous theorems in statistics (the Law of Large Numbers and the Central Limit Theorem ) are concerned exactly with how $\overline{X}_N$ behaves under various different assumptions about the common distribution. Indeed, it can be shown that in almost all sensible cases $\overline{X}_N \rightarrow \mathbb{E}X \text{ as } N \rightarrow \infty$ (given an appropriate sense of convergence for random variables): the average of the sample converges to the expectation as the size of the sample increases. This is a highly intuitive outcome that can be derived entirely from the axioms of probability, and it is an incredibly satisfying result! Some historical formulations of probability began with the postulate that this is true, so it is a testament to how effectively the axioms “clean up” the theory.

Conclusion

Axiomatic and measure theoretic probability is a topic more than worth studying. It is very difficult, if not impossible, to have a full understanding of probability (particularly stochastic processes) without eventually going down this path. I hope that I have here shown that the elementary ideas are not just esoteric abstractions, but that they are well supported by a very natural intuition. It is the technical need to deal with annoying pathologies (like the Banach-Tarski paradox) that necessitates some technical machinery. But, the underlying intuition comes down to nothing but the measure of volumes in Nature’s Dartboard. Indeed, most abstractions in mathematics have some completely natural and intelligible intuition behind them!

Remark (Some book recommendations): The first book about probability I read was An Intermediate Course in Probability by Allan Gut. However, this book is not measure theoretic. Instead, this book by the same author can serve as a reasonable introduction, or the well known book by Billingsley which is more-or-less the canonical book to recommend. Another great book which is somewhat less concerned with the details of measure theory, focusing more on probability, is Probability: Theory and Examples which contains all of the classic results and, as the name suggestions, a lot of very nice examples. However, I only read this book long after I became well acquainted with measure-theoretic probability. Since its treatment of measure is rather more limited, it’s difficult for me to know whether or not this is a good book to start with.

Remark (Other topics): There are a number of other rather famous books broadly in the area of machine learning and statistics – examples being the following: Theoretical Statistics , Pattern Recognition and Machine Learning , Bayesian Data Analysis , Deep Learning , and Mathematics for Machine Learning (this last one I have not myself read). These books are perfectly good, but they are emphatically not about probability theory – they are applied books in machine learning and statistics. To learn probability theory itself, stick with the above.

Group Theory, Symmetry, and a Brain Teaser

Sun, 19 Feb 2023 00:00:00 -0800

I review some elementary discrete math concepts, use them to describe and solve a neat brain-teaser, and generalize the solution into a form of robust breadth first search.

Introduction

Recently, a friend introduced me to the following brain teaser:

Problem Statement: Suppose you are kidnapped, blindfolded, and placed into a room. Your kidnappers hand you a plate with four coins on it, arranged in a $2 \times 2$ square. They will let you leave if you can flip all of the coins so that they are face up, in a finite amount of time (they ain’t got all day!). Since you can’t see, you can ask the interrogator if you’ve successfully placed all the coins heads up (if you’d like to increase the immersion, you can imagine you are actually blind, otherwise you could just take off the blindfold). The interrogator won’t lie to you, but will try to mess with you: any time the answer is “no”, they will rotate the plate before giving it back to you. The amount by which it is rotated is arbitrary – it could be random, follow a fixed pattern, or be adversarial.

The first things coming to mind with this puzzle is that you could just iterate through every possible sequence of flips, asking the interrogator whether they’re all heads up after each flip. This possibility is immediately shut down by their adversarial rotations – the interrogator can easily foil this plan.

Another possibility is through random flips. Randomization is an effective tool because the interrogator cannot determine your next move by finding a pattern in your behaviour (if you randomize effectively, even you don’t know your next move!). If you simply pick up all the coins at random, flip them randomly, and then ask if you’ve solved the puzzle, it is guaranteed (by the Borell-Cantelli Lemma ) that you will eventually stumble upon the solution. However, this is where the caveat about finite time comes in. In probability theory, it is sometimes a point of confusion what is actually meant by a random variable being finite, so let’s spell it out. If $N(\omega)$ is the number of flips taken, given a state of the universe $\omega$, the Borell-Cantelli Lemma tells us that $N(\omega) < \infty$ almost surely, so $N$ is bounded, but it is not uniformly bounded (over all possible events $\omega$). What we want is that there is some real number $N_{\text{max}} < \infty$ such that $N(\omega) \le N_{\text{max}}$ almost-surely, and this cannot be done by the random flipping strategy.

That all being said, the solution I arrived upon does not include any randomization. Before explaining the solution, I’m going to take a detour through the ideas that came to my mind, and some of the formalisms that can be used to understand the puzzle, its solution, and potential generalization.

A Mathematical Preamble

The purpose of this section is to review some mathematical formalism and terminology used in reasoning about the coins-on-a-plate puzzle. I have a limited background in discrete math or abstract algebra, so even though these concepts are rather elementary, it has been good for me to write things down. Indeed, reviewing elementary concepts facilitates Chunking (an essential part of learning ). If all one is interested in is the solution to the puzzle, or if they are already well familiar with these concepts, they can skip to Solving the Puzzle .

Equivalence Classes

In mathematics, an equivalence relation $\sim$ on a set $X$ is a binary relation satisfying the following properties:

Reflexivity: $\forall x \in X\ x \sim x$
Symmetry: $\forall x, y \in X\ x \sim y \iff y \sim x$
Transitivity: $∀ x, y, z ∈ X\ x ∼ y, y ∼ z \implies x ∼ z.$

The most famous equivalence relation is “ordinary” equality: ‘$=$’, but there are various more exotic equivalence relations: equivalence up to rotations in geometry, equivalence almost surely in probability, equivalence except on measure-zero sets in the theory of $L_p$ spaces, etc.

When dealing with a finite number of objects (finiteness is not necessary in some/many of the cases I talk about, but I keep the assumption to avoid accidentally writing something that is technically wrong for infinite sets), equivalence classes easily arise from partitions of the set. That is, if $X$ is a finite set, and $X = \bigcup_{i = 1}^M X_i$ for subsets $X_i \subseteq X$ which are all disjoint $i \ne j \implies X_i \cap X_j = \emptyset$ (i.e., the $X_i$ are a partition $X$) we can define an equivalence relation on $X$ by \[x \sim y \iff (\exists i: x \in X_i \wedge y \in X_i),\] where $\wedge$ indicates “and”. Let us quickly check that this is a bona-fide equivalence relation: Let $x \in X$. There is an $X_i$ in the partition such that $x \in X_i$ and hence “also” $x \in X_i$ verifying it is reflexive. That it is symmetric arises since $(x \in X_i \wedge y \in X_i) \iff (y \in X_i \wedge x \in X_i)$. Finally, it is transitive since the $X_i$ are a partition without any overlap: if $x, y$ are in the same set $X_i$ and $y, z$ are both in another set $X_j$, it must be that $X_i = X_j$ (and hence $x, z \in X_i$) since $y$ must be in exactly one of the $X_i$. Conversely, the reader can check that any set partition also induces an equivalence relation. Thus, set partitions and equivalence relations are equivalent.

Given an equivalence relation $\sim$ on a set $X$, the corresponding partition (i.e., the partition induced by $\sim$) is commonly denoted as $X / \sim$ as the quotient set. We then say that elements of a set in the partition (that is, elements $x \in X_i$ for some $X_i \in X / \sim$, which is a subset $X_i \subseteq X$) are equivalent up to $\sim$.

Functions, Permutations, and Groups

Any finite set with $N$ elements can be represented simply as the set of integers from $1$ to $N$ as in $X = \{1, 2, \ldots, N\}$ (a set which is commonly denoted $[N]$). It does not matter what the elements of the set actually are, you can simply refer to the first, second, etc., in an arbitrary order.

Now, what are the functions on finite sets? Well, a function $f: X \rightarrow X$ is just a mapping from some element of $x \in X$ (without loss, an integer between $1$ and $N$) to another element $f(x) \in X$. If the function is a one-to-one function (i.e., it is injective) whose domain is all of $X$, then it must also (why?) be an onto function (i.e., a surjection) and thus it is a bijection and in this context such functions are called permutations . These are nothing but re-orderings of the integers, usually denoted by $\sigma$ (or other Greek letter, instead of the more generic $f$) as in “$\sigma$ re-orders the sequence $(1, 2, \ldots, N)$ into $(\sigma(1), \sigma(2), \ldots, \sigma(N))$”.

Since permutations are bijections, they have an inverse. Moreover, the composition operation amongst permutations is associative (why?): for permutations $\sigma, \pi, \tau$ on $X$, it is the case that $(\sigma \circ \pi) \circ \tau = \sigma \circ (\pi \circ \tau)$, where, e.g., $(\sigma \circ \pi)(x) = \sigma(\pi(x))$. This means that (with the trivial “doing nothing” permutation $Id(x) = x$ serving as the identity) the set of all permutations of $X$, call it \[G_X = \{\sigma: X \rightarrow X\ | \ \sigma \text{ is a bijection}\},\] along with the composition operation constitutes a (finite) Group $(G_X, \circ)$. This particular group $G_X$ of all bijections is particularly special, and is referred to as the Symmetric Group $S_N$ on $N$ elements – it is worthwhile noting that the size of this group grows rather quickly: there are $N!$ bijections in $S_N$.

Cycles and Decomposition

Given a permutation $\sigma \in S_N$, an $r$-cycle is a sequence of $r \ge 1$ numbers $i_1, \ldots, i_r$ such that $i_{k + 1} = \sigma(i_k)$ and $i_r = i_1$. Given an $r$-cycle of $\sigma$, we can collect each $i_1, \ldots, i_r$ into a set $C \subseteq [N]$ called an orbit and restrict $\sigma$ to this cycle so that $\sigma_C$ is equal to $\sigma$ in $C$, and the identity outside of $C$ as in $\forall x \in C:\ \sigma_C(x) = \sigma(x)$ and $\forall x \in C^{\mathsf{c}}:\ \sigma_C(x) = x$. Remarkably, permutations can be decomposed into a composition of their cycles:

Theorem (Cyclic Decomposition): Let $\sigma \in S_N$. Then, there exists a partition $C_1, \ldots, C_m$ of $[N]$ such that $\sigma = \sigma_{C_1} \circ \cdots \circ \sigma_{C_m}.$ Moreover, each $C_i$ is an orbit of a cycle of $\sigma$, and for any cycle of $\sigma$ there is a corresponding orbit $C_i$.

proof (sketch): Since $\sigma$ acts on a finite set, there exists at least one cycle, call its orbit $C_1$. We have the partition $[N] = C_1^{\mathsf{c}} \cup C_1$ and the decomposition $\sigma = \sigma_{C_1^{\mathsf{c}}} \circ \sigma_{C_1}$ since $\sigma_{C_1}(x) = x$ on $C_1^{\mathsf{c}}$. Now, $C_1$ contains at least one element, and $\sigma_{C_1^{\mathsf{C}}}$ is another permutation when restricted to $C_1^{\mathsf{c}}$, so the theorem follows by induction. $\square$

From the partitioning of $[N]$ by the orbits of a permutation, we see that any permutation also induces a collection of equivalence classes (exactly the orbits of the permutation)! Within each of these equivalence classes, we can say that the elements are identical up to the operation of $\sigma$. As well, we say that any set $C$ such that $x \in C \implies \sigma(x) \in C$ is invariant to $\sigma$ – clearly, the partitions induced by $\sigma$ are all invariant sets. These formalisms can be used to understand the structure of the coins-on-a-plate puzzle.

Solving the Puzzle

The main idea for solving the puzzle revolves around identifying particular partitions of the set of all states of the game, thus identifying equivalence relations. I then define some natural operations that one can apply to the coins on the plate and reason about how those operations map elements of one equivalence class into another. It turns out that we can apply these operations in a particular order such that we are guaranteed to eventually reach the solution.

Representation and Rotations

Abstractly, we can represent the state of each coin as either being heads $H$ or tails $T$. Since the coins are arranged in a square, we can think of the state of the whole puzzle as being the sequence of states of each individual coin following a left-to-right and top-to-bottom ordering. Notice that we’ve already encountered an equivalence class: we naturally think of aligning the coins to look as though they are on a “right-angle” when facing us – following this convention, a whole set of angles of the plate spanning $90^\circ$ ($45^\circ$ in either direction) all represent equivalent configurations. If the interrogator rotates the plate by, say, $37^\circ$ before handing it back to us, we should be able to feel around the coins to rotate this back to $0^\circ$. Thus, when we speak of an arrangement of coins on the plate, we are already really speaking of a whole equivalence class, all of which are equivalent up to rotation back to the nearest right-angle alignment of the coins. The upshot here is that listing the coins in this order results in a finite representation of the set of all possible states (of which there are $2^4 = 16$) as elements of the set \[S = \Bigl\{{HH \atop HH}, {TH \atop HH}, {HT \atop HH}, \ldots, {TT \atop TH}, {TT \atop TT}\Bigr\},\] which is naturally isomorphic (i.e., equivalent) to the set $[16] = \{1, \ldots, 16\}$.

Any such rotation operation can now naturally be thought of as a permutation $\texttt{rotate}^n$, where $n$ indicates the number of $90^\circ$ angles the plate is clockwise rotated. For example, $\texttt{rotate} \Bigl({HT \atop HH}\Bigr) = {HH \atop HT}$ and $\texttt{rotate}^2 \Bigl({HT \atop HH}\Bigr) = {HH \atop TH}$, etc. The power notation with $n$ is naturally suggestive since $n$ rotations by $90^\circ$ is equivalent to applying the single $\texttt{rotate}$ permutation $n$ times. Since we don’t know how many times the interrogator will apply $\texttt{rotate}$ before giving us the plate back, we need to work with equivalence classes of states which are equivalent up to the action of this permutation.

Remark: It is an exercise for the reader to enumerate all of the equivalence classes induced by $\texttt{rotate}$. I should note at this point though that I stumbled upon the appropriate sets and operations for solving the puzzle through intuition, and only went backwards afterwards to add some formalism. In my experience, this is typical of mathematical problem solving and research – you begin by understanding and solving particular problems and special cases ad-hoc through intuition, and then go back to formalize the ideas, which opens up new ideas and generalizations, which you begin by solving ad-hoc, and then later formalize, which leads to more questions…

Solved States

The next key observation involves another permutation and collection of equivalence classes. Consider a $\texttt{flip\_all}$ operation – you can apply this operator to the coins on the plate by flipping every coin onto its other face. Since this $\texttt{flip\_all}$ operation is just a permutation, it decomposes the space $S$ into equivalence classes (whose elements are all equivalent up to the operation of $\texttt{flip\_all}$). Examples include $\Bigl\{ {HT \atop HT}, {TH \atop TH} \Bigr\}, \Bigl\{ {HT \atop TH}, {TH \atop HT} \Bigr\}$, etc, which are all orbits containing just two elements. Another important example is \[S_\star = \Bigl\{{HH \atop HH}, {TT \atop TT}\Bigr\},\] which is the equivalence class (of $\texttt{flip\_all}$) containing the solved state ${HH \atop HH}$. The importance of this class arises as follows.

Consider the following query, which I’ll refer to as $\texttt{solved?}(s)$: Ask the interrogator if the puzzle in state $s$ is solved. If it is, you’re done. If not, the plate is returned to you after application of some number of $\texttt{rotate}$ permutations. Now, consider the composite query ($\texttt{solved?}(s)$ is not formally a permutation, so I use the word “query”): $\texttt{solved?} \circ \texttt{flip\_all}$, which in words describes: first flip all the coins onto their other side and then ask the interrogator if the puzzle is solved. Therefore, since $\texttt{rotate}$ does not change the number of heads or tails in the configuration (i.e., both elements of $S_\star$ are fixed-points of $\texttt{rotate}$) if you reach a state $s \in S_\star$ then you can first query $\texttt{solved?}$ and if the plate is returned to you (which means you must have asked about the configuration ${TT \atop TT}$) then you know that the state $s$ has not changed and you can apply the composite query $\texttt{solved?} \circ \texttt{flip\_all}$ which is now guaranteed to result in successfully solving the puzzle.

Thus, if you do this every time you want to know if the puzzle is solved, then you have effectively solved the puzzle anytime all of the coins are all face up, or all face down – if they are face down, then they will be face up the second time you ask. Thus, the two states in $S_\star$ “coins all face up” and “coins all face down” form an equivalence class of “solved states” according to the composite query described above – I’ll call this composite query $\texttt{solved?}^\star$. We can now frame our problem as reaching any state in $S_\star$.

Three More Equivalence Classes

Consider the following three sets: \[S_1 = \Bigl\{{HH \atop HT}, {HH \atop TH}, {HT \atop HH}, {TH \atop HH}, {TT \atop TH}, {TT \atop HT}, {TH \atop TT}, {HT \atop TT}\Bigr\},\] which has either a single head or a single tail; \[S_2^{(d)} = \Bigl\{{HT \atop TH}, {TH \atop HT}\Bigr\},\] with two heads and two tails arranged diagonally from one and other; and \[S_2^{(s)} = \Bigl\{{HH \atop TT}, {HT \atop HT}, {TT \atop HH}, {TH \atop TH}\Bigr\},\] consisting again of two heads and two tails, but now sitting side-by-side. Critically, the sets we have so far defined form a partition of the space: \[S = S_\star \cup S_1 \cup S_2^{(s)} \cup S_2^{(d)},\] and thus more equivalence relations given by membership in these sets. Moreover, each of these sets is invariant to both permutations $\texttt{rotate}$ and $\texttt{flip\_all}$. Indeed, $S_2^{(d)}$ is exactly one of the equivalence classes induced by both, $S_2^{(s)}$ is an equivalence class induced by $\texttt{rotate}$ and is the union of two equivalence classes of $\texttt{flip\_all}$, and $S_1$ is the union of two equivalence classes of $\texttt{rotate}$ along with the corresponding ones of $\texttt{flip\_all}$. Since the elements of the sets are all equivalent to one and other (in the sense of being members of a partition, and in the sense of rotational symmetry) we can treat all members of the class simultaneously. These sets, along with three more operations, will allow us to formalize a solution to the puzzle.

Due to the invariance of these sets to the $\texttt{rotate}$ permutation, the action of the rotation introduced by the interrogator when you query $\texttt{solved?}$ does not have any affect on the membership of $s$ in one of these sets. This is, of course, not true of any arbitrary subset of $S$, and hence the choice of sets above is in some sense “cunning” (and now we see, well motivated by the consideration of equivalence classes!). Indeed, rotations applied to $\tilde{S} = \{{HH \atop TT}, {TT \atop HH}\}$ could produce something in $\tilde{S}$, but it could also produce something in $\{{HT \atop HT}, {TH \atop TH}\}$!

More Permutations on $S$

The additional permutations we need to solve the puzzle are: $\texttt{flip\_one}$, which flips over an arbitrary single coin; $\texttt{flip\_side}$, which flips over two side-by-side coins; and $\texttt{flip\_diag}$, which flips over arbitrary cross-diagonal coins. These functions are not uniquely defined (e.g., which coin do you flip over for $\texttt{flip\_one}$) but the choice of which particular permutation you use does not matter, so I’m glossing over this.

We can reason through the effects of how these map between our equivalence classes as follows (I skip the consideration of how $S_\star$ is affected since we will always query $\texttt{solved?}^\star$ between each operation, so if we hit $S_\star$ we are done):

\begin{equation} \begin{aligned} \texttt{flip\_one}: S_1 \rightarrow S_2^{(d)} \cup S_2^{(s)} \cup S_\star\\ \texttt{flip\_one}: S_2^{(d)} \cup S_2^{(s)} \rightarrow S_1, \end{aligned} \end{equation}

since if $s \in S_1$ and we flip over a single coin, then we must now have either two heads and two tails, or we happened to flip over the “right” coin and reach a solved state. Alternatively, if we begin with two $H$s and two $T$s, then flipping one coin over necessarily puts us back in $S_1$ with either three $H$s and one $T$, or three $T$s and one $H$. For the side-by-side flip operator:

\begin{equation} \begin{aligned} \texttt{flip\_side}: S_1 \rightarrow S_1\\ \texttt{flip\_side}: S_2^{(d)} \rightarrow S_2^{(s)}\\ \texttt{flip\_side}: S_2^{(s)} \rightarrow S_\star \cup S_2^{(d)}, \end{aligned} \end{equation}

since if there is one head (or one tail), application of $\texttt{flip\_side}$ must necessarily result in three heads or tails; if there are heads diagonal from one and other (i.e., $s \in S_2^{(d)}$) then $\texttt{flip\_side}$ must necessarily flip one head and one tail, and thus result in side-by-side heads and tails; and if $s \in S_2^{(s)}$ then we either get lucky and flip the “right” coins into a solved state, or we flip the “wrong” coins into a diagonal state $S_2^{(d)}$. Finally, for the diagonal flipping operator:

\begin{equation} \begin{aligned} \texttt{flip\_diag}: S_1 \rightarrow S_1\\ \texttt{flip\_diag}: S_2^{(d)} \rightarrow S_\star\\ \texttt{flip\_diag}: S_2^{(s)} \rightarrow S_2^{(s)}, \end{aligned} \end{equation}

by similar reasoning. This is the key to the puzzle. If we are faced with some $s \in S_2^{(d)}$, then application of $\texttt{flip\_diag}$ is guaranteed to place us into the solved state $S_\star$. The only difficulty now is that we cannot know whether or not $s \in S_2^{(d)}$! Regardless, this feels like progress.

Transition Diagramming

The simple enumeration of the action of the operators $\texttt{flip\_one}, \texttt{flip\_side}, \texttt{flip\_diag}$ on the sets $S_1, S_2^{(s)}, S_2^{(d)}$ described in the previous section can be difficult to visualize. Our intuition is that we should try to find some appropriate order in which to apply these operations which will guarantee that we eventually reach the state $S_\star$, but how? We can make progress towards this task by drawing a diagram of all the possible transitions:

One thing we can notice from this diagram is that some operations map from a set and back into itself, i.e., some of the sets described above are also invariant to $\texttt{flip\_diag}$ or $\texttt{flip\_side}$. For example, $\texttt{flip\_diag}$ maps from $S_2^{(s)}$ back into itself, as well as from $S_1$ and back into itself. Recall that there is also an implicit self-loop corresponding to the $\texttt{rotate}$ permutation for each set, i.e., we can safely ask if the puzzle is solved (following the logic described in Solved States ) without the interrogator being able to effect the state we are in.

A Solution

Using the diagram above, we can deduce a solution to the puzzle. Start in any arbitrary state $s \in S$: first, make the $\texttt{solved?}^\star$ query, if $s \in S_\star$ then we are done (that was easy!), otherwise, we are in a state \[s \in S \setminus S_\star = S_1 \cup S_2^{(d)} \cup S_2^{(s)}.\] Apply the operator $\texttt{flip\_diag}$ and then query $\texttt{solved?}^\star$ again. If we happened to be in $S_2^{(d)}$, then the puzzle is solved, otherwise (notice the self-loops) we must be in $s \in S_2^{(s)} \cup S_1$. Now, apply $\texttt{flip\_side}$, followed by $\texttt{solved?}^\star$, $\texttt{flip\_diag}$, and $\texttt{solved?}^\star$ again. If we were in $S_2^{(s)}$, then $\texttt{flip\_side}$ transitioned us into $S_2^{(d)}$, and $\texttt{flip\_diag}$ brought us into $S_\star$. If we were in $S_1$, then we are still in $S_1$, since this set is invariant to both $\texttt{flip\_diag}$ and $\texttt{flip\_side}$. To solve the puzzle from here, we apply $\texttt{flip\_one}$ and query $\texttt{solved?}^\star$. If that query failed, then we must be in $S_2^{(d)} \cup S_2^{(s)}$ in which case we can apply $\texttt{flip\_diag}$ and query $\texttt{solved}?^\star$, which will finish the puzzle if we were in $S_2^{(d)}$. Otherwise, we are in $S_2^{(s)}$ and applying $\texttt{flip\_side}$ followed by $\texttt{solved?}^\star$ and then again $\texttt{flip\_diag}$ which is finally guaranteed to place us into $S_\star$, wherein a final $\texttt{solved?}^\star$ query is the end.

We have solved the puzzle – regardless of our starting position, and regardless of how random or adversarial are the rotations to the plate of coins done by the interrogator (they can even know our exact plan!), the above policy will eventually move us into the solved state $S_\star$. To summarize the policy, we apply the following order of operations, querying $\texttt{solved?}^\star$ between each operator: $\texttt{flip\_diag}, \texttt{flip\_side}, \texttt{flip\_diag}, \texttt{flip\_one}, \texttt{flip\_diag}, \texttt{flip\_side}, \texttt{flip\_diag}$.

Computational Generalizations

Solving this particular brain-teaser is satisfying, but a solution is not a good solution to me unless it is illustrative or indicative of a more general pattern. Indeed, I wrote out the formalism above exactly for the purpose of trying to reason through a more general idea. To introduce the idea, suppose we have $N$ states $V = [N] \overset{\Delta}{=} \{1, 2, \ldots, N\}$ and $K$ functions $\mathcal{F} = \{f_k: V \rightarrow V\ |\ k \in [K]\}$ on $V$. Given this data, we can define a graph $\mathcal{G} = (V, \mathcal{E})$ where the set of vertices is naturally $V$ and the edges are tuples $\mathcal{E} \subseteq \{(i, j, f)\ |\ i, j \in V, f \in \mathcal{F}\}$ where the edges $(i, j, f) \in \mathcal{E}$ is present in the graph if there is a function $f \in \mathcal{F}$ which maps from node $i$ to node $j$. Each node will have $K$ edges emanating from it – one for each function. Let’s suppose as well that there is a fixed target state $s^\star \in S$ and that this state is invariant to every function application $f_k(s^\star) = s^\star$, i.e., when we reach the target state, we know it. This graph is analogous to the graph I drew for the coin problem.

Now, the interesting part of this comes in when we suppose we do not know what state we are in. The new question is: can we find a search algorithm which will find a sequence of functions leading to a goal state, but which does not require knowledge of what vertex it is currently visiting? Abstractly, given a target state $s^\star$, can we find a finite (ideally minimal) sequence of functions $f_1, f_2, \ldots, f_M \in \mathcal{F}$ which when applied from an arbitrary starting point, are guaranteed to eventually reach the target state $s^\star$?

Uncertain Breadth First Search

There may exist a better algorithm, but this problem can be solved by means of breadth first search. However, we cannot just apply BFS directly upon the graph $\mathcal{G}$, since we don’t know our starting point. Instead, we’ll apply BFS on a graph over the power set $2^V$ of $V$ (i.e., the set of all subsets of $V$). We generalize the application of $f \in \mathcal{F}$ to act on $2^V$ by the definition \[ W\subseteq V:\ f(W) = \{f(w)\ |\ w \in W\},\] i.e., we apply $f$ to every element of the subset. This gives us an edge relation for a graph $\mathcal{G}^\star = (2^V, \mathcal{E}^\star)$: we have \[(U, W) \in \mathcal{E}^\star \iff \bigl(\exists f \in \mathcal{F}:\ f(U) = W\bigr).\] The goal is to find a path through this “power-graph” leading to the state $S_\star = \{s^\star\}$, the singleton set containing only the goal state.

A completely hacked-together implementation of this idea for the coins-on-a-plate problem is given here:

import typing as ty

from functools import reduce
from collections import deque

# Some nicer names for the nodes in the graph
Node = str
S1: Node = "S1"
S2d: Node = "S2d"
S2s: Node = "S2s"
S_star: Node = "S_star"

TransitionFunction = ty.Callable[[Node], ty.Set[Node]]

# To easily keep track of what has been visited
def _hash(s: ty.Set[Node]):
    return hash("".join(str(hash(n)) for n in s))

# Computes the set of next nodes obtained by applying `op` from some uncertainty set
def _next_nodes(nodes: ty.Set[Node], op: TransitionFunction):
    return reduce(lambda acc, new_nodes: acc | new_nodes, (op(w) for w in nodes), set())

# The robust implementation of BFS
def uncertain_bfs(nodes: ty.Set[Node], ops: ty.Dict[str, TransitionFunction]):
    visited = {_hash(nodes)}
    queue = deque([(("START",), nodes)])
    while queue:
        path, nodes = queue.pop()
        for name, op in ops.items():
            new_nodes = _next_nodes(nodes, op)
            if (hsh := _hash(new_nodes)) not in visited:
                visited.add(hsh)
                queue.append(((*path, name), new_nodes))
            if new_nodes == {S_star}:
                print(f"SUCCESS: {(*path, name)}")
                return
    print("FAILURE!")

# Helper to make transition functions with S_star being an absorbing state
def _make_op(mapping: ty.Dict[Node, Node]) -> TransitionFunction:
    def op(v: Node) -> ty.Set[Node]:
        if v == S_star:
            return {S_star}
        return mapping[v]
    return op

# The operations on the coins
flip_diag = _make_op({S1: {S1}, S2d: {S_star}, S2s: {S2s}})
flip_side = _make_op({S1: {S1}, S2d: {S2s}, S2s: {S2d, S_star}})
flip_one = _make_op({S1: {S2d, S_star, S2s}, S2d: {S1}, S2s: {S1}})

# The possible starting positions
nodes = {S1, S2d, S2s, S_star}

# And names for the transition functions
transition_functions = {"flip_diag": flip_diag, "flip_one": flip_one, "flip_side": flip_side}

# Which we finally run the algorithm on
uncertain_bfs(nodes, transition_functions)

\[\texttt{SUCCESS: (‘START’, ‘flip\_diag’, ‘flip\_side’, ‘flip\_diag’, ‘flip\_one’, ‘flip\_diag’, ‘flip\_side’, ‘flip\_diag’)}\]

And indeed, this code outputs the same policy I came up with ad-hoc in the previous section.

Conclusion

I thought that the puzzle described at the beginning of this article was very neat and wanted to write down a complete solution. We saw how the idea of symmetries were crucial for solving the puzzle, and that these symmetries can be described through the language of equivalence classes. Moreover, the remarkable cyclic decomposition theorem for permutations (nothing but bijections on finite sets) tell us that we can partition all the possible states of the coins on the plate into a collection of cycles and equivalence classes. Some of these sets are invariant to certain operations on the coins, and therefore create self-loops in a graph describing how the state of the puzzle evolves. Diagramming this graph allowed us to write down an ad-hoc solution to the coins-on-a-plate puzzle.

The idea of searching in a graph when you don’t know your starting point generalized to a form of robust breadth-first search. In this algorithm, we keep track of an entire uncertainty set of nodes that we can plausibly be, and stop once that set contains nothing but the goal state. I have few ideas for what such an algorithm might be practically useful for, but there it is!

Lyapunov Stability, Linear Systems, and Semidefinite Programming

Sat, 17 Dec 2022 00:00:00 -0800

Dynamical systems are ubiquitous models occuring in science, engineering, and mathematics. Not only are they used to model real-world dynamic phenomena like the dynamics of chemical plants, population growth, and physical engineered systems, they can also be applied to model algorithms themselves. This post focuses on linear dynamical systems, their analysis by means of semidefinite programming, and connections with control theory through the computation of quadratic functionals of their paths.

Introduction

This post introduces the Lyapunov approach to stability and convergence analysis. While this approach and its connection to semidefinite programming may seem like “overkill” for linear systems (where you can just look at the eigenvalues of the system matrix), I believe it is an insightful starting point. Analysis based on Lyapunov functions generalizes naturally to both nonlinear and stochastic systems, and the connection with semidefinite programming can be exploited to derive an analysis which may be, in some sense, optimal.

References

Much of the material in this post is derived or inspired from: Boyd, Stephen, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear matrix inequalities in system and control theory. Society for industrial and applied mathematics, 1994. This is an easily accessible book on linear matrix inequalities. I have also drawn some ideas and inspiration from the Convex Optimization book (also freely available ) Boyd, Stephen, Stephen P. Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. and have made use of cvxpy for many some of the examples: Diamond, Steven, and Stephen Boyd. “CVXPY: A Python-embedded modeling language for convex optimization.” The Journal of Machine Learning Research 17, no. 1 (2016): 2909-2913.

Linear Dynamical Systems

Linear dynamical systems are generally characterized by a matrix $A \in \R^{n \times n}$ which describes how the state vector $x \in \R^n$ changes through time. For continuous time dynamical systems, we work with ordinary differential equations

\begin{equation} \frac{\mathsf{d}}{\mathsf{d} t} x(t) = Ax(t), \end{equation}

usually written simply as $\dot{x} = Ax$ where it is to be understood that $x$ is a continuous function of time $t$ and $\dot{x}$ denotes differentiation with respect to $t$. In this case, $A$ describes a velocity field in state space – each point $x \in \R^n$ is associated to some velocity $Ax$. Analogously, for discrete time dynamical systems, we have an iterative relationship

\begin{equation} x(t + 1) = Ax(t), \end{equation}

where in this case the matrix $A$ specifies the next state of the system.

Remark: I like to use $t$ for both discrete and continuous time. The reader is trusted to recognize which is which (or if it matters at all) from the context.

Somewhat more generally, we could work with systems having some constant offset $b \in \R^n$ as in $\dot{x} = Ax + b$. However, if $A$ is a full-rank matrix (meaning that it is invertible), then this additional offset term is not interesting. Indeed, we can just as-well shift the coordinates of the system and instead work with $\dot{z} = Az$ where $z = x - A^{-1} b$. Anything interesting that can be established for $x$ can be done so by first doing it for $z$. If the matrix $A$ is not invertible, then there is a subspace which the matrix $A$ has no effect upon, and this subspace can be analyzed separately. It will become more-or-less obvious how to do this as we proceed, so I will make the standing assumption that $A$ is full-rank and $b = 0$.

Remark: The case of linear systems with a general input $\dot{x}(t) = Ax(t) + Bu(t)$ on the other hand are quite interesting, and is likely to be the topic of some future post.

Examples

Firstly, it should be understood that much of the analysis of nonlinear dynamical systems, i.e., ordinary differential equations of the form $\dot{x} = f(x)$, comes down to the linear case. The reason lies in the Hartman-Grobman Theorem (which we have also already seen ): the nonlinear system $\dot{x} = f(x)$ can be approximated by the linear dynamical system $\dot{x} = Ax$ for in a neighbourhood of an equilibrium point (i.e., point $x_e$ such that $f(x_e) = 0$) with the matrix $A = \mathsf{D} f(x_e)$, the derivative of $f$. More examples follow.

Gradient Descent

One of the modern drivers of interest in systems theory is for applications in machine learning, or more specifically, optimization algorithms. One of the most famous classical algorithms for optimization is gradient descent: given a function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ that we want to minimize, the gradient descent algorithm proceeds through the iterates \[x(t + 1) = x(t) - \alpha \nabla f(x(t)),\] where $\nabla f(x) = (\mathsf{D} f(x))^{\mathsf{T}}$ is the gradient and $\alpha > 0$ is a step-size parameter. In general, this is a nonlinear system. However, there is a particularly important special case when $\nabla f(x)$ is a linear function of $x$: when $f(x) = \frac{1}{2}x^{\mathsf{T}} Q x$ is a quadratic function. In this case, $\nabla f(x) = Qx$ and gradient descent \[x(t + 1) = (I - \alpha Q)x(t)\] is a linear system with $A = I - α Q.$

Electronic Circuits

Another interesting example comes from electrical engineering. In the design of electronic circuits, there are devices called capacitors (basically two metal plates sandwiched next to each other) which can store a small electric charge and then subsequently release it. Similarly, inductors (coils of wire) store magnetic energy, and later release it.

Using these devices, along with simple resistors, we can construct an RLC circuit – a circuit consisting of resitors, inductors, and capacitors (the ‘L’ for inductors coming from the scientist Heinrich Lenz). One of the simplest such circuits is to just place these three elements in a loop with each other.

Using Kirchoff’s voltage law , which says that the total voltage differences around a loop must sum to zero, we can obtain the equation $V_R(t) + V_L(t) + V_C(t)$ where $V_R, V_L, V_C$ are the voltages across the resistor, inductor, and capacitor at time $t$, respectively.

The needed facts for this analysis are the the current-voltage relationships: $V_R(t) = RI_R(t),$ $V_L(t) = L\dot{I}_L(t)$, and $V_C(t) = \frac{1}{C}\int_0^t I_C(\tau)\mathsf{d}\tau$ with $I_R, I_L, I_C$ being the currents passing through the devices, and $R, L, C$ denoting the “size” of the devices in units of Ohms (resistance), Henries (inductance), and Farads (capacitance). These are nothing but functions which describe how the devices operate. The other key observation is that, since these devices are all placed in series one after the other along the same wire: $I_R = I_L = I_C$! Thus, we have obtained the integro-differential equation:

\begin{equation} \begin{aligned} RI(t) + L\dot{I}(t) + \frac{1}{C}\int_0^t I(\tau) \mathsf{d} \tau &= 0\\ \implies R\dot{I}(t) + L\ddot{I}(t) + \frac{1}{C}I(t) &= 0, \end{aligned}\notag \end{equation}

which by differentiating through the whole thing results in a second order ODE. We now intend to play a clever trick with this equation – a common source of linear systems. We think of the state variables as being $x(t) = \bigl(\dot{I}(t), I(t)\bigr)$, and by using the relationship between these derivatives described above we obtain:

\begin{equation} \begin{bmatrix} \ddot{I}\\ \dot{I} \end{bmatrix}= \begin{bmatrix} -\frac{R}{L} & -\frac{1}{CL}\\ 1 & 0\\ \end{bmatrix} \begin{bmatrix} \dot{I}\\ I \end{bmatrix}, \notag \end{equation}

which is a linear dynamical system. Simulation of this system is straightforward.

from scipy.integrate import solve_ivp

def simulate(t, R, C, L, I0=1.0, I_dot0=0.0):
    A = np.array([[0, 1.0], [-1.0 / (C * L), -R / L]])

    def ode(I):
        return A @ I

    res = solve_ivp(
        fun=lambda t, y: ode(y),
        t_span=(t[0], t[-1]),
        y0=np.array([I0, I_dot0]),
        t_eval=t
    )
    return (res.y[0, :], res.y[1, :])

The reader unfamiliar with this trick is encouraged to study this example, as the technique is very commonly used. The behaviour of the system can now be understood by analyzing the dynamics of this ODE given some set of initial conditions $I(0), \dot{I}(0)$.

Semidefinite Programming

In convex optimization, a semidefinite program is an optimization problem involving a matrix variable $X$, a linear cost function $J(x) = \mathsf{tr}\ C^{\mathsf{T}} X = \sum_{i, j} C_{ij} X_{ij}$, a linear equality constraint $\mathcal{A}(X) = 0$, and a constraint $X \succeq 0$, which means that $X$ must be a positive semidefinite matrix. If you are not familiar with what this is, it is a matrix which is (1) symmetric and (2) has non-negative eigenvalues. A generic SDP is as follows:

\begin{equation} \begin{aligned} \underset{X \in \R^{n \times n}}{\text{minimize}}&\quad \mathsf{tr}\ C^{\mathsf{T}} X\\ \text{subject to} &\quad \mathcal{A}(X) = 0\\ &\quad X \succeq 0. \end{aligned}\notag \end{equation}

The linear function $\mathcal{A}(X) = 0$ can be any linear function of $X$ – examples include $AX = 0$, $AXA^{\mathsf{T}} - X + Q = 0$, etc. Of course, much more general SDPs are possible, but this will be adequate for our purposes.

Practically speaking, solving (relatively small: $n < 100$) SDPs is fairly straightforward. Here is a basic CVX Program to solve an SDP we will encounter shortly:

from math import inf
import scipy.linalg as la
import cvxpy as cvx


def solve_lyapunov_sdp(A, g, verbose=True):
    """Solves the feasibility problem P > 0, (1 - g) * P - A' P A > 0."""
    n, _ = A.shape
    P = cvx.Variable(shape=(n, n), symmetric=True)
    I = np.eye(n)

    objective = cvx.Minimize(0)
    constraints = [
        P >> I,  # P is positive semi-definite
        (1 - g) * P - A.T @ P @ A >> I
        ]
    problem = cvx.Problem(objective, constraints=constraints)
    problem.solve(solver="CVXOPT", verbose=verbose)
    return problem.value, P.value

Given a matrix $A$, and a constant $\gamma$, this program will solve a semidefinite feasibility program to find a matrix $P \succeq I$ such that $(1 - \gamma) P - A^{\mathsf{T}} P A \succeq I$. This is actually just a computational means of solving a special type of feasability problem called a linear matrix inequality (LMI). Usually, LMIs are strict inequalities where it is required to find $P \succ 0$ and $(1 - \gamma) P - A^{\mathsf{T}} P A \succ 0$, and any such $P$ will do. However, since optimization problems are not usually well defined when strict inequalities are involved (since the feasible region is then not closed, c.f., Weierstrass’ Theorem ) it is necessary to modify the LMI to involve non-strict inequalities. Doing this in a naive way, simply replacing $\succ$ by $\succeq$, may result in a trivial output $P = 0$. However, since the LMI is linear in $P$, we can just scale the inequality and ask that it be $\succeq I$ – indeed, if any $P \succ 0$ exists which satisfies the inequality, there must also exist one satisfying $P \succeq I$.

The meaning of this LMI will be explained next.

Lyapunov Stability Theory

The most important question to answer when it comes to linear systems is stability. The term stability is used to invoke the idea of a system which, when slightly perturbed, returns back to its nominal position. However, it might be somewhat of a misnomer, as what is often really meant is convergence: given some starting point $x(0)$, does the linear system satisfy $x(t) \rightarrow 0\ \text{as}\ t \rightarrow \infty$. There are many methods of determining this, the most natural being an analysis of the eigenvalues of the system matrix $A$. Analyzing the eigenvalues however, is a method which is quite particular to the case of simple linear systems. Another method, which has a much greater deal of potential for generalization, is the method of analysis by means of Lyapunov Stability Theory.

Lyapunov Stability

One of the most important techniques for establishing system stability is Lyapunov’s direct method – I’ll explain the idea for discrete time systems, although much of the literature focuses on the continuous time case.

The idea is to find a so-called Lyapunov function $V: \R^n \rightarrow \R$ on the state space which captures some generalized notion of “kinetic energy” of the system. A Lyapunov function must satisfy a few properties. The first of which is that $V(0) = 0,$ but $V(x) > 0$ everwhere else (i.e., everywhere that $x \ne 0$). I will call this property positivity, and it is this property that makes a Lyapunov function analogous to a generalized notion of kinetic energy: it is positive except at an equilibrium point when the system stops moving. The next key property is that $V$ must be constructed such that its value decreases along the path traced by the system. That is, if it can be shown that $V(x(t + 1)) < V(x(t))$ for the system $x(t + 1) = Ax(t)$, with $x(0) = x_0 \in \R^n$ being an arbitrary initial condition, then the system can be expected to converge towards $0$.

These two properties (positivity and decreasing along trajectories) are not the whole story though. The additional technical property of coerciveness is required: $||x|| \rightarrow \infty \implies V(x) \rightarrow \infty$. This appears like an esoteric technical requirement, but it is essential. If $V$ is not coercive, then the level sets of $V$, the sets $L_\alpha = \{x \in \R^n\ |\ V(x) \le \alpha\}$ may be unbounded, and the trajectory $x(t)$ can drift off $x(t) \rightarrow \infty$ while $V(x(t))$ is stil decreasing.

In the linear case, excellent Lyapunov function candidates (i.e., functions $V$ which you hope will serve as Lyapunov functions) are quadratic forms $V(x) = x^{\mathsf{T}} P x$, for some matrix $P \in \R^{n \times n}$. It is quite easy to determine when these functions are positive – this is the case exactly when $P \succ 0$ (the matrix $P$ is positive-definite), and moreover, $V(x)$ will be coercive when $P \succ 0$ (the matrix $P$ is positive definite). Thus, it makes sense to refer to positive coercive functions $V$ as being positive definite; we can refer to a Lyapunov function for a system as being a positive definite function which is decreasing along trajectories of the system.

So, in the linear case we have the candidate function $V(x) = x^{\mathsf{T}} P x$ and the requirement $P > 0$. All that remains is to figure out if $V(x(t + 1)) < V(x(t))$ along trajectories of the system. This can be established by verifying $V(Ax) < V(x)$ (why?), that is:

\begin{equation} \begin{aligned} \forall x \in \R^n:\ V(Ax) - V(x) < 0 &\iff \forall x \in \R^n:\ x^{\mathsf{T}}A^{\mathsf{T}}PAx - x^{\mathsf{T}}Px < 0\\ &\iff \forall x \in \R^n:\ x^{\mathsf{T}}\bigl(A^{\mathsf{T}} P A - P\bigr)x < 0\\ &\iff A^{\mathsf{T}} P A - P \prec 0. \end{aligned}\notag \end{equation}

If we are able to simply find some matrix $P \succ 0$ such that $V(x)$ is a Lyapunov function, then we have established the convergence of our system. The astute reader will notice that this is a linear matrix inequality and can be solved by means of the above semidefinite feasibility program (with $\gamma = 0$).

Remark: Lyapunov stability is a fundamental tool for establishing convergence in nonlinear systems. One of the most elegant examples is that of the gradient flow $\dot{x} = -\nabla f(x)$ for a strongly convex function $f$ (making this assumption for simplicity). In this case, the function $f$ itself can serve as a Lyapunov function: That is, $\frac{\mathsf{d}}{\mathsf{d}t} f(x) = \nabla f(x)^{\mathsf{T}} \dot{x} = -||\nabla f(x)||_2^2 < 0$ except at $x = x^\star$, the minimizer where $\nabla f(x^\star) = 0$. Incidentally, this also tells us that the gradient flow monotonically decreases the function $f$. Convergence proofs of many algorithms come down to constructing an appropriate Lyapunov function.

Example

Consider a simple example with

\begin{equation} A = \begin{bmatrix} 0.750 & 1.00\\ -0.667 & 0.111 \end{bmatrix}. \notag \end{equation}

Notice that $A$ does not have any particularly simple properties (triangularity, diagonal dominance…) that would allow us to determine at a glance (unless we are rather speedy when it comes to calculating eigenvalues of $2 \times 2$ matrices!) that the system $x(t + 1) = Ax(t)$ is stable. However, we can plug this matrix into our trusty semidefinite program and obtain a matrix

\begin{equation} P = \begin{bmatrix} 8.419 & 3.636\\ 3.636 & 12.109 \end{bmatrix}, \notag \end{equation}

which proves stability by Lyapunov’s method.

I’ve plotted above a figure showing the value of various functions of the state $x(t)$ for the example. The main observation is that while they all are roughly “decreasing”, it is only the Lyapunov function which is decreasing monotonically. That is not to say that this Lyapunov function is the only function with this property – there are many Lyapunov functions. More subtly, it may also be possible to find starting points such that some other arbitrary function decreases monotonically towards zero from that particular system initialization, but a Lyapunov function will do so from any starting point.

Fast Convergence

Can we modify Lyapunov’s method to somehow establish fast convergence? The answer is yes. Consider the stronger condition:

\begin{equation} \begin{aligned} V(Ax) - V(x) &< -\gamma V(x)\\ \implies V(Ax) &< (1 - \gamma)V(x)\\ \implies V(A^2 x) &< (1 - \gamma)V(Ax) < (1 - \gamma)^2 V(x)\\ &\vdots\\ \implies V(A^T x) &< (1 - \gamma)^T V(x). \end{aligned}\notag \end{equation}

which is establishing a rate of convergence on the Lyapunov function – after $T$ iterations of the system, the value must have decreased by a factor of $(1 - \gamma)^T$. It is quite convenient to recognize that the existence of a quadratic Lyapunov function $V_\gamma$ which verifies this fast convergence is equivalent to finding an ordinary quadratic Lyapunov function for the modified system with $A_\gamma = \frac{1}{\sqrt{1 - \gamma}} A$.

How do we incorporate the search for some such $\gamma$ into our SDP? All we need is that $A^{\mathsf{T}}PA - (1 - \gamma)P \prec 0$, and indeed, we would like to find the largest satisfactory $\gamma$. Unfortunately, I do not believe it is possible to directly encode optimization over $\gamma$ into a convex SDP, since the product $γ P $, where both $\gamma$ and $P$ are variables, is non-convex. Instead, since $\gamma \in \R$ is just a single parameter, we can apply bisection to find an optimal $\gamma$:

from math import inf

def is_feasible(A, g, verbose=True):
    """Checks if P > 0, (1 - g) * P - A' P A > 0 is feasible."""
    val, _ = solve_lyapunov_sdp(A, g, verbose=verbose)
    return val < inf

def find_optimal_convergence_rate(A, num_iter=20):
    right = 1.0
    left = 0.0
    for it in range(num_iter):
        g = left + (right - left) / 2
        if is_feasible(A, g, verbose=False):
            left = g
        else:
            right = g
    return g

By solving the feasability SDP multiple times, we obtain a value of $\gamma$ such that $V(A^T x) < (1 - \gamma)^T V(x)$, establishing a fast convergence rate for the system. It should be noted that specialized algorithms to solve the Lyapunov inequality $A^{\mathsf{T}} P A - P \prec 0$, but doing so by semidefinite programming opens up a more general array of possibilities, which I turn to next.

Remark: It is possible to represent this optimization problem as a joint optimization problem over $P, \gamma$, but not as a convex optimization problem. However, the resulting problem is quasiconvex.

Stability of Uncertain Linear Systems

So far, I’ve talked about linear systems with a single and perfectly well known matrix $A$. However, in many applications it may be the case that the matrix $A$ is not known exactly, or that it is only an approximation of a real system. For this reason, we would like to establish some degree of robust stability guarantee. To this end, let us suppose that we have some uncertainty set $\mathbf{A} \subset \R^{n \times n}$ and that our system matrix $A \in \mathbf{A} \subset \R^{n \times n}$. If this is the case, then we can expect our linear system to satisfy a linear recursive inclusion:

\begin{equation} x(t + 1) \in \mathbf{A} x(t) \overset{\Delta}{=} \{A x(t)\ |\ A \in \mathbf{A}\}. \notag \end{equation}

That is, the next point $x(t + 1)$ is obtained from $x(t)$ by simple multiplication by a matrix $A$, we just don’t know which matrix $A$; all we know is that $A \in \mathbf{A}$. Amazingly, establishing the stability of this system can still be done by looking for a Lyapunov function $V(x) = x^{\mathsf{T}} P x$ with $P \succ 0$ and such that $\forall x \in \R^n, A \in \mathbf{A}:\ V(A x) - V(x) < 0$ and this is equivalent to the collection of LMIs: \[P \succ 0, \forall A \in \mathbf{A}:\ A^{\mathsf{T}} P A - P \prec 0.\]

Of course, not all such LMIs can be solved – it depends on the model of $\mathbf{A}$. A quite flexible model is furnished by polytopic sets: $\mathbf{A} = \mathsf{conv}\{ A_k\ |\ k \in [K] \}$, that is, the convex hull of a finite number of matrices $A_k$, i.e., for any $A \in \mathbf{A}$ there exists some $\mu \in [0, 1]$ and matrices $A_i, A_j$ such that $A = \mu A_i + (1 - \mu) A_j$. These sets can be used to construct arbitrarily accurate approximations of any convex set $\mathbf{A}$.

Then, using the fact that $V(x)$ is convex if $P \succ 0$, we have $V(\mu A_i x + (1 - \mu) A_j x) \le \mu V(A_i x) + (1 - \mu) V(A_j x)$ and therefore a sufficient condition for the stability of the recursive inclusion is to find some $P \succ 0$ which satisfies the system of $K$ LMIs: \[P \succ 0, \forall k \in [K]:\ A_k^{\mathsf{T}} P A_k - P \prec 0.\]

Speaking intuitively, if such a $P$ exists for some uncertain linear system, we can make a conclusion about the robust stability of the system.

Example – Gain Margins and Momentum Gradient Descent

One of the main motivations for considering uncertain linear systems arises from control theory. A very common situation is that we have some system matrix $A$, as well as a gain matrix $K$ which we have designed. The gain matrix $K$ is intended to be used to steer the system $x(t + 1) = A x(t)$ towards zero by means of a feedback control $x(t + 1) = Ax(t) - Kx(t)$. Importantly, the system $A$ may very well be unstable, in which case the matrix $K$ is absolutely essential.

Suppose that we have some $K$ in hand which makes the controlled system $x(t + 1) = (A - K)x(t)$ stable. The idea of a gain margin is to determine if for some interval $[\alpha_1, \alpha_2]$ that the modified system $x(t + 1) = (A - \alpha K) x(t)$ remains stable, for any $\alpha \in [\alpha_1, \alpha_2]$. If this interval is wide, then it may give us some assurance that the matrix $K$ is robust, in some sense. The collection of system matrices \[\mathbf{A} = \{A - \alpha K\ |\ \alpha_1 \le \alpha \le \alpha_2\}\] forms a natural polytopic set.

Gradient descent constitutes a perfectly good example of this situation. For a quadratic function $f(x) = \frac{1}{2}x^{\mathsf{T}} Q x$ we have non-stable natural dynamics $x(t + 1) = x(t)$, a controller $Q$, and the stepsize $\alpha$ fills in for the gain margin, giving us $x(t + 1) = (I - \alpha Q)x(t)$. To make this a bit more interesting, we can augment this system with a momentum term by introducing a velocity $v(t)$:

\begin{equation} \begin{aligned} v(t + 1) &= \gamma v(t) + \alpha Q x(t)\\ x(t + 1) &= x(t) - v(t + 1). \end{aligned}\notag \end{equation}

These equations are known as gradient descent with momentum . The idea is that if we are heading in some “nominal” direction $v(t)$, we might as well keep going in that direction, and just add the gradient step $\alpha Qx(t)$ to our velocity. A slight modification of this momentum scheme, Nesterov’s momentum, is known to be optimal in a certain sense.

In any case, the ordinary Momentum gradient descent algorithm corresponds to the (block) system matrix

\begin{equation} A_{\gamma, \alpha} = \begin{bmatrix} \gamma I & \alpha Q\\ -\gamma I & I - \alpha Q \end{bmatrix}\notag \end{equation}

for the joint system $\bigl(v(t), x(t)\bigr)$.

Now, if you can construct a Lyapunov function (by solving a semidefinite program) for $\gamma, \alpha \in \{\gamma_1, \gamma_2\} \times \{\alpha_1, \alpha_2\}$ (i.e., all four corners of a square) then you will have proof that the algorithm will converge for every pair $\gamma, \alpha \in [\gamma_1, \gamma_2] \times [\alpha_1, \alpha_2]$.

Remark: This example is quite simplistic, but it is illustrative of general and powerful techniques . Specifically, if you can construct and solve an appropriate SDP analytically, then it can serve as a constructive proof of convergence. As well, SDPs can potentially be used to find optimal parameters for algorithms by similar means as were used earlier to calculate convergence rates by means of bisection search.

Quadratic Integrals

As a final illustration of the power of the Lyapunov approach, let’s consider how to evaluate infinite quadratic integrals (or summations, in the discrete case). Say we have a linear system $x(t + 1) = Ax(t)$, and are interested in computing the value of a of the path of this system according to the quadratic functional

\begin{equation} \begin{aligned} J_Q(x_0) = \sum_{t = 0}^\infty x(t)^{\mathsf{T}} Q x(t); x(0) = x_0, \end{aligned}\notag \end{equation}

where $Q \succ 0$ is some arbitrary positive definite matrix. Such functions arise often in control theory.

To see the connection with Lyapunov theory, recognize that we can fully expand out the summation in terms of $x_0$ by using the fact that $x(t) = A^t x_0$:

\begin{equation} \begin{aligned} J_Q(x_0) &= \sum_{t = 0}^\infty \bigl(A^t x_0 \bigr)^{\mathsf{T}} Q \bigl(A^t x_0)\\ &= x_0^{\mathsf{T}} \Bigl( \sum_{t = 0}^\infty (A^t)^{\mathsf{T}} Q A^t \Bigr)x_0\\ &= x_0^{\mathsf{T}} Q x_0 + \Bigl( \sum_{t = 1}^\infty (A^t)^{\mathsf{T}} Q A^t \Bigr)x_0\\ &= x_0^{\mathsf{T}} \Bigl(Q + \sum_{t = 1}^\infty (A^t)^{\mathsf{T}} Q A^t \Bigr)x_0\\ &= x_0^{\mathsf{T}} \Bigl(Q + A^{\mathsf{T}} \bigl[\sum_{t = 0}^\infty (A^t)^{\mathsf{T}} Q A^t \bigr]A \Bigr)x_0\\ &= x_0^{\mathsf{T}} Q x_0 + J(Ax_0). \end{aligned}\notag \end{equation}

Thus, since $J$ is a quadratic function of $x_0$, there must be some $P$ such that $J(x_0) = x_0^{\mathsf{T}} P x_0$ and therefore $J(x)$ satisfies, for any starting point $x \in \R^n$:

\begin{equation} \begin{aligned} J(x) &= x^{\mathsf{T}} P x\\ &= x^{\mathsf{T}}[Q + A^{\mathsf{T}} P A] x, \end{aligned}\notag \end{equation}

or in other words, $P - A^{\mathsf{T}} P A = Q$. What this tells us is that if we have a matrix $P \succ 0$ verifying the Lyapunov inequality $P - A^{\mathsf{T}} P A \succ 0$, then not only do we have a Lyapunov function, but there is some positive definite matrix $Q = P - A^{\mathsf{T}} P A$ such that $x^\mathsf{T} P x$ is the value of the quadratic functional $J_Q(x)$. Conversely, if we can solve the Lyapunov equation $P - A^{\mathsf{T}} P A = Q$ for some $P \succ 0$, then we have a Lyapunov function, as well as a function for easily evaluating the value of $J_Q(x)$, namely, $J_Q(x) = x^{\mathsf{T}} P x$.

I intend to explore the ideas surrounding quadratic functionals in more detail in a future post. These functions can be used to tie together semidefinite programming duality, stochastic linear systems (with random disturbances), as well as the classical methods of stochastic optimal control: the Kalman filter and the linear quadratic regulator.

Remark: Lyapunov’s equation $P - A^{\mathsf{T}} P A = Q$ is a linear equation in $P$ and can be solved efficiently. In Python, the function scipy.linalg.solve_discrete_lyapunov can be used for this purpose. Lyapunov equations arise in a number of different ways, so it is a useful pattern to have in mind.

Conclusion

My purpose in writing this has been to explore the Lyapunov approach to analyzing the stability of linear dynamical systems. This is only one of many possible approaches to this analysis, but it also generalizes (in various directions) much more easily than does a direct analysis of the eigenvalues of the system matrix. Indeed, the Lyapunov approach can also lead to global convergence theorems for nonlinear systems, whereas the Hartman-Grobman theorem combined with an eigenvalue analysis can only lead to a local convergence result. Moreover, the semidefinite programming perspective on this problem can also generalize greatly, and opens the door to constructing optimal algorithms. The tradeoff with the Lyapunov approach is that constructing an appropriate Lyapunov function can be extremely difficult, and there is no ready-made recipe for doing so – it often comes down to your creativity!

Solving Equations with Jacobi Iteration

Sun, 11 Dec 2022 00:00:00 -0800

Jacobi iteration is a natural idea for solving certain types of nonlinear equations, and reduces to a famous algorithm for linear systems. This post discusses the algorithm, its convergence, benefits and drawbacks, along with a discussion of examples and pretty pictures 🖼️.

Introduction

The method of Jacobi Iteration, named after the 19th century mathematician Carl Gustav Jacobi (the same Jacobi for whom the Jacobian in calculus is named after), is a numerical method for solving systems of equations. It is particularly famous as a classical iterative algorithm for solving linear systems. One of the main reasons one might want to use Jacobi iteration in practice is that it admits of a naturally parallel implementation, and can thus scale to very large and complex systems, and even to problems without any closed form representation of the system we want to solve (like a simulator, or a black-box machine learning algorithm).

In this post, I’ll introduce the basic mathematical intuition of Jacobi iteration, along with some examples of how and where it might arise. I’ll also give a brief analysis of convergence for linear systems of equations, where the concept of a diagonally dominant matrix arises. I also attempt to extend the intuition of the linear case to nonlinear systems by using techniques from the theory of ordinary differential equations.

Systems of Equations

We are all familiar with the grade-school notion of a mathematical equation. Particularly famous are the quadratic polynomial equations, e.g., $x^2 - x - 2 = 0$, which has exactly the solutions $x = 2$ and $x = -1$. For another example, the trigonometric equation $\mathsf{sin}(\frac{(2x + 1) \pi}{2}) - 1 = 0$ has any $x$ being an integer multiple of $2$, i.e., $x \in 2\Z$ as a solution.

What is meant by a system of equations is simply a multitude of ordinary equations that need to be satisfied simultaneously. For instance, if we combined the above two examples into the system of two equations and one variable we would be left with the system

\begin{equation} \notag \begin{aligned} x^2 + x - 2 &= 0\\ \mathsf{sin}\bigl(\frac{(2x + 1) \pi}{2}\bigr) - 1 &= 0, \end{aligned} \end{equation}

which now has a single unique simultaneous solution $x = 2$.

In general, we can write systems of equations using a single multivariate function $F: \R^m \rightarrow \R^n$ which takes $m$ variables $x = (x_1, x_2, \ldots, x_m)$ as in $F(x)$, and outputs $n$ values $F(x) = \bigl(F_1(x), F_2(x), \ldots, F_n(x)\bigr)$. It is a rule of thumb (certainly not an actual rule) that if there are $n$ equations, you can expect to be able to solve for $n$ variables, i.e., the function $F$ is “square” with $m = n$. Assuming this square case is by no-means essential, but it simplifies many of our examples, so we will run with this case.

The ultimate goal of solving systems of equations is to find some $x \in \R^n$ such that $F(x) = 0$. We can motivate such problems with a few illustrative examples.

Minimizing Functions

Consider a function $f: \R^n \rightarrow \R$ which we want to minimize, i.e., to find an $x^\star \in \R^n$ such that $f(x^\star) \le f(x)$ for every other $x$ in some neighbourhood of $x^\star$. One might want to think of $f(x)$ perhaps as a design objective (find the best design according to the cost function $f$), or as a machine learning loss function, etc. It is a theorem (the first order necessary conditions) that for differentiable functions $f$, any minimizer $x^\star$ must necessarily satisfy the derivative condition $\mathsf{D} f(x^\star) = 0$, where $\mathsf{D} f: \R^n \rightarrow \R^n$ is the derivative of $f$.

Economic Equilibria

In economics, many believe that prices of goods in an economy are determined by the matching of supply and demand. That is, if $\mathcal{D}(p)$ is a demand curve and $\mathcal{S}(p)$ is a supply curve, then we expect the price $p$ to be found from solving the nonlinear equation $\mathcal{D}(p) = \mathcal{S}(p)$. I have much more to say about this example in Section Example: Supply and Demand with Substitution , where I apply Jacobi iteration to a simple coffee and tea economy where the two goods serve as partial substitutes for one and other.

Robotic Manipulators

Robotic arms with stiff linkages can be modeled by the angles $\theta$ of their joints. For example, the position of your hand (your end affector) in space can be determined as a function of the length of your upper arm and your forearm, along with the angles formed at your elbow and shoulder (including the multiple dimensions of rotation your shoulder is capable of). Specifically, we might wish to describe the position of your hand in space by a function $F(\theta)$ of these joints. The problem of determining the appropriate angular settings of your joints, in order to place your hand at a point $p \in \R^3$ in space, is a problem of solving the system of equations $F(\theta) - p = 0$, and one which your brain apparently solves with remarkable ease.

Jacobi Iteration

The ideal of Jacobi iteration is to split up the system $F(x) = 0$ of $n$ equations in $n$ unknowns, into a sequence of simpler equations of one variable and one unknown. For equation $i$, we imagine that every variable except $x_i$, let’s call them $x_{-i}$ is held fixed, and that we don’t care about the value of any function except $F_i$. We then solve the univariate equation $F_i(x_i; x_{-i}) = 0$ (This notation is common in game theory, I hope that it is understood) to find the single value of $x_i$ that results in function $F_i$ being satisfied.

This process is carried out, possibly in parallel, simultaneously for each equation to obtain a new set of points which we hope is closer to satisfying the full system of equations.

Remark: Proceeding sequentially, rather than simultaneously, by updating each $x_i$ value immediately after finding a univariate solution, and before finding the next one, is an algorithm called Gauss-Seidel Iteration. The reader is encouraged to meditate upon the difference.

A pseudo-code algorithm implementing Jacobi iteration is given as follows:

Initialize $x(0) \in \R^n$
for $t = 1, 2, \ldots$
- parallel for each $i \in [n]$
  - find $\tilde{x}_i$ such that $F_i(\tilde{x}_i, x_{-i}(t)) = 0$ // i.e., solve the $i^{th}$ equation
- set $x(t + 1) = \tilde{x}$

In order that this algorithm be an appropriate choice for your problem, it is at least necessary that the step requiring that we “solve the $i^{th}$ equation” can be reliably carried out. However, even if solving each of the sub-problems is challenging, we may benefit from the straight-forward parallelism offered by finding each $\tilde{x}_i$ simultaneously.

The other major issue is convergence, i.e., does the algorithm actually find a solution? We can get fairly clear answers to this question in the linear case by applying dynamical systems theory. As is so often the case, the theory for linear functions serves as a stepping stone to building intuition more generally.

The Case of Linear Equations

Whether or not the algorithm is actually able to find some $x$ such that $F(x) = 0$ (assuming at the very least that such an $x$ exists!) is highly problem dependent and in practice may require various “clever tweaks”, or benefit from restarting the algorithm from a wide range of initial points $x(0)$. However, for the case of linear systems of equations $Ax = b$ (with $A \in \R^{n \times n}$ a square matrix with real entries), the algorithm is both elegantly simple, and admits of an easily verifiable sufficient condition for convergence known as diagonal dominance.

To understand how this works, lets solve for $x_i$ such that the $i^{th}$ equation is satisfied. That is,

\begin{equation} \notag \begin{aligned} \sum_{j = 1}^n A_{ij} x_j &= b_i\\ \iff A_{ii} x_i &= b_i - \sum_{j: j \ne i} A_{ij} x_j\\ \iff x_i &= \frac{1}{A_{ii}}\bigl(b_i - \sum_{j: j \ne i}A_{ij} x_j \bigr), \end{aligned} \end{equation}

where we should notice that we already require the diagonal elements of $A$ to be non-zero (otherwise we couldn’t divide).

Focusing for a moment on the term $\sum_{j \ne i} A_{ij} x_j$, this is nothing but the $i^{th}$ element in the matrix multiplication $Ax$ minus the component $A_{ii} x_i$, that is, $\sum_{j \ne i} A_{ij} x_j = (Ax)_i - A_{ii} x_i$. Using this, we can write the parallel updates to the entire vector $x$, and the entire Jacobi iteration algorithm as:

\begin{equation} \notag x(t + 1) = D^{-1} (b - Mx(t)), \end{equation}

where $D = \mathsf{dg}(A)$ (the diagonal elements of $A$), $M = A - D$ is all of the off-diagonal elements of $A$, and $x(0) \in \R^n$ is an arbitrary starting point for the sequence of candidate solutions $x(t)$. When people refer to “Jacobi iteration”, it is usually this algorithm in particular that they are referring to.

Convergence

The hope is that the iterations $x(t)$ converge to a solution $x^\star$, i.e., $x(t) \rightarrow x^\star$, where $Ax^\star = b$.

To see why we might expect this to happen, imagine that $x(t)$ converges to a fixed point of the algorithm, i.e., some $x(\infty)$ satisfying $x(\infty) = D^{-1} (b - M x(\infty))$. For such a point it holds that:

\begin{equation} \notag \begin{aligned} x(\infty) &= D^{-1} (b - M x(\infty))\\ Dx(\infty) &= b - M x(\infty)\\ (D + M)x(\infty) &= b\\ Ax(\infty) &= b, \end{aligned} \end{equation}

which is a solution to the equation $Ax = b$!

To establish the convergence of the iterations, it is well known (this is likey to be a topic of a future post! 😜) that for linear systems $x(t + 1) = a + K x(t)$, $x(t)$ will converge whenever $\rho(K) < 1$, where $\rho(K) = \text{max}_i\ |\lambda_i(K)|$ is the largest eigenvalue magnitude, a quantity called the spectral radius. For the case of Jacobi iteration, we require that $\rho(D^{-1} M) < 1$.

Using the Gershgorin circle theorem , and the fact that the diagonal values of $M$ are all $0$ (by construction), it holds that every eigenvalue of $D^{-1} M$ lies within a distance $|D_{ii}^{-1}|\sum_{j \ne i} |M_{ij}|$ of $0$. A condition on $A$ which guarantees this is diagonal dominance: $\sum_{j \ne i} |A_{ij}| < |A_{ii}|$ for each $i$. If this holds, then the eigenvalues $\lambda_i$ of $A$ are “close” to the diagonals of $A_{ii}$, so such matrices are in some sense “almost diagonal”. Since linear systems $Dx = b$, for diagonal $D$, are easy to solve, it is not surprising that there is a simple iterative algorithm for solving linear systems which are diagonally dominant. Moreover, if the diagonals of $A$ are also non-zero, $A$ will necessarily have non-zero eigenvalues, and therefore it will be full-rank, invertible, and admit of unique solutions. This is all summarized with the following theorem.

Theorem: Let $A \in \R^{n \times n}$ be a square matrix with real entries. Suppose that $A$ has a non-zero diagonal and is diagonally dominant. Then, for any vector $b \in \R^n$, there exists a unique solution $x^\star$ to the linear equation $Ax = b$ and Jacobi iteration converges, from any initial condition, to the solution $x^\star$.

Linear $2 \times 2$ Example

Some straightforward Python code implementing linear Jacobi iteration is provided in the listing below. A realistic implementation should have a method of detecting divergence. As well, checking the norm of the distance to the solution on every iteration is relatively expensive – it essentially doubles the computational effort.

def solve_linear_system(A, b, x0=None, eps=1e-6, maxiter=np.inf):
    """Solves the linear system Ax = b by Jacobi iteration.

    The algorithm's starting point is x0.  The algorithm is only guaranteed
    to converge if A is diagonally dominant.
    """
    Dinv_M, Dinv_b = (A - np.diagflat(D)) / np.diag(A)[:, None], b / np.diag(A)

    x = np.array(x0) if x0 is not None else np.zeros_like(b)

    it = 0
    while it < maxiter and np.linalg.norm(A @ x - b) > eps:
        x = Dinv_b - Dinv_M @ x
        it += 1

    return x

For problems in $\R^2$ (i.e., with just two variables) it is quite straight-forward to produce nice-looking figures. The following figure plots the flow of an associated ODE, as well as an “SOR” modification (both to be explained shortly 💁), along with the discrete iterates of the Jacobi iteration algorithm for the matrix

\[ A = \begin{bmatrix} 4/5 & 3 / 5 \\ -6 / 5 & 7/5\end{bmatrix}, \]

and $b = 0$. The case $b = 0$ is without loss of generality for the purpose of plotting. Even though finding a solution to the equation $Ax = 0$ is trivial in this case, it is equivalent to re-orienting the center of our coordinate system upon the solution $A^{-1} b$. It is also worth noting that this matrix is row-wise diagonally dominant (corresponding to our definition) but not column wise.

If you squint closely at this figure, you might even believe that the lines joining the discrete iterates are tangent to the flow lines in the background…

Relaxation and Associated Flows

Our figure above is constructed by plotting the vector field from the flow of a closely related ordinary differential equation. Let’s use $t$ to denote time, and as in the previous example, assume without loss that $b = 0$.

Merely as a device to construct a differential equation, imagine that each step of the algorithm takes $\Delta$ “algorithm time”. Precisely, let us write

\begin{equation} \notag \begin{aligned} x(t + \Delta) &= -D^{-1} Mx(t)\\ \iff x(t + \Delta) - x(t) &= -(I + D^{-1} M)x(t))\\ \end{aligned} \end{equation}

Now, we have an analogy with continually taking steps $-(I + D^{-1} M)x(t)$ to update the value of $x(t)$. If we reduce the size of these steps by $\Delta$ and then take a limit as $\Delta \rightarrow 0$

\begin{equation} \notag \begin{aligned} x(t + \Delta) - x(t) &= -\Delta(I + D^{-1} M)x(t))\\ \iff \frac{1}{\Delta} [x(t + \Delta) - x(t)] &= -(I + D^{-1} M)x(t))\\ \overset{\Delta \rightarrow 0}{\implies} \dot{x}(t) &= -(I + D^{-1} M)x(t)), \end{aligned} \end{equation}

we obtain a linear ordinary differential equation.

Remark: The stablity of linear ODEs can be established by checking that the eigenvalues of the corresponding system matrix have negative real part. In our case, due to the negative sign, we require that $\mathfrak{R}[\lambda_i(I + D^{-1} M)] > 0$ hold for every $i$, where $\lambda_i(A)$ is the $i^{th}$ eigenvalue of $A$. Since $D^{-1}M$ has $0$ on the main diagonal (by construction) the Gershgorin circle theorem tells us that the eigenvalues lie within Gershgorin discs centered at $1$. Clearly, the condition $\rho(D^{-1} M) < 1$ is sufficient for the stability of this ODE. But, thinking intuitively, should one expect that this ODE be “more likely” to converge than the discrete algorithm? The reader is encouraged to visualize this situation, and to think about this stability condition.

Let’s play with this a bit. What if we didn’t take a full limit towards $\Delta \rightarrow 0$? Rearranging the above equations results in another discrete algorithm

\begin{equation} \notag x(t + \Delta) = (1 - \Delta) x(t) -\Delta D^{-1} Mx(t)), \end{equation}

where $\Delta > 0$ is a parameter of the algorithm, with $\Delta = 1$ corresponding to ordinary Jacobi iteration.

This algorithm is now picking the next point to lie somewhere along the line connecting $x$ and the nominal next step $s(x)$, that is: as $x \leftarrow (1 - \Delta) x + \Delta s(x)$. This is a general pattern called Successive Over-relaxation (SOR), and can be applied to any iterative algorithm which takes as input a point $x$, and outputs the next point as $x \leftarrow s(x)$. When $\Delta < 1$, the next point will lie somewhere in the interval $[x, s(x)]$ between the current point and the nominal next step (the “relaxation” part of SOR); and $\Delta > 1$ means that the step goes beyond $s(x)$ to move a further distance (the “over” part). The value $\Delta = 0$ corresponds to the vanilla version of the algorithm. I don’t think that the case $\Delta < 0$ has any sensible use (at least not directly in this context) as you would be going backwards in some sense, but maybe it’s an interesting possibility to think about.

The reason that the SOR algorithm can be expected to converge for a broader collection of $A$ matrices than ordinary Jacobi iteration, is that the space of matrices satisfying $\mathfrak{R}[\lambda_i(I + D^{-1} M)] > 0$ is larger than those which satisfy $\rho(D^{-1} M) < 1$ (can you see why?). However, for a matrix $A$ such that $D^{-1}M$ has an eigenvalue with real part less than $-1$, both algorithms will diverge.

The case of Nonlinear Equations

For the case of nonlinear equations, we can use tools from ordinary differential equations to say something about the convergence of the associated Jacobi flow, and then extend that intuition through successive over-relaxation to try to understand something about how the discrete algorithm will behave.

To this end, let us go back to the nonlinear equation $F(x) = 0$ that we want to solve. Supposing that the Jacobi iteration algorithm can be implemented for $F$ (i.e., each equation in the system can be solved in the “diagonal” variable), we can write the algorithm with the notation

\begin{equation} \notag x(t + 1) = \mathcal{M}(x(t)), \end{equation}

where for any $x$, $y = \mathcal{M}(x)$ is a vector such that $F_i(x_{-i}, y_i) = 0$, that is, solves the $i^{th}$ equation in the $i^{th}$ variable. Following the same trick as we had in the linear case, we can construct an ODE:

\begin{equation} \notag \begin{aligned} \frac{x(t + \Delta) - x(t)}{\Delta} &= \mathcal{M}(x(t)) - x(t)\\ \overset{\Delta \rightarrow 0}{\implies} \dot{x}(t) &= \mathcal{M}(x(t)) - x(t). \end{aligned} \end{equation}

Thus, we can hope to understand the convergence of nonlinear Jacobi iteration by analyzing the convergence of the nonlinear system of ODEs associated to the function $\mathcal{M}$. Firstly, a fixed point of this ODE is, similarly to the linear case, a solution to the nonlinear equation. So, perhaps we can understand the convergence of $x(t)$ for points where $\mathcal{M}(x(t)) \approx x(t)$? The method for doing this type of local analysis of ODE convergence is the Hartman-Grobman Theorem (HGT from hereon). Intuitively, our hope is that if we initialize the algorithm somewhere “close” to a solution, that it will actually converge to that solution. That is, we hope that the ODE is locally asymptotically stable.

The HGT essentially tells us that if the Jacobian $A = \mathsf{D}\mathcal{M}(x^\star) - I$ of the system at an equilibrium point $x^\star$ is stable, i.e., $\dot{x} = Ax$ converges to $x^\star$, then there is a local neighbourhood around $x^\star$ such that the original nonlinear ODE $\dot{x} = \mathcal{M}(x) - x$ converges to $x^\star$.

This statement is still pretty abstract, since we don’t really have a handle on what $\mathcal{M}$ is explicitly. Another abstract tool that we could apply here is the Implicit Function Theorem , which would allow us to calculate $\mathsf{D}\mathcal{M}$ in terms of derivatives of the original function $F$. While this approach can certainly get us somewhere, I’d like to make a closer analogy with the linear case. To this end, let’s make a simplifying assumption about the structure of $F$:

\begin{equation} \notag F_i(x) = f_i(x_i) + g_i(x_{-i})\ \forall i, \end{equation}

that is, the $i^{th}$ equation consists of an individual function $f_i: \mathbb{R} \rightarrow \mathbb{R}$ of $x_i$, along with additional coupling function $g_{i}: \mathbb{R}^{n - 1} \rightarrow \mathbb{R}^{}$ involving the remainder of the variables. We can now write down what $\mathcal{M}_i$ is explicitly:

\begin{equation} \notag \mathcal{M}_i(x) = f_i^{-1} (-g_i(x_{-i})). \end{equation}

In the linear case, $f_i^{-1}(z) = z / A_{ii}$ and $g_i(x_{-i}) = \sum_{j: j \ne i} A_{ij} x_j - b_i$. Considering now the derivatives of $\mathcal{M}_i,$ we just need to combine the Inverse Function Theorem along with the chain rule to obtain

\begin{equation} \notag \mathsf{D}\mathcal{M}_i(x) = -\frac{1}{f_i^\prime \circ \mathcal{M}_i(x)} \begin{bmatrix}\frac{\partial g_i(x_{-i})}{\partial x_1} & \cdots & \frac{\partial g_i(x_{-i})}{\partial x_{i - 1}} & 0 & \frac{\partial g_i(x_{-i})}{\partial x_{i + 1}} & \cdots &\frac{\partial g_i(x_{-i})}{\partial x_{n}}\end{bmatrix} \end{equation}

where $f^\prime \circ \mathcal{M}_i(x) = f^\prime (f_i^{-1} (-g_{i}(x_{-i})))$. Thus, in exact analogy with the linear case, we can write

\begin{equation} \notag \mathsf{D}\mathcal{M}(x) = -D(x)^{-1} M(x), \end{equation}

where $D(x)$ is a diagonal matrix consisting of the derivatives $f_i^\prime$ evaluated at the point $\mathcal{M}_i(x)$ and $M(x)$ is the Jacobian $\mathsf{D}F$ of $F$ itself, but with the diagonal elements zeroed out.

Returning to the HGT, we need to consider the stability of the linear ODE defined by the matrix $A = \mathsf{D}\mathcal{M}(x^\star) - I$. Since $\mathcal{M}(x^\star) = x^\star$, we have that $D_i(x^\star) = f_i^\prime(x^\star_i)$. Using this fact, we are inspired to make a definition of a locally diagonally dominant function. We will say that $F$ is a locally diagonally dominant (around an equilibrium point $x^\star$) if it has the form $F_i(x) = f_i(x_i) + g_i(x_{-i})$ and \[|f_i^\prime(x_i^\star)| > \sum_{j: j \ne i} |\frac{\partial g_i (x^\star_{-i})}{\partial x_j}|\] for every $i$.

Using what we know from the linear case, combined with the HGT, and the fact that the SOR scheme is an Euler Method for the ODE (which converges towards the ODE itself as $\Delta \rightarrow 0$), we have a theorem about local convergence of nonlinear Jacobi iteration:

Theorem: Let $F: \mathbb{R}^n \rightarrow \mathbb{R}^n$ be a smooth function. Suppose there exists some $x^\star$ such that $F(x^\star) = 0$ and that $F(x^\star)$ is locally diagonally dominant in a neighbourhood of $x^\star$. Then, there exists a neighbourhood $\mathcal{N}$ of $x^\star$ and a $\Delta > 0$ such that SOR nonlinear Jacobi iteration with step-size $\Delta$ converges to $x^\star$ from any initial point $x \in \mathcal{N}$.

Of course, the linear case corresponds to the functions $f_i(x_i) = A_{ii} x_i$ and $g_i(x_{-i}) = \sum_{j: j \ne i} A_{ij} x_j - b_i$, wherein we recover from above the convergence theorem of linear Jacobi iteration.

Example: Supply and Demand with Substitution

Let’s cook up an example from economics where we’ll try to work out the equilibrium prices of goods in an economy. A typical economic model (at least so far as I know… I’m no economist!), is to find the prices at which the supply of a good is matched to the demand for that good. The story is that when prices are high, manufacturers will scramble to produce that good in order to sell it for a large profit, which will drive down the price for the good through competition. Similarly, when prices are low, demand will be very high, since you can get a lot of utility out of consuming the good in comparison to keeping the small amount of money. This can be expected to drive up prices. Through these competing effects pushing prices up and down, it is hoped that prices will equilibriate at some fixed value through a process of tâtonnement .

The critical aspect that will make this model an interesting example for solving nonlinear equations is a substitution effect . To get more specific, suppose that when the prices for all $n$ goods are given by $p \in \R^n$, the supply for good $i$ is described by a function $\mathcal{S}_i(p) = S_i(p_i)$ and the demand for this good is \[\mathcal{D}_i(p) = D_i(p_i) + T_i(p_{-i}).\] What this latter equation says is that there is some nominal demand $D_i(p_i)$ given by a scalar function of the price of that good, as well as the functions $T_i(p_{-i})$ which serves to model the substitution effect: the demand for good $i$ also depends upon the prices of other goods in the economy. A set of equilibrium prices $p^\star \in \R^n$ (which may not be unique!) in this economy are prices such that \[\mathcal{S_i}(p^\star) - \mathcal{D}_i(p^\star) = 0\ \forall i \in [n],\] a nonlinear equation.

To make this more concrete, suppose we have an economy with two goods: coffee and tea. As the price of coffee increases, demand for coffee also falls. But, the demand for tea rises, since some coffee drinkers that are tightening their belt and drinking less coffee, start drinking more tea. Let’s write this all down mathematically. For the demand I’ll use functions $D_i(p_i) = \frac{k_i}{1 + p_i}$, modelling the fact that demand should be decreasing toward zero as price increases, and that there is some maximum demand $k_i$ when the good is free (I don’t think it’s possible to consume an infinite amount of tea or coffee! ☕). Picking another perfectly good function, I’ll model the substitution effect with a hyperbolic tangent: $T_i(p_{-i}) = c_i \mathsf{tanh}(p_{-i})$. This function has a plateau at $c_i$, as $p_{-i} \rightarrow \infty$, so the ratio $c_c / k_\tau$ is measuring the proportion of tea drinkers that would switch to coffee as the price of tea increases. On the supply side, I’ll use a nice S-shaped function $S_i(p_i) = \frac{m_i p_i}{1 + p_i}$ where the maximum producible amount of the good reaches another plateau at $m_i$. This results in the nonlinear system of equations (using $c$ for coffee and $\tau$ for tea):

\begin{equation} \notag \begin{aligned} \text{coffee:}\quad& \frac{m_c p_c}{1 + p_c} - \frac{k_c}{1 + p_c} - c_c \mathsf{tanh}(p_\tau) = 0,\\ \text{tea:}\quad& \frac{m_\tau p_\tau}{1 + p_\tau} - \frac{k_\tau}{1 + p_\tau} - c_\tau \mathsf{tanh}(p_c) = 0. \end{aligned} \end{equation}

Solving these equations individually for $p_c, p_\tau$ we obtain a nonlinear Jacobi algorithm:

\begin{equation} \notag \begin{aligned} p_c(t + 1) &= \frac{k_c + c_c \mathsf{tanh}(p_\tau(t))}{m_c - c_c \mathsf{tanh}(p_\tau(t))}\\ p_\tau(t + 1) &= \frac{k_\tau + c_\tau \mathsf{tanh}(p_c(t))}{m_\tau - c_\tau \mathsf{tanh}(p_c(t))}. \end{aligned} \end{equation}

In order for this to constitute a locally diagonally dominant system, we need to check the derivatives. It is a fairly straightforward calculation which, after cancelling some terms, results in the requirements \[\frac{m_i + k_i}{(1 + p_i^\star)^2} > c_i(1 - \mathsf{tanh}^2 (p_{-i}^\star)).\] The idea of “diagonal dominance” is clear in this equation: $m_i, k_i$ are coefficients controlling the importance of the “diagonal” part of the function, with $c_i$ controlling the coupling. If the coupling is weak: $m_i + k_i \gg c_i$, then it can be expected that the system is diagonally dominant, and an equilibrium should be close to the “nominal” equilibrium $p_i^\star \approx k_i / m_i$. Incidentally, this should give us a decent starting point for initializing Jacobi iteration.

Here’s a generic implementation of nonlinear Jacobi iteration:

from typing import List, Callable


def jacobi_iteration(x, f_inv: List[Callable], g: List[Callable]):
    x_next = np.zeros_like(x)
    all_i = np.arange(len(x))
    for i, (f_inv_i, g_i) in enumerate(zip(f_inv, g)):
        x_next[i] = f_inv_i(-g_i(x[all_i != i]))
    return x_next


def jacobi_flow(x, f_inv: List[Callable], g: List[Callable]):
    return jacobi_iteration(x, f_inv, g) - x


def solve_nonlinear_system(
    F: Callable,
    f_inv: List[Callable],
    g: List[Callable],
    x0=None,
    eps=1e-6,
    maxiter=np.inf,
    D=0
):
    """Solves the nonlinear system F_i(x) = f_i(x_i) + g_i(x_{-i}) = 0.

    The algorithm's starting point is x0.  The algorithm is only guaranteed
    to converge if F is locally diagonally dominant around an equilibrium, and
    x0 is chosen nearby that equilibrium point.
    """
    x = np.array(x0) if x0 is not None else np.zeros(len(g))
    _iter = partial(jacobi_iteration, f_inv=f_inv, g=g)
    all_i = np.arange(len(x))

    it = 0
    while it < maxiter and np.linalg.norm(F(x)) > eps:
        x = D * x + (1 - D) * _iter(x)
        it += 1
    return x

The code follows a functional programming style that mimics the mathematical description as closely as possible. A particular implementation of our coffee-tea economy, including useful functions for plotting the progress of the algorithm, might look something like this:

from math import tanh
from functools import partial


def coffee_tea_solver(k, m, c, D=0.0):
    """A particular example with coffee and tea..."""

    def F(p):
        x0 = (m[0] * p[0] - k[0]) / (1 + p[0]) - c[0] * tanh(p[1])
        x1 = (m[1] * p[1] - k[1]) / (1 + p[1]) - c[1] * tanh(p[0])
        return (x0, x1)

    def f_inv(z, _k, _m):
        return (_k + z) / (_m - z)

    def g(p, _c):
        return -_c * tanh(p)

    f_invs = [
        partial(f_inv, _k=k[0], _m=m[0]),
        partial(f_inv, _k=k[1], _m=m[1]),
    ]
    gs = [partial(g, _c=c[0]), partial(g, _c=c[1])]

    _jacobi_iterator = partial(jacobi_iteration, f_inv=f_invs, g=gs)
    _jacobi_flow = partial(jacobi_flow, f_inv=f_invs, g=gs)
    _solver = partial(
        solve_nonlinear_system,
        F=F,
        f_inv=f_invs,
        g=gs,
        eps=1e-6,
        maxiter=1000,
        D=D
    )
    return _jacobi_iterator, _jacobi_flow, _solver, F

Here’s a plot of the flow of this system for the parmeter settings

\begin{equation} \begin{aligned} &k_c = 3/2, k_\tau = 2\\ &m_c = 5/3, m_\tau = 1/2\\ &c_c = 4/3, c_\tau = 1/3, \end{aligned} \end{equation}

which, at least intuitively, results in a system corresponding to our intuition of diagonal dominance. Increasing the values of $c_c, c_\tau$ results in a breakdown of the algorithm and a failure to converge, either as a result of domain errors, or running to infinity. The SOR technique described earlier can potentially be used to increase the domain of convergence, but that is a matter for experimentation.

For this particular coffee-tea system, there is surely a lot more analysis that one could do in order to determine clearer convergence criteria, the ranges of parameters that will result in convergence, etc. Such analysis is worthwhile in actual practice, but we’re here for fun 🙃.

Conclusion

Our main point is in understanding the Jacobi iteration algorithm for solving systems of equations. The key assumption for this algorithm is that the system be diagonally dominant. Intuitively, this means that the equation associated to the $i^{th}$ variable be only weakly coupled with the remaining variables $x_{-i}$. We’ve seen how to formalize this in the case of linear equations, and how similar “rough guide” criteria can be obtained from the Hartman-Grobman Theorem in the nonlinear case.

In actual practice, Jacobi iteration is not likely to be the best nonlinear equation solver to use, though it depends upon the problem domain. The fact that it breaks down a system of equations into the parallel solution of univariate equations (for which completely general black-box algorithms like the Bisection Method are available) gives it a rather large potential domain of applicability. If you find yourself confronted with the need to solve a large system of nonlinear equations, available only through slow and black-box evaluation, and where the equations can be expected to be weakly coupled, Jacobi iteration might be for you 🤞.

Introduction

Fri, 09 Dec 2022 00:00:00 -0800

The title of this blog, Quant out of Water, is more-or-less the first thing that came to mind. However, as I (as of 2022) work as a quant at an hedge fund, and I wanted to write a blog that was not explicitly about finance, this title reflects that motivation. I produced the fish with money using a stable diffusion model. I hope you enjoy some of my writings.

You can learn more on the “about me” page.

Quant Out of Water

Constraint Programming, Puzzles, ILP, and z3

Introduction

Constraint Programming

Integer Linear Programming

Satisfiability Modulo Theories

Solving the Vanilla 24-7 Puzzle

Solving the 4-in-1 Puzzle

Conclusion

All About Quadratic Forms

Introduction

From Whence?

Completing the Square

Optimization

Convexity

Minimizing Quadratic Forms

The Shape of a Quadratic Form

Subspace Constraints

Convex and Concave Subspaces

Linear Algebra

Schur Complements

Partial Minimization

Definiteness Conditions

Square Roots and Cholesky Factorization

Probability and Statistics

Computations with the Mean and the Variance

Sampling

Linear Minimum Mean Squared Error Estimation

Linear Regression

Conclusion

Generalized Eigenvalue Problems and Trace Optimization

Introduction

Spectral Theory

The Spectral Theorem

The Courant-Fischer Variational Characterization

Generalized Eigenvalues

Interpretations and Intuition

Applications and Computations

Principle Component Analysis

Weighted Principle Component Analysis

Rayleigh Quotients and Fisher’s Linear Discriminant

Example

Optimizing Signal to Noise Ratio

Conclusion

Nature's Dartboard: The Axioms of Probability

Mathematics and Axiomatization

The Axioms of Probability

Samples and Events

Nature’s Dartboard

The Banach-Tarski Paradox: Volume and Measure

Random Variables

Expectation

Data and Sampling – the Passage to Statistics

Independent and Identically Distributed Random Variables

Conclusion

Group Theory, Symmetry, and a Brain Teaser

Introduction

A Mathematical Preamble

Equivalence Classes

Functions, Permutations, and Groups

Cycles and Decomposition

Solving the Puzzle

Representation and Rotations

Solved States

Three More Equivalence Classes

More Permutations on \(S\)

Transition Diagramming

A Solution

Computational Generalizations

Uncertain Breadth First Search

Conclusion

Lyapunov Stability, Linear Systems, and Semidefinite Programming

Introduction

References

Linear Dynamical Systems

Examples

Gradient Descent

Electronic Circuits

Semidefinite Programming

Lyapunov Stability Theory