To use ordinary least squares to find the line
y
=
m
x
+
b
that minimizes the distance in the y coordinate
We begin with a set of n points given by
(
x
0
,
y
0
)
,
(
x
1
,
y
1
)
,
(
x
2
,
y
2
)
,
…
,
(
x
n
-
1
,
y
n
-
1
)
To find the least squares line, we want to minimize this function
F
(
m
,
b
)
=
∑
i
=
0
n
-
1
[
y
i
-
(
m
x
i
+
b
)
]
2
To minimize the function, we need the partial derivatives to be zero.
∂
∂
b
F
(
m
,
b
)
=
-
2
∑
i
=
0
n
-
1
[
y
i
-
(
m
x
i
+
b
)
]
∂
∂
m
F
(
m
,
b
)
=
-
2
∑
i
=
0
n
-
1
x
i
[
y
i
-
(
m
x
i
+
b
)
]
This can be rewritten as the following system of equations
b
n
+
m
∑
i
=
0
n
-
1
x
i
=
∑
i
=
0
n
-
1
y
i
b
∑
i
=
0
n
-
1
x
i
+
m
∑
i
=
0
n
-
1
x
i
2
=
∑
i
=
0
n
-
1
x
i
y
i
Writing this as a matrix equation, we have
[
a
1
1
a
1
2
a
2
1
a
2
2
]
[
b
m
]
=
[
c
1
c
2
]
where the labels correspond to the values in the equations (i.e.
a
1
1
=
n
)
The solution is given by finding the inverse
[
b
m
]
=
1
a
1
1
a
2
2
-
a
1
2
a
1
2
[
a
2
2
-
a
1
2
-
a
2
1
a
1
1
]
[
c
1
c
2
]
This gives the solution
b
=
a
2
2
c
1
-
a
1
2
c
2
a
1
1
a
2
2
-
a
1
2
a
1
2
=
∑
i
=
0
n
-
1
x
i
2
∑
i
=
0
n
-
1
y
i
-
∑
i
=
0
n
-
1
x
i
∑
i
=
0
n
-
1
x
i
y
i
n
∑
i
=
0
n
-
1
x
i
2
-
(
∑
i
=
0
n
-
1
x
i
)
2
m
=
a
1
1
c
2
-
a
2
1
c
1
a
1
1
a
2
2
-
a
1
2
a
1
2
=
n
∑
i
=
0
n
-
1
x
i
y
i
-
∑
i
=
0
n
-
1
x
i
∑
i
=
0
n
-
1
y
i
n
∑
i
=
0
n
-
1
x
i
2
-
(
∑
i
=
0
n
-
1
x
i
)
2
For the homogeneous equation
A
x
+
B
y
+
C
=
0
, we have
A
=
∑
i
=
0
n
-
1
x
i
∑
i
=
0
n
-
1
y
i
-
n
∑
i
=
0
n
-
1
x
i
y
i
B
=
n
∑
i
=
0
n
-
1
x
i
2
-
(
∑
i
=
0
n
-
1
x
i
)
2
C
=
∑
i
=
0
n
-
1
x
i
∑
i
=
0
n
-
1
x
i
y
i
-
∑
i
=
0
n
-
1
x
i
2
∑
i
=
0
n
-
1
y
i
Using vector notation and setting
1
→
=
(
1
,
1
,
…
1
)
, we have
A
=
(
1
→
⋅
x
→
)
(
1
→
⋅
y
→
)
-
(
1
→
⋅
1
→
)
(
x
→
⋅
y
→
)
B
=
(
1
→
⋅
1
→
)
(
x
→
⋅
x
→
)
-
(
1
→
⋅
x
→
)
(
1
→
⋅
x
→
)
C
=
(
1
→
⋅
x
→
)
(
x
→
⋅
y
→
)
-
(
x
→
⋅
x
→
)
(
1
→
⋅
y
→
)