DS&AData Structures and Algorithmshttp://dsaa.werp.site/feed2018-05-14T15:00:00+00:00Exercises: Algorithm Correctness<p>Before attempting these exercises, you should read the posts on <a href="/post/is-your-algorithm-correct/">algorithm correctness</a>, <a href="/post/invariants/">invariants</a> and <a href="/post/variants/">variants</a>.</p><!-- pagebreak --><h3>Exercises</h3><ol class="exercises"><li>In English, write down the preconditions (if any) and postconditions of the following simple algorithms:
<ol><li><a href="/post/algorithmic-plans/#best-so-far">Search for the maximum value</a>
</li><li><a href="/post/search-algorithms/#linear-search">Linear search</a>
</li><li><a href="/post/search-algorithms/#binary-search">Binary search</a>
</li></ol>
</li><li>Write as many as possible of the preconditions and postconditions in 1. as assertions in Python.
</li><li class="language-python"> The algorithms below take integer inputs.
For each algorithm,
<ul><li>Test enough inputs to discover the preconditions and postconditions, and write them as <code>assert</code> statements.
</li><li>Write <code>print(locals())</code> before the loop and at the end of the loop body, to <a href="/post/invariants/#tracing">trace</a> the execution.
</li><li>Use your traces to find the loop invariant, and add <code>assert</code> statements to check it.
</li><li>Use your traces to find the loop variant.
</li></ul>
Algorithm A:
<pre class="code-listing"><code>def algorithm_a(x, y):
p = x
q = y
while q > 0:
p += 1
q -= 1
return p</code></pre>
Algorithm B:
<pre class="code-listing"><code>def algorithm_b(x, y):
p = x
q = y
while q > 0:
p -= 1
q -= 1
return p</code></pre>
Algorithm C:
<pre class="code-listing"><code>def algorithm_c(x, y):
p = 0
i = 0
while i < y:
p += x
i += 1
return p</code></pre>
</li></ol>
http://dsaa.werp.site/post/exercises-algorithm-correctness/2018-05-14T15:00:00+00:00Variants<p>An algorithm is <a href="https://en.wikipedia.org/wiki/Correctness_(computer_science)">correct</a> if it always returns the correct result.</p><p><a href="/post/is-your-algorithm-correct/#assertions">Assertions</a> are our main tool for proving algorithms are correct; they state that some condition will be true whenever a particular line of code is reached.
The goal is to assert that, <a href="https://en.wikipedia.org/wiki/Postcondition">when the algorithm finishes running</a>, the result is correct.
If this assertion never fails, then the algorithm will never return a wrong result.</p><!-- pagebreak --><h4>An unfortunate problem</h4><div class="language-python"><p>
Consider the following sorting algorithm; assume the <code>is_sorted</code> function tests whether <code>lst</code> is sorted or not.</p><pre class="code-listing"><code>def infinite_loop_sort(lst):
while not is_sorted(lst):
# do nothing
pass</code></pre><p>This loop does nothing!
Certainly, it makes no progress towards sorting the list.
Yet we can be absolutely sure it never returns a wrong result, because it never returns a result at all!</p><p>This is really a problem.
If we write a <code>return</code> statement, and <code>assert</code> that the result is correct, the techniques we developed in <a href="/post/invariants/#using-the-loop-condition">the previous post</a> actually allow us to “prove” that this clearly incorrect algorithm is “correct”:</p><pre class="code-listing"><code>def infinite_loop_sort(lst):
while not is_sorted(lst):
# do nothing
pass
assert is_sorted(lst)
return lst</code></pre><p>The “proof” is as follows.
At the end of the loop, the assertion cannot fail, because if <code>is_sorted(lst)</code> is false, the loop would not have stopped.
So, the algorithm never returns a wrong result.
</p></div><h4>Termination</h4><p>What we want to prove is that the algorithm always returns a correct result.
But assertions only let us say that <em>if</em> the algorithm returns a result, then the result will be correct.﻿<a class="ptr">(1)</a></p><p>We also need to prove that the algorithm will return a result at all — i.e. that it will eventually <a href="https://en.wikipedia.org/wiki/Termination_analysis">terminate</a>, or finish running.
In this post, we’ll explore the idea of <a href="https://en.wikipedia.org/wiki/Loop_variant">variants</a>, which are needed to prove that algorithms terminate.
The good news is that variants are simpler and easier to use than <a href="/post/invariants/">invariants</a>.</p><h4>Analogy</h4><p>In the game of <a href="https://en.wikipedia.org/wiki/Chess">chess</a>, a <a href="https://en.wikipedia.org/wiki/Pawn_(chess)">pawn</a> can only move forwards.
If you keep moving the same pawn, eventually it must reach the end of the board.</p>
<script type="text/javascript" src="http://dsaa.werp.site/js/chess/invariants.js"></script>
<link rel="stylesheet" property="stylesheet" href="/js/chess/chess.css">
<div id="chess_pawn" class="chess-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<p>The <em>y</em> coordinate gets smaller on every move, so it must eventually reach 0.
This proves that the pawn will reach the end of the board in finitely many moves — it can’t get stuck in an “infinite loop”.</p><p>In other words, the <em>y</em> coordinate is a <a href="https://en.wikipedia.org/wiki/Loop_variant">variant</a>.</p><h4>Example: factorial</h4><p>The following algorithm computes the <a href="https://en.wikipedia.org/wiki/Factorial">factorial</a> of <code>n</code>:</p><pre class="code-listing language-python"><code>def factorial(n):
if n < 0:
raise ValueError('n must not be negative')
r = 1
while n > 0:
r *= n
n -= 1
return r</code></pre><p>In this loop, <code>n</code> is a variant — it gets smaller on each iteration, and the loop stops when <code>n</code> is 0.
Therefore, this algorithm will always terminate.</p><h4>Example: insertion sort</h4><p>Consider the <a href="/post/iterative-sorting/#insertion-sort">insertion sort</a> algorithm.</p><pre class="code-listing language-python"><code>def insertion_sort(lst):
n = len(lst)
for i in range(n):
j = i
while j > 0 and lst[j-1] > lst[j]:
tmp = lst[j]
lst[j] = lst[j-1]
lst[j-1] = tmp
j -= 1</code></pre><div class="language-python"><p>
The inner <a href="https://en.wikipedia.org/wiki/While_loop">while loop</a> must terminate, because <code>j</code> gets smaller on each iteration, and the loop terminates when <code>j</code> is 0.
If the other condition <code>lst[j-1] > lst[j]</code> is false, then the loop might terminate sooner — but it will terminate.
</p></div><p>So, <code>j</code> is a loop variant.</p><h4>Example: exponentiation</h4><p>In <a href="/post/invariants/">the previous post</a> we proved that the following algorithm never returns a wrong result:</p><pre class="code-listing language-python"><code>def power(x, n):
if n < 0:
raise ValueError('n must not be negative')
r = 1
i = 0
while i < n:
r *= x
i += 1
return r</code></pre><p>Let’s also prove that the loop must terminate.
This example is a bit different, since there is no variable “counting down”, and the loop stops when <code>i</code> reaches <code>n</code>, not when something reaches 0.</p><div class="language-python"><p>
This loop terminates because <code>i</code> keeps getting <em>bigger,</em> so it must eventually reach <code>n</code>.
Put another way, the difference <code>n - i</code> keeps getting smaller.
For example, consider the <a href="/post/invariants/#tracing">trace</a> from computing <code>power(3, 4)</code>:
</p></div><div style="display:flex; flex-direction:column; align-items:center;"><table class="tex-tbl"><style scoped type="text/css">td:nth-child(1){text-align:center;}td:nth-child(2){text-align:center;}tr>*:nth-child(2){border-right-width:1px;border-right-style:solid;}td:nth-child(3){text-align:center;}</style><thead><tr><th>n </th><th> i </th><th> n - i
</th></tr></thead><tbody><tr><td>4 </td><td> 0 </td><td> 4 </td></tr><tr><td>
4 </td><td> 1 </td><td> 3 </td></tr><tr><td>
4 </td><td> 2 </td><td> 2 </td></tr><tr><td>
4 </td><td> 3 </td><td> 1 </td></tr><tr><td>
4 </td><td> 4 </td><td> 0
</td></tr></tbody></table></div><div class="language-python"><p>
The expression <code>n - i</code> gets smaller on every iteration, because <code>i</code> gets bigger but <code>n</code> stays the same.
When <code>n - i</code> reaches 0, the condition <code>i < n</code> becomes false and so the loop terminates.</p><p>So, <code>n - i</code> is a loop variant.
</p></div><h4>Example: binary search</h4><p>Consider the <a href="/post/search-algorithms/#binary-search">binary search</a> algorithm:</p><div class="language-python"><pre class="code-listing"><code>def binary_search(lst, target):
left = 0
right = len(lst) - 1
while left <= right:
middle = (left + right) // 2
value = lst[middle]
if value == target:
return middle
elif value < target:
left = middle + 1
else:
right = middle - 1
return -1</code></pre><p>This example is a bit more complicated, because there’s no variable counting up or down to a fixed limit.</p><p>Intuitively, binary search should terminate because the “range of possibilities” keeps getting smaller — this range is from <code>left</code> to <code>right</code>, so <code>right - left</code> could be a loop variant.</p><p>On each iteration, one of three things happens:
</p><ul><li>The algorithm might terminate immediately — so, no problem.
</li><li>Or <code>left</code> becomes <code>middle + 1</code>.
</li><li>Or <code>right</code> becomes <code>middle - 1</code>.
</li></ul><p><a href="/post/invariants/#using-the-loop-condition">The loop condition</a> is <code>left <= right</code>, and <code>middle</code> is halfway between them.
So, either <code>left</code> gets bigger, or <code>right</code> gets smaller — either way, <code>right - left</code> gets smaller.</p><p>To prove the loop terminates, we need to show that <code>left <= right</code> eventually becomes false.
This happens when <code>right - left</code> reaches −1, so the proof is complete.﻿<a class="ptr">(2)</a>
</p></div><h4>Loop variants</h4><p>After these examples, hopefully it’s clear what <em>loop variants</em> are, and how we can use them to prove algorithms terminate.
But let’s also give a proper definition.</p><p>A <a href="https://en.wikipedia.org/wiki/Loop_variant">loop variant</a> is an <a href="https://en.wikipedia.org/wiki/Expression_(computer_science)">expression</a> which:</p><ul><li>Has an <a href="https://en.wikipedia.org/wiki/Integer_(computer_science)">integer</a> value.
</li><li>Gets at least 1 smaller on every iteration of the loop.
</li><li>When it gets below some threshold (usually 0), the loop condition becomes false.
</li></ul><p>Any loop with a variant is guaranteed to terminate, because an integer can’t keep getting smaller forever without falling below the threshold.
Once it reaches the threshold, the loop condition becomes false, so the loop terminates.</p><h4>A note on ‘for’ loops</h4><p>In this post we’ve only analysed <a href="https://en.wikipedia.org/wiki/While_loop">while loops</a>.
On the other hand, many algorithms use <a href="https://en.wikipedia.org/wiki/For_loop">for loops</a>:</p><div class="language-python"><pre class="code-listing"><code>for i in range(n):
# ...</code></pre><p>If a <code>for</code> loop iterates over a finite collection, such as a list, set or dictionary, or a finite range of integers, then the number of iterations is strictly limited to the size of that collection or range, so it can’t be an infinite loop.﻿<a class="ptr">(3)</a></p><p>If necessary, we can still use variants to prove that a <code>for</code> loop terminates, by writing an equivalent <code>while</code> loop.
This example iterates over the same range, and has the variant <code>n - i</code>.</p><pre class="code-listing"><code>i = 0
while i < n:
# ...
i += 1</code></pre></div><h4>Footnotes</h4><ol id="footnotes"><li>Fundamentally, an assertion says that <em>when</em> a line of code is reached, the condition will be true.
There is no way to assert that a line of code <em>will</em> be reached.</li><li>The variant is guaranteed to reach −1 for the same reason it’s guaranteed to reach 0.
Alternatively, we could choose <code>right - left + 1</code> as a variant.</li><li>This assumes the size doesn’t change during the loop.
It’s generally a mistake to make a collection larger while you’re iterating over it with a <code>for</code> loop, and many languages will raise an error if they detect this happening.</li></ol>
http://dsaa.werp.site/post/variants/2018-05-14T12:00:00+00:00Invariants<p>An <a href="/post/is-your-algorithm-correct/#assertions">assertion</a> is a claim about the state of a running program — it says that some <a href="/post/booleans/">logical condition</a> should be true whenever a particular line of code is reached.
In <a href="/post/is-your-algorithm-correct/">the previous post</a>, we only considered assertions in code where each line is executed once, in sequence.</p><p>Of course, real algorithms are more complicated, because <a href="https://en.wikipedia.org/wiki/Control_flow#Loops">loops</a> and <a href="https://en.wikipedia.org/wiki/Recursion_(computer_science)">recursion</a> cause some code to be executed multiple times, not in a simple sequence.
In this post we’ll explore <a href="https://en.wikipedia.org/wiki/Invariant_(computer_science)">invariants</a>, which are needed to prove correctness of non-trivial algorithms.</p><!-- pagebreak --><h4>Analogy</h4><p>Invariants are a very abstract idea, and can be difficult to understand.
We’ll start with an analogy unrelated to algorithms: <a href="https://en.wikipedia.org/wiki/Bishop_(chess)">bishops</a> in the game of <a href="https://en.wikipedia.org/wiki/Chess">chess</a>.
Don’t worry if you don’t play chess — all you need to know is that bishops move diagonally.</p>
<script type="text/javascript" src="http://dsaa.werp.site/js/chess/invariants.js"></script>
<link rel="stylesheet" property="stylesheet" href="/js/chess/chess.css">
<div id="chess_bishop" class="chess-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<p>However many moves you make, this bishop will always be on a white square.
This is because:
</p><ul><li>It starts on a white square, and
</li><li>Any move from a white square lands on a white square.
</li></ul><p>
The <a href="/post/is-your-algorithm-correct/#assertions">assertion</a> that “the bishop is on a white square” is called an <em>invariant</em>.
It remains true even though the bishop’s position can change.</p><p>An invariant is an assertion — a statement of fact.
For our purposes, the colour of the bishop’s square is <strong>not</strong> an invariant, because although it does stay the same, the square colour is white or black, not true or false.</p><h4>Induction</h4><p>We said the bishop starts on a white square, and any move from a white square lands on another white square — therefore it will still be on a white square <em>however many moves you make</em>.</p><p>This last part of the argument is subtle.
We said the invariant is preserved after <em>one</em> move, therefore it’s preserved after <em>any number of</em> moves.
This kind of argument is called <a href="https://en.wikipedia.org/wiki/Mathematical_induction">induction</a>.</p><p>This is less obvious than it might seem:
</p><ul><li>I am in the library.
</li><li>If I walk one step, I will still be in the library.
</li><li>Therefore, I am stuck in the library forever.
</li></ul><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/trollface.png">
</p></div><p>This is clearly nonsense, but it seems like the same kind of argument.</p><p>The problem is the second part.
It may be true that from my current location, the <em>first</em> step is not enough to exit the library.
However, it’s certainly possible that I could later go from inside the library to outside by walking one step.</p><p>On the other hand, the bishop can never move from a white square to a black square, not on the first move or any later move; so it really <em>is</em> stuck on white squares.</p><h4>Example</h4><div class="language-python"><p>
Let’s see an example of an invariant in an algorithm.
Consider the following simple function which calculates <code>x ** n</code> by multiplication:</p><pre class="code-listing"><code>def power(x, n):
r = 1
i = 0
while i < n:
r *= x
i += 1
return r</code></pre><p>The algorithm uses a loop to multiply <code>r</code> by <code>x</code> the right number of times.
The variable <code>i</code> counts how many multiplications have been made so far; at each step <code>r</code> should equal <code>x ** i</code>.</p><p>So, even though <code>i</code> and <code>r</code> change in value, the assertion <code>r == x ** i</code> is an <em>invariant</em> of this loop.
</p></div><h4 id="tracing">Tracing</h4><div class="language-python"><p>
One way to see how an algorithm works is to <a href="https://en.wikipedia.org/wiki/Tracing_(software)">trace</a> it — that is, work through the execution step-by-step, and write down the values of each variable at each step.
We can do this with pen and paper, or by adding <code>print</code> statements to the code.﻿<a class="ptr">(1)</a></p><p>For example, the table below shows a trace of all four variables, at each step of computing <code>power(3, 4)</code>.
</p></div><div style="display:flex; flex-direction:column; align-items:center;"><table class="tex-tbl"><style type="text/css" scoped>td:nth-child(1){text-align:center;}td:nth-child(2){text-align:center;}td:nth-child(3){text-align:center;}td:nth-child(4){text-align:center;}tr>*:nth-child(4){border-right-style:solid;border-right-width:1px;}td:nth-child(5){text-align:center;}</style><thead><tr><th>x </th><th> n </th><th> r </th><th> i </th><th> r == x ** i
</th></tr></thead><tbody><tr><td>3 </td><td> 4 </td><td> 1 </td><td> 0 </td><td> True </td></tr><tr><td>
3 </td><td> 4 </td><td> 3 </td><td> 1 </td><td> True </td></tr><tr><td>
3 </td><td> 4 </td><td> 9 </td><td> 2 </td><td> True </td></tr><tr><td>
3 </td><td> 4 </td><td> 27 </td><td> 3 </td><td> True </td></tr><tr><td>
3 </td><td> 4 </td><td> 81 </td><td> 4 </td><td> True
</td></tr></tbody></table></div><div class="language-python"><p>
Before and after each iteration of the loop, the invariant <code>r == x ** i</code> remains true, even as the values of <code>r</code> and <code>i</code> change.</p><p>Tracing is also a good way to discover invariants, if you don’t already know what they should be.
You can identify this relationship between the variables by looking at the trace, just as you might notice by looking at the chess board that the bishop is always on a white square.
</p></div><h4>Loop invariants</h4><div class="language-python"><p>
We’ve identified that <code>r == x ** i</code> is a <a href="https://en.wikipedia.org/wiki/Loop_invariant">loop invariant</a>.
This means:
</p><ul><li>It’s true before the loop begins.
</li><li><em>If</em> it’s true before an iteration of the loop, <em>then</em> it will still be true afterwards.
</li><li>Therefore it’s true on every iteration, and will still be true when the loop stops.
</li></ul><p>Let’s annotate the loop with some assertions:</p><pre class="code-listing"><code>def power(x, n):
# the invariant must be true before the loop
r = 1
i = 0
assert r == x ** i
while i < n:
# if the invariant is true here,
assert r == x ** i
r *= x
i += 1
# then it's still true here
assert r == x ** i
return r</code></pre><p>Before the loop, <code>x ** i</code> is just <code>x ** 0</code>.
<a href="https://en.wikipedia.org/wiki/Exponentiation#Zero_exponent">This is 1</a>, and <code>r</code> is also 1, so the invariant is true to begin with.﻿<a class="ptr">(2)</a></p><p>We also need to show that the loop preserves the invariant.
The loop multiplies <code>r</code> by <code>x</code>, and increases <code>i</code> by 1.
So, the new value of <code>r</code> is <code>(x ** i) * x</code>, and the new value of <code>x ** i</code> is <code>x ** (i + 1)</code>.
<a href="https://en.wikipedia.org/wiki/Exponentiation#Identities_and_properties">These are equal</a>, so the invariant is preserved.</p><p>Therefore the invariant will remain true, however many times the loop iterates.
</p></div><h4>More invariants</h4><div class="language-python"><p>
What else can we prove about this loop?</p><p>The loop continues while <code>i < n</code>, and <code>i</code> increases by 1 on each iteration.
In some sense, it is obvious that the loop stops when <code>i == n</code>.
But how do we actually prove this?
It turns out we need another loop invariant.</p><p>Certainly, <code>i == n</code> isn’t an invariant; it’s not even true before the loop.
On the other hand, by looking at the trace you may notice something:
</p></div><div style="display:flex; flex-direction:column; align-items:center;"><table class="tex-tbl"><style type="text/css" scoped>td:nth-child(1){text-align:center;}td:nth-child(2){text-align:center;}</style><thead><tr><th>n </th><th> i
</th></tr></thead><tbody><tr><td>4 </td><td> 0 </td></tr><tr><td>
4 </td><td> 1 </td></tr><tr><td>
4 </td><td> 2 </td></tr><tr><td>
4 </td><td> 3 </td></tr><tr><td>
4 </td><td> 4
</td></tr></tbody></table></div><div class="language-python"><p>
Don’t think too hard; the simple relationship between these variables is <code>i <= n</code>.</p><pre class="code-listing"><code>while i < n:
# if the invariant is true here,
assert i <= n
r *= x
i += 1
# then it's still true here
assert i <= n</code></pre><p>Let’s check whether <code>i <= n</code> really is an invariant.
</p><ul><li>Before the loop, <code>i = 0</code>, so the invariant holds if <code>0 <= n</code>.
</li><li>For the loop to make an iteration, the condition <code>i < n</code> must be true.
The loop increases <code>i</code> by 1, but if <code>i < n</code> then <code>i + 1 <= n</code>.
So the invariant is preserved.
</li></ul><p>There are two points arising from this proof; both are worth addressing.
</p></div><h4>Discovering preconditions</h4><div class="language-python"><p>
The invariant must hold before the loop begins.
This means we need <code>0 <= n</code> to be true — but this is not guaranteed!</p><p>By trying to prove the invariant, we actually discovered a <a href="https://en.wikipedia.org/wiki/Precondition">precondition</a> for our algorithm — <code>n</code> must not be negative.
To avoid incorrect behaviour, we should reject the input if it’s invalid:</p><pre class="code-listing"><code>def power(x, n):
if n < 0:
raise ValueError('n must not be negative')
assert n >= 0
# ...</code></pre><p>By raising an error if <code>n</code> is negative, we can safely assert <code>n >= 0</code>.
Now the invariant <code>i <= n</code> is guaranteed to hold before the loop, because <code>i = 0</code>.
</p></div><h4 id="using-the-loop-condition">Using the loop condition</h4><div class="language-python"><p>
To prove that the loop preserves the invariant <code>i <= n</code>, we had to show that <em>if</em> it was true at the start of an iteration, then it would still be true afterwards.</p><p>This worked because at the start of an iteration, we can assert <code>i < n</code>.
This assertion can never fail, because otherwise the loop would not have made another iteration.﻿<a class="ptr">(3)</a></p><pre class="code-listing"><code>while i < n:
assert i < n
# ...
assert i >= n</code></pre><p>We can also assert <code>i >= n</code> when the loop stops.
This assertion can’t fail; if it were false, the loop wouldn’t have stopped.﻿<a class="ptr">(4)</a></p><p>This is enough to prove that <code>i == n</code> after the loop:
</p><ul><li>The loop invariant <code>i <= n</code> must be true after the loop.
</li><li>The assertion <code>i >= n</code> is also true after the loop.
</li><li>The only way this can happen is if <code>i == n</code>.
</li></ul></div><p>This may seem like a lot of effort to prove something obvious; on the other hand, although it’s a simple example, it demonstrates some important principles and techniques.</p><h4>Conclusion</h4><div class="language-python"><p>
Using invariants, we proved that <code>r == x ** i</code> and <code>i == n</code> are <a href="https://en.wikipedia.org/wiki/Postcondition">postconditions</a> of this loop.
Together, these imply <code>r == x ** n</code>, meaning the algorithm returns the correct result.
</p></div><p>We also fixed a bug by discovering a precondition for the algorithm to be correct; <code>n</code> must not be negative.
This isn’t unusual; many bugs can be discovered by testing, but if we hadn’t tried proving the algorithm correct, we might never have thought about negative inputs.</p><h4>Footnotes</h4><ol id="footnotes"><li>In Python, <span class="language-python"><code>print(locals())</code></span> is a simple way to print the entire current state of a function.</li><li>There is a special case when <code>x</code> is zero; 0<sup>0</sup> is usually undefined.
However, in Python <code>0 ** 0</code> is 1, and this doesn’t cause any problems here.</li><li>A similar argument can be made for if/else statements:
<pre class="code-listing language-python"><code>if i < n:
assert i < n
# ...
else:
assert i >= n
# ...</code></pre></li><li>This argument requires that there are no <code>break</code> statements inside the loop.</li></ol>
http://dsaa.werp.site/post/invariants/2018-05-14T11:00:00+00:00Is Your Algorithm Correct?<p>When we write an algorithm to solve a <a href="/post/specifying-problems/">problem</a>, the algorithm is only <a href="https://en.wikipedia.org/wiki/Correctness_(computer_science)">correct</a> if it does actually solve that problem.
That should be obvious!
But how do we know whether an algorithm is correct?</p><p>This is a genuine concern because algorithms often <em>aren’t</em> correct, and we need to debug them.
We need a way to analyse an algorithm to decide if it really does solve the problem or not.</p><!-- pagebreak --><p>
So far, we’ve done this by <a href="/post/a-problem-solving-process/#testing">testing</a>.
By actually running the algorithm with particular inputs, we can see whether it gives the expected (correct) output.
If it doesn’t, that proves the algorithm is incorrect.
On the other hand, testing alone isn’t enough to be sure the algorithm is correct — only that the algorithm gives the correct output for the few inputs we tested.</p><h4>Example</h4><p>Consider the following simple algorithm which computes <code>x * x * x * x</code> with two multiplications instead of three:</p><pre class="code-listing language-python"><code>def fourth_power(x):
y = x * x
z = y * y
return z</code></pre><p>With some testing, we can confirm the output is correct in some test cases:</p><pre class="code-listing language-python"><code>>>> fourth_power(2)
16
>>> fourth_power(-3)
81
>>> fourth_power(0)
0</code></pre><p>This is evidence, but not proof, that the algorithm is correct.
To prove it, instead of trying some particular inputs, we can <a href="https://en.wikipedia.org/wiki/Symbolic_execution">imagine running the algorithm</a> without knowing the value of <code>x</code>:</p><ul><li>After the first line, <code>y</code> is equal to <code>x * x</code>.
</li><li>After the second line, <code>z</code> is equal to <code>y * y</code>.
</li><li>Therefore using both facts, <code>z</code> is equal to <code>(x * x) * (x * x)</code>, which is the correct result because the brackets don’t matter.
</li></ul><p>So, we have proved this algorithm is correct!</p><h4 id="assertions">Assertions</h4><p>The key technique we used was to state facts at particular points in the code, and use those to <a href="https://en.wikipedia.org/wiki/Deductive_reasoning">deduce</a> other facts at other points in the code.
These statements are called <a href="https://en.wikipedia.org/wiki/Assertion_(software_development)">assertions</a>, and we can actually make them in Python:</p><pre class="code-listing language-python"><code>def fourth_power(x):
y = x * x
assert y == x * x
z = y * y
assert z == y * y
assert z == x * x * x * x
return z</code></pre><p>Each <span class="language-python"><code>assert</code></span> statement makes a claim about the <a href="https://en.wikipedia.org/wiki/State_(computer_science)#Program_state">program state</a> at a particular point in its execution; “whenever this line is reached, this condition is true”.</p><p>The assertions are not part of the algorithm — think of them like <a href="https://en.wikipedia.org/wiki/Comment_(computer_programming)">comments</a> explaining why the algorithm is correct.
However, unlike comments, the computer will check to make sure assertions are true.</p><p>That has a cost in running time — this function now does seven multiplications instead of two — so normally we’d disable assertion-checking once we’ve finished testing the software.</p><p>If the algorithm <em>isn’t</em> correct, then an assertion will fail, raising an error.
For example, the following function is supposed to calculate <code>x ** 2</code>, but doesn’t:</p><pre class="code-listing language-python"><code>>>> def incorrect_square(x):
... y = x + x
... assert y == x ** 2
... return y
...
>>> incorrect_square(5)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in incorrect_square
AssertionError</code></pre><p>We can see that the assertion failed.
To make debugging easier, we could include a message:</p><pre class="code-listing language-python"><code>>>> def incorrect_square(x):
... y = x + x
... assert y == x ** 2, "{} ** 2 isn't {}".format(x, y)
... return y
...
>>> incorrect_square(5)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in incorrect_square
AssertionError: 5 ** 2 isn't 10</code></pre><h4>Preconditions and postconditions</h4><p>An assertion made before a section of code is a <a href="https://en.wikipedia.org/wiki/Precondition">precondition</a>, and an assertion made afterwards is a <a href="https://en.wikipedia.org/wiki/Postcondition">postcondition</a>.</p><div class="language-python"><p>
Consider the instruction <code>z = y * y</code> in the <code>fourth_power</code> algorithm:</p><pre class="code-listing"><code>assert y == x * x
z = y * y
assert z == x * x * x * x</code></pre><p>This instruction has a precondition and a postcondition.
Both are assertions, but they serve quite different roles in this section of code: we <em>assume</em> the precondition is true, and <em>deduce</em> that the postcondition is true.</p><p>The precondition, instruction and postcondition together are called a <a href="https://en.wikipedia.org/wiki/Hoare_logic#Hoare_triple">Hoare triple</a>.
Hoare triples are useful because they let us focus on each part of the code separately; we only have to show that <em>if</em> the precondition is true, <em>then</em> the postcondition will be true.</p><p>This works because the precondition of one section of code is the postcondition of another — in this case, <code>y == x * x</code> is the postcondition of <code>y = x * x</code>, so this assertion is clearly valid.</p><p>Likewise, the postcondition <code>z == x * x * x * x</code> is the precondition for the next instruction, <code>return z</code>.
</p></div><h4>Correctness</h4><p>For an algorithm to be correct, it must always return the correct output.
To prove this, we need to specify the output with an assertion — in this case, the problem was to compute <code>x * x * x * x</code>, so the postcondition <code>z == x * x * x * x</code> is exactly what’s required.</p><p>More generally, to prove an algorithm is correct, we must specify the correct output as the algorithm’s postcondition, then prove it really is a postcondition.</p><p>The algorithm as a whole may also have a precondition — for example, the <a href="/post/search-algorithms/#binary-search">binary search</a> algorithm only works if the input list is in order.</p><h4>Conclusion</h4><p>The purpose of an assertion is to make a claim about the program state, which should always be true when a particular line of the code is reached.
If an assertion fails, it indicates the code before it is incorrect.</p><p>Assertions are the basic components of correctness proofs.
By dividing an algorithm into sections of code, we can prove for each section that <em>if</em> the precondition is true, <em>then</em> the postcondition will also be true.</p><p>However, in this post we only saw how to do this for simple algorithms, where each section of code is executed in sequence.
In the next post we’ll see how to extend this reasoning to algorithms with loops.</p><div class="warning"><h4>Warning</h4><p>
Only assert things that <em>should always</em> be true.
Don’t use assertions to handle error conditions which really could happen.</p><div class="language-python"><p>
For example, if <code>get_user_data</code> returns <code>None</code> when the user doesn’t exist, never do this:</p><pre class="code-listing"><code>data = get_user_data(user)
assert data is not None, "User doesn't exist"
save_data_to_file(data)</code></pre><p>This is bad because if assertions are disabled, the computer won’t check them, so no error will be raised and the file could be corrupted.
If you want to raise an error, do it yourself:</p><pre class="code-listing"><code>data = get_user_data(user)
if data is None:
raise ValueError("User doesn't exist")
save_data_to_file(data)</code></pre></div></div><h4>Footnotes</h4><ol id="footnotes"></ol>
http://dsaa.werp.site/post/is-your-algorithm-correct/2018-05-14T10:00:00+00:00Exercises: Algorithm Efficiency<p>Before attempting these exercises, you should read the posts on <a href="https://dsaa.werp.site/post/how-fast-is-your-algorithm/">dynamic analysis</a> and <a href="https://dsaa.werp.site/post/static-analysis/">static analysis</a>.</p><!-- pagebreak --><p>
You will need the following two files:</p><ul><li><a href="/js/efficiency/sorting.py">sorting.py</a> — code for <a href="/post/iterative-sorting/#selection-sort">selection sort</a>, <a href="/post/iterative-sorting/#insertion-sort">insertion sort</a> and <a href="/post/recursive-sorting/#merge-sort">merge sort</a>.
</li><li><a href="/js/efficiency/timing.py">timing.py</a> — for measuring running times.
</li></ul><p>You should read both files carefully, to see what functions they provide.
To make use of these libraries, put both in the same directory, and run <strong>timing.py</strong> in Python’s interactive mode.</p><h3>Exercises</h3><ol class="exercises"><li class="language-python"> Use the following code to compare the three sorting algorithms for lists of length 1000:
<ul><li><code>time_algorithm(selection_sort, 1000, 200)</code>
</li><li><code>time_algorithm(insertion_sort, 1000, 200)</code>
</li><li><code>time_algorithm(merge_sort, 1000, 200)</code>
</li></ul>
Which algorithm is faster?
</li><li>Which algorithm is faster for lists of length 10?
What about length 100?
</li><li class="language-python"> Use the following code to measure the running time of selection sort for list lengths up to 2000, with 100 repetitions each.
This may take a few minutes to complete.
<ul><li><code>vary_list_length(selection_sort, 2000, 100, 100)</code>
</li></ul>
Copy the output into a spreadsheet, and plot a chart showing how the running time depends on the list length.
</li><li>Measure the running times for insertion sort and merge sort, for the same list lengths.
Plot your results for all three algorithms on the same chart.
Which algorithm is most efficient for long lists?
</li><li>Change the <code>generate_random_list</code> function to investigate the effect of other kinds of input.
Try:
<ul><li>Lists which are already in order.
</li><li>Lists which are completely out of order.
</li><li>Lists of strings, instead of numbers.
(You will need a way to generate random strings.)
</li></ul>
</li></ol>
http://dsaa.werp.site/post/exercises-algorithm-efficiency/2018-05-07T15:00:00+00:00Choose the Right Data Structures<p>In an earlier post we saw the importance of <a href="/post/choose-the-right-data-types/">choosing the right data types</a> when designing an algorithm.
But we don’t just choose a <a href="https://en.wikipedia.org/wiki/Data_type">type</a> — whether we’re aware of it or not, we are also choosing a <a href="https://en.wikipedia.org/wiki/Data_structure">data structure</a> which implements that type.</p><p>Both choices can have a big impact on how efficient an algorithm is; let’s consider some examples.</p><!-- pagebreak --><h3>Smallest missing integer</h3><p>Our first example is an exercise from <a href="https://app.codility.com/programmers/lessons/4-counting_elements/missing_integer/">Codility</a>:</p><ul><li><em>Problem:</em> given a list of integers, find the smallest positive integer that does not occur in it.
<ul><li><em>Input:</em> a list of integers.
</li><li><em>Output:</em> the smallest integer missing from the list.
</li></ul>
</li><li class="language-json"><em>Example:</em> for the list <code>[1, 2, 3]</code>, the correct output is <code>4</code>.
</li><li class="language-json"><em>Example:</em> for the list <code>[3, 1, 5, 4]</code>, the correct output is <code>2</code>.
</li></ul><p>This is a straightforward problem with a straightforward solution.
Unfortunately, the straightforward solution is a bad one!</p><h4>Bad solution</h4><p>We can solve this problem directly by testing if each number 1, 2, 3<span>…</span> is missing or not.
Since we try them in order, the first missing number we find must be the smallest one.
There is a subproblem of testing whether a number is missing from the list; fortunately, Python lists support the <code>not in</code> operator.
Here’s the solution:
</p><pre class="code-listing language-python"><code>def smallest_missing_integer(numbers):
i = 1
while True:
if i not in numbers:
return i
i += 1</code></pre><p>This solution is bad because it’s unnecessarily slow.</p><p>To see why, consider how the <code>not in</code> operator works; the only way to test if a number is missing from a list is by <a href="/post/search-algorithms/#linear-search-variations">linear search</a>.
That is, although Python makes <code>not in</code> look like a <a href="/post/static-analysis/#basic-operations">basic operation</a>, what it’s actually doing is more like this:</p><pre class="code-listing language-python"><code>def not_in(lst, target):
for value in lst:
if value == target:
return False
return True
def smallest_missing_integer(numbers):
i = 1
while True:
if not_in(numbers, i):
return i
i += 1</code></pre><p>This code works exactly the same way, but now it’s clear that <code>numbers</code> is being searched over and over again.
The <code>not_in</code> function is <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a>, but the algorithm calls it from inside a loop,﻿<a class="ptr">(1)</a>
so the overall complexity is <a href="/post/big-o-notation/#quadratic">O(<i>n</i><sup>2</sup>)</a>.
Codility will reject this solution, because it’s not efficient enough.</p><h4>Good solution</h4><p>The problem isn’t the <code>not in</code> operator — it’s the data structure.
Testing whether or not <code>numbers</code> contains <code>i</code> is only slow because <code>numbers</code> is a list.</p><p>In contrast, the complexity of the <em>contains</em> operation on a Python <a href="/post/abstract-data-types/#set">set</a> is <a href="/post/big-o-notation/#constant">O(1)</a>.﻿<a class="ptr">(2)</a>
Here’s the solution using a set:</p><pre class="code-listing language-python"><code>def smallest_missing_integer(numbers):
numbers_set = set(numbers)
i = 1
while True:
if i not in numbers_set:
return i
i += 1</code></pre><p>The code is almost exactly the same as before, but now there is no searching.
This algorithm is <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a>, which is efficient enough to satisfy Codility.</p><h4>Moral of the story</h4><p>Beware of <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a> operations disguised as basic operations, like <code>in</code> or <code>not in</code> on a list.</p><p>Set data structures are designed so that the <em>contains</em> operation is faster than searching a list.
If you want to test whether some values are in (or not in) a collection, use a set.</p><h3>Most common value</h3><p>Our next example is another simple problem:</p><ul><li><em>Problem:</em> given a list, find the value which occurs the most times.
<ul><li><em>Input:</em> a list.
</li><li><em>Output:</em> the value with the most occurrences in the list.
</li></ul>
</li><li class="language-json"><em>Example:</em> the most common value in <code>[3, 2, 1, 3]</code> is <code>3</code>.
</li><li class="language-json"><em>Example:</em> the most common value in <code>["f", "o", "o"]</code> is <code>"o"</code>.
</li></ul><p>This is a realistic problem; in statistics, the most common value in a list of numbers is called the <a href="https://en.wikipedia.org/wiki/Mode_(statistics)">mode</a>.</p><h4>Bad solution</h4><p>We can solve this problem by directly applying “plans”: <a href="/post/algorithmic-plans/#for-each">for each value</a>, <a href="/post/algorithmic-plans/#counter">count</a> the occurrences, and keep track of which value has <a href="/post/algorithmic-plans/#best-so-far">the most occurrences so far</a>.
</p><pre class="code-listing language-python"><code># subproblem: count occurrences of a value in a list
def count_occurrences(lst, value):
counter = 0
for x in lst:
if x == value:
counter += 1
return counter
def most_common_value(lst):
# find the highest count
most_common = lst[0]
highest_count = count_occurrences(lst, most_common)
for value in lst:
count = count_occurrences(lst, value)
if count > highest_count:
most_common = value
highest_count = count
return most_common</code></pre><p>This works, but it’s a bit complicated, and inefficient.
The <code>count_occurrences</code> function is <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a>, but the algorithm calls it from inside a loop, so it’s <a href="/post/big-o-notation/#quadratic">O(<i>n</i><sup>2</sup>)</a>.</p><p>From another perspective, every time this algorithm sees a value, it counts the occurrences, even if they’ve already been counted before.</p><p>There is a simpler, faster solution.</p><h4>Good solution</h4><p>We can count every value in <a href="https://en.wikipedia.org/wiki/One-pass_algorithm">one pass</a> through the list, using a <a href="/post/abstract-data-types/#dictionary">dictionary</a> to store the counters.
Each time we see a value, we’ve either seen it before or we haven’t:</p><ul><li>If the value is already in the dictionary, increase its counter by 1.
</li><li>Otherwise, this is the first occurrence, so set its counter to 1.
</li></ul><p>After we’ve counted the occurrences of every value, we can search the dictionary for the one with the highest counter.
Here’s the solution:</p><pre class="code-listing language-python"><code>def most_common_value(lst):
# count occurrences of every value
occurrences = dict()
for value in lst:
if value in occurrences:
occurrences[value] += 1
else:
occurrences[value] = 1
# find the highest count
most_common = lst[0]
highest_count = occurrences[most_common]
for value, count in occurrences.items():
if count > highest_count:
most_common = value
highest_count = count
return most_common</code></pre><p>The total amount of code is about the same as before, but now there is no “loop within a loop”.
There are two loops, but both are <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a>, so the algorithm is <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a> in total.﻿<a class="ptr">(3)</a></p><div class="specific-to language-python"><h4>Python-specific detail</h4><p>
Python’s <a href="https://docs.python.org/3/library/collections.html#collections.Counter">collections</a> module has a class called <code>Counter</code>, which is a dictionary for counting occurrences.
The following code works the same way as above, but is simpler:</p><pre class="code-listing"><code>from collections import Counter
def most_common_value(lst):
# count occurrences of every value
occurrences = Counter(lst)
# ...</code></pre></div><h4>Moral of the story</h4><p>If you need to count multiple things, count them all in the same loop.
Do as much “work” as you can in <a href="https://en.wikipedia.org/wiki/One-pass_algorithm">one pass</a>.</p><p>Instead of calling the same function again and again with the same inputs to find the same results, store those results in a <a href="/post/abstract-data-types/#dictionary">dictionary</a>.﻿<a class="ptr">(4)</a></p><h3>Substitution cipher</h3><p>Our final example is a <a href="https://en.wikipedia.org/wiki/Substitution_cipher">simple encryption scheme</a> which works by <em>substituting</em> letters for other letters.
The substitutions are defined by a “plaintext alphabet” and a “ciphertext alphabet”; for example, the plaintext alphabet is usually</p><div style="display:flex; flex-direction:column; align-items:center;"><p><code>"ABCDEFGHIJKLMNOPQRSTUVWXYZ"</code>
</p></div><p>and the ciphertext alphabet could be</p><div style="display:flex; flex-direction:column; align-items:center;"><p><code>"CEKTOBYZJAPDLVMRXHWNQSIUFG"</code>
</p></div><p>Using these alphabets, <code>"A"</code> is encrypted as <code>"C"</code>, <code>"B"</code> is encrypted as <code>"E"</code>, and so on.
Symbols which aren’t in the plaintext alphabet, like <code>" "</code> or <code>"."</code>, are not substituted.</p><ul><li><em>Problem:</em> encrypt a message.
<ul><li><em>Input:</em> a plaintext alphabet.
</li><li><em>Input:</em> a ciphertext alphabet.
</li><li><em>Input:</em> a message.
</li><li><em>Output:</em> the encrypted message.
</li></ul>
</li><li><em>Example:</em> with the ciphertext alphabet <code>"NRJMAUEFTXBHVPQOZILKYDGWCS"</code> and the usual plaintext alphabet, the message
<pre class="code-listing"><code>FEW FALSE IDEAS HAVE MORE FIRMLY GRIPPED THE MINDS OF SO
MANY INTELLIGENT MEN THAN THE ONE THAT, IF THEY JUST TRIED,
THEY COULD INVENT A CIPHER THAT NO ONE COULD BREAK.
- DAVID KAHN</code></pre>
is encrypted as
<pre class="code-listing"><code>UAG UNHLA TMANL FNDA VQIA UTIVHC EITOOAM KFA VTPML QU LQ
VNPC TPKAHHTEAPK VAP KFNP KFA QPA KFNK, TU KFAC XYLK KITAM,
KFAC JQYHM TPDAPK N JTOFAI KFNK PQ QPA JQYHM RIANB.
- MNDTM BNFP</code></pre>
</li></ul><p>This isn’t a secure cryptographic scheme, but there are real reasons to substitute some symbols with others.
For example, we might want to replace symbols like <code>"é"</code> with <code>"e"</code> when the <a href="https://en.wikipedia.org/wiki/Typeface">font</a> doesn’t support them.</p><h4>Bad solution</h4><p>We can encrypt a symbol by finding its index in the plaintext alphabet, and getting the substitute from the same index in the ciphertext alphabet.
If the symbol is not in the plaintext alphabet, no substitution will be made.</p><p>Here’s the solution:</p><pre class="code-listing language-python"><code>def encrypt(plaintext_alphabet, ciphertext_alphabet, message):
output = ""
for symbol in message:
# if symbol is in the plaintext alphabet, substitute it
i = plaintext_alphabet.find(symbol)
if i >= 0:
symbol = ciphertext_alphabet[i]
output += symbol
return output</code></pre><p>This works, but it is not efficient.</p><p>You may have noticed that the <code>find</code> method is a <a href="/post/search-algorithms/#linear-search">linear search</a>, and we’re doing this repeatedly with the same few symbols.
Following the advice above, it would be better to store the plaintext-to-ciphertext substitutions in a <a href="/post/abstract-data-types/#dictionary">dictionary</a>.</p><p>But there’s a worse, more subtle problem.
Every time we add a symbol to the output string, <a href="https://wiki.python.org/moin/PythonSpeed/PerformanceTips#String_Concatenation">the whole string is copied</a> to make a new string.
If <i>n</i> is the message length, then the algorithm is <a href="/post/big-o-notation/#quadratic">O(<i>n</i><sup>2</sup>)</a> just because of string concatenation.</p><h4>Good solution</h4><p>Let’s fix both problems; we can begin by building a dictionary of substitutions.
Then, to build the output string, we should put the parts in a <a href="/post/abstract-data-types/#list">list</a> and <a href="https://docs.python.org/3/library/stdtypes.html#str.join">join</a> them all at once.
Here’s the improved solution:</p><pre class="code-listing language-python"><code>def encrypt(plaintext_alphabet, ciphertext_alphabet, message):
# build a dictionary of substitutions
substitutes = dict()
for i, symbol in enumerate(plaintext_alphabet):
substitutes[symbol] = ciphertext_alphabet[i]
# collect the output in a list
output = []
for symbol in message:
# if symbol is in the plaintext alphabet, substitute it
if symbol in substitutes:
symbol = substitutes[symbol]
output.append(symbol)
# join the parts all at once
return ''.join(output)</code></pre><p>The code is now more complicated, but we replaced the <code>find</code> method with a dictionary’s <em>get</em> operation, and we replaced the string concatenation with a list’s <em>append</em> operation.
Both of these are <a href="/post/big-o-notation/#constant">O(1)</a> on Python’s built-in data structures, and the algorithm is now <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a> — a big improvement.</p><h4>Moral of the story</h4><p>If you repeatedly search one sequence to find an index in another, build a dictionary instead.
Dictionary data structures are designed to make the <em>get</em> operation efficient.</p><p>Never build a long string by concatenation.
Instead, collect the parts in a list, then join them all together at once.</p><h4>Footnotes</h4><ol id="footnotes"><li>The first missing integer is at most <i>n</i> + 1, so the main loop runs <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a> times in the worst case.</li><li>Python’s sets are a kind of <a href="/post/the-hash-map-data-structure/">hash map</a>.
Another efficient set data structure specifically for positive integers is a <a href="https://en.wikipedia.org/wiki/Bit_array">bit set</a>.</li><li>The first loop is <a href="/post/big-o-notation/#linear">O(<i>n</i>)</a> assuming that accessing a key in the dictionary is <a href="/post/big-o-notation/#constant">O(1)</a>.
This is true for Python dictionaries, which are a kind of <a href="/post/the-hash-map-data-structure/">hash map</a>.</li><li>That is, assuming there are no <a href="https://en.wikipedia.org/wiki/Side_effect_(computer_science)">side-effects</a> of calling the function.</li></ol>
http://dsaa.werp.site/post/choose-the-right-data-structures/2018-05-07T13:00:00+00:00‘Big O’ Notation<p>By making a <a href="/post/static-analysis/">series of assumptions</a> and considering only “large” inputs, we can analyse how efficient an algorithm is without actually running it.
The result of this analysis is a <em>mathematical formula</em> called the <a href="https://en.wikipedia.org/wiki/Time_complexity">complexity</a> (or <em>time complexity</em>) of the algorithm.
For example, we derived the formula <em>n</em><sup>2</sup> for the <a href="/post/iterative-sorting/#selection-sort">selection sort</a> algorithm.</p><p>Using <a href="https://en.wikipedia.org/wiki/Big_O_notation">standard notation</a>, we would say that selection sort’s complexity is O(<em>n</em><sup>2</sup>), or that selection sort is an O(<em>n</em><sup>2</sup>) algorithm.</p><p>This formula says, very roughly, how much “work” the algorithm has to do as a function of <em>n,</em> which represents the “input size”.
In this post, we’ll see how to make sense of this formula, and how to derive it with as little algebra as possible.</p><!-- pagebreak --><h4>Complexity and running time</h4><p>Before we begin, we should ask: is this even a useful kind of analysis?
Perhaps surprisingly, after all the assumptions and approximations, this <em>n</em><sup>2</sup> formula does say something useful about selection sort.</p><p>The chart on the left shows running times for selection sort, <a href="/post/how-fast-is-your-algorithm/">measured by actually running it</a>.
On the right is simply a plot of the equation <em>T</em> = <em>n</em><sup>2</sup>.</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/selection-sort-n-squared.png">
</p></div><p>This is remarkably accurate, except the scales are completely different — the running times scale from 0 to 160 <a href="https://en.wikipedia.org/wiki/Millisecond">ms</a>, while the complexity scales from 0 to 4 million.
The scale on the right has no real meaning; the scale on the left measures real time, but this would be different on different computers or in different programming languages anyway.﻿<a class="ptr">(1)</a></p><p>This is not a fluke.
Here are running times for the <a href="/post/search-algorithms/#linear-search">linear search</a> algorithm,﻿<a class="ptr">(2)</a>
with its complexity function <em>T</em> = <em>n</em> plotted on the right:</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/linear-search-n.png">
</p></div><p>The complexity function still gives a fairly accurate estimate of how the running time depends on <em>n</em>.</p><p>Here are running times for <a href="/post/search-algorithms/#binary-search">binary search</a>, alongside its complexity function <em>T</em> = log <em>n</em> on the right:</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/binary-search-log-n.png">
</p></div><p>Again, the complexity function is fairly accurate except for scale.</p><h4 id="big-o-definition">‘Big O’ notation</h4><p>The complexity gives a pretty accurate estimate of running time, but with no sense of scale.
This is because we justified our <a href="/post/static-analysis">simplifications and approximations</a> by saying it would still be “roughly proportional” to the running time.</p><p>If the complexity is <em>n</em><sup>2</sup> then the real running time might be approximately <em>n</em><sup>2</sup>, or 5<em>n</em><sup>2</sup>, or 100<em>n</em><sup>2</sup>; these functions all have the same “shape” but different scales.﻿<a class="ptr">(3)</a></p><p>To make it clear that we don’t know the scale, we use <a href="https://en.wikipedia.org/wiki/Big_O_notation">the capital letter O</a> to mean “roughly proportional to”.﻿<a class="ptr">(4)</a>
For example, when we say selection sort’s complexity is O(<em>n</em><sup>2</sup>), this means its running time is roughly proportional to <em>n</em><sup>2</sup>.</p><p>There are many possible complexity functions, but some are very common.
The following are in order from most efficient to least efficient.</p><h3 id="constant">O(1)</h3><p>An algorithm’s complexity is O(1) if its running time doesn’t depend on the input size at all.
This means that on average, the algorithm performs some fixed number of <a href="/post/static-analysis/#basic-operations">basic operations</a>, whatever the input is.</p><p>An O(1) algorithm is called <a href="https://en.wikipedia.org/wiki/Time_complexity#Constant_time">constant time</a>, because there is a fixed constant amount of time it takes on average, regardless of <em>n</em>.
For example, here are running times for the <em>contains</em> operation on Python’s built-in <a href="/post/abstract-data-types/#set">set</a> data structure:</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/set-contains-1.png">
</p></div><p>Every basic operation is O(1) by definition.
Others include:
</p><ul><li><em>Get</em> and <em>set</em> on an <a href="/post/arrays/">array</a>.
</li><li><em>Get, set</em> and <em>append*</em> on an <a href="/post/the-array-list-data-structure/">array-list</a>.
</li><li><em>Insert</em> and <em>remove</em> at the start of a <a href="/post/linked-list-data-structure/">linked list</a>.
</li><li><em>Get, set*, remove*</em> and <em>contains-key</em> on a <a href="/post/the-hash-map-data-structure/">hash map</a>.
</li><li><em>Insert*, remove*</em> and <em>contains</em> on a <em>hash set</em> data structure.
</li></ul><p>The operations marked * are O(1) <a href="https://en.wikipedia.org/wiki/Amortized_analysis">on average</a>, but sometimes require copying every element to a new array, which is O(<em>n</em>).</p><h3 id="logarithmic">O(log <em>n</em>)</h3><p>The <a href="https://en.wikipedia.org/wiki/Logarithm">logarithm</a> of a number is, roughly, how many times you can divide it by a given base before you get to 1.﻿<a class="ptr">(5)</a>
This is relevant to <a href="/post/algorithm-strategies/#divide-and-conquer">divide and conquer</a> algorithms, because it determines how many “divide” steps there are.
The logarithm of <em>n</em> is written as log <em>n</em>.</p><p>If the complexity of an algorithm is O(log <em>n</em>), then we call it <a href="https://en.wikipedia.org/wiki/Time_complexity#Logarithmic_time">logarithmic time</a>.
Larger inputs will take more time, but not proportionally — for example, <a href="/post/search-algorithms/#binary-search">binary search</a> only takes one more iteration for a list of length 200 than for a list of length 100.</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/binary-search-log-n.png">
</p></div><p>O(log <em>n</em>) algorithms include:
</p><ul><li>Binary search.
</li><li><em>Contains, insert</em> and <em>remove</em> on a <a href="https://en.wikipedia.org/wiki/Binary_search_tree">binary search tree</a>.
</li><li><em>Get, set, insert</em> and <em>remove</em> on a <a href="https://en.wikipedia.org/wiki/Skip_list">skip list</a>.
</li><li><em>Poll</em> from a <a href="https://en.wikipedia.org/wiki/Binary_heap">binary heap</a> priority queue.
</li></ul><h3 id="linear">O(<em>n</em>)</h3><p>If the complexity of an algorithm is O(<em>n</em>) then the running time is roughly proportional to the input size.
If the input is twice as large, the algorithm will take about twice as much time.
This is called <a href="https://en.wikipedia.org/wiki/Time_complexity#Linear_time">linear time</a>.</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/linear-search-n.png">
</p></div><p>Algorithms usually have linear complexity because they do a fixed amount of “work” for each item in some collection.
Examples include:
</p><ul><li>Linear search.
</li><li>Finding the minimum value, maximum value or sum of a list.
</li><li><em>Insert</em> and <em>remove</em> on an <a href="/post/the-array-list-data-structure/">array-list</a>.
</li><li><em>Get, set, insert</em> and <em>remove</em> on a <a href="/post/the-linked-list-data-structure/">linked list</a>.
</li><li>The “merge” procedure in the <a href="/post/recursive-sorting/#merge-sort">merge sort</a> algorithm.
</li></ul><h3 id="n-log-n">O(<em>n</em> log <em>n</em>)</h3><p>A complexity of O(<em>n</em> log <em>n</em>) is common for efficient sorting algorithms, or other algorithms which take advantage of sorting, such as <a href="https://en.wikipedia.org/wiki/Sweep_line_algorithm">sweep line</a> algorithms in geometry.﻿<a class="ptr">(6)</a>
For example, here are running times for <a href="/post/recursive-sorting/#merge-sort">merge sort</a>:</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/merge-sort-n-log-n.png">
</p></div><p>It’s hard to visually tell that this is not a straight line, but for larger <em>n</em> the curve is steeper.
Therefore, for large enough <em>n</em> the algorithm will be slower than any <a href="#linear">linear complexity</a> algorithm.</p><p>Algorithms with O(<em>n</em> log <em>n</em>) complexity include:
</p><ul><li><a href="/post/recursive-sorting/#merge-sort">Merge sort</a>, <a href="https://en.wikipedia.org/wiki/Quicksort">quicksort</a> and <a href="https://en.wikipedia.org/wiki/Heapsort">heap sort</a>.
</li><li>The <a href="https://en.wikipedia.org/wiki/Fast_Fourier_transform">fast Fourier transform</a> used in audio and image processing.
</li><li>The <a href="https://en.wikibooks.org/wiki/Algorithm_Implementation/Geometry/Convex_hull/Monotone_chain">monotone chain</a> algorithm for finding <a href="https://en.wikipedia.org/wiki/Convex_hull">convex hulls</a>.
</li></ul><h3 id="quadratic">O(<em>n</em><sup>2</sup>)</h3><p>An algorithm is O(<em>n</em><sup>2</sup>), or <em>quadratic time,</em> if the running time is roughly proportional to <em>n</em><sup>2</sup>.
This is usually because of a “loop in a loop”; the algorithm has an inner loop which is O(<em>n</em>), and an outer loop which runs the inner loop <em>n</em> times, doing a total of <em>n</em> <span>×</span> <em>n</em> or <em>n</em><sup>2</sup> basic operations.</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/selection-sort-n-squared.png">
</p></div><p>Quadratic complexity is common for problems where simply iterating over a collection isn’t enough, such as sorting.
<a href="/post/iterative-sorting/#selection-sort">Selection sort</a>, <a href="/post/iterative-sorting/#insertion-sort">insertion sort</a> and <a href="https://en.wikipedia.org/wiki/Bubble_sort">bubble sort</a> are all O(<em>n</em><sup>2</sup>).</p><p>Also, many naive algorithms are O(<em>n</em><sup>2</sup>) because they use an O(<em>n</em>) operation on a data structure, when a different data structure would do the same operation in constant time.</p><h4>Comparing algorithms</h4><p>The common complexities above are in order from most efficient to least efficient.
To decide between two algorithms, we can often just go by complexity:</p><ul><li>Merge sort is O(<em>n</em> log <em>n</em>), while selection sort is O(<em>n</em><sup>2</sup>) — so merge sort is faster.
</li><li><em>Getting</em> from an array-list is O(1), but from a linked list it’s O(<em>n</em>) — so <em>getting</em> from an array-list is faster.
</li></ul><p>These conclusions are correct, at least assuming <em>n</em> is large enough for this analysis to be valid.
It’s also a lot easier than doing a <a href="/post/how-fast-is-your-algorithm/">dynamic analysis</a> for both algorithms, since we don’t have to try lots of inputs, avoid measurement errors, or run every algorithm on the same computer to get comparable results.</p><p>On the other hand, when two algorithms have the same complexity, we can’t decide between them so easily — for example, selection sort and insertion sort are both O(<em>n</em><sup>2</sup>) — so we would do dynamic analysis to see which is faster.</p><h4>Deriving the complexity of an algorithm</h4><p>In the previous post we derived an exact formula for how many comparisons <a href="/post/iterative-sorting/#selection-sort">selection sort</a> would do, and then made approximations to get the formula <em>n</em><sup>2</sup>.
This took a fair bit of algebra, but using some simple rules it’s possible to derive the complexity much more easily.</p><p>Start with the code:</p><pre class="code-listing language-python"><code>def selection_sort(lst):
n = len(lst)
for i in range(n):
min_index = i
for j in range(i+1, n):
if lst[j] < lst[min_index]:
min_index = j
tmp = lst[i]
lst[i] = lst[min_index]
lst[min_index] = tmp</code></pre><p>Any <a href="#constant">fixed sequence of basic operations</a> is O(1), so we can simplify by writing how much “work” each part of the code takes:</p><pre class="code-listing language-python"><code>def selection_sort(lst):
# O(1)
for i in range(n):
# O(1)
for j in range(i+1, n):
# O(1)
# O(1)</code></pre><p>A loop multiplies the amount of “work” to be done.
The outer loop iterates <em>n</em> times; the inner loop iterates over a range whose size is proportional to <em>n</em>.</p><pre class="code-listing language-python"><code>def selection_sort(lst):
# O(1)
# O(n) times:
# O(1)
# O(n) times:
# O(1)
# O(1)</code></pre><p>Simplifying:</p><pre class="code-listing language-python"><code>def selection_sort(lst):
# O(1)
# O(n) times:
# O(1)
# O(n * 1) == O(n)
# O(1)</code></pre><p>O(<em>n</em>) is bigger than O(1), so the O(<em>n</em>) “dominates” the work inside the loop:</p><pre class="code-listing language-python"><code>def selection_sort(lst):
# O(1)
# O(n) times:
# O(n)</code></pre><p>Simplifying:</p><pre class="code-listing language-python"><code>def selection_sort(lst):
# O(1)
# O(n * n) == O(n ** 2)</code></pre><p>Finally, the O(1) is “dominated” by the O(<em>n</em><sup>2</sup>), so the overall complexity is O(<em>n</em><sup>2</sup>).</p><h4>Footnotes</h4><ol id="footnotes"><li>The time measurements were made in Python, and are quite different in scale to the <a href="/post/how-fast-is-your-algorithm/">measurements of insertion sort</a> which were made in Javascript.</li><li>Searching is much faster than sorting, so these times are in <a href="https://en.wikipedia.org/wiki/Microsecond">microseconds</a>.
These measurements are in the worst case, when the target is not in the list.</li><li>Ultimately, static analysis can never reveal the true scale without knowing how fast the <a href="/post/static-analysis/#basic-operations">basic operations</a> are.</li><li>Strictly speaking, it means “<em>at most</em> roughly proportional to”.
There is a <a href="https://en.wikipedia.org/wiki/Big_O_notation#Formal_definition">formal mathematical definition</a>, but for most purposes an informal understanding is good enough.</li><li>Because we’re ignoring scale, it doesn’t actually matter what the logarithm base is.</li><li>In fact O(<em>n</em> log <em>n</em>) is the lowest complexity theoretically possible for a <a href="/post/comparisons-and-swaps/">comparison-based</a> sorting algorithm, so all sorting algorithms which achieve this are <a href="https://en.wikipedia.org/wiki/Asymptotically_optimal_algorithm">asymptotically optimal</a>.</li></ol>
http://dsaa.werp.site/post/-big-o-notation/2018-05-07T12:00:00+00:00Static Analysis<p>In <a href="/post/how-fast-is-your-algorithm">the previous post</a> we analysed how efficient the <a href="/post/iterative-sorting/#insertion-sort">insertion sort</a> algorithm is.
We did <a href="https://en.wikipedia.org/wiki/Dynamic_program_analysis">dynamic analysis</a> — we analysed the algorithm by actually running it.</p><p>Technically, we analysed an <em>implementation</em> of the algorithm; perhaps a cleverer implementation could be more efficient.
From one perspective, the implementation is what we really care about:
</p><ul><li>The implementation is what will actually be run in the real world.
</li><li>A more efficient implementation will make a real piece of software faster.
</li></ul><p>On the other hand, it’s hard to do an accurate dynamic analysis:
</p><ul><li>There will always be some measurement error.
</li><li>It might take a long time to do the analysis.
</li><li>Running time depends on external factors like how fast the computer is, or what other programs are running.
Results from different computers, or even the same computer on different days, may be incomparable.
</li></ul><p>In this post, we’ll get around these problems by making a series of simplifications.
We’ll do a kind of <a href="https://en.wikipedia.org/wiki/Static_program_analysis">static analysis</a> called <a href="https://en.wikipedia.org/wiki/Big_O_notation">asymptotic analysis</a> — this means we’ll analyse algorithms without running them, and we’ll assume the inputs are large.</p><!-- pagebreak --><h4>Informal static analysis</h4><p>Before we begin, let’s consider how we’ve already been analysing efficiency informally.
For example, when we analysed <a href="/post/the-array-list-data-structure/">array-lists</a> and <a href="/post/the-linked-list-data-structure/">linked lists</a>, we talked about how many steps it took to perform list operations like <em>get</em> and <em>insert:</em></p><ul><li><em>Getting</em> from an array-list is faster than a linked list, because the array-list can read the value directly from memory, whereas in a linked list the computer might have to follow many pointers.
</li><li><em>Inserting</em> is slower near the start of an array-list than near the end, since it requires more reading/writing to create space for the new value in the array.
</li><li><em>Inserting</em> is faster near the start of a linked list than near the end, since it requires following fewer pointers.
</li></ul><p>This is nothing like dynamic analysis.
Instead of running the algorithms and measuring the time, this analysis is hypothetical — what <em>would</em> happen <em>if</em> we run them?
And instead of measuring time, we’re measuring how much “work” the algorithm has to do.</p><p>Before we go any further, it is well worth revisiting the the interactive models of <a href="/post/the-array-list-data-structure/">array-lists</a> and <a href="/post/the-linked-list-data-structure/">linked lists</a>, and thinking carefully about how many steps it takes to perform each list operation.
How is the number of steps represented in each model?
When does it depend on the length of the list?</p><h4 id="basic-operations">Simplification: count basic operations</h4><p>The time it takes a program to run is roughly proportional to how many “basic operations” the computer will execute.
These include things like <a href="/post/memory-and-pointers/">reading or writing memory</a>, <a href="https://en.wikipedia.org/wiki/Arithmetic#Arithmetic_operations">arithmetic operations</a>, <a href="https://en.wikipedia.org/wiki/Bitwise_operation">bitwise operations</a>, <a href="https://en.wikipedia.org/wiki/Relational_operator">comparison operations</a> and <a href="https://en.wikipedia.org/wiki/Logical_connective">logical operations</a> — the most basic steps of a computation, which take a constant time to execute.</p><p>So, we can judge the efficiency of an algorithm by counting how many basic operations it will perform.
This simplification is reasonable, because more basic operations should mean proportionally more time.</p><p>On the other hand, different basic operations take different times — for example, adding <a href="https://en.wikipedia.org/wiki/Integer_(computer_science)">integers</a> is generally faster than dividing <a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic">floating-point numbers</a>.
So this simplification isn’t always valid, but it makes the analysis much easier, and usually still gets the right answer.</p><p>It’s worth clarifying some things:
</p><ul><li>A “basic operation” doesn’t need to be a single <a href="https://en.wikipedia.org/wiki/Machine_code">CPU instruction</a>, as long as it takes a constant amount of time.
</li><li>We’re not talking about the number of instructions in the code; if a <a href="https://en.wikipedia.org/wiki/Control_flow#Loops">loop</a> causes the same instruction to be executed 1000 times, then we should count it 1000 times.
</li><li>Not every <a href="https://en.wikipedia.org/wiki/Language_primitive">language primitive</a> is a basic operation.
For example, Python’s <code>in</code> operator actually does a <a href="/post/search-algorithms/#linear-search-variations">linear search</a> when you use it on a <a href="/post/abstract-data-types/#list">list</a>, so the number of basic operations depends on the length of the list.
</li></ul><h4>Simplification: assume large inputs</h4><p>The amount of “work” an algorithm has to do will <a href="/post/how-fast-is-your-algorithm/#input-size">depend on how large the input is</a>.
This could be the length of a <a href="/post/abstract-data-types/#list">list</a>, the size of a <a href="/post/abstract-data-types/#set">set</a> or <a href="/post/abstract-data-types/#dictionary">dictionary</a>, or the number of nodes in a <a href="/post/trees/">tree</a> or <a href="/post/graphs/">graph</a>.</p><p>Computers are blazingly fast at small tasks; even inefficient algorithms are usually fast enough when the input is small.
On the other hand, when the input is large, the differences between algorithms can be enormous.</p><p>Consider how many <a href="/post/comparisons-and-swaps/#comparisons">comparison</a> operations <a href="/post/search-algorithms/">linear and binary search</a> might have to perform, depending on the length of the list:</p><div style="display:flex; flex-direction:column; align-items:center;"><table class="tex-tbl"><style type="text/css" scoped>td:nth-child(1){text-align:right;}td:nth-child(2){text-align:right;}td:nth-child(3){text-align:right;}</style><thead><tr><th>n </th><th> Linear search </th><th> Binary search
</th></tr></thead><tbody><tr><td>10 </td><td> 10 </td><td> 4 </td></tr><tr><td>
100 </td><td> 100 </td><td> 7 </td></tr><tr><td>
1,000 </td><td> 1,000 </td><td> 10 </td></tr><tr><td>
10,000 </td><td> 10,000 </td><td> 14 </td></tr><tr><td>
100,000 </td><td> 100,000 </td><td> 17 </td></tr><tr><td>
1,000,000 </td><td> 1,000,000 </td><td> 20
</td></tr></tbody></table></div><p>How about <a href="/post/iterative-sorting/#selection-sort">selection sort</a> and <a href="/post/recursive-sorting/#merge-sort">merge sort</a>?</p><div style="display:flex; flex-direction:column; align-items:center;"><table class="tex-tbl"><style type="text/css" scoped>td:nth-child(1){text-align:right;}td:nth-child(2){text-align:right;}td:nth-child(3){text-align:right;}</style><thead><tr><th>n </th><th> Selection sort </th><th> Merge sort
</th></tr></thead><tbody><tr><td>10 </td><td> 45 </td><td> 40 </td></tr><tr><td>
100 </td><td> 4,950 </td><td> 700 </td></tr><tr><td>
1,000 </td><td> 499,500 </td><td> 10,000 </td></tr><tr><td>
10,000 </td><td> 49,995,000 </td><td> 140,000 </td></tr><tr><td>
100,000 </td><td> 4,999,950,000 </td><td> 1,700,000 </td></tr><tr><td>
1,000,000 </td><td> 499,999,500,000 </td><td> 20,000,000
</td></tr></tbody></table></div><p>Algebraically, we say <em>n</em> is the “input size”, and we can mathematically estimate how many basic operations an algorithm will perform, as a <a href="https://en.wikipedia.org/wiki/Function_(mathematics)">function</a> of <em>n</em>.﻿<a class="ptr">(1)</a>
When one algorithm is faster than another, it might be difficult to judge for small <em>n,</em> but for large <em>n</em> it’s not even close.</p><p>This is important because it will justify all of our other simplifications.
When the difference between algorithms is this great, accuracy barely matters — we can make big approximations without invalidating the analysis.</p><h4>Simplification: count <em>one</em> basic operation</h4><p>Consider an algorithm which finds the <a href="/post/algorithmic-plans/#running-total">sum</a> of a list of numbers:</p><pre class="code-listing language-python"><code>def list_sum(numbers):
total = 0
for x in numbers:
total += x
return total</code></pre><p>This algorithm uses a few different basic operations:
</p><ul><li>Reading a value from <code>numbers</code>.
</li><li>Adding a value to <code>total</code>.
</li><li>Assigning a value to <code>total</code>.
</li></ul><p>For a list of <em>n</em> numbers, the algorithm would “read” <em>n</em> times, “add” <em>n</em> times, and “assign” <em>n</em> + 1 times (including the initial assignment <code>total = 0</code>).
So the total number of basic operations is 3<em>n</em> + 1.</p><p>We can simplify things by just counting the “add” operations.
If the running time is roughly proportional to the 3<em>n</em> + 1 total, then it’s also roughly proportional to the <em>n</em> “add” operations.</p><p>Assuming <em>n</em> is large, the difference between <em>n</em> and 3<em>n</em> + 1 is not really important.
Whether merge sort does 20 million or 60 million basic operations, that’s still less than the <em>half a trillion</em> for selection sort.</p><p>So, it’s fine to count just additions, or just comparisons, and ignore other basic operations — as long as the number is still proportional to the total “work” the algorithm has to do.</p><h4>Simplification: ignore small terms and constants</h4><p>The <a href="/post/iterative-sorting/#selection-sort">selection sort</a> algorithm iteratively finds the smallest unsorted value, and swaps it into place.
If there are <em>n</em> unsorted values, finding the smallest one requires exactly <em>n</em> <span>−</span> 1 comparisons.﻿<a class="ptr">(2)</a></p><p>The unsorted part of the list gets smaller on each iteration; for example, the total number of comparisons for a list of length 10 would be:
</p><div style="display:flex; flex-direction:column; align-items:center;"><p>9 + 8 + 7 + 6 + 5 + 4 + 3 + 2 + 1 = 45</p></div><p>
The <a href="https://en.wikipedia.org/wiki/Triangular_number">formula for this total</a> is:
</p><div style="display:flex; flex-direction:column; align-items:center;"><p>(<em>n</em> <span>−</span> 1) + <span>⋯</span> + 1 = <span class="tex-frac"><span><em>n</em>(<em>n</em> <span>−</span> 1)</span><span>2</span></span></p></div><p>
When <em>n</em> is large, the difference between <em>n</em> and <em>n</em> <span>−</span> 1 doesn’t matter:
</p><div style="display:flex; flex-direction:column; align-items:center;"><p><span class="tex-frac"><span><em>n</em>(<em>n</em> <span>−</span> 1)</span><span>2</span></span> <span>≈</span> <span class="tex-frac"><span><em>n</em> <span>×</span> <em>n</em></span><span>2</span></span> = <span class="tex-frac"><span><em>n</em><sup>2</sup></span><span>2</span></span></p></div><p>By the same reasoning as before, the factor of 1/2 doesn’t really matter, so <em>n</em><sup>2</sup> is a good enough approximation for how many comparisons selection sort will do.
These approximations are fine, so long as we’re only interested in large inputs.</p><h4>Conclusion</h4><p>It’s worth remarking on how abstract this is — given an <em>algorithm</em> we can derive a <em>mathematical formula</em> for how efficient it is.
This formula is called the <a href="https://en.wikipedia.org/wiki/Time_complexity">complexity</a>, or <em>time complexity</em> of the algorithm.</p><p>The complexity formula doesn’t really tell us how much time the algorithm will take to run; it very roughly measures how much “work” the algorithm has to do, as a function of how “large” the input is.</p><p>In the next post, we’ll see how to make sense of complexities, and use them to compare algorithms against each other.
We’ll also see a much easier way to derive the complexity of an algorithm, requiring less algebra.</p><h4>Footnotes</h4><ol id="footnotes"><li>An estimate as a function of <em>n</em> may be specific to a <a href="/post/how-fast-is-your-algorithm/#input-variations">kind of input</a> — usually either random inputs, or the “worst case”.</li><li>The first value is automatically the “<a href="/post/algorithmic-plans/#best-so-far">best so far</a>”, but the remaining <em>n</em> <span>−</span> 1 values must be compared.</li></ol>
http://dsaa.werp.site/post/static-analysis/2018-05-07T11:00:00+00:00How Fast is Your Algorithm?<p>Efficiency is important; people don’t like waiting for computers to slowly do things that ought to be fast.</p>
<script src="http://dsaa.werp.site/js/efficiency/timing.js" type="text/javascript"></script>
<div id="one_plus_one" class="adt-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<p>So far we’ve been thinking about this in vague terms; we know, for example:</p><ul><li><em>Getting</em> from an <a href="/post/the-array-list-data-structure/">array-list</a> is faster than from a <a href="/post/the-linked-list-data-structure/">linked list</a>,
</li><li><a href="/post/search-algorithms/#binary-search">Binary search</a> is faster than <a href="/post/search-algorithms/#linear-search">linear search</a>,
</li><li><a href="/post/recursive-sorting/#merge-sort">Merge sort</a> is faster than <a href="/post/iterative-sorting/#selection-sort">selection sort</a>.
</li></ul><p>Now it’s time to go into detail: how do we know how fast an algorithm is?</p><!-- pagebreak --><h4>An analogy</h4><p>In the real world, there’s an obvious way to find out how long something takes: do it, and measure the time with a stopwatch.</p>
<div id="stopwatch" class="adt-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<p>Try using this stopwatch to measure how long the “1 + 1” model takes to finish.</p><p>Of course, computing 1 + 1 doesn’t realistically take this long; computers are too fast to measure manually with a stopwatch.
This is just an analogy; really, we would make the computer start the timer, run the algorithm and stop the timer all automatically.</p><p>But this analogy does demonstrate some relevant issues:
</p><ul><li>Neither you, nor the computer, can do two things at exactly the same time.﻿<a class="ptr">(1)</a>
The measurement includes the time between starting the stopwatch and starting the computation.
</li><li>If you repeat the measurement, you might get a different result.
</li><li>Measuring the time for 1 + 1 doesn’t necessarily tell us how fast the same algorithm would be with different inputs; would 2 + 2 be twice as slow, four times as slow, or the same speed?
</li></ul><p>So, when we measure how long an algorithm takes to run, we should:
</p><ul><li>Avoid doing anything else while the timer is running — for example, if the algorithm takes a <a href="/post/abstract-data-types/#list">list</a> as input, create the list <em>before</em> starting the timer.
</li><li>Measure repeatedly and take an <a href="https://en.wikipedia.org/wiki/Average">average</a>.
</li><li>Try different inputs to see what factors affect the running time.
</li></ul><p>Let’s illustrate all of these points by measuring the <a href="/post/iterative-sorting/#insertion-sort">insertion sort</a> algorithm.
To follow along, you’ll need the code for it.</p><p>The rest of the models on this page are real — they genuinely run the insertion sort algorithm in your browser, and measure the running time.
This is called <a href="https://en.wikipedia.org/wiki/Dynamic_program_analysis">dynamic analysis</a>, because we’re analysing the algorithm by actually running it.﻿<a class="ptr">(2)</a></p><div class="warning"><h4>Warning</h4><p><a href="https://developers.google.com/web/updates/2018/02/meltdown-spectre#high-resolution_timers">For security reasons</a>, many browsers don’t provide Javascript access to a high-resolution clock.
Therefore, the times shown here may be inaccurate, or the models may not work at all.
Follow along in Python to see more accurate results.
</p></div><h4>Timing a function</h4><p>Let’s generate a list of 100 random numbers, and see how long it takes the <em>insertion sort</em> algorithm to sort this list.</p><p>We’ll use the <code>perf_counter</code> function from Python’s built-in <a href="https://docs.python.org/3/library/time.html#time.perf_counter">time</a> library.
The running time of the <code>insertion_sort</code> function is the end time minus the start time.
We’ll multiply by 1000 to measure in <a href="https://en.wikipedia.org/wiki/Millisecond">milliseconds</a> instead of seconds.</p><pre class="code-listing language-python"><code>from random import random
from time import perf_counter
def generate_random_list(n):
lst = []
for i in range(n):
lst.append(random())
return lst
def time_insertion_sort(n):
# generate random list
lst = generate_random_list(n)
# measure running time
start_time = perf_counter()
insertion_sort(lst)
end_time = perf_counter()
# return time in milliseconds
return 1000 * (end_time - start_time)
t = time_insertion_sort(100)
print('Time:', t, 'ms')</code></pre><p>Try it out:</p>
<div id="time_insertion_sort_1" class="adt-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<h4>Taking an average</h4><p>If we repeat the measurement, we don’t always get the same result.
This isn’t just because the list is random — the clock itself is not perfectly accurate.
The clock error will be small, but so are the times we’re measuring.</p><p>To get a more accurate measurement, we can execute the <em>insertion sort</em> algorithm repeatedly, and divide the total time by the number of repetitions to get an average.
For example, let’s do 500 repetitions.</p><p>To do this properly, we should generate 500 different lists.﻿<a class="ptr">(3)</a></p><pre class="code-listing language-python"><code>def time_insertion_sort(n, repetitions):
# generate random lists
lists = []
for i in range(repetitions):
lst = generate_random_list(n)
lists.append(lst)
# measure running time
start_time = perf_counter()
for lst in lists:
insertion_sort(lst)
end_time = perf_counter()
# return average in milliseconds
return 1000 * (end_time - start_time) / repetitions
t = time_insertion_sort(100, 500)
print('Average time:', t, 'ms')</code></pre><p>Try it out:</p>
<div id="time_insertion_sort_2" class="adt-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<h4 id="input-size">Trying different input sizes</h4><p>The amount of input data usually has a big effect on how long an algorithm takes to run; so we should try lists of many different lengths.
We’ll try lists of length 100, 200, 300, 400, and so on up to 2000.</p><p>This code will measure the average time for each length, and print the results in a <a href="https://en.wikipedia.org/wiki/Tab-separated_values">tab-separated</a> format.</p><pre class="code-listing language-python"><code>def vary_list_length(max_n, step_size, repetitions):
# print headers, tab-separated
print('n', 'average time (ms)', sep='\t')
for n in range(0, max_n+1, step_size):
t = time_insertion_sort(n, repetitions)
# print results, tab-separated
print(n, t, sep='\t')
# try n = 100, 200, ..., 2000 with 200 repetitions each
vary_list_length(2000, 100, 200)</code></pre><p>Try it out.
It will take some time to run; reduce the number of repetitions if it’s too slow.</p>
<div id="time_insertion_sort_3" class="adt-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<p>This data is easy to import into a spreadsheet, so we can plot a chart of the average running times:</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/insertion-sort-chart.png">
</p></div><p>If you’re following along in Python, then your measurements may be quite different, but your chart should have a similar curved shape.</p><p>In particular, the chart is not a straight line — when the list is twice as long, insertion sort is <strong>not</strong> simply twice as slow; in this experiment, it was about <em>four</em> times as slow.</p><h4 id="input-variations">More input variations</h4><p>We can learn more about how efficient <em>insertion sort</em> is by trying different kinds of inputs — not just different list lengths.
Real data usually isn’t completely random, so it could be useful to see whether the running time is affected by how disordered the input list is.
We could try:</p><ul><li class="language-json"> Lists which are already in order, like <code>[1, 2, 3, 4, 5]</code>.
</li><li class="language-json"> Lists which are completely out of order, like <code>[5, 4, 3, 2, 1]</code>.
</li><li class="language-json"> Lists which are almost in order, like <code>[1, 3, 4, 2, 5]</code>.
</li><li class="language-json"> Lists where most of the values are the same, like <code>[1, 1, 1, 3, 1]</code>.
</li></ul><p>You can try out these options with the model below.
If you’re following along in Python, you’ll need to change the <code>generate_random_list</code> function.</p>
<div id="time_insertion_sort_4" class="adt-model">(This interactive feature requires Javascript to be enabled in your browser.)</div>
<p>Putting these all together shows that the kind of input really makes a big difference to how fast the algorithm is:</p><div style="display:flex; flex-direction:column; align-items:center;"><p><img src="http://dsaa.werp.site/js/efficiency/insertion-sort-variations.png">
</p></div><h4>Footnotes</h4><ol id="footnotes"><li>CPUs with multiple cores can execute multiple instructions at once, but it is practically impossible to guarantee that two particular instructions will be exactly simultaneous.</li><li>Technically, we’re analysing an <em>implementation</em> of the algorithm; an alternative implementation of the same algorithm, e.g. in a different programming language, could give different results.</li><li>This isn’t just because an average over different inputs is more reliable.
The algorithm is <a href="https://en.wikipedia.org/wiki/In-place_algorithm">in-place</a>, so if we used the same list 500 times, it would only be unsorted for the first run.</li></ol>
http://dsaa.werp.site/post/how-fast-is-your-algorithm/2018-05-07T10:00:00+00:00Exercises: Algorithm Design<p>Before attempting these exercises, you should read the posts on <a href="/post/algorithm-strategies/">algorithm strategies</a> and <a href="/post/choose-the-right-data-types/">choosing data types</a>.</p><!-- pagebreak --><h3>Scenario</h3><p>You are developing a website where users can search for and compare mobile phones.
The user chooses the two criteria (e.g. screen size and battery capacity), and a database query returns a list of phone models.</p><p>This list contains tuples of three values; the first value is the model name, and the second and third values are the user’s criteria.
For example:</p><pre class="code-listing language-python"><code>search_results = [
("Samsung Galaxy S8", 5.8, 3000),
("Samsung Galaxy S7", 5.1, 3000),
("Samsung Galaxy A5 2017", 5.2, 3000),
("Apple iPhone 8", 4.7, 1820),
("Huawei Honor View 10", 5.99, 3750),
("Apple iPhone X", 5.8, 2716),
("Samsung Galaxy S8 Plus", 6.2, 3500),
("Samsung Galaxy J5 2017", 5.2, 3000),
("Huawei Mate 10 Pro", 6, 4000),
("Apple iPhone 7", 4.7, 1960),
("Motorola Moto G5S Plus", 5.5, 3000),
("Apple iPhone 8 Plus", 5.5, 2670),
("Google Pixel 2 XL", 6, 3520),
("Huawei P Smart", 5.65, 3000),
("OnePlus 5T", 6, 3300)
]</code></pre><p><span class="language-python">The format for each tuple is <code>(name, screen_size, battery_capacity)</code>.</span></p><p>Since there may be many search results, it is your task to narrow down this list.
The rule for doing this is:
</p><ul><li>One phone “dominates” another phone if it is better on one criterion, and at least as good on the other criterion.
</li><li>A phone should only be shown to the user if it is not “dominated” by any other phone.
</li></ul><p>We assume that larger numbers are better — i.e. the user prefers a bigger screen, more battery capacity, and so on.﻿<a class="ptr">(1)</a>
For example:
</p><ul><li class="language-json"><code>"Samsung Galaxy S8"</code> dominates <code>"Samsung Galaxy S7"</code> because it has a bigger screen and the same battery capacity.
</li><li class="language-json"><code>"Apple iPhone X"</code> has a bigger screen but a smaller battery capacity than <code>"Motorola Moto G5S Plus"</code>, so neither dominates the other.
</li></ul><p>When solving the exercises, use the small list above and the longer list in the following file to test your functions:
</p><ul><li><a href="/js/problems/phone_models.py">phone_models.py</a> — test data (2223 phones).
</li></ul><p>In the small list, the phones not dominated by another phone are:
</p><pre class="code-listing"><code>Samsung Galaxy S8 Plus
Huawei Mate 10 Pro</code></pre><h3>Exercises</h3><ol class="exercises"><li>Is any of these phones dominated by all other phones?
</li><li class="language-python"> Write a function <code>def dominates(phone1, phone2):</code> which returns a <a href="/post/booleans/">Boolean</a> indicating whether <code>phone1</code> dominates <code>phone2</code>.
</li><li class="language-python"> Write a function <code>def is_dominated(phone, lst):</code> which returns a <a href="/post/booleans/">Boolean</a> indicating whether <code>phone</code> is dominated by any other phone in <code>lst</code>.
</li><li class="language-python"> Write a function <code>def show_best_results(lst):</code> which prints the names of the phones that aren’t dominated.
What strategy does your algorithm use?
</li><li class="language-python"> Write a function which sorts <code>search_results</code> in-place by screen size, from largest to smallest.
Phones with the same screen size should be ordered by battery capacity, from largest to smallest. <br>
You may use the built-in <code>sort()</code> method, or implement an in-place sorting algorithm of your choice.
</li><li>Using the sorted list, write a function which prints the names of the phones that aren’t dominated.
Your algorithm must only iterate over the list once. <br>
(Hint: keep track of the <a href="/post/algorithmic-plans/#best-so-far">best battery capacity so far</a>.)
</li></ol><h4>Footnotes</h4><ol id="footnotes"><li>If the user prefers e.g. a smaller screen or a lower price, we could make those numbers negative to avoid complicating the algorithm.</li></ol>
http://dsaa.werp.site/post/exercises-algorithm-design/2018-04-30T17:00:00+00:00