Theory of Computing Notes

Computer Science 461 - Fall 2023

Jump to week:

Week 1 Notes

Day Section Topic
Mon, Aug 21 1.3 Proof techniques
Wed, Aug 23 2.1 - 2.2 Notation & encodings
Fri, Aug 25 3.1 - 3.7 Boolean circuits

Monday, Aug 21

Today we talked about proof techniques, particularly proof by induction. We looked at these two examples:

  1. Prove that if a set AA has nn elements, then AA has 2n2^n subsets.

The second example had to do with the famous Tower’s of Hanoi puzzle( see https://youtu.be/SMleU0oeGLg).

  1. Use induction to prove that it is always possible to move the disks from one peg to another by moving one disk at a time without breaking the rules.

Wednesday, Aug 23

Today we reviewed mathematical notation, including some new notation we will be using. Then we talked about encoding a finite set using binary strings. We observed that every countable set SS can be encoded with a 1-to-1 function from S{0,1}*S \rightarrow \{0,1\}^*. We discussed specific encodings such as how to encode the integers \mathbb{Z} and the rationals \mathbb{Q}.

At the end we considered the set {0,1}={f:f:{0,1}}\{0,1\}^\infty = \{f : f : \mathbb{N}\rightarrow \{0,1\} \} which you can think of as the set of all infinitely long strings of zeros and ones. We claimed, but did not have time to prove that there is no 1-to-1 encoding function E:{0,1}{0,1}*E : \{0,1\}^\infty \rightarrow \{0,1\}^*.

Friday, Aug 25

Today we finished the proof that there is no 1-to-1 encoding function from {0,1}\{0,1\}^\infty to {0,1}*\{0,1\}^*.

Then we talked about Boolean circuits which are formed by AND, OR, and NOT gates.

  1. Show that you can construction the function IF A THEN B ELSE C for any Boolean inputs A, B, C using AND, OR, and NOT gates.

  2. Use mathematical induction to prove the following:

Theorem. Every function f:{0,1}n{0,1}f:\{0,1\}^n \rightarrow \{0,1\} can be represented by a Boolean circuit.

We finished by talking about how NAND gates are also universal in the sense that any function that can be constructed from AND, OR, NOT gates can also be constructed with NAND gates.


Week 2 Notes

Day Section Topic
Mon, Aug 28 Impossible computer programs
Wed, Aug 30 2.1 Intro to finite automata
Fri, Sep 1 2.2 - 2.3 Regular languages

Monday, Aug 28

Today we talked about functions f:{0,1}*{0,1}f:\{0,1\}^* \rightarrow \{0,1\} that cannot be computed by a computer program.

Wednesday, Aug 30

Today we introduced finite automata. We started with three examples:

  1. An automatic door at a grocery store with sensors on the front and rear. It opens if the sensor on the front is active. It won’t close until neither of the sensors are active.

  2. An automatic toll both that accepts nickles, dimes, and quarters. The gate won’t open until it receives 25 cents.

  3. A combination lock needs three numbers entered correctly (in order) before it will open.

With these examples we introduced state diagrams, and the definition of a deterministic finite automata (DFA).

  1. Then we constructed a DFA that can compute the function f:{0,1}*{0,1}f: \{0,1\}^* \rightarrow \{0,1\} which returns 1 when the input string has an odd number of 1’s and 0 otherwise.

Friday, Sep 1

Today we started with these questions about DFAs:

  1. For the DFA shown below:

    1. What is the transition function?
    2. Describe the function f:{0,1}*{0,1}f:\{0,1\}^* \rightarrow \{0,1\} that this DFA computes.
  2. Draw the state diagram for a DFA that computes whether a binary string contains 011.

  3. Modify the DFA from the last problem so that it computer whether a binary string ends with 011.

  4. How many states would a DFA need if you wanted to check whether a binary string has a 1 in the third position from the last?

After we did these examples, we defined languages which are just subsets of strings in Σ*\Sigma^*. Then we gave a recursive definition of regular languages and looked at some simple examples.


Week 3 Notes

Day Section Topic
Wed, Sep 6 2.4 Nondeterministic finite automata
Fri, Sep 8 2.6 NFAs and regular languages

Wednesday, Sep 6

Today we looked at an alternative definition of a regular language: A language LΣ*L \subseteq \Sigma^* is regular if and only if there is a DFA that accepts xΣ*x \in \Sigma^* if and only if xLx \in L. To prove that this definition is equivalent to the one we gave last week, we need to prove three things:

  1. The union of two regular languages is regular.

  2. The concatenation of two regular languages is regular.

  3. The Kleene star of a regular language is regular.

We proved #1 in class by thinking about how to construct a new DFA MM that accepts L1L2L_1 \cup L_2 using DFAs M1M_1 and M2M_2 that accept L1L_1 and L2L_2 respectively. To prove this, we answered these questions:

  1. If the machine MM is built by running both M1=(Q1,Σ,δ1,q1,F1)M_1 = (Q_1, \Sigma, \delta_1, q_1, F_1) and M2=(Q2,Σ,δ2,q2,F2)M_2 = (Q_2, \Sigma, \delta_2, q_2, F_2) simultaneously, what are the possible states of MM?

  2. What are the accepting states of MM?

  3. What are the initial states for MM?

  4. What is the transition function for MM?

To prove that regular languages are closed under concatenation, we introduced a new idea: non-deterministic finite automata (NFAs).

We looked at this example:

We answered these questions:

  1. Which states are active at each step as we read the input string 010110?

  2. Does this NFA accept the string 010110?

  3. Describe the set of all strings in {0,1}*\{0,1\}^* that this NFA will accept.

Friday, September 8

Today we continued talking about NFAs. We’ve been following the textbook by Meshwari & Smid pretty closely this week, so I strongly recommend reading skimming Section 2.6 which is what we covered today.

We started with some examples of NFAs.

  1. Last week we talked about how many states a DFA would need to check if a string in {0,1}*\{0,1\}^* has a 1 in its third to last entry. We saw that a DFA needs at least 8 states for that language. But an NFA can do the job with only four states. Try to construct an example. Hint: What if the NFA “guesses” every time it sees a 1 that that might be the third to last entry. What should it’s states be from that point on?

  2. Construct an NFA that accepts a string in {0}*\{0\}^* iff the length of the string is a multiple of 2 or 3. Can you construct a DFA that does the same thing and with the same number of states?

After these examples, we explained two important ideas:

  1. You can easily construct an NFA to check if a string is in L1L2L_1 L_2 when L1L_1 and L2L_2 are regular languages.

  2. It is also easy to check if xL*x \in L^* when LL is a regular language.

For practice, see if you can construct an NFA that checks if xL1L2x \in L_1 \cup L_2 when L1L_1 and L2L_2 are regular languages.


Week 4 Notes

Day Section Topic
Mon, Sep 11 2.5 NFAs and DFAs are equivalent
Wed, Sep 13 2.7 - 2.8 Regular expressions
Fri, Sep 15 2.9 The pumping lemma

Monday, September 11

Today we explained the surprising fact that any language an NFA can recognize can also be recognized by a DFA. The idea is to convert an NFA N=(Q,Σ,δ,q,F)N = (Q, \Sigma, \delta, q, F) into a DFA MM which has states that are subsets of the states of NN, i.e., the states of MM are the power set 2Q2^Q of the states of NN. Then you have to make a transition function from sets of states to sets of states using the transition function δ\delta.

We looked at this example (again) to see how the idea works:

Notice that this NFA starts at state 0, and no matter what input we receive, state 0 stays active. So there is no way to reach a set of active states that does not contain state 0. So that cuts the number of elements we need to worry about in 2Q2^Q by half.

We also talked about regular expressions.

Recursive Definition. A regular expression over an alphabet Σ\Sigma is a string ee with symbols from the extended alphabet Σ{(,),*,|}\Sigma \cup \{(,),*,|\} that has one of the following forms:

  1. ee is a single symbol in Σ\Sigma.
  2. e=e1e2e = e_1e_2
  3. e=e1|e2e = e_1|e_2 where e1e_1 and e2e_2 are regular expressions.
  4. e=(e1)e = (e_1) where e1e_1 is a regular expression
  5. e=(e1)*e = (e_1)* where e1e_1 is a regular expression

We also accept the empty set \varnothing and the empty string "" as regular expressions.

Regular expressions are used to match sets of strings (i.e., languages over Σ\Sigma). e1e2e_1e_2 matches any string that is a concatenation of a string matched by e1e_1 with a string matched by e2e_2. The regular expression e1|e2e_1|e_2 matches anything matched by e1e_1 or e2e_2. Finally (e1)*(e_1)* matches any finite concatenation of strings that e1e_1 matches (including no repetitions).

  1. Let Σ={a,b,c,,z}\Sigma = \{a,b,c,\ldots,z\}. What strings does the regular expression pet(dog|cat) match?

  2. What strings does the regular expression pet(dog|cat|bird)* match?

  3. Find a regular expression over the alphabet Σ={0,1}\Sigma = \{0,1\} that matches all strings that start with a 1, end with a 1, and have an even number of zeros between.

Wednesday, September 13

Today we mentioned the following theorem:

Theorem. A language is regular if and only if it can be represented by a regular expression.

This means that NFAs, DFAs, and RegEx’s are all equivalent to each other in terms of what they can compute. Then we looked at how to convert NFAs to RegEx’s and vice versa. We did the following examples.

  1. Use the ideas from last week about how to build NFAs to find the union, concatenation, and Kleene star of languages to convert to following regular expression (ab|a)* over the alphabet Σ = {a,b} to an NFA.

Then we described an algorithm for converting NFAs to regular expressions. Note that the Maheshwwari & Smid textbook describes a different approach to convert an NFA into a regular expression in Section 2.8. We did the following example:

  1. Let Σ={A,C,G,T}. Convert the following NFA to a regular expression:

We finished today by talking about some of the short-hand and quirks in regular expression libraries for some programming languages.

Friday, September 15

Today we talked about the pumping lemma. We used the pumping lemma to prove that the following languages over Σ = {0,1} are not regular:

  1. L={w:w has an equal number of 0 and 1’s}L = \{w : w \text{ has an equal number of 0 and 1's}\}

  2. L={ww:w0,1*}L = \{ww : w ∈ {0,1}^* \}

  3. L={0n2:n}L = \{0^{n^2} : n \in \mathbb{N}\}.

Here is a nice meme proof using the pumping lemma from Barak textbook.

Week 5 Notes

Day Section Topic
Mon, Sep 18 2.9 Non-regular languages
Wed, Sep 20 Review
Fri, Sep 22 Midterm 1

Monday, September 18

Today we looked at more examples of regular and non-regular languages.

  1. L={0n2:n}L = \{0^{n^2} : n \in \mathbb{N}\}.

  2. L={w:w contains an equal number of 01 and 10 substrings}L = \{w : w \text{ contains an equal number of } 01 \text{ and } 10 \text{ substrings}\}.

Many programming languages, including Python & Javascript allow backreferences to previous groups in a regular expression. A group is a part of the regular expression inside parentheses. The special symbol \1 refers to anything matched by the first group in the regular expression. Similarly \2 refers back to anything matched by the second group, and so on. For example: the regular expression "([a-z]*) \1" would match “word word” or “dog dog”.

  1. Explain why regular expressions with backreferences are not really regular expressions (at least not according to our definition).

We finished with this last example. Consider the language F={aibjck:i,j,k0 and if i=1 then j=k}.F = \{a^i b^j c^k : i, j, k \ge 0 \text{ and if } i = 1 \text{ then } j = k\}.

  1. Explain why there is a pumping number pp (in fact p=1p=1 works) such that any string sFs \in F can be “pumped”.

  2. Despite this, explain why FF is not a regular language.

  3. Why doesn’t that contradict the pumping lemma?


Week 6 Notes

Day Section Topic
Mon, Sep 25 3.1 - 3.2 Context-free grammars
Wed, Sep 27 3.3 Parsing and parse-trees
Fri, Sep 29 3.8 The pumping lemma for context-free languages

Monday, September 25

Today we introduced context-free grammars. We started with some simple examples from Wikipedia (https://en.wikipedia.org/wiki/Context-free_grammar#Examples) to illustrate the definition.

  1. We found a context free grammar that generates the (non-regular) language L={anbn:n}L = \{a^n b^n : n \in \mathbb{N}\}.

  2. We looked at the example of a CFG and used it to identify the parts (V,Σ,R,S)(V,\Sigma, R, S) in the formal definition.

    S         →  <subject> <verb>
    <subject> →  <article> <noun>
    <article> →  a | the
    <noun>    →  boy | girl | dog
    <verb>    →  runs | jumps | swims
  1. Find a context-free grammar for the language {w{a,b}*:w has an equal number of a’s and b’s}\{w \in \{a,b\}^* : w \text{ has an equal number of } a\text{'s and } b \text{'s} \}.

  2. Describe how to generate the string (a + a) * a using the context free grammar below:

    <expr>    →  <expr> + <term> | <term>
    <term>    →  <term> * <factor> | <factor>
    <factor>  →  (<expr>) | a

We finished by looking at this example that gives the idea of how a programming language can be though of as a context-free language.

Wednesday, September 27

Today, we briefly went over the first midterm exam. I talked about the grading scale and went over some of the problems .

Today we spent more time discussing context free grammars. We talked about left derivations and right derivations

We looked at this example:

EE+E|E*E|(E)|aE \rightarrow ~E+E~ | ~E*E~ | ~(E)~ | ~a~

  1. Use this CFG to derive the string a+a*aa+a*a. Try to use a left derivation. Then use a right derivation.

  2. Show that that string actually has more than one left derivation.

A grammar is ambiguous if there are strings with more than one left derivation. If every left-derivation is unique, then the grammar is unambiguous. Note that some CFGs have unambiguous grammars, but so do not. This alternate grammar generates the same language as our first example, but is unambiguous:

    <expr>    →  <expr> + <term> | <term>
    <term>    →  <term> * <factor> | <factor>
    <factor>  →  (<expr>) | a

We looked at this ambiguous grammar.

  1. Show that the sentence “the girl touches the boy with the flower” has two left derivations and those derivations correspond to two different meanings this sentence can have.

Friday, September 29

Today we discussed the pumping lemma for context-free languages. This is a little more complicated than the pumping lemma for regular languages. We drew some pictures of parse trees to give an intuitive explanation for why this new pumping lemma is true (see page 126 in Maheshwari & Smid). In particular, we looked at how the string a+a*a gets generated by the grammar:

    E →  E + T | T
    T →  T * F | F
    F →  (E) | a

In the parse tree, there is a branch where the variable T gets repeated, and that lets us “pump”.

We used the pumping lemma to prove that the following languages are not context-free:

  1. L={anbncn:n}L = \{a^n b^n c^n : n \in \mathbb{N}\}.

  2. L={an2:n}L = \{a^{n^2} : n \in \mathbb{N}\}.

  3. L={w=w:w{a,b}*}L = \{w = w : w \in \{a, b\}^* \}.

We also talked about how there are always an uncountable number of languages over a finite alphabet Σ\Sigma (since the set of languages is the power set of Σ*\Sigma^*), and a computer can only compute a countable set of languages, so it is not surprising that most languages are not context-free.


Week 7 Notes

Day Section Topic
Mon, Oct 2 3.5 - 3.6 Pushdown automata
Wed, Oct 4 3.7 Pushdown automata & context-free grammars
Fri, Oct 6 4.1 - 4.2 Definition of Turing machines

Monday, October 2

Today we talked about nondeterministic pushdown automata (NDPA) which are basically just NFAs with a stack. We started with this example:

This NDPA accepts the language L={anbn:n}L = \{a^n b^n : n \in \mathbb{N}\}. Notice that each arrow has three parts, for example the first loop is labeled a,ϵ/aa, \epsilon/a which means you can take this loop if you read an aa from the input string, and pop ϵ\epsilon (i.e., nothing) from the stack, and then push aa onto the stack. We will always use this notation (read, pop/push) for each arc.

NDPAs work just like NFAs, except they have this extra rule: You only accept a string if you finish in an accepting state and the stack is empty.

Some things to note:

  1. Since the machine is nondeterministic, you can have more than one state (and more than one stack) branching off whenever you have a choice about which arrow to take.

  2. If you are in a state and there is no valid arrow to take, then that state dies.

We looked at these questions:

  1. For the NPDA below, describe the language that it accepts.

  2. Describe an NPDA that accepts the language L={anbm:nm}L = \{ a^n b^m : n \ge m \}.

  3. Describe an NPDA with just one state that accepts the language of balanced parentheses.

Wednesday, October 4

Today we talked about the equivalence between CFGs and NPDAs. We sketched the proof of one direction: that if you have a CFG, then there is an NPDA that accepted the same language.

The idea of the proof is to create an NPDA with three states. In state q0q_0 you immediately transition to q1q_1 reading nothing and push S$ on the stack (where $ is a special symbol to mark the bottom of the stack). Then in state q1q_1 you have two different types of transitions that loop back to q1q_1:

  1. Transitions where you read nothing and pop a variable off the stack, then push the output of one of the grammar rules for that variable back onto the stack. (I called these red transitions.) You have one red transition for each of the rules in the grammar.

  2. Transitions where you read a symbol from the input and pop that symbol off the stack. (I called these blue transitions.) You have one blue transition for each of the terminals in the grammar.

Finally, you have a final transition to the accepting state q2q_2 which involves reading nothing and popping the $ off the stack. If you have finished reading the input when you do this, you will accept the string.

We illustrated how this idea works using the grammar and the input string aabbbb.

    S →  AB
    A →  aAb | ε
    B →  bB | b

We finished by describing how Deterministic Pushdown Automata (DPDAs) are different from Nondeterministic Pushdown Automata (NPDAs) and the hierarchy of different languages that we have seen so far. We looked at this example DPDA:

Friday, October 5

Today we introduced Turing machines (TMs) which will be the most general model of computation that we will discuss. We looked at two examples of TMs:

  1. First we drew a state diagram for a TM that accepts the language {w=w:w{a,b}*}\{w=w : w \in \{a,b\}^* \}.

  2. Second we described an algorithm for a TM that accepts the language {a2n:n0}.\{a^{2^n} : n \ge 0\}.

We defined what it means for a Turing machine to accept or recognize a language if it accepts every string in the language. We also defined when a Turing machine decides a language, and how that is different than just accepting/recognizing. Both of the example Turing Machines above actually decide their languages, since they will successfully reject any input that doesn’t match a valid string (they won’t get stuck running forever without halting).


Week 8 Notes

Day Section Topic
Mon, Oct 9 4.2 Turing computable functions
Wed, Oct 11 4.3 - 4.4 Church-Turing thesis
Fri, Oct 13 5.5 - 5.7 Enumerators

Monday, October 9

Today we started with this example:

  1. Construct a Turing machine that reads an input string w{a,b}*w \in \{a,b\}^* and leaves wwww on the tape when it halts.

This wasn’t to hard to describe the algorithm, but it got a little complicated when we made the state diagram. Once we did that, we defined Turing computable functions.

Definition. A function f:Σ*Σ*f: \Sigma^* \rightarrow \Sigma^* is Turing computable if there is a Turing machine that halts with f(w)f(w) on the tape whenever it is given a tape with wΣ*w \in \Sigma^* as input.

  1. Explain why the function f(w)=wwwwf(w) = wwww is Turing computable.

  2. Prove that if f,g:Σ*Σ*f, g : \Sigma^* \rightarrow \Sigma^* are both Turing computable functions, then so is the composition fgf \circ g.

  3. Explain why the function f(w)={1 if length of w is even.0 otherwise.f(w) = \begin{cases} 1 & \text{ if length of } w \text{ is even.} \\ 0 & \text{ otherwise.} \end{cases} is Turing computable. Hint: Every regular language is Turing decidable.

We finished by describing how this function (which comes up in the Collatz-Conjecture) is Turing computable: f(n)={n/2 if n is even,3n+1 if n is odd.f(n) = \begin{cases} n/2 & \text{ if } n \text{ is even,} \\ 3n+1 & \text{ if } n \text{ is odd.} \end{cases}

Wednesday, October 11

Today we mentioned but did not prove that any problem that a multi-tape Turing machine can solve, can also be solved by a single-tape Turing machine. You can find a readable proof in Section 4.3 in the book.

Instead, we summarized what we know so far:

  1. Turing machines are able to decide any regular or context-free language. That is, if LΣ*L \subset \Sigma^* is a context-free language, then the function f:Σ*{0,1}f: \Sigma^* \rightarrow \{0,1\} defined by f(w)=1f(w) = 1 if and only if wLw \in L is Turing computable.

  2. If ff and gg are Turing computable functions, then so is the composition fgf \circ g.

  3. If f:Σ*{0,1}f: \Sigma^* \rightarrow \{0,1\}, and g,h:Σ*Σ*g, h: \Sigma^* \rightarrow \Sigma^* are Turing computable, then so is the function

    IF f(x) THEN g(x) ELSE h(x).

  4. So here is a question: Can a Turing machine have a for-loop, where it repeats some function nn times? Something like this Python example where function ff is applied nn times?

for i in range(n): 
    x = f(x)
  1. What about a while-loop?
while(f(x)):
    x = g(x)

I think we get the idea that you can implement pretty much all of the basic building blocks of most programming languages using Turing machines. There are many other models of computing that can be proven to be Turing complete, that is, they can compute the same functions as Turing machines. But it is believed that there is no “effective method” of computation that can solve problems that Turing machines cannot solve. The Church-Turing thesis says that an computational task that we can describe how to solve with an algorithm can be computed by a Turing machine.

We finished by discussing Hilbert’s 10th problem: Is there an algorithm to determine whether a multivariable polynomial with integer coefficients has an integer root. We looked at some example polynomials like p(x,y)=(x1)2+(y3)2=x2+y22x6y+10p(x,y) = (x-1)^2 + (y-3)^2 = x^2 + y^2 - 2x - 6y + 10 and q(x,y)=x2+y22x6y+11.q(x,y) = x^2 + y^2 - 2x - 6y + 11. It turns out that no Turing machine can decide the language of integer multivariable polynomials with integer roots. But we did describe an algorithm that recognizes that language.

Friday, October 13

Today we introduced enumerators. An enumerator for a language LΣ*L \subseteq \Sigma^* is a Turing machines that computes a function f:Lf: \mathbb{N}\rightarrow L that is onto. We started by proving the following:

“Easy” Lemma. If LΣ*L \subset \Sigma^* is decidable, then there is an enumerator for LL.

To prove this lemma, we came up with the following algorithm. In class I didn’t include the just_started variable which caused a minor issue in the proof.

input n
# use a decider function to return the n-th string in the language
s = ""
for i in range(n):
    just_started = True
    while just_started or not decider(s):
        just_started = False
        increment(s)
return s

“Harder” Theorem. A language LΣ*L \subset \Sigma^* is recognizable if and only if there is an enumerator for LL.

It was relatively easy to prove the ()(\Leftarrow) direction as this algorithm demonstrates:

input w
# use an enumerator function to return True if w is in L
n = 0
while not w == enumerater(n):
    n++
return True

The other direction is harder to prove. We sketched a proof at the board, but the important thing to understand is this: Why doesn’t the algorithm for the “Easy Lemma” prove the ()(\Rightarrow) case of the “Harder Theorem”?


Week 9 Notes

Day Section Topic
Wed, Oct 18 B9.1 Universal Turing machines
Fri, Oct 20 5.1-5.3 The halting problem & Rice’s theorem

Wednesday, October 18

Today we introduced universal Turing machines. We use them to consider this language:

Accept={M,w:M is a Turing machine and w is a string that M accepts}.\text{Accept} = \{ \langle M, w \rangle : M \text{ is a Turing machine and } w \text{ is a string that } M \text{ accepts}\}.

  1. Explain why Accept is Turing recognizable.

Then we gave a diagonalization proof that Accept is not Turing decidable. Suppose there was a decider for Accept. Then we could make a table with rows for each Turing machine MiM_i and columns every Turing machine encoding Mj\langle M_j\rangle and entries 1 or 0 depending on whether MiM_i accepts Mj\langle M_j \rangle.

  1. What would happen if we created a TM NN inputs any TM encoding M\langle M \rangle and then does the opposite of what DD does on M,M\langle M, \langle M \rangle \rangle. What would NN do when you input N\langle N \rangle into NN?

  2. Why is that a contradiction?

We finished by observed that there are only a countably infinite number of possible Turing machines, but there is an uncountable infinite number of languages, so there must be lots of languages that are not decidable!

Friday, October 20

Today we proved that the language

Halt={M,w:M is a TM that halts on w}\text{Halt} = \{ \langle M, w \rangle : M \text{ is a TM that halts on }w \}

is undecidable. Here is the proof:

If Halt were decidable, then we could use the following algorithm to decide if M,wAccept\langle M, w \rangle \in \text{Accept}:

  1. Check if M,wHalt\langle M, w \rangle \in \text{Halt}.
  2. If it is, run MM on ww and accept or reject accordingly.
  3. If not, then reject.

Since this algorithm would decide the language Accept from last time, but we proved that the language Accept is undecidable, we conclude that Halt must also be undecidable.

Then we proved Rice’s theorem (our book also has a proof in Section 5.3, my proof was shorter, but I glossed over some of the details). We used Rice’s theorem to prove that these two languages are undecidable.

  1. Infinte={M:M is a TM and L(M) is infinite}\text{Infinte} = \{ \langle M \rangle : M \text{ is a TM and } L(M) \text{ is infinite}\}.

  2. Empty={M:M is a TM and L(M)=}\text{Empty} = \{ \langle M \rangle : M \text{ is a TM and } L(M) = \varnothing \}.


Week 10 Notes

Day Section Topic
Mon, Oct 23 Review
Wed, Oct 25 Midterm 2
Fri, Oct 27 no class

Monday, October 23

Today we reviewed that material from CFGs and NPDAs through TMs. We did some examples:

  1. Show that there is no CFG or NPDA that can generate/accept this language: {ww:w{a,b,c}*}\{w w : w \in \{a,b,c\}^*\}.

  2. Find a NPDA that accepts the language {(ab)ncn:n}\{(ab)^n c^n : n \in \mathbb{N}\}.

  3. Use Rice’s theorem to prove that the language {M:M is a TM that accepts 111}\{\langle M \rangle : M \text{ is a TM that accepts } 111\} is undecidable.

  4. Why can’t you use Rice’s theorem to prove that the language {M:M is a TM with 3 states}\{\langle M \rangle : M \text{ is a TM with 3 states}\} is undecidable?

  5. Explain why the complement of the language HALT={M,w:M is a TM that accepts w}\text{HALT} = \{\langle M, w \rangle : M \text{ is a TM that accepts } w \} is uncomputable (i.e., not even recognizable).


Week 11 Notes

Day Section Topic
Mon, Oct 30 6.1 The running time of algorithms
Wed, Nov 1 6.2 The complexity class P
Fri, Nov 3 6.2 The complexity class P - con’d

Monday, October 30

Today we introduced big-O notation and used it to talk about running times of algorithms. Note that There are lots of important examples in Barak Chapter 12 of algorithms and their running times.

  1. Which is true? (logn)3O(n)(\log n)^3 \in O(n) or n(logn)3n \in (\log n)^3?

We proved the following facts about big-O notation:

Theorem (Only Dominant Terms Matter). If f:[0,)f: \mathbb{N}\rightarrow [0,\infty) is a sum of nonnegative functions f1,,fkf_1, \ldots, f_k, then there is at least one function fjf_j such that O(f)=O(fj)O(f) = O(f_j). We call that function a dominant term for ff.

Theorem (Constant Multiples Don’t Matter). For any f:[0,)f: \mathbb{N}\rightarrow [0,\infty) and constant c>0c > 0, O(cf)=O(f)O(cf) = O(f).

Theorem (Products of Functions). If f1O(g1)f_1 \in O(g_1) and f2O(g2)f_2 \in O(g_2), then f1f2O(g1g2)f_1f_2 \in O(g_1 g_2).

We also reviewed this heirarchy of functions:

logarithmspolynomialsexponentialsfactorials\text{logarithms} \ll \text{polynomials} \ll \text{exponentials} \ll \text{factorials}

We used these ideas to find the dominant term for these expressions:

  1. n4+n3logn+(logn)5n^4 + n^3 \log n + (\log n)^5.

  2. n100+2nn^{100} + 2^n.

So why do we use big-O notation? It makes it easier to talk about a lot of formulas. For example, an algorithm that requires 7(n+3)n!k!(nk)!+1677(n+3) \dfrac{n!}{k!(n-k)!}+167 steps can be described simply as being O(nk+1)O(n^{k+1}).

Caution 1. Big-O notation only gives an upper bound for how fast functions grow. Functions in O(f)O(f) don’t have to grow at the same rate as ff, they can also grow much slower.

Caution 2. You should always write the simplest possible expression inside big-O notation. For example O(5n3+logn)=O(n3)O(5n^3 + \log n) = O(n^3), so just write O(n3)O(n^3).

We finished by talking about how we use Big-O notation to describe the runtime of algorithms such as:

  1. The TM from Midterm 2 that decrements a binary string by 1.

  2. Testing whether a number NN \in \mathbb{N} (represented in binary) is prime.

Wednesday, November 1

Today we talked about the running times of some algorithms. We started with this example:

  1. Prove that the language {w{0,1}*:w is divisible by seven}\{w \in \{0,1\}^* : w \text{ is divisible by seven} \} can be decided in O(n)O(n) time.

We defined class P which is the set of all decidable languages LL for which there exists a Turing machine MM and a constant kk \in \mathbb{N} such that MM can decide LL in O(nk)O(n^k) time.

  1. Prove that if K,L{0,1}*K, L \subset \{0,1\}^* are both in P, then the concatenation KLKL is also in P.

  2. Consider the PATH problem which involves deciding this language: PATH={G,s,t:G is a graph containing a path from s to t}.\text{PATH} = \{ \langle G, s, t \rangle : G \text{ is a graph containing a path from }s \text{ to } t \}. Prove that the following algorithm runs in polynomial time.

One important fact that I forgot to mention in class is the following:

Fact. If a Turing machine can decide a language in polynomial time, then so can a Python or C++ program.

Friday, November 3

Today we proved that all Context Free Languages (CFLs) are in class P. Before we proved this, we introduced the idea of Chomsky’s normal form. We looked at this example CFG

SaaS|ϵS \rightarrow aaS \,|\, \epsilon

and we observed that the following grammar in Chomsky normal form is equivalent, i.e., it generates the same language.

SAA|AB|ϵAaBACCAB|AA\begin{align*} S &\rightarrow AA \,|\, AB | \epsilon \\ A &\rightarrow a \\ B &\rightarrow AC \\ C &\rightarrow AB \,|\, AA \end{align*}

We did not go into the details of how to convert a CFG to Chomsky normal form, but you can find details of how to do that in Section 3.4 in Maheshwari & Smid. Instead, we did the following workshop which let us figure out the running time of an algorithm to decide whether a string of length nn can be generated by a grammar in Chomsky normal form.


Week 12 Notes

Day Section Topic
Mon, Nov 6 6.3 The complexity class NP
Wed, Nov 8 6.4 Nondeterministic TMs
Fri, Nov 10 6.5 The SAT problem

Monday, November 6

Today we introduced class NP. A language is in class NP if there is a solution string ss for every wLw \in L and an algorithm called a verifier that accepts w,s\langle w, s \rangle in polynomial time in the length of wLw \in L. We looked at the following examples:

  1. COMPOSITE ={w{0,1}*:w is a binary number that is not prime}= \{ w \in \{0,1\}^* : w \text{ is a binary number that is not prime}\}.

  2. TRIANGLE ={G:G is a graph with a complete subgraph with 3 vertices}= \{\langle G \rangle : G \text{ is a graph with a complete subgraph with }3\text{ vertices}\}.

In each of these examples, you need to prove that if you are given a special string called the solution, then you can find a polynomial time algorithm that will use the solution to verify that a string ww is in the language.

We also proved this theorem:

Theorem. P \subseteq NP.

Along the way, we mentioned some interesting new results:

It is still unknown whether P \ne NP.

Wednesday, November 8

Today we talked about what the name NP stands for. It refers to problems that are solvable in polynomial time by a nondeterministic Turing machine. We defined nondeterministic Turing machines, and then we proved the following:

Theorem. Any language that can be decided (or recognized) by a nondeterministic TM can be decided (recognized) by a TM.

Theorem. A language LL is in class NP if and only if there is a nondeterministic TM that decides LL in polynomial time.

Friday, November 10

Today we introduced the Boolean satisfiability problem (SAT). Recall that a Boolean formula/circuit (involving AND, OR, and NOT gates) can evaluate any function f:{0,1}n{0,1}f:\{0,1\}^n \rightarrow \{0,1\}. For example, ϕ=(xy)(xz)\phi = (\bar{x} \wedge y) \vee (x \wedge \bar{z}) is a Boolean formula/circuit. The Boolean satisfiability problem involves deciding if a Boolean formula can be satisfied. That is, if there are values of the input variable that can make the output TRUE. In other words, we want to decide if a Boolean formula ϕ\phi is in the language SAT={ϕ:ϕ is a satisfiable Boolean formula}.SAT = \{ \langle \phi \rangle : \phi \text{ is a satisfiable Boolean formula} \}.

  1. Give an example of a Boolean formula that is not satisfiable.

  2. Prove that SAT is decidable.

A Boolean formula is in conjuctive normal form (CNF) if it combines a finite number of clauses made by connecting literals (either variables or negated variables) with AND statements, and then the clauses are all joined together by OR statements. For example
ϕ=(xy)(xz)\phi = (\bar{x} \wedge y) \vee (x \wedge \bar{z}) is in CNF.

An important special case of the SAT problem is the 3SAT problem, which only looks at Boolean formulas in CNF where the clauses each involve only 3 literals. So a 3-CNF formula would be something like: (x1x2x3)(x3x5x6)(x3x6x4)(x4x5x6).(x_1 \vee \bar{x_2} \vee x_3 ) \wedge (\bar{x_3} \vee x_5 \vee x_6 ) \wedge (\bar{x_3} \vee x_6 \vee \bar{x_4}) \wedge (x_4 \vee x_5 \vee x_6 ).

  1. If a Boolean formula has nn-variables, then how many different 3-clauses are possible?

Instead of using a brute force algorithm to decide 3SAT, we will take a different approach and prove that solving the 3SAT problem is not much harder than deciding this language:

CLIQUE={G,k:G is a graph with a k-clique}.\text{CLIQUE} = \{ \langle G, k \rangle : G \text{ is a graph with a } k\text{-clique} \}.

  1. Prove that CLIQUE is in class NP.

The key to link 3SAT and CLIQUE is to draw a graph where every vertex corresponds to a literal in one of the clauses, and we draw edges according to the following rules:

Then, if a Boolean formula in 3-CNF has kk-clauses, then it is decidable iff the corresponding graph has a kk-clique (as the following image illustrates):

3-satisfiability

Week 13 Notes

Day Section Topic
Mon, Nov 13 6.5 Polynomial time reductions
Wed, Nov 15 6.5 Hamilton path problem
Fri, Nov 17 6.5 NP-Complete languages
Mon, Nov 20 TBA

Monday, November 13

Today we introduced polynomial time reductions.

Definition. A language AA is polynomial-time reducible (or polyreducible) to language BB (denoted APBA \le_P B) if there is a function f:Σ*Σ*f: \Sigma^* \rightarrow \Sigma^* such that

  1. fFPf \in \text{FP}, i.e., ff can be computed in polynomial time, and
  2. wAw \in A if and only if f(w)Bf(w) \in B.

We started by proving that SAT P\le_P 3SAT. To prove that we considered the following transformation:

For any clause (x1x2x3x5x6x8)(x_1 \vee \bar{x}_2 \vee x_3 \vee \bar{x}_5 \vee x_6 \vee x_8), you can replace the last two literals by a dummy variable d1d_1 and then add another clause with d1\bar{d}_1 and the two literals you removed.

  1. Explain why this transformation will preserve whether or not a boolean formula in CNF is satisfiable.

  2. Explain why this transformation can be computed in polynomial time.

We looked at this theorem:

Theorem. If APBA \le_P B and BB is in class-P, then so is AA.

  1. Give an algorithm proof for this theorem.

We also proved that 3SAT P\le_P CLIQUE by looking at how long it takes to construct the graph in the example from last class. We finished by defining NP-complete.

Wednesday, November 15

Today we went over Homework 10. We didn’t cover any new material, but we did introduce the following language:

FIB={w[09]*:w is a decimal representation of a Fibonacci number}.\text{FIB} = \{w \in [0-9]^* : w \text{ is a decimal representation of a Fibonacci number} \}.

So FIB contains {1,2,3,5,8,13,21,34,55,}\{1,2,3,5,8,13,21,34,55,\ldots\}.

Then we asked whether the following decimal strings are in FIB*^*:

We also determined that FIB \in P since the following algorithm runs in less than O(n)O(n) time:

Fib = [1,1]
while w < Fib[-1]:
    Fib.append(Fib[-2]+Fib[-1])
if w == Fib[-1]:
    return true
else: 
    return false

Friday, November 17

Today I started by announcing this extra credit project. You can do it on your own, or bring your laptop to class on Monday and we’ll work on it then.

Then we introduced the classes NP-complete and NP-hard.

Definition. A language LL is NP-complete if

  1. LNPL \in \text{NP}, and
  2. Every other NP language AA has a polynomial-time reduction to LL. (i.e., APLA \le_P L).

Any language satisfying property 2 is called NP-hard whether or not it satisfies property 1.

We prove that the relationship P\le_P (polynomial-time reducibility) is transitive, and used that to prove this theorem:

Theorem. If BPCB \le_P C and BB is NP-Hard, then so is CC.

  1. Prove that if APBA \le_P B and BPB \in \text{P}, then APA \in \text{P}.

We finished by talking about the Cook-Levin theorem

Theorem (Cook-Levin). SAT is NP-complete.

We sketched a proof. First we observe that for any ANPA \in \text{NP} there is a polynomial time verifier VV that can decide if wAw \in A with the help of a solution string ss. Assuming that both ww and ss are encoded in {0,1}*\{0,1\}^*, we can reduce the question of whether wAw \in A to a question about whether there is a solution s{0,1}nks \in \{0,1\}^{n^k} such that VV accepts w,s\langle w, s \rangle. If we think of ww as fixed, then the question of whether there is a solution is equivalent to asking whether the Boolean function

f(s)={1 if V accepts w,s0 otherwise.f(s) = \begin{cases} 1 & \text{ if } V \text{ accepts } \langle w, s \rangle \\ 0 & \text{ otherwise.} \end{cases}

is satisfiable.

Since that question is a Boolean function f:{0,1}nk{0,1}f: \{0,1\}^{n^k} \rightarrow \{0,1\}, there is a Boolean formula involving AND, OR, and NOT for ff. We just have to prove that we can construct that formula in polynomial time.

To prove that you can construct a formula for ff in polynomial time, observe that VV only has a finite number of states which can be described by a state string q{0,1}mq \in \{0,1\}^m. There is also exactly one position on the tape that the head is pointing to at a time. Call that position the head variable hh. Since at each step VV reads the head variable, writes a new value for the head variable and then moves to a new state, while moving the head variable left or right on the tape, you can create a Boolean formula to simulation the action of one step of VV on qq and hh in constant time. Then each step will correspond to a different Boolean formula with the same state variable qq but possibly different head variables. After O(nk)O(n^k) steps, the verifier will accept or reject, so we can create a Boolean formula to simulate VV in O(nk)O(n^k) time.

Monday, November 20

Today we talked about homework 11. We also talked about the extra credit project. In particular, I suggested that the following hint might help write an algorithm to solve problem 3 from homework 10:

The algorithm to decide if wL*w \in L^* only needs to loop through substrings of ww that start at the beginning of ww.

To decide if a substring that starts at the beginning of ww is in L*L^*, just check two things:


Week 14 Notes

Day Section Topic
Mon, Nov 27 review
Wed, Nov 29 Midterm 3
Fri, Dec 1
Mon, Dec 4

Monday, November 27

Today we focused on review for the midterm on Wednesday. The midterm will cover the following topics:

  1. Factoring large integers is hard (it can be done in nondeterministic polynomial time, but most computer scientists believe it cannot be done in polynomial time). For example, the integer 82264459 factors as (9049)(9091). Explain the difference between a function that factors an integer and a function that can verify if a factorization is correct. What would the two functions input? Why would the later function definitely run in polynomial time even if the first function cannot?

  2. A language is in class EXP if it can be decided in O(2(nk))O(2^{(n^k)}) time for some integer kk. Prove that NPEXP\text{NP} \subseteq \text{EXP}. Note: It is an open problem whether or not NP=EXP\text{NP} = \text{EXP}.

  3. Explain why the problem of finding the median entry in an array of integers is polynomial time reducible to the problem of sorting an array of integers.

We also reviewed some of questions from HW11. In particular, we talked about problem #4 and how you can write a polynomial time nondeterminsitic algorithm by including a random number generator in your pseudocode. Then if any of the random results accepts the input, then the whole algorithm does.

Friday, December 1

Today we reviewed the Chomsky heirarchy of languages, and we looked at how the different complexity classes (P, NP, EXP) fall in that heirarchy. We also review problem 2 from the midterm. We also did the following exercise in class:

  1. Prove that HALT={M,w:M halts on input w}\text{HALT} = \{ \langle M, w \rangle : M \text{ halts on input } w \} is NP-Hard.

Monday, December 4

We reviewed some of the material we haven’t seen in a while. We looked at these questions.

  1. Consider the regular expression 1(0|1)*(+|-)1(0|1)*. What language does this regular expression represent? Give a simple description of the strings in that language.

  2. Create an NFA that recognizes that language.

  3. Since the language is regular, it is also context free. Create a context free grammar that generates the language.

  4. Consider the language L={ww=0:w is any binary number}L = \{ w - w = 0 : w \text{ is any binary number}\}. Prove that this language is not context free. (Use the pumping lemma for CFLs).

  5. Show that the language LL from the previous problem is decidable by describing a Turing machine algorithm that can decide LL. What is the run-time for that algorithm in big-O notation?

  6. What is an example of a language that is context free but not regular?

  7. Describe an NPDA that recognizes your language from the previous question.