\input{../../cs446.tex}
\usepackage{graphicx,amssymb,amsmath}
\sloppy
\newcommand{\ignore}[1]{}

\oddsidemargin 0in
\evensidemargin 0in
\textwidth 6.5in
\topmargin -0.5in
\textheight 9.0in

\begin{document}

\assignment{Spring 2017}{$1$}{Jan  $25^{th}$, 2017}{Feb $3^{rd}$, 2017}

\begin{footnotesize}
\begin{itemize}
\item Feel free to talk to other members of the class in doing the homework.  I am
more concerned that you learn how to solve the problem than that you
demonstrate that you solved it entirely on your own.  You should, however,
write down your solution yourself.  Please try to keep the solution brief and
clear.

\item Please use Piazza first if you have questions about the homework.
  Also feel free to send us e-mails and come to office hours.

\item Please, no handwritten solutions. You will submit your solution 
  manuscript as a single pdf file. Please use \LaTeX to typeset your solutions.
  We have provided resources to get you started including homework templates. 
  In addition, you also have to submit two output files for question 3.

\item Please present your algorithms in both pseudocode and English.  That is, give
a precise formulation of your algorithm as pseudocode and {\em also} explain
in one or two concise paragraphs what your algorithm does.  Be aware that
pseudocode is much simpler and more abstract than real code.

\item The homework is due at 11:59 PM on the due date. We will be using
Compass for collecting the homework assignments. Please submit your solution manuscript as a pdf file via Compass 
(\texttt{http://compass2g.illinois.edu}). Please do NOT hand in a hard copy of your write-up.
Contact the TAs if you are having technical difficulties in 
submitting the assignment. 
\end{itemize}
\end{footnotesize}


\begin{enumerate}

\item[1.] [Learning Conjunctions - 40 points] (Based on Mitchell, exercise 2.9)\\
Consider a learning problem where each instance is a Boolean vector over $n$
variables $(x_1, \ldots, x_n)$ and is labeled either positive (1) or negative
(-1).  Thus, a typical instance would be
\begin{displaymath}
(1, 0, \ldots, 1) = (x_1 = 1, x_2 = 0, \ldots, x_n = 1)
\end{displaymath}
Now consider a hypothesis space ${\mathbf H}$ consisting of all {\em
  Conjunctions} over these variables.  For example, one hypothesis
could be
\begin{displaymath}
(x_1 = 1 \land x_5 = 0 \land x_7 = 1) \; \equiv \; (x_1 \land \neg x_5 \land x_7)
\end{displaymath}
For example, an instance $(1, 0, 1, 0, 0, 0, 1, \ldots, 0)$ will be labeled
as positive (1) and $(0, 0, 1, 0, 1, 1, 0, \ldots, 1)$ as negative (-1),
according to the above hypothesis.

\begin{enumerate}
\item[a.]
Propose an algorithm that accepts a sequence of labeled training examples
and outputs a hypothesis $h \in \mathbf{H}$ that is consistent with all the training examples,
if one exists.  If there are no consistent hypotheses, your algorithm should
indicate as such and halt. [10 points]

\item[b.]
Prove the correctness of your algorithm.  That is, argue that the hypothesis
it produces labels all the training examples correctly. [10 points]

\item[c.]
Analyze the complexity of your algorithm as a function of the number of
variables $(n)$ and the number of training examples $(m)$.  Your algorithm
should run in time polynomial in $n$ and $m$ (if it fails to, you should rethink your algorithm). [10 points]

\item[d.]  Assume that there exists a conjunction in ${\mathbf H}$
  that is consistent with all the data you will ever see.  Now, you are
  given a new, unlabeled example that is not part of your training
  set.  What can you say about the ability of the hypothesis derived
  by your algorithm in $(a.)$~to produce the correct label for this
  example?  [10 points]
  
\end{enumerate}

\item[2.] [Linear Algebra Review - 10 points] \\
	Assume that $\vec{w}\in \mathbb{R}^n$ and $\theta$
    is a scalar. A hyper-plane in $\mathbb{R}^n$ is the set, $
    \{ \vec{x}: \vec{x} \in \mathbb{R}^n, \vec{w}^T \vec{x} + \theta = 0 \}$.
	\begin{enumerate}
	\item [a.] \textbf{[5 points]} The distance between a point $\vec{x}_0 \in
	\mathbb{R}^n$ and the hyperplane $\vec{w}^T \vec{x} + \theta = 0$
	can be described as the solution of the following optimization problem:
	\begin{align*}
    	\min_{\vec{x}} & \qquad \| \vec{x}_0 - \vec{x} \|^2 \\
		\textrm{subject to } & \quad \vec{w}^T \vec{x} + \theta = 0
	\end{align*}
	However, there is an analytical solution (i.e. closed form solution) to find the distance between
	the point $\vec{x}_0$ and the hyperplane $\vec{w}^T \vec{x} + \theta = 0$. 
	Derive the solution. 
	\item [b.]  \textbf{[5 points]} Assume that we have two hyper-planes,
	$\vec{w}^T \vec{x} + \theta_1 = 0$ and $\vec{w}^T \vec{x} + \theta_2 = 0$. 
	What is the distance between these two hyper-planes?
	\end{enumerate}

\item[3.] [Finding a Linear Discriminant Function via Linear Programming - 50 points] \\
The goal of this problem is to develop a different formulation for the problem
of learning linear discriminant functions and use it to solve the problem of
learning conjunctions and the badges problem
discussed in class.
Some subproblems to question 3 may ask you to compose multiple programs or respond to multiple issues.
Anything you are expected to {\bf program, turn in, or write about in your report} will be in boldface.


\begin{definition}[Linear Program]
A {\em linear program} can be stated as follows:

Let 
$A$ be an $m \times n$ real-valued matrix,
$\vec{b} \in \mathbb{R}^m$, and $\vec{c} \in \mathbb{R}^n$.
Find a $\vec{t} \in \mathbb{R}^n$, that minimizes the linear function
\begin{eqnarray*}
  & & z(\vec{t}) = \vec{c}~^T \vec{t} \\
\textrm{subject to} & & A \vec{t} \geq \vec{b}
\end{eqnarray*}
\end{definition}

In the linear programming terminology, $\vec{c}$ is often referred
to as the {\em cost vector} and $z(\vec{t})$ is referred to as the {\em objective
function}.\footnote{Note that you don't need
to really know how to solve a linear program since we will use Matlab to obtain the
solution (although you may wish to brush up on Linear Algebra). 
Also see the appendix for more information on linear programming.}
We can use this framework to define the problem of learning a linear
discriminant function.\footnote{This discussion closely
  parallels the linear programming representation found in 
  {\em Pattern Classification}, by Duda, Hart, and Stork.}

\textbf{The Learning Problem:}\footnote{Note that the notation used in the
Learning Problem is
{\bf unrelated} to the one used in the Linear Program definition. You may want to
figure out the correspondence.} \hspace{2mm}
Let $\vec{x_1}, \vec{x_2}, \ldots, \vec{x_m}$ 
represent $m$ samples, where each sample $\vec{x_i}\in \mathbb{R}^n$ is an $n$-dimensional
vector, and $\vec{y} \in \{-1, 1\}^m$ is an $m \times 1$
vector representing the respective labels of each of the $m$ samples. Let
$\vec{w} \in \mathbb{R}^n$ be an $n \times 1$ vector representing the weights of the
linear discriminant function, and $\theta$ be the threshold value. 

We {\em predict} $\vec{x_i}$ to be a {\em positive} example if
$\vec{w}^T \vec{x_i} + \theta \geq 0$. On the other hand, we {\em predict}
$\vec{x_i}$ to be a {\em negative} example if $\vec{w}^T \vec{x_i} + \theta < 0$.

We hope that the learned linear function can separate the data set.  
That is, for the true labels we hope
\begin{equation}
\label{eq:separable}
y_i = \begin{cases}
 1 & \mbox{if } \vec{w}^T \vec{x_i} + \theta \ge 0 \\
-1 & \mbox{if } \vec{w}^T \vec{x_i} + \theta < 0. \\
\end{cases}
\end{equation}
%\begin{equation}
%\textrm{if } \left\{
%\begin{array}{cc}
%\vec{w}^T \vec{x_i} + \theta \geq 0, & y_i = 1 \\
%\vec{w}^T \vec{x_i} + \theta < 0, & y_i = -1
%\end{array}
%\right. \label{eq:separable}
%\end{equation}

In order to find a good linear separator, we propose the following linear program:
\begin{eqnarray}
  \min_{\vec{w}, \theta, \delta} & & \delta  \label{eq:lin_prog_discriminant_obj}\\
 \textrm{subject to } & & y_i(\vec{w}^T \vec{x_i} + \theta) \geq 1 - \delta, \qquad \forall (\vec{x_i},y_i) \in D  \label{eq:lin_prog_discriminant_constraint}\\
  & & \delta \geq 0  \label{eq:lin_prog_discriminant_bound}
\end{eqnarray}
where $D$ is the data set of all training examples.
\begin{enumerate}


\item[a.] Understanding linear separability {\em [15 points]} 
  \begin{enumerate}
  \item [a.1] {\em[8 points]} A data set
      $D=\{(\vec{x_i},y_i)\}_{i=1}^m$ that satisfies 
      condition (\ref{eq:separable}) above is called
      {\em linearly separable}. 
      {\bf Show that the data set
      $D$ is linearly separable 
      if and only if there exists
      a hyperplane ($\vec{w'}, \theta'$) that satisfies 
      condition
      \eqref{eq:lin_prog_discriminant_constraint} with $\delta = 0$} 
	  (Need a hint?\footnote{{\bf Hint:} Assume that $D$ is linearly separable. $D$ is a
      finite set, so it has a positive example that is {\em closest}
      to the hyperplane among all positive examples. Similarly, there
      is a negative example that is {\em closest} to the hyperplane
      among negative examples. Consider their distances and use them
      to show that condition
      (\ref{eq:lin_prog_discriminant_constraint}) holds. Then show the
      other direction.
	  }).
      Conclude that $D$ is linearly separable iff the optimal solution
      to the linear program
      [\eqref{eq:lin_prog_discriminant_obj} to \eqref{eq:lin_prog_discriminant_bound}]
      attains $\delta = 0$.
      {\bf What can we say about the linear separability of the data set
      if there exists a hyperplane that satisfies condition
      \eqref{eq:lin_prog_discriminant_constraint} with $\delta > 0$?}

  \item [a.2] {\em [2 points]} {\bf Show that there is a
      trivial optimal solution for the following linear
      program:}
    \begin{eqnarray*}
      \min_{\vec{w},\theta,\delta} & & \delta  \\
%\min_{\delta} & & \delta  \\
      \textrm{subject to } & & y_i(\vec{w}^T \vec{x_i} + \theta) \geq - \delta, \qquad \forall (x_i,y_i) \in D \\
      && \delta \geq 0  \\
    \end{eqnarray*}
	{\bf Show the optimal solution and use it to (very briefly)
	explain why we chose to formulate the linear program as
	[\eqref{eq:lin_prog_discriminant_obj} to \eqref{eq:lin_prog_discriminant_bound}]
	instead.}

\item [a.3] {\em [5 points]} 
Let $\vec{x_1} \in \mathbb{R}^n$, $\vec{x_1}^T =
\begin{bmatrix}
  1 & 1 & \ldots & 1
\end{bmatrix}$ and $y_1 = 1$. 
Let
$\vec{x_2} \in \mathbb{R}^n$, $\vec{x_2}^T =
\begin{bmatrix}
  -1 & -1 & \ldots & -1
\end{bmatrix}$ and $y_2 = -1$.
The data set $D$ is defined as
\begin{equation*}
    D = \{ (\vec{x_1},y_1), (\vec{x_2},y_2)\}.
\end{equation*}
Consider the formulation in
[\eqref{eq:lin_prog_discriminant_obj} to \eqref{eq:lin_prog_discriminant_bound}]
applied to
$D$. {\bf Show the set of all possible optimal solutions (solve this problem by hand)}.

  \end{enumerate}


\item[b.] Solving linear programming problems {\em [35 points]}

	This problem requires developing some Matlab code.
    We have provided you a collection of Matlab files.
    You will need to edit some of these files to answer the following questions.
    {\bf In addition to your answers to the following questions,
    include as part of your solution manuscript the Matlab code you wrote in 
    \begin{itemize}
      \item {\tt findLinearDiscriminant.m}
      \item {\tt computeLabel.m}
      \item {\tt plot2dSeparator.m}
      \item {\tt findLinearThreshold.m.}
    \end{itemize}
    Please do NOT submit these files 
    separately.} A summary of what to submit is listed in the end of this problem.

  \begin{enumerate}
  \item [b.1] {\em [Writing LP - 5 points]} {\bf Rewrite the linear program
	[\eqref{eq:lin_prog_discriminant_obj} to \eqref{eq:lin_prog_discriminant_bound}]
	  into a ``standard form'' linear program (the form used in Definition $1$).}
      %Please write the solution in your document using equations.
      \ \\

      {\bf Now, implement     
      a Matlab function called {\tt findLinearDiscriminant}
      (by editing {\tt findLinearDiscriminant.m})
      to find a linear discriminant function that separates
      a given data set using the linear programming formulation described above} (you can refer to the appendix for a primer on solving linear programs using Matlab).
      Your Matlab function must take {\tt data} as input,
      and produce the linear separator, i.e., 
      the weights {\tt w} ($\vec{w}$), the threshold {\tt theta} ($\theta$), 
      and the value of the objective at the minimum, i.e., {\tt delta} ($\delta$).
      We will read our data from files. 
      A data file consists of lines that contain a sequence of binary values
      followed by either 1, indicating a positive label, or by -1, indicating a negative label.
      A sample line might be ``\verb+0 1 -1+'', which corresponds to
      $\vec{x}=[0,1]^T, y=-1$.
      In order to read files in this format,
      we provided a Matlab function called {\tt readFeatures}
      that takes a file name and the dimension of the feature vector $\vec{x}$ (denoted $n$)
      and returns an $m \times (n+1)$ matrix called {\tt data},
      where $m$ is the number of data points.

  \item [b.2] {\em [Learning Conjunctions as an LP - 10 points]} There are two parts 
    in this question. {\bf First, generate a data set that represents a 
      monotone conjunction\footnote{A conjunction with no negated variables.
      This file should be 4 lines with 3 numbers per line.} 
      over two variables manually. Write this data set into {\tt hw1sample2d.txt}}.
      We provided a Matlab function called {\tt plot2dData} to plot the labeled
      data points given by {\tt data}. 
      {\bf Now, implement
      a Matlab function called {\tt plot2dSeparator}
      (by editing {\tt plot2DSeparator.m}) that takes
      {\tt w} and {\tt theta} as input, and plots the corresponding separator line.
      For your dataset, find the linear separator and plot it together with your data,
      and include the figure in your solution.}
      \ \\

      In addition,
      we provided a data set in {\tt hw1conjunction.txt} that is consistent
      with a conjunctive concept with $n=10$.
      {\bf Use your linear program to learn the target concept in  {\tt hw1conjunction.txt}.
      State the linear discriminant function returned and {\em succinctly}
      explain your results. 
      For example, what conjunctive concept is
      represented by the hypothesis returned, and how can this be known
      from the resulting hyperplane from your function (or can this be
      known). What can be said about $\delta$ and $\theta$? In addition to your
      explanations, you should also give the actual weights, $\delta$ and
      $\theta$ reported by your program.}
      \ \\

      You can use the script {\tt learnConjunctions} to run these two 
      experiments and obtain the materials you need to submit in your answer.
      Note that this script outputs the model of the second experiment in a
      file named {\tt p3b2-model.txt}. {\bf You also need to submit {\tt p3b2-model.txt}}. 
  
\item [b.3] {\em [Learning Badges - 10 points]} 
      You will solve the badges problem using your linear program.
      The input files {\tt badges.train}, and {\tt badges.test}
      contain
      the names of the participants and the badge labels.
      We will consider features of the following form.
      Let $\Sigma$ be the set of characters of interest.
      Let $D$ be the set of character positions of interest.
      For each $(d,\sigma) \in D \times \Sigma$, we have a binary variable
      $x_{(d,\sigma)}$ that is set to $1$ if the $d$-th character of the name is 
      the character $\sigma$ (ignoring case), or to $0$ otherwise.
      The feature vector $\vec{x}$ then has dimensions $|\Sigma||D|$.
      Given $\Sigma$ ({\tt alphabet}) and $D$ ({\tt positions}),
      the Matlab function {\tt writeBadgeFeatures} computes the feature vector
      and writes it to a file using the data file format.
      
      In this part, you will use the Matlab script {\tt learnBadges}.
      It defines an alphabet consisting of 30 characters, and 
      a position list containing the first ten positions.
      This script uses your {\tt findLinearDiscriminant} to obtain a linear separator.
      In order to evalute your result,
      we provided a code to compute the accuracy of this separator both in the training
      set and in the test set.
      The function {\tt computeAccuracy} makes calls to the {\tt computeLabel} function,
      which should predict the label for the given feature vector using the linear separator
      represented by $w$, and $\theta$.
      {\bf In order for accuracy computations to work,
      you need to implement the {\tt computeLabel} function.}
      
      In addition to accuracy, {\tt learnBadges} creates a figure to illustrate the 
      weight vector returned by your linear program.
      The $x-$axis shows the characters, the $y-$axis shows the positions, and the 
      pixel colors at location $(\sigma,d)$ denote the weight corresponding to the feature $x_{(d,\sigma)}$.
      
      {\bf Now, run this script and explain $\delta$, the accuracies, and the generated figure
      in your solution (Be sure to include the figure in your solution).}
      
      Next, experiment with different feature vectors by changing {\tt alphabet} and/or {\tt positions}
      in {\tt learnBadges} script.
      {\bf Find one {\tt alphabet} and {\tt positions} such that the linear separator computed
      from the resulting feature vector has a perfect accuracy ($=1$) in the training dataset,
      but has an imperfect accuracy ($<1$) in the test dataset.
      Give the figure showing the weight vector for this linear separator.
      How does this figure compare to the figure you obtained before?}

\item [b.4] {\em [Learning Multiple Hyperplanes - 10 points]}
      For this part, you will use the Matlab script {\tt learnMultipleHyperplanes}.
      The script generates a data matrix that contains 20 two-dimensional real-valued examples
      with positive and negative labels.
      First, the script finds a linear separator using your linear program.
      
      Now, consider that you are given the weights, but not the threshold,
      and the goal is to solve the linear program to find the threshold
      that minimizes $\delta$.
      {\bf Your job is to implement the function {\tt findLinearThreshold} that takes the data
      and $w$, and returns $\theta$, and $\delta$.}
      
      The script has three different weight vectors, and for each of them it calls your function
      and then plots the resulting linear function using {\tt 
      plot2dSeparator}. It also generates a text file {\tt p3b4-model.txt} 
      which stores the four different models. 
      {\bf Run this script, and explain the results (Be sure to include the 
      generated figures in your solution and upload {\tt p3b4-model.txt}).
      Which hyperplanes separated the data exactly? 
      What $\delta$ values have you obtained, and what does they tell us?
      Is one hyperplane better than another for classification?
      From these examples, can we infer whether or not
      the solution to the linear program
      ([\eqref{eq:lin_prog_discriminant_obj} to \eqref{eq:lin_prog_discriminant_bound}]) is unique?}

\end{enumerate}


\end{enumerate}
\item[] {\bf What to Submit}
  \begin{itemize}
    \item A pdf file which contains 
      \begin{itemize}
        \item your answers to each question,
        \item the code snippets of {\tt findLinearDiscriminant.m}, {\tt computeLabel.m}, {\tt plot2dSeparator.m}, and {\tt findLinearThreshold.m}, 
        \item figures generated by {\tt learnConjunctions.m}, {\tt learnBadges.m}, and {\tt learnMultipleHyperplanes.m}.
      \end{itemize}
    \item {\tt p3b2-model.txt} and {\tt p3b4-model.txt}. You will generate these two
      files while executing {\tt learnConjunctions.m} and {\tt learnMultipleHyperplanes.m}.
    \item Please upload the above three files on Compass. (\texttt{http://compass2g.illinois.edu})
  \end{itemize}

\end{enumerate}

\newpage

\section*{Appendix: Linear Programming}

In this appendix, we will walk through a simple linear programming
example. If you want to read more on the topic, a good reference is {\em Linear Programming: Foundations and
Extensions} by Vanderbei.  Some classic texts include {\em Linear Programming} by Chvatal;
and {\em Combinatorial Optimization: Algorithms and Complexity} by Papadimitrou
and Steiglitz (in particular, the beginning of chapter $2$ may be helpful).  A
widely available (albeit incomplete) reference is {\em Introduction to
Algorithms} by Cormen, Leiserson, Rivest, and Stein.

\vspace{3mm}

\textbf{Example: }
Consider the following problem.\footnote{The problem closely resembles an
instantiation of the {\em diet problem} by G.~J.~Stigler, {\em The Cost of
Subsistence}, 1945.} You are given a choice of three foods, namely eggs at a
cost of $\$0.10$ an egg, pasta at a cost of $\$0.05$ a bowl, and yogurt at a cost
of $\$0.25$ a cup.  An egg $(t_1)$ provides $3$ portions of protein, $1$ portion of
carbohydrates, and $2$ portions of fat.  A bowl of pasta $(t_2)$ provides $1$ portion of
protein, $3$ portions of carbohydrates, and no portions of fat.  A cup of yogurt $(t_3)$
provides $2$ portions of protein, $2$ portions of carbohydrates, and $1$ portion of
fat.  You are required to consume at least $7$ portions of protein and $9$
portions of carbohydrates per day and are not allowed to consume more than $4$
portions of fat.  In addition, you obviously may not consume a negative amount
of any food.  The objective now is to find the cheapest combination of foods
that still meet your daily nutritional requirements. 

This can be written as the following linear program:

\begin{equation}
z = 0.1t_1 + 0.05t_2 + 0.25t_3 \to \min
\end{equation}
\begin{equation}
3t_1 + t_2 + 2t_3 \geq 7 \\
\end{equation}
\begin{equation}
t_1 + 3t_2 + 2t_3 \geq 9 \\
\end{equation}
\begin{equation}
-2t_1 - t_3 \geq -4 \\
\label{constraint3}
\end{equation}
\begin{equation}
t_i \geq 0 \hspace{0.3in} \forall i
\end{equation}

Note that inequality (\ref{constraint3}) of the LP is equivalent to $2t_1 +
t_3 < 4$. This corresponds to:

\begin{tabular}{cccc}
${\mathbf A} = \left ( \begin{array}{ccc}3 & 1 & 2\\1 & 3 & 2\\-2 & 0 & -1 \end{array} \right ) $
&
$\vec{b} = \left ( \begin{array}{c}7\\9\\-4\end{array} \right ) $
&
$\vec{c} = \left ( \begin{array}{c}0.1\\0.05\\0.25\end{array} \right ) $
&
$\vec{t} = \left ( \begin{array}{c}t_1\\t_2\\t_3\end{array} \right ) $
\end{tabular}

\vspace{3mm} 

To solve this program using Matlab:
\begin{verbatim}
     c = [0.1; 0.05; 0.25];
     A = [3 1 2; 1 3 2; -2 0 -1];
     b = [7; 9; -4];
     lowerBound = zeros(3, 1); % this constrains t >= 0
     [t, z] = linprog(c, -A, -b, [], [], lowerBound)
\end{verbatim}

The results of this linear program show that to meet your nutritional
requirements at a minimum cost, you should eat $1.5$ eggs, $2.5$ bowls of
pasta, and no cups of yogurt, and the cost for such a diet is $\$0.275$.

\end{document}

