\input{cs446.tex}
\usepackage{amsmath,url,graphicx,amssymb}
\usepackage{soul,xcolor}
\sloppy
\usepackage{ulem}
\oddsidemargin 0in
\evensidemargin 0in
\textwidth 6.5in
\topmargin -0.5in
\textheight 9.0in
\newcommand{\ignore}[1]{}

\newcommand{\tight}[1]{\!#1\!}
\newcommand{\loose}[1]{\;#1\;}

\begin{document}
\setstcolor{red}

\assignment{Spring 2017}{6}{April $12^{th}$, $2017$}{April $20^{th}$, $2017$}

\begin{footnotesize}
\begin{itemize}
\item Feel free to talk to other members of the class in doing the homework.  I am
more concerned that you learn how to solve the problem than that you
demonstrate that you solved it entirely on your own.  You should, however,
write down your solution yourself.  Please try to keep the solution brief and
clear.

\item Please use Piazza first if you have questions about the homework.
  Also feel free to send us e-mails and come to office hours.

\item Please, no handwritten solutions. You will submit your solution manuscript as a single pdf file.


\item The homework is due at 11:59 PM on the due date. We will be using
Compass for collecting the homework assignments. Please submit your solution manuscript as a pdf file via Compass 
(\texttt{http://compass2g.illinois.edu}). Please do NOT hand in a hard copy of your write-up.
Contact the TAs if you are having technical difficulties in 
submitting the assignment.

\item No code is needed for any of these problems. You can do the
calculations however you please. You need to turn in only the report. Please name 
your report as \texttt{$\langle$NetID$\rangle$-hw6.pdf}.

\end{itemize}
\end{footnotesize}



\begin{enumerate}
\item {\bf [Na\"ive Bayes and Learning Threshold Functions - 25 points]}

Consider the Boolean function $f_{TH(4,9)}$.  This is a threshold function
defined on the 9 dimensional Boolean cube as follows: given an instance $x$,
$f_{TH(4,9)}(x) = 1$ if and only if 4 or more of $x$'s components are 1. \textcolor{red}{The 9-dimensional Boolean cube is a function $f: \{0,1\}^9 \rightarrow \{0,1\}$.}
\begin{enumerate}
\item {\bf [5 points]} Show that $f_{TH(4,9)}$ has a linear decision surface over the 
$9$ dimensional Boolean cube.
\item {\bf [10 points]}
Assume that you are given data sampled according to the uniform distribution
over the Boolean cube $\{0, 1\}^9$ and labeled according to $f_{TH(4,9)}$.
Use na\"ive Bayes to learn a hypothesis that predicts these labels.  What is
the hypothesis generated by the na\"ive Bayes algorithm?  (You may assume that
you have seen all the data required to get accurate estimates of the
probabilities).
\item {\bf [5 points]}
Show that the final hypothesis in (b) does not represent this function.
\item {\bf [5 points]}
Are the na\"ive Bayes assumptions satisfied by $f_{TH(4,9)}$?  Justify.
\end{enumerate}

\item {\bf [Multivariate Poisson na\"ive Bayes - 30 points]}
In this question, we consider the problem of classifying piazza posts ($Y$) into two categories: student posts ($A$), and instructor posts ($B$). For every post, we have two attributes: number of words ($X_1$), and number of mathematical symbols ($X_2$). We assume that each attribute ($X_i$,\ $i=1,2$) is related to a post category ($A$/$B$) via a Poisson distribution\footnote{http://en.wikipedia.org/wiki/Poisson\_distribution} with a particular mean ($\lambda^A_i$/$\lambda^B_i$). That is 
\[ Pr[X_i=x | Y = A] = \frac{e^{-\lambda^A_i} (\lambda^A_i)^x }{x!} ~~~\text{ and } ~~~ Pr[X_i=x | Y = B] = \frac{e^{-\lambda^B_i} (\lambda^B_i)^x }{x!} \text{ for } i =1,2 \]

\begin{table}[!h]
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
$X_1$ & $X_2$ & $Y$ \\
\hline
$0$ & $3$ & $A$ \\
$4$ & $8$ & $A$ \\
$2$ & $4$ & $A$ \\
$6$ & $2$ & $B$ \\
$3$ & $5$ & $B$ \\
$2$ & $1$ & $B$ \\
$5$ & $4$ & $B$ \\
\hline
\end{tabular}
\caption{Dataset for Poisson na\"ive Bayes}
\label{tab:p1}
\end{center}
\end{table}

Assume that the given data in Table~\ref{tab:p1} is generated by a Poisson na\"ive Bayes model. You will use this
 data to develop a na\"ive Bayes predictor over the Poisson distribution.
\begin{table}[!h]
\begin{center}
\begin{tabular}{|rp{1in}|rp{1in}|}
\hline
$\Pr(Y\tight{=}A)=$ & & $\Pr(Y\tight{=}B)=$ & \\ \hline
$\lambda^A_1=$ & & $\lambda^B_1=$ & \\ \hline
$\lambda^A_2=$ & & $\lambda^B_2=$ & \\ \hline
\end{tabular}
\caption{Parameters for Poisson na\"ive Bayes}
\label{tab:poissonNBparams}
\end{center}
\end{table}

\begin{enumerate}
\item {\bf [10 points]} Compute the prior probabilities and parameter values, i.e., fill out Table~\ref{tab:poissonNBparams}. [Hint: Use MLE to compute the $\lambda$'s]
\Comment{Note for future TAs: make sure to change the verbiage here. Students complained for losing points because they only filled up the table but didn't show intermediate results.}
\item {\bf [10 points]} Based on the parameter values from 
Table~\ref{tab:poissonNBparams}, compute
\begin{equation*}
\frac{\Pr(X_1\tight{=}2 , X_2\tight{=}3 \loose{|} Y\tight{=}A)}{\Pr(X_1\tight{=}2 , X_2\tight{=}3 \loose{|} Y\tight{=}B)}
\end{equation*}
\item {\bf [5 points]} 
Derive an algebraic expression for the Poisson na\"ive Bayes predictor for $Y$ in terms of the 
parameters estimated from the data.
% set (not specifically Table~\ref{tab:gaussianNBdata}). 
%\sout{Also write down how you will compute these probabilities.}

\item {\bf [5 points]}
Use the parameters estimated from the data given in Table~\ref{tab:p1} to 
create a Poisson na\"ive Bayes classifier. What will the classifier predict as the value of $Y$, given the data point:
$X_1\tight{=}2, X_2\tight{=}3$? 

\end{enumerate}

\item {\bf [Na\"ive Bayes over Multinomial Distribution - 35 points]}

In this question, we will look into training a na\"ive Bayes classifier with a model that uses 
a multinomial distribution to represent documents. Assume that all the documents are 
written in a language which has only three words $a$, $b$, and $c$. All the 
documents have exactly $n$ words (each word can be either $a$, $b$, or $c$). We 
are given a labeled document collection $\{D_1, D_2, \ldots, D_m\}$. The label $y_i$ 
of document $D_i$ is $1$ or $0$, indicating whether $D_i$ is ``\texttt{good}'' or 
``\texttt{bad}''.

This model uses the multinominal distribution in the following way: Given the 
$i^{th}$ document $D_i$, we denote by $a_i$ (respectively, $b_i$, $c_i$) the 
number of times that word $a$ (respectively, $b$, $c$) appears in $D_i$. Therefore, 
$a_i + b_i + c_i = |D_i| = n$. We define
\[ \Pr(D_i | y = 1) = \frac{n!}{a_i! b_i! c_i!} \alpha_1^{a_i} \beta_1^{b_i} \gamma_1^{c_i} \]
where $\alpha_1$ (respectively, $\beta_1$, $\gamma_1$) is the probability that word 
$a$ (respectively, $b$, $c$) appears in a ``\texttt{good}'' document. Therefore,
$\alpha_1 + \beta_1 + \gamma_1 = 1$. Similarly,
\[ \Pr(D_i | y = 0) = \frac{n!}{a_i! b_i! c_i!} \alpha_0^{a_i} \beta_0^{b_i} \gamma_0^{c_i} \]
where $\alpha_0$ (respectively, $\beta_0$, $\gamma_0$) is the probability that word 
$a$ (respectively, $b$, $c$) appears in a ``\texttt{bad}'' document. Therefore, 
$\alpha_0 + \beta_0 + \gamma_0 = 1$.

\begin{enumerate}
\item {\bf [2 points]} What information do we lose when we represent documents using the aforementioned model?
\item {\bf [5 points]} Write down the expression for the log likelihood of the document $D_i$, 
$\log \Pr(D_i, y_i)$. Assume that the prior probability, $Pr(y_i = 1)$ is $\theta$.
\item {\bf [28 points]} Derive the expression for the maximum likelihood estimates for parameters $\alpha_1$, $\beta_1$, $\gamma_1$, 
$\alpha_0$, $\beta_0$, and $\gamma_0$.
\end{enumerate}

\textbf{Submission note:} You need not show the derivation of all six parameters separately. Some parameters are symmetric to others, and so, once you derive the expression for one, you can directly write down the expression for others. 

\textbf{Grading note: 8 points} for the derivation of one of the parameters, \textbf{4 points} each for the remaining five parameter expressions.

\item {\bf [Dice Roll - 10 points]}

Consider a scheme to generate a series of numbers as follows: For each element in the series, first a dice is rolled. If it comes up as one of the numbers from 1 to 5, it is shown to the user. On the other hand, if the dice roll comes up as 6, then the dice is rolled for the second time, and the outcome of this roll is shown to the user.

Assume that the probability of a dice roll coming up as 6 is $p$. Also assume if a dice roll doesn't come up as 6, then the remaining numbers are equally likely. Suppose you see a sequence \texttt{3463661622} generated based on the scheme given above.  What is the most likely value of $p$ for this given sequence? 
\end{enumerate}
\end{document}
