Deep learning techniques now touch on data systems of all varieties. Sometimes, deep learning is a product; sometimes, deep learning optimizes a pipeline; sometimes, deep learning provides critical insights; and sometimes, deep learning sheds light on neuroscience. The purpose of this course is to deconstruct the hype by teaching deep learning theories, models, skills, and applications that are useful for applications. The class will focus on the following 5 questions

Not every problem should be tackled with deep learning. It’s data-hungry, catastrophically forgets, extrapolates poorly, cannot easily perform estimates of uncertainty, cannot easily transfer knowledge, and cannot be audited. Deep learning has been wildly successful in certain domains (CV, NLP, and RL come to mind), and students will understand how to determine what kinds of models are best for which problem settings.

Suppose a problem has variable-length input. There are a range of popular ways to deal with such problems: truncation, padding, aggregated convolution, and recurrence. For any given problem, some approaches are preferable. We will cover popular models, discuss how to find interesting niche research, and gain intuition for the best approaches.

The course begins with the mechanics of modern deep learning frameworks and autograd systems. Students will build on this knowledge by implementing widely-used models, cutting-edge research, and an open project of their choosing on a unique dataset. We will also cover interesting research on how to design deep learning systems (e.g. choosing the depth and the number of hidden neurons).

The success of a deep learning project often depends on the interpretation and presentation of results. The course will cover how to choose evaluation criteria, how to determine next steps from the empirical performance of a model, which visualizations are the most useful and intuitive, and how to write up applied findings for both technical and non-technical audiences. Perhaps most importantly, the course will discuss when to use deep learning rather than simpler models (e.g. SVM, shallow networks).

For many innovations in deep learning research, neuroscience has directly or indirectly acted as an inspiration. We will thus, along with the material on implementations always teach ideas from neuroscience, presented compactly, about how DL ideas may relate to biology.

Must have a background in Machine Learning such as CIS 519 / CIS 520 / ESE 546 or equivalent. A background in linear algebra, probability and Python is also highly recommended.

CIS 522 will be held on **Tuesdays/Thursdays from 12-1:30pm, in Wu and Chen Auditorium (Levine 101)**.
The course consists of five major components:

Homework Assignments | 40% |

Final Project | 55% |

Exam (Have cancelled exam, please see @464 for details) | 0% |

Lecture Attendance | 5% |

There will be five equally weighted homework assignments in the course. Homework assignments will always be due on Thursday and are approximately bi-weekly:

: This is a very short homework that will introduce you to the tools / infrastructure of the course. It provides a tutorial on Google Colab, Tensorboard, using the GPU, PDB, and how to submit with our infrastructure.**Homework 0**: This is an introductory ML homework to the course that will have students brush up on the knowledge that they’ve gained from CIS 519 / 520 / an equivalent ML course. This homework will have students experiment with different classical approaches to classify the Wisconsin Breast Cancer dataset using Scikit Learn and PyTorch and analyze the dataset using Pandas and Numpy. The students will set up all of this on Google Colab, which will be used for the remainder of the course because it allows free usage of a GPU / TPU for computing.**Homework 1**: The goal of this homework is to give students more experience with creating and training neural networks in PyTorch using tensorboard to visualize the training progress. Additionally, students will then be tasked to debug a variety of different pre-trained neural networks provided to them, to build an intuition for different training failure modes as well as procedures for correcting them.**Homework 2**: This assignment provides a challenging introduction to deep learning in computer vision. Students will work in groups of two (2) to implement a Convolutional Neural Network for classification, comparing this to the simple Feed Forward Network / classical approaches explored in the previous homework assignments. In addition, students will implement a variety of state of the art algorithms in Computer Vision (tentatively such as a VAE, CVAE, GAN, etc).**Homework 3**: In the past, students worked with a Reddit Sarcasm Dataset to detect sarcasm on Reddit. Students created a word embedding while varying encoding, featurization and network model to classify whether reddit excerpts exhibit sarcasm. The assignment / dataset for this semester is to be determined.**Homework 4**: Students explore the power of reinforcement learning in an interactive way, using Deep Q-Learning to develop an agent that can successfully play a game/control an environment (game to be determined) in a provided environment. Students will be able to, as the agent progresses, visualize the learned policy, applying it to the given environment to solve the problem and also look at achieved rewards corresponding to the learnt policy.**Homework 5**

In this class, you will have a total of 5 late days for all of the homework assignments. You are free to use any of the remaining late days for whichever assignment you choose. Please do not let us know if you intend to use a late day, please submit normally on Gradescope and Gradescope will handle it automatically. After the five late days have been used, submitting an assignment a day late will be a **25% penalty**. You will not receive credit for any assignment that is submitted past 5 days late.

Each student will deeply explore a domain and topic in deep learning of their choice (for instance GANs in Computer Vision, Transformers in NLP, etc) and draft an abstract for their proposed project idea. Students will then be paired in groups of (3) based on similar domain / topic interest and work for the remainder of the semester to implement a range of models in the topic, comparing the neural and non-neural approaches’ respective efficacies. More details on this will be added soon.

On completion of the core course material (Introduction to Deep Learning, Computer Vision, Natural Language Processing, and Reinforcement Learning), there will be an exam testing all the material presented thus far in the course. The exam is meant to allow students to obtain a firm grasp of the material presented in the class, but not as to be a source of major stress for the students. Therefore, it is weighted 15% of the overall class grade.

Lecture attendance is mandatory for students in the course. This is imperative to facilitate effective discussion and absorption of the presented material. As such, attendance will be taken via PollEverywhere every lecture at multiple times throughout the lecture to not only account for attendance, but also as a way to interactively engage in the material and inform us as instructors as to what concepts need further elaboration. The lecture polls are not graded on correctness but solely on completion.

Every Friday there will be an optional recitation held by the TAs where the TAs will review the material presented in class as well as provide helpful tips with regards to implementation / debugging of the models presented. We highly recommend that if you can, you attend these recitations as they will be very helpful for the homework assignments and overall understanding of course material.

There will be office hours held every week by the TAs and professor. In office hours we will provide with help regarding elucidating the material presented in lecture / recitation as well as provide debugging help with the various assignments. TAs will not provide code, but will help you debug your existing code (debugging neural networks is challenging, so please come if you are stuck!)

Collaboration is allowed to the following extent in the course:

- For
**group assignments / the final project**, students are permitted to view each other’s code, help each other debug, and view each other’s LaTeX,**within their group**. - For
**individual assignments**students are not permitted to view each other’s code, help each other debug or view each other’s LaTeX. Students may, however, discuss the concepts, questions and assignment at a high level amongst each other (i.e. clarifications but not giving each other solutions), however, all LaTeX**must be done individually**as must all code. - For
**both group and individual assignments**students may use the internet for debugging help (i.e. stack overflow), read medium (and similar) articles to gleam further understanding of the material presented, and of course official documentation (i.e. for PyTorch, NumPy, Pandas, etc.).**However, copying of code from the internet is strictly forbidden, all code submitted must be your own work.**We will be running all code you submit against plagiarism detection software.

**Any student caught in violation of the collaboration policy above will be reported to the Office of Student Conduct (OSC) as well as given a 0 for that assignment / project. If a student copied from another student, that other student will also be reported to OSC and given a 0 on the homework. Please do not cheat, you are only hurting yourself!**