Course Description

Deep learning techniques now touch on data systems of all varieties. Sometimes, deep learning is a product; sometimes, deep learning optimizes a pipeline; sometimes, deep learning provides critical insights; and sometimes, deep learning sheds light on neuroscience. The purpose of this course is to deconstruct the hype by teaching deep learning theories, models, skills, and applications that are useful for applications. The class will focus on the following 5 questions

How do we decide which problems to tackle with deep learning?

Not every problem should be tackled with deep learning. It’s data-hungry, catastrophically forgets, extrapolates poorly, cannot easily perform estimates of uncertainty, cannot easily transfer knowledge, and cannot be audited. Deep learning has been wildly successful in certain domains (CV, NLP, and RL come to mind), and students will understand how to determine what kinds of models are best for which problem settings.

Given a problem setting, how do we determine what model is best?

Suppose a problem has variable-length input. There are a range of popular ways to deal with such problems: truncation, padding, aggregated convolution, and recurrence. For any given problem, some approaches are preferable. We will cover popular models, discuss how to find interesting niche research, and gain intuition for the best approaches.

What’s the best way to implement said model?

The course begins with the mechanics of modern deep learning frameworks and autograd systems. Students will build on this knowledge by implementing widely-used models, cutting-edge research, and an open project of their choosing on a unique dataset. We will also cover interesting research on how to design deep learning systems (e.g. choosing the depth and the number of hidden neurons).

How can we best visualize, explain, and justify our findings?

The success of a deep learning project often depends on the interpretation and presentation of results. The course will cover how to choose evaluation criteria, how to determine next steps from the empirical performance of a model, which visualizations are the most useful and intuitive, and how to write up applied findings for both technical and non-technical audiences. Perhaps most importantly, the course will discuss when to use deep learning rather than simpler models (e.g. SVM, shallow networks).

How can neuroscience inspire deep learning?

For many innovations in deep learning research, neuroscience has directly or indirectly acted as an inspiration. We will thus, along with the material on implementations always teach ideas from neuroscience, presented compactly, about how DL ideas may relate to biology.


Must have a background in Machine Learning such as CIS 519 / CIS 520 / ESE 546 or equivalent. A background in linear algebra, probability and Python is also highly recommended.

Course Logistics

CIS 522 will be held on Tuesdays/Thursdays from 12-1:30pm, in Wu and Chen Auditorium (Levine 101). The course consists of five major components:

Homework Assignments 40%
Final Project 55%
Exam (Have cancelled exam, please see @464 for details) 0%
Lecture Attendance 5%

Homework Assignments

There will be five equally weighted homework assignments in the course. Homework assignments will always be due on Thursday and are approximately bi-weekly:

Late Policy

In this class, you will have a total of 5 late days for all of the homework assignments. You are free to use any of the remaining late days for whichever assignment you choose. Please do not let us know if you intend to use a late day, please submit normally on Gradescope and Gradescope will handle it automatically. After the five late days have been used, submitting an assignment a day late will be a 25% penalty. You will not receive credit for any assignment that is submitted past 5 days late.

Final Project

Each student will deeply explore a domain and topic in deep learning of their choice (for instance GANs in Computer Vision, Transformers in NLP, etc) and draft an abstract for their proposed project idea. Students will then be paired in groups of (3) based on similar domain / topic interest and work for the remainder of the semester to implement a range of models in the topic, comparing the neural and non-neural approaches’ respective efficacies. More details on this will be added soon.


On completion of the core course material (Introduction to Deep Learning, Computer Vision, Natural Language Processing, and Reinforcement Learning), there will be an exam testing all the material presented thus far in the course. The exam is meant to allow students to obtain a firm grasp of the material presented in the class, but not as to be a source of major stress for the students. Therefore, it is weighted 15% of the overall class grade.

Lecture Attendance

Lecture attendance is mandatory for students in the course. This is imperative to facilitate effective discussion and absorption of the presented material. As such, attendance will be taken via PollEverywhere every lecture at multiple times throughout the lecture to not only account for attendance, but also as a way to interactively engage in the material and inform us as instructors as to what concepts need further elaboration. The lecture polls are not graded on correctness but solely on completion.

Recitation Attendance

Every Friday there will be an optional recitation held by the TAs where the TAs will review the material presented in class as well as provide helpful tips with regards to implementation / debugging of the models presented. We highly recommend that if you can, you attend these recitations as they will be very helpful for the homework assignments and overall understanding of course material.

Office Hours

There will be office hours held every week by the TAs and professor. In office hours we will provide with help regarding elucidating the material presented in lecture / recitation as well as provide debugging help with the various assignments. TAs will not provide code, but will help you debug your existing code (debugging neural networks is challenging, so please come if you are stuck!)

Collaboration Policy

Collaboration is allowed to the following extent in the course:

Any student caught in violation of the collaboration policy above will be reported to the Office of Student Conduct (OSC) as well as given a 0 for that assignment / project. If a student copied from another student, that other student will also be reported to OSC and given a 0 on the homework. Please do not cheat, you are only hurting yourself!