CIS 700-002: Data-Driven Robotic Perception and Control (Fall 2020)

A robotic agent’s actions must be influenced by what it observes in its surroundings, and also in turn influence what it can observe in the future. As the renowned cognitive scientist Gibson once said, “We see in order to move and we move in order to see.”

Perception and control have long been studied disjointly across many disciplines: computer vision, robotics, control theory, reinforcement learning, and cognitive science. Through presentations and discussions of recent academic papers across these disciplines, interspersed with lectures, this course will aim to synthesize a common understanding of recent advances in data-driven methods to close the robotic perception-action loop, to actively analyze the strengths and weaknesses of current approaches, and to identify interesting open questions and possible directions for future research. Planned topics include: self-supervised representation learning, active perception, intrinsic motivation and exploration, video prediction and dynamics model learning, model-based and model-free reinforcement learning for visual control, visual servoing and control theoretical approaches, imitation learning, causality in ML, physics-based deep learning techniques, object-based representations, and meta- and transfer learning.

Early weeks of the course will mainly focus on lectures by the instructor, and the majority of the weeks will consist of student presentations, experiments, and paper discussions. The class discussions will follow the format of a reviewer panel, with assigned proponents and opponents for each paper.

Students will be responsible for:

  • writing two paper reviews each week for the assigned readings prior to in-class discussion, and posting public summaries on Piazza
  • participating in discussions during class, sometimes leading them as an assigned “proponent” or “opponent” for a paper (probably once in the course of the class, details depend on enrolment)
  • completing two or three programming assignments
  • presenting twice in class, possibly with a partner (details depend on enrollment): once on external papers, and once on an experiment
  • conducting an experiment based on a paper that we cover in class
  • completing a research-oriented final project with a partner

Grades will be based on:

  • 25% participation (includes attendance, in-class discussions, paper reviews)
  • 15% coding assignments
  • 35% presentations
  • 25% final project (includes proposal, presentation, final paper)

The format for this course is modeled after my PhD advisor, Kristen Grauman’s wonderful CS381V course.