The final project will be done in groups of 2-4 (3 is ideal).

  • Project proposals due Tuesday, November 1.  Include
    • Names of group members
    • Brief description of what problem/challenge you are addressing
    • Description of what you plan to do, including what data you plan to use
  • Submit via Latte

Projects should be chosen from one of the 3 major areas of the course:

  • Speech recognition
  • Task-based, frame-based dialog systems
  • Statistical dialog modeling

Some notes:

  • Projects are a significant part of your portfolio when interviewing for jobs.  Take that into account when deciding what to work on.
  • Some topics will be related to things we’ll cover in the later parts of the course.  If you’d like to work on a topic in statistical dialog modeling, I would be happy to talk with you earlier about the kinds of things you can do and point you to data and examples so you can get started before we get to the material.
  • Joint projects for multiple courses is allowed with the approval of both professors.
  • The topic ideas below are some things that I see in industry and research that would make a good project, but there are plenty more and I’m sure I’ll be adding to it, so these are just suggestions, not a list to pick from.


Speech Recognition

Speech recognition progress has been dominated by dumping in more data and appying increasingly complex DNNs.  However, there are still challenges in getting targeted domains to work really well, which is what you need for effective task oriented dialogs.  Pick a specific domain (e.g. ordering a pizza–we have some data for that) and look at what you can do in language modeling to be able to improve performance and quickly add new words/likelihoods, for example if your pizza place is running a special on a new “fantabulous pizza” this week.

Task oriented dialog system

We will be doing a class project with Dialogflow to build an interactive dialog system that can be launched as a bot on slack.  For a class project, you could use the same framework, but build a system that goes beyond what is typically using this type of framework by extending it to use context in NLP and dialog management, and providing multimodal, mixed initiative, and other advanced dialog capabilities. It will be important to select an application that requires these more advanced capabilities.

Chat dialog system

Using the AlexaPrize systems as a model, develop a system that will chat, however rather than allowing  open ended conversations, restrict it to a domain, such as travel or sports, and actually model the knowledge, so there is some understanding of both what the system says and what the user says.  In particular allow the user to ask follow up questions about what the system says.

Statistical dialog modeling

Over the past few years, there have been a sequence of “Dialog State Tracking Challenges”, in which participants apply mainly machine learning techniques to determine what the next state in a dialog should be.  The earliest ones were very simple domains and the data was data from actual human-system interaction on deployed systems.  So on the one hand, the input was noisy and on the other hand the state sequence couldn’t be any more sophisticated than an existing system could handle.  Despite these drawbacks, they made significant progress.

The latest challenge (DSTC7) is based on real human-human dialog.  There are two domains, Tech Support Chat about Ubuntu and a Student Advisor chat line to help students select courses.  In both cases, since the domains and possible states are so large and complex, the task is simplified.  The goal is given test dialogs, at particular defined points in the dialog, predict which of a set of utterances was the actual one in the reference dialog.

A reasonable project would be to use a DNN toolkit like Tensorflow to complete the task (with no assumption that it would have competitive performance–the challenge went on all summer), but at least get a performance that is better than chance.  The presentation/paper should also include what would be the plan if you were actually going to compete.

Another project would be to repeat one of the earlier challenges and use the published papers and your ideas to get competitive performance (after all, you can read what the winners did).  Caveat is we only have data for the earlier ones.

Note that performing well will unfortunately not result in your being sent to the workshop, which is Hawaii.