Course description

Course website: www.cs136a.mmeteer.com

Speech recognition has had a huge resurgence in the past few years, both commercially and in the underlying technology. We can talk to Google, Siri, and Alexa to get information and carry out routine tasks. However, the technology is far from perfect when compared to human performance. There are still errors in recognizing the words, understanding the speakers intentions or providing an adequate response.

This course will touch on all of the components in a spoken dialog system, including speech recognition, natural language understanding, dialog state tracking, and response generation. For each topic area in the course, there will be two important parts:

  • Understanding the paradigms, formalism and algorithms in current approaches in each components of the system,
  • Getting hands on experience in using the top research and industry tools to build models and system components.

The main topics and examples of what we’ll cover are shown below:

Topic Approaches Tools
Speech recognition Architecture, phonetics, feature extraction, hidden markov models, weighted finite state transducers, Gaussian mixture models Kaldi, OpenFST, SRILM, NIST SCLite
NLP and frame-based dialog systems Intents & Entities, dialog context Dialogflow, Watson Conversation, Alexa
Dialog research Dialog state tracking, end-to-end dialog modeling, response generation Machine learning and DNN packages


Another goal of the course is to go beyond traditional lectures, quizzes and structured assignments (though they will be present) to types of learning that students will encounter throughout your careers.  So you will be asked to read papers that you might not have sufficient background for but will nonetheless help you to understand directions in the field.  You will be asked to figure out how systems work by downloading code and trying it out or following tutorials without knowing what all the steps mean but can help you evaluate whether it’s an area worth pursuing.  And, most importantly, you will be asked to work in teams where you’ll learn from each other and teach each other.

In our field, the underlying technology is changing faster than we can keep up.  The only constant is learning and working together.  We don’t need to be experts in the content.  We do need to know how to approach it and evaluate how relevant it is.  These are the skills you’ll need going forward.


Topics and assignments for each class are posted on the schedule page. Please check this reguarly, as it may change throughout the year.

Details on the assignments are posted on the assignments page. Again, please check this reguarly, I’ll update it as the assignments get closer.


There will be the following types of gradable elements in class. Due dates will be posted on the schedule page and announced in class. No extensions will be considered after the due date of the assignment for any reason and extensions will only be considered for well articulated reasons. If that reason is because you didn’t understand the problem or weren’t able to access data, etc, then it needs to be well in advance of the actual due date. Bottom line: Start early and communicate.

Policy on working together: Unless it is specifically stated in the assignment, all assignments must be done independently. However, when working with 3rd party toolsets, you may collaborate on getting the tools installed and running. In order to make this collaboration fair for everyone, you must post questions and answers on the class blog (tools TBD), even if it’s just a summary of a hallway conversation. If it was helpful, share it.

Assignment Types:

  • Show competency in the approaches presented in class and in readings through take home quizzes (10%)
  • Show competency in the tools by completing tutorials and directed assignments (20%)
  • Show ability to go beyond the course material in research through small group research and presentation projects (25%)
  • Show ability to go beyond the course material in system development by a small group application project in one of the three topic areas. (35%)
  • Participation (10%)

If you are a student with a documented disability on record at Brandeis University and which to have reasonable accommodations made for you in this class, please see me immediately.

Success in this 4 credit hour course is based on the expectation that students will spend a minimum of 9 hours of study time per week in preparation for class (readings, papers, discussion sections, preparation for exams, etc.).