Peter A. Heeman
Associate Research Professor
Center for Spoken Language Understanding
Computer Science and Electrical Engineering
Peter Heeman's research interests focus around the theme of spoken
dialogue processing. Spoken dialogue interfaces to computer systems
have the advantage that they can let people use the communication
style that they are accustomed for interacting with a computer system.
However, there are many barriers standing in our way.
In spontaneous speech, speakers will often utter more than one
contribution during their turn of speaking. Unlike written text,
there are no explicit punctuation marks that delimit one utterance
from the next. Furthermore, due to the online nature of spontaneous
speech, speakers sometimes need to revise what they have just said, by
making a speech repair. Hence, a first step in understanding
spontaneous speech is to segment the speech into distinct
utterances and determine the speaker's intended contributions by
resolving any speech repairs that might have occurred.
Segmenting speech into utterances and resolving speech repairs is
intimately intertwined with the speech recognition task of determining
what words the speaker is speaking. Speech repairs and utterance
boundaries cause disruptions in the local context, both acoustic and
in predicting the next word given the previous words. We have proposed
that speech repairs and utterance boundaries (as well as
part-of-speech tags and discourse markers) be resolved during the
speech recognition task. Towards this end, we have become interested
in language models that can be used during speech recognition to help
the speech recognizer to prune alternative acoustic hypothesis, and
augmenting them to take into account phenomena of spontaneous speech.
This work centers around using machine learning techniques, such as
decision trees, to learn spontaneous speech phenomena.
Dr. Heeman is also interested in dialogue management. During a
dialogue, conversants must attune themselves towards whether they are
being understood and whether they understand their partner, as well as
whether they are reaching agreement. Current spoken dialogue systems
tend to ignore this facet of conversation, and blindly assume that the
conversation is proceeding without difficulty. But if difficulties
(misunderstandings or nonunderstandings) arise, the system will have
problems following the dialogue, leading to user frustration. In
order to make dialogue systems more user-friendly and easier to use, a
dialogue system must participate and collaborate in grounding
both its own utterances and the utterances of the user. Peter Heeman
has built a computational model of how participants collaborate in
In understanding the user's contributions to the dialogue, the
dialogue system needs to make use of the discourse state to narrow
down the possible interpretations. This usage of discourse knowledge
needs to be applied as early on processing as possible, even during
the task of speech recognition. Peter Heeman's prior work in
incorporating modeling speech repairs and utterance segmentation is a
first step towards incorporating more knowledge into the speech
recognition task and in enlarging this task to incorporate processing
that typically is done after recognition. Peter's work in
automatically identifying discourse markers, words such as ``so'' and
``but'' that relate the current utterance to the discourse state, also
shows the potential of incorporating discourse processing into the speech
recognition problem of language modeling.
||Sympolic Approaches to Artificial Intelligence
||Statistical Natural Language Processing
||Spoken Language Systems
||Ph.D. 2008, now at Nuance.
||M.Sc. 2005, now at Google.
||M.Sc. 2001, Ph.D. student at the University of Rochester
||Ph.D. 2001 (on-campus adviser), now an Associate Professor at University of Portland
||Ph.D. 2000 (on-campus advisor), now at Intel
||M.Sc. 1999, now a professor at University of Zulia State in Venezuela
||M.Sc. 1998, now at Microsoft
A complete list of publications are located here.
- F. Yang, P. Heeman, and A. Kun. An
investigation of Interruptions and Resumptions in Multi-Tasking
Dialogues. In Computational Linguistics Journal,
Vol. 37-1, 2011.
- P. Heeman, A. McMillin, and J. Scott Yaruss. Computer-Assisted
Disfluency Counts for Stuttered Speech. In Proceedings of
the 12th Annual Conference of the International Speech
Communication Association (Interspeech), Florence Italy,
- P. Heeman, R. Lunsford, E. Selfridge, L. Black, and J. van Santen.
Autism and Interactional Aspects of
Dialogue. In 11th Annual SIGdial
Meeting on Discourse and Dialogue, Tokyo, September 2010.
- F. Yang and P. Heeman. Initiave Conflicts in Task-Oriented Dialogue.
In Computer Speech and Language, Vol. 24, num. 3, pgs. 175-189, 2010.
- E. Selfridge and P. Heeman. Importance-Driven
Turn-Bidding for Spoken Dialogue Systems. In Proceedings
of the 48th Annual Meeting of the Association for Computational
Linguistics, pages 177-185, Uppsala Sweden, July 2010.
- P. Heeman. Representing the Reinforcement
Learning State in a Negotiation Dialogue. In Proceedings of
the IEEE workshop on Automatic Speech Recognition and
Understanding, Merano Italy, December 2009.
- F. Yang, P. Heeman, K. Hollingshead, and S. Strayer.
DialogueView: Annotating and viewing dialogue with multiple levels of
abstraction. In Natural Language Engineering, Vol. 14,
pgs. 3-32, 2008.
- P. Heeman. Combining Reinforcement
Learning with Information-State Update Rules. In Proceedings
of the North American Chapter of the Association for Computational
Linguistics Annual Meeting, pages 268-275, Rochester NY, April 2007.
- M. English and P. Heeman. Learning
mixed initiative dialog strategies by using reinforcement learning on
both conversants. In Proceedings of the Human Language
Technology Conference and Conference on Empirical Methods in Natural
Language Processing, pages 1011-1018, Vancouver Canada, October
- P. Heeman. Modeling Spontaneous Speech
Events during Recognition. In First International Conference
on Universal Access in Human-Computer Interaction, New Orleans,
- P. Heeman and J. Allen. Speech
Repairs, Intonational Phrases and Discourse Markers: Modeling
Speakers' Utterances in Spoken Dialog, Computational
Linguistics, Vol. 25-4, pgs. 527-572, 1999.
- P. Heeman and G. Hirst. Collaborating on Referring Expressions. In
Compuational Linguistics, Vol. 21-3, 1995.