Science.gov

Sample records for lipreading

  1. Craig Lipreading Inventory.

    ERIC Educational Resources Information Center

    DiJohnson, Albert; And Others

    This speech inventory used in a study of aurally handicapped children (see TM 001 129) measures lipreading ability. The 33-item inventory represents a meaningful sample of phonemes in Standard American English. This measure can be presented either live or by 16 millimeter film. See also TM 001 130, 131, and 133. (CK)

  2. Recent developments in automated lip-reading

    NASA Astrophysics Data System (ADS)

    Bowden, Richard; Cox, Stephen; Harvey, Richard; Lan, Yuxuan; Ong, Eng-Jon; Owen, Gari; Theobald, Barry-John

    2013-10-01

    Human lip-readers are increasingly being presented as useful in the gathering of forensic evidence but, like all humans, suffer from unreliability. Here we report the results of a long-term study in automatic lip-reading with the objective of converting video-to-text (V2T). The V2T problem is surprising in that some aspects that look tricky, such as real-time tracking of the lips on poor-quality interlaced video from hand-held cameras, but prove to be relatively tractable. Whereas the problem of speaker independent lip-reading is very demanding due to unpredictable variations between people. Here we review the problem of automatic lip-reading for crime fighting and identify the critical parts of the problem.

  3. Lip-reading enhancement for law enforcement

    NASA Astrophysics Data System (ADS)

    Theobald, Barry J.; Harvey, Richard; Cox, Stephen J.; Lewis, Colin; Owen, Gari P.

    2006-09-01

    Accurate lip-reading techniques would be of enormous benefit for agencies involved in counter-terrorism and other law-enforcement areas. Unfortunately, there are very few skilled lip-readers, and it is apparently a difficult skill to transmit, so the area is under-resourced. In this paper we investigate the possibility of making the lip-reading task more amenable to a wider range of operators by enhancing lip movements in video sequences using active appearance models. These are generative, parametric models commonly used to track faces in images and video sequences. The parametric nature of the model allows a face in an image to be encoded in terms of a few tens of parameters, while the generative nature allows faces to be re-synthesised using the parameters. The aim of this study is to determine if exaggerating lip-motions in video sequences by amplifying the parameters of the model improves lip-reading ability. We also present results of lip-reading tests undertaken by experienced (but non-expert) adult subjects who claim to use lip-reading in their speech recognition process. The results, which are comparisons of word error-rates on unprocessed and processed video, are mixed. We find that there appears to be the potential to improve the word error rate but, for the method to improve the intelligibility there is need for more sophisticated tracking and visual modelling. Our technique can also act as an expression or visual gesture amplifier and so has applications to animation and the presentation of information via avatars or synthetic humans.

  4. The relationship between two visual communication systems: reading and lipreading.

    PubMed

    Williams, A

    1982-12-01

    To explore the relationship between reading and lipreading and to determine whether readers and lipreaders use similar strategies to comprehend verbal messages, 60 female junior and sophomore high school students--30 good and 30 poor readers--were given a filmed lipreading test, a test to measure eye-voice span, a test of cloze ability, and a test of their ability to comprehend printed material presented one word at a time in the absence of an opportunity to regress or scan ahead. The results of this study indicated that (a) there is a significant relationship between reading and lipreading ability; (b) although good readers may be either good or poor lipreaders, poor readers are more likely to be poor than good lipreaders; (c) there are similarities in the strategies used by readers and lipreaders in their approach to comprehending spoken and written material; (d) word-by-word reading of continuous prose appears to be a salient characteristic of both poor reading and poor lipreading ability; and (c) good readers and lipreaders do not engage in word-by-word reading but rather use a combination of visual and linguistic cues to interpret written and spoken messages.

  5. Some normative data on lip-reading skills (L)

    PubMed Central

    Altieri, Nicholas A.; Pisoni, David B.; Townsend, James T.

    2011-01-01

    The ability to obtain reliable phonetic information from a talker’s face during speech perception is an important skill. However, lip-reading abilities vary considerably across individuals. There is currently a lack of normative data on lip-reading abilities in young normal-hearing listeners. This letter describes results obtained from a visual-only sentence recognition experiment using CUNY sentences and provides the mean number of words correct and the standard deviation for different sentence lengths. Additionally, the method for calculating T-scores is provided to facilitate the conversion between raw and standardized scores. This metric can be utilized by clinicians and researchers in lip-reading studies. This statistic provides a useful benchmark for determining whether an individual’s lip-reading score falls within the normal range, or whether it is above or below this range. PMID:21786870

  6. Lipreading in School-Age Children: The Roles of Age, Hearing Status, and Cognitive Ability

    ERIC Educational Resources Information Center

    Tye-Murray, Nancy; Hale, Sandra; Spehar, Brent; Myerson, Joel; Sommers, Mitchell S.

    2014-01-01

    Purpose: The study addressed three research questions: Does lipreading improve between the ages of 7 and 14 years? Does hearing loss affect the development of lipreading? How do individual differences in lipreading relate to other abilities? Method: Forty children with normal hearing (NH) and 24 with hearing loss (HL) were tested using 4…

  7. Lip-reading abilities in a subject with congenital prosopagnosia.

    PubMed

    Wathour, J; Decat, M; Vander Linden, F; Deggouj, N

    2015-01-01

    We present the case of an individual with congenital prosopagnosia or "face blindness", a disorder where the ability to recognize faces is impaired. We studied the lip-reading ability and audiovisual perception of this subject using a DVD with four conditions (audiovisual congruent, auditory, visual, and audiovisual incongruent) and compared results with a normal patient cohort. The patient had no correct responses in the visual lip-reading task; whereas, he improved in the audiovisual congruent task. In the audiovisual incongruent task, the patient provided one response; thus, he was able to lip-read. (He was able to use lip-reading/to use labial informations) This patient perceived only global dynamic facial movements, not the fine ones. He had a sufficient complementary use of lip-reading in audiovisual tasks, but not visual ones. These data are consistent with abnormal development of the pathways used for visual speech perception and associated with second-order face processing disorders and normal development of the audiovisual network for speech perception.

  8. Tactual display of consonant voicing as a supplement to lipreading

    NASA Astrophysics Data System (ADS)

    Yuan, Hanfeng; Reed, Charlotte M.; Durlach, Nathaniel I.

    2005-08-01

    This research is concerned with the development and evaluation of a tactual display of consonant voicing to supplement the information available through lipreading for persons with profound hearing impairment. The voicing cue selected is based on the envelope onset asynchrony derived from two different filtered bands (a low-pass band and a high-pass band) of speech. The amplitude envelope of each of the two bands was used to modulate a different carrier frequency which in turn was delivered to one of the two fingers of a tactual stimulating device. Perceptual evaluations of speech reception through this tactual display included the pairwise discrimination of consonants contrasting voicing and identification of a set of 16 consonants under conditions of the tactual cue alone (T), lipreading alone (L), and the combined condition (L+T). The tactual display was highly effective for discriminating voicing at the segmental level and provided a substantial benefit to lipreading on the consonant-identification task. No such benefits of the tactual cue were observed, however, for lipreading of words in sentences due perhaps to difficulties in integrating the tactual and visual cues and to insufficient training on the more difficult task of connected-speech reception.

  9. Experience with a talker can transfer across modalities to facilitate lipreading.

    PubMed

    Sanchez, Kauyumari; Dias, James W; Rosenblum, Lawrence D

    2013-10-01

    Rosenblum, Miller, and Sanchez (Psychological Science, 18, 392-396, 2007) found that subjects first trained to lip-read a particular talker were then better able to perceive the auditory speech of that same talker, as compared with that of a novel talker. This suggests that the talker experience a perceiver gains in one sensory modality can be transferred to another modality to make that speech easier to perceive. An experiment was conducted to examine whether this cross-sensory transfer of talker experience could occur (1) from auditory to lip-read speech, (2) with subjects not screened for adequate lipreading skill, (3) when both a familiar and an unfamiliar talker are presented during lipreading, and (4) for both old (presentation set) and new words. Subjects were first asked to identify a set of words from a talker. They were then asked to perform a lipreading task from two faces, one of which was of the same talker they heard in the first phase of the experiment. Results revealed that subjects who lip-read from the same talker they had heard performed better than those who lip-read a different talker, regardless of whether the words were old or new. These results add further evidence that learning of amodal talker information can facilitate speech perception across modalities and also suggest that this information is not restricted to previously heard words.

  10. Some observations on computer lip-reading: moving from the dream to the reality

    NASA Astrophysics Data System (ADS)

    Bear, Helen L.; Owen, Gari; Harvey, Richard; Theobald, Barry-John

    2014-10-01

    In the quest for greater computer lip-reading performance there are a number of tacit assumptions which are either present in the datasets (high resolution for example) or in the methods (recognition of spoken visual units called "visemes" for example). Here we review these and other assumptions and show the surprising result that computer lip-reading is not heavily constrained by video resolution, pose, lighting and other practical factors. However, the working assumption that visemes, which are the visual equivalent of phonemes, are the best unit for recognition does need further examination. We conclude that visemes, which were defined over a century ago, are unlikely to be optimal for a modern computer lip-reading system.

  11. Phonological Processing in Deaf Children: When Lipreading and Cues Are Incongruent

    ERIC Educational Resources Information Center

    Alegria, J.; Lechat, J.

    2005-01-01

    Deaf children exposed to Cued Speech (CS), either before age two (early) or later at school (late), were presented with pseudowords with and without CS. The main goal was to establish the way in which lipreading and CS combine to produce unitary percepts, similar to audiovisual integration in speech perception, when participants are presented with…

  12. Lipreading, Processing Speed, and Working Memory in Younger and Older Adults

    ERIC Educational Resources Information Center

    Feld, Julia E.; Sommers, Mitchell S.

    2009-01-01

    Purpose: To examine several cognitive and perceptual abilities--including working memory (WM), information processing speed (PS), perceptual closure, and perceptual disembedding skill--as factors contributing to individual differences in lipreading performance and to examine how patterns in predictor variables change across age groups. Method:…

  13. Silent lipreading and covert speech production suppress processing of non-linguistic sounds in auditory cortex

    PubMed Central

    Balk, Marja H.; Kari, Heini; Kauramäki, Jaakko; Ahveninen, Jyrki; Sams, Mikko; Autti, Taina; Jääskeläinen, Iiro P.

    2013-01-01

    Previous studies have suggested that speech motor system mediates suppression by silent lipreading of electromagnetic auditory cortex responses to pure tones at about 100 ms from sound onset. We used sparse sampling functional magnetic resonance imaging (fMRI) at 3 Tesla to map auditory-cortex foci of suppressant effects during silent lipreading and covert self-production. Streams of video clips were presented simultaneously with 1/3 octave noise bursts centered at 250 Hz (low frequency, LF) or 2000 Hz (mid-frequency, MF), or during no auditory stimulation. In different conditions, the subjects were a) to press a button whenever they lipread the face articulate the same consecutive Finnish vowels /a/, /i/, /o/, and /y/, b) covertly selfproducing vowels while viewing still face image, or c) to press a button whenever a circle pictured on top of the lips expanded into oval shape of the same orientation twice in a row. The regions of interest (ROIs) within the superior temporal lobes of each hemisphere were defined by contrasting MF and LF stimulation against silence. Contrasting the nonlinguistic (i.e., expanding circle) vs. linguistic (i.e., lipreading and covert self-production) conditions within these ROIs showed significant suppression of hemodynamic activity to MF sounds in the linguistic condition in left hemisphere first transverse sulcus (FTS) and right hemisphere superior temporal gyrus (STG) lateral to Heschl’s sulcus (HS). These findings suggest that the speech motor system mediates suppression of auditory-cortex processing of non-linguistic sounds during silent lipreading and covert self-production in left hemisphere FST and right hemisphere STG lateral to HS. PMID:23956838

  14. Lipreading and covert speech production similarly modulate human auditory-cortex responses to pure tones.

    PubMed

    Kauramäki, Jaakko; Jääskeläinen, Iiro P; Hari, Riitta; Möttönen, Riikka; Rauschecker, Josef P; Sams, Mikko

    2010-01-27

    Watching the lips of a speaker enhances speech perception. At the same time, the 100 ms response to speech sounds is suppressed in the observer's auditory cortex. Here, we used whole-scalp 306-channel magnetoencephalography (MEG) to study whether lipreading modulates human auditory processing already at the level of the most elementary sound features, i.e., pure tones. We further envisioned the temporal dynamics of the suppression to tell whether the effect is driven by top-down influences. Nineteen subjects were presented with 50 ms tones spanning six octaves (125-8000 Hz) (1) during "lipreading," i.e., when they watched video clips of silent articulations of Finnish vowels /a/, /i/, /o/, and /y/, and reacted to vowels presented twice in a row; (2) during a visual control task; (3) during a still-face passive control condition; and (4) in a separate experiment with a subset of nine subjects, during covert production of the same vowels. Auditory-cortex 100 ms responses (N100m) were equally suppressed in the lipreading and covert-speech-production tasks compared with the visual control and baseline tasks; the effects involved all frequencies and were most prominent in the left hemisphere. Responses to tones presented at different times with respect to the onset of the visual articulation showed significantly increased N100m suppression immediately after the articulatory gesture. These findings suggest that the lipreading-related suppression in the auditory cortex is caused by top-down influences, possibly by an efference copy from the speech-production system, generated during both own speech and lipreading. PMID:20107058

  15. Lipreading and covert speech production similarly modulate human auditory-cortex responses to pure tones

    PubMed Central

    Kauramäki, Jaakko; Jääskeläinen, Iiro P.; Hari, Riitta; Möttönen, Riikka; Rauschecker, Josef P.; Sams, Mikko

    2010-01-01

    Watching the lips of a speaker enhances speech perception. At the same time, the 100-ms response to speech sounds is suppressed in the observer’s auditory cortex. Here, we used whole-scalp 306-channel magnetoencephalography (MEG) to study whether lipreading modulates human auditory processing already at the level of the most elementary sound features, i.e., pure tones. We further envisioned the temporal dynamics of the suppression to tell whether the effect is driven by top-down influences. Nineteen subjects were presented with 50-ms tones spanning six octaves (125–8000 Hz) (1) during “lipreading”, i.e. when they watched video clips of silent articulations of Finnish vowels /a/, /i/, /o/, /y/, and reacted to vowels presented twice in a row, (2) during a visual control task, (3) during a still-face passive control condition, and, in a separate experiment with a subset of nine subjects, (4) during covert production of the same vowels. Auditory-cortex 100-ms responses (N100m) were equally suppressed in the lipreading and covert speech-production tasks compared with the visual control and baseline tasks; the effects involved all frequencies and were most prominent in the left hemisphere. Responses to tones presented at different times with respect to the onset of the visual articulation showed significantly increased N100m suppression immediately after the articulatory gesture. These findings suggest that the lipreading-related suppression in the auditory cortex is caused by top-down influences, possibly by an efference copy from the speech-production system, generated during both own speech and lipreading. PMID:20107058

  16. There Goes the Neighborhood: Lipreading and the Structure of the Mental Lexicon

    PubMed Central

    Feld, Julia; Sommers, Mitchell

    2010-01-01

    A central question in spoken word recognition research is whether words are recognized relationally, in the context of other words in the mental lexicon [1, 2]. The current research evaluated metrics for measuring the influence of the mental lexicon on visually perceived (lipread) spoken word recognition. Lexical competition (the extent to which perceptually similar words influence recognition of a stimulus word) was quantified using metrics that are well-established in the literature, as well as a novel statistical method for calculating perceptual confusability, based on the Phi-square statistic. The Phi-square statistic proved an effective measure for assessing lexical competition and explained significant variance in visual spoken word recognition beyond that accounted for by traditional metrics. Because these values include the influence of all words in the lexicon (rather than only perceptually very similar words), it suggests that even perceptually distant words may receive some activation, and therefore provide competition, during spoken word recognition. This work supports and extends earlier research [3] that proposed a common recognition system underlying auditory and visual spoken word recognition and provides support for the use of the Phi-square statistic for quantifying lexical competition. PMID:21170172

  17. Audiovisual Speech Integration and Lipreading in Autism

    ERIC Educational Resources Information Center

    Smith, Elizabeth G.; Bennetto, Loisa

    2007-01-01

    Background: During speech perception, the ability to integrate auditory and visual information causes speech to sound louder and be more intelligible, and leads to quicker processing. This integration is important in early language development, and also continues to affect speech comprehension throughout the lifespan. Previous research shows that…

  18. Multi-pose lipreading and audio-visual speech recognition

    NASA Astrophysics Data System (ADS)

    Estellers, Virginia; Thiran, Jean-Philippe

    2012-12-01

    In this article, we study the adaptation of visual and audio-visual speech recognition systems to non-ideal visual conditions. We focus on overcoming the effects of a changing pose of the speaker, a problem encountered in natural situations where the speaker moves freely and does not keep a frontal pose with relation to the camera. To handle these situations, we introduce a pose normalization block in a standard system and generate virtual frontal views from non-frontal images. The proposed method is inspired by pose-invariant face recognition and relies on linear regression to find an approximate mapping between images from different poses. We integrate the proposed pose normalization block at different stages of the speech recognition system and quantify the loss of performance related to pose changes and pose normalization techniques. In audio-visual experiments we also analyze the integration of the audio and visual streams. We show that an audio-visual system should account for non-frontal poses and normalization techniques in terms of the weight assigned to the visual stream in the classifier.

  19. Effects of Context Type on Lipreading and Listening Performance and Implications for Sentence Processing

    ERIC Educational Resources Information Center

    Spehar, Brent; Goebel, Stacey; Tye-Murray, Nancy

    2015-01-01

    Purpose: This study compared the use of 2 different types of contextual cues (sentence based and situation based) in 2 different modalities (visual only and auditory only). Method: Twenty young adults were tested with the Illustrated Sentence Test (Tye-Murray, Hale, Spehar, Myerson, & Sommers, 2014) and the Speech Perception in Noise Test…

  20. Cued Speech and the Reception of Spoken Language.

    ERIC Educational Resources Information Center

    Nicholls, Gaye H.; Ling, Daniel

    1982-01-01

    The Australian study investigated the effect of cued speech on the speech reception abilities of 18 profoundly hearing impaired children under seven conditions of presentation: audition; lipreading; audition and lipreading; cues; audition and cues; lipreading and cues; and audition, lipreading, and cues. (Author)

  1. 77 FR 24554 - Culturally Significant Objects Imported for Exhibition; Determinations: “Quay Brothers: On...

    Federal Register 2010, 2011, 2012, 2013, 2014

    2012-04-24

    ... Pharmacist's Prescription for Lip-Reading Puppets'' AGENCY: State Department. ACTION: Notice. SUMMARY: Notice... Lip-Reading Puppets'' imported from abroad for temporary exhibition within the United States, are...

  2. A Study of the Combined Use of a Hearing Aid and Tactual Aid in an Adult with Profound Hearing Loss

    ERIC Educational Resources Information Center

    Reed, Charlotte M.; Delhorne, Lorraine A.

    2006-01-01

    This study examined the benefits of the combined used of a hearing aid and tactual aid to supplement lip-reading in the reception of speech and for the recognition of environmental sounds in an adult with profound hearing loss. Speech conditions included lip-reading alone (L), lip-reading + tactual aid (L+TA) lip-reading + hearing aid (L+HA) and…

  3. A 12-consonant confusion study on a multiple-channel cochlear implant patient.

    PubMed

    Dowell, R C; Martin, L F; Tong, Y C; Clark, G M; Seligman, P M; Patrick, J F

    1982-12-01

    A consonant confusion study was undertaken on a multiple-channel cochlear implant patient using a wearable speech processing device. This patient suffered from total bilateral deafness acquired postlingually. The consonants /b/, /p/, /m/, /v/, /f/, /d/, /t/, /n/, /z/, /s/, /g/, /k/ were presented in a VCV context with the vowel /a/ as in father by a male and female speaker under three conditions: lipreading alone; electrical stimulation alone using the wearable speech processor and multiple-channel cochlear implant; lipreading in conjunction with electrical stimulation. No significant difference was detected between the results for the male and female speakers. The percentage correct scores for the pooled results of both speakers were lipreading alone--30%; electrical stimulation alone--48%; lipreading with electrical stimulation--70%. Performance was significantly better for lipreading with electrical stimulation than for lipreading alone and for electrical stimulation alone than for lipreading alone. An information transmission analysis demonstrated the effective integration of visual and auditory information for lipreading with electrical stimulation. There was a significant improvement in performance for the electrical stimulation alone condition over the 2 months of the study in contrast to no such improvement for lipreading alone.

  4. Computer Aided Lip Reading Training Tool

    ERIC Educational Resources Information Center

    Sarmasik, Gamze; Dalkilic, Gokhan; Kut, Alp; Cebi, Yalcin; Serbetcioglu, Bulent

    2007-01-01

    Worldwide auditory-verbal education is becoming widespread for deaf children. But many prelingually, late-diagnosed deaf children and adults may utilize neither hearing aids nor cochlear implants and needed the support of lip-reading. Therefore, lip-reading skill remains to be important for oral education programmes of hearing impaired. The…

  5. Phonetic Recalibration Only Occurs in Speech Mode

    ERIC Educational Resources Information Center

    Vroomen, Jean; Baart, Martijn

    2009-01-01

    Upon hearing an ambiguous speech sound dubbed onto lipread speech, listeners adjust their phonetic categories in accordance with the lipread information (recalibration) that tells what the phoneme should be. Here we used sine wave speech (SWS) to show that this tuning effect occurs if the SWS sounds are perceived as speech, but not if the sounds…

  6. Effects of a Wearable, Tactile Aid on Language Comprehension of Prelingual Profoundly Deaf Children.

    ERIC Educational Resources Information Center

    Proctor, Adele

    Factors influencing the use of nonacoustic aids (such as visual displays and tactile devices) with the hearing impaired are reviewed. The benefits of tactile devices in improving speech reading/lipreading and speech are pointed out. Tactile aids which provide information on rhythm, rate, intensity, and duration of speech increase lipreading and…

  7. Syllable Reception by Hearing-Impaired Children Trained from Infancy in Auditory-Oral Programs.

    ERIC Educational Resources Information Center

    Ling, Daniel; And Others

    1981-01-01

    Under both the audition alone and combined audition and lipreading conditions, Ss' performance was superior to that previously reported for profoundly hearing impaired children trained in schools in which a "multisensory" approach is typically used. (Author)

  8. Development of a speech autocuer

    NASA Technical Reports Server (NTRS)

    Bedles, R. L.; Kizakvich, P. N.; Lawson, D. T.; Mccartney, M. L.

    1980-01-01

    A wearable, visually based prosthesis for the deaf based upon the proven method for removing lipreading ambiguity known as cued speech was fabricated and tested. Both software and hardware developments are described, including a microcomputer, display, and speech preprocessor.

  9. What's Hearing Loss?

    MedlinePlus

    ... Medical Words En Español What Other Kids Are Reading Back-to-School Butterflies? Read This Chloe & Nurb ... like these to communicate: speechreading (also called lip-reading), which involves looking closely at a person's lips, ...

  10. Development of a speech autocuer

    NASA Astrophysics Data System (ADS)

    Bedles, R. L.; Kizakvich, P. N.; Lawson, D. T.; McCartney, M. L.

    1980-12-01

    A wearable, visually based prosthesis for the deaf based upon the proven method for removing lipreading ambiguity known as cued speech was fabricated and tested. Both software and hardware developments are described, including a microcomputer, display, and speech preprocessor.

  11. Modalities of memory: is reading lips like hearing voices?

    PubMed

    Maidment, David W; Macken, Bill; Jones, Dylan M

    2013-12-01

    Functional similarities in verbal memory performance across presentation modalities (written, heard, lipread) are often taken to point to a common underlying representational form upon which the modalities converge. We show here instead that the pattern of performance depends critically on presentation modality and different mechanisms give rise to superficially similar effects across modalities. Lipread recency is underpinned by different mechanisms to auditory recency, and while the effect of an auditory suffix on an auditory list is due to the perceptual grouping of the suffix with the list, the corresponding effect with lipread speech is due to misidentification of the lexical content of the lipread suffix. Further, while a lipread suffix does not disrupt auditory recency, an auditory suffix does disrupt recency for lipread lists. However, this effect is due to attentional capture ensuing from the presentation of an unexpected auditory event, and is evident both with verbal and nonverbal auditory suffixes. These findings add to a growing body of evidence that short-term verbal memory performance is determined by modality-specific perceptual and motor processes, rather than by the storage and manipulation of phonological representations. PMID:24041834

  12. Is automated conversion of video to text a reality?

    NASA Astrophysics Data System (ADS)

    Bowden, Richard; Cox, Stephen J.; Harvey, Richard W.; Lan, Yuxuan; Ong, Eng-Jon; Owen, Gari; Theobald, Barry-John

    2012-10-01

    A recent trend in law enforcement has been the use of Forensic lip-readers. Criminal activities are often recorded on CCTV or other video gathering systems. Knowledge of what suspects are saying enriches the evidence gathered but lip-readers, by their own admission, are fallible so, based on long term studies of automated lip-reading, we are investigating the possibilities and limitations of applying this technique under realistic conditions. We have adopted a step-by-step approach and are developing a capability when prior video information is available for the suspect of interest. We use the terminology video-to-text (V2T) for this technique by analogy with speech-to-text (S2T) which also has applications in security and law-enforcement.

  13. Tones for Profoundly Deaf Tone-Language Speakers.

    ERIC Educational Resources Information Center

    Ching, Teresa

    A study assessed the practical use of the simplified speech pattern approach to teaching lipreading in a tone language by comparing performance using an acoustic hearing-aid and a Sivo-aid in a tone labelling task. After initial assessment, subjects were given training to enhance perception of lexically contrastive tones, then post-tested. The…

  14. Effects of English Cued Speech on Speech Perception, Phonological Awareness and Literacy: A Case Study of a 9-Year-Old Deaf Boy Using a Cochlear Implant

    ERIC Educational Resources Information Center

    Rees, Rachel; Bladel, Judith

    2013-01-01

    Many studies have shown that French Cued Speech (CS) can enhance lipreading and the development of phonological awareness and literacy in deaf children but, as yet, there is little evidence that these findings can be generalized to English CS. This study investigated the possible effects of English CS on the speech perception, phonological…

  15. Visual Cues and Listening Effort: Individual Variability

    ERIC Educational Resources Information Center

    Picou, Erin M.; Ricketts, Todd A; Hornsby, Benjamin W. Y.

    2011-01-01

    Purpose: To investigate the effect of visual cues on listening effort as well as whether predictive variables such as working memory capacity (WMC) and lipreading ability affect the magnitude of listening effort. Method: Twenty participants with normal hearing were tested using a paired-associates recall task in 2 conditions (quiet and noise) and…

  16. Services for the Deaf: Check This Out.

    ERIC Educational Resources Information Center

    Lake County Public Library, Merrillville, IN.

    The Lake County (Indiana) Public Library provides a guide to materials for the deaf. The fiction list is annotated. Unannotated lists include materials on the deaf child, books on manual communication, resource books (law, education, directories, guides), poetry, lipreading materials, general information on deafness and the deaf, biographies,…

  17. THE HARD OF HEARING. PRENTICE-HALL FOUNDATIONS OF SPEECH PATHOLOGY SERIES.

    ERIC Educational Resources Information Center

    O'NEILL, JOHN J.

    BASIC INFORMATION ABOUT TESTING, DIAGNOSING, AND REHABILITATING THE HARD OF HEARING IS OFFERED IN THIS INTRODUCTORY TEXT. THE PHYSICS OF SOUND, AUDITORY THEORY, ANATOMY AND PATHOLOGY OF THE EAR, AND DIAGNOSTIC ROUTINES ARE DISCUSSED. A CHAPTER ON AURAL REHABILITATION INCLUDES AN OVERVIEW OF LIPREADING AND AUDITORY TRAINING TECHNIQUES FOR ADULTS…

  18. The What? When? and How? of Teaching Language to Deaf Children - Preschool and Primary Grades.

    ERIC Educational Resources Information Center

    Hogan, Sister James Lorene

    Three levels of work in language development for preschool and primary age deaf children are presented, along with suggested daily schedules and yearly programs. Skills covered are speech, lipreading, auditory training, and language. Instructions are given for teaching activities in the areas of the various parts of speech and types of sentences.…

  19. "The Business of Life": Educating Catholic Deaf Children in Late Nineteenth-Century England

    ERIC Educational Resources Information Center

    Mangion, Carmen M.

    2012-01-01

    Much of the debates in late nineteenth-century Britain regarding the education of deaf children revolved around communication. For many Victorians, sign language was unacceptable; many proponents of oralism attempted to "normalise" the hearing impaired by replacing deaf methods of communication with spoken language and lipreading. While debates on…

  20. "All Methods--and Wedded to None": The Deaf Education Methods Debate and Progressive Educational Reform in Toronto, Canada, 1922-1945

    ERIC Educational Resources Information Center

    Ellis, Jason A.

    2014-01-01

    This article is about the deaf education methods debate in the public schools of Toronto, Canada. The author demonstrates how pure oralism (lip-reading and speech instruction to the complete exclusion of sign language) and day school classes for deaf schoolchildren were introduced as a progressive school reform in 1922. Plans for further oralist…

  1. A LINGUISTIC APPROACH TO THE EDUCATION OF AURALLY-HANDICAPPED CHILDREN.

    ERIC Educational Resources Information Center

    LOWELL, EDGAR L.; WOODWARD, MARY F.

    IT IS HELD THAT ALTHOUGH PRESENT TUTORING TECHNIQUES IN LIPREADING FREQUENTLY ACHIEVE REMARKABLE RESULTS, THEY MIGHT BE BETTER UNDERSTOOD AND MADE MORE EFFECTIVE BY THE SYSTEMATIC APPLICATION OF LINGUISTIC CONCEPTS AND METHODS TO THE PROBLEMS INVOLVED. THE INVESTIGATION HAS BEEN CONCENTRATED ON A STUDY OF THE VISUAL PERCEPTIBILITY OF ENGLISH…

  2. SELF TEACHING IN THE DEVELOPMENT OF SPEECHREADING IN DEAF CHILDREN.

    ERIC Educational Resources Information Center

    NEYHUS, ARTHUR I.

    THE EFFECTIVENESS OF MOTION PICTURE FILMS AS A TEACHING DEVICE IN THE DEVELOPMENT OF LIPREADING SKILLS AND THE USE OF A CARTRIDGE-LOAD, SELF-WINDING EIGHT MILLIMETER PROJECTOR AS A TEACHING TOOL WERE STUDIED. IT WAS HYPOTHESIZED THAT DEAF AND HARD OF HEARING CHILDREN WOULD LEARN PRESCRIBED VOCABULARY MORE QUICKLY BY AUTOINSTRUCTIONAL FILM METHODS…

  3. Intermodal timing relations and audio-visual speech recognition by normal-hearing adults.

    PubMed

    McGrath, M; Summerfield, Q

    1985-02-01

    Audio-visual identification of sentences was measured as a function of audio delay in untrained observers with normal hearing; the soundtrack was replaced by rectangular pulses originally synchronized to the closing of the talker's vocal folds and then subjected to delay. When the soundtrack was delayed by 160 ms, identification scores were no better than when no acoustical information at all was provided. Delays of up to 80 ms had little effect on group-mean performance, but a separate analysis of a subgroup of better lipreaders showed a significant trend of reduced scores with increased delay in the range from 0-80 ms. A second experiment tested the interpretation that, although the main disruptive effect of the delay occurred on a syllabic time scale, better lipreaders might be attempting to use intermodal timing cues at a phonemic level. Normal-hearing observers determined whether a 120-Hz complex tone started before or after the opening of a pair of liplike Lissajou figures. Group-mean difference limens (70.7% correct DLs) were - 79 ms (sound leading) and + 138 ms (sound lagging), with no significant correlation between DLs and sentence lipreading scores. It was concluded that most observers, whether good lipreaders or not, possess insufficient sensitivity to intermodal timing cues in audio-visual speech for them to be used analogously to voice onset time in auditory speech perception. The results of both experiments imply that delays of up to about 40 ms introduced by signal-processing algorithms in aids to lipreading should not materially affect audio-visual speech understanding.

  4. Auditory, visual, and auditory-visual perception of vowels by hearing-impaired children.

    PubMed

    Hack, Z C; Erber, N P

    1982-03-01

    The vowels (foreign letters in text) were presented through auditory, visual, and combined auditory-visual modalities to hearing-impaired children having good, intermediate, and poor auditory work-recognition skills. When they received acoustic information only, children with good word-recognition skills confused neighboring vowels (i.e., those having similar formant frequencies). Children with intermediate work-recognition skills demonstrated this same difficulty and confused front and back vowels. Children with poor word-recognition skills identified vowels mainly on the basis of temporal and intensity cues. Through lipreading alone, all three groups distinguished spread from rounded vowels but could not reliably identify vowels within the categories. The first two groups exhibited only moderate difficulty in identifying vowels audiovisually. The third group, although showing a small amount of improvement over lipreading alone, still experienced difficulty in identifying vowels through combined auditory and visual modes.

  5. Bimodal bilingualism as multisensory training?: Evidence for improved audiovisual speech perception after sign language exposure.

    PubMed

    Williams, Joshua T; Darcy, Isabelle; Newman, Sharlene D

    2016-02-15

    The aim of the present study was to characterize effects of learning a sign language on the processing of a spoken language. Specifically, audiovisual phoneme comprehension was assessed before and after 13 weeks of sign language exposure. L2 ASL learners performed this task in the fMRI scanner. Results indicated that L2 American Sign Language (ASL) learners' behavioral classification of the speech sounds improved with time compared to hearing nonsigners. Results indicated increased activation in the supramarginal gyrus (SMG) after sign language exposure, which suggests concomitant increased phonological processing of speech. A multiple regression analysis indicated that learner's rating on co-sign speech use and lipreading ability was correlated with SMG activation. This pattern of results indicates that the increased use of mouthing and possibly lipreading during sign language acquisition may concurrently improve audiovisual speech processing in budding hearing bimodal bilinguals. PMID:26740404

  6. A multiple-channel cochlear implant: an evaluation using open-set CID sentences.

    PubMed

    Clark, G M; Tong, Y C; Martin, L F

    1981-04-01

    A multiple-channel cochlear implant and speech processor have been used in two postlingually deaf adult patients with a total hearing loss, to enable them to perceive varying degrees of running speech. The results have veen confirmed with open-set CID everyday sentence tests. Using the implant alone, the patients obtained 8% and 14% scores with prerecorded material, and 34% and 36% scores for "live" presentations. This was equivalent to the perception of 35% of connected discourse. When the implant was used in conjuction with lipreading, improvements of 188% and 386% were obtained over lipreading alone, and the scores were 68% and 98% which were equivalent to the perception of 60% and 95% of connected discourse. PMID:7219008

  7. A multiple-channel cochlear implant and wearable speech-processor: an audiological evaluation.

    PubMed

    Tong, Y C; Clark, G M; Dowell, R C; Martin, L F; Seligman, P M; Patrick, J F

    1981-01-01

    Stand audiological tests were administered to a totally deaf multiple-channel cochlear implant patient with a wearable speech-processor in a monitored sound field under the following conditions: a wearable unit activated alone (WA), lipreading with the wearable unit off (LA), and wearable unit activated in combination with lipreading (WL). Thresholds obtained for narrow-band noise signals indicated that the wearable unit allowed the patient to detect a variety of sounds at different frequencies. The results obtained in closed-set word tests and open-set word and sentence tests showed significant improvements in word and sentence scores from LA to WL. In the open-set (C.I.D.) sentence test, the patient scored 22% for LA and 76% for WL. The WL score of 76% correlates with a satisfactory performance in understanding connected speech. The patient also scored 30% correct in a test involving the recognition of environmental sounds.

  8. A sex difference in visual influence on heard speech.

    PubMed

    Irwin, Julia R; Whalen, D H; Fowler, Carol A

    2006-05-01

    Reports of sex differences in language processing are inconsistent and are thought to vary by task type and difficulty. In two experiments, we investigated a sex difference in visual influence onheard speech (the McGurk effect). First, incongruent consonant-vowel stimuli were presented where the visual portion of the signal was brief (100 msec) or full (temporally equivalent to the auditory). Second, to determine whether men and women differed in their ability to extract visual speech information from these brief stimuli, the same stimuli were presented to new participants with an additional visual-only (lipread) condition. In both experiments, women showed a significantly greater visual influence on heard speech than did men for the brief visual stimuli. No sex differences for the full stimuli or in the ability to lipread were found. These findings indicate that the more challenging brief visual stimuli elicit sex differences in the processing of audiovisual speech.

  9. Electrophysiological evidence for speech-specific audiovisual integration.

    PubMed

    Baart, Martijn; Stekelenburg, Jeroen J; Vroomen, Jean

    2014-01-01

    Lip-read speech is integrated with heard speech at various neural levels. Here, we investigated the extent to which lip-read induced modulations of the auditory N1 and P2 (measured with EEG) are indicative of speech-specific audiovisual integration, and we explored to what extent the ERPs were modulated by phonetic audiovisual congruency. In order to disentangle speech-specific (phonetic) integration from non-speech integration, we used Sine-Wave Speech (SWS) that was perceived as speech by half of the participants (they were in speech-mode), while the other half was in non-speech mode. Results showed that the N1 obtained with audiovisual stimuli peaked earlier than the N1 evoked by auditory-only stimuli. This lip-read induced speeding up of the N1 occurred for listeners in speech and non-speech mode. In contrast, if listeners were in speech-mode, lip-read speech also modulated the auditory P2, but not if listeners were in non-speech mode, thus revealing speech-specific audiovisual binding. Comparing ERPs for phonetically congruent audiovisual stimuli with ERPs for incongruent stimuli revealed an effect of phonetic stimulus congruency that started at ~200 ms after (in)congruence became apparent. Critically, akin to the P2 suppression, congruency effects were only observed if listeners were in speech mode, and not if they were in non-speech mode. Using identical stimuli, we thus confirm that audiovisual binding involves (partially) different neural mechanisms for sound processing in speech and non-speech mode.

  10. How is the McGurk effect modulated by Cued Speech in deaf and hearing adults?

    PubMed

    Bayard, Clémence; Colin, Cécile; Leybaert, Jacqueline

    2014-01-01

    Speech perception for both hearing and deaf people involves an integrative process between auditory and lip-reading information. In order to disambiguate information from lips, manual cues from Cued Speech may be added. Cued Speech (CS) is a system of manual aids developed to help deaf people to clearly and completely understand speech visually (Cornett, 1967). Within this system, both labial and manual information, as lone input sources, remain ambiguous. Perceivers, therefore, have to combine both types of information in order to get one coherent percept. In this study, we examined how audio-visual (AV) integration is affected by the presence of manual cues and on which form of information (auditory, labial or manual) the CS receptors primarily rely. To address this issue, we designed a unique experiment that implemented the use of AV McGurk stimuli (audio /pa/ and lip-reading /ka/) which were produced with or without manual cues. The manual cue was congruent with either auditory information, lip information or the expected fusion. Participants were asked to repeat the perceived syllable aloud. Their responses were then classified into four categories: audio (when the response was /pa/), lip-reading (when the response was /ka/), fusion (when the response was /ta/) and other (when the response was something other than /pa/, /ka/ or /ta/). Data were collected from hearing impaired individuals who were experts in CS (all of which had either cochlear implants or binaural hearing aids; N = 8), hearing-individuals who were experts in CS (N = 14) and hearing-individuals who were completely naïve of CS (N = 15). Results confirmed that, like hearing-people, deaf people can merge auditory and lip-reading information into a single unified percept. Without manual cues, McGurk stimuli induced the same percentage of fusion responses in both groups. Results also suggest that manual cues can modify the AV integration and that their impact differs between hearing and deaf people.

  11. How is the McGurk effect modulated by Cued Speech in deaf and hearing adults?

    PubMed Central

    Bayard, Clémence; Colin, Cécile; Leybaert, Jacqueline

    2014-01-01

    Speech perception for both hearing and deaf people involves an integrative process between auditory and lip-reading information. In order to disambiguate information from lips, manual cues from Cued Speech may be added. Cued Speech (CS) is a system of manual aids developed to help deaf people to clearly and completely understand speech visually (Cornett, 1967). Within this system, both labial and manual information, as lone input sources, remain ambiguous. Perceivers, therefore, have to combine both types of information in order to get one coherent percept. In this study, we examined how audio-visual (AV) integration is affected by the presence of manual cues and on which form of information (auditory, labial or manual) the CS receptors primarily rely. To address this issue, we designed a unique experiment that implemented the use of AV McGurk stimuli (audio /pa/ and lip-reading /ka/) which were produced with or without manual cues. The manual cue was congruent with either auditory information, lip information or the expected fusion. Participants were asked to repeat the perceived syllable aloud. Their responses were then classified into four categories: audio (when the response was /pa/), lip-reading (when the response was /ka/), fusion (when the response was /ta/) and other (when the response was something other than /pa/, /ka/ or /ta/). Data were collected from hearing impaired individuals who were experts in CS (all of which had either cochlear implants or binaural hearing aids; N = 8), hearing-individuals who were experts in CS (N = 14) and hearing-individuals who were completely naïve of CS (N = 15). Results confirmed that, like hearing-people, deaf people can merge auditory and lip-reading information into a single unified percept. Without manual cues, McGurk stimuli induced the same percentage of fusion responses in both groups. Results also suggest that manual cues can modify the AV integration and that their impact differs between hearing and deaf people

  12. How to improve communication with deaf children in the dental clinic.

    PubMed

    Alsmark, Silvia San Bernardino; García, Joaquín; Martínez, María Rosa Mourelle; López, Nuria Esther Gallardo

    2007-12-01

    It may be difficult for hearing-impaired people to communicate with people who hear. In the health care area, there is often little awareness of the communication barriers faced by the deaf and, in dentistry, the attitude adopted towards the deaf is not always correct. A review is given of the basic rules and advice given for communicating with the hearing-impaired. The latter are classified in three groups - lip-readers, sign language users and those with hearing aids. The advice given varies for the different groups although the different methods of communication are often combined (e.g. sign language plus lip-reading, hearing-aids plus lip-reading). Treatment of hearing-impaired children in the dental clinic must be personalised. Each child is different, depending on the education received, the communication skills possessed, family factors (degree of parental protection, etc.), the existence of associated problems (learning difficulties), degree of loss of hearing, age, etc. PMID:18059242

  13. Visual abilities are important for auditory-only speech recognition: evidence from autism spectrum disorder.

    PubMed

    Schelinski, Stefanie; Riedel, Philipp; von Kriegstein, Katharina

    2014-12-01

    In auditory-only conditions, for example when we listen to someone on the phone, it is essential to fast and accurately recognize what is said (speech recognition). Previous studies have shown that speech recognition performance in auditory-only conditions is better if the speaker is known not only by voice, but also by face. Here, we tested the hypothesis that such an improvement in auditory-only speech recognition depends on the ability to lip-read. To test this we recruited a group of adults with autism spectrum disorder (ASD), a condition associated with difficulties in lip-reading, and typically developed controls. All participants were trained to identify six speakers by name and voice. Three speakers were learned by a video showing their face and three others were learned in a matched control condition without face. After training, participants performed an auditory-only speech recognition test that consisted of sentences spoken by the trained speakers. As a control condition, the test also included speaker identity recognition on the same auditory material. The results showed that, in the control group, performance in speech recognition was improved for speakers known by face in comparison to speakers learned in the matched control condition without face. The ASD group lacked such a performance benefit. For the ASD group auditory-only speech recognition was even worse for speakers known by face compared to speakers not known by face. In speaker identity recognition, the ASD group performed worse than the control group independent of whether the speakers were learned with or without face. Two additional visual experiments showed that the ASD group performed worse in lip-reading whereas face identity recognition was within the normal range. The findings support the view that auditory-only communication involves specific visual mechanisms. Further, they indicate that in ASD, speaker-specific dynamic visual information is not available to optimize auditory

  14. Visual abilities are important for auditory-only speech recognition: evidence from autism spectrum disorder.

    PubMed

    Schelinski, Stefanie; Riedel, Philipp; von Kriegstein, Katharina

    2014-12-01

    In auditory-only conditions, for example when we listen to someone on the phone, it is essential to fast and accurately recognize what is said (speech recognition). Previous studies have shown that speech recognition performance in auditory-only conditions is better if the speaker is known not only by voice, but also by face. Here, we tested the hypothesis that such an improvement in auditory-only speech recognition depends on the ability to lip-read. To test this we recruited a group of adults with autism spectrum disorder (ASD), a condition associated with difficulties in lip-reading, and typically developed controls. All participants were trained to identify six speakers by name and voice. Three speakers were learned by a video showing their face and three others were learned in a matched control condition without face. After training, participants performed an auditory-only speech recognition test that consisted of sentences spoken by the trained speakers. As a control condition, the test also included speaker identity recognition on the same auditory material. The results showed that, in the control group, performance in speech recognition was improved for speakers known by face in comparison to speakers learned in the matched control condition without face. The ASD group lacked such a performance benefit. For the ASD group auditory-only speech recognition was even worse for speakers known by face compared to speakers not known by face. In speaker identity recognition, the ASD group performed worse than the control group independent of whether the speakers were learned with or without face. Two additional visual experiments showed that the ASD group performed worse in lip-reading whereas face identity recognition was within the normal range. The findings support the view that auditory-only communication involves specific visual mechanisms. Further, they indicate that in ASD, speaker-specific dynamic visual information is not available to optimize auditory

  15. The Self-Advantage in Visual Speech Processing Enhances Audiovisual Speech Recognition in Noise

    PubMed Central

    Tye-Murray, Nancy; Spehar, Brent P.; Myerson, Joel; Hale, Sandra; Sommers, Mitchell S.

    2014-01-01

    Individuals lipread themselves more accurately than they lipread others when only the visual speech signal is available (Tye-Murray, Spehar, Myerson, Hale, & Sommers, 2013). This self-advantage for vision-only speech recognition is consistent with the common-coding hypothesis (Prinz, 1997), which posits (1) that observing an action activates the same motor plan representation as actually performing that action and (2) that observing one’s own actions activates motor plan representations more than the others’ actions because of greater congruity between percepts and corresponding motor plans. The present study extends this line of research to audiovisual speech recognition by examining whether there is a self-advantage when the visual signal is added to the auditory signal under poor listening conditions. Participants were assigned to sub-groups for round-robin testing in which each participant was paired with every member of their subgroup, including themselves, serving as both talker and listener/observer. On average, the benefit participants obtained from the visual signal when they were the talker was greater than when the talker was someone else and also was greater than the benefit others obtained from observing as well as listening to them. Moreover, the self-advantage in audiovisual speech recognition was significant after statistically controlling for individual differences in both participants’ ability to benefit from a visual speech signal and the extent to which their own visual speech signal benefited others. These findings are consistent with our previous finding of a self-advantage in lipreading and with the hypothesis of a common code for action perception and motor plan representation. PMID:25421408

  16. Phonetic recalibration does not depend on working memory

    PubMed Central

    Baart, Martijn

    2010-01-01

    Listeners use lipread information to adjust the phonetic boundary between two speech categories (phonetic recalibration, Bertelson et al. 2003). Here, we examined phonetic recalibration while listeners were engaged in a visuospatial or verbal memory working memory task under different memory load conditions. Phonetic recalibration was—like selective speech adaptation—not affected by a concurrent verbal or visuospatial memory task. This result indicates that phonetic recalibration is a low-level process not critically depending on processes used in verbal- or visuospatial working memory. PMID:20437168

  17. Effects of aging on audio-visual speech integration.

    PubMed

    Huyse, Aurélie; Leybaert, Jacqueline; Berthommier, Frédéric

    2014-10-01

    This study investigated the impact of aging on audio-visual speech integration. A syllable identification task was presented in auditory-only, visual-only, and audio-visual congruent and incongruent conditions. Visual cues were either degraded or unmodified. Stimuli were embedded in stationary noise alternating with modulated noise. Fifteen young adults and 15 older adults participated in this study. Results showed that older adults had preserved lipreading abilities when the visual input was clear but not when it was degraded. The impact of aging on audio-visual integration also depended on the quality of the visual cues. In the visual clear condition, the audio-visual gain was similar in both groups and analyses in the framework of the fuzzy-logical model of perception confirmed that older adults did not differ from younger adults in their audio-visual integration abilities. In the visual reduction condition, the audio-visual gain was reduced in the older group, but only when the noise was stationary, suggesting that older participants could compensate for the loss of lipreading abilities by using the auditory information available in the valleys of the noise. The fuzzy-logical model of perception confirmed the significant impact of aging on audio-visual integration by showing an increased weight of audition in the older group. PMID:25324091

  18. Phonetic matching of auditory and visual speech develops during childhood: evidence from sine-wave speech.

    PubMed

    Baart, Martijn; Bortfeld, Heather; Vroomen, Jean

    2015-01-01

    The correspondence between auditory speech and lip-read information can be detected based on a combination of temporal and phonetic cross-modal cues. Here, we determined the point in developmental time at which children start to effectively use phonetic information to match a speech sound with one of two articulating faces. We presented 4- to 11-year-olds (N=77) with three-syllabic sine-wave speech replicas of two pseudo-words that were perceived as non-speech and asked them to match the sounds with the corresponding lip-read video. At first, children had no phonetic knowledge about the sounds, and matching was thus based on the temporal cues that are fully retained in sine-wave speech. Next, we trained all children to perceive the phonetic identity of the sine-wave speech and repeated the audiovisual (AV) matching task. Only at around 6.5 years of age did the benefit of having phonetic knowledge about the stimuli become apparent, thereby indicating that AV matching based on phonetic cues presumably develops more slowly than AV matching based on temporal cues.

  19. Effects of aging on audio-visual speech integration.

    PubMed

    Huyse, Aurélie; Leybaert, Jacqueline; Berthommier, Frédéric

    2014-10-01

    This study investigated the impact of aging on audio-visual speech integration. A syllable identification task was presented in auditory-only, visual-only, and audio-visual congruent and incongruent conditions. Visual cues were either degraded or unmodified. Stimuli were embedded in stationary noise alternating with modulated noise. Fifteen young adults and 15 older adults participated in this study. Results showed that older adults had preserved lipreading abilities when the visual input was clear but not when it was degraded. The impact of aging on audio-visual integration also depended on the quality of the visual cues. In the visual clear condition, the audio-visual gain was similar in both groups and analyses in the framework of the fuzzy-logical model of perception confirmed that older adults did not differ from younger adults in their audio-visual integration abilities. In the visual reduction condition, the audio-visual gain was reduced in the older group, but only when the noise was stationary, suggesting that older participants could compensate for the loss of lipreading abilities by using the auditory information available in the valleys of the noise. The fuzzy-logical model of perception confirmed the significant impact of aging on audio-visual integration by showing an increased weight of audition in the older group.

  20. Speech Analysis Based On Image Information from Lip Movement

    NASA Astrophysics Data System (ADS)

    Talha, Kamil S.; Wan, Khairunizam; Za'ba, S. K.; Mohamad Razlan, Zuradzman; B, Shahriman A.

    2013-12-01

    Deaf and hard of hearing people often have problems being able to understand and lip read other people. Usually deaf and hard of hearing people feel left out of conversation and sometimes they are actually ignored by other people. There are a variety of ways hearing-impaired person can communicate and gain accsss to the information. Communication support includes both technical and human aids. Human aids include interpreters, lip-readers and note-takers. Interpreters translate the Sign Language and must therefore be qualified. In this paper, vision system is used to track movements of the lip. In the experiment, the proposed system succesfully can differentiate 11 type of phonemes and then classified it to the respective viseme group. By using the proposed system the hearing-impaired persons could practise pronaunciations by themselve without support from the instructor.

  1. PERVALE-S: a new cognitive task to assess deaf people’s ability to perceive basic and social emotions

    PubMed Central

    Mestre, José M.; Larrán, Cristina; Herrero, Joaquín; Guil, Rocío; de la Torre, Gabriel G.

    2015-01-01

    A poorly understood aspect of deaf people (DP) is how their emotional information is processed. Verbal ability is key to improve emotional knowledge in people. Nevertheless, DP are unable to distinguish intonation, intensity, and the rhythm of language due to lack of hearing. Some DP have acquired both lip-reading abilities and sign language, but others have developed only sign language. PERVALE-S was developed to assess the ability of DP to perceive both social and basic emotions. PERVALE-S presents different sets of visual images of a real deaf person expressing both basic and social emotions, according to the normative standard of emotional expressions in Spanish Sign Language. Emotional expression stimuli were presented at two different levels of intensity (1: low; and 2: high) because DP do not distinguish an object in the same way as hearing people (HP) do. Then, participants had to click on the more suitable emotional expression. PERVALE-S contains video instructions (given by a sign language interpreter) to improve DP’s understanding about how to use the software. DP had to watch the videos before answering the items. To test PERVALE-S, a sample of 56 individuals was recruited (18 signers, 8 lip-readers, and 30 HP). Participants also performed a personality test (High School Personality Questionnaire adapted) and a fluid intelligence (Gf) measure (RAPM). Moreover, all deaf participants were rated by four teachers for the deaf. Results: there were no significant differences between deaf and HP in performance in PERVALE-S. Confusion matrices revealed that embarrassment, envy, and jealousy were worse perceived. Age was just related to social-emotional tasks (but not in basic emotional tasks). Emotional perception ability was related mainly to warmth and consciousness, but negatively related to tension. Meanwhile, Gf was related to only social-emotional tasks. There were no gender differences. PMID:26300828

  2. Accessibility of spoken, written, and sign language in Landau-Kleffner syndrome: a linguistic and functional MRI study.

    PubMed

    Sieratzki, J S; Calvert, G A; Brammer, M; David, A; Woll, B

    2001-06-01

    Landau-Kleffner syndrome (LKS) is an acquired aphasia which begins in childhood and is thought to arise from an epileptic disorder within the auditory speech cortex. Although the epilepsy usually subsides at puberty, a severe communication impairment often persists. Here we report on a detailed study of a 26-year old, left-handed male, with onset of LKS at age 5 years, who is aphasic for English but who learned British Sign Language (BSL) at age 13. We have investigated his skills in different language modalities, recorded EEGs during wakefulness, sleep, and under conditions of auditory stimulation, measured brain stem auditory-evoked potentials (BAEP), and performed functional MRI (fMRI) during a range of linguistic tasks. Our investigation demonstrated severe restrictions in comprehension and production of spoken English as well as lip-reading, while reading was comparatively less impaired. BSL was by far the most efficient mode of communication. All EEG recordings were normal, while BAEP showed minor abnormalities. fMRI revealed: 1) powerful and extensive bilateral (R > L) activation of auditory cortices in response to heard speech, much stronger than when listening to music; 2) very little response to silent lip-reading; 3) strong activation in the temporo-parieto-occipital association cortex, exclusively in the right hemisphere (RH), when viewing BSL signs. Analysis of these findings provides novel insights into the disturbance of the auditory speech cortex which underlies LKS and its diagnostic evaluation by fMRI, and underpins a strategy of restoring communication abilities in LKS through a natural sign language of the deaf (with Video)

  3. Audibility and visual biasing in speech perception

    NASA Astrophysics Data System (ADS)

    Clement, Bart Richard

    Although speech perception has been considered a predominantly auditory phenomenon, large benefits from vision in degraded acoustic conditions suggest integration of audition and vision. More direct evidence of this comes from studies of audiovisual disparity that demonstrate vision can bias and even dominate perception (McGurk & MacDonald, 1976). It has been observed that hearing-impaired listeners demonstrate more visual biasing than normally hearing listeners (Walden et al., 1990). It is argued here that stimulus audibility must be equated across groups before true differences can be established. In the present investigation, effects of visual biasing on perception were examined as audibility was degraded for 12 young normally hearing listeners. Biasing was determined by quantifying the degree to which listener identification functions for a single synthetic auditory /ba-da-ga/ continuum changed across two conditions: (1)an auditory-only listening condition; and (2)an auditory-visual condition in which every item of the continuum was synchronized with visual articulations of the consonant-vowel (CV) tokens /ba/ and /ga/, as spoken by each of two talkers. Audibility was altered by presenting the conditions in quiet and in noise at each of three signal-to- noise (S/N) ratios. For the visual-/ba/ context, large effects of audibility were found. As audibility decreased, visual biasing increased. A large talker effect also was found, with one talker eliciting more biasing than the other. An independent lipreading measure demonstrated that this talker was more visually intelligible than the other. For the visual-/ga/ context, audibility and talker effects were less robust, possibly obscured by strong listener effects, which were characterized by marked differences in perceptual processing patterns among participants. Some demonstrated substantial biasing whereas others demonstrated little, indicating a strong reliance on audition even in severely degraded acoustic

  4. Hearing Loss: Communicating With the Patient Who Is Deaf or Hard of Hearing.

    PubMed

    McKee, Michael M; Moreland, Christopher; Atcherson, Samuel R; Zazove, Philip

    2015-07-01

    Hearing loss impairs health care communication and adversely affects patient satisfaction, treatment adherence, and use of health services. Hearing loss is the third most common chronic health condition among older patients after hypertension and arthritis, but only 15% to 18% of older adults are screened for hearing loss during health maintenance examinations. Patients with hearing loss may be reluctant to disclose it because of fear of ageism, perceptions of disability, and vanity. Lipreading and note writing often are ineffective ways to communicate with deaf and hard of hearing (DHH) patients who use American Sign Language; use of medical sign language interpreters is preferred. A variety of strategies can improve the quality of health care communication for DHH patients, such as the physician facing the patient, listening attentively, and using visual tools. Physicians should learn what hearing loss means to the DHH patient. Deaf American Sign Language users may not perceive hearing loss as a disability but as a cultural identity. Patients' preferred communication strategies will vary. Relay services, electronic communication, and other telecommunications methods can be helpful, but family physicians and medical staff should learn from each DHH patient about which communication strategies will work best. PMID:26161525

  5. Segmentation of human face using gradient-based approach

    NASA Astrophysics Data System (ADS)

    Baskan, Selin; Bulut, M. Mete; Atalay, Volkan

    2001-04-01

    This paper describes a method for automatic segmentation of facial features such as eyebrows, eyes, nose, mouth and ears in color images. This work is an initial step for wide range of applications based on feature-based approaches, such as face recognition, lip-reading, gender estimation, facial expression analysis, etc. Human face can be characterized by its skin color and nearly elliptical shape. For this purpose, face detection is performed using color and shape information. Uniform illumination is assumed. No restrictions on glasses, make-up, beard, etc. are imposed. Facial features are extracted using the vertically and horizontally oriented gradient projections. The gradient of a minimum with respect to its neighbor maxima gives the boundaries of a facial feature. Each facial feature has a different horizontal characteristic. These characteristics are derived by extensive experimentation with many face images. Using fuzzy set theory, the similarity between the candidate and the feature characteristic under consideration is calculated. Gradient-based method is accompanied by the anthropometrical information, for robustness. Ear detection is performed using contour-based shape descriptors. This method detects the facial features and circumscribes each facial feature with the smallest rectangle possible. AR database is used for testing. The developed method is also suitable for real-time systems.

  6. Auditory Midbrain Implant: Research and Development Towards a Second Clinical Trial

    PubMed Central

    Lim, Hubert H.; Lenarz, Thomas

    2015-01-01

    The cochlear implant is considered one of the most successful neural prostheses to date, which was made possible by visionaries who continued to develop the cochlear implant through multiple technological and clinical challenges. However, patients without a functional auditory nerve or implantable cochlea cannot benefit from a cochlear implant. The focus of the paper is to review the development and translation of a new type of central auditory prosthesis for this group of patients, which is known as the auditory midbrain implant (AMI) and is designed for electrical stimulation within the inferior colliculus. The rationale and results for the first AMI clinical study using a multi-site single-shank array will be presented initially. Although the AMI has achieved encouraging results in terms of safety and improvements in lip-reading capabilities and environmental awareness, it has not yet provided sufficient speech perception. Animal and human data will then be presented to show that a two-shank AMI array can potentially improve hearing performance by targeting specific neurons of the inferior colliculus. Modifications to the AMI array design, stimulation strategy, and surgical approach have been made that are expected to improve hearing performance in the patients implanted with a two-shank array in an upcoming clinical trial funded by the National Institutes of Health. Positive outcomes from this clinical trial will motivate new efforts and developments toward improving central auditory prostheses for those who cannot sufficiently benefit from cochlear implants. PMID:25613994

  7. Automatic lip reading by using multimodal visual features

    NASA Astrophysics Data System (ADS)

    Takahashi, Shohei; Ohya, Jun

    2013-12-01

    Since long time ago, speech recognition has been researched, though it does not work well in noisy places such as in the car or in the train. In addition, people with hearing-impaired or difficulties in hearing cannot receive benefits from speech recognition. To recognize the speech automatically, visual information is also important. People understand speeches from not only audio information, but also visual information such as temporal changes in the lip shape. A vision based speech recognition method could work well in noisy places, and could be useful also for people with hearing disabilities. In this paper, we propose an automatic lip-reading method for recognizing the speech by using multimodal visual information without using any audio information such as speech recognition. First, the ASM (Active Shape Model) is used to track and detect the face and lip in a video sequence. Second, the shape, optical flow and spatial frequencies of the lip features are extracted from the lip detected by ASM. Next, the extracted multimodal features are ordered chronologically so that Support Vector Machine is performed in order to learn and classify the spoken words. Experiments for classifying several words show promising results of this proposed method.

  8. [Central auditory prosthesis].

    PubMed

    Lenarz, T; Lim, H; Joseph, G; Reuter, G; Lenarz, M

    2009-06-01

    Deaf patients with severe sensory hearing loss can benefit from a cochlear implant (CI), which stimulates the auditory nerve fibers. However, patients who do not have an intact auditory nerve cannot benefit from a CI. The majority of these patients are neurofibromatosis type 2 (NF2) patients who developed neural deafness due to growth or surgical removal of a bilateral acoustic neuroma. The only current solution is the auditory brainstem implant (ABI), which stimulates the surface of the cochlear nucleus in the brainstem. Although the ABI provides improvement in environmental awareness and lip-reading capabilities, only a few NF2 patients have achieved some limited open set speech perception. In the search for alternative procedures our research group in collaboration with Cochlear Ltd. (Australia) developed a human prototype auditory midbrain implant (AMI), which is designed to electrically stimulate the inferior colliculus (IC). The IC has the potential as a new target for an auditory prosthesis as it provides access to neural projections necessary for speech perception as well as a systematic map of spectral information. In this paper the present status of research and development in the field of central auditory prostheses is presented with respect to technology, surgical technique and hearing results as well as the background concepts of ABI and AMI. PMID:19517084

  9. Language access and theory of mind reasoning: evidence from deaf children in bilingual and oralist environments.

    PubMed

    Meristo, Marek; Falkman, Kerstin W; Hjelmquist, Erland; Tedoldi, Mariantonia; Surian, Luca; Siegal, Michael

    2007-09-01

    This investigation examined whether access to sign language as a medium for instruction influences theory of mind (ToM) reasoning in deaf children with similar home language environments. Experiment 1 involved 97 deaf Italian children ages 4-12 years: 56 were from deaf families and had LIS (Italian Sign Language) as their native language, and 41 had acquired LIS as late signers following contact with signers outside their hearing families. Children receiving bimodal/bilingual instruction in LIS together with Sign-Supported and spoken Italian significantly outperformed children in oralist schools in which communication was in Italian and often relied on lipreading. Experiment 2 involved 61 deaf children in Estonia and Sweden ages 6-16 years. On a wide variety of ToM tasks, bilingually instructed native signers in Estonian Sign Language and spoken Estonian succeeded at a level similar to age-matched hearing children. They outperformed bilingually instructed late signers and native signers attending oralist schools. Particularly for native signers, access to sign language in a bilingual environment may facilitate conversational exchanges that promote the expression of ToM by enabling children to monitor others' mental states effectively. PMID:17723042

  10. Prototype to product—developing a commercially viable neural prosthesis

    NASA Astrophysics Data System (ADS)

    Seligman, Peter

    2009-12-01

    The Cochlear implant or 'Bionic ear' is a device that enables people who do not get sufficient benefit from a hearing aid to communicate with the hearing world. The Cochlear implant is not an amplifier, but a device that electrically stimulates the auditory nerve in a way that crudely mimics normal hearing, thus providing a hearing percept. Many recipients are able to understand running speech without the help of lipreading. Cochlear implants have reached a stage of maturity where there are now 170 000 recipients implanted worldwide. The commercial development of these devices has occurred over the last 30 years. This development has been multidisciplinary, including audiologists, engineers, both mechanical and electrical, histologists, materials scientists, physiologists, surgeons and speech pathologists. This paper will trace the development of the device we have today, from the engineering perspective. The special challenges of designing an active device that will work in the human body for a lifetime will be outlined. These challenges include biocompatibility, extreme reliability, safety, patient fitting and surgical issues. It is emphasized that the successful development of a neural prosthesis requires the partnership of academia and industry.

  11. ERP adaptation provides direct evidence for early mirror neuron activation in the inferior parietal lobule.

    PubMed

    Möhring, Nicole; Brandt, Emily S L; Mohr, Bettina; Pulvermüller, Friedemann; Neuhaus, Andres H

    2014-10-01

    Mirror neuron systems are frequently investigated by assessing overlapping brain activity during observation and execution of actions; however, distinct neuronal subpopulations may be activated that fall below the spatial resolution of magnetic resonance techniques. This shortfall can be resolved using repetition suppression paradigms that identify physiological adaptation processes caused by repeated activation of identical neuronal circuits. Here, event-related potentials were used to investigate the time course of mirror neuron circuit activation using repetition suppression within and across action observation and action execution modalities. In a lip-reading and speech production paradigm, the N170 component indexed stimulus repetition by adapting to both cross-modal and intra-modal repetitions in the left hemisphere. Neuronal source localization revealed activation of the left inferior parietal lobule during cross-modal relative to intra-modal trials. These results provide support for the position that the same neuronal circuits are activated in perceiving and performing articulatory actions. Moreover, our data strongly suggest that inferior parietal lobule mirror neurons are activated relatively early in time, which indicates partly automatic processes of linguistic perception and mirroring. Repetition suppression paradigms therefore help to elucidate neuronal correlates of different cognitive processes and may serve as a starting point for advanced electrophysiological research on mirror neurons.

  12. ERP adaptation provides direct evidence for early mirror neuron activation in the inferior parietal lobule.

    PubMed

    Möhring, Nicole; Brandt, Emily S L; Mohr, Bettina; Pulvermüller, Friedemann; Neuhaus, Andres H

    2014-10-01

    Mirror neuron systems are frequently investigated by assessing overlapping brain activity during observation and execution of actions; however, distinct neuronal subpopulations may be activated that fall below the spatial resolution of magnetic resonance techniques. This shortfall can be resolved using repetition suppression paradigms that identify physiological adaptation processes caused by repeated activation of identical neuronal circuits. Here, event-related potentials were used to investigate the time course of mirror neuron circuit activation using repetition suppression within and across action observation and action execution modalities. In a lip-reading and speech production paradigm, the N170 component indexed stimulus repetition by adapting to both cross-modal and intra-modal repetitions in the left hemisphere. Neuronal source localization revealed activation of the left inferior parietal lobule during cross-modal relative to intra-modal trials. These results provide support for the position that the same neuronal circuits are activated in perceiving and performing articulatory actions. Moreover, our data strongly suggest that inferior parietal lobule mirror neurons are activated relatively early in time, which indicates partly automatic processes of linguistic perception and mirroring. Repetition suppression paradigms therefore help to elucidate neuronal correlates of different cognitive processes and may serve as a starting point for advanced electrophysiological research on mirror neurons. PMID:25017963

  13. Sizing up the competition: quantifying the influence of the mental lexicon on auditory and visual spoken word recognition.

    PubMed

    Strand, Julia F; Sommers, Mitchell S

    2011-09-01

    Much research has explored how spoken word recognition is influenced by the architecture and dynamics of the mental lexicon (e.g., Luce and Pisoni, 1998; McClelland and Elman, 1986). A more recent question is whether the processes underlying word recognition are unique to the auditory domain, or whether visually perceived (lipread) speech may also be sensitive to the structure of the mental lexicon (Auer, 2002; Mattys, Bernstein, and Auer, 2002). The current research was designed to test the hypothesis that both aurally and visually perceived spoken words are isolated in the mental lexicon as a function of their modality-specific perceptual similarity to other words. Lexical competition (the extent to which perceptually similar words influence recognition of a stimulus word) was quantified using metrics that are well-established in the literature, as well as a statistical method for calculating perceptual confusability based on the phi-square statistic. Both auditory and visual spoken word recognition were influenced by modality-specific lexical competition as well as stimulus word frequency. These findings extend the scope of activation-competition models of spoken word recognition and reinforce the hypothesis (Auer, 2002; Mattys et al., 2002) that perceptual and cognitive properties underlying spoken word recognition are not specific to the auditory domain. In addition, the results support the use of the phi-square statistic as a better predictor of lexical competition than metrics currently used in models of spoken word recognition.

  14. The contribution of fundamental frequency, amplitude envelope, and voicing duration cues to speechreading in normal-hearing subjects.

    PubMed

    Grant, K W; Ardell, L H; Kuhl, P K; Sparks, D W

    1985-02-01

    The ability to combine speechreading (i.e., lipreading) with prosodic information extracted from the low-frequency regions of speech was evaluated with three normally hearing subjects. The subjects were tested in a connected discourse tracking procedure which measures the rate at which spoken text can be repeated back without any errors. Receptive conditions included speechreading alone (SA), speechreading plus amplitude envelope cues (AM), speechreading plus fundamental frequency cues (FM), and speechreading plus intensity-modulated fundamental frequency cues (AM + FM). In a second experiment, one subject was further tested in a speechreading plus voicing duration cue condition (DUR). Speechreading performance was best in the AM + FM condition (83.6 words per minute,) and worst in the SA condition (41.1 words per minute). Tracking levels in the AM, FM, and DUR conditions were 73.7, 73.6, and 65.4 words per minute, respectively. The average tracking rate obtained when subjects were allowed to listen to the talker's normal (unfiltered) speech (NS condition) was 108.3 words per minute. These results demonstrate that speechreaders can use information related to the rhythm, stress, and intonation patterns of speech to improve their speechreading performance.

  15. Speechreading development in deaf and hearing children: introducing a new Test of Child Speechreading (ToCS)

    PubMed Central

    Kyle, Fiona Elizabeth; Campbell, Ruth; Mohammed, Tara; Coleman, Mike; MacSweeney, Mairéad

    2016-01-01

    Purpose We describe the development of a new Test of Child Speechreading (ToCS) specifically designed for use with deaf and hearing children. Speechreading is a skill which is required for deaf children to access the language of the hearing community. ToCS is a deaf-friendly, computer-based test that measures child speechreading (silent lipreading) at three psycholinguistic levels: words, sentences and short stories. The aims of the study were to standardize ToCS with deaf and hearing children and investigate the effects of hearing status, age and linguistic complexity on speechreading ability. Method 86 severely and profoundly deaf and 91 hearing children aged between 5 and 14 years participated. The deaf children were from a range of language and communication backgrounds and their preferred mode of communication varied. Results: Speechreading skills significantly improved with age for both deaf and hearing children. There was no effect of hearing status on speechreading ability and deaf and hearing showed similar performance across all subtests on ToCS. Conclusions The Test of Child Speechreading (ToCS) is a valid and reliable assessment of speechreading ability in school-aged children that can be used to measure individual differences in performance in speechreading ability. PMID:23275416

  16. The deaf utilize phonological representations in visually presented verbal memory tasks.

    PubMed

    Okada, Rieko; Nakagawa, Jun; Takahashi, Muneyoshi; Kanaka, Noriko; Fukamauchi, Fumihiko; Watanabe, Katsumi; Namatame, Miki; Matsuda, Tetsuya

    2015-01-01

    The phonological abilities of congenitally deaf individuals are inferior to those of people who can hear. However, deaf individuals can acquire spoken languages by utilizing orthography and lip-reading. The present study used functional magnetic resonance imaging (fMRI) to show that deaf individuals utilize phonological representations via a mnemonic process. We compared the brain activation of deaf and hearing participants while they memorized serially visually presented Japanese kana letters (Kana), finger alphabets (Finger), and Arabic letters (Arabic). Hearing participants did not know which finger alphabets corresponded to which language sounds, whereas deaf participants did. All of the participants understood the correspondence between Kana and their language sounds. None of the participants knew the correspondence between Arabic and their language sounds, so this condition was used as a baseline. We found that the left superior temporal gyrus (STG) was activated by phonological representations in the deaf group when memorizing both Kana and Finger. Additionally, the brain areas associated with phonological representations for Finger in the deaf group were the same as the areas for Kana in the hearing group. Overall, despite the fact that they are superior in visual information processing, deaf individuals utilize phonological rather than visual representations in visually presented verbal memory.

  17. Auditory midbrain implant: research and development towards a second clinical trial.

    PubMed

    Lim, Hubert H; Lenarz, Thomas

    2015-04-01

    The cochlear implant is considered one of the most successful neural prostheses to date, which was made possible by visionaries who continued to develop the cochlear implant through multiple technological and clinical challenges. However, patients without a functional auditory nerve or implantable cochlea cannot benefit from a cochlear implant. The focus of the paper is to review the development and translation of a new type of central auditory prosthesis for this group of patients that is known as the auditory midbrain implant (AMI) and is designed for electrical stimulation within the inferior colliculus. The rationale and results for the first AMI clinical study using a multi-site single-shank array will be presented initially. Although the AMI has achieved encouraging results in terms of safety and improvements in lip-reading capabilities and environmental awareness, it has not yet provided sufficient speech perception. Animal and human data will then be presented to show that a two-shank AMI array can potentially improve hearing performance by targeting specific neurons of the inferior colliculus. A new two-shank array, stimulation strategy, and surgical approach are planned for the AMI that are expected to improve hearing performance in the patients who will be implanted in an upcoming clinical trial funded by the National Institutes of Health. Positive outcomes from this clinical trial will motivate new efforts and developments toward improving central auditory prostheses for those who cannot sufficiently benefit from cochlear implants. This article is part of a Special Issue entitled .

  18. From Mimicry to Language: A Neuroanatomically Based Evolutionary Model of the Emergence of Vocal Language.

    PubMed

    Poliva, Oren

    2016-01-01

    The auditory cortex communicates with the frontal lobe via the middle temporal gyrus (auditory ventral stream; AVS) or the inferior parietal lobule (auditory dorsal stream; ADS). Whereas the AVS is ascribed only with sound recognition, the ADS is ascribed with sound localization, voice detection, prosodic perception/production, lip-speech integration, phoneme discrimination, articulation, repetition, phonological long-term memory and working memory. Previously, I interpreted the juxtaposition of sound localization, voice detection, audio-visual integration and prosodic analysis, as evidence that the behavioral precursor to human speech is the exchange of contact calls in non-human primates. Herein, I interpret the remaining ADS functions as evidence of additional stages in language evolution. According to this model, the role of the ADS in vocal control enabled early Homo (Hominans) to name objects using monosyllabic calls, and allowed children to learn their parents' calls by imitating their lip movements. Initially, the calls were forgotten quickly but gradually were remembered for longer periods. Once the representations of the calls became permanent, mimicry was limited to infancy, and older individuals encoded in the ADS a lexicon for the names of objects (phonological lexicon). Consequently, sound recognition in the AVS was sufficient for activating the phonological representations in the ADS and mimicry became independent of lip-reading. Later, by developing inhibitory connections between acoustic-syllabic representations in the AVS and phonological representations of subsequent syllables in the ADS, Hominans became capable of concatenating the monosyllabic calls for repeating polysyllabic words (i.e., developed working memory). Finally, due to strengthening of connections between phonological representations in the ADS, Hominans became capable of encoding several syllables as a single representation (chunking). Consequently, Hominans began vocalizing and

  19. Brain networks engaged in audiovisual integration during speech perception revealed by persistent homology-based network filtration.

    PubMed

    Kim, Heejung; Hahm, Jarang; Lee, Hyekyoung; Kang, Eunjoo; Kang, Hyejin; Lee, Dong Soo

    2015-05-01

    The human brain naturally integrates audiovisual information to improve speech perception. However, in noisy environments, understanding speech is difficult and may require much effort. Although the brain network is supposed to be engaged in speech perception, it is unclear how speech-related brain regions are connected during natural bimodal audiovisual or unimodal speech perception with counterpart irrelevant noise. To investigate the topological changes of speech-related brain networks at all possible thresholds, we used a persistent homological framework through hierarchical clustering, such as single linkage distance, to analyze the connected component of the functional network during speech perception using functional magnetic resonance imaging. For speech perception, bimodal (audio-visual speech cue) or unimodal speech cues with counterpart irrelevant noise (auditory white-noise or visual gum-chewing) were delivered to 15 subjects. In terms of positive relationship, similar connected components were observed in bimodal and unimodal speech conditions during filtration. However, during speech perception by congruent audiovisual stimuli, the tighter couplings of left anterior temporal gyrus-anterior insula component and right premotor-visual components were observed than auditory or visual speech cue conditions, respectively. Interestingly, visual speech is perceived under white noise by tight negative coupling in the left inferior frontal region-right anterior cingulate, left anterior insula, and bilateral visual regions, including right middle temporal gyrus, right fusiform components. In conclusion, the speech brain network is tightly positively or negatively connected, and can reflect efficient or effortful processes during natural audiovisual integration or lip-reading, respectively, in speech perception.

  20. From Mimicry to Language: A Neuroanatomically Based Evolutionary Model of the Emergence of Vocal Language

    PubMed Central

    Poliva, Oren

    2016-01-01

    The auditory cortex communicates with the frontal lobe via the middle temporal gyrus (auditory ventral stream; AVS) or the inferior parietal lobule (auditory dorsal stream; ADS). Whereas the AVS is ascribed only with sound recognition, the ADS is ascribed with sound localization, voice detection, prosodic perception/production, lip-speech integration, phoneme discrimination, articulation, repetition, phonological long-term memory and working memory. Previously, I interpreted the juxtaposition of sound localization, voice detection, audio-visual integration and prosodic analysis, as evidence that the behavioral precursor to human speech is the exchange of contact calls in non-human primates. Herein, I interpret the remaining ADS functions as evidence of additional stages in language evolution. According to this model, the role of the ADS in vocal control enabled early Homo (Hominans) to name objects using monosyllabic calls, and allowed children to learn their parents' calls by imitating their lip movements. Initially, the calls were forgotten quickly but gradually were remembered for longer periods. Once the representations of the calls became permanent, mimicry was limited to infancy, and older individuals encoded in the ADS a lexicon for the names of objects (phonological lexicon). Consequently, sound recognition in the AVS was sufficient for activating the phonological representations in the ADS and mimicry became independent of lip-reading. Later, by developing inhibitory connections between acoustic-syllabic representations in the AVS and phonological representations of subsequent syllables in the ADS, Hominans became capable of concatenating the monosyllabic calls for repeating polysyllabic words (i.e., developed working memory). Finally, due to strengthening of connections between phonological representations in the ADS, Hominans became capable of encoding several syllables as a single representation (chunking). Consequently, Hominans began vocalizing and

  1. From Mimicry to Language: A Neuroanatomically Based Evolutionary Model of the Emergence of Vocal Language.

    PubMed

    Poliva, Oren

    2016-01-01

    The auditory cortex communicates with the frontal lobe via the middle temporal gyrus (auditory ventral stream; AVS) or the inferior parietal lobule (auditory dorsal stream; ADS). Whereas the AVS is ascribed only with sound recognition, the ADS is ascribed with sound localization, voice detection, prosodic perception/production, lip-speech integration, phoneme discrimination, articulation, repetition, phonological long-term memory and working memory. Previously, I interpreted the juxtaposition of sound localization, voice detection, audio-visual integration and prosodic analysis, as evidence that the behavioral precursor to human speech is the exchange of contact calls in non-human primates. Herein, I interpret the remaining ADS functions as evidence of additional stages in language evolution. According to this model, the role of the ADS in vocal control enabled early Homo (Hominans) to name objects using monosyllabic calls, and allowed children to learn their parents' calls by imitating their lip movements. Initially, the calls were forgotten quickly but gradually were remembered for longer periods. Once the representations of the calls became permanent, mimicry was limited to infancy, and older individuals encoded in the ADS a lexicon for the names of objects (phonological lexicon). Consequently, sound recognition in the AVS was sufficient for activating the phonological representations in the ADS and mimicry became independent of lip-reading. Later, by developing inhibitory connections between acoustic-syllabic representations in the AVS and phonological representations of subsequent syllables in the ADS, Hominans became capable of concatenating the monosyllabic calls for repeating polysyllabic words (i.e., developed working memory). Finally, due to strengthening of connections between phonological representations in the ADS, Hominans became capable of encoding several syllables as a single representation (chunking). Consequently, Hominans began vocalizing and

  2. How can audiovisual pathways enhance the temporal resolution of time-compressed speech in blind subjects?

    PubMed

    Hertrich, Ingo; Dietrich, Susanne; Ackermann, Hermann

    2013-01-01

    In blind people, the visual channel cannot assist face-to-face communication via lipreading or visual prosody. Nevertheless, the visual system may enhance the evaluation of auditory information due to its cross-links to (1) the auditory system, (2) supramodal representations, and (3) frontal action-related areas. Apart from feedback or top-down support of, for example, the processing of spatial or phonological representations, experimental data have shown that the visual system can impact auditory perception at more basic computational stages such as temporal signal resolution. For example, blind as compared to sighted subjects are more resistant against backward masking, and this ability appears to be associated with activity in visual cortex. Regarding the comprehension of continuous speech, blind subjects can learn to use accelerated text-to-speech systems for "reading" texts at ultra-fast speaking rates (>16 syllables/s), exceeding by far the normal range of 6 syllables/s. A functional magnetic resonance imaging study has shown that this ability, among other brain regions, significantly covaries with BOLD responses in bilateral pulvinar, right visual cortex, and left supplementary motor area. Furthermore, magnetoencephalographic measurements revealed a particular component in right occipital cortex phase-locked to the syllable onsets of accelerated speech. In sighted people, the "bottleneck" for understanding time-compressed speech seems related to higher demands for buffering phonological material and is, presumably, linked to frontal brain structures. On the other hand, the neurophysiological correlates of functions overcoming this bottleneck, seem to depend upon early visual cortex activity. The present Hypothesis and Theory paper outlines a model that aims at binding these data together, based on early cross-modal pathways that are already known from various audiovisual experiments on cross-modal adjustments during space, time, and object recognition. PMID

  3. Auditory midbrain implant: research and development towards a second clinical trial.

    PubMed

    Lim, Hubert H; Lenarz, Thomas

    2015-04-01

    The cochlear implant is considered one of the most successful neural prostheses to date, which was made possible by visionaries who continued to develop the cochlear implant through multiple technological and clinical challenges. However, patients without a functional auditory nerve or implantable cochlea cannot benefit from a cochlear implant. The focus of the paper is to review the development and translation of a new type of central auditory prosthesis for this group of patients that is known as the auditory midbrain implant (AMI) and is designed for electrical stimulation within the inferior colliculus. The rationale and results for the first AMI clinical study using a multi-site single-shank array will be presented initially. Although the AMI has achieved encouraging results in terms of safety and improvements in lip-reading capabilities and environmental awareness, it has not yet provided sufficient speech perception. Animal and human data will then be presented to show that a two-shank AMI array can potentially improve hearing performance by targeting specific neurons of the inferior colliculus. A new two-shank array, stimulation strategy, and surgical approach are planned for the AMI that are expected to improve hearing performance in the patients who will be implanted in an upcoming clinical trial funded by the National Institutes of Health. Positive outcomes from this clinical trial will motivate new efforts and developments toward improving central auditory prostheses for those who cannot sufficiently benefit from cochlear implants. This article is part of a Special Issue entitled . PMID:25613994

  4. Visual speech discrimination and identification of natural and synthetic consonant stimuli

    PubMed Central

    Files, Benjamin T.; Tjan, Bosco S.; Jiang, Jintao; Bernstein, Lynne E.

    2015-01-01

    From phonetic features to connected discourse, every level of psycholinguistic structure including prosody can be perceived through viewing the talking face. Yet a longstanding notion in the literature is that visual speech perceptual categories comprise groups of phonemes (referred to as visemes), such as /p, b, m/ and /f, v/, whose internal structure is not informative to the visual speech perceiver. This conclusion has not to our knowledge been evaluated using a psychophysical discrimination paradigm. We hypothesized that perceivers can discriminate the phonemes within typical viseme groups, and that discrimination measured with d-prime (d’) and response latency is related to visual stimulus dissimilarities between consonant segments. In Experiment 1, participants performed speeded discrimination for pairs of consonant-vowel spoken nonsense syllables that were predicted to be same, near, or far in their perceptual distances, and that were presented as natural or synthesized video. Near pairs were within-viseme consonants. Natural within-viseme stimulus pairs were discriminated significantly above chance (except for /k/-/h/). Sensitivity (d’) increased and response times decreased with distance. Discrimination and identification were superior with natural stimuli, which comprised more phonetic information. We suggest that the notion of the viseme as a unitary perceptual category is incorrect. Experiment 2 probed the perceptual basis for visual speech discrimination by inverting the stimuli. Overall reductions in d’ with inverted stimuli but a persistent pattern of larger d’ for far than for near stimulus pairs are interpreted as evidence that visual speech is represented by both its motion and configural attributes. The methods and results of this investigation open up avenues for understanding the neural and perceptual bases for visual and audiovisual speech perception and for development of practical applications such as visual lipreading

  5. Using space and time to encode vibrotactile information: toward an estimate of the skin's achievable throughput.

    PubMed

    Novich, Scott D; Eagleman, David M

    2015-10-01

    Touch receptors in the skin can relay various forms of abstract information, such as words (Braille), haptic feedback (cell phones, game controllers, feedback for prosthetic control), and basic visual information such as edges and shape (sensory substitution devices). The skin can support such applications with ease: They are all low bandwidth and do not require a fine temporal acuity. But what of high-throughput applications? We use sound-to-touch conversion as a motivating example, though others abound (e.g., vision, stock market data). In the past, vibrotactile hearing aids have demonstrated improvement in speech perceptions in the deaf. However, a sound-to-touch sensory substitution device that works with high efficacy and without the aid of lipreading has yet to be developed. Is this because skin simply does not have the capacity to effectively relay high-throughput streams such as sound? Or is this because the spatial and temporal properties of skin have not been leveraged to full advantage? Here, we begin to address these questions with two experiments. First, we seek to determine the best method of relaying information through the skin using an identification task on the lower back. We find that vibrotactile patterns encoding information in both space and time yield the best overall information transfer estimate. Patterns encoded in space and time or "intensity" (the coupled coding of vibration frequency and force) both far exceed performance of only spatially encoded patterns. Next, we determine the vibrotactile two-tacton resolution on the lower back-the distance necessary for resolving two vibrotactile patterns. We find that our vibratory motors conservatively require at least 6 cm of separation to resolve two independent tactile patterns (>80 % correct), regardless of stimulus type (e.g., spatiotemporal "sweeps" versus single vibratory pulses). Six centimeter is a greater distance than the inter-motor distances used in Experiment 1 (2.5 cm), which

  6. Auditory brainstem implants: current neurosurgical experiences and perspective.

    PubMed

    Matthies, C; Thomas, S; Moshrefi, M; Lesinski-Schiedat, A; Frohne, C; Battmer, R D; Lenarz, T; Samii, M

    2000-01-01

    The objective of this study was to present aspects of the current treatment protocol, such as patient evaluation and selection for therapy, multimodality monitoring for optimal auditory brainstem implant (ABI) positioning and radiological evaluation, that might have an impact on the functional results of ABI. Out of a series of 145 patients with bilateral vestibular schwannomas 10 patients received an ABI, eight of which are reported here. Patient selection was based on disease course, clinical and radiological criteria (according to the Hannover evaluation and prognosis scaling of neurofibromatosis type 2 (NF2)), extensive otological test battery and psycho-social factors. ABI placement was controlled by multimodality electrophysiological monitoring in order to activate the auditory pathway and to prevent false stimulation of the cranial nerve nuclei or long sensory or motor tracts. Results of hearing function were correlated with patients' ages, duration of deafness, tumour extension, tumour-induced compression or deformation of the brainstem, and numbers of activated electrodes without any side-effects. Out of 59 patients with pre-operative deafness eight patients received an ABI of the Nucleus 22 type. All these patients became continuous users without any side effects and experienced improved quality of life. Speech reception in combination with lip-reading was markedly improved, with further improvement over a long period. A short duration of deafness may be favourable for achieving good results, while age was not a relevant factor. Lateral recess obstruction may necessitate a more meticulous dissection, but did not prevent good placement of the ABI in the lateral recess. Pre-existing brainstem compression did not prevent good results, but brainstem deformation and ipsi- and contralateral distortion were followed by a less favourable outcome. Among the factors that can be influenced by the therapy management are the selection of patients with a slow

  7. Home-based Early Intervention on Auditory and Speech Development in Mandarin-speaking Deaf Infants and Toddlers with Chronological Aged 7–24 Months

    PubMed Central

    Yang, Ying; Liu, Yue-Hui; Fu, Ming-Fu; Li, Chun-Lin; Wang, Li-Yan; Wang, Qi; Sun, Xi-Bin

    2015-01-01

    Background: Early auditory and speech development in home-based early intervention of infants and toddlers with hearing loss younger than 2 years are still spare in China. This study aimed to observe the development of auditory and speech in deaf infants and toddlers who were fitted with hearing aids and/or received cochlear implantation between the chronological ages of 7–24 months, and analyze the effect of chronological age and recovery time on auditory and speech development in the course of home-based early intervention. Methods: This longitudinal study included 55 hearing impaired children with severe and profound binaural deafness, who were divided into Group A (7–12 months), Group B (13–18 months) and Group C (19–24 months) based on the chronological age. Categories auditory performance (CAP) and speech intelligibility rating scale (SIR) were used to evaluate auditory and speech development at baseline and 3, 6, 9, 12, 18, and 24 months of habilitation. Descriptive statistics were used to describe demographic features and were analyzed by repeated measures analysis of variance. Results: With 24 months of hearing intervention, 78% of the patients were able to understand common phrases and conversation without lip-reading, 96% of the patients were intelligible to a listener. In three groups, children showed the rapid growth of trend features in each period of habilitation. CAP and SIR scores have developed rapidly within 24 months after fitted auxiliary device in Group A, which performed much better auditory and speech abilities than Group B (P < 0.05) and Group C (P < 0.05). Group B achieved better results than Group C, whereas no significant differences were observed between Group B and Group C (P > 0.05). Conclusions: The data suggested the early hearing intervention and home-based habilitation benefit auditory and speech development. Chronological age and recovery time may be major factors for aural verbal outcomes in hearing impaired children

  8. One-dimensional scales for pain evaluation adopted in Italian nursing practice: giving preference to deaf patients.

    PubMed

    Palese, Alvisa; Salvador, Linda; Cozzi, Donatella

    2011-01-01

    Despite the increasing attention given to pain, little is known about how deaf patients communicate their pain and which pain scales they prefer to use. Studies of the validity of various scales often specify conditions that exclude them. With the aim to explore the preferred pain evaluation scale and the method of administration when evaluating deaf patients, a descriptive phenomenology of qualitative research study was undertaken and articulated in two phases. In the first phase, a purposeful sample of 10 nurses with experience in the care of deaf clients was studied using focus groups to collect data regarding which pain scale they used and the methods they used to administer the scales in clinical settings during care to deaf patients. In the second phase, a purposeful sample of 16 deaf people was engaged in multiple focus groups to analyze a set of one-dimension scales that emerged from the first phase of the study with nurses and to discuss their preferences for pain scales. Nurses who participated in the focus group reported using the numerical rating scale, visual analogue scale, Faces Pain Scale, and the Iowa Pain Thermometer (IPT) scale when caring for deaf people. Deaf patients involved in the second phase of this study preferred the IPT scale. Participants also noted the interference of environmental factors such as dimly lit rooms or glaring lights in situations that required lipreading for communication of pain such as in operating rooms. It was concluded that decisions regarding how to administer pain scales to deaf persons need to consider the preferences and the values of the patients. To avoid the risk of misunderstanding the pain of deaf patients, practice guidelines and strategies related to measuring pain in deaf persons should be specified by deaf associations at international, national, and local levels. Utilization of a simple sign language even at an international level could guarantee security in the communication of the pain between