Analyzing Multimodal Interaction within a Classroom Setting

ERIC Educational Resources Information Center

Moura, Heloisa

2006-01-01

Human interactions are multimodal in nature. From simple to complex forms of transferal of information, human beings draw on a multiplicity of communicative modes, such as intonation and gaze, to make sense of everyday experiences. Likewise, the learning process, either within traditional classrooms or Virtual Learning Environments, is shaped by…

Challenges in Transcribing Multimodal Data: A Case Study

ERIC Educational Resources Information Center

Helm, Francesca; Dooly, Melinda

2017-01-01

Computer-mediated communication (CMC) once meant principally text-based communication mediated by computers, but rapid technological advances in recent years have heralded an era of multimodal communication with a growing emphasis on audio and video synchronous interaction. As CMC, in all its variants (text chats, video chats, forums, blogs, SMS,…

Development of a Dynamically Configurable, Object-Oriented Framework for Distributed, Multi-modal Computational Aerospace Systems Simulation

NASA Technical Reports Server (NTRS)

Afjeh, Abdollah A.; Reed, John A.

2003-01-01

The following reports are presented on this project:A first year progress report on: Development of a Dynamically Configurable,Object-Oriented Framework for Distributed, Multi-modal Computational Aerospace Systems Simulation; A second year progress report on: Development of a Dynamically Configurable, Object-Oriented Framework for Distributed, Multi-modal Computational Aerospace Systems Simulation; An Extensible, Interchangeable and Sharable Database Model for Improving Multidisciplinary Aircraft Design; Interactive, Secure Web-enabled Aircraft Engine Simulation Using XML Databinding Integration; and Improving the Aircraft Design Process Using Web-based Modeling and Simulation.

Meaning-Making in Online Language Learner Interactions via Desktop Videoconferencing

ERIC Educational Resources Information Center

Satar, H. Müge

2016-01-01

Online language learning and teaching in multimodal contexts has been identified as one of the key research areas in computer-aided learning (CALL) (Lamy, 2013; White, 2014). This paper aims to explore meaning-making in online language learner interactions via desktop videoconferencing (DVC) and in doing so illustrate multimodal transcription and…

Multimodal Research: Addressing the Complexity of Multimodal Environments and the Challenges for CALL

ERIC Educational Resources Information Center

Tan, Sabine; O'Halloran, Kay L.; Wignell, Peter

2016-01-01

Multimodality, the study of the interaction of language with other semiotic resources such as images and sound resources, has significant implications for computer assisted language learning (CALL) with regards to understanding the impact of digital environments on language teaching and learning. In this paper, we explore recent manifestations of…

A multimodal parallel architecture: A cognitive framework for multimodal interactions.

PubMed

Cohn, Neil

2016-01-01

Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech-gesture and text-image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated "grammatical" sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoff's (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it "dominates" the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the "linguistic" system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality. Copyright © 2015.

Multimodal Neuroelectric Interface Development

NASA Technical Reports Server (NTRS)

Trejo, Leonard J.; Wheeler, Kevin R.; Jorgensen, Charles C.; Totah, Joseph (Technical Monitor)

2001-01-01

This project aims to improve performance of NASA missions by developing multimodal neuroelectric technologies for augmented human-system interaction. Neuroelectric technologies will add completely new modes of interaction that operate in parallel with keyboards, speech, or other manual controls, thereby increasing the bandwidth of human-system interaction. We recently demonstrated the feasibility of real-time electromyographic (EMG) pattern recognition for a direct neuroelectric human-computer interface. We recorded EMG signals from an elastic sleeve with dry electrodes, while a human subject performed a range of discrete gestures. A machine-teaming algorithm was trained to recognize the EMG patterns associated with the gestures and map them to control signals. Successful applications now include piloting two Class 4 aircraft simulations (F-15 and 757) and entering data with a "virtual" numeric keyboard. Current research focuses on on-line adaptation of EMG sensing and processing and recognition of continuous gestures. We are also extending this on-line pattern recognition methodology to electroencephalographic (EEG) signals. This will allow us to bypass muscle activity and draw control signals directly from the human brain. Our system can reliably detect P-rhythm (a periodic EEG signal from motor cortex in the 10 Hz range) with a lightweight headset containing saline-soaked sponge electrodes. The data show that EEG p-rhythm can be modulated by real and imaginary motions. Current research focuses on using biofeedback to train of human subjects to modulate EEG rhythms on demand, and to examine interactions of EEG-based control with EMG-based and manual control. Viewgraphs on these neuroelectric technologies are also included.

Multimodal and ubiquitous computing systems: supporting independent-living older users.

PubMed

Perry, Mark; Dowdall, Alan; Lines, Lorna; Hone, Kate

2004-09-01

We document the rationale and design of a multimodal interface to a pervasive/ubiquitous computing system that supports independent living by older people in their own homes. The Millennium Home system involves fitting a resident's home with sensors--these sensors can be used to trigger sequences of interaction with the resident to warn them about dangerous events, or to check if they need external help. We draw lessons from the design process and conclude the paper with implications for the design of multimodal interfaces to ubiquitous systems developed for the elderly and in healthcare, as well as for more general ubiquitous computing applications.

Learning multimodal dictionaries.

PubMed

Monaci, Gianluca; Jost, Philippe; Vandergheynst, Pierre; Mailhé, Boris; Lesage, Sylvain; Gribonval, Rémi

2007-09-01

Real-world phenomena involve complex interactions between multiple signal modalities. As a consequence, humans are used to integrate at each instant perceptions from all their senses in order to enrich their understanding of the surrounding world. This paradigm can be also extremely useful in many signal processing and computer vision problems involving mutually related signals. The simultaneous processing of multimodal data can, in fact, reveal information that is otherwise hidden when considering the signals independently. However, in natural multimodal signals, the statistical dependencies between modalities are in general not obvious. Learning fundamental multimodal patterns could offer deep insight into the structure of such signals. In this paper, we present a novel model of multimodal signals based on their sparse decomposition over a dictionary of multimodal structures. An algorithm for iteratively learning multimodal generating functions that can be shifted at all positions in the signal is proposed, as well. The learning is defined in such a way that it can be accomplished by iteratively solving a generalized eigenvector problem, which makes the algorithm fast, flexible, and free of user-defined parameters. The proposed algorithm is applied to audiovisual sequences and it is able to discover underlying structures in the data. The detection of such audio-video patterns in audiovisual clips allows to effectively localize the sound source on the video in presence of substantial acoustic and visual distractors, outperforming state-of-the-art audiovisual localization algorithms.

Potential of Cognitive Computing and Cognitive Systems

NASA Astrophysics Data System (ADS)

Noor, Ahmed K.

2015-01-01

Cognitive computing and cognitive technologies are game changers for future engineering systems, as well as for engineering practice and training. They are major drivers for knowledge automation work, and the creation of cognitive products with higher levels of intelligence than current smart products. This paper gives a brief review of cognitive computing and some of the cognitive engineering systems activities. The potential of cognitive technologies is outlined, along with a brief description of future cognitive environments, incorporating cognitive assistants - specialized proactive intelligent software agents designed to follow and interact with humans and other cognitive assistants across the environments. The cognitive assistants engage, individually or collectively, with humans through a combination of adaptive multimodal interfaces, and advanced visualization and navigation techniques. The realization of future cognitive environments requires the development of a cognitive innovation ecosystem for the engineering workforce. The continuously expanding major components of the ecosystem include integrated knowledge discovery and exploitation facilities (incorporating predictive and prescriptive big data analytics); novel cognitive modeling and visual simulation facilities; cognitive multimodal interfaces; and cognitive mobile and wearable devices. The ecosystem will provide timely, engaging, personalized / collaborative, learning and effective decision making. It will stimulate creativity and innovation, and prepare the participants to work in future cognitive enterprises and develop new cognitive products of increasing complexity. http://www.aee.odu.edu/cognitivecomp

Unraveling Students' Interaction around a Tangible Interface Using Multimodal Learning Analytics

ERIC Educational Resources Information Center

Schneider, Bertrand; Blikstein, Paulo

2015-01-01

In this paper, we describe multimodal learning analytics (MMLA) techniques to analyze data collected around an interactive learning environment. In a previous study (Schneider & Blikstein, submitted), we designed and evaluated a Tangible User Interface (TUI) where dyads of students were asked to learn about the human hearing system by…

Audio-visual affective expression recognition

NASA Astrophysics Data System (ADS)

Huang, Thomas S.; Zeng, Zhihong

2007-11-01

Automatic affective expression recognition has attracted more and more attention of researchers from different disciplines, which will significantly contribute to a new paradigm for human computer interaction (affect-sensitive interfaces, socially intelligent environments) and advance the research in the affect-related fields including psychology, psychiatry, and education. Multimodal information integration is a process that enables human to assess affective states robustly and flexibly. In order to understand the richness and subtleness of human emotion behavior, the computer should be able to integrate information from multiple sensors. We introduce in this paper our efforts toward machine understanding of audio-visual affective behavior, based on both deliberate and spontaneous displays. Some promising methods are presented to integrate information from both audio and visual modalities. Our experiments show the advantage of audio-visual fusion in affective expression recognition over audio-only or visual-only approaches.

Brain-computer interaction research at the Computer Vision and Multimedia Laboratory, University of Geneva.

PubMed

Pun, Thierry; Alecu, Teodor Iulian; Chanel, Guillaume; Kronegg, Julien; Voloshynovskiy, Sviatoslav

2006-06-01

This paper describes the work being conducted in the domain of brain-computer interaction (BCI) at the Multimodal Interaction Group, Computer Vision and Multimedia Laboratory, University of Geneva, Geneva, Switzerland. The application focus of this work is on multimodal interaction rather than on rehabilitation, that is how to augment classical interaction by means of physiological measurements. Three main research topics are addressed. The first one concerns the more general problem of brain source activity recognition from EEGs. In contrast with classical deterministic approaches, we studied iterative robust stochastic based reconstruction procedures modeling source and noise statistics, to overcome known limitations of current techniques. We also developed procedures for optimal electroencephalogram (EEG) sensor system design in terms of placement and number of electrodes. The second topic is the study of BCI protocols and performance from an information-theoretic point of view. Various information rate measurements have been compared for assessing BCI abilities. The third research topic concerns the use of EEG and other physiological signals for assessing a user's emotional status.

Multimodal interactions in typically and atypically developing children: natural versus artificial environments.

PubMed

Giannopulu, Irini

2013-11-01

This review addresses the central role played by multimodal interactions in neurocognitive development. We first analyzed our studies of multimodal verbal and nonverbal cognition and emotional interactions within neuronal, that is, natural environments in typically developing children. We then tried to relate them to the topic of creating artificial environments using mobile toy robots to neurorehabilitate severely autistic children. By doing so, both neural/natural and artificial environments are considered as the basis of neuronal organization and reorganization. The common thread underlying the thinking behind this approach revolves around the brain's intrinsic properties: neuroplasticity and the fact that the brain is neurodynamic. In our approach, neural organization and reorganization using natural or artificial environments aspires to bring computational perspectives into cognitive developmental neuroscience.

Adding Pluggable and Personalized Natural Control Capabilities to Existing Applications

PubMed Central

Lamberti, Fabrizio; Sanna, Andrea; Carlevaris, Gilles; Demartini, Claudio

2015-01-01

Advancements in input device and sensor technologies led to the evolution of the traditional human-machine interaction paradigm based on the mouse and keyboard. Touch-, gesture- and voice-based interfaces are integrated today in a variety of applications running on consumer devices (e.g., gaming consoles and smartphones). However, to allow existing applications running on desktop computers to utilize natural interaction, significant re-design and re-coding efforts may be required. In this paper, a framework designed to transparently add multi-modal interaction capabilities to applications to which users are accustomed is presented. Experimental observations confirmed the effectiveness of the proposed framework and led to a classification of those applications that could benefit more from the availability of natural interaction modalities. PMID:25635410

Adding pluggable and personalized natural control capabilities to existing applications.

PubMed

Lamberti, Fabrizio; Sanna, Andrea; Carlevaris, Gilles; Demartini, Claudio

2015-01-28

Advancements in input device and sensor technologies led to the evolution of the traditional human-machine interaction paradigm based on the mouse and keyboard. Touch-, gesture- and voice-based interfaces are integrated today in a variety of applications running on consumer devices (e.g., gaming consoles and smartphones). However, to allow existing applications running on desktop computers to utilize natural interaction, significant re-design and re-coding efforts may be required. In this paper, a framework designed to transparently add multi-modal interaction capabilities to applications to which users are accustomed is presented. Experimental observations confirmed the effectiveness of the proposed framework and led to a classification of those applications that could benefit more from the availability of natural interaction modalities.

See You See Me: the Role of Eye Contact in Multimodal Human-Robot Interaction.

PubMed

Xu, Tian Linger; Zhang, Hui; Yu, Chen

2016-05-01

We focus on a fundamental looking behavior in human-robot interactions - gazing at each other's face. Eye contact and mutual gaze between two social partners are critical in smooth human-human interactions. Therefore, investigating at what moments and in what ways a robot should look at a human user's face as a response to the human's gaze behavior is an important topic. Toward this goal, we developed a gaze-contingent human-robot interaction system, which relied on momentary gaze behaviors from a human user to control an interacting robot in real time. Using this system, we conducted an experiment in which human participants interacted with the robot in a joint attention task. In the experiment, we systematically manipulated the robot's gaze toward the human partner's face in real time and then analyzed the human's gaze behavior as a response to the robot's gaze behavior. We found that more face looks from the robot led to more look-backs (to the robot's face) from human participants and consequently created more mutual gaze and eye contact between the two. Moreover, participants demonstrated more coordinated and synchronized multimodal behaviors between speech and gaze when more eye contact was successfully established and maintained.

When a robot is social: spatial arrangements and multimodal semiotic engagement in the practice of social robotics.

PubMed

Alac, Morana; Movellan, Javier; Tanaka, Fumihide

2011-12-01

Social roboticists design their robots to function as social agents in interaction with humans and other robots. Although we do not deny that the robot's design features are crucial for attaining this aim, we point to the relevance of spatial organization and coordination between the robot and the humans who interact with it. We recover these interactions through an observational study of a social robotics laboratory and examine them by applying a multimodal interactional analysis to two moments of robotics practice. We describe the vital role of roboticists and of the group of preverbal infants, who are involved in a robot's design activity, and we argue that the robot's social character is intrinsically related to the subtleties of human interactional moves in laboratories of social robotics. This human involvement in the robot's social agency is not simply controlled by individual will. Instead, the human-machine couplings are demanded by the situational dynamics in which the robot is lodged.

Generating Multimodal References

ERIC Educational Resources Information Center

van der Sluis, Ielka; Krahmer, Emiel

2007-01-01

This article presents a new computational model for the generation of multimodal referring expressions (REs), based on observations in human communication. The algorithm is an extension of the graph-based algorithm proposed by Krahmer, van Erk, and Verleg (2003) and makes use of a so-called Flashlight Model for pointing. The Flashlight Model…

Construction of a multimodal CT-video chest model

NASA Astrophysics Data System (ADS)

Byrnes, Patrick D.; Higgins, William E.

2014-03-01

Bronchoscopy enables a number of minimally invasive chest procedures for diseases such as lung cancer and asthma. For example, using the bronchoscope's continuous video stream as a guide, a physician can navigate through the lung airways to examine general airway health, collect tissue samples, or administer a disease treatment. In addition, physicians can now use new image-guided intervention (IGI) systems, which draw upon both three-dimensional (3D) multi-detector computed tomography (MDCT) chest scans and bronchoscopic video, to assist with bronchoscope navigation. Unfortunately, little use is made of the acquired video stream, a potentially invaluable source of information. In addition, little effort has been made to link the bronchoscopic video stream to the detailed anatomical information given by a patient's 3D MDCT chest scan. We propose a method for constructing a multimodal CT-video model of the chest. After automatically computing a patient's 3D MDCT-based airway-tree model, the method next parses the available video data to generate a positional linkage between a sparse set of key video frames and airway path locations. Next, a fusion/mapping of the video's color mucosal information and MDCT-based endoluminal surfaces is performed. This results in the final multimodal CT-video chest model. The data structure constituting the model provides a history of those airway locations visited during bronchoscopy. It also provides for quick visual access to relevant sections of the airway wall by condensing large portions of endoscopic video into representative frames containing important structural and textural information. When examined with a set of interactive visualization tools, the resulting fused data structure provides a rich multimodal data source. We demonstrate the potential of the multimodal model with both phantom and human data.

User Localization During Human-Robot Interaction

PubMed Central

Alonso-Martín, F.; Gorostiza, Javi F.; Malfaz, María; Salichs, Miguel A.

2012-01-01

This paper presents a user localization system based on the fusion of visual information and sound source localization, implemented on a social robot called Maggie. One of the main requisites to obtain a natural interaction between human-human and human-robot is an adequate spatial situation between the interlocutors, that is, to be orientated and situated at the right distance during the conversation in order to have a satisfactory communicative process. Our social robot uses a complete multimodal dialog system which manages the user-robot interaction during the communicative process. One of its main components is the presented user localization system. To determine the most suitable allocation of the robot in relation to the user, a proxemic study of the human-robot interaction is required, which is described in this paper. The study has been made with two groups of users: children, aged between 8 and 17, and adults. Finally, at the end of the paper, experimental results with the proposed multimodal dialog system are presented. PMID:23012577

User localization during human-robot interaction.

PubMed

Alonso-Martín, F; Gorostiza, Javi F; Malfaz, María; Salichs, Miguel A

2012-01-01

This paper presents a user localization system based on the fusion of visual information and sound source localization, implemented on a social robot called Maggie. One of the main requisites to obtain a natural interaction between human-human and human-robot is an adequate spatial situation between the interlocutors, that is, to be orientated and situated at the right distance during the conversation in order to have a satisfactory communicative process. Our social robot uses a complete multimodal dialog system which manages the user-robot interaction during the communicative process. One of its main components is the presented user localization system. To determine the most suitable allocation of the robot in relation to the user, a proxemic study of the human-robot interaction is required, which is described in this paper. The study has been made with two groups of users: children, aged between 8 and 17, and adults. Finally, at the end of the paper, experimental results with the proposed multimodal dialog system are presented.

Affective Computing and the Impact of Gender and Age

PubMed Central

Rukavina, Stefanie; Gruss, Sascha; Hoffmann, Holger; Tan, Jun-Wen; Walter, Steffen; Traue, Harald C.

2016-01-01

Affective computing aims at the detection of users’ mental states, in particular, emotions and dispositions during human-computer interactions. Detection can be achieved by measuring multimodal signals, namely, speech, facial expressions and/or psychobiology. Over the past years, one major approach was to identify the best features for each signal using different classification methods. Although this is of high priority, other subject-specific variables should not be neglected. In our study, we analyzed the effect of gender, age, personality and gender roles on the extracted psychobiological features (derived from skin conductance level, facial electromyography and heart rate variability) as well as the influence on the classification results. In an experimental human-computer interaction, five different affective states with picture material from the International Affective Picture System and ULM pictures were induced. A total of 127 subjects participated in the study. Among all potentially influencing variables (gender has been reported to be influential), age was the only variable that correlated significantly with psychobiological responses. In summary, the conducted classification processes resulted in 20% classification accuracy differences according to age and gender, especially when comparing the neutral condition with four other affective states. We suggest taking age and gender specifically into account for future studies in affective computing, as these may lead to an improvement of emotion recognition accuracy. PMID:26939129

See You See Me: the Role of Eye Contact in Multimodal Human-Robot Interaction

PubMed Central

XU, TIAN (LINGER); ZHANG, HUI; YU, CHEN

2016-01-01

We focus on a fundamental looking behavior in human-robot interactions – gazing at each other’s face. Eye contact and mutual gaze between two social partners are critical in smooth human-human interactions. Therefore, investigating at what moments and in what ways a robot should look at a human user’s face as a response to the human’s gaze behavior is an important topic. Toward this goal, we developed a gaze-contingent human-robot interaction system, which relied on momentary gaze behaviors from a human user to control an interacting robot in real time. Using this system, we conducted an experiment in which human participants interacted with the robot in a joint attention task. In the experiment, we systematically manipulated the robot’s gaze toward the human partner’s face in real time and then analyzed the human’s gaze behavior as a response to the robot’s gaze behavior. We found that more face looks from the robot led to more look-backs (to the robot’s face) from human participants and consequently created more mutual gaze and eye contact between the two. Moreover, participants demonstrated more coordinated and synchronized multimodal behaviors between speech and gaze when more eye contact was successfully established and maintained. PMID:28966875

Multimodality: a basis for augmentative and alternative communication--psycholinguistic, cognitive, and clinical/educational aspects.

PubMed

Loncke, Filip T; Campbell, Jamie; England, Amanda M; Haley, Tanya

2006-02-15

Message generating is a complex process involving a number of processes, including the selection of modes to use. When expressing a message, human communicators typically use a combination of modes. This phenomenon is often termed multimodality. This article explores the use of models that explain multimodality as an explanatory framework for augmentative and alternative communication (AAC). Multimodality is analysed from a communication, psycholinguistic, and cognitive perspective. Theoretical and applied topics within AAC can be explained or described within the multimodality framework considering iconicity, simultaneous communication, lexical organization, and compatibility of communication modes. Consideration of multimodality is critical to understanding underlying processes in individuals who use AAC and individuals who interact with them.

Volume curtaining: a focus+context effect for multimodal volume visualization

NASA Astrophysics Data System (ADS)

Fairfield, Adam J.; Plasencia, Jonathan; Jang, Yun; Theodore, Nicholas; Crawford, Neil R.; Frakes, David H.; Maciejewski, Ross

2014-03-01

In surgical preparation, physicians will often utilize multimodal imaging scans to capture complementary information to improve diagnosis and to drive patient-specific treatment. These imaging scans may consist of data from magnetic resonance imaging (MR), computed tomography (CT), or other various sources. The challenge in using these different modalities is that the physician must mentally map the two modalities together during the diagnosis and planning phase. Furthermore, the different imaging modalities will be generated at various resolutions as well as slightly different orientations due to patient placement during scans. In this work, we present an interactive system for multimodal data fusion, analysis and visualization. Developed with partners from neurological clinics, this work discusses initial system requirements and physician feedback at the various stages of component development. Finally, we present a novel focus+context technique for the interactive exploration of coregistered multi-modal data.

A Physics-driven Neural Networks-based Simulation System (PhyNNeSS) for multimodal interactive virtual environments involving nonlinear deformable objects

PubMed Central

De, Suvranu; Deo, Dhannanjay; Sankaranarayanan, Ganesh; Arikatla, Venkata S.

2012-01-01

Background While an update rate of 30 Hz is considered adequate for real time graphics, a much higher update rate of about 1 kHz is necessary for haptics. Physics-based modeling of deformable objects, especially when large nonlinear deformations and complex nonlinear material properties are involved, at these very high rates is one of the most challenging tasks in the development of real time simulation systems. While some specialized solutions exist, there is no general solution for arbitrary nonlinearities. Methods In this work we present PhyNNeSS - a Physics-driven Neural Networks-based Simulation System - to address this long-standing technical challenge. The first step is an off-line pre-computation step in which a database is generated by applying carefully prescribed displacements to each node of the finite element models of the deformable objects. In the next step, the data is condensed into a set of coefficients describing neurons of a Radial Basis Function network (RBFN). During real-time computation, these neural networks are used to reconstruct the deformation fields as well as the interaction forces. Results We present realistic simulation examples from interactive surgical simulation with real time force feedback. As an example, we have developed a deformable human stomach model and a Penrose-drain model used in the Fundamentals of Laparoscopic Surgery (FLS) training tool box. Conclusions A unique computational modeling system has been developed that is capable of simulating the response of nonlinear deformable objects in real time. The method distinguishes itself from previous efforts in that a systematic physics-based pre-computational step allows training of neural networks which may be used in real time simulations. We show, through careful error analysis, that the scheme is scalable, with the accuracy being controlled by the number of neurons used in the simulation. PhyNNeSS has been integrated into SoFMIS (Software Framework for Multimodal Interactive Simulation) for general use. PMID:22629108

Investigation of protein selectivity in multimodal chromatography using in silico designed Fab fragment variants.

PubMed

Karkov, Hanne Sophie; Krogh, Berit Olsen; Woo, James; Parimal, Siddharth; Ahmadian, Haleh; Cramer, Steven M

2015-11-01

In this study, a unique set of antibody Fab fragments was designed in silico and produced to examine the relationship between protein surface properties and selectivity in multimodal chromatographic systems. We hypothesized that multimodal ligands containing both hydrophobic and charged moieties would interact strongly with protein surface regions where charged groups and hydrophobic patches were in close spatial proximity. Protein surface property characterization tools were employed to identify the potential multimodal ligand binding regions on the Fab fragment of a humanized antibody and to evaluate the impact of mutations on surface charge and hydrophobicity. Twenty Fab variants were generated by site-directed mutagenesis, recombinant expression, and affinity purification. Column gradient experiments were carried out with the Fab variants in multimodal, cation-exchange, and hydrophobic interaction chromatographic systems. The results clearly indicated that selectivity in the multimodal system was different from the other chromatographic modes examined. Column retention data for the reduced charge Fab variants identified a binding site comprising light chain CDR1 as the main electrostatic interaction site for the multimodal and cation-exchange ligands. Furthermore, the multimodal ligand binding was enhanced by additional hydrophobic contributions as evident from the results obtained with hydrophobic Fab variants. The use of in silico protein surface property analyses combined with molecular biology techniques, protein expression, and chromatographic evaluations represents a previously undescribed and powerful approach for investigating multimodal selectivity with complex biomolecules. © 2015 Wiley Periodicals, Inc.

Computer-assisted surgical planning and automation of laser delivery systems

NASA Astrophysics Data System (ADS)

Zamorano, Lucia J.; Dujovny, Manuel; Dong, Ada; Kadi, A. Majeed

1991-05-01

This paper describes a 'real time' surgical treatment planning interactive workstation, utilizing multimodality imaging (computer tomography, magnetic resonance imaging, digital angiography) that has been developed to provide the neurosurgeon with two-dimensional multiplanar and three-dimensional 'display' of a patient's lesion.

HCI∧2 framework: a software framework for multimodal human-computer interaction systems.

PubMed

Shen, Jie; Pantic, Maja

2013-12-01

This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed software framework, which is called the HCI∧2 Framework, is built upon publish/subscribe (P/S) architecture. It implements a shared-memory-based data transport protocol for message delivery and a TCP-based system management protocol. The latter ensures that the integrity of system structure is maintained at runtime. With the inclusion of bridging modules, the HCI∧2 Framework is interoperable with other software frameworks including Psyclone and ActiveMQ. In addition to the core communication middleware, we also present the integrated development environment (IDE) of the HCI∧2 Framework. It provides a complete graphical environment to support every step in a typical MHCI system development process, including module development, debugging, packaging, and management, as well as the whole system management and testing. The quantitative evaluation indicates that our framework outperforms other similar tools in terms of average message latency and maximum data throughput under a typical single PC scenario. To demonstrate HCI∧2 Framework's capabilities in integrating heterogeneous modules, we present several example modules working with a variety of hardware and software. We also present an example of a full system developed using the proposed HCI∧2 Framework, which is called the CamGame system and represents a computer game based on hand-held marker(s) and low-cost camera(s).

Interactive multi-mode blade impact analysis

NASA Technical Reports Server (NTRS)

Alexander, A.; Cornell, R. W.

1978-01-01

The theoretical methodology used in developing an analysis for the response of turbine engine fan blades subjected to soft-body (bird) impacts is reported, and the computer program developed using this methodology as its basis is described. This computer program is an outgrowth of two programs that were previously developed for the purpose of studying problems of a similar nature (a 3-mode beam impact analysis and a multi-mode beam impact analysis). The present program utilizes an improved missile model that is interactively coupled with blade motion which is more consistent with actual observations. It takes into account local deformation at the impact area, blade camber effects, and the spreading of the impacted missile mass on the blade surface. In addition, it accommodates plate-type mode shapes. The analysis capability in this computer program represents a significant improvement in the development of the methodology for evaluating potential fan blade materials and designs with regard to foreign object impact resistance.

A new strategic neurosurgical planning tool for brainstem cavernous malformations using interactive computer graphics with multimodal fusion images.

PubMed

Kin, Taichi; Nakatomi, Hirofumi; Shojima, Masaaki; Tanaka, Minoru; Ino, Kenji; Mori, Harushi; Kunimatsu, Akira; Oyama, Hiroshi; Saito, Nobuhito

2012-07-01

In this study, the authors used preoperative simulation employing 3D computer graphics (interactive computer graphics) to fuse all imaging data for brainstem cavernous malformations. The authors evaluated whether interactive computer graphics or 2D imaging correlated better with the actual operative field, particularly in identifying a developmental venous anomaly (DVA). The study population consisted of 10 patients scheduled for surgical treatment of brainstem cavernous malformations. Data from preoperative imaging (MRI, CT, and 3D rotational angiography) were automatically fused using a normalized mutual information method, and then reconstructed by a hybrid method combining surface rendering and volume rendering methods. With surface rendering, multimodality and multithreshold techniques for 1 tissue were applied. The completed interactive computer graphics were used for simulation of surgical approaches and assumed surgical fields. Preoperative diagnostic rates for a DVA associated with brainstem cavernous malformation were compared between conventional 2D imaging and interactive computer graphics employing receiver operating characteristic (ROC) analysis. The time required for reconstruction of 3D images was 3-6 hours for interactive computer graphics. Observation in interactive mode required approximately 15 minutes. Detailed anatomical information for operative procedures, from the craniotomy to microsurgical operations, could be visualized and simulated three-dimensionally as 1 computer graphic using interactive computer graphics. Virtual surgical views were consistent with actual operative views. This technique was very useful for examining various surgical approaches. Mean (±SEM) area under the ROC curve for rate of DVA diagnosis was significantly better for interactive computer graphics (1.000±0.000) than for 2D imaging (0.766±0.091; p<0.001, Mann-Whitney U-test). The authors report a new method for automatic registration of preoperative imaging data from CT, MRI, and 3D rotational angiography for reconstruction into 1 computer graphic. The diagnostic rate of DVA associated with brainstem cavernous malformation was significantly better using interactive computer graphics than with 2D images. Interactive computer graphics was also useful in helping to plan the surgical access corridor.

Design of a compact low-power human-computer interaction equipment for hand motion

NASA Astrophysics Data System (ADS)

Wu, Xianwei; Jin, Wenguang

2017-01-01

Human-Computer Interaction (HCI) raises demand of convenience, endurance, responsiveness and naturalness. This paper describes a design of a compact wearable low-power HCI equipment applied to gesture recognition. System combines multi-mode sense signals: the vision sense signal and the motion sense signal, and the equipment is equipped with the depth camera and the motion sensor. The dimension (40 mm × 30 mm) and structure is compact and portable after tight integration. System is built on a module layered framework, which contributes to real-time collection (60 fps), process and transmission via synchronous confusion with asynchronous concurrent collection and wireless Blue 4.0 transmission. To minimize equipment's energy consumption, system makes use of low-power components, managing peripheral state dynamically, switching into idle mode intelligently, pulse-width modulation (PWM) of the NIR LEDs of the depth camera and algorithm optimization by the motion sensor. To test this equipment's function and performance, a gesture recognition algorithm is applied to system. As the result presents, general energy consumption could be as low as 0.5 W.

Shared periodic performer movements coordinate interactions in duo improvisations.

PubMed

Eerola, Tuomas; Jakubowski, Kelly; Moran, Nikki; Keller, Peter E; Clayton, Martin

2018-02-01

Human interaction involves the exchange of temporally coordinated, multimodal cues. Our work focused on interaction in the visual domain, using music performance as a case for analysis due to its temporally diverse and hierarchical structures. We made use of two improvising duo datasets-(i) performances of a jazz standard with a regular pulse and (ii) non-pulsed, free improvizations-to investigate whether human judgements of moments of interaction between co-performers are influenced by body movement coordination at multiple timescales. Bouts of interaction in the performances were manually annotated by experts and the performers' movements were quantified using computer vision techniques. The annotated interaction bouts were then predicted using several quantitative movement and audio features. Over 80% of the interaction bouts were successfully predicted by a broadband measure of the energy of the cross-wavelet transform of the co-performers' movements in non-pulsed duos. A more complex model, with multiple predictors that captured more specific, interacting features of the movements, was needed to explain a significant amount of variance in the pulsed duos. The methods developed here have key implications for future work on measuring visual coordination in musical ensemble performances, and can be easily adapted to other musical contexts, ensemble types and traditions.

3D hierarchical spatial representation and memory of multimodal sensory data

NASA Astrophysics Data System (ADS)

Khosla, Deepak; Dow, Paul A.; Huber, David J.

2009-04-01

This paper describes an efficient method and system for representing, processing and understanding multi-modal sensory data. More specifically, it describes a computational method and system for how to process and remember multiple locations in multimodal sensory space (e.g., visual, auditory, somatosensory, etc.). The multimodal representation and memory is based on a biologically-inspired hierarchy of spatial representations implemented with novel analogues of real representations used in the human brain. The novelty of the work is in the computationally efficient and robust spatial representation of 3D locations in multimodal sensory space as well as an associated working memory for storage and recall of these representations at the desired level for goal-oriented action. We describe (1) A simple and efficient method for human-like hierarchical spatial representations of sensory data and how to associate, integrate and convert between these representations (head-centered coordinate system, body-centered coordinate, etc.); (2) a robust method for training and learning a mapping of points in multimodal sensory space (e.g., camera-visible object positions, location of auditory sources, etc.) to the above hierarchical spatial representations; and (3) a specification and implementation of a hierarchical spatial working memory based on the above for storage and recall at the desired level for goal-oriented action(s). This work is most useful for any machine or human-machine application that requires processing of multimodal sensory inputs, making sense of it from a spatial perspective (e.g., where is the sensory information coming from with respect to the machine and its parts) and then taking some goal-oriented action based on this spatial understanding. A multi-level spatial representation hierarchy means that heterogeneous sensory inputs (e.g., visual, auditory, somatosensory, etc.) can map onto the hierarchy at different levels. When controlling various machine/robot degrees of freedom, the desired movements and action can be computed from these different levels in the hierarchy. The most basic embodiment of this machine could be a pan-tilt camera system, an array of microphones, a machine with arm/hand like structure or/and a robot with some or all of the above capabilities. We describe the approach, system and present preliminary results on a real-robotic platform.

Multimodal emotional state recognition using sequence-dependent deep hierarchical features.

PubMed

Barros, Pablo; Jirak, Doreen; Weber, Cornelius; Wermter, Stefan

2015-12-01

Emotional state recognition has become an important topic for human-robot interaction in the past years. By determining emotion expressions, robots can identify important variables of human behavior and use these to communicate in a more human-like fashion and thereby extend the interaction possibilities. Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots. Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion. Our model uses a hierarchical feature representation to deal with spontaneous emotions, and learns how to integrate multiple modalities for non-verbal emotion recognition, making it suitable to be used in an HRI scenario. Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-the-art approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions. Copyright © 2015 The Authors. Published by Elsevier Ltd.. All rights reserved.

EEG Classification for Hybrid Brain-Computer Interface Using a Tensor Based Multiclass Multimodal Analysis Scheme

PubMed Central

Ji, Hongfei; Li, Jie; Lu, Rongrong; Gu, Rong; Cao, Lei; Gong, Xiaoliang

2016-01-01

Electroencephalogram- (EEG-) based brain-computer interface (BCI) systems usually utilize one type of changes in the dynamics of brain oscillations for control, such as event-related desynchronization/synchronization (ERD/ERS), steady state visual evoked potential (SSVEP), and P300 evoked potentials. There is a recent trend to detect more than one of these signals in one system to create a hybrid BCI. However, in this case, EEG data were always divided into groups and analyzed by the separate processing procedures. As a result, the interactive effects were ignored when different types of BCI tasks were executed simultaneously. In this work, we propose an improved tensor based multiclass multimodal scheme especially for hybrid BCI, in which EEG signals are denoted as multiway tensors, a nonredundant rank-one tensor decomposition model is proposed to obtain nonredundant tensor components, a weighted fisher criterion is designed to select multimodal discriminative patterns without ignoring the interactive effects, and support vector machine (SVM) is extended to multiclass classification. Experiment results suggest that the proposed scheme can not only identify the different changes in the dynamics of brain oscillations induced by different types of tasks but also capture the interactive effects of simultaneous tasks properly. Therefore, it has great potential use for hybrid BCI. PMID:26880873

EEG Classification for Hybrid Brain-Computer Interface Using a Tensor Based Multiclass Multimodal Analysis Scheme.

PubMed

Ji, Hongfei; Li, Jie; Lu, Rongrong; Gu, Rong; Cao, Lei; Gong, Xiaoliang

2016-01-01

Electroencephalogram- (EEG-) based brain-computer interface (BCI) systems usually utilize one type of changes in the dynamics of brain oscillations for control, such as event-related desynchronization/synchronization (ERD/ERS), steady state visual evoked potential (SSVEP), and P300 evoked potentials. There is a recent trend to detect more than one of these signals in one system to create a hybrid BCI. However, in this case, EEG data were always divided into groups and analyzed by the separate processing procedures. As a result, the interactive effects were ignored when different types of BCI tasks were executed simultaneously. In this work, we propose an improved tensor based multiclass multimodal scheme especially for hybrid BCI, in which EEG signals are denoted as multiway tensors, a nonredundant rank-one tensor decomposition model is proposed to obtain nonredundant tensor components, a weighted fisher criterion is designed to select multimodal discriminative patterns without ignoring the interactive effects, and support vector machine (SVM) is extended to multiclass classification. Experiment results suggest that the proposed scheme can not only identify the different changes in the dynamics of brain oscillations induced by different types of tasks but also capture the interactive effects of simultaneous tasks properly. Therefore, it has great potential use for hybrid BCI.

A new multimodal interactive way of subjective scoring of 3D video quality of experience

NASA Astrophysics Data System (ADS)

Kim, Taewan; Lee, Kwanghyun; Lee, Sanghoon; Bovik, Alan C.

2014-03-01

People that watch today's 3D visual programs, such as 3D cinema, 3D TV and 3D games, experience wide and dynamically varying ranges of 3D visual immersion and 3D quality of experience (QoE). It is necessary to be able to deploy reliable methodologies that measure each viewers subjective experience. We propose a new methodology that we call Multimodal Interactive Continuous Scoring of Quality (MICSQ). MICSQ is composed of a device interaction process between the 3D display and a separate device (PC, tablet, etc.) used as an assessment tool, and a human interaction process between the subject(s) and the device. The scoring process is multimodal, using aural and tactile cues to help engage and focus the subject(s) on their tasks. Moreover, the wireless device interaction process makes it possible for multiple subjects to assess 3D QoE simultaneously in a large space such as a movie theater, and at di®erent visual angles and distances.

Interactive natural language acquisition in a multi-modal recurrent neural architecture

NASA Astrophysics Data System (ADS)

Heinrich, Stefan; Wermter, Stefan

2018-01-01

For the complex human brain that enables us to communicate in natural language, we gathered good understandings of principles underlying language acquisition and processing, knowledge about sociocultural conditions, and insights into activity patterns in the brain. However, we were not yet able to understand the behavioural and mechanistic characteristics for natural language and how mechanisms in the brain allow to acquire and process language. In bridging the insights from behavioural psychology and neuroscience, the goal of this paper is to contribute a computational understanding of appropriate characteristics that favour language acquisition. Accordingly, we provide concepts and refinements in cognitive modelling regarding principles and mechanisms in the brain and propose a neurocognitively plausible model for embodied language acquisition from real-world interaction of a humanoid robot with its environment. In particular, the architecture consists of a continuous time recurrent neural network, where parts have different leakage characteristics and thus operate on multiple timescales for every modality and the association of the higher level nodes of all modalities into cell assemblies. The model is capable of learning language production grounded in both, temporal dynamic somatosensation and vision, and features hierarchical concept abstraction, concept decomposition, multi-modal integration, and self-organisation of latent representations.

Multimodal neuroelectric interface development

NASA Technical Reports Server (NTRS)

Trejo, Leonard J.; Wheeler, Kevin R.; Jorgensen, Charles C.; Rosipal, Roman; Clanton, Sam T.; Matthews, Bryan; Hibbs, Andrew D.; Matthews, Robert; Krupka, Michael

2003-01-01

We are developing electromyographic and electroencephalographic methods, which draw control signals for human-computer interfaces from the human nervous system. We have made progress in four areas: 1) real-time pattern recognition algorithms for decoding sequences of forearm muscle activity associated with control gestures; 2) signal-processing strategies for computer interfaces using electroencephalogram (EEG) signals; 3) a flexible computation framework for neuroelectric interface research; and d) noncontact sensors, which measure electromyogram or EEG signals without resistive contact to the body.

Multimodal interaction for human-robot teams

NASA Astrophysics Data System (ADS)

Burke, Dustin; Schurr, Nathan; Ayers, Jeanine; Rousseau, Jeff; Fertitta, John; Carlin, Alan; Dumond, Danielle

2013-05-01

Unmanned ground vehicles have the potential for supporting small dismounted teams in mapping facilities, maintaining security in cleared buildings, and extending the team's reconnaissance and persistent surveillance capability. In order for such autonomous systems to integrate with the team, we must move beyond current interaction methods using heads-down teleoperation which require intensive human attention and affect the human operator's ability to maintain local situational awareness and ensure their own safety. This paper focuses on the design, development and demonstration of a multimodal interaction system that incorporates naturalistic human gestures, voice commands, and a tablet interface. By providing multiple, partially redundant interaction modes, our system degrades gracefully in complex environments and enables the human operator to robustly select the most suitable interaction method given the situational demands. For instance, the human can silently use arm and hand gestures for commanding a team of robots when it is important to maintain stealth. The tablet interface provides an overhead situational map allowing waypoint-based navigation for multiple ground robots in beyond-line-of-sight conditions. Using lightweight, wearable motion sensing hardware either worn comfortably beneath the operator's clothing or integrated within their uniform, our non-vision-based approach enables an accurate, continuous gesture recognition capability without line-of-sight constraints. To reduce the training necessary to operate the system, we designed the interactions around familiar arm and hand gestures.

A unified coding strategy for processing faces and voices

PubMed Central

Yovel, Galit; Belin, Pascal

2013-01-01

Both faces and voices are rich in socially-relevant information, which humans are remarkably adept at extracting, including a person's identity, age, gender, affective state, personality, etc. Here, we review accumulating evidence from behavioral, neuropsychological, electrophysiological, and neuroimaging studies which suggest that the cognitive and neural processing mechanisms engaged by perceiving faces or voices are highly similar, despite the very different nature of their sensory input. The similarity between the two mechanisms likely facilitates the multi-modal integration of facial and vocal information during everyday social interactions. These findings emphasize a parsimonious principle of cerebral organization, where similar computational problems in different modalities are solved using similar solutions. PMID:23664703

Vestibular system: the many facets of a multimodal sense.

PubMed

Angelaki, Dora E; Cullen, Kathleen E

2008-01-01

Elegant sensory structures in the inner ear have evolved to measure head motion. These vestibular receptors consist of highly conserved semicircular canals and otolith organs. Unlike other senses, vestibular information in the central nervous system becomes immediately multisensory and multimodal. There is no overt, readily recognizable conscious sensation from these organs, yet vestibular signals contribute to a surprising range of brain functions, from the most automatic reflexes to spatial perception and motor coordination. Critical to these diverse, multimodal functions are multiple computationally intriguing levels of processing. For example, the need for multisensory integration necessitates vestibular representations in multiple reference frames. Proprioceptive-vestibular interactions, coupled with corollary discharge of a motor plan, allow the brain to distinguish actively generated from passive head movements. Finally, nonlinear interactions between otolith and canal signals allow the vestibular system to function as an inertial sensor and contribute critically to both navigation and spatial orientation.

Shared periodic performer movements coordinate interactions in duo improvisations

PubMed Central

Jakubowski, Kelly; Moran, Nikki; Keller, Peter E.

2018-01-01

Human interaction involves the exchange of temporally coordinated, multimodal cues. Our work focused on interaction in the visual domain, using music performance as a case for analysis due to its temporally diverse and hierarchical structures. We made use of two improvising duo datasets—(i) performances of a jazz standard with a regular pulse and (ii) non-pulsed, free improvizations—to investigate whether human judgements of moments of interaction between co-performers are influenced by body movement coordination at multiple timescales. Bouts of interaction in the performances were manually annotated by experts and the performers’ movements were quantified using computer vision techniques. The annotated interaction bouts were then predicted using several quantitative movement and audio features. Over 80% of the interaction bouts were successfully predicted by a broadband measure of the energy of the cross-wavelet transform of the co-performers’ movements in non-pulsed duos. A more complex model, with multiple predictors that captured more specific, interacting features of the movements, was needed to explain a significant amount of variance in the pulsed duos. The methods developed here have key implications for future work on measuring visual coordination in musical ensemble performances, and can be easily adapted to other musical contexts, ensemble types and traditions. PMID:29515867

A multimodal interface to resolve the Midas-Touch problem in gaze controlled wheelchair.

PubMed

Meena, Yogesh Kumar; Cecotti, Hubert; Wong-Lin, KongFatt; Prasad, Girijesh

2017-07-01

Human-computer interaction (HCI) research has been playing an essential role in the field of rehabilitation. The usability of the gaze controlled powered wheelchair is limited due to Midas-Touch problem. In this work, we propose a multimodal graphical user interface (GUI) to control a powered wheelchair that aims to help upper-limb mobility impaired people in daily living activities. The GUI was designed to include a portable and low-cost eye-tracker and a soft-switch wherein the wheelchair can be controlled in three different ways: 1) with a touchpad 2) with an eye-tracker only, and 3) eye-tracker with soft-switch. The interface includes nine different commands (eight directions and stop) and integrated within a powered wheelchair system. We evaluated the performance of the multimodal interface in terms of lap-completion time, the number of commands, and the information transfer rate (ITR) with eight healthy participants. The analysis of the results showed that the eye-tracker with soft-switch provides superior performance with an ITR of 37.77 bits/min among the three different conditions (p<;0.05). Thus, the proposed system provides an effective and economical solution to the Midas-Touch problem and extended usability for the large population of disabled users.

The Effects of Instructor-Avatar Immediacy in Second Life, an Immersive and Interactive Three-Dimensional Virtual Environment

ERIC Educational Resources Information Center

Lawless-Reljic, Sabine Karine

2010-01-01

Growing interest of educational institutions in desktop 3D graphic virtual environments for hybrid and distance education prompts questions on the efficacy of such tools. Virtual worlds, such as Second Life[R], enable computer-mediated immersion and interactions encompassing multimodal communication channels including audio, video, and text-.…

Mandarin Students' Perceptions of Multimodal Interaction in a Web Conferencing Environment: A Satisfaction Survey

ERIC Educational Resources Information Center

Tseng, Jun-Jie

2015-01-01

A major indicator of whether online courses have been effective and successful is student satisfaction. Copious research points to lack of interaction as the most cited reason for student dissatisfaction. To improve this problem, new Computer-Mediated Communication (CMC) technology could be considered as an option to enhance the online learning…

Multimodal system for the planning and guidance of bronchoscopy

NASA Astrophysics Data System (ADS)

Higgins, William E.; Cheirsilp, Ronnarit; Zang, Xiaonan; Byrnes, Patrick

2015-03-01

Many technical innovations in multimodal radiologic imaging and bronchoscopy have emerged recently in the effort against lung cancer. Modern X-ray computed-tomography (CT) scanners provide three-dimensional (3D) high-resolution chest images, positron emission tomography (PET) scanners give complementary molecular imaging data, and new integrated PET/CT scanners combine the strengths of both modalities. State-of-the-art bronchoscopes permit minimally invasive tissue sampling, with vivid endobronchial video enabling navigation deep into the airway-tree periphery, while complementary endobronchial ultrasound (EBUS) reveals local views of anatomical structures outside the airways. In addition, image-guided intervention (IGI) systems have proven their utility for CT-based planning and guidance of bronchoscopy. Unfortunately, no IGI system exists that integrates all sources effectively through the complete lung-cancer staging work flow. This paper presents a prototype of a computer-based multimodal IGI system that strives to fill this need. The system combines a wide range of automatic and semi-automatic image-processing tools for multimodal data fusion and procedure planning. It also provides a flexible graphical user interface for follow-on guidance of bronchoscopy/EBUS. Human-study results demonstrate the system's potential.

Ghost-in-the-Machine reveals human social signals for human-robot interaction.

PubMed

Loth, Sebastian; Jettka, Katharina; Giuliani, Manuel; de Ruiter, Jan P

2015-01-01

We used a new method called "Ghost-in-the-Machine" (GiM) to investigate social interactions with a robotic bartender taking orders for drinks and serving them. Using the GiM paradigm allowed us to identify how human participants recognize the intentions of customers on the basis of the output of the robotic recognizers. Specifically, we measured which recognizer modalities (e.g., speech, the distance to the bar) were relevant at different stages of the interaction. This provided insights into human social behavior necessary for the development of socially competent robots. When initiating the drink-order interaction, the most important recognizers were those based on computer vision. When drink orders were being placed, however, the most important information source was the speech recognition. Interestingly, the participants used only a subset of the available information, focussing only on a few relevant recognizers while ignoring others. This reduced the risk of acting on erroneous sensor data and enabled them to complete service interactions more swiftly than a robot using all available sensor data. We also investigated socially appropriate response strategies. In their responses, the participants preferred to use the same modality as the customer's requests, e.g., they tended to respond verbally to verbal requests. Also, they added redundancy to their responses, for instance by using echo questions. We argue that incorporating the social strategies discovered with the GiM paradigm in multimodal grammars of human-robot interactions improves the robustness and the ease-of-use of these interactions, and therefore provides a smoother user experience.

A multimodal interface for real-time soldier-robot teaming

NASA Astrophysics Data System (ADS)

Barber, Daniel J.; Howard, Thomas M.; Walter, Matthew R.

2016-05-01

Recent research and advances in robotics have led to the development of novel platforms leveraging new sensing capabilities for semantic navigation. As these systems becoming increasingly more robust, they support highly complex commands beyond direct teleoperation and waypoint finding facilitating a transition away from robots as tools to robots as teammates. Supporting future Soldier-Robot teaming requires communication capabilities on par with human-human teams for successful integration of robots. Therefore, as robots increase in functionality, it is equally important that the interface between the Soldier and robot advances as well. Multimodal communication (MMC) enables human-robot teaming through redundancy and levels of communications more robust than single mode interaction. Commercial-off-the-shelf (COTS) technologies released in recent years for smart-phones and gaming provide tools for the creation of portable interfaces incorporating MMC through the use of speech, gestures, and visual displays. However, for multimodal interfaces to be successfully used in the military domain, they must be able to classify speech, gestures, and process natural language in real-time with high accuracy. For the present study, a prototype multimodal interface supporting real-time interactions with an autonomous robot was developed. This device integrated COTS Automated Speech Recognition (ASR), a custom gesture recognition glove, and natural language understanding on a tablet. This paper presents performance results (e.g. response times, accuracy) of the integrated device when commanding an autonomous robot to perform reconnaissance and surveillance activities in an unknown outdoor environment.

Optimization Model for Web Based Multimodal Interactive Simulations.

PubMed

Halic, Tansel; Ahn, Woojin; De, Suvranu

2015-07-15

This paper presents a technique for optimizing the performance of web based multimodal interactive simulations. For such applications where visual quality and the performance of simulations directly influence user experience, overloading of hardware resources may result in unsatisfactory reduction in the quality of the simulation and user satisfaction. However, optimization of simulation performance on individual hardware platforms is not practical. Hence, we present a mixed integer programming model to optimize the performance of graphical rendering and simulation performance while satisfying application specific constraints. Our approach includes three distinct phases: identification, optimization and update . In the identification phase, the computing and rendering capabilities of the client device are evaluated using an exploratory proxy code. This data is utilized in conjunction with user specified design requirements in the optimization phase to ensure best possible computational resource allocation. The optimum solution is used for rendering (e.g. texture size, canvas resolution) and simulation parameters (e.g. simulation domain) in the update phase. Test results are presented on multiple hardware platforms with diverse computing and graphics capabilities to demonstrate the effectiveness of our approach.

Optimization Model for Web Based Multimodal Interactive Simulations

PubMed Central

Halic, Tansel; Ahn, Woojin; De, Suvranu

2015-01-01

This paper presents a technique for optimizing the performance of web based multimodal interactive simulations. For such applications where visual quality and the performance of simulations directly influence user experience, overloading of hardware resources may result in unsatisfactory reduction in the quality of the simulation and user satisfaction. However, optimization of simulation performance on individual hardware platforms is not practical. Hence, we present a mixed integer programming model to optimize the performance of graphical rendering and simulation performance while satisfying application specific constraints. Our approach includes three distinct phases: identification, optimization and update. In the identification phase, the computing and rendering capabilities of the client device are evaluated using an exploratory proxy code. This data is utilized in conjunction with user specified design requirements in the optimization phase to ensure best possible computational resource allocation. The optimum solution is used for rendering (e.g. texture size, canvas resolution) and simulation parameters (e.g. simulation domain) in the update phase. Test results are presented on multiple hardware platforms with diverse computing and graphics capabilities to demonstrate the effectiveness of our approach. PMID:26085713

Semantic Entity-Component State Management Techniques to Enhance Software Quality for Multimodal VR-Systems.

PubMed

Fischbach, Martin; Wiebusch, Dennis; Latoschik, Marc Erich

2017-04-01

Modularity, modifiability, reusability, and API usability are important software qualities that determine the maintainability of software architectures. Virtual, Augmented, and Mixed Reality (VR, AR, MR) systems, modern computer games, as well as interactive human-robot systems often include various dedicated input-, output-, and processing subsystems. These subsystems collectively maintain a real-time simulation of a coherent application state. The resulting interdependencies between individual state representations, mutual state access, overall synchronization, and flow of control implies a conceptual close coupling whereas software quality asks for a decoupling to develop maintainable solutions. This article presents five semantics-based software techniques that address this contradiction: Semantic grounding, code from semantics, grounded actions, semantic queries, and decoupling by semantics. These techniques are applied to extend the well-established entity-component-system (ECS) pattern to overcome some of this pattern's deficits with respect to the implied state access. A walk-through of central implementation aspects of a multimodal (speech and gesture) VR-interface is used to highlight the techniques' benefits. This use-case is chosen as a prototypical example of complex architectures with multiple interacting subsystems found in many VR, AR and MR architectures. Finally, implementation hints are given, lessons learned regarding maintainability pointed-out, and performance implications discussed.

Virtual microscopy: merging of computer mediated communication and intuitive interfacing

NASA Astrophysics Data System (ADS)

de Ridder, Huib; de Ridder-Sluiter, Johanna G.; Kluin, Philip M.; Christiaans, Henri H. C. M.

2009-02-01

Ubiquitous computing (or Ambient Intelligence) is an upcoming technology that is usually associated with futuristic smart environments in which information is available anytime anywhere and with which humans can interact in a natural, multimodal way. However spectacular the corresponding scenarios may be, it is equally challenging to consider how this technology may enhance existing situations. This is illustrated by a case study from the Dutch medical field: central quality reviewing for pathology in child oncology. The main goal of the review is to assess the quality of the diagnosis based on patient material. The sharing of knowledge in social face-to-face interaction during such meeting is an important advantage. At the same time there is the disadvantage that the experts from the seven Dutch academic medical centers have to travel to the review meeting and that the required logistics to collect and bring patient material and data to the meeting is cumbersome and time-consuming. This paper focuses on how this time-consuming, nonefficient way of reviewing can be replaced by a virtual collaboration system by merging technology supporting Computer Mediated Collaboration and intuitive interfacing. This requires insight in the preferred way of communication and collaboration as well as knowledge about preferred interaction style with a virtual shared workspace.

Sustaining Multimodal Language Learner Interactions Online

ERIC Educational Resources Information Center

Satar, H. Müge

2015-01-01

Social presence is considered an important quality in computer-mediated communication as it promotes willingness in learners to take risks through participation in interpersonal exchanges (Kehrwald, 2008) and makes communication more natural (Lowenthal, 2010). While social presence has mostly been investigated through questionnaire data and…

Numerical investigation of nonlinear interactions between multimodal guided waves and delamination in composite structures

NASA Astrophysics Data System (ADS)

Shen, Yanfeng

2017-04-01

This paper presents a numerical investigation of the nonlinear interactions between multimodal guided waves and delamination in composite structures. The elastodynamic wave equations for anisotropic composite laminate were formulated using an explicit Local Interaction Simulation Approach (LISA). The contact dynamics was modeled using the penalty method. In order to capture the stick-slip contact motion, a Coulomb friction law was integrated into the computation procedure. A random gap function was defined for the contact pairs to model distributed initial closures or openings to approximate the nature of rough delamination interfaces. The LISA procedure was coded using the Compute Unified Device Architecture (CUDA), which enables the highly parallelized computation on powerful graphic cards. Several guided wave modes centered at various frequencies were investigated as the incident wave. Numerical case studies of different delamination locations across the thickness were carried out. The capability of different wave modes at various frequencies to trigger the Contact Acoustic Nonlinearity (CAN) was studied. The correlation between the delamination size and the signal nonlinearity was also investigated. Furthermore, the influence from the roughness of the delamination interfaces was discussed as well. The numerical investigation shows that the nonlinear features of wave delamination interactions can enhance the evaluation capability of guided wave Structural Health Monitoring (SHM) system. This paper finishes with discussion, concluding remarks, and suggestions for future work.

Emerging Affordances in Telecollaborative Multimodal Interactions

ERIC Educational Resources Information Center

Dey-Plissonneau, Aparajita; Blin, Françoise

2016-01-01

Drawing on Gibson's (1977) theory of affordances, Computer-Assisted Language Learning (CALL) affordances are a combination of technological, social, educational, and linguistic affordances (Blin, 2016). This paper reports on a preliminary study that sought to identify the emergence of affordances during an online video conferencing session between…

Revealing Spatial Variation and Correlation of Urban Travels from Big Trajectory Data

NASA Astrophysics Data System (ADS)

Li, X.; Tu, W.; Shen, S.; Yue, Y.; Luo, N.; Li, Q.

2017-09-01

With the development of information and communication technology, spatial-temporal data that contain rich human mobility information are growing rapidly. However, the consistency of multi-mode human travel behind multi-source spatial-temporal data is not clear. To this aim, we utilized a week of taxies' and buses' GPS trajectory data and smart card data in Shenzhen, China to extract city-wide travel information of taxi, bus and metro and tested the correlation of multi-mode travel characteristics. Both the global correlation and local correlation of typical travel indicator were examined. The results show that: (1) Significant differences exist in of urban multi-mode travels. The correlation between bus travels and taxi travels, metro travel and taxi travels are globally low but locally high. (2) There are spatial differences of the correlation relationship between bus, metro and taxi travel. These findings help us understanding urban travels deeply therefore facilitate both the transport policy making and human-space interaction research.

Building adaptive connectionist-based controllers: review of experiments in human-robot interaction, collective robotics, and computational neuroscience

NASA Astrophysics Data System (ADS)

Billard, Aude

2000-10-01

This paper summarizes a number of experiments in biologically inspired robotics. The common feature to all experiments is the use of artificial neural networks as the building blocks for the controllers. The experiments speak in favor of using a connectionist approach for designing adaptive and flexible robot controllers, and for modeling neurological processes. I present 1) DRAMA, a novel connectionist architecture, which has general property for learning time series and extracting spatio-temporal regularities in multi-modal and highly noisy data; 2) Robota, a doll-shaped robot, which imitates and learns a proto-language; 3) an experiment in collective robotics, where a group of 4 to 15 Khepera robots learn dynamically the topography of an environment whose features change frequently; 4) an abstract, computational model of primate ability to learn by imitation; 5) a model for the control of locomotor gaits in a quadruped legged robot.

The integration of emotional and symbolic components in multimodal communication

PubMed Central

Mehu, Marc

2015-01-01

Human multimodal communication can be said to serve two main purposes: information transfer and social influence. In this paper, I argue that different components of multimodal signals play different roles in the processes of information transfer and social influence. Although the symbolic components of communication (e.g., verbal and denotative signals) are well suited to transfer conceptual information, emotional components (e.g., non-verbal signals that are difficult to manipulate voluntarily) likely take a function that is closer to social influence. I suggest that emotion should be considered a property of communicative signals, rather than an entity that is transferred as content by non-verbal signals. In this view, the effect of emotional processes on communication serve to change the quality of social signals to make them more efficient at producing responses in perceivers, whereas symbolic components increase the signals’ efficiency at interacting with the cognitive processes dedicated to the assessment of relevance. The interaction between symbolic and emotional components will be discussed in relation to the need for perceivers to evaluate the reliability of multimodal signals. PMID:26217280

Generation and Evaluation of User Tailored Responses in Multimodal Dialogue

ERIC Educational Resources Information Center

Walker, M. A.; Whittaker, S. J.; Stent, A.; Maloor, P.; Moore, J.; Johnston, M.; Vasireddy, G.

2004-01-01

When people engage in conversation, they tailor their utterances to their conversational partners, whether these partners are other humans or computational systems. This tailoring, or adaptation to the partner takes place in all facets of human language use, and is based on a "mental model" or a "user model" of the conversational partner. Such…

Criteria for Evaluating a Game-Based CALL Platform

ERIC Educational Resources Information Center

Ní Chiaráin, Neasa; Ní Chasaide, Ailbhe

2017-01-01

Game-based Computer-Assisted Language Learning (CALL) is an area that currently warrants attention, as task-based, interactive, multimodal games increasingly show promise for language learning. This area is inherently multidisciplinary--theories from second language acquisition, games, and psychology must be explored and relevant concepts from…

Real-time skin feature identification in a time-sequential video stream

NASA Astrophysics Data System (ADS)

Kramberger, Iztok

2005-04-01

Skin color can be an important feature when tracking skin-colored objects. Particularly this is the case for computer-vision-based human-computer interfaces (HCI). Humans have a highly developed feeling of space and, therefore, it is reasonable to support this within intelligent HCI, where the importance of augmented reality can be foreseen. Joining human-like interaction techniques within multimodal HCI could, or will, gain a feature for modern mobile telecommunication devices. On the other hand, real-time processing plays an important role in achieving more natural and physically intuitive ways of human-machine interaction. The main scope of this work is the development of a stereoscopic computer-vision hardware-accelerated framework for real-time skin feature identification in the sense of a single-pass image segmentation process. The hardware-accelerated preprocessing stage is presented with the purpose of color and spatial filtering, where the skin color model within the hue-saturation-value (HSV) color space is given with a polyhedron of threshold values representing the basis of the filter model. An adaptive filter management unit is suggested to achieve better segmentation results. This enables the adoption of filter parameters to the current scene conditions in an adaptive way. Implementation of the suggested hardware structure is given at the level of filed programmable system level integrated circuit (FPSLIC) devices using an embedded microcontroller as their main feature. A stereoscopic clue is achieved using a time-sequential video stream, but this shows no difference for real-time processing requirements in terms of hardware complexity. The experimental results for the hardware-accelerated preprocessing stage are given by efficiency estimation of the presented hardware structure using a simple motion-detection algorithm based on a binary function.

Eyeblink Synchrony in Multimodal Human-Android Interaction.

PubMed

Tatsukawa, Kyohei; Nakano, Tamami; Ishiguro, Hiroshi; Yoshikawa, Yuichiro

2016-12-23

As the result of recent progress in technology of communication robot, robots are becoming an important social partner for humans. Behavioral synchrony is understood as an important factor in establishing good human-robot relationships. In this study, we hypothesized that biasing a human's attitude toward a robot changes the degree of synchrony between human and robot. We first examined whether eyeblinks were synchronized between a human and an android in face-to-face interaction and found that human listeners' eyeblinks were entrained to android speakers' eyeblinks. This eyeblink synchrony disappeared when the android speaker spoke while looking away from the human listeners but was enhanced when the human participants listened to the speaking android while touching the android's hand. These results suggest that eyeblink synchrony reflects a qualitative state in human-robot interactions.

Android Based Behavioral Biometric Authentication via Multi-Modal Fusion

DTIC Science & Technology

2014-06-12

such as the way he or she uses the mouse, or interacts with the Graphical User Interface (GUI) [9]. Described simply, standard biometrics is determined...as a login screen on a standard computer. Active authentication is authentication that occurs dynamically throughout interaction with the device. A...because they are higher level constructs in themselves. The Android framework was specifically used for capturing the multitouch gestures: pinch and zoom

Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains

PubMed Central

Gerdes, Antje B. M.; Wieser, Matthias J.; Alpers, Georg W.

2014-01-01

In everyday life, multiple sensory channels jointly trigger emotional experiences and one channel may alter processing in another channel. For example, seeing an emotional facial expression and hearing the voice’s emotional tone will jointly create the emotional experience. This example, where auditory and visual input is related to social communication, has gained considerable attention by researchers. However, interactions of visual and auditory emotional information are not limited to social communication but can extend to much broader contexts including human, animal, and environmental cues. In this article, we review current research on audiovisual emotion processing beyond face-voice stimuli to develop a broader perspective on multimodal interactions in emotion processing. We argue that current concepts of multimodality should be extended in considering an ecologically valid variety of stimuli in audiovisual emotion processing. Therefore, we provide an overview of studies in which emotional sounds and interactions with complex pictures of scenes were investigated. In addition to behavioral studies, we focus on neuroimaging, electro- and peripher-physiological findings. Furthermore, we integrate these findings and identify similarities or differences. We conclude with suggestions for future research. PMID:25520679

Ghost-in-the-Machine reveals human social signals for human–robot interaction

PubMed Central

Loth, Sebastian; Jettka, Katharina; Giuliani, Manuel; de Ruiter, Jan P.

2015-01-01

We used a new method called “Ghost-in-the-Machine” (GiM) to investigate social interactions with a robotic bartender taking orders for drinks and serving them. Using the GiM paradigm allowed us to identify how human participants recognize the intentions of customers on the basis of the output of the robotic recognizers. Specifically, we measured which recognizer modalities (e.g., speech, the distance to the bar) were relevant at different stages of the interaction. This provided insights into human social behavior necessary for the development of socially competent robots. When initiating the drink-order interaction, the most important recognizers were those based on computer vision. When drink orders were being placed, however, the most important information source was the speech recognition. Interestingly, the participants used only a subset of the available information, focussing only on a few relevant recognizers while ignoring others. This reduced the risk of acting on erroneous sensor data and enabled them to complete service interactions more swiftly than a robot using all available sensor data. We also investigated socially appropriate response strategies. In their responses, the participants preferred to use the same modality as the customer’s requests, e.g., they tended to respond verbally to verbal requests. Also, they added redundancy to their responses, for instance by using echo questions. We argue that incorporating the social strategies discovered with the GiM paradigm in multimodal grammars of human–robot interactions improves the robustness and the ease-of-use of these interactions, and therefore provides a smoother user experience. PMID:26582998

Effects of urea on selectivity and protein-ligand interactions in multimodal cation exchange chromatography.

PubMed

Holstein, Melissa A; Parimal, Siddharth; McCallum, Scott A; Cramer, Steven M

2013-01-08

Nuclear magnetic resonance (NMR) and molecular dynamics (MD) simulations were employed in concert with chromatography to provide insight into the effect of urea on protein-ligand interactions in multimodal (MM) chromatography. Chromatographic experiments with a protein library in ion exchange (IEX) and MM systems indicated that, while urea had a significant effect on protein retention and selectivity for a range of proteins in MM systems, the effects were much less pronounced in IEX. NMR titration experiments carried out with a multimodal ligand, and isotopically enriched human ubiquitin indicated that, while the ligand binding face of ubiquitin remained largely intact in the presence of urea, the strength of binding was decreased. MD simulations were carried out to provide further insight into the effect of urea on MM ligand binding. These results indicated that, while the overall ligand binding face of ubiquitin remained the same, there was a reduction in the occupancy of the MM ligand interaction region along with subtle changes in the residues involved in these interactions. This work demonstrates the effectiveness of urea in enhancing selectivity in MM chromatographic systems and also provides an in-depth analysis of how MM ligand-protein interactions are altered in the presence of this fluid phase modifier.

Beyond image quality: designing engaging interactions with digital products

NASA Astrophysics Data System (ADS)

de Ridder, Huib; Rozendaal, Marco C.

2008-02-01

Ubiquitous computing (or Ambient Intelligence) promises a world in which information is available anytime anywhere and with which humans can interact in a natural, multimodal way. In such world, perceptual image quality remains an important criterion since most information will be displayed visually, but other criteria such as enjoyment, fun, engagement and hedonic quality are emerging. This paper deals with engagement, the intrinsically enjoyable readiness to put more effort into exploring and/or using a product than strictly required, thus attracting and keeping user's attention for a longer period of time. The impact of the experienced richness of an interface, both visually and degree of possible manipulations, was investigated in a series of experiments employing game-like user interfaces. This resulted in the extension of an existing conceptual framework relating engagement to richness by means of two intermediating variables, namely experienced challenge and sense of control. Predictions from this revised framework are evaluated against results of an earlier experiment assessing the ergonomic and hedonic qualities of interactive media. Test material consisted of interactive CD-ROM's containing presentations of three companies for future customers.

Human likeness: cognitive and affective factors affecting adoption of robot-assisted learning systems

NASA Astrophysics Data System (ADS)

Yoo, Hosun; Kwon, Ohbyung; Lee, Namyeon

2016-07-01

With advances in robot technology, interest in robotic e-learning systems has increased. In some laboratories, experiments are being conducted with humanoid robots as artificial tutors because of their likeness to humans, the rich possibilities of using this type of media, and the multimodal interaction capabilities of these robots. The robot-assisted learning system, a special type of e-learning system, aims to increase the learner's concentration, pleasure, and learning performance dramatically. However, very few empirical studies have examined the effect on learning performance of incorporating humanoid robot technology into e-learning systems or people's willingness to accept or adopt robot-assisted learning systems. In particular, human likeness, the essential characteristic of humanoid robots as compared with conventional e-learning systems, has not been discussed in a theoretical context. Hence, the purpose of this study is to propose a theoretical model to explain the process of adoption of robot-assisted learning systems. In the proposed model, human likeness is conceptualized as a combination of media richness, multimodal interaction capabilities, and para-social relationships; these factors are considered as possible determinants of the degree to which human cognition and affection are related to the adoption of robot-assisted learning systems.

The Interactive Origin and the Aesthetic Modelling of Image-Schemas and Primary Metaphors.

PubMed

Martínez, Isabel C; Español, Silvia A; Pérez, Diana I

2018-06-02

According to the theory of conceptual metaphor, image-schemas and primary metaphors are preconceptual structures configured in human cognition, based on sensory-motor environmental activity. Focusing on the way both non-conceptual structures are embedded in early social interaction, we provide empirical evidence for the interactive and intersubjective ontogenesis of image-schemas and primary metaphors. We present the results of a multimodal image-schematic microanalysis of three interactive infant-directed performances (the composition of movement, touch, speech, and vocalization that adults produce for-and-with the infants). The microanalyses show that adults aesthetically highlight the image-schematic structures embedded in the multimodal composition of the performance, and that primary metaphors are also lived as embedded in these inter-enactive experiences. The findings allow corroborating that the psychological domains of cognition and affection are not in rivalry or conflict but rather intertwined in meaningful experiences.

Students' Multimodal Construction of the Work-Energy Concept

NASA Astrophysics Data System (ADS)

Tang, Kok-Sing; Chee Tan, Seng; Yeo, Jennifer

2011-09-01

This article examines the role of multimodalities in representing the concept of work-energy by studying the collaborative discourse of a group of ninth-grade physics students engaging in an inquiry-based instruction. Theorising a scientific concept as a network of meaning relationships across semiotic modalities situated in human activity, this article analyses the students' interactions through their use of natural language, mathematical symbolism, depiction, and gestures, and examines the intertextual meanings made through the integration of these modalities. Results indicate that the thematic integration of multimodalities is both difficult and necessary for students in order to construct a scientific understanding that is congruent with the physics curriculum. More significantly, the difficulties in multimodal integration stem from the subtle differences in the categorical, quantitative, and spatial meanings of the work-energy concept whose contrasts are often not made explicit to the students. The implications of these analyses and findings for science teaching and educational research are discussed.

Construction of Multi-Mode Affective Learning System: Taking Affective Design as an Example

ERIC Educational Resources Information Center

Lin, Hao-Chiang Koong; Su, Sheng-Hsiung; Chao, Ching-Ju; Hsieh, Cheng-Yen; Tsai, Shang-Chin

2016-01-01

This study aims to design a non-simultaneous distance instruction system with affective computing, which integrates interactive agent technology with the curricular instruction of affective design. The research subjects were 78 students, and prototype assessment and final assessment were adopted to assess the interface and usability of the system.…

Eyeblink Synchrony in Multimodal Human-Android Interaction

PubMed Central

Tatsukawa, Kyohei; Nakano, Tamami; Ishiguro, Hiroshi; Yoshikawa, Yuichiro

2016-01-01

As the result of recent progress in technology of communication robot, robots are becoming an important social partner for humans. Behavioral synchrony is understood as an important factor in establishing good human-robot relationships. In this study, we hypothesized that biasing a human’s attitude toward a robot changes the degree of synchrony between human and robot. We first examined whether eyeblinks were synchronized between a human and an android in face-to-face interaction and found that human listeners’ eyeblinks were entrained to android speakers’ eyeblinks. This eyeblink synchrony disappeared when the android speaker spoke while looking away from the human listeners but was enhanced when the human participants listened to the speaking android while touching the android’s hand. These results suggest that eyeblink synchrony reflects a qualitative state in human-robot interactions. PMID:28009014

Tangible interactive system for document browsing and visualisation of multimedia data

NASA Astrophysics Data System (ADS)

Rytsar, Yuriy; Voloshynovskiy, Sviatoslav; Koval, Oleksiy; Deguillaume, Frederic; Topak, Emre; Startchik, Sergei; Pun, Thierry

2006-01-01

In this paper we introduce and develop a framework for document interactive navigation in multimodal databases. First, we analyze the main open issues of existing multimodal interfaces and then discuss two applications that include interaction with documents in several human environments, i.e., the so-called smart rooms. Second, we propose a system set-up dedicated to the efficient navigation in the printed documents. This set-up is based on the fusion of data from several modalities that include images and text. Both modalities can be used as cover data for hidden indexes using data-hiding technologies as well as source data for robust visual hashing. The particularities of the proposed robust visual hashing are described in the paper. Finally, we address two practical applications of smart rooms for tourism and education and demonstrate the advantages of the proposed solution.

Interactivity in Educational Apps for Young Children: A Multimodal Analysis

ERIC Educational Resources Information Center

Blitz-Raith, Alexandra H.; Liu, Jianxin

2017-01-01

Interactivity is an important indicator of an educational app's reception. Since most educational apps are multimodal, it justifies a methodological initiative to understand meaningful involvement of multimodality in enacting and even amplifying interactivity in an educational app. Yet research so far has largely concentrated on algorithm…

Multimodal human communication--targeting facial expressions, speech content and prosody.

PubMed

Regenbogen, Christina; Schneider, Daniel A; Gur, Raquel E; Schneider, Frank; Habel, Ute; Kellermann, Thilo

2012-05-01

Human communication is based on a dynamic information exchange of the communication channels facial expressions, prosody, and speech content. This fMRI study elucidated the impact of multimodal emotion processing and the specific contribution of each channel on behavioral empathy and its prerequisites. Ninety-six video clips displaying actors who told self-related stories were presented to 27 healthy participants. In two conditions, all channels uniformly transported only emotional or neutral information. Three conditions selectively presented two emotional channels and one neutral channel. Subjects indicated the actors' emotional valence and their own while fMRI was recorded. Activation patterns of tri-channel emotional communication reflected multimodal processing and facilitative effects for empathy. Accordingly, subjects' behavioral empathy rates significantly deteriorated once one source was neutral. However, emotionality expressed via two of three channels yielded activation in a network associated with theory-of-mind-processes. This suggested participants' effort to infer mental states of their counterparts and was accompanied by a decline of behavioral empathy, driven by the participants' emotional responses. Channel-specific emotional contributions were present in modality-specific areas. The identification of different network-nodes associated with human interactions constitutes a prerequisite for understanding dynamics that underlie multimodal integration and explain the observed decline in empathy rates. This task might also shed light on behavioral deficits and neural changes that accompany psychiatric diseases. Copyright © 2012 Elsevier Inc. All rights reserved.

Multimodal Word Meaning Induction From Minimal Exposure to Natural Text.

PubMed

Lazaridou, Angeliki; Marelli, Marco; Baroni, Marco

2017-04-01

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models (DSMs) have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a set of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition. Copyright © 2017 Cognitive Science Society, Inc.

Toward in vivo diagnosis of skin cancer using multimode imaging dermoscopy: (II) molecular mapping of highly pigmented lesions

NASA Astrophysics Data System (ADS)

Vasefi, Fartash; MacKinnon, Nicholas; Farkas, Daniel L.

2014-03-01

We have developed a multimode imaging dermoscope that combines polarization and hyperspectral imaging with a computationally rapid analytical model. This approach employs specific spectral ranges of visible and near infrared wavelengths for mapping the distribution of specific skin bio-molecules. This corrects for the melanin-hemoglobin misestimation common to other systems, without resorting to complex and computationally intensive tissue optical models that are prone to inaccuracies due to over-modeling. Various human skin measurements including a melanocytic nevus, and venous occlusion conditions were investigated and compared with other ratiometric spectral imaging approaches. Access to the broad range of hyperspectral data in the visible and near-infrared range allows our algorithm to flexibly use different wavelength ranges for chromophore estimation while minimizing melanin-hemoglobin optical signature cross-talk.

Rehabilitation exoskeletal robotics. The promise of an emerging field.

PubMed

Pons, José L

2010-01-01

Exoskeletons are wearable robots exhibiting a close cognitive and physical interaction with the human user. These are rigid robotic exoskeletal structures that typically operate alongside human limbs. Scientific and technological work on exoskeletons began in the early 1960s but have only recently been applied to rehabilitation and functional substitution in patients suffering from motor disorders. Key topics for further development of exoskeletons in rehabilitation scenarios include the need for robust human-robot multimodal cognitive interaction, safe and dependable physical interaction, true wearability and portability, and user aspects such as acceptance and usability. This discussion provides an overview of these aspects and draws conclusions regarding potential future research directions in robotic exoskeletons.

A hardware and software architecture to deal with multimodal and collaborative interactions in multiuser virtual reality environments

NASA Astrophysics Data System (ADS)

Martin, P.; Tseu, A.; Férey, N.; Touraine, D.; Bourdot, P.

2014-02-01

Most advanced immersive devices provide collaborative environment within several users have their distinct head-tracked stereoscopic point of view. Combining with common used interactive features such as voice and gesture recognition, 3D mouse, haptic feedback, and spatialized audio rendering, these environments should faithfully reproduce a real context. However, even if many studies have been carried out on multimodal systems, we are far to definitively solve the issue of multimodal fusion, which consists in merging multimodal events coming from users and devices, into interpretable commands performed by the application. Multimodality and collaboration was often studied separately, despite of the fact that these two aspects share interesting similarities. We discuss how we address this problem, thought the design and implementation of a supervisor that is able to deal with both multimodal fusion and collaborative aspects. The aim of this supervisor is to ensure the merge of user's input from virtual reality devices in order to control immersive multi-user applications. We deal with this problem according to a practical point of view, because the main requirements of this supervisor was defined according to a industrial task proposed by our automotive partner, that as to be performed with multimodal and collaborative interactions in a co-located multi-user environment. In this task, two co-located workers of a virtual assembly chain has to cooperate to insert a seat into the bodywork of a car, using haptic devices to feel collision and to manipulate objects, combining speech recognition and two hands gesture recognition as multimodal instructions. Besides the architectural aspect of this supervisor, we described how we ensure the modularity of our solution that could apply on different virtual reality platforms, interactive contexts and virtual contents. A virtual context observer included in this supervisor in was especially designed to be independent to the content of the virtual scene of targeted application, and is use to report high-level interactive and collaborative events. This context observer allows the supervisor to merge these interactive and collaborative events, but is also used to deal with new issues coming from our observation of two co-located users in an immersive device performing this assembly task. We highlight the fact that when speech recognition features are provided to the two users, it is required to automatically detect according to the interactive context, whether the vocal instructions must be translated into commands that have to be performed by the machine, or whether they take a part of the natural communication necessary for collaboration. Information coming from this context observer that indicates a user is looking at its collaborator, is important to detect if the user is talking to its partner. Moreover, as the users are physically co-localised and head-tracking is used to provide high fidelity stereoscopic rendering, and natural walking navigation in the virtual scene, we have to deals with collision and screen occlusion between the co-located users in the physical work space. Working area and focus of each user, computed and reported by the context observer is necessary to prevent or avoid these situations.

Interactive multi-spectral analysis of more than one Sonrai village in Niger, West Africa

NASA Technical Reports Server (NTRS)

Reining, P.; Egbert, D. D.

1975-01-01

Use of LANDSAT data and an interaction system is considered for identifying and measuring small scale compact human settlements (villages) for demographic and anthropological studies. Because village components are not uniformly distributed within any one village, they apparently are multimodal, spectrally. Therefore, the functions of location and enumeration are kept separate. Measurement of a known village is compared with CCT response.

Enhanced Particle Swarm Optimization Algorithm: Efficient Training of ReaxFF Reactive Force Fields.

PubMed

Furman, David; Carmeli, Benny; Zeiri, Yehuda; Kosloff, Ronnie

2018-06-12

Particle swarm optimization (PSO) is a powerful metaheuristic population-based global optimization algorithm. However, when it is applied to nonseparable objective functions, its performance on multimodal landscapes is significantly degraded. Here we show that a significant improvement in the search quality and efficiency on multimodal functions can be achieved by enhancing the basic rotation-invariant PSO algorithm with isotropic Gaussian mutation operators. The new algorithm demonstrates superior performance across several nonlinear, multimodal benchmark functions compared with the rotation-invariant PSO algorithm and the well-established simulated annealing and sequential one-parameter parabolic interpolation methods. A search for the optimal set of parameters for the dispersion interaction model in the ReaxFF- lg reactive force field was carried out with respect to accurate DFT-TS calculations. The resulting optimized force field accurately describes the equations of state of several high-energy molecular crystals where such interactions are of crucial importance. The improved algorithm also presents better performance compared to a genetic algorithm optimization method in the optimization of the parameters of a ReaxFF- lg correction model. The computational framework is implemented in a stand-alone C++ code that allows the straightforward development of ReaxFF reactive force fields.

Multimodal visualization interface for data management, self-learning and data presentation.

PubMed

Van Sint Jan, S; Demondion, X; Clapworthy, G; Louryan, S; Rooze, M; Cotten, A; Viceconti, M

2006-10-01

A multimodal visualization software, called the Data Manager (DM), has been developed to increase interdisciplinary communication around the topic of visualization and modeling of various aspects of the human anatomy. Numerous tools used in Radiology are integrated in the interface that runs on standard personal computers. The available tools, combined to hierarchical data management and custom layouts, allow analyzing of medical imaging data using advanced features outside radiological premises (for example, for patient review, conference presentation or tutorial preparation). The system is free, and based on an open-source software development architecture, and therefore updates of the system for custom applications are possible.

A Passive Learning Sensor Architecture for Multimodal Image Labeling: An Application for Social Robots.

PubMed

Gutiérrez, Marco A; Manso, Luis J; Pandya, Harit; Núñez, Pedro

2017-02-11

Object detection and classification have countless applications in human-robot interacting systems. It is a necessary skill for autonomous robots that perform tasks in household scenarios. Despite the great advances in deep learning and computer vision, social robots performing non-trivial tasks usually spend most of their time finding and modeling objects. Working in real scenarios means dealing with constant environment changes and relatively low-quality sensor data due to the distance at which objects are often found. Ambient intelligence systems equipped with different sensors can also benefit from the ability to find objects, enabling them to inform humans about their location. For these applications to succeed, systems need to detect the objects that may potentially contain other objects, working with relatively low-resolution sensor data. A passive learning architecture for sensors has been designed in order to take advantage of multimodal information, obtained using an RGB-D camera and trained semantic language models. The main contribution of the architecture lies in the improvement of the performance of the sensor under conditions of low resolution and high light variations using a combination of image labeling and word semantics. The tests performed on each of the stages of the architecture compare this solution with current research labeling techniques for the application of an autonomous social robot working in an apartment. The results obtained demonstrate that the proposed sensor architecture outperforms state-of-the-art approaches.

Game Design Narrative for Learning: Appropriating Adventure Game Design Narrative Devices and Techniques for the Design of Interactive Learning Environments

ERIC Educational Resources Information Center

Dickey, Michele D.

2006-01-01

The purpose of this conceptual analysis is to investigate how contemporary video and computer games might inform instructional design by looking at how narrative devices and techniques support problem solving within complex, multimodal environments. Specifically, this analysis presents a brief overview of game genres and the role of narrative in…

The integration of audio-tactile information is modulated by multimodal social interaction with physical contact in infancy.

PubMed

Tanaka, Yukari; Kanakogi, Yasuhiro; Kawasaki, Masahiro; Myowa, Masako

2018-04-01

Interaction between caregivers and infants is multimodal in nature. To react interactively and smoothly to such multimodal signals, infants must integrate all these signals. However, few empirical infant studies have investigated how multimodal social interaction with physical contact facilitates multimodal integration, especially regarding audio - tactile (A-T) information. By using electroencephalogram (EEG) and event-related potentials (ERPs), the present study investigated how neural processing involved in A-T integration is modulated by tactile interaction. Seven- to 8-months-old infants heard one pseudoword both whilst being tickled (multimodal 'A-T' condition), and not being tickled (unimodal 'A' condition). Thereafter, their EEG was measured during the perception of the same words. Compared to the A condition, the A-T condition resulted in enhanced ERPs and higher beta-band activity within the left temporal regions, indicating neural processing of A-T integration. Additionally, theta-band activity within the middle frontal region was enhanced, which may reflect enhanced attention to social information. Furthermore, differential ERPs correlated with the degree of engagement in the tickling interaction. We provide neural evidence that the integration of A-T information in infants' brains is facilitated through tactile interaction with others. Such plastic changes in neural processing may promote harmonious social interaction and effective learning in infancy. Copyright © 2017 The Authors. Published by Elsevier Ltd.. All rights reserved.

A strategic map for high-impact virtual experience design

NASA Astrophysics Data System (ADS)

Faste, Haakon; Bergamasco, Massimo

2009-02-01

We have employed methodologies of human centered design to inspire and guide the engineering of a definitive low-cost aesthetic multimodal experience intended to stimulate cultural growth. Using a combination of design research, trend analysis and the programming of immersive virtual 3D worlds, over 250 innovative concepts have been brainstormed, prototyped, evaluated and refined. These concepts have been used to create a strategic map for the development of highimpact virtual art experiences, the most promising of which have been incorporated into a multimodal environment programmed in the online interactive 3D platform XVR. A group of test users have evaluated the experience as it has evolved, using a multimodal interface with stereo vision, 3D audio and haptic feedback. This paper discusses the process, content, results, and impact on our engineering laboratory that this research has produced.

A Multimodal Emotion Detection System during Human-Robot Interaction

PubMed Central

Alonso-Martín, Fernando; Malfaz, María; Sequeira, João; Gorostiza, Javier F.; Salichs, Miguel A.

2013-01-01

In this paper, a multimodal user-emotion detection system for social robots is presented. This system is intended to be used during human–robot interaction, and it is integrated as part of the overall interaction system of the robot: the Robotics Dialog System (RDS). Two modes are used to detect emotions: the voice and face expression analysis. In order to analyze the voice of the user, a new component has been developed: Gender and Emotion Voice Analysis (GEVA), which is written using the Chuck language. For emotion detection in facial expressions, the system, Gender and Emotion Facial Analysis (GEFA), has been also developed. This last system integrates two third-party solutions: Sophisticated High-speed Object Recognition Engine (SHORE) and Computer Expression Recognition Toolbox (CERT). Once these new components (GEVA and GEFA) give their results, a decision rule is applied in order to combine the information given by both of them. The result of this rule, the detected emotion, is integrated into the dialog system through communicative acts. Hence, each communicative act gives, among other things, the detected emotion of the user to the RDS so it can adapt its strategy in order to get a greater satisfaction degree during the human–robot dialog. Each of the new components, GEVA and GEFA, can also be used individually. Moreover, they are integrated with the robotic control platform ROS (Robot Operating System). Several experiments with real users were performed to determine the accuracy of each component and to set the final decision rule. The results obtained from applying this decision rule in these experiments show a high success rate in automatic user emotion recognition, improving the results given by the two information channels (audio and visual) separately. PMID:24240598

Sharing a Multimodal Corpus to Study Webcam-Mediated Language Teaching

ERIC Educational Resources Information Center

Guichon, Nicolas

2017-01-01

This article proposes a methodology to create a multimodal corpus that can be shared with a group of researchers in order to analyze synchronous online pedagogical interactions. Epistemological aspects involved in studying online interactions from a multimodal and semiotic perspective are addressed. Then, issues and challenges raised by corpus…

Time-dependent, multimode interaction analysis of the gyroklystron amplifier

DOE Office of Scientific and Technical Information (OSTI.GOV)

Swati, M. V., E-mail: swati.mv.ece10@iitbhu.ac.in; Chauhan, M. S.; Jain, P. K.

2016-08-15

In this paper, a time-dependent multimode nonlinear analysis for the gyroklystron amplifier has been developed by extending the analysis of gyrotron oscillators by employing the self-consistent approach. The nonlinear analysis developed here has been validated by taking into account the reported experimental results for a 32.3 GHz, three cavity, second harmonic gyroklystron operating in the TE{sub 02} mode. The analysis has been used to estimate the temporal RF growth in the operating mode as well as the nearby competing modes. Device gain and bandwidth have been computed for different drive powers and frequencies. The effect of various beam parameters, such asmore » beam voltage, beam current, and pitch factor, has also been studied. The computational results have estimated the gyroklystron saturated RF power ∼319 kW at 32.3 GHz with efficiency ∼23% and gain ∼26.3 dB with device bandwidth ∼0.027% (8 MHz) for a 70 kV, 20 A electron beam. The computed results are found to be in agreement with the experimental values within 10%.« less

The Effects of Multimodal Mobile Communications on Cooperative Team Interactions Executing Distributed Tasks

DTIC Science & Technology

2013-07-01

AFRL-RH-WP-TP-2013-0046 The Effects of Multimodal Mobile Communications on Cooperative Team Interactions Executing Distributed Tasks Gregory...3. DATES COVERED (From - To) 31-07-13 Interim 01 August 2011 – 01 August 2013 4. TITLE AND SUBTITLE The Effects of Multimodal Mobile... multimodal communication capabilities can con- tribute to the effectiveness and efficiency of real-time, task outcome and per- formance. In this paper, we

Visually induced plasticity of auditory spatial perception in macaques.

PubMed

Woods, Timothy M; Recanzone, Gregg H

2004-09-07

When experiencing spatially disparate visual and auditory stimuli, a common percept is that the sound originates from the location of the visual stimulus, an illusion known as the ventriloquism effect. This illusion can persist for tens of minutes, a phenomenon termed the ventriloquism aftereffect. The underlying neuronal mechanisms of this rapidly induced plasticity remain unclear; indeed, it remains untested whether similar multimodal interactions occur in other species. We therefore tested whether macaque monkeys experience the ventriloquism aftereffect similar to the way humans do. The ability of two monkeys to determine which side of the midline a sound was presented from was tested before and after a period of 20-60 min in which the monkeys experienced either spatially identical or spatially disparate auditory and visual stimuli. In agreement with human studies, the monkeys did experience a shift in their auditory spatial perception in the direction of the spatially disparate visual stimulus, and the aftereffect did not transfer across sounds that differed in frequency by two octaves. These results show that macaque monkeys experience the ventriloquism aftereffect similar to the way humans do in all tested respects, indicating that these multimodal interactions are a basic phenomenon of the central nervous system.

Human perceptual deficits as factors in computer interface test and evaluation

DOE Office of Scientific and Technical Information (OSTI.GOV)

Bowser, S.E.

1992-06-01

Issues related to testing and evaluating human computer interfaces are usually based on the machine rather than on the human portion of the computer interface. Perceptual characteristics of the expected user are rarely investigated, and interface designers ignore known population perceptual limitations. For these reasons, environmental impacts on the equipment will more likely be defined than will user perceptual characteristics. The investigation of user population characteristics is most often directed toward intellectual abilities and anthropometry. This problem is compounded by the fact that some deficits capabilities tend to be found in higher-than-overall population distribution in some user groups. The testmore » and evaluation community can address the issue from two primary aspects. First, assessing user characteristics should be extended to include tests of perceptual capability. Secondly, interface designs should use multimode information coding.« less

Mothers' multimodal information processing is modulated by multimodal interactions with their infants.

PubMed

Tanaka, Yukari; Fukushima, Hirokata; Okanoya, Kazuo; Myowa-Yamakoshi, Masako

2014-10-17

Social learning in infancy is known to be facilitated by multimodal (e.g., visual, tactile, and verbal) cues provided by caregivers. In parallel with infants' development, recent research has revealed that maternal neural activity is altered through interaction with infants, for instance, to be sensitive to infant-directed speech (IDS). The present study investigated the effect of mother- infant multimodal interaction on maternal neural activity. Event-related potentials (ERPs) of mothers were compared to non-mothers during perception of tactile-related words primed by tactile cues. Only mothers showed ERP modulation when tactile cues were incongruent with the subsequent words, and only when the words were delivered with IDS prosody. Furthermore, the frequency of mothers' use of those words was correlated with the magnitude of ERP differentiation between congruent and incongruent stimuli presentations. These results suggest that mother-infant daily interactions enhance multimodal integration of the maternal brain in parenting contexts.

Mothers' multimodal information processing is modulated by multimodal interactions with their infants

PubMed Central

Tanaka, Yukari; Fukushima, Hirokata; Okanoya, Kazuo; Myowa-Yamakoshi, Masako

2014-01-01

Social learning in infancy is known to be facilitated by multimodal (e.g., visual, tactile, and verbal) cues provided by caregivers. In parallel with infants' development, recent research has revealed that maternal neural activity is altered through interaction with infants, for instance, to be sensitive to infant-directed speech (IDS). The present study investigated the effect of mother- infant multimodal interaction on maternal neural activity. Event-related potentials (ERPs) of mothers were compared to non-mothers during perception of tactile-related words primed by tactile cues. Only mothers showed ERP modulation when tactile cues were incongruent with the subsequent words, and only when the words were delivered with IDS prosody. Furthermore, the frequency of mothers' use of those words was correlated with the magnitude of ERP differentiation between congruent and incongruent stimuli presentations. These results suggest that mother-infant daily interactions enhance multimodal integration of the maternal brain in parenting contexts. PMID:25322936

The two types of stethoscope systems for respiration system diagnostics of the human body

NASA Astrophysics Data System (ADS)

Abashkin, Vladimir; Achimova, Elena

2003-12-01

An acoustic multimode fiber optic sensors for medical diagnostics based upon the shutter principle has been elaborated with semiconductor laser diode as light source. The construction and the method of component preparation are described. Other type of stethoscope is electrical one. Both stethoscopes are four channels. The kinetics and dynamic vibrations and sounds of the human body can be detected, acquired and then processing by personal computer for medical diagnostics.

Research reports: 1990 NASA/ASEE Summer Faculty Fellowship Program

NASA Technical Reports Server (NTRS)

Anderson, Loren A. (Editor); Beymer, Mark A. (Editor)

1990-01-01

A collection of technical reports on research conducted by the participants in this program is presented. The topics covered include: human-computer interface software, multimode fiber optic communication links, electrochemical impedance spectroscopy, rocket-triggered lightning, robotics, a flammability study of thin polymeric film materials, a vortex shedding flowmeter, modeling of flow systems, monomethyl hydrazine vapor detection, a rocket noise filter system using digital filters, computer programs, lower body negative pressure, closed ecological systems, and others. Several reports with respect to space shuttle orbiters are presented.

Techniques for efficient, real-time, 3D visualization of multi-modality cardiac data using consumer graphics hardware.

PubMed

Levin, David; Aladl, Usaf; Germano, Guido; Slomka, Piotr

2005-09-01

We exploit consumer graphics hardware to perform real-time processing and visualization of high-resolution, 4D cardiac data. We have implemented real-time, realistic volume rendering, interactive 4D motion segmentation of cardiac data, visualization of multi-modality cardiac data and 3D display of multiple series cardiac MRI. We show that an ATI Radeon 9700 Pro can render a 512x512x128 cardiac Computed Tomography (CT) study at 0.9 to 60 frames per second (fps) depending on rendering parameters and that 4D motion based segmentation can be performed in real-time. We conclude that real-time rendering and processing of cardiac data can be implemented on consumer graphics cards.

Multimodality Inferring of Human Cognitive States Based on Integration of Neuro-Fuzzy Network and Information Fusion Techniques

NASA Astrophysics Data System (ADS)

Yang, G.; Lin, Y.; Bhattacharya, P.

2007-12-01

To achieve an effective and safe operation on the machine system where the human interacts with the machine mutually, there is a need for the machine to understand the human state, especially cognitive state, when the human's operation task demands an intensive cognitive activity. Due to a well-known fact with the human being, a highly uncertain cognitive state and behavior as well as expressions or cues, the recent trend to infer the human state is to consider multimodality features of the human operator. In this paper, we present a method for multimodality inferring of human cognitive states by integrating neuro-fuzzy network and information fusion techniques. To demonstrate the effectiveness of this method, we take the driver fatigue detection as an example. The proposed method has, in particular, the following new features. First, human expressions are classified into four categories: (i) casual or contextual feature, (ii) contact feature, (iii) contactless feature, and (iv) performance feature. Second, the fuzzy neural network technique, in particular Takagi-Sugeno-Kang (TSK) model, is employed to cope with uncertain behaviors. Third, the sensor fusion technique, in particular ordered weighted aggregation (OWA), is integrated with the TSK model in such a way that cues are taken as inputs to the TSK model, and then the outputs of the TSK are fused by the OWA which gives outputs corresponding to particular cognitive states under interest (e.g., fatigue). We call this method TSK-OWA. Validation of the TSK-OWA, performed in the Northeastern University vehicle drive simulator, has shown that the proposed method is promising to be a general tool for human cognitive state inferring and a special tool for the driver fatigue detection.

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications.

PubMed

Corneanu, Ciprian Adrian; Simon, Marc Oliu; Cohn, Jeffrey F; Guerrero, Sergio Escalera

2016-08-01

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

Cognitive engineering models: A prerequisite to the design of human-computer interaction in complex dynamic systems

NASA Technical Reports Server (NTRS)

Mitchell, Christine M.

1993-01-01

This chapter examines a class of human-computer interaction applications, specifically the design of human-computer interaction for the operators of complex systems. Such systems include space systems (e.g., manned systems such as the Shuttle or space station, and unmanned systems such as NASA scientific satellites), aviation systems (e.g., the flight deck of 'glass cockpit' airplanes or air traffic control) and industrial systems (e.g., power plants, telephone networks, and sophisticated, e.g., 'lights out,' manufacturing facilities). The main body of human-computer interaction (HCI) research complements but does not directly address the primary issues involved in human-computer interaction design for operators of complex systems. Interfaces to complex systems are somewhat special. The 'user' in such systems - i.e., the human operator responsible for safe and effective system operation - is highly skilled, someone who in human-machine systems engineering is sometimes characterized as 'well trained, well motivated'. The 'job' or task context is paramount and, thus, human-computer interaction is subordinate to human job interaction. The design of human interaction with complex systems, i.e., the design of human job interaction, is sometimes called cognitive engineering.

Adhesion of multimode adhesives to enamel and dentin after one year of water storage.

PubMed

Vermelho, Paulo Moreira; Reis, André Figueiredo; Ambrosano, Glaucia Maria Bovi; Giannini, Marcelo

2017-06-01

This study aimed to evaluate the ultramorphological characteristics of tooth-resin interfaces and the bond strength (BS) of multimode adhesive systems to enamel and dentin. Multimode adhesives (Scotchbond Universal (SBU) and All-Bond Universal) were tested in both self-etch and etch-and-rinse modes and compared to control groups (Optibond FL and Clearfil SE Bond (CSB)). Adhesives were applied to human molars and composite blocks were incrementally built up. Teeth were sectioned to obtain specimens for microtensile BS and TEM analysis. Specimens were tested after storage for either 24 h or 1 year. SEM analyses were performed to classify the failure pattern of beam specimens after BS testing. Etching increased the enamel BS of multimode adhesives; however, BS decreased after storage for 1 year. No significant differences in dentin BS were noted between multimode and control in either evaluation period. Storage for 1 year only reduced the dentin BS for SBU in self-etch mode. TEM analysis identified hybridization and interaction zones in dentin and enamel for all adhesives. Silver impregnation was detected on dentin-resin interfaces after storage of specimens for 1 year only with the SBU and CSB. Storage for 1 year reduced enamel BS when adhesives are applied on etched surface; however, BS of multimode adhesives did not differ from those of the control group. In dentin, no significant difference was noted between the multimode and control group adhesives, regardless of etching mode. In general, multimode adhesives showed similar behavior when compared to traditional adhesive techniques. Multimode adhesives are one-step self-etching adhesives that can also be used after enamel/dentin phosphoric acid etching, but each product may work better in specific conditions.

Fostering Students' Science Inquiry through App Affordances of Multimodality, Collaboration, Interactivity, and Connectivity

ERIC Educational Resources Information Center

Beach, Richard; O'Brien, David

2015-01-01

This study examined 6th graders' use of the VoiceThread app as part of a science inquiry project on photosynthesis and carbon dioxide emissions in terms of their ability to engage in causal reasoning and their use of the affordances of multimodality, collaboration, interactivity, and connectivity. Students employed multimodal production using…

Using "human state aware" robots to enhance physical human-robot interaction in a cooperative scenario.

PubMed

Guerrero, Carlos Rodriguez; Fraile Marinero, Juan Carlos; Turiel, Javier Perez; Muñoz, Victor

2013-11-01

Human motor performance, speed and variability are highly susceptible to emotional states. This paper reviews the impact of the emotions on the motor control performance, and studies the possibility of improving the perceived skill/challenge relation on a multimodal neural rehabilitation scenario, by means of a biocybernetic controller that modulates the assistance provided by a haptic controlled robot in reaction to undesirable physical and mental states. Results from psychophysiological, performance and self assessment data for closed loop experiments in contrast with their open loop counterparts, suggest that the proposed method had a positive impact on the overall challenge/skill relation leading to an enhanced physical human-robot interaction experience. Copyright © 2013 Elsevier Ireland Ltd. All rights reserved.

Metaphors for the Nature of Human-Computer Interaction in an Empowering Environment: Interaction Style Influences the Manner of Human Accomplishment.

ERIC Educational Resources Information Center

Weller, Herman G.; Hartson, H. Rex

1992-01-01

Describes human-computer interface needs for empowering environments in computer usage in which the machine handles the routine mechanics of problem solving while the user concentrates on its higher order meanings. A closed-loop model of interaction is described, interface as illusion is discussed, and metaphors for human-computer interaction are…

Multi-modal gesture recognition using integrated model of motion, audio and video

NASA Astrophysics Data System (ADS)

Goutsu, Yusuke; Kobayashi, Takaki; Obara, Junya; Kusajima, Ikuo; Takeichi, Kazunari; Takano, Wataru; Nakamura, Yoshihiko

2015-07-01

Gesture recognition is used in many practical applications such as human-robot interaction, medical rehabilitation and sign language. With increasing motion sensor development, multiple data sources have become available, which leads to the rise of multi-modal gesture recognition. Since our previous approach to gesture recognition depends on a unimodal system, it is difficult to classify similar motion patterns. In order to solve this problem, a novel approach which integrates motion, audio and video models is proposed by using dataset captured by Kinect. The proposed system can recognize observed gestures by using three models. Recognition results of three models are integrated by using the proposed framework and the output becomes the final result. The motion and audio models are learned by using Hidden Markov Model. Random Forest which is the video classifier is used to learn the video model. In the experiments to test the performances of the proposed system, the motion and audio models most suitable for gesture recognition are chosen by varying feature vectors and learning methods. Additionally, the unimodal and multi-modal models are compared with respect to recognition accuracy. All the experiments are conducted on dataset provided by the competition organizer of MMGRC, which is a workshop for Multi-Modal Gesture Recognition Challenge. The comparison results show that the multi-modal model composed of three models scores the highest recognition rate. This improvement of recognition accuracy means that the complementary relationship among three models improves the accuracy of gesture recognition. The proposed system provides the application technology to understand human actions of daily life more precisely.

Motion-sensor fusion-based gesture recognition and its VLSI architecture design for mobile devices

NASA Astrophysics Data System (ADS)

Zhu, Wenping; Liu, Leibo; Yin, Shouyi; Hu, Siqi; Tang, Eugene Y.; Wei, Shaojun

2014-05-01

With the rapid proliferation of smartphones and tablets, various embedded sensors are incorporated into these platforms to enable multimodal human-computer interfaces. Gesture recognition, as an intuitive interaction approach, has been extensively explored in the mobile computing community. However, most gesture recognition implementations by now are all user-dependent and only rely on accelerometer. In order to achieve competitive accuracy, users are required to hold the devices in predefined manner during the operation. In this paper, a high-accuracy human gesture recognition system is proposed based on multiple motion sensor fusion. Furthermore, to reduce the energy overhead resulted from frequent sensor sampling and data processing, a high energy-efficient VLSI architecture implemented on a Xilinx Virtex-5 FPGA board is also proposed. Compared with the pure software implementation, approximately 45 times speed-up is achieved while operating at 20 MHz. The experiments show that the average accuracy for 10 gestures achieves 93.98% for user-independent case and 96.14% for user-dependent case when subjects hold the device randomly during completing the specified gestures. Although a few percent lower than the conventional best result, it still provides competitive accuracy acceptable for practical usage. Most importantly, the proposed system allows users to hold the device randomly during operating the predefined gestures, which substantially enhances the user experience.

How to decompose arbitrary continuous-variable quantum operations.

PubMed

Sefi, Seckin; van Loock, Peter

2011-10-21

We present a general, systematic, and efficient method for decomposing any given exponential operator of bosonic mode operators, describing an arbitrary multimode Hamiltonian evolution, into a set of universal unitary gates. Although our approach is mainly oriented towards continuous-variable quantum computation, it may be used more generally whenever quantum states are to be transformed deterministically, e.g., in quantum control, discrete-variable quantum computation, or Hamiltonian simulation. We illustrate our scheme by presenting decompositions for various nonlinear Hamiltonians including quartic Kerr interactions. Finally, we conclude with two potential experiments utilizing offline-prepared optical cubic states and homodyne detections, in which quantum information is processed optically or in an atomic memory using quadratic light-atom interactions. © 2011 American Physical Society

Multi-mode horn antenna simulation

NASA Technical Reports Server (NTRS)

Dod, L. R.; Wolf, J. D.

1980-01-01

Radiation patterns were computed for a circular multimode horn antenna using waveguide electric field radiation expressions. The circular multimode horn was considered as a possible reflector feed antenna for the Large Antenna Multifrequency Microwave Radiometer (LAMMR). This horn antenna uses a summation of the TE sub 11 deg and TM sub 11 deg modes to generate far field primary radiation patterns with equal E and H plane beamwidths and low sidelobes. A computer program for the radiation field expressions using the summation of waveguide radiation modes is described. The sensitivity of the multimode horn antenna radiation patterns to phase variations between the two modes is given. Sample radiation pattern calculations for a reflector feed horn for LAMMR are shown. The multimode horn antenna provides a low noise feed suitable for radiometric applications.

Collaborative interactive visualization: exploratory concept

NASA Astrophysics Data System (ADS)

Mokhtari, Marielle; Lavigne, Valérie; Drolet, Frédéric

2015-05-01

Dealing with an ever increasing amount of data is a challenge that military intelligence analysts or team of analysts face day to day. Increased individual and collective comprehension goes through collaboration between people. Better is the collaboration, better will be the comprehension. Nowadays, various technologies support and enhance collaboration by allowing people to connect and collaborate in settings as varied as across mobile devices, over networked computers, display walls, tabletop surfaces, to name just a few. A powerful collaboration system includes traditional and multimodal visualization features to achieve effective human communication. Interactive visualization strengthens collaboration because this approach is conducive to incrementally building a mental assessment of the data meaning. The purpose of this paper is to present an overview of the envisioned collaboration architecture and the interactive visualization concepts underlying the Sensemaking Support System prototype developed to support analysts in the context of the Joint Intelligence Collection and Analysis Capability project at DRDC Valcartier. It presents the current version of the architecture, discusses future capabilities to help analyst(s) in the accomplishment of their tasks and finally recommends collaboration and visualization technologies allowing to go a step further both as individual and as a team.

Advanced Interactive Display Formats for Terminal Area Traffic Control

NASA Technical Reports Server (NTRS)

Grunwald, Arthur J.; Shaviv, G. E.

1999-01-01

This research project deals with an on-line dynamic method for automated viewing parameter management in perspective displays. Perspective images are optimized such that a human observer will perceive relevant spatial geometrical features with minimal errors. In order to compute the errors at which observers reconstruct spatial features from perspective images, a visual spatial-perception model was formulated. The model was employed as the basis of an optimization scheme aimed at seeking the optimal projection parameter setting. These ideas are implemented in the context of an air traffic control (ATC) application. A concept, referred to as an active display system, was developed. This system uses heuristic rules to identify relevant geometrical features of the three-dimensional air traffic situation. Agile, on-line optimization was achieved by a specially developed and custom-tailored genetic algorithm (GA), which was to deal with the multi-modal characteristics of the objective function and exploit its time-evolving nature.

Mobile Technologies for the Discovery, Analysis, and Engineering of the Global Microbiome.

PubMed

Ballard, Zachary S; Brown, Calvin; Ozcan, Aydogan

2018-04-24

The microbiome has been heralded as a gauge of and contributor to both human health and environmental conditions. Current challenges in probing, engineering, and harnessing the microbiome stem from its microscopic and nanoscopic nature, diversity and complexity of interactions among its members and hosts, as well as the spatiotemporal sampling and in situ measurement limitations induced by the restricted capabilities and norm of existing technologies, leaving some of the constituents of the microbiome unknown. To facilitate significant progress in the microbiome field, deeper understanding of the constituents' individual behavior, interactions with others, and biodiversity are needed. Also crucial is the generation of multimodal data from a variety of subjects and environments over time. Mobile imaging and sensing technologies, particularly through smartphone-based platforms, can potentially meet some of these needs in field-portable, cost-effective, and massively scalable manners by circumventing the need for bulky, expensive instrumentation. In this Perspective, we outline how mobile sensing and imaging technologies could lead the way to unprecedented insight into the microbiome, potentially shedding light on various microbiome-related mysteries of today, including the composition and function of human, animal, plant, and environmental microbiomes. Finally, we conclude with a look at the future, propose a computational microbiome engineering and optimization framework, and discuss its potential impact and applications.

All the School's a Stage: A Multimodal Interaction Analysis of a School Administrator's Literate Life as Dramaturgical Metaphor

ERIC Educational Resources Information Center

Tomlin, Dru DeLance

2013-01-01

In "Images of Leadership" (1991), Bolman and Deal identified four "frames" that school administrators use when making decisions: structural, symbolic, human resource and political. They discovered that the latter two frames, which focus on relationships, partnerships, and communication, were most frequently identified as…

Is Human-Computer Interaction Social or Parasocial?

ERIC Educational Resources Information Center

Sundar, S. Shyam

Conducted in the attribution-research paradigm of social psychology, a study examined whether human-computer interaction is fundamentally social (as in human-human interaction) or parasocial (as in human-television interaction). All 30 subjects (drawn from an undergraduate class on communication) were exposed to an identical interaction with…

Fourier domain asymmetric cryptosystem for privacy protected multimodal biometric security

NASA Astrophysics Data System (ADS)

Choudhury, Debesh

2016-04-01

We propose a Fourier domain asymmetric cryptosystem for multimodal biometric security. One modality of biometrics (such as face) is used as the plaintext, which is encrypted by another modality of biometrics (such as fingerprint). A private key is synthesized from the encrypted biometric signature by complex spatial Fourier processing. The encrypted biometric signature is further encrypted by other biometric modalities, and the corresponding private keys are synthesized. The resulting biometric signature is privacy protected since the encryption keys are provided by the human, and hence those are private keys. Moreover, the decryption keys are synthesized using those private encryption keys. The encrypted signatures are decrypted using the synthesized private keys and inverse complex spatial Fourier processing. Computer simulations demonstrate the feasibility of the technique proposed.

New Sociotechnical Insights in Interaction Design

NASA Astrophysics Data System (ADS)

Abdelnour-Nocera, José; Mørch, Anders I.

New challenges are facing interaction design. On one hand because of advances in technology - pervasive, ubiquitous, multimodal and adaptive computing - are changing the nature of interaction. On the other, web 2.0, massive multiplayer games and collaboration software extends the boundaries of HCI to deal with interaction in settings of remote communication and collaboration. The aim of this workshop is to provide a forum for HCI practitioners and researchers interested in knowledge from the social sciences to discuss how sociotechnical insights can be used to inform interaction design, and more generally how social science methods and theories can help to enrich the conceptual framework of systems development and participatory design. Position papers submissions are invited to address key aspects of current research and practical case studies.

Multimodality and interactivity: connecting properties of serious games with educational outcomes.

PubMed

Ritterfeld, Ute; Shen, Cuihua; Wang, Hua; Nocera, Luciano; Wong, Wee Ling

2009-12-01

Serious games have become an important genre of digital media and are often acclaimed for their potential to enhance deeper learning because of their unique technological properties. Yet the discourse has largely remained at a conceptual level. For an empirical evaluation of educational games, extra effort is needed to separate intertwined and confounding factors in order to manipulate and thus attribute the outcome to one property independent of another. This study represents one of the first attempts to empirically test the educational impact of two important properties of serious games, multimodality and interactivity, through a partial 2 x 3 (interactive, noninteractive by high, moderate, low in multimodality) factorial between-participants follow-up experiment. Results indicate that both multimodality and interactivity contribute to educational outcomes individually. Implications for educational strategies and future research directions are discussed.

Design of a Multi-mode Flight Deck Decision Support System for Airborne Conflict Management

NASA Technical Reports Server (NTRS)

Barhydt, Richard; Krishnamurthy, Karthik

2004-01-01

NASA Langley has developed a multi-mode decision support system for pilots operating in a Distributed Air-Ground Traffic Management (DAG-TM) environment. An Autonomous Operations Planner (AOP) assists pilots in performing separation assurance functions, including conflict detection, prevention, and resolution. Ongoing AOP design has been based on a comprehensive human factors analysis and evaluation results from previous human-in-the-loop experiments with airline pilot test subjects. AOP considers complex flight mode interactions and provides flight guidance to pilots consistent with the current aircraft control state. Pilots communicate goals to AOP by setting system preferences and actively probing potential trajectories for conflicts. To minimize training requirements and improve operational use, AOP design leverages existing alerting philosophies, displays, and crew interfaces common on commercial aircraft. Future work will consider trajectory prediction uncertainties, integration with the TCAS collision avoidance system, and will incorporate enhancements based on an upcoming air-ground coordination experiment.

Pilot-Vehicle Interface

DTIC Science & Technology

1993-11-01

way is to develop a crude but working model of an entire system. The other is by developing a realistic model of the user interface , leaving out most...devices or by incorporating software for a more user -friendly interface . Automation introduces the possibility of making data entry errors. Multimode...across various human- computer interfaces . 127 a Memory: Minimize the amount of information that the user must maintain in short-term memory

Visual exploration and analysis of human-robot interaction rules

NASA Astrophysics Data System (ADS)

Zhang, Hui; Boyles, Michael J.

2013-01-01

We present a novel interaction paradigm for the visual exploration, manipulation and analysis of human-robot interaction (HRI) rules; our development is implemented using a visual programming interface and exploits key techniques drawn from both information visualization and visual data mining to facilitate the interaction design and knowledge discovery process. HRI is often concerned with manipulations of multi-modal signals, events, and commands that form various kinds of interaction rules. Depicting, manipulating and sharing such design-level information is a compelling challenge. Furthermore, the closed loop between HRI programming and knowledge discovery from empirical data is a relatively long cycle. This, in turn, makes design-level verification nearly impossible to perform in an earlier phase. In our work, we exploit a drag-and-drop user interface and visual languages to support depicting responsive behaviors from social participants when they interact with their partners. For our principal test case of gaze-contingent HRI interfaces, this permits us to program and debug the robots' responsive behaviors through a graphical data-flow chart editor. We exploit additional program manipulation interfaces to provide still further improvement to our programming experience: by simulating the interaction dynamics between a human and a robot behavior model, we allow the researchers to generate, trace and study the perception-action dynamics with a social interaction simulation to verify and refine their designs. Finally, we extend our visual manipulation environment with a visual data-mining tool that allows the user to investigate interesting phenomena such as joint attention and sequential behavioral patterns from multiple multi-modal data streams. We have created instances of HRI interfaces to evaluate and refine our development paradigm. As far as we are aware, this paper reports the first program manipulation paradigm that integrates visual programming interfaces, information visualization, and visual data mining methods to facilitate designing, comprehending, and evaluating HRI interfaces.

Bird impact analysis package for turbine engine fan blades

NASA Technical Reports Server (NTRS)

Hirschbein, M. S.

1982-01-01

A computer program has been developed to analyze the gross structural response of turbine engine fan blades subjected to bird strikes. The program couples a NASTRAN finite element model and modal analysis of a fan blade with a multi-mode bird impact analysis computer program. The impact analysis uses the NASTRAN blade model and a fluid jet model of the bird to interactively calculate blade loading during a bird strike event. The analysis package is computationaly efficient, easy to use and provides a comprehensive history of the gross structual blade response. Example cases are presented for a representative fan blade.

On the Rhetorical Contract in Human-Computer Interaction.

ERIC Educational Resources Information Center

Wenger, Michael J.

1991-01-01

An exploration of the rhetorical contract--i.e., the expectations for appropriate interaction--as it develops in human-computer interaction revealed that direct manipulation interfaces were more likely to establish social expectations. Study results suggest that the social nature of human-computer interactions can be examined with reference to the…

Vanderbilt University Institute of Imaging Science Center for Computational Imaging XNAT: A multimodal data archive and processing environment.

PubMed

Harrigan, Robert L; Yvernault, Benjamin C; Boyd, Brian D; Damon, Stephen M; Gibney, Kyla David; Conrad, Benjamin N; Phillips, Nicholas S; Rogers, Baxter P; Gao, Yurui; Landman, Bennett A

2016-01-01

The Vanderbilt University Institute for Imaging Science (VUIIS) Center for Computational Imaging (CCI) has developed a database built on XNAT housing over a quarter of a million scans. The database provides framework for (1) rapid prototyping, (2) large scale batch processing of images and (3) scalable project management. The system uses the web-based interfaces of XNAT and REDCap to allow for graphical interaction. A python middleware layer, the Distributed Automation for XNAT (DAX) package, distributes computation across the Vanderbilt Advanced Computing Center for Research and Education high performance computing center. All software are made available in open source for use in combining portable batch scripting (PBS) grids and XNAT servers. Copyright © 2015 Elsevier Inc. All rights reserved.

Virtual workstation - A multimodal, stereoscopic display environment

NASA Astrophysics Data System (ADS)

Fisher, S. S.; McGreevy, M.; Humphries, J.; Robinett, W.

1987-01-01

A head-mounted, wide-angle, stereoscopic display system controlled by operator position, voice and gesture has been developed for use in a multipurpose interface environment. The system provides a multisensory, interactive display environment in which a user can virtually explore a 360-degree synthesized or remotely sensed environment and can viscerally interact with its components. Primary applications of the system are in telerobotics, management of large-scale integrated information systems, and human factors research. System configuration, application scenarios, and research directions are described.

Multimodal Strategies Allowing Corrective Feedback to Be Softened during Webconferencing-Supported Interactions

ERIC Educational Resources Information Center

Wigham, Ciara R.; Vidal, Julie

2016-01-01

This paper focuses on corrective feedback and examines how trainee-teachers use different semiotic resources to soften feedback sequences during synchronous online interactions. The ISMAEL corpus of webconferencing-supported L2 interactions in French provided data for this qualitative study. Using multimodal transcriptions, the analysis describes…

Good Student/Bad Student: Situated Identities in the Figured Worlds of School and Creative Multimodal Production

ERIC Educational Resources Information Center

Jocius, Robin

2017-01-01

This study situates young adolescents' multimodal composing practices within two figured worlds--school and creative multimodal production. In a microanalysis of two focal students' multimodal processes and products, I trace how pedagogical, interactional, and semiotic resources both reified and challenged students' developing identities as…

Occupational stress in human computer interaction.

PubMed

Smith, M J; Conway, F T; Karsh, B T

1999-04-01

There have been a variety of research approaches that have examined the stress issues related to human computer interaction including laboratory studies, cross-sectional surveys, longitudinal case studies and intervention studies. A critical review of these studies indicates that there are important physiological, biochemical, somatic and psychological indicators of stress that are related to work activities where human computer interaction occurs. Many of the stressors of human computer interaction at work are similar to those stressors that have historically been observed in other automated jobs. These include high workload, high work pressure, diminished job control, inadequate employee training to use new technology, monotonous tasks, por supervisory relations, and fear for job security. New stressors have emerged that can be tied primarily to human computer interaction. These include technology breakdowns, technology slowdowns, and electronic performance monitoring. The effects of the stress of human computer interaction in the workplace are increased physiological arousal; somatic complaints, especially of the musculoskeletal system; mood disturbances, particularly anxiety, fear and anger; and diminished quality of working life, such as reduced job satisfaction. Interventions to reduce the stress of computer technology have included improved technology implementation approaches and increased employee participation in implementation. Recommendations for ways to reduce the stress of human computer interaction at work are presented. These include proper ergonomic conditions, increased organizational support, improved job content, proper workload to decrease work pressure, and enhanced opportunities for social support. A model approach to the design of human computer interaction at work that focuses on the system "balance" is proposed.

Virtual personal assistance

NASA Astrophysics Data System (ADS)

Aditya, K.; Biswadeep, G.; Kedar, S.; Sundar, S.

2017-11-01

Human computer communication has growing demand recent days. The new generation of autonomous technology aspires to give computer interfaces emotional states that relate and consider user as well as system environment considerations. In the existing computational model is based an artificial intelligent and externally by multi-modal expression augmented with semi human characteristics. But the main problem with is multi-model expression is that the hardware control given to the Artificial Intelligence (AI) is very limited. So, in our project we are trying to give the Artificial Intelligence (AI) more control on the hardware. There are two main parts such as Speech to Text (STT) and Text to Speech (TTS) engines are used accomplish the requirement. In this work, we are using a raspberry pi 3, a speaker and a mic as hardware and for the programing part, we are using python scripting.

Experimental Tests of Normative Group Influence and Representation Effects in Computer-Mediated Communication: When Interacting Via Computers Differs from Interacting With Computers.

ERIC Educational Resources Information Center

Lee, Eun-Ju; Nass, Clifford

2002-01-01

Presents two experiments to address the questions of if and how normative social influence operates in anonymous computer-mediated communication and human-computer interaction. Finds that the perception of interaction partner (human vs. computer) moderated the group conformity effect such that the undergraduate student subjects expressed greater…

Offspring Generation Method for interactive Genetic Algorithm considering Multimodal Preference

NASA Astrophysics Data System (ADS)

Ito, Fuyuko; Hiroyasu, Tomoyuki; Miki, Mitsunori; Yokouchi, Hisatake

In interactive genetic algorithms (iGAs), computer simulations prepare design candidates that are then evaluated by the user. Therefore, iGA can predict a user's preferences. Conventional iGA problems involve a search for a single optimum solution, and iGA were developed to find this single optimum. On the other hand, our target problems have several peaks in a function and there are small differences among these peaks. For such problems, it is better to show all the peaks to the user. Product recommendation in shopping sites on the web is one example of such problems. Several types of preference trend should be prepared for users in shopping sites. Exploitation and exploration are important mechanisms in GA search. To perform effective exploitation, the offspring generation method (crossover) is very important. Here, we introduced a new offspring generation method for iGA in multimodal problems. In the proposed method, individuals are clustered into subgroups and offspring are generated in each group. The proposed method was applied to an experimental iGA system to examine its effectiveness. In the experimental iGA system, users can decide on preferable t-shirts to buy. The results of the subjective experiment confirmed that the proposed method enables offspring generation with consideration of multimodal preferences, and the proposed mechanism was also shown not to adversely affect the performance of preference prediction.

Computer-Based Multimodal Composing Activities, Self-Revision, and L2 Acquisition through Writing

ERIC Educational Resources Information Center

Dzekoe, Richmond

2017-01-01

This study investigated how 22 advanced-low proficiency ESL students used computer-based multimodal composing activities (CBMCAs) to facilitate self-revision and learn English through academic writing in the USA. The CBMCAs involved a combination of writing, listening, visual analysis, and speaking activities. The research was framed within an…

Multimodal Learning Analytics and Education Data Mining: Using Computational Technologies to Measure Complex Learning Tasks

ERIC Educational Resources Information Center

Blikstein, Paulo; Worsley, Marcelo

2016-01-01

New high-frequency multimodal data collection technologies and machine learning analysis techniques could offer new insights into learning, especially when students have the opportunity to generate unique, personalized artifacts, such as computer programs, robots, and solutions engineering challenges. To date most of the work on learning analytics…

Using multiple metaphors and multimodalities as a semiotic resource when teaching year 2 students computational strategies

NASA Astrophysics Data System (ADS)

Mildenhall, Paula; Sherriff, Barbara

2017-06-01

Recent research indicates that using multimodal learning experiences can be effective in teaching mathematics. Using a social semiotic lens within a participationist framework, this paper reports on a professional learning collaboration with a primary school teacher designed to explore the use of metaphors and modalities in mathematics instruction. This video case study was conducted in a year 2 classroom over two terms, with the focus on building children's understanding of computational strategies. The findings revealed that the teacher was able to successfully plan both multimodal and multiple metaphor learning experiences that acted as semiotic resources to support the children's understanding of abstract mathematics. The study also led to implications for teaching when using multiple metaphors and multimodalities.

Polarization-Sensitive Hyperspectral Imaging in vivo: A Multimode Dermoscope for Skin Analysis

NASA Astrophysics Data System (ADS)

Vasefi, Fartash; MacKinnon, Nicholas; Saager, Rolf B.; Durkin, Anthony J.; Chave, Robert; Lindsley, Erik H.; Farkas, Daniel L.

2014-05-01

Attempts to understand the changes in the structure and physiology of human skin abnormalities by non-invasive optical imaging are aided by spectroscopic methods that quantify, at the molecular level, variations in tissue oxygenation and melanin distribution. However, current commercial and research systems to map hemoglobin and melanin do not correlate well with pathology for pigmented lesions or darker skin. We developed a multimode dermoscope that combines polarization and hyperspectral imaging with an efficient analytical model to map the distribution of specific skin bio-molecules. This corrects for the melanin-hemoglobin misestimation common to other systems, without resorting to complex and computationally intensive tissue optical models. For this system's proof of concept, human skin measurements on melanocytic nevus, vitiligo, and venous occlusion conditions were performed in volunteers. The resulting molecular distribution maps matched physiological and anatomical expectations, confirming a technologic approach that can be applied to next generation dermoscopes and having biological plausibility that is likely to appeal to dermatologists.

A molecular modeling based method to predict elution behavior and binding patches of proteins in multimodal chromatography.

PubMed

Banerjee, Suvrajit; Parimal, Siddharth; Cramer, Steven M

2017-08-18

Multimodal (MM) chromatography provides a powerful means to enhance the selectivity of protein separations by taking advantage of multiple weak interactions that include electrostatic, hydrophobic and van der Waals interactions. In order to increase our understanding of such phenomena, a computationally efficient approach was developed that combines short molecular dynamics simulations and continuum solvent based coarse-grained free energy calculations in order to study the binding of proteins to Self Assembled Monolayers (SAM) presenting MM ligands. Using this method, the free energies of protein-MM SAM binding over a range of incident orientations of the protein can be determined. The resulting free energies were then examined to identify the more "strongly bound" orientations of different proteins with two multimodal surfaces. The overall free energy of protein-MM surface binding was then determined and correlated to retention factors from isocratic chromatography. This correlation, combined with analytical expressions from the literature, was then employed to predict protein gradient elution salt concentrations as well as selectivity reversals with different MM resin systems. Patches on protein surfaces that interacted strongly with MM surfaces were also identified by determining the frequency of heavy atom contacts with the atoms of the MM SAMs. A comparison of these patches to Electrostatic Potential and hydrophobicity maps indicated that while all of these patches contained significant positive charge, only the highest frequency sites also possessed hydrophobicity. The ability to identify key binding patches on proteins may have significant impact on process development for the separation of bioproduct related impurities. Copyright © 2017 Elsevier B.V. All rights reserved.

Human-computer interaction in multitask situations

NASA Technical Reports Server (NTRS)

Rouse, W. B.

1977-01-01

Human-computer interaction in multitask decisionmaking situations is considered, and it is proposed that humans and computers have overlapping responsibilities. Queueing theory is employed to model this dynamic approach to the allocation of responsibility between human and computer. Results of simulation experiments are used to illustrate the effects of several system variables including number of tasks, mean time between arrivals of action-evoking events, human-computer speed mismatch, probability of computer error, probability of human error, and the level of feedback between human and computer. Current experimental efforts are discussed and the practical issues involved in designing human-computer systems for multitask situations are considered.

Prospectus on Multi-Modal Aspects of Human Factors in Transportation

DOT National Transportation Integrated Search

1991-02-01

This prospectus identifies and discusses a series of human factors : issues which are critical to transportation safety and productivity, and : examines the potential benefits that can accrue from taking a multi-modal : approach to human factors rese...

Modeling Human-Computer Decision Making with Covariance Structure Analysis.

ERIC Educational Resources Information Center

Coovert, Michael D.; And Others

Arguing that sufficient theory exists about the interplay between human information processing, computer systems, and the demands of various tasks to construct useful theories of human-computer interaction, this study presents a structural model of human-computer interaction and reports the results of various statistical analyses of this model.…

Enhancing Learning through Human Computer Interaction

ERIC Educational Resources Information Center

McKay, Elspeth, Ed.

2007-01-01

Enhancing Learning Through Human Computer Interaction is an excellent reference source for human computer interaction (HCI) applications and designs. This "Premier Reference Source" provides a complete analysis of online business training programs and e-learning in the higher education sector. It describes a range of positive outcomes for linking…

Parametric Representation of the Speaker's Lips for Multimodal Sign Language and Speech Recognition

NASA Astrophysics Data System (ADS)

Ryumin, D.; Karpov, A. A.

2017-05-01

In this article, we propose a new method for parametric representation of human's lips region. The functional diagram of the method is described and implementation details with the explanation of its key stages and features are given. The results of automatic detection of the regions of interest are illustrated. A speed of the method work using several computers with different performances is reported. This universal method allows applying parametrical representation of the speaker's lipsfor the tasks of biometrics, computer vision, machine learning, and automatic recognition of face, elements of sign languages, and audio-visual speech, including lip-reading.

Reconceptualising Understandings of Texts, Readers and Contexts: One English Teacher's Response to Using Multimodal Texts and Interactive Whiteboards

ERIC Educational Resources Information Center

Kitson, Lisbeth

2011-01-01

The comprehension of multimodal texts is now a key concern with the release of the Australian National Curriculum for English (ACARA, 2010). However, the nature of multimodal texts, the diversity of readers in classrooms, and the complex technological environments through which multimodal texts are mediated, requires English teachers to reconsider…

Development and implementation of an integrated, multi-modality, user-centered interactive dietary change program

PubMed Central

Glasgow, Russell E.; Christiansen, Steve; Smith, K. Sabina; Stevens, Victor J.; Toobert, Deborah J.

2009-01-01

Computer-tailored behavior change programs offer the potential for reaching large populations at a much lower cost than individual or group-based programs. However, few of these programs to date appear to integrate behavioral theory with user choice, or combine different electronic modalities. We describe the development of an integrated CD-ROM and interactive voice response dietary change intervention that combines behavioral problem-solving theory with a high degree of user choice. The program, WISE CHOICES, is being evaluated as part of an ongoing trial. This paper describes the program development, emphasizing how user preferences are accommodated, and presents implementation and user satisfaction data. The program was successfully implemented; the linkages among the central database, the CD-ROM and the automated telephone components were robust, and participants liked the program almost as well as a counselor-delivered dietary change condition. Multi-modality programs that emphasize the strengths of each approach appear to be feasible. Future research is needed to determine the program impact and cost-effectiveness compared with counselor-delivered intervention. PMID:18711204

Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face.

PubMed

Kreifelts, Benjamin; Ethofer, Thomas; Huberle, Elisabeth; Grodd, Wolfgang; Wildgruber, Dirk

2010-07-01

Multimodal integration of nonverbal social signals is essential for successful social interaction. Previous studies have implicated the posterior superior temporal sulcus (pSTS) in the perception of social signals such as nonverbal emotional signals as well as in social cognitive functions like mentalizing/theory of mind. In the present study, we evaluated the relationships between trait emotional intelligence (EI) and fMRI activation patterns in individual subjects during the multimodal perception of nonverbal emotional signals from voice and face. Trait EI was linked to hemodynamic responses in the right pSTS, an area which also exhibits a distinct sensitivity to human voices and faces. Within all other regions known to subserve the perceptual audiovisual integration of human social signals (i.e., amygdala, fusiform gyrus, thalamus), no such linked responses were observed. This functional difference in the network for the audiovisual perception of human social signals indicates a specific contribution of the pSTS as a possible interface between the perception of social information and social cognition. (c) 2009 Wiley-Liss, Inc.

Design of a 3D Navigation Technique Supporting VR Interaction

NASA Astrophysics Data System (ADS)

Boudoin, Pierre; Otmane, Samir; Mallem, Malik

2008-06-01

Multimodality is a powerful paradigm to increase the realness and the easiness of the interaction in Virtual Environments (VEs). In particular, the search for new metaphors and techniques for 3D interaction adapted to the navigation task is an important stage for the realization of future 3D interaction systems that support multimodality, in order to increase efficiency and usability. In this paper we propose a new multimodal 3D interaction model called Fly Over. This model is especially devoted to the navigation task. We present a qualitative comparison between Fly Over and a classical navigation technique called gaze-directed steering. The results from preliminary evaluation on the IBISC semi-immersive Virtual Reality/Augmented Realty EVR@ platform show that Fly Over is a user friendly and efficient navigation technique.

VoxelStats: A MATLAB Package for Multi-Modal Voxel-Wise Brain Image Analysis.

PubMed

Mathotaarachchi, Sulantha; Wang, Seqian; Shin, Monica; Pascoal, Tharick A; Benedet, Andrea L; Kang, Min Su; Beaudry, Thomas; Fonov, Vladimir S; Gauthier, Serge; Labbe, Aurélie; Rosa-Neto, Pedro

2016-01-01

In healthy individuals, behavioral outcomes are highly associated with the variability on brain regional structure or neurochemical phenotypes. Similarly, in the context of neurodegenerative conditions, neuroimaging reveals that cognitive decline is linked to the magnitude of atrophy, neurochemical declines, or concentrations of abnormal protein aggregates across brain regions. However, modeling the effects of multiple regional abnormalities as determinants of cognitive decline at the voxel level remains largely unexplored by multimodal imaging research, given the high computational cost of estimating regression models for every single voxel from various imaging modalities. VoxelStats is a voxel-wise computational framework to overcome these computational limitations and to perform statistical operations on multiple scalar variables and imaging modalities at the voxel level. VoxelStats package has been developed in Matlab(®) and supports imaging formats such as Nifti-1, ANALYZE, and MINC v2. Prebuilt functions in VoxelStats enable the user to perform voxel-wise general and generalized linear models and mixed effect models with multiple volumetric covariates. Importantly, VoxelStats can recognize scalar values or image volumes as response variables and can accommodate volumetric statistical covariates as well as their interaction effects with other variables. Furthermore, this package includes built-in functionality to perform voxel-wise receiver operating characteristic analysis and paired and unpaired group contrast analysis. Validation of VoxelStats was conducted by comparing the linear regression functionality with existing toolboxes such as glim_image and RMINC. The validation results were identical to existing methods and the additional functionality was demonstrated by generating feature case assessments (t-statistics, odds ratio, and true positive rate maps). In summary, VoxelStats expands the current methods for multimodal imaging analysis by allowing the estimation of advanced regional association metrics at the voxel level.

Can Robots and Humans Get Along?

DOE Office of Scientific and Technical Information (OSTI.GOV)

Scholtz, Jean

2007-06-01

Now that robots have moved into the mainstream—as vacuum cleaners, lawn mowers, autonomous vehicles, tour guides, and even pets—it is important to consider how everyday people will interact with them. A robot is really just a computer, but many researchers are beginning to understand that human-robot interactions are much different than human-computer interactions. So while the metrics used to evaluate the human-computer interaction (usability of the software interface in terms of time, accuracy, and user satisfaction) may also be appropriate for human-robot interactions, we need to determine whether there are additional metrics that should be considered.

The Dimensionality and Correlates of Flow in Human-Computer Interactions.

ERIC Educational Resources Information Center

Webster, Jane; And Others

1993-01-01

Defines playfulness in human-computer interactions in terms of flow theory and explores the dimensionality of the flow concept. Two studies are reported that investigated the factor structure and correlates of flow in human-computer interactions: one examined MBA students using Lotus 1-2-3 spreadsheet software, and one examined employees using…

Use of Multi-Modal Media and Tools in an Online Information Literacy Course: College Students' Attitudes and Perceptions

ERIC Educational Resources Information Center

Chen, Hsin-Liang; Williams, James Patrick

2009-01-01

This project studies the use of multi-modal media objects in an online information literacy class. One hundred sixty-two undergraduate students answered seven surveys. Significant relationships are found among computer skills, teaching materials, communication tools and learning experience. Multi-modal media objects and communication tools are…

EEGLAB, SIFT, NFT, BCILAB, and ERICA: new tools for advanced EEG processing.

PubMed

Delorme, Arnaud; Mullen, Tim; Kothe, Christian; Akalin Acar, Zeynep; Bigdely-Shamlo, Nima; Vankov, Andrey; Makeig, Scott

2011-01-01

We describe a set of complementary EEG data collection and processing tools recently developed at the Swartz Center for Computational Neuroscience (SCCN) that connect to and extend the EEGLAB software environment, a freely available and readily extensible processing environment running under Matlab. The new tools include (1) a new and flexible EEGLAB STUDY design facility for framing and performing statistical analyses on data from multiple subjects; (2) a neuroelectromagnetic forward head modeling toolbox (NFT) for building realistic electrical head models from available data; (3) a source information flow toolbox (SIFT) for modeling ongoing or event-related effective connectivity between cortical areas; (4) a BCILAB toolbox for building online brain-computer interface (BCI) models from available data, and (5) an experimental real-time interactive control and analysis (ERICA) environment for real-time production and coordination of interactive, multimodal experiments.

Interactions of Multimodal Ligands with Proteins: Insights into Selectivity Using Molecular Dynamics Simulations.

PubMed

Parimal, Siddharth; Garde, Shekhar; Cramer, Steven M

2015-07-14

Fundamental understanding of protein-ligand interactions is important to the development of efficient bioseparations in multimodal chromatography. Here we employ molecular dynamics (MD) simulations to investigate the interactions of three different proteins--ubiquitin, cytochrome C, and α-chymotrypsinogen A, sampling a range of charge from +1e to +9e--with two multimodal chromatographic ligands containing similar chemical moieties--aromatic, carboxyl, and amide--in different structural arrangements. We use a spherical harmonic expansion to analyze ligand and individual moiety density profiles around the proteins. We find that the Capto MMC ligand, which contains an additional aliphatic group, displays stronger interactions than Nuvia CPrime ligand with all three proteins. Studying the ligand densities at the moiety level suggests that hydrophobic interactions play a major role in determining the locations of high ligand densities. Finally, the greater structural flexibility of the Capto MMC ligand compared to that of the Nuvia cPrime ligand allows for stronger structural complementarity and enables stronger hydrophobic interactions. These subtle and not-so-subtle differences in binding affinities and modalities for multimodal ligands can result in significantly different binding behavior towards proteins with important implications for bioprocessing.

Promoting Multilingual Communicative Competence through Multimodal Academic Learning Situations

ERIC Educational Resources Information Center

Kyppö, Anna; Natri, Teija

2016-01-01

This paper presents information on the factors affecting the development of multilingual and multicultural communicative competence in interactive multimodal learning environments in an academic context. The interdisciplinary course in multilingual interaction offered at the University of Jyväskylä aims to enhance students' competence in…

Pollution going multimodal: the complex impact of the human-altered sensory environment on animal perception and performance

PubMed Central

Halfwerk, Wouter; Slabbekoorn, Hans

2015-01-01

Anthropogenic sensory pollution is affecting ecosystems worldwide. Human actions generate acoustic noise, emanate artificial light and emit chemical substances. All of these pollutants are known to affect animals. Most studies on anthropogenic pollution address the impact of pollutants in unimodal sensory domains. High levels of anthropogenic noise, for example, have been shown to interfere with acoustic signals and cues. However, animals rely on multiple senses, and pollutants often co-occur. Thus, a full ecological assessment of the impact of anthropogenic activities requires a multimodal approach. We describe how sensory pollutants can co-occur and how covariance among pollutants may differ from natural situations. We review how animals combine information that arrives at their sensory systems through different modalities and outline how sensory conditions can interfere with multimodal perception. Finally, we describe how sensory pollutants can affect the perception, behaviour and endocrinology of animals within and across sensory modalities. We conclude that sensory pollution can affect animals in complex ways due to interactions among sensory stimuli, neural processing and behavioural and endocrinal feedback. We call for more empirical data on covariance among sensory conditions, for instance, data on correlated levels in noise and light pollution. Furthermore, we encourage researchers to test animal responses to a full-factorial set of sensory pollutants in the presence or the absence of ecologically important signals and cues. We realize that such approach is often time and energy consuming, but we think this is the only way to fully understand the multimodal impact of sensory pollution on animal performance and perception. PMID:25904319

Towards Better Human Robot Interaction: Understand Human Computer Interaction in Social Gaming Using a Video-Enhanced Diary Method

NASA Astrophysics Data System (ADS)

See, Swee Lan; Tan, Mitchell; Looi, Qin En

This paper presents findings from a descriptive research on social gaming. A video-enhanced diary method was used to understand the user experience in social gaming. From this experiment, we found that natural human behavior and gamer’s decision making process can be elicited and speculated during human computer interaction. These are new information that we should consider as they can help us build better human computer interfaces and human robotic interfaces in future.

Multimodal Imaging of Human Brain Activity: Rational, Biophysical Aspects and Modes of Integration

PubMed Central

Blinowska, Katarzyna; Müller-Putz, Gernot; Kaiser, Vera; Astolfi, Laura; Vanderperren, Katrien; Van Huffel, Sabine; Lemieux, Louis

2009-01-01

Until relatively recently the vast majority of imaging and electrophysiological studies of human brain activity have relied on single-modality measurements usually correlated with readily observable or experimentally modified behavioural or brain state patterns. Multi-modal imaging is the concept of bringing together observations or measurements from different instruments. We discuss the aims of multi-modal imaging and the ways in which it can be accomplished using representative applications. Given the importance of haemodynamic and electrophysiological signals in current multi-modal imaging applications, we also review some of the basic physiology relevant to understanding their relationship. PMID:19547657

Multimodal integration of anatomy and physiology classes: How instructors utilize multimodal teaching in their classrooms

NASA Astrophysics Data System (ADS)

McGraw, Gerald M., Jr.

Multimodality is the theory of communication as it applies to social and educational semiotics (making meaning through the use of multiple signs and symbols). The term multimodality describes a communication methodology that includes multiple textual, aural, and visual applications (modes) that are woven together to create what is referred to as an artifact. Multimodal teaching methodology attempts to create a deeper meaning to course content by activating the higher cognitive areas of the student's brain, creating a more sustained retention of the information (Murray, 2009). The introduction of multimodality educational methodologies as a means to more optimally engage students has been documented within educational literature. However, studies analyzing the distribution and penetration into basic sciences, more specifically anatomy and physiology, have not been forthcoming. This study used a quantitative survey design to determine the degree to which instructors integrated multimodality teaching practices into their course curricula. The instrument used for the study was designed by the researcher based on evidence found in the literature and sent to members of three associations/societies for anatomy and physiology instructors: the Human Anatomy and Physiology Society; the iTeach Anatomy & Physiology Collaborate; and the American Physiology Society. Respondents totaled 182 instructor members of two- and four-year, private and public higher learning colleges collected from the three organizations collectively with over 13,500 members in over 925 higher learning institutions nationwide. The study concluded that the expansion of multimodal methodologies into anatomy and physiology classrooms is at the beginning of the process and that there is ample opportunity for expansion. Instructors continue to use lecture as their primary means of interaction with students. Email is still the major form of out-of-class communication for full-time instructors. Instructors with greater than 16 years of teaching anatomy and physiology are less likely to use video or animation in their classroom than instructors with fewer years.

Representation, Classification and Information Fusion for Robust and Efficient Multimodal Human States Recognition

ERIC Educational Resources Information Center

Li, Ming

2013-01-01

The goal of this work is to enhance the robustness and efficiency of the multimodal human states recognition task. Human states recognition can be considered as a joint term for identifying/verifing various kinds of human related states, such as biometric identity, language spoken, age, gender, emotion, intoxication level, physical activity, vocal…

Multimodal imaging of the human knee down to the cellular level

NASA Astrophysics Data System (ADS)

Schulz, G.; Götz, C.; Müller-Gerbl, M.; Zanette, I.; Zdora, M.-C.; Khimchenko, A.; Deyhle, H.; Thalmann, P.; Müller, B.

2017-06-01

Computed tomography reaches the best spatial resolution for the three-dimensional visualization of human tissues among the available nondestructive clinical imaging techniques. Nowadays, sub-millimeter voxel sizes are regularly obtained. Regarding investigations on true micrometer level, lab-based micro-CT (μCT) has become gold standard. The aim of the present study is firstly the hierarchical investigation of a human knee post mortem using hard X-ray μCT and secondly a multimodal imaging using absorption and phase contrast modes in order to investigate hard (bone) and soft (cartilage) tissues on the cellular level. After the visualization of the entire knee using a clinical CT, a hierarchical imaging study was performed using the lab-system nanotom® m. First, the entire knee was measured with a pixel length of 65 μm. The highest resolution with a pixel length of 3 μm could be achieved after extracting cylindrically shaped plugs from the femoral bones. For the visualization of the cartilage, grating-based phase contrast μCT (I13-2, Diamond Light Source) was performed. With an effective voxel size of 2.3 μm it was possible to visualize individual chondrocytes within the cartilage.

Discriminating between intentional and unintentional gaze fixation using multimodal-based fuzzy logic algorithm for gaze tracking system with NIR camera sensor

NASA Astrophysics Data System (ADS)

Naqvi, Rizwan Ali; Park, Kang Ryoung

2016-06-01

Gaze tracking systems are widely used in human-computer interfaces, interfaces for the disabled, game interfaces, and for controlling home appliances. Most studies on gaze detection have focused on enhancing its accuracy, whereas few have considered the discrimination of intentional gaze fixation (looking at a target to activate or select it) from unintentional fixation while using gaze detection systems. Previous research methods based on the use of a keyboard or mouse button, eye blinking, and the dwell time of gaze position have various limitations. Therefore, we propose a method for discriminating between intentional and unintentional gaze fixation using a multimodal fuzzy logic algorithm applied to a gaze tracking system with a near-infrared camera sensor. Experimental results show that the proposed method outperforms the conventional method for determining gaze fixation.

Stepwise Connectivity of the Modal Cortex Reveals the Multimodal Organization of the Human Brain

PubMed Central

Sepulcre, Jorge; Sabuncu, Mert R.; Yeo, Thomas B.; Liu, Hesheng; Johnson, Keith A.

2012-01-01

How human beings integrate information from external sources and internal cognition to produce a coherent experience is still not well understood. During the past decades, anatomical, neurophysiological and neuroimaging research in multimodal integration have stood out in the effort to understand the perceptual binding properties of the brain. Areas in the human lateral occipito-temporal, prefrontal and posterior parietal cortices have been associated with sensory multimodal processing. Even though this, rather patchy, organization of brain regions gives us a glimpse of the perceptual convergence, the articulation of the flow of information from modality-related to the more parallel cognitive processing systems remains elusive. Using a method called Stepwise Functional Connectivity analysis, the present study analyzes the functional connectome and transitions from primary sensory cortices to higher-order brain systems. We identify the large-scale multimodal integration network and essential connectivity axes for perceptual integration in the human brain. PMID:22855814

Adaptive wavefront shaping for controlling nonlinear multimode interactions in optical fibres

NASA Astrophysics Data System (ADS)

Tzang, Omer; Caravaca-Aguirre, Antonio M.; Wagner, Kelvin; Piestun, Rafael

2018-06-01

Recent progress in wavefront shaping has enabled control of light propagation inside linear media to focus and image through scattering objects. In particular, light propagation in multimode fibres comprises complex intermodal interactions and rich spatiotemporal dynamics. Control of physical phenomena in multimode fibres and its applications are in their infancy, opening opportunities to take advantage of complex nonlinear modal dynamics. Here, we demonstrate a wavefront shaping approach for controlling nonlinear phenomena in multimode fibres. Using a spatial light modulator at the fibre input, real-time spectral feedback and a genetic algorithm optimization, we control a highly nonlinear multimode stimulated Raman scattering cascade and its interplay with four-wave mixing via a flexible implicit control on the superposition of modes coupled into the fibre. We show versatile spectrum manipulations including shifts, suppression, and enhancement of Stokes and anti-Stokes peaks. These demonstrations illustrate the power of wavefront shaping to control and optimize nonlinear wave propagation.

Language evolution and human-computer interaction

NASA Technical Reports Server (NTRS)

Grudin, Jonathan; Norman, Donald A.

1991-01-01

Many of the issues that confront designers of interactive computer systems also appear in natural language evolution. Natural languages and human-computer interfaces share as their primary mission the support of extended 'dialogues' between responsive entities. Because in each case one participant is a human being, some of the pressures operating on natural languages, causing them to evolve in order to better support such dialogue, also operate on human-computer 'languages' or interfaces. This does not necessarily push interfaces in the direction of natural language - since one entity in this dialogue is not a human, this is not to be expected. Nonetheless, by discerning where the pressures that guide natural language evolution also appear in human-computer interaction, we can contribute to the design of computer systems and obtain a new perspective on natural languages.

The Study of Surface Computer Supported Cooperative Work and Its Design, Efficiency, and Challenges

ERIC Educational Resources Information Center

Hwang, Wu-Yuin; Su, Jia-Han

2012-01-01

In this study, a Surface Computer Supported Cooperative Work paradigm is proposed. Recently, multitouch technology has become widely available for human-computer interaction. We found it has great potential to facilitate more awareness of human-to-human interaction than personal computers (PCs) in colocated collaborative work. However, other…

Multimodal Pressure-Flow Analysis: Application of Hilbert Huang Transform in Cerebral Blood Flow Regulation

NASA Astrophysics Data System (ADS)

Lo, Men-Tzung; Hu, Kun; Liu, Yanhui; Peng, C.-K.; Novak, Vera

2008-12-01

Quantification of nonlinear interactions between two nonstationary signals presents a computational challenge in different research fields, especially for assessments of physiological systems. Traditional approaches that are based on theories of stationary signals cannot resolve nonstationarity-related issues and, thus, cannot reliably assess nonlinear interactions in physiological systems. In this review we discuss a new technique called multimodal pressure flow (MMPF) method that utilizes Hilbert-Huang transformation to quantify interaction between nonstationary cerebral blood flow velocity (BFV) and blood pressure (BP) for the assessment of dynamic cerebral autoregulation (CA). CA is an important mechanism responsible for controlling cerebral blood flow in responses to fluctuations in systemic BP within a few heart-beats. The MMPF analysis decomposes BP and BFV signals into multiple empirical modes adaptively so that the fluctuations caused by a specific physiologic process can be represented in a corresponding empirical mode. Using this technique, we showed that dynamic CA can be characterized by specific phase delays between the decomposed BP and BFV oscillations, and that the phase shifts are significantly reduced in hypertensive, diabetics and stroke subjects with impaired CA. Additionally, the new technique can reliably assess CA using both induced BP/BFV oscillations during clinical tests and spontaneous BP/BFV fluctuations during resting conditions.

Interactive visualization and analysis of multimodal datasets for surgical applications.

PubMed

Kirmizibayrak, Can; Yim, Yeny; Wakid, Mike; Hahn, James

2012-12-01

Surgeons use information from multiple sources when making surgical decisions. These include volumetric datasets (such as CT, PET, MRI, and their variants), 2D datasets (such as endoscopic videos), and vector-valued datasets (such as computer simulations). Presenting all the information to the user in an effective manner is a challenging problem. In this paper, we present a visualization approach that displays the information from various sources in a single coherent view. The system allows the user to explore and manipulate volumetric datasets, display analysis of dataset values in local regions, combine 2D and 3D imaging modalities and display results of vector-based computer simulations. Several interaction methods are discussed: in addition to traditional interfaces including mouse and trackers, gesture-based natural interaction methods are shown to control these visualizations with real-time performance. An example of a medical application (medialization laryngoplasty) is presented to demonstrate how the combination of different modalities can be used in a surgical setting with our approach.

Developing a multimodal biometric authentication system using soft computing methods.

PubMed

Malcangi, Mario

2015-01-01

Robust personal authentication is becoming ever more important in computer-based applications. Among a variety of methods, biometric offers several advantages, mainly in embedded system applications. Hard and soft multi-biometric, combined with hard and soft computing methods, can be applied to improve the personal authentication process and to generalize the applicability. This chapter describes the embedded implementation of a multi-biometric (voiceprint and fingerprint) multimodal identification system based on hard computing methods (DSP) for feature extraction and matching, an artificial neural network (ANN) for soft feature pattern matching, and a fuzzy logic engine (FLE) for data fusion and decision.

Multimodal imaging of human cerebellum - merging X-ray phase microtomography, magnetic resonance microscopy and histology

NASA Astrophysics Data System (ADS)

Schulz, Georg; Waschkies, Conny; Pfeiffer, Franz; Zanette, Irene; Weitkamp, Timm; David, Christian; Müller, Bert

2012-11-01

Imaging modalities including magnetic resonance imaging and X-ray computed tomography are established methods in daily clinical diagnosis of human brain. Clinical equipment does not provide sufficient spatial resolution to obtain morphological information on the cellular level, essential for applying minimally or non-invasive surgical interventions. Therefore, generic data with lateral sub-micrometer resolution have been generated from histological slices post mortem. Sub-cellular spatial resolution, lost in the third dimension as a result of sectioning, is obtained using magnetic resonance microscopy and micro computed tomography. We demonstrate that for human cerebellum grating-based X-ray phase tomography shows complementary contrast to magnetic resonance microscopy and histology. In this study, the contrast-to-noise values of magnetic resonance microscopy and phase tomography were comparable whereas the spatial resolution in phase tomography is an order of magnitude better. The registered data with their complementary information permit the distinct segmentation of tissues within the human cerebellum.

Multimodal Interaction on English Testing Academic Assessment

ERIC Educational Resources Information Center

Magal-Royo, T.; Gimenez-Lopez, J. L.; Garcia Laborda, Jesus

2012-01-01

Multimodal interaction methods applied to learning environments of the English language will be a line for future research from the use of adapted mobile phones or PDAs. Today's mobile devices allow access and data entry in a synchronized manner through different channels. At the academic level we made the first analysis of English language…

Contradictory Explorative Assessment. Multimodal Teacher/Student Interaction in Scandinavian Digital Learning Environments

ERIC Educational Resources Information Center

Kjällander, Susanne

2018-01-01

Assessment in the much-discussed digital divide in Scandinavian technologically advanced schools, is the study object of this article. Interaction is studied to understand assessment; and to see how assessment can be didactically designed to recognise students' learning. With a multimodal, design theoretical perspective on learning teachers' and…

Visual and tactile interfaces for bi-directional human robot communication

NASA Astrophysics Data System (ADS)

Barber, Daniel; Lackey, Stephanie; Reinerman-Jones, Lauren; Hudson, Irwin

2013-05-01

Seamless integration of unmanned and systems and Soldiers in the operational environment requires robust communication capabilities. Multi-Modal Communication (MMC) facilitates achieving this goal due to redundancy and levels of communication superior to single mode interaction using auditory, visual, and tactile modalities. Visual signaling using arm and hand gestures is a natural method of communication between people. Visual signals standardized within the U.S. Army Field Manual and in use by Soldiers provide a foundation for developing gestures for human to robot communication. Emerging technologies using Inertial Measurement Units (IMU) enable classification of arm and hand gestures for communication with a robot without the requirement of line-of-sight needed by computer vision techniques. These devices improve the robustness of interpreting gestures in noisy environments and are capable of classifying signals relevant to operational tasks. Closing the communication loop between Soldiers and robots necessitates them having the ability to return equivalent messages. Existing visual signals from robots to humans typically require highly anthropomorphic features not present on military vehicles. Tactile displays tap into an unused modality for robot to human communication. Typically used for hands-free navigation and cueing, existing tactile display technologies are used to deliver equivalent visual signals from the U.S. Army Field Manual. This paper describes ongoing research to collaboratively develop tactile communication methods with Soldiers, measure classification accuracy of visual signal interfaces, and provides an integration example including two robotic platforms.

Application of Virtual Navigation with Multimodality Image Fusion in Foramen Ovale Cannulation.

PubMed

Qiu, Xixiong; Liu, Weizong; Zhang, Mingdong; Lin, Hengzhou; Zhou, Shoujun; Lei, Yi; Xia, Jun

2017-11-01

Idiopathic trigeminal neuralgia (ITN) can be effectively treated with radiofrequency thermocoagulation. However, this procedure requires cannulation of the foramen ovale, and conventional cannulation methods are associated with high failure rates. Multimodality imaging can improve the accuracy of cannulation because each imaging method can compensate for the drawbacks of the other. We aim to determine the feasibility and accuracy of percutaneous foramen ovale cannulation under the guidance of virtual navigation with multimodality image fusion in a self-designed anatomical model of human cadaveric heads. Five cadaveric head specimens were investigated in this study. Spiral computed tomography (CT) scanning clearly displayed the foramen ovale in all five specimens (10 foramina), which could not be visualized using two-dimensional ultrasound alone. The ultrasound and spiral CT images were fused, and percutaneous cannulation of the foramen ovale was performed under virtual navigation. After this, spiral CT scanning was immediately repeated to confirm the accuracy of the cannulation. Postprocedural spiral CT confirmed that the ultrasound and CT images had been successfully fused for all 10 foramina, which were accurately and successfully cannulated. The success rates of both image fusion and cannulation were 100%. Virtual navigation with multimodality image fusion can substantially facilitate foramen ovale cannulation and is worthy of clinical application. © 2017 American Academy of Pain Medicine. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com

EEGLAB, SIFT, NFT, BCILAB, and ERICA: New Tools for Advanced EEG Processing

PubMed Central

Delorme, Arnaud; Mullen, Tim; Kothe, Christian; Akalin Acar, Zeynep; Bigdely-Shamlo, Nima; Vankov, Andrey; Makeig, Scott

2011-01-01

We describe a set of complementary EEG data collection and processing tools recently developed at the Swartz Center for Computational Neuroscience (SCCN) that connect to and extend the EEGLAB software environment, a freely available and readily extensible processing environment running under Matlab. The new tools include (1) a new and flexible EEGLAB STUDY design facility for framing and performing statistical analyses on data from multiple subjects; (2) a neuroelectromagnetic forward head modeling toolbox (NFT) for building realistic electrical head models from available data; (3) a source information flow toolbox (SIFT) for modeling ongoing or event-related effective connectivity between cortical areas; (4) a BCILAB toolbox for building online brain-computer interface (BCI) models from available data, and (5) an experimental real-time interactive control and analysis (ERICA) environment for real-time production and coordination of interactive, multimodal experiments. PMID:21687590

A multimodal interface device for online board games designed for sight-impaired people.

PubMed

Caporusso, Nicholas; Mkrtchyan, Lusine; Badia, Leonardo

2010-03-01

Online games between remote opponents playing over computer networks are becoming a common activity of everyday life. However, computer interfaces for board games are usually based on the visual channel. For example, they require players to check their moves on a video display and interact by using pointing devices such as a mouse. Hence, they are not suitable for visually impaired people. The present paper discusses a multipurpose system that allows especially blind and deafblind people playing chess or other board games over a network, therefore reducing their disability barrier. We describe and benchmark a prototype of a special interactive haptic device for online gaming providing a dual tactile feedback. The novel interface of this proposed device is able to guarantee not only a better game experience for everyone but also an improved quality of life for sight-impaired people.

Human-Computer Interaction and Sociological Insight: A Theoretical Examination and Experiment in Building Affinity in Small Groups

ERIC Educational Resources Information Center

Oren, Michael Anthony

2011-01-01

The juxtaposition of classic sociological theory and the, relatively, young discipline of human-computer interaction (HCI) serves as a powerful mechanism for both exploring the theoretical impacts of technology on human interactions as well as the application of technological systems to moderate interactions. It is the intent of this dissertation…

Haptic-Multimodal Flight Control System Update

NASA Technical Reports Server (NTRS)

Goodrich, Kenneth H.; Schutte, Paul C.; Williams, Ralph A.

2011-01-01

The rapidly advancing capabilities of autonomous aircraft suggest a future where many of the responsibilities of today s pilot transition to the vehicle, transforming the pilot s job into something akin to driving a car or simply being a passenger. Notionally, this transition will reduce the specialized skills, training, and attention required of the human user while improving safety and performance. However, our experience with highly automated aircraft highlights many challenges to this transition including: lack of automation resilience; adverse human-automation interaction under stress; and the difficulty of developing certification standards and methods of compliance for complex systems performing critical functions traditionally performed by the pilot (e.g., sense and avoid vs. see and avoid). Recognizing these opportunities and realities, researchers at NASA Langley are developing a haptic-multimodal flight control (HFC) system concept that can serve as a bridge between today s state of the art aircraft that are highly automated but have little autonomy and can only be operated safely by highly trained experts (i.e., pilots) to a future in which non-experts (e.g., drivers) can safely and reliably use autonomous aircraft to perform a variety of missions. This paper reviews the motivation and theoretical basis of the HFC system, describes its current state of development, and presents results from two pilot-in-the-loop simulation studies. These preliminary studies suggest the HFC reshapes human-automation interaction in a way well-suited to revolutionary ease-of-use.

Human-Computer Interaction and Virtual Environments

NASA Technical Reports Server (NTRS)

Noor, Ahmed K. (Compiler)

1995-01-01

The proceedings of the Workshop on Human-Computer Interaction and Virtual Environments are presented along with a list of attendees. The objectives of the workshop were to assess the state-of-technology and level of maturity of several areas in human-computer interaction and to provide guidelines for focused future research leading to effective use of these facilities in the design/fabrication and operation of future high-performance engineering systems.

Pollution going multimodal: the complex impact of the human-altered sensory environment on animal perception and performance.

PubMed

Halfwerk, Wouter; Slabbekoorn, Hans

2015-04-01

Anthropogenic sensory pollution is affecting ecosystems worldwide. Human actions generate acoustic noise, emanate artificial light and emit chemical substances. All of these pollutants are known to affect animals. Most studies on anthropogenic pollution address the impact of pollutants in unimodal sensory domains. High levels of anthropogenic noise, for example, have been shown to interfere with acoustic signals and cues. However, animals rely on multiple senses, and pollutants often co-occur. Thus, a full ecological assessment of the impact of anthropogenic activities requires a multimodal approach. We describe how sensory pollutants can co-occur and how covariance among pollutants may differ from natural situations. We review how animals combine information that arrives at their sensory systems through different modalities and outline how sensory conditions can interfere with multimodal perception. Finally, we describe how sensory pollutants can affect the perception, behaviour and endocrinology of animals within and across sensory modalities. We conclude that sensory pollution can affect animals in complex ways due to interactions among sensory stimuli, neural processing and behavioural and endocrinal feedback. We call for more empirical data on covariance among sensory conditions, for instance, data on correlated levels in noise and light pollution. Furthermore, we encourage researchers to test animal responses to a full-factorial set of sensory pollutants in the presence or the absence of ecologically important signals and cues. We realize that such approach is often time and energy consuming, but we think this is the only way to fully understand the multimodal impact of sensory pollution on animal performance and perception. © 2015 The Author(s) Published by the Royal Society. All rights reserved.

Multimodal Interaction with Speech, Gestures and Haptic Feedback in a Media Center Application

NASA Astrophysics Data System (ADS)

Turunen, Markku; Hakulinen, Jaakko; Hella, Juho; Rajaniemi, Juha-Pekka; Melto, Aleksi; Mäkinen, Erno; Rantala, Jussi; Heimonen, Tomi; Laivo, Tuuli; Soronen, Hannu; Hansen, Mervi; Valkama, Pellervo; Miettinen, Toni; Raisamo, Roope

We demonstrate interaction with a multimodal media center application. Mobile phone-based interface includes speech and gesture input and haptic feedback. The setup resembles our long-term public pilot study, where a living room environment containing the application was constructed inside a local media museum allowing visitors to freely test the system.

Using the Interactive Whiteboard to Resource Continuity and Support Multimodal Teaching in a Primary Science Classroom

ERIC Educational Resources Information Center

Gillen, J.; Littleton, K.; Twiner, A.; Staarman, J. K.; Mercer, N.

2008-01-01

All communication is inherently multimodal, and understandings of science need to be multidimensional. The interactive whiteboard offers a range of potential benefits to the primary science classroom in terms of relative ease of integration of a number of presentational and ICT functions, which, taken together, offers new opportunities for…

Intraoperative imaging-guided cancer surgery: from current fluorescence molecular imaging methods to future multi-modality imaging technology.

PubMed

Chi, Chongwei; Du, Yang; Ye, Jinzuo; Kou, Deqiang; Qiu, Jingdan; Wang, Jiandong; Tian, Jie; Chen, Xiaoyuan

2014-01-01

Cancer is a major threat to human health. Diagnosis and treatment using precision medicine is expected to be an effective method for preventing the initiation and progression of cancer. Although anatomical and functional imaging techniques such as radiography, computed tomography (CT), magnetic resonance imaging (MRI) and positron emission tomography (PET) have played an important role for accurate preoperative diagnostics, for the most part these techniques cannot be applied intraoperatively. Optical molecular imaging is a promising technique that provides a high degree of sensitivity and specificity in tumor margin detection. Furthermore, existing clinical applications have proven that optical molecular imaging is a powerful intraoperative tool for guiding surgeons performing precision procedures, thus enabling radical resection and improved survival rates. However, detection depth limitation exists in optical molecular imaging methods and further breakthroughs from optical to multi-modality intraoperative imaging methods are needed to develop more extensive and comprehensive intraoperative applications. Here, we review the current intraoperative optical molecular imaging technologies, focusing on contrast agents and surgical navigation systems, and then discuss the future prospects of multi-modality imaging technology for intraoperative imaging-guided cancer surgery.

Intraoperative Imaging-Guided Cancer Surgery: From Current Fluorescence Molecular Imaging Methods to Future Multi-Modality Imaging Technology

PubMed Central

Chi, Chongwei; Du, Yang; Ye, Jinzuo; Kou, Deqiang; Qiu, Jingdan; Wang, Jiandong; Tian, Jie; Chen, Xiaoyuan

2014-01-01

Cancer is a major threat to human health. Diagnosis and treatment using precision medicine is expected to be an effective method for preventing the initiation and progression of cancer. Although anatomical and functional imaging techniques such as radiography, computed tomography (CT), magnetic resonance imaging (MRI) and positron emission tomography (PET) have played an important role for accurate preoperative diagnostics, for the most part these techniques cannot be applied intraoperatively. Optical molecular imaging is a promising technique that provides a high degree of sensitivity and specificity in tumor margin detection. Furthermore, existing clinical applications have proven that optical molecular imaging is a powerful intraoperative tool for guiding surgeons performing precision procedures, thus enabling radical resection and improved survival rates. However, detection depth limitation exists in optical molecular imaging methods and further breakthroughs from optical to multi-modality intraoperative imaging methods are needed to develop more extensive and comprehensive intraoperative applications. Here, we review the current intraoperative optical molecular imaging technologies, focusing on contrast agents and surgical navigation systems, and then discuss the future prospects of multi-modality imaging technology for intraoperative imaging-guided cancer surgery. PMID:25250092

Multimodal Learning Clubs

ERIC Educational Resources Information Center

Casey, Heather

2012-01-01

Multimodal learning clubs link principles of motivation and engagement with 21st century technological tools and texts to support content area learning. The author describes how a sixth grade health teacher and his class incorporated multimodal learning clubs into a unit of study on human body systems. The students worked collaboratively online…

Human Factors Considerations in System Design

NASA Technical Reports Server (NTRS)

Mitchell, C. M. (Editor); Vanbalen, P. M. (Editor); Moe, K. L. (Editor)

1983-01-01

Human factors considerations in systems design was examined. Human factors in automated command and control, in the efficiency of the human computer interface and system effectiveness are outlined. The following topics are discussed: human factors aspects of control room design; design of interactive systems; human computer dialogue, interaction tasks and techniques; guidelines on ergonomic aspects of control rooms and highly automated environments; system engineering for control by humans; conceptual models of information processing; information display and interaction in real time environments.

Fast multi-core based multimodal registration of 2D cross-sections and 3D datasets.

PubMed

Scharfe, Michael; Pielot, Rainer; Schreiber, Falk

2010-01-11

Solving bioinformatics tasks often requires extensive computational power. Recent trends in processor architecture combine multiple cores into a single chip to improve overall performance. The Cell Broadband Engine (CBE), a heterogeneous multi-core processor, provides power-efficient and cost-effective high-performance computing. One application area is image analysis and visualisation, in particular registration of 2D cross-sections into 3D image datasets. Such techniques can be used to put different image modalities into spatial correspondence, for example, 2D images of histological cuts into morphological 3D frameworks. We evaluate the CBE-driven PlayStation 3 as a high performance, cost-effective computing platform by adapting a multimodal alignment procedure to several characteristic hardware properties. The optimisations are based on partitioning, vectorisation, branch reducing and loop unrolling techniques with special attention to 32-bit multiplies and limited local storage on the computing units. We show how a typical image analysis and visualisation problem, the multimodal registration of 2D cross-sections and 3D datasets, benefits from the multi-core based implementation of the alignment algorithm. We discuss several CBE-based optimisation methods and compare our results to standard solutions. More information and the source code are available from http://cbe.ipk-gatersleben.de. The results demonstrate that the CBE processor in a PlayStation 3 accelerates computational intensive multimodal registration, which is of great importance in biological/medical image processing. The PlayStation 3 as a low cost CBE-based platform offers an efficient option to conventional hardware to solve computational problems in image processing and bioinformatics.

Reflection Effects in Multimode Fiber Systems Utilizing Laser Transmitters

NASA Technical Reports Server (NTRS)

Bates, Harry E.

1991-01-01

A number of optical communication lines are now in use at NASA-Kennedy for the transmission of voice, computer data, and video signals. Now, all of these channels use a single carrier wavelength centered near 1300 or 1550 nm. Engineering tests in the past have given indications of the growth of systematic and random noise in the RF spectrum of a fiber network as the number of connector pairs is increased. This noise seems to occur when a laser transmitter is used instead of a LED. It has been suggested that the noise is caused by back reflections created at connector fiber interfaces. Experiments were performed to explore the effect of reflection on the transmitting laser under conditions of reflective feedback. This effort included computer integration of some of the instrumentation in the fiber optic lab using the Lab View software recently acquired by the lab group. The main goal was to interface the Anritsu Optical and RF spectrum analyzers to the MacIntosh II computer so that laser spectra and network RF spectra could be simultaneously and rapidly acquired in a form convenient for analysis. Both single and multimode fiber is installed at Kennedy. Since most are multimode, this effort concentrated on multimode systems.

Reflection effects in multimode fiber systems utilizing laser transmitters

NASA Astrophysics Data System (ADS)

Bates, Harry E.

1991-11-01

A number of optical communication lines are now in use at NASA-Kennedy for the transmission of voice, computer data, and video signals. Now, all of these channels use a single carrier wavelength centered near 1300 or 1550 nm. Engineering tests in the past have given indications of the growth of systematic and random noise in the RF spectrum of a fiber network as the number of connector pairs is increased. This noise seems to occur when a laser transmitter is used instead of a LED. It has been suggested that the noise is caused by back reflections created at connector fiber interfaces. Experiments were performed to explore the effect of reflection on the transmitting laser under conditions of reflective feedback. This effort included computer integration of some of the instrumentation in the fiber optic lab using the Lab View software recently acquired by the lab group. The main goal was to interface the Anritsu Optical and RF spectrum analyzers to the MacIntosh II computer so that laser spectra and network RF spectra could be simultaneously and rapidly acquired in a form convenient for analysis. Both single and multimode fiber is installed at Kennedy. Since most are multimode, this effort concentrated on multimode systems.

Development of an E-Prime Based Computer Simulation of an Interactive Human Rights Violation Negotiation Script (Developpement d’un Programme de Simulation par Ordinateur Fonde sur le Logiciel E Prime pour la Negociation Interactive en cas de Violation des Droits de la Personne)

DTIC Science & Technology

2010-12-01

Base ( CFB ) Kingston. The computer simulation developed in this project is intended to be used for future research and as a possible training platform...DRDC Toronto No. CR 2010-055 Development of an E-Prime based computer simulation of an interactive Human Rights Violation negotiation script...Abstract This report describes the method of developing an E-Prime computer simulation of an interactive Human Rights Violation (HRV) negotiation. An

A multimodal image guiding system for Navigated Ultrasound Bronchoscopy (EBUS): A human feasibility study

PubMed Central

Hofstad, Erlend Fagertun; Amundsen, Tore; Langø, Thomas; Bakeng, Janne Beate Lervik; Leira, Håkon Olav

2017-01-01

Background Endobronchial ultrasound transbronchial needle aspiration (EBUS-TBNA) is the endoscopic method of choice for confirming lung cancer metastasis to mediastinal lymph nodes. Precision is crucial for correct staging and clinical decision-making. Navigation and multimodal imaging can potentially improve EBUS-TBNA efficiency. Aims To demonstrate the feasibility of a multimodal image guiding system using electromagnetic navigation for ultrasound bronchoschopy in humans. Methods Four patients referred for lung cancer diagnosis and staging with EBUS-TBNA were enrolled in the study. Target lymph nodes were predefined from the preoperative computed tomography (CT) images. A prototype convex probe ultrasound bronchoscope with an attached sensor for position tracking was used for EBUS-TBNA. Electromagnetic tracking of the ultrasound bronchoscope and ultrasound images allowed fusion of preoperative CT and intraoperative ultrasound in the navigation software. Navigated EBUS-TBNA was used to guide target lymph node localization and sampling. Navigation system accuracy was calculated, measured by the deviation between lymph node position in ultrasound and CT in three planes. Procedure time, diagnostic yield and adverse events were recorded. Results Preoperative CT and real-time ultrasound images were successfully fused and displayed in the navigation software during the procedures. Overall navigation accuracy (11 measurements) was 10.0 ± 3.8 mm, maximum 17.6 mm, minimum 4.5 mm. An adequate sample was obtained in 6/6 (100%) of targeted lymph nodes. No adverse events were registered. Conclusions Electromagnetic navigated EBUS-TBNA was feasible, safe and easy in this human pilot study. The clinical usefulness was clearly demonstrated. Fusion of real-time ultrasound, preoperative CT and electromagnetic navigational bronchoscopy provided a controlled guiding to level of target, intraoperative overview and procedure documentation. PMID:28182758

Graphic Disruptions: Comics, Disability and De-Canonizing Composition

ERIC Educational Resources Information Center

Walters, Shannon

2015-01-01

The study of comics is an important part of the project of critiquing normative assumptions underlying multimodality and composition. Extending the efforts of the authors of "Multimodality in Motion"--which explains that "multimodality as it is commonly used implies an ableist understanding of the human composer" (Yergeau et…

Multimodal Language Learner Interactions via Desktop Videoconferencing within a Framework of Social Presence: Gaze

ERIC Educational Resources Information Center

Satar, H. Muge

2013-01-01

Desktop videoconferencing (DVC) offers many opportunities for language learning through its multimodal features. However, it also brings some challenges such as gaze and mutual gaze, that is, eye-contact. This paper reports some of the findings of a PhD study investigating social presence in DVC interactions of English as a Foreign Language (EFL)…

Multi-Modal Interaction for Robotic Mules

DTIC Science & Technology

2014-02-26

Multi-Modal Interaction for Robotic Mules Glenn Taylor, Mike Quist, Matt Lanting, Cory Dunham , Patrick Theisen, Paul Muench Abstract...Taylor, Mike Quist, Matt Lanting, Cory Dunham , and Patrick Theisen are with Soar Technology, Inc. (corresponding author: 734-887- 7620; email: glenn...soartech.com; quist@soartech.com; matt.lanting@soartech.com; dunham @soartech.com; patrick.theisen@soartech.com Paul Muench is with US Army TARDEC

Human-Computer Interaction: A Review of the Research on Its Affective and Social Aspects.

ERIC Educational Resources Information Center

Deaudelin, Colette; Dussault, Marc; Brodeur, Monique

2003-01-01

Discusses a review of 34 qualitative and non-qualitative studies related to affective and social aspects of student-computer interactions. Highlights include the nature of the human-computer interaction (HCI); the interface, comparing graphic and text types; and the relation between variables linked to HCI, mainly trust, locus of control,…

Cortical inter-hemispheric circuits for multimodal vocal learning in songbirds.

PubMed

Paterson, Amy K; Bottjer, Sarah W

2017-10-15

Vocal learning in songbirds and humans is strongly influenced by social interactions based on sensory inputs from several modalities. Songbird vocal learning is mediated by cortico-basal ganglia circuits that include the SHELL region of lateral magnocellular nucleus of the anterior nidopallium (LMAN), but little is known concerning neural pathways that could integrate multimodal sensory information with SHELL circuitry. In addition, cortical pathways that mediate the precise coordination between hemispheres required for song production have been little studied. In order to identify candidate mechanisms for multimodal sensory integration and bilateral coordination for vocal learning in zebra finches, we investigated the anatomical organization of two regions that receive input from SHELL: the dorsal caudolateral nidopallium (dNCL SHELL ) and a region within the ventral arcopallium (Av). Anterograde and retrograde tracing experiments revealed a topographically organized inter-hemispheric circuit: SHELL and dNCL SHELL , as well as adjacent nidopallial areas, send axonal projections to ipsilateral Av; Av in turn projects to contralateral SHELL, dNCL SHELL , and regions of nidopallium adjacent to each. Av on each side also projects directly to contralateral Av. dNCL SHELL and Av each integrate inputs from ipsilateral SHELL with inputs from sensory regions in surrounding nidopallium, suggesting that they function to integrate multimodal sensory information with song-related responses within LMAN-SHELL during vocal learning. Av projections share this integrated information from the ipsilateral hemisphere with contralateral sensory and song-learning regions. Our results suggest that the inter-hemispheric pathway through Av may function to integrate multimodal sensory feedback with vocal-learning circuitry and coordinate bilateral vocal behavior. © 2017 Wiley Periodicals, Inc.

Toward a Neuroscientific Understanding of Play: A Dimensional Coding Framework for Analyzing Infant–Adult Play Patterns

PubMed Central

Neale, Dave; Clackson, Kaili; Georgieva, Stanimira; Dedetas, Hatice; Scarpate, Melissa; Wass, Sam; Leong, Victoria

2018-01-01

Play during early life is a ubiquitous activity, and an individual’s propensity for play is positively related to cognitive development and emotional well-being. Play behavior (which may be solitary or shared with a social partner) is diverse and multi-faceted. A challenge for current research is to converge on a common definition and measurement system for play – whether examined at a behavioral, cognitive or neurological level. Combining these different approaches in a multimodal analysis could yield significant advances in understanding the neurocognitive mechanisms of play, and provide the basis for developing biologically grounded play models. However, there is currently no integrated framework for conducting a multimodal analysis of play that spans brain, cognition and behavior. The proposed coding framework uses grounded and observable behaviors along three dimensions (sensorimotor, cognitive and socio-emotional), to compute inferences about playful behavior in a social context, and related social interactional states. Here, we illustrate the sensitivity and utility of the proposed coding framework using two contrasting dyadic corpora (N = 5) of mother-infant object-oriented interactions during experimental conditions that were either non-conducive (Condition 1) or conducive (Condition 2) to the emergence of playful behavior. We find that the framework accurately identifies the modal form of social interaction as being either non-playful (Condition 1) or playful (Condition 2), and further provides useful insights about differences in the quality of social interaction and temporal synchronicity within the dyad. It is intended that this fine-grained coding of play behavior will be easily assimilated with, and inform, future analysis of neural data that is also collected during adult–infant play. In conclusion, here, we present a novel framework for analyzing the continuous time-evolution of adult–infant play patterns, underpinned by biologically informed state coding along sensorimotor, cognitive and socio-emotional dimensions. We expect that the proposed framework will have wide utility amongst researchers wishing to employ an integrated, multimodal approach to the study of play, and lead toward a greater understanding of the neuroscientific basis of play. It may also yield insights into a new biologically grounded taxonomy of play interactions. PMID:29618994

Toward a Neuroscientific Understanding of Play: A Dimensional Coding Framework for Analyzing Infant-Adult Play Patterns.

PubMed

Neale, Dave; Clackson, Kaili; Georgieva, Stanimira; Dedetas, Hatice; Scarpate, Melissa; Wass, Sam; Leong, Victoria

2018-01-01

Play during early life is a ubiquitous activity, and an individual's propensity for play is positively related to cognitive development and emotional well-being. Play behavior (which may be solitary or shared with a social partner) is diverse and multi-faceted. A challenge for current research is to converge on a common definition and measurement system for play - whether examined at a behavioral, cognitive or neurological level. Combining these different approaches in a multimodal analysis could yield significant advances in understanding the neurocognitive mechanisms of play, and provide the basis for developing biologically grounded play models. However, there is currently no integrated framework for conducting a multimodal analysis of play that spans brain, cognition and behavior. The proposed coding framework uses grounded and observable behaviors along three dimensions (sensorimotor, cognitive and socio-emotional), to compute inferences about playful behavior in a social context, and related social interactional states. Here, we illustrate the sensitivity and utility of the proposed coding framework using two contrasting dyadic corpora ( N = 5) of mother-infant object-oriented interactions during experimental conditions that were either non-conducive (Condition 1) or conducive (Condition 2) to the emergence of playful behavior. We find that the framework accurately identifies the modal form of social interaction as being either non-playful (Condition 1) or playful (Condition 2), and further provides useful insights about differences in the quality of social interaction and temporal synchronicity within the dyad. It is intended that this fine-grained coding of play behavior will be easily assimilated with, and inform, future analysis of neural data that is also collected during adult-infant play. In conclusion, here, we present a novel framework for analyzing the continuous time-evolution of adult-infant play patterns, underpinned by biologically informed state coding along sensorimotor, cognitive and socio-emotional dimensions. We expect that the proposed framework will have wide utility amongst researchers wishing to employ an integrated, multimodal approach to the study of play, and lead toward a greater understanding of the neuroscientific basis of play. It may also yield insights into a new biologically grounded taxonomy of play interactions.

Multi-mode of Four and Six Wave Parametric Amplified Process

NASA Astrophysics Data System (ADS)

Zhu, Dayu; Yang, Yiheng; Zhang, Da; Liu, Ruizhou; Ma, Danmeng; Li, Changbiao; Zhang, Yanpeng

2017-03-01

Multiple quantum modes in correlated fields are essential for future quantum information processing and quantum computing. Here we report the generation of multi-mode phenomenon through parametric amplified four- and six-wave mixing processes in a rubidium atomic ensemble. The multi-mode properties in both frequency and spatial domains are studied. On one hand, the multi-mode behavior is dominantly controlled by the intensity of external dressing effect, or nonlinear phase shift through internal dressing effect, in frequency domain; on the other hand, the multi-mode behavior is visually demonstrated from the images of the biphoton fields directly, in spatial domain. Besides, the correlation of the two output fields is also demonstrated in both domains. Our approach supports efficient applications for scalable quantum correlated imaging.

Multi-mode of Four and Six Wave Parametric Amplified Process.

PubMed

Zhu, Dayu; Yang, Yiheng; Zhang, Da; Liu, Ruizhou; Ma, Danmeng; Li, Changbiao; Zhang, Yanpeng

2017-03-03

Multiple quantum modes in correlated fields are essential for future quantum information processing and quantum computing. Here we report the generation of multi-mode phenomenon through parametric amplified four- and six-wave mixing processes in a rubidium atomic ensemble. The multi-mode properties in both frequency and spatial domains are studied. On one hand, the multi-mode behavior is dominantly controlled by the intensity of external dressing effect, or nonlinear phase shift through internal dressing effect, in frequency domain; on the other hand, the multi-mode behavior is visually demonstrated from the images of the biphoton fields directly, in spatial domain. Besides, the correlation of the two output fields is also demonstrated in both domains. Our approach supports efficient applications for scalable quantum correlated imaging.

Applications of Elpasolites as a Multimode Radiation Sensor

NASA Astrophysics Data System (ADS)

Guckes, Amber

This study consists of both computational and experimental investigations. The computational results enabled detector design selections and confirmed experimental results. The experimental results determined that the CLYC scintillation detector can be applied as a functional and field-deployable multimode radiation sensor. The computational study utilized MCNP6 code to investigate the response of CLYC to various incident radiations and to determine the feasibility of its application as a handheld multimode sensor and as a single-scintillator collimated directional detection system. These simulations include: • Characterization of the response of the CLYC scintillator to gamma-rays and neutrons; • Study of the isotopic enrichment of 7Li versus 6Li in the CLYC for optimal detection of both thermal neutrons and fast neutrons; • Analysis of collimator designs to determine the optimal collimator for the single CLYC sensor directional detection system to assay gamma rays and neutrons; Simulations of a handheld CLYC multimode sensor and a single CLYC scintillator collimated directional detection system with the optimized collimator to determine the feasibility of detecting nuclear materials that could be encountered during field operations. These nuclear materials include depleted uranium, natural uranium, low-enriched uranium, highly-enriched uranium, reactor-grade plutonium, and weapons-grade plutonium. The experimental study includes the design, construction, and testing of both a handheld CLYC multimode sensor and a single CLYC scintillator collimated directional detection system. Both were designed in the Inventor CAD software and based on results of the computational study to optimize its performance. The handheld CLYC multimode sensor is modular, scalable, low?power, and optimized for high count rates. Commercial?off?the?shelf components were used where possible in order to optimize size, increase robustness, and minimize cost. The handheld CLYC multimode sensor was successfully tested to confirm its ability for gamma-ray and neutron detection, and gamma?ray and neutron spectroscopy. The sensor utilizes wireless data transfer for possible radiation mapping and network?centric deployment. The handheld multimode sensor was tested by performing laboratory measurements with various gamma-ray sources and neutron sources. The single CLYC scintillator collimated directional detection system is portable, robust, and capable of source localization and identification. The collimator was designed based on the results of the computational study and is constructed with high density polyethylene (HDPE) and lead (Pb). The collimator design and construction allows for the directional detection of gamma rays and fast neutrons utilizing only one scintillator which is interchangeable. For this study, a CLYC-7 scintillator was used. The collimated directional detection system was tested by performing laboratory directional measurements with various gamma-ray sources, 252Cf and a 239PuBe source.

Individual Functional ROI Optimization via Maximization of Group-wise Consistency of Structural and Functional Profiles

PubMed Central

Li, Kaiming; Guo, Lei; Zhu, Dajiang; Hu, Xintao; Han, Junwei; Liu, Tianming

2013-01-01

Studying connectivities among functional brain regions and the functional dynamics on brain networks has drawn increasing interest. A fundamental issue that affects functional connectivity and dynamics studies is how to determine the best possible functional brain regions or ROIs (regions of interest) for a group of individuals, since the connectivity measurements are heavily dependent on ROI locations. Essentially, identification of accurate, reliable and consistent corresponding ROIs is challenging due to the unclear boundaries between brain regions, variability across individuals, and nonlinearity of the ROIs. In response to these challenges, this paper presents a novel methodology to computationally optimize ROIs locations derived from task-based fMRI data for individuals so that the optimized ROIs are more consistent, reproducible and predictable across brains. Our computational strategy is to formulate the individual ROI location optimization as a group variance minimization problem, in which group-wise consistencies in functional/structural connectivity patterns and anatomic profiles are defined as optimization constraints. Our experimental results from multimodal fMRI and DTI data show that the optimized ROIs have significantly improved consistency in structural and functional profiles across individuals. These improved functional ROIs with better consistency could contribute to further study of functional interaction and dynamics in the human brain. PMID:22281931

Multimodal swept-source spectrally encoded scanning laser ophthalmoscopy and optical coherence tomography at 400 kHz

NASA Astrophysics Data System (ADS)

El-Haddad, Mohamed T.; Joos, Karen M.; Patel, Shriji N.; Tao, Yuankai K.

2017-02-01

Multimodal imaging systems that combine scanning laser ophthalmoscopy (SLO) and optical coherence tomography (OCT) have demonstrated the utility of concurrent en face and volumetric imaging for aiming, eye tracking, bulk motion compensation, mosaicking, and contrast enhancement. However, this additional functionality trades off with increased system complexity and cost because both SLO and OCT generally require dedicated light sources, galvanometer scanners, relay and imaging optics, detectors, and control and digitization electronics. We previously demonstrated multimodal ophthalmic imaging using swept-source spectrally encoded SLO and OCT (SS-SESLO-OCT). Here, we present system enhancements and a new optical design that increase our SS-SESLO-OCT data throughput by >7x and field-of-view (FOV) by >4x. A 200 kHz 1060 nm Axsun swept-source was optically buffered to 400 kHz sweep-rate, and SESLO and OCT were simultaneously digitized on dual input channels of a 4 GS/s digitizer at 1.2 GS/s per channel using a custom k-clock. We show in vivo human imaging of the anterior segment out to the limbus and retinal fundus over a >40° FOV. In addition, nine overlapping volumetric SS-SESLO-OCT volumes were acquired under video-rate SESLO preview and guidance. In post-processing, all nine SESLO images and en face projections of the corresponding OCT volumes were mosaicked to show widefield multimodal fundus imaging with a >80° FOV. Concurrent multimodal SS-SESLO-OCT may have applications in clinical diagnostic imaging by enabling aiming, image registration, and multi-field mosaicking and benefit intraoperative imaging by allowing for real-time surgical feedback, instrument tracking, and overlays of computationally extracted image-based surrogate biomarkers of disease.

A Multimodal Search Engine for Medical Imaging Studies.

PubMed

Pinho, Eduardo; Godinho, Tiago; Valente, Frederico; Costa, Carlos

2017-02-01

The use of digital medical imaging systems in healthcare institutions has increased significantly, and the large amounts of data in these systems have led to the conception of powerful support tools: recent studies on content-based image retrieval (CBIR) and multimodal information retrieval in the field hold great potential in decision support, as well as for addressing multiple challenges in healthcare systems, such as computer-aided diagnosis (CAD). However, the subject is still under heavy research, and very few solutions have become part of Picture Archiving and Communication Systems (PACS) in hospitals and clinics. This paper proposes an extensible platform for multimodal medical image retrieval, integrated in an open-source PACS software with profile-based CBIR capabilities. In this article, we detail a technical approach to the problem by describing its main architecture and each sub-component, as well as the available web interfaces and the multimodal query techniques applied. Finally, we assess our implementation of the engine with computational performance benchmarks.

Radioactive Nanomaterials for Multimodality Imaging

PubMed Central

Chen, Daiqin; Dougherty, Casey A.; Yang, Dongzhi; Wu, Hongwei; Hong, Hao

2016-01-01

Nuclear imaging techniques, including primarily positron emission tomography (PET) and single-photon emission computed tomography (SPECT), can provide quantitative information for a biological event in vivo with ultra-high sensitivity, however, the comparatively low spatial resolution is their major limitation in clinical application. By convergence of nuclear imaging with other imaging modalities like computed tomography (CT), magnetic resonance imaging (MRI) and optical imaging, the hybrid imaging platforms can overcome the limitations from each individual imaging technique. Possessing versatile chemical linking ability and good cargo-loading capacity, radioactive nanomaterials can serve as ideal imaging contrast agents. In this review, we provide a brief overview about current state-of-the-art applications of radioactive nanomaterials in the circumstances of multimodality imaging. We present strategies for incorporation of radioisotope(s) into nanomaterials along with applications of radioactive nanomaterials in multimodal imaging. Advantages and limitations of radioactive nanomaterials for multimodal imaging applications are discussed. Finally, a future perspective of possible radioactive nanomaterial utilization is presented for improving diagnosis and patient management in a variety of diseases. PMID:27227167

Human-Robot Teams for Unknown and Uncertain Environments

NASA Technical Reports Server (NTRS)

Fong, Terry

2015-01-01

Man-robot interaction is the study of interactions between humans and robots. It is often referred as HRI by researchers. Human-robot interaction is a multidisciplinary field with contributions from human-computer interaction, artificial intelligence.

Thalamic and cortical pathways supporting auditory processing

PubMed Central

Lee, Charles C.

2012-01-01

The neural processing of auditory information engages pathways that begin initially at the cochlea and that eventually reach forebrain structures. At these higher levels, the computations necessary for extracting auditory source and identity information rely on the neuroanatomical connections between the thalamus and cortex. Here, the general organization of these connections in the medial geniculate body (thalamus) and the auditory cortex is reviewed. In addition, we consider two models organizing the thalamocortical pathways of the non-tonotopic and multimodal auditory nuclei. Overall, the transfer of information to the cortex via the thalamocortical pathways is complemented by the numerous intracortical and corticocortical pathways. Although interrelated, the convergent interactions among thalamocortical, corticocortical, and commissural pathways enable the computations necessary for the emergence of higher auditory perception. PMID:22728130

Agent Interaction with Human Systems in Complex Environments: Requirements for Automating the Function of CapCom in Apollo 17

NASA Technical Reports Server (NTRS)

Clancey, William J.

2003-01-01

A human-centered approach to computer systems design involves reframing analysis in terms of people interacting with each other, not only human-machine interaction. The primary concern is not how people can interact with computers, but how shall we design computers to help people work together? An analysis of astronaut interactions with CapCom on Earth during one traverse of Apollo 17 shows what kind of information was conveyed and what might be automated today. A variety of agent and robotic technologies are proposed that deal with recurrent problems in communication and coordination during the analyzed traverse.

A multimodal approach to estimating vigilance using EEG and forehead EOG.

PubMed

Zheng, Wei-Long; Lu, Bao-Liang

2017-04-01

Covert aspects of ongoing user mental states provide key context information for user-aware human computer interactions. In this paper, we focus on the problem of estimating the vigilance of users using EEG and EOG signals. The PERCLOS index as vigilance annotation is obtained from eye tracking glasses. To improve the feasibility and wearability of vigilance estimation devices for real-world applications, we adopt a novel electrode placement for forehead EOG and extract various eye movement features, which contain the principal information of traditional EOG. We explore the effects of EEG from different brain areas and combine EEG and forehead EOG to leverage their complementary characteristics for vigilance estimation. Considering that the vigilance of users is a dynamic changing process because the intrinsic mental states of users involve temporal evolution, we introduce continuous conditional neural field and continuous conditional random field models to capture dynamic temporal dependency. We propose a multimodal approach to estimating vigilance by combining EEG and forehead EOG and incorporating the temporal dependency of vigilance into model training. The experimental results demonstrate that modality fusion can improve the performance compared with a single modality, EOG and EEG contain complementary information for vigilance estimation, and the temporal dependency-based models can enhance the performance of vigilance estimation. From the experimental results, we observe that theta and alpha frequency activities are increased, while gamma frequency activities are decreased in drowsy states in contrast to awake states. The forehead setup allows for the simultaneous collection of EEG and EOG and achieves comparative performance using only four shared electrodes in comparison with the temporal and posterior sites.

Combined multi-modal photoacoustic tomography, optical coherence tomography (OCT) and OCT angiography system with an articulated probe for in vivo human skin structure and vasculature imaging

PubMed Central

Liu, Mengyang; Chen, Zhe; Zabihian, Behrooz; Sinz, Christoph; Zhang, Edward; Beard, Paul C.; Ginner, Laurin; Hoover, Erich; Minneman, Micheal P.; Leitgeb, Rainer A.; Kittler, Harald; Drexler, Wolfgang

2016-01-01

Cutaneous blood flow accounts for approximately 5% of cardiac output in human and plays a key role in a number of a physiological and pathological processes. We show for the first time a multi-modal photoacoustic tomography (PAT), optical coherence tomography (OCT) and OCT angiography system with an articulated probe to extract human cutaneous vasculature in vivo in various skin regions. OCT angiography supplements the microvasculature which PAT alone is unable to provide. Co-registered volumes for vessel network is further embedded in the morphologic image provided by OCT. This multi-modal system is therefore demonstrated as a valuable tool for comprehensive non-invasive human skin vasculature and morphology imaging in vivo. PMID:27699106

Interactions, Intersections and Improvisations: Studying the Multimodal Texts and Classroom Talk of Six- to Seven-Year-Olds

ERIC Educational Resources Information Center

Pahl, Kate

2009-01-01

This article examines the relationship between children's talk in the classroom and their multimodal texts. The article uses an analytic framework derived from Bourdieu's concept of habitus to examine how 6-7-year-old children's regular ways of being and doing can be found in their multimodal texts together with their talk (Bourdieu, 1977, 1990).…

A Framework and Toolkit for the Construction of Multimodal Learning Interfaces

DTIC Science & Technology

1998-04-29

human communication modalities in the context of a broad class of applications, specifically those that support state manipulation via parameterized actions. The multimodal semantic model is also the basis for a flexible, domain independent, incrementally trainable multimodal interpretation algorithm based on a connectionist network. The second major contribution is an application framework consisting of reusable components and a modular, distributed system architecture. Multimodal application developers can assemble the components in the framework into a new application,

Designing an Automated Assessment of Public Speaking Skills Using Multimodal Cues

ERIC Educational Resources Information Center

Chen, Lei; Feng, Gary; Leong, Chee Wee; Joe, Jilliam; Kitchen, Christopher; Lee, Chong Min

2016-01-01

Traditional assessments of public speaking skills rely on human scoring. We report an initial study on the development of an automated scoring model for public speaking performances using multimodal technologies. Task design, rubric development, and human rating were conducted according to standards in educational assessment. An initial corpus of…

Fast multi-core based multimodal registration of 2D cross-sections and 3D datasets

PubMed Central

2010-01-01

Background Solving bioinformatics tasks often requires extensive computational power. Recent trends in processor architecture combine multiple cores into a single chip to improve overall performance. The Cell Broadband Engine (CBE), a heterogeneous multi-core processor, provides power-efficient and cost-effective high-performance computing. One application area is image analysis and visualisation, in particular registration of 2D cross-sections into 3D image datasets. Such techniques can be used to put different image modalities into spatial correspondence, for example, 2D images of histological cuts into morphological 3D frameworks. Results We evaluate the CBE-driven PlayStation 3 as a high performance, cost-effective computing platform by adapting a multimodal alignment procedure to several characteristic hardware properties. The optimisations are based on partitioning, vectorisation, branch reducing and loop unrolling techniques with special attention to 32-bit multiplies and limited local storage on the computing units. We show how a typical image analysis and visualisation problem, the multimodal registration of 2D cross-sections and 3D datasets, benefits from the multi-core based implementation of the alignment algorithm. We discuss several CBE-based optimisation methods and compare our results to standard solutions. More information and the source code are available from http://cbe.ipk-gatersleben.de. Conclusions The results demonstrate that the CBE processor in a PlayStation 3 accelerates computational intensive multimodal registration, which is of great importance in biological/medical image processing. The PlayStation 3 as a low cost CBE-based platform offers an efficient option to conventional hardware to solve computational problems in image processing and bioinformatics. PMID:20064262

The Role of Bcl-xL in Synergistic Induction of Apoptosis by Mapatumumab and Oxaliplatin in Combination with Hyperthermia on Human Colon Cancer

PubMed Central

Song, Xinxin; Kim, Seog-Young; Lee, Yong J.

2012-01-01

Colorectal cancer is the third leading cause of cancer-related mortality in the world. The main cause of death of colorectal cancer is hepatic metastases which can be treated using isolated hepatic perfusion (IHP), allowing treatment of colorectal metastasis with various methods. In this study we present a novel potent multimodality strategy comprising humanized death receptor 4 (DR4) antibody mapatumumab (Mapa) in combination with oxaliplatin and hyperthermia to treat human colon cancer cells. Oxaliplatin and hyperthermia sensitized colon cancer cells to Mapa in the mitochondrial dependent apoptotic pathway and increased reactive oxygen species production, leading to Bcl-xL phosphorylation at Serine 62 in a c-Jun N-terminal kinase (JNK)-dependent manner. Overexpression of Bcl-xL reduced the efficacy of the multimodality treatment, while phosphorylation of Bcl-xL decreased its anti-apoptotic activity. The multimodality treatment dissociated Bcl-xL from Bax, allowing Bax oligomerization to induce cytochrome c release from mitochondria. In addition, the multimodality treatment significantly inhibited colorectal cancer xenografts’ tumor growth. The successful outcome of this study will support the application of multimodality strategy to colorectal hepatic metastases. PMID:23051936

The impact of using computer decision-support software in primary care nurse-led telephone triage: interactional dilemmas and conversational consequences.

PubMed

Murdoch, Jamie; Barnes, Rebecca; Pooler, Jillian; Lattimer, Valerie; Fletcher, Emily; Campbell, John L

2015-02-01

Telephone triage represents one strategy to manage demand for face-to-face GP appointments in primary care. Although computer decision-support software (CDSS) is increasingly used by nurses to triage patients, little is understood about how interaction is organized in this setting. Specifically any interactional dilemmas this computer-mediated setting invokes; and how these may be consequential for communication with patients. Using conversation analytic methods we undertook a multi-modal analysis of 22 audio-recorded telephone triage nurse-caller interactions from one GP practice in England, including 10 video-recordings of nurses' use of CDSS during triage. We draw on Goffman's theoretical notion of participation frameworks to make sense of these interactions, presenting 'telling cases' of interactional dilemmas nurses faced in meeting patient's needs and accurately documenting the patient's condition within the CDSS. Our findings highlight troubles in the 'interactional workability' of telephone triage exposing difficulties faced in aligning the proximal and wider distal context that structures CDSS-mediated interactions. Patients present with diverse symptoms, understanding of triage consultations, and communication skills which nurses need to negotiate turn-by-turn with CDSS requirements. Nurses therefore need to have sophisticated communication, technological and clinical skills to ensure patients' presenting problems are accurately captured within the CDSS to determine safe triage outcomes. Dilemmas around how nurses manage and record information, and the issues of professional accountability that may ensue, raise questions about the impact of CDSS and its use in supporting nurses to deliver safe and effective patient care. Copyright © 2014 Elsevier Ltd. All rights reserved.

Implementations of the CC'01 Human-Computer Interaction Guidelines Using Bloom's Taxonomy

ERIC Educational Resources Information Center

Manaris, Bill; Wainer, Michael; Kirkpatrick, Arthur E.; Stalvey, RoxAnn H.; Shannon, Christine; Leventhal, Laura; Barnes, Julie; Wright, John; Schafer, J. Ben; Sanders, Dean

2007-01-01

In today's technology-laden society human-computer interaction (HCI) is an important knowledge area for computer scientists and software engineers. This paper surveys existing approaches to incorporate HCI into computer science (CS) and such related issues as the perceived gap between the interests of the HCI community and the needs of CS…

Computer Human Interaction for Image Information Systems.

ERIC Educational Resources Information Center

Beard, David Volk

1991-01-01

Presents an approach to developing viable image computer-human interactions (CHI) involving user metaphors for comprehending image data and methods for locating, accessing, and displaying computer images. A medical-image radiology workstation application is used as an example, and feedback and evaluation methods are discussed. (41 references) (LRW)

Personalized, relevance-based Multimodal Robotic Imaging and augmented reality for Computer Assisted Interventions.

PubMed

Navab, Nassir; Fellow, Miccai; Hennersperger, Christoph; Frisch, Benjamin; Fürst, Bernhard

2016-10-01

In the last decade, many researchers in medical image computing and computer assisted interventions across the world focused on the development of the Virtual Physiological Human (VPH), aiming at changing the practice of medicine from classification and treatment of diseases to that of modeling and treating patients. These projects resulted in major advancements in segmentation, registration, morphological, physiological and biomechanical modeling based on state of art medical imaging as well as other sensory data. However, a major issue which has not yet come into the focus is personalizing intra-operative imaging, allowing for optimal treatment. In this paper, we discuss the personalization of imaging and visualization process with particular focus on satisfying the challenging requirements of computer assisted interventions. We discuss such requirements and review a series of scientific contributions made by our research team to tackle some of these major challenges. Copyright © 2016. Published by Elsevier B.V.

Prosodic alignment in human-computer interaction

NASA Astrophysics Data System (ADS)

Suzuki, N.; Katagiri, Y.

2007-06-01

Androids that replicate humans in form also need to replicate them in behaviour to achieve a high level of believability or lifelikeness. We explore the minimal social cues that can induce in people the human tendency for social acceptance, or ethopoeia, toward artifacts, including androids. It has been observed that people exhibit a strong tendency to adjust to each other, through a number of speech and language features in human-human conversational interactions, to obtain communication efficiency and emotional engagement. We investigate in this paper the phenomena related to prosodic alignment in human-computer interactions, with particular focus on human-computer alignment of speech characteristics. We found that people exhibit unidirectional and spontaneous short-term alignment of loudness and response latency in their speech in response to computer-generated speech. We believe this phenomenon of prosodic alignment provides one of the key components for building social acceptance of androids.

Simultaneous measurement of breathing rate and heart rate using a microbend multimode fiber optic sensor

NASA Astrophysics Data System (ADS)

Chen, Zhihao; Lau, Doreen; Teo, Ju Teng; Ng, Soon Huat; Yang, Xiufeng; Kei, Pin Lin

2014-05-01

We propose and demonstrate the feasibility of using a highly sensitive microbend multimode fiber optic sensor for simultaneous measurement of breathing rate (BR) and heart rate (HR). The sensing system consists of a transceiver, microbend multimode fiber, and a computer. The transceiver is comprised of an optical transmitter, an optical receiver, and circuits for data communication with the computer via Bluetooth. Comparative experiments conducted between the sensor and predicate commercial physiologic devices showed an accuracy of ±2 bpm for both BR and HR measurement. Our preliminary study of simultaneous measurement of BR and HR in a clinical trial conducted on 11 healthy subjects during magnetic resonance imaging (MRI) also showed very good agreement with measurements obtained from conventional MR-compatible devices.

Conceptualizing, Designing, and Investigating Locative Media Use in Urban Space

NASA Astrophysics Data System (ADS)

Diamantaki, Katerina; Rizopoulos, Charalampos; Charitos, Dimitris; Kaimakamis, Nikos

This chapter investigates the social implications of locative media (LM) use and attempts to outline a theoretical framework that may support the design and implementation of location-based applications. Furthermore, it stresses the significance of physical space and location awareness as important factors that influence both human-computer interaction and computer-mediated communication. The chapter documents part of the theoretical aspect of the research undertaken as part of LOcation-based Communication Urban NETwork (LOCUNET), a project that aims to investigate the way users interact with one another (human-computer-human interaction aspect) and with the location-based system itself (human-computer interaction aspect). A number of relevant theoretical approaches are discussed in an attempt to provide a holistic theoretical background for LM use. Additionally, the actual implementation of the LOCUNET system is described and some of the findings are discussed.

The Study on Human-Computer Interaction Design Based on the Users’ Subconscious Behavior

NASA Astrophysics Data System (ADS)

Li, Lingyuan

2017-09-01

Human-computer interaction is human-centered. An excellent interaction design should focus on the study of user experience, which greatly comes from the consistence between design and human behavioral habit. However, users’ behavioral habits often result from subconsciousness. Therefore, it is smart to utilize users’ subconscious behavior to achieve design's intention and maximize the value of products’ functions, which gradually becomes a new trend in this field.

Utilizing Multi-Modal Literacies in Middle Grades Science

ERIC Educational Resources Information Center

Saurino, Dan; Ogletree, Tamra; Saurino, Penelope

2010-01-01

The nature of literacy is changing. Increased student use of computer-mediated, digital, and visual communication spans our understanding of adolescent multi-modal capabilities that reach beyond the traditional conventions of linear speech and written text in the science curriculum. Advancing technology opens doors to learning that involve…

Risk-Based Neuro-Grid Architecture for Multimodal Biometrics

NASA Astrophysics Data System (ADS)

Venkataraman, Sitalakshmi; Kulkarni, Siddhivinayak

Recent research indicates that multimodal biometrics is the way forward for a highly reliable adoption of biometric identification systems in various applications, such as banks, businesses, government and even home environments. However, such systems would require large distributed datasets with multiple computational realms spanning organisational boundaries and individual privacies.

Fast and Robust Registration of Multimodal Remote Sensing Images via Dense Orientated Gradient Feature

NASA Astrophysics Data System (ADS)

Ye, Y.

2017-09-01

This paper presents a fast and robust method for the registration of multimodal remote sensing data (e.g., optical, LiDAR, SAR and map). The proposed method is based on the hypothesis that structural similarity between images is preserved across different modalities. In the definition of the proposed method, we first develop a pixel-wise feature descriptor named Dense Orientated Gradient Histogram (DOGH), which can be computed effectively at every pixel and is robust to non-linear intensity differences between images. Then a fast similarity metric based on DOGH is built in frequency domain using the Fast Fourier Transform (FFT) technique. Finally, a template matching scheme is applied to detect tie points between images. Experimental results on different types of multimodal remote sensing images show that the proposed similarity metric has the superior matching performance and computational efficiency than the state-of-the-art methods. Moreover, based on the proposed similarity metric, we also design a fast and robust automatic registration system for multimodal images. This system has been evaluated using a pair of very large SAR and optical images (more than 20000 × 20000 pixels). Experimental results show that our system outperforms the two popular commercial software systems (i.e. ENVI and ERDAS) in both registration accuracy and computational efficiency.

Image-guided thoracic surgery in the hybrid operation room.

PubMed

Ujiie, Hideki; Effat, Andrew; Yasufuku, Kazuhiro

2017-01-01

There has been an increase in the use of image-guided technology to facilitate minimally invasive therapy. The next generation of minimally invasive therapy is focused on advancement and translation of novel image-guided technologies in therapeutic interventions, including surgery, interventional pulmonology, radiation therapy, and interventional laser therapy. To establish the efficacy of different minimally invasive therapies, we have developed a hybrid operating room, known as the guided therapeutics operating room (GTx OR) at the Toronto General Hospital. The GTx OR is equipped with multi-modality image-guidance systems, which features a dual source-dual energy computed tomography (CT) scanner, a robotic cone-beam CT (CBCT)/fluoroscopy, high-performance endobronchial ultrasound system, endoscopic surgery system, near-infrared (NIR) fluorescence imaging system, and navigation tracking systems. The novel multimodality image-guidance systems allow physicians to quickly, and accurately image patients while they are on the operating table. This yield improved outcomes since physicians are able to use image guidance during their procedures, and carry out innovative multi-modality therapeutics. Multiple preclinical translational studies pertaining to innovative minimally invasive technology is being developed in our guided therapeutics laboratory (GTx Lab). The GTx Lab is equipped with similar technology, and multimodality image-guidance systems as the GTx OR, and acts as an appropriate platform for translation of research into human clinical trials. Through the GTx Lab, we are able to perform basic research, such as the development of image-guided technologies, preclinical model testing, as well as preclinical imaging, and then translate that research into the GTx OR. This OR allows for the utilization of new technologies in cancer therapy, including molecular imaging, and other innovative imaging modalities, and therefore enables a better quality of life for patients, both during and after the procedure. In this article, we describe capabilities of the GTx systems, and discuss the first-in-human technologies used, and evaluated in GTx OR.

The Promise of Interactive Video: An Affective Search.

ERIC Educational Resources Information Center

Hon, David

1983-01-01

Argues that factors that create a feeling of interactivity in the human situation--response time, spontaneity, lack of distractors--should be included as prime elements in the design of human/machine systems, e.g., computer assisted instruction and interactive video. A computer/videodisc learning system for cardio-pulmonary resuscitation and its…

The Human-Computer Interaction of Cross-Cultural Gaming Strategy

ERIC Educational Resources Information Center

Chakraborty, Joyram; Norcio, Anthony F.; Van Der Veer, Jacob J.; Andre, Charles F.; Miller, Zachary; Regelsberger, Alexander

2015-01-01

This article explores the cultural dimensions of the human-computer interaction that underlies gaming strategies. The article is a desktop study of existing literature and is organized into five sections. The first examines the cultural aspects of knowledge processing. The social constructs technology interaction is discussed. Following this, the…

Biological characterization of preclinical Bioluminescent Osteosarcoma Orthotopic Mouse (BOOM) model: A multi-modality approach

PubMed Central

Garimella, Rama; Eskew, Jeff; Bhamidi, Priyanka; Vielhauer, George; Hong, Yan; Anderson, H. Clarke; Tawfik, Ossama; Rowe, Peter

2013-01-01

Osteosarcoma (OS) is a bone malignancy that affects children and adolescents. It is a highly aggressive tumor and typically metastasizes to lungs. Despite aggressive chemotherapy and surgical treatments, the current 5 year survival rate is 60–70%. Clinically relevant models are needed to understand OS pathobiology, metastatic progression from bones to lungs, and ultimately, to develop more efficacious treatment strategies and improve survival rates in OS patients with metastasis. The main goal of this study was to develop and characterize an in vivo OS model that will allow non-invasive tracking of tumor progression in real time, and aid in studying OS pathobiology, and screening of potential therapeutic agents against OS. In this study, we have used a multi-modality approach using bioluminescent imaging, electron microscopy, micro-computed tomography, and histopathology to develop and characterize a preclinical Bioluminescent Osteosarcoma Orthotopic Mouse (BOOM) model, using 143B human OS cell line. The results of this study clearly demonstrate that the BOOM model represents the clinical disease as evidenced by a spectrum of changes associated with tumor establishment, progression and metastasis, and detection of known OS biomarkers in the primary and metastatic tumor tissue. Key novel findings of this study include: (a) multimodality approach for extensive characterization of the BOOM model using 143B human OS cell line; (b) evidence of renal metastasis in OS orthotopic model using 143B cells; (c) evidence of Runx2 expression in the metastatic lung tissue; and (d) evidence of the presence of extracellular membrane vesicles and myofibroblasts in the BOOM model. PMID:25688332

Evaluation of an eye-pointer interaction device for human-computer interaction.

PubMed

Cáceres, Enrique; Carrasco, Miguel; Ríos, Sebastián

2018-03-01

Advances in eye-tracking technology have led to better human-computer interaction, and involve controlling a computer without any kind of physical contact. This research describes the transformation of a commercial eye-tracker for use as an alternative peripheral device in human-computer interactions, implementing a pointer that only needs the eye movements of a user facing a computer screen, thus replacing the need to control the software by hand movements. The experiment was performed with 30 test individuals who used the prototype with a set of educational videogames. The results show that, although most of the test subjects would prefer a mouse to control the pointer, the prototype tested has an empirical precision similar to that of the mouse, either when trying to control its movements or when attempting to click on a point of the screen.

Cyberpsychology: a human-interaction perspective based on cognitive modeling.

PubMed

Emond, Bruno; West, Robert L

2003-10-01

This paper argues for the relevance of cognitive modeling and cognitive architectures to cyberpsychology. From a human-computer interaction point of view, cognitive modeling can have benefits both for theory and model building, and for the design and evaluation of sociotechnical systems usability. Cognitive modeling research applied to human-computer interaction has two complimentary objectives: (1) to develop theories and computational models of human interactive behavior with information and collaborative technologies, and (2) to use the computational models as building blocks for the design, implementation, and evaluation of interactive technologies. From the perspective of building theories and models, cognitive modeling offers the possibility to anchor cyberpsychology theories and models into cognitive architectures. From the perspective of the design and evaluation of socio-technical systems, cognitive models can provide the basis for simulated users, which can play an important role in usability testing. As an example of application of cognitive modeling to technology design, the paper presents a simulation of interactive behavior with five different adaptive menu algorithms: random, fixed, stacked, frequency based, and activation based. Results of the simulation indicate that fixed menu positions seem to offer the best support for classification like tasks such as filing e-mails. This research is part of the Human-Computer Interaction, and the Broadband Visual Communication research programs at the National Research Council of Canada, in collaboration with the Carleton Cognitive Modeling Lab at Carleton University.

Simulation of the «COSMONAUT-ROBOT» System Interaction on the Lunar Surface Based on Methods of Machine Vision and Computer Graphics

NASA Astrophysics Data System (ADS)

Kryuchkov, B. I.; Usov, V. M.; Chertopolokhov, V. A.; Ronzhin, A. L.; Karpov, A. A.

2017-05-01

Extravehicular activity (EVA) on the lunar surface, necessary for the future exploration of the Moon, involves extensive use of robots. One of the factors of safe EVA is a proper interaction between cosmonauts and robots in extreme environments. This requires a simple and natural man-machine interface, e.g. multimodal contactless interface based on recognition of gestures and cosmonaut's poses. When travelling in the "Follow Me" mode (master/slave), a robot uses onboard tools for tracking cosmonaut's position and movements, and on the basis of these data builds its itinerary. The interaction in the system "cosmonaut-robot" on the lunar surface is significantly different from that on the Earth surface. For example, a man, dressed in a space suit, has limited fine motor skills. In addition, EVA is quite tiring for the cosmonauts, and a tired human being less accurately performs movements and often makes mistakes. All this leads to new requirements for the convenient use of the man-machine interface designed for EVA. To improve the reliability and stability of human-robot communication it is necessary to provide options for duplicating commands at the task stages and gesture recognition. New tools and techniques for space missions must be examined at the first stage of works in laboratory conditions, and then in field tests (proof tests at the site of application). The article analyzes the methods of detection and tracking of movements and gesture recognition of the cosmonaut during EVA, which can be used for the design of human-machine interface. A scenario for testing these methods by constructing a virtual environment simulating EVA on the lunar surface is proposed. Simulation involves environment visualization and modeling of the use of the "vision" of the robot to track a moving cosmonaut dressed in a spacesuit.

Awareware: Narrowcasting Attributes for Selective Attention, Privacy, and Multipresence

NASA Astrophysics Data System (ADS)

Cohen, Michael; Newton Fernando, Owen Noel

The domain of cscw, computer-supported collaborative work, and DSC, distributed synchronous collaboration, spans real-time interactive multiuser systems, shared information spaces, and applications for teleexistence and artificial reality, including collaborative virtual environments ( cves) (Benford et al., 2001). As presence awareness systems emerge, it is important to develop appropriate interfaces and architectures for managing multimodal multiuser systems. Especially in consideration of the persistent connectivity enabled by affordable networked communication, shared distributed environments require generalized control of media streams, techniques to control source → sink transmissions in synchronous groupware, including teleconferences and chatspaces, online role-playing games, and virtual concerts.

Design Science in Human-Computer Interaction: A Model and Three Examples

ERIC Educational Resources Information Center

Prestopnik, Nathan R.

2013-01-01

Humanity has entered an era where computing technology is virtually ubiquitous. From websites and mobile devices to computers embedded in appliances on our kitchen counters and automobiles parked in our driveways, information and communication technologies (ICTs) and IT artifacts are fundamentally changing the ways we interact with our world.…

Evaluation of a computerized aid for creating human behavioral representations of human-computer interaction.

PubMed

Williams, Kent E; Voigt, Jeffrey R

2004-01-01

The research reported herein presents the results of an empirical evaluation that focused on the accuracy and reliability of cognitive models created using a computerized tool: the cognitive analysis tool for human-computer interaction (CAT-HCI). A sample of participants, expert in interacting with a newly developed tactical display for the U.S. Army's Bradley Fighting Vehicle, individually modeled their knowledge of 4 specific tasks employing the CAT-HCI tool. Measures of the accuracy and consistency of task models created by these task domain experts using the tool were compared with task models created by a double expert. The findings indicated a high degree of consistency and accuracy between the different "single experts" in the task domain in terms of the resultant models generated using the tool. Actual or potential applications of this research include assessing human-computer interaction complexity, determining the productivity of human-computer interfaces, and analyzing an interface design to determine whether methods can be automated.

A collaborative interaction and visualization multi-modal environment for surgical planning.

PubMed

Foo, Jung Leng; Martinez-Escobar, Marisol; Peloquin, Catherine; Lobe, Thom; Winer, Eliot

2009-01-01

The proliferation of virtual reality visualization and interaction technologies has changed the way medical image data is analyzed and processed. This paper presents a multi-modal environment that combines a virtual reality application with a desktop application for collaborative surgical planning. Both visualization applications can function independently but can also be synced over a network connection for collaborative work. Any changes to either application is immediately synced and updated to the other. This is an efficient collaboration tool that allows multiple teams of doctors with only an internet connection to visualize and interact with the same patient data simultaneously. With this multi-modal environment framework, one team working in the VR environment and another team from a remote location working on a desktop machine can both collaborate in the examination and discussion for procedures such as diagnosis, surgical planning, teaching and tele-mentoring.

On the Benefits of Multimodal Annotations for Vocabulary Uptake from Reading

ERIC Educational Resources Information Center

Boers, Frank; Warren, Paul; Grimshaw, Gina; Siyanova-Chanturia, Anna

2017-01-01

Several research articles published in the realm of Computer Assisted Language Learning (CALL) have reported evidence of the benefits of multimodal annotations, i.e. the provision of pictorial as well as verbal clarifications, for vocabulary uptake from reading. Almost invariably, these publications account for the observed benefits with reference…

Sharing Solutions: Persistence and Grounding in Multimodal Collaborative Problem Solving

ERIC Educational Resources Information Center

Dillenbourg, Pierre; Traum, David

2006-01-01

This article reports on an exploratory study of the relationship between grounding and problem solving in multimodal computer-mediated collaboration. This article examines two different media, a shared whiteboard and a MOO environment that includes a text chat facility. A study was done on how the acknowledgment rate (how often partners give…

MCA-NMF: Multimodal Concept Acquisition with Non-Negative Matrix Factorization

PubMed Central

Mangin, Olivier; Filliat, David; ten Bosch, Louis; Oudeyer, Pierre-Yves

2015-01-01

In this paper we introduce MCA-NMF, a computational model of the acquisition of multimodal concepts by an agent grounded in its environment. More precisely our model finds patterns in multimodal sensor input that characterize associations across modalities (speech utterances, images and motion). We propose this computational model as an answer to the question of how some class of concepts can be learnt. In addition, the model provides a way of defining such a class of plausibly learnable concepts. We detail why the multimodal nature of perception is essential to reduce the ambiguity of learnt concepts as well as to communicate about them through speech. We then present a set of experiments that demonstrate the learning of such concepts from real non-symbolic data consisting of speech sounds, images, and motions. Finally we consider structure in perceptual signals and demonstrate that a detailed knowledge of this structure, named compositional understanding can emerge from, instead of being a prerequisite of, global understanding. An open-source implementation of the MCA-NMF learner as well as scripts and associated experimental data to reproduce the experiments are publicly available. PMID:26489021

Gestural and symbolic development among apes and humans: support for a multimodal theory of language evolution.

PubMed

Gillespie-Lynch, Kristen; Greenfield, Patricia M; Lyn, Heidi; Savage-Rumbaugh, Sue

2014-01-01

What are the implications of similarities and differences in the gestural and symbolic development of apes and humans?This focused review uses as a starting point our recent study that provided evidence that gesture supported the symbolic development of a chimpanzee, a bonobo, and a human child reared in language-enriched environments at comparable stages of communicative development. These three species constitute a complete clade, species possessing a common immediate ancestor. Communicative behaviors observed among all species in a clade are likely to have been present in the common ancestor. Similarities in the form and function of many gestures produced by the chimpanzee, bonobo, and human child suggest that shared non-verbal skills may underlie shared symbolic capacities. Indeed, an ontogenetic sequence from gesture to symbol was present across the clade but more pronounced in child than ape. Multimodal expressions of communicative intent (e.g., vocalization plus persistence or eye-contact) were normative for the child, but less common for the apes. These findings suggest that increasing multimodal expression of communicative intent may have supported the emergence of language among the ancestors of humans. Therefore, this focused review includes new studies, since our 2013 article, that support a multimodal theory of language evolution.

Gestural and symbolic development among apes and humans: support for a multimodal theory of language evolution

PubMed Central

Gillespie-Lynch, Kristen; Greenfield, Patricia M.; Lyn, Heidi; Savage-Rumbaugh, Sue

2014-01-01

What are the implications of similarities and differences in the gestural and symbolic development of apes and humans?This focused review uses as a starting point our recent study that provided evidence that gesture supported the symbolic development of a chimpanzee, a bonobo, and a human child reared in language-enriched environments at comparable stages of communicative development. These three species constitute a complete clade, species possessing a common immediate ancestor. Communicative behaviors observed among all species in a clade are likely to have been present in the common ancestor. Similarities in the form and function of many gestures produced by the chimpanzee, bonobo, and human child suggest that shared non-verbal skills may underlie shared symbolic capacities. Indeed, an ontogenetic sequence from gesture to symbol was present across the clade but more pronounced in child than ape. Multimodal expressions of communicative intent (e.g., vocalization plus persistence or eye-contact) were normative for the child, but less common for the apes. These findings suggest that increasing multimodal expression of communicative intent may have supported the emergence of language among the ancestors of humans. Therefore, this focused review includes new studies, since our 2013 article, that support a multimodal theory of language evolution. PMID:25400607

Two-photon quantum walk in a multimode fiber

PubMed Central

Defienne, Hugo; Barbieri, Marco; Walmsley, Ian A.; Smith, Brian J.; Gigan, Sylvain

2016-01-01

Multiphoton propagation in connected structures—a quantum walk—offers the potential of simulating complex physical systems and provides a route to universal quantum computation. Increasing the complexity of quantum photonic networks where the walk occurs is essential for many applications. We implement a quantum walk of indistinguishable photon pairs in a multimode fiber supporting 380 modes. Using wavefront shaping, we control the propagation of the two-photon state through the fiber in which all modes are coupled. Excitation of arbitrary output modes of the system is realized by controlling classical and quantum interferences. This report demonstrates a highly multimode platform for multiphoton interference experiments and provides a powerful method to program a general high-dimensional multiport optical circuit. This work paves the way for the next generation of photonic devices for quantum simulation, computing, and communication. PMID:27152325

Pedagogical Agents as Learning Companions: The Impact of Agent Emotion and Gender

ERIC Educational Resources Information Center

Kim, Yanghee; Baylor, A. L.; Shen, E.

2007-01-01

The potential of emotional interaction between human and computer has recently interested researchers in human-computer interaction. The instructional impact of this interaction in learning environments has not been established, however. This study examined the impact of emotion and gender of a pedagogical agent as a learning companion (PAL) on…

Evaluation of protein adsorption and preferred binding regions in multimodal chromatography using NMR

PubMed Central

Chung, Wai Keen; Freed, Alexander S.; Holstein, Melissa A.; McCallum, Scott A.; Cramer, Steven M.

2010-01-01

NMR titration experiments with labeled human ubiquitin were employed in concert with chromatographic data obtained with a library of ubiquitin mutants to study the nature of protein adsorption in multimodal (MM) chromatography. The elution order of the mutants on the MM resin was significantly different from that obtained by ion-exchange chromatography. Further, the chromatographic results with the protein library indicated that mutations in a defined region induced greater changes in protein affinity to the solid support. Chemical shift mapping and determination of dissociation constants from NMR titration experiments with the MM ligand and isotopically enriched ubiquitin were used to determine and rank the relative binding affinities of interaction sites on the protein surface. The results with NMR confirmed that the protein possessed a distinct preferred binding region for the MM ligand in agreement with the chromatographic results. Finally, coarse-grained ligand docking simulations were employed to study the modes of interaction between the MM ligand and ubiquitin. The use of NMR titration experiments in concert with chromatographic data obtained with protein libraries represents a previously undescribed approach for elucidating the structural basis of protein binding affinity in MM chromatographic systems. PMID:20837551

Computer-aided, multi-modal, and compression diffuse optical studies of breast tissue

NASA Astrophysics Data System (ADS)

Busch, David Richard, Jr.

Diffuse Optical Tomography and Spectroscopy permit measurement of important physiological parameters non-invasively through ˜10 cm of tissue. I have applied these techniques in measurements of human breast and breast cancer. My thesis integrates three loosely connected themes in this context: multi-modal breast cancer imaging, automated data analysis of breast cancer images, and microvascular hemodynamics of breast under compression. As per the first theme, I describe construction, testing, and the initial clinical usage of two generations of imaging systems for simultaneous diffuse optical and magnetic resonance imaging. The second project develops a statistical analysis of optical breast data from many spatial locations in a population of cancers to derive a novel optical signature of malignancy; I then apply this data-derived signature for localization of cancer in additional subjects. Finally, I construct and deploy diffuse optical instrumentation to measure blood content and blood flow during breast compression; besides optics, this research has implications for any method employing breast compression, e.g., mammography.

The eXperience Induction Machine: A New Paradigm for Mixed-Reality Interaction Design and Psychological Experimentation

NASA Astrophysics Data System (ADS)

Bernardet, Ulysses; Bermúdez I Badia, Sergi; Duff, Armin; Inderbitzin, Martin; Le Groux, Sylvain; Manzolli, Jônatas; Mathews, Zenon; Mura, Anna; Väljamäe, Aleksander; Verschure, Paul F. M. J.

The eXperience Induction Machine (XIM) is one of the most advanced mixed-reality spaces available today. XIM is an immersive space that consists of physical sensors and effectors and which is conceptualized as a general-purpose infrastructure for research in the field of psychology and human-artifact interaction. In this chapter, we set out the epistemological rational behind XIM by putting the installation in the context of psychological research. The design and implementation of XIM are based on principles and technologies of neuromorphic control. We give a detailed description of the hardware infrastructure and software architecture, including the logic of the overall behavioral control. To illustrate the approach toward psychological experimentation, we discuss a number of practical applications of XIM. These include the so-called, persistent virtual community, the application in the research of the relationship between human experience and multi-modal stimulation, and an investigation of a mixed-reality social interaction paradigm.

Adaptive multimodal interaction in mobile augmented reality: A conceptual framework

NASA Astrophysics Data System (ADS)

Abidin, Rimaniza Zainal; Arshad, Haslina; Shukri, Saidatul A'isyah Ahmad

2017-10-01

Recently, Augmented Reality (AR) is an emerging technology in many mobile applications. Mobile AR was defined as a medium for displaying information merged with the real world environment mapped with augmented reality surrounding in a single view. There are four main types of mobile augmented reality interfaces and one of them are multimodal interfaces. Multimodal interface processes two or more combined user input modes (such as speech, pen, touch, manual gesture, gaze, and head and body movements) in a coordinated manner with multimedia system output. In multimodal interface, many frameworks have been proposed to guide the designer to develop a multimodal applications including in augmented reality environment but there has been little work reviewing the framework of adaptive multimodal interface in mobile augmented reality. The main goal of this study is to propose a conceptual framework to illustrate the adaptive multimodal interface in mobile augmented reality. We reviewed several frameworks that have been proposed in the field of multimodal interfaces, adaptive interface and augmented reality. We analyzed the components in the previous frameworks and measure which can be applied in mobile devices. Our framework can be used as a guide for designers and developer to develop a mobile AR application with an adaptive multimodal interfaces.

ATPP: A Pipeline for Automatic Tractography-Based Brain Parcellation

PubMed Central

Li, Hai; Fan, Lingzhong; Zhuo, Junjie; Wang, Jiaojian; Zhang, Yu; Yang, Zhengyi; Jiang, Tianzi

2017-01-01

There is a longstanding effort to parcellate brain into areas based on micro-structural, macro-structural, or connectional features, forming various brain atlases. Among them, connectivity-based parcellation gains much emphasis, especially with the considerable progress of multimodal magnetic resonance imaging in the past two decades. The Brainnetome Atlas published recently is such an atlas that follows the framework of connectivity-based parcellation. However, in the construction of the atlas, the deluge of high resolution multimodal MRI data and time-consuming computation poses challenges and there is still short of publically available tools dedicated to parcellation. In this paper, we present an integrated open source pipeline (https://www.nitrc.org/projects/atpp), named Automatic Tractography-based Parcellation Pipeline (ATPP) to realize the framework of parcellation with automatic processing and massive parallel computing. ATPP is developed to have a powerful and flexible command line version, taking multiple regions of interest as input, as well as a user-friendly graphical user interface version for parcellating single region of interest. We demonstrate the two versions by parcellating two brain regions, left precentral gyrus and middle frontal gyrus, on two independent datasets. In addition, ATPP has been successfully utilized and fully validated in a variety of brain regions and the human Brainnetome Atlas, showing the capacity to greatly facilitate brain parcellation. PMID:28611620

Online mentalising investigated with functional MRI.

PubMed

Kircher, Tilo; Blümel, Isabelle; Marjoram, Dominic; Lataster, Tineke; Krabbendam, Lydia; Weber, Jochen; van Os, Jim; Krach, Sören

2009-05-01

For successful interpersonal communication, inferring intentions, goals or desires of others is highly advantageous. Increasingly, humans also interact with computers or robots. In this study, we sought to determine to what degree an interactive task, which involves receiving feedback from social partners that can be used to infer intent, engaged the medial prefrontal cortex, a region previously associated with Theory of Mind processes among others. Participants were scanned using fMRI as they played an adapted version of the Prisoner's Dilemma Game with alleged human and computer partners who were outside the scanner. The medial frontal cortex was activated when both human and computer partner were played, while the direct contrast revealed significantly stronger signal change during the human-human interaction. The results suggest a link between activity in the medial prefrontal cortex and the partner played in a mentalising task. This signal change was also present for to the computers partner. Implying agency or a will to non-human actors might be an innate human resource that could lead to an evolutionary advantage.

Effects of Sex Steroids in the Human Brain.

PubMed

Nguyen, Tuong-Vi; Ducharme, Simon; Karama, Sherif

2017-11-01

Sex steroids are thought to play a critical developmental role in shaping both cortical and subcortical structures in the human brain. Periods of profound changes in sex steroids invariably coincide with the onset of sex differences in mental health vulnerability, highlighting the importance of sex steroids in determining sexual differentiation of the brain. Yet, most of the evidence for the central effects of sex steroids relies on non-human studies, as several challenges have limited our understanding of these effects in humans: the lack of systematic assessment of the human sex steroid metabolome, the different developmental trajectories of specific sex steroids, the impact of genetic variation and epigenetic changes, and the plethora of interactions between sex steroids, sex chromosomes, neurotransmitters, and other hormonal systems. Here we review how multimodal strategies may be employed to bridge the gap between the basic and clinical understanding of sex steroid-related changes in the human brain.

Computer modeling and simulation of human movement. Applications in sport and rehabilitation.

PubMed

Neptune, R R

2000-05-01

Computer modeling and simulation of human movement plays an increasingly important role in sport and rehabilitation, with applications ranging from sport equipment design to understanding pathologic gait. The complex dynamic interactions within the musculoskeletal and neuromuscular systems make analyzing human movement with existing experimental techniques difficult but computer modeling and simulation allows for the identification of these complex interactions and causal relationships between input and output variables. This article provides an overview of computer modeling and simulation and presents an example application in the field of rehabilitation.

The Human-Computer Interface and Information Literacy: Some Basics and Beyond.

ERIC Educational Resources Information Center

Church, Gary M.

1999-01-01

Discusses human/computer interaction research, human/computer interface, and their relationships to information literacy. Highlights include communication models; cognitive perspectives; task analysis; theory of action; problem solving; instructional design considerations; and a suggestion that human/information interface may be a more appropriate…

Enhancing Human-Computer Interaction Design Education: Teaching Affordance Design for Emerging Mobile Devices

ERIC Educational Resources Information Center

Faiola, Anthony; Matei, Sorin Adam

2010-01-01

The evolution of human-computer interaction design (HCID) over the last 20 years suggests that there is a growing need for educational scholars to consider new and more applicable theoretical models of interactive product design. The authors suggest that such paradigms would call for an approach that would equip HCID students with a better…

Integrating HCI into IDT: Charting the Human Computer Interaction Competencies Necessary for Instructional Media Production Coursework

ERIC Educational Resources Information Center

Brown, Abbie; Sugar, William

2004-01-01

A report on the efforts made to describe the range of human-computer interaction skills necessary to complete a program of study in Instructional Design Technology. Educators responsible for instructional media production courses have not yet articulated which among the wide range of possible interactions students must master for instructional…

On the feasibility of concurrent human TMS-EEG-fMRI measurements

PubMed Central

Reithler, Joel; Schuhmann, Teresa; de Graaf, Tom; Uludağ, Kâmil; Goebel, Rainer; Sack, Alexander T.

2013-01-01

Simultaneously combining the complementary assets of EEG, functional MRI (fMRI), and transcranial magnetic stimulation (TMS) within one experimental session provides synergetic results, offering insights into brain function that go beyond the scope of each method when used in isolation. The steady increase of concurrent EEG-fMRI, TMS-EEG, and TMS-fMRI studies further underlines the added value of such multimodal imaging approaches. Whereas concurrent EEG-fMRI enables monitoring of brain-wide network dynamics with high temporal and spatial resolution, the combination with TMS provides insights in causal interactions within these networks. Thus the simultaneous use of all three methods would allow studying fast, spatially accurate, and distributed causal interactions in the perturbed system and its functional relevance for intact behavior. Concurrent EEG-fMRI, TMS-EEG, and TMS-fMRI experiments are already technically challenging, and the three-way combination of TMS-EEG-fMRI might yield additional difficulties in terms of hardware strain or signal quality. The present study explored the feasibility of concurrent TMS-EEG-fMRI studies by performing safety and quality assurance tests based on phantom and human data combining existing commercially available hardware. Results revealed that combined TMS-EEG-fMRI measurements were technically feasible, safe in terms of induced temperature changes, allowed functional MRI acquisition with comparable image quality as during concurrent EEG-fMRI or TMS-fMRI, and provided artifact-free EEG before and from 300 ms after TMS pulse application. Based on these empirical findings, we discuss the conceptual benefits of this novel complementary approach to investigate the working human brain and list a number of precautions and caveats to be heeded when setting up such multimodal imaging facilities with current hardware. PMID:23221407

Why we interact: on the functional role of the striatum in the subjective experience of social interaction.

PubMed

Pfeiffer, Ulrich J; Schilbach, Leonhard; Timmermans, Bert; Kuzmanovic, Bojana; Georgescu, Alexandra L; Bente, Gary; Vogeley, Kai

2014-11-01

There is ample evidence that human primates strive for social contact and experience interactions with conspecifics as intrinsically rewarding. Focusing on gaze behavior as a crucial means of human interaction, this study employed a unique combination of neuroimaging, eye-tracking, and computer-animated virtual agents to assess the neural mechanisms underlying this component of behavior. In the interaction task, participants believed that during each interaction the agent's gaze behavior could either be controlled by another participant or by a computer program. Their task was to indicate whether they experienced a given interaction as an interaction with another human participant or the computer program based on the agent's reaction. Unbeknownst to them, the agent was always controlled by a computer to enable a systematic manipulation of gaze reactions by varying the degree to which the agent engaged in joint attention. This allowed creating a tool to distinguish neural activity underlying the subjective experience of being engaged in social and non-social interaction. In contrast to previous research, this allows measuring neural activity while participants experience active engagement in real-time social interactions. Results demonstrate that gaze-based interactions with a perceived human partner are associated with activity in the ventral striatum, a core component of reward-related neurocircuitry. In contrast, interactions with a computer-driven agent activate attention networks. Comparisons of neural activity during interaction with behaviorally naïve and explicitly cooperative partners demonstrate different temporal dynamics of the reward system and indicate that the mere experience of engagement in social interaction is sufficient to recruit this system. Copyright © 2014 Elsevier Inc. All rights reserved.

Multimodality as a Sociolinguistic Resource

ERIC Educational Resources Information Center

Collister, Lauren Brittany

2013-01-01

This work explores the use of multimodal communication in a community of expert "World of Warcraft"® players and its impact on politeness, identity, and relationships. Players in the community regularly communicated using three linguistic modes quasi-simultaneously: text chat, voice chat, and face-to-face interaction. Using the…

A multimodal approach to estimating vigilance using EEG and forehead EOG

NASA Astrophysics Data System (ADS)

Zheng, Wei-Long; Lu, Bao-Liang

2017-04-01

Objective. Covert aspects of ongoing user mental states provide key context information for user-aware human computer interactions. In this paper, we focus on the problem of estimating the vigilance of users using EEG and EOG signals. Approach. The PERCLOS index as vigilance annotation is obtained from eye tracking glasses. To improve the feasibility and wearability of vigilance estimation devices for real-world applications, we adopt a novel electrode placement for forehead EOG and extract various eye movement features, which contain the principal information of traditional EOG. We explore the effects of EEG from different brain areas and combine EEG and forehead EOG to leverage their complementary characteristics for vigilance estimation. Considering that the vigilance of users is a dynamic changing process because the intrinsic mental states of users involve temporal evolution, we introduce continuous conditional neural field and continuous conditional random field models to capture dynamic temporal dependency. Main results. We propose a multimodal approach to estimating vigilance by combining EEG and forehead EOG and incorporating the temporal dependency of vigilance into model training. The experimental results demonstrate that modality fusion can improve the performance compared with a single modality, EOG and EEG contain complementary information for vigilance estimation, and the temporal dependency-based models can enhance the performance of vigilance estimation. From the experimental results, we observe that theta and alpha frequency activities are increased, while gamma frequency activities are decreased in drowsy states in contrast to awake states. Significance. The forehead setup allows for the simultaneous collection of EEG and EOG and achieves comparative performance using only four shared electrodes in comparison with the temporal and posterior sites.

Parallel workflow tools to facilitate human brain MRI post-processing

PubMed Central

Cui, Zaixu; Zhao, Chenxi; Gong, Gaolang

2015-01-01

Multi-modal magnetic resonance imaging (MRI) techniques are widely applied in human brain studies. To obtain specific brain measures of interest from MRI datasets, a number of complex image post-processing steps are typically required. Parallel workflow tools have recently been developed, concatenating individual processing steps and enabling fully automated processing of raw MRI data to obtain the final results. These workflow tools are also designed to make optimal use of available computational resources and to support the parallel processing of different subjects or of independent processing steps for a single subject. Automated, parallel MRI post-processing tools can greatly facilitate relevant brain investigations and are being increasingly applied. In this review, we briefly summarize these parallel workflow tools and discuss relevant issues. PMID:26029043

A novel automated method for doing registration and 3D reconstruction from multi-modal RGB/IR image sequences

NASA Astrophysics Data System (ADS)

Kirby, Richard; Whitaker, Ross

2016-09-01

In recent years, the use of multi-modal camera rigs consisting of an RGB sensor and an infrared (IR) sensor have become increasingly popular for use in surveillance and robotics applications. The advantages of using multi-modal camera rigs include improved foreground/background segmentation, wider range of lighting conditions under which the system works, and richer information (e.g. visible light and heat signature) for target identification. However, the traditional computer vision method of mapping pairs of images using pixel intensities or image features is often not possible with an RGB/IR image pair. We introduce a novel method to overcome the lack of common features in RGB/IR image pairs by using a variational methods optimization algorithm to map the optical flow fields computed from different wavelength images. This results in the alignment of the flow fields, which in turn produce correspondences similar to those found in a stereo RGB/RGB camera rig using pixel intensities or image features. In addition to aligning the different wavelength images, these correspondences are used to generate dense disparity and depth maps. We obtain accuracies similar to other multi-modal image alignment methodologies as long as the scene contains sufficient depth variations, although a direct comparison is not possible because of the lack of standard image sets from moving multi-modal camera rigs. We test our method on synthetic optical flow fields and on real image sequences that we created with a multi-modal binocular stereo RGB/IR camera rig. We determine our method's accuracy by comparing against a ground truth.

Multimode and single-mode fibers for data center and high-performance computing applications

NASA Astrophysics Data System (ADS)

Bickham, Scott R.

2016-03-01

Data center (DC) and high performance computing (HPC) applications have traditionally used a combination of copper, multimode fiber and single-mode fiber interconnects with relative percentages that depend on factors such as the line rate, reach and connectivity costs. The balance between these transmission media has increasingly shifted towards optical fiber due to the reach constraints of copper at data rates of 10 Gb/s and higher. The percentage of single-mode fiber deployed in the DC has also grown slightly since 2014, coinciding with the emergence of mega DCs with extended distance needs beyond 100 m. This trend will likely continue in the next few years as DCs expand their capacity from 100G to 400G, increase the physical size of their facilities and begin to utilize silicon-photonics transceiver technology. However there is a still a need for the low-cost and high-density connectivity, and this is sustaining the deployment of multimode fiber for links <= 100 m. In this paper, we discuss options for single-mode and multimode fibers in DCs and HPCs and introduce a reduced diameter multimode fiber concept which provides intra-and inter-rack connectivity as well as compatibility with silicon-photonic transceivers operating at 1310 nm. We also discuss the trade-offs between single-mode fiber attributes such as bend-insensitivity, attenuation and mode field diameter and their roles in capacity and connectivity in data centers.

Novel design of interactive multimodal biofeedback system for neurorehabilitation.

PubMed

Huang, He; Chen, Y; Xu, W; Sundaram, H; Olson, L; Ingalls, T; Rikakis, T; He, Jiping

2006-01-01

A previous design of a biofeedback system for Neurorehabilitation in an interactive multimodal environment has demonstrated the potential of engaging stroke patients in task-oriented neuromotor rehabilitation. This report explores the new concept and alternative designs of multimedia based biofeedback systems. In this system, the new interactive multimodal environment was constructed with abstract presentation of movement parameters. Scenery images or pictures and their clarity and orientation are used to reflect the arm movement and relative position to the target instead of the animated arm. The multiple biofeedback parameters were classified into different hierarchical levels w.r.t. importance of each movement parameter to performance. A new quantified measurement for these parameters were developed to assess the patient's performance both real-time and offline. These parameters were represented by combined visual and auditory presentations with various distinct music instruments. Overall, the objective of newly designed system is to explore what information and how to feedback information in interactive virtual environment could enhance the sensorimotor integration that may facilitate the efficient design and application of virtual environment based therapeutic intervention.

Multimode optical dermoscopy (SkinSpect) analysis for skin with melanocytic nevus

NASA Astrophysics Data System (ADS)

Vasefi, Fartash; MacKinnon, Nicholas; Saager, Rolf; Kelly, Kristen M.; Maly, Tyler; Chave, Robert; Booth, Nicholas; Durkin, Anthony J.; Farkas, Daniel L.

2016-04-01

We have developed a multimode dermoscope (SkinSpect™) capable of illuminating human skin samples in-vivo with spectrally-programmable linearly-polarized light at 33 wavelengths between 468nm and 857 nm. Diffusely reflected photons are separated into collinear and cross-polarized image paths and images captured for each illumination wavelength. In vivo human skin nevi (N = 20) were evaluated with the multimode dermoscope and melanin and hemoglobin concentrations were compared with Spatially Modulated Quantitative Spectroscopy (SMoQS) measurements. Both systems show low correlation between their melanin and hemoglobin concentrations, demonstrating the ability of the SkinSpect™ to separate these molecular signatures and thus act as a biologically plausible device capable of early onset melanoma detection.

Toward Usable Interactive Analytics: Coupling Cognition and Computation

DOE Office of Scientific and Technical Information (OSTI.GOV)

Endert, Alexander; North, Chris; Chang, Remco

Interactive analytics provide users a myriad of computational means to aid in extracting meaningful information from large and complex datasets. Much prior work focuses either on advancing the capabilities of machine-centric approaches by the data mining and machine learning communities, or human-driven methods by the visualization and CHI communities. However, these methods do not yet support a true human-machine symbiotic relationship where users and machines work together collaboratively and adapt to each other to advance an interactive analytic process. In this paper we discuss some of the inherent issues, outlining what we believe are the steps toward usable interactive analyticsmore » that will ultimately increase the effectiveness for both humans and computers to produce insights.« less

Hierarchical Spatial Concept Formation Based on Multimodal Information for Human Support Robots.

PubMed

Hagiwara, Yoshinobu; Inoue, Masakazu; Kobayashi, Hiroyoshi; Taniguchi, Tadahiro

2018-01-01

In this paper, we propose a hierarchical spatial concept formation method based on the Bayesian generative model with multimodal information e.g., vision, position and word information. Since humans have the ability to select an appropriate level of abstraction according to the situation and describe their position linguistically, e.g., "I am in my home" and "I am in front of the table," a hierarchical structure of spatial concepts is necessary in order for human support robots to communicate smoothly with users. The proposed method enables a robot to form hierarchical spatial concepts by categorizing multimodal information using hierarchical multimodal latent Dirichlet allocation (hMLDA). Object recognition results using convolutional neural network (CNN), hierarchical k-means clustering result of self-position estimated by Monte Carlo localization (MCL), and a set of location names are used, respectively, as features in vision, position, and word information. Experiments in forming hierarchical spatial concepts and evaluating how the proposed method can predict unobserved location names and position categories are performed using a robot in the real world. Results verify that, relative to comparable baseline methods, the proposed method enables a robot to predict location names and position categories closer to predictions made by humans. As an application example of the proposed method in a home environment, a demonstration in which a human support robot moves to an instructed place based on human speech instructions is achieved based on the formed hierarchical spatial concept.

Hierarchical Spatial Concept Formation Based on Multimodal Information for Human Support Robots

PubMed Central

Hagiwara, Yoshinobu; Inoue, Masakazu; Kobayashi, Hiroyoshi; Taniguchi, Tadahiro

2018-01-01

In this paper, we propose a hierarchical spatial concept formation method based on the Bayesian generative model with multimodal information e.g., vision, position and word information. Since humans have the ability to select an appropriate level of abstraction according to the situation and describe their position linguistically, e.g., “I am in my home” and “I am in front of the table,” a hierarchical structure of spatial concepts is necessary in order for human support robots to communicate smoothly with users. The proposed method enables a robot to form hierarchical spatial concepts by categorizing multimodal information using hierarchical multimodal latent Dirichlet allocation (hMLDA). Object recognition results using convolutional neural network (CNN), hierarchical k-means clustering result of self-position estimated by Monte Carlo localization (MCL), and a set of location names are used, respectively, as features in vision, position, and word information. Experiments in forming hierarchical spatial concepts and evaluating how the proposed method can predict unobserved location names and position categories are performed using a robot in the real world. Results verify that, relative to comparable baseline methods, the proposed method enables a robot to predict location names and position categories closer to predictions made by humans. As an application example of the proposed method in a home environment, a demonstration in which a human support robot moves to an instructed place based on human speech instructions is achieved based on the formed hierarchical spatial concept. PMID:29593521

Applying Spatial Audio to Human Interfaces: 25 Years of NASA Experience

NASA Technical Reports Server (NTRS)

Begault, Durand R.; Wenzel, Elizabeth M.; Godfrey, Martine; Miller, Joel D.; Anderson, Mark R.

2010-01-01

From the perspective of human factors engineering, the inclusion of spatial audio within a human-machine interface is advantageous from several perspectives. Demonstrated benefits include the ability to monitor multiple streams of speech and non-speech warning tones using a cocktail party advantage, and for aurally-guided visual search. Other potential benefits include the spatial coordination and interaction of multimodal events, and evaluation of new communication technologies and alerting systems using virtual simulation. Many of these technologies were developed at NASA Ames Research Center, beginning in 1985. This paper reviews examples and describes the advantages of spatial sound in NASA-related technologies, including space operations, aeronautics, and search and rescue. The work has involved hardware and software development as well as basic and applied research.

Multi-modality molecular imaging: pre-clinical laboratory configuration

NASA Astrophysics Data System (ADS)

Wu, Yanjun; Wellen, Jeremy W.; Sarkar, Susanta K.

2006-02-01

In recent years, the prevalence of in vivo molecular imaging applications has rapidly increased. Here we report on the construction of a multi-modality imaging facility in a pharmaceutical setting that is expected to further advance existing capabilities for in vivo imaging of drug distribution and the interaction with their target. The imaging instrumentation in our facility includes a microPET scanner, a four wavelength time-domain optical imaging scanner, a 9.4T/30cm MRI scanner and a SPECT/X-ray CT scanner. An electronics shop and a computer room dedicated to image analysis are additional features of the facility. The layout of the facility was designed with a central animal preparation room surrounded by separate laboratory rooms for each of the major imaging modalities to accommodate the work-flow of simultaneous in vivo imaging experiments. This report will focus on the design of and anticipated applications for our microPET and optical imaging laboratory spaces. Additionally, we will discuss efforts to maximize the daily throughput of animal scans through development of efficient experimental work-flows and the use of multiple animals in a single scanning session.

Molecular simulations of multimodal ligand-protein binding: elucidation of binding sites and correlation with experiments.

PubMed

Freed, Alexander S; Garde, Shekhar; Cramer, Steven M

2011-11-17

Multimodal chromatography, which employs more than one mode of interaction between ligands and proteins, has been shown to have unique selectivity and high efficacy for protein purification. To test the ability of free solution molecular dynamics (MD) simulations in explicit water to identify binding regions on the protein surface and to shed light on the "pseudo affinity" nature of multimodal interactions, we performed MD simulations of a model protein ubiquitin in aqueous solution of free ligands. Comparisons of MD with NMR spectroscopy of ubiquitin mutants in solutions of free ligands show a good agreement between the two with regard to the preferred binding region on the surface of the protein and several binding sites. MD simulations also identify additional binding sites that were not observed in the NMR experiments. "Bound" ligands were found to be sufficiently flexible and to access a number of favorable conformations, suggesting only a moderate loss of ligand entropy in the "pseudo affinity" binding of these multimodal ligands. Analysis of locations of chemical subunits of the ligand on the protein surface indicated that electrostatic interaction units were located on the periphery of the preferred binding region on the protein. The analysis of the electrostatic potential, the hydrophobicity maps, and the binding of both acetate and benzene probes were used to further study the localization of individual ligand moieties. These results suggest that water-mediated electrostatic interactions help the localization and orientation of the MM ligand to the binding region with additional stability provided by nonspecific hydrophobic interactions.

Parkinson Patients' Initial Trust in Avatars: Theory and Evidence.

PubMed

Javor, Andrija; Ransmayr, Gerhard; Struhal, Walter; Riedl, René

2016-01-01

Parkinson's disease (PD) is a neurodegenerative disease that affects the motor system and cognitive and behavioral functions. Due to these impairments, PD patients also have problems in using the computer. However, using computers and the Internet could help these patients to overcome social isolation and enhance information search. Specifically, avatars (defined as virtual representations of humans) are increasingly used in online environments to enhance human-computer interaction by simulating face-to-face interaction. Our laboratory experiment investigated how PD patients behave in a trust game played with human and avatar counterparts, and we compared this behavior to the behavior of age, income, education and gender matched healthy controls. The results of our study show that PD patients trust avatar faces significantly more than human faces. Moreover, there was no significant difference between initial trust of PD patients and healthy controls in avatar faces, while PD patients trusted human faces significantly less than healthy controls. Our data suggests that PD patients' interaction with avatars may constitute an effective way of communication in situations in which trust is required (e.g., a physician recommends intake of medication). We discuss the implications of these results for several areas of human-computer interaction and neurological research.

Parkinson Patients’ Initial Trust in Avatars: Theory and Evidence

PubMed Central

Javor, Andrija; Ransmayr, Gerhard; Struhal, Walter; Riedl, René

2016-01-01

Parkinson’s disease (PD) is a neurodegenerative disease that affects the motor system and cognitive and behavioral functions. Due to these impairments, PD patients also have problems in using the computer. However, using computers and the Internet could help these patients to overcome social isolation and enhance information search. Specifically, avatars (defined as virtual representations of humans) are increasingly used in online environments to enhance human-computer interaction by simulating face-to-face interaction. Our laboratory experiment investigated how PD patients behave in a trust game played with human and avatar counterparts, and we compared this behavior to the behavior of age, income, education and gender matched healthy controls. The results of our study show that PD patients trust avatar faces significantly more than human faces. Moreover, there was no significant difference between initial trust of PD patients and healthy controls in avatar faces, while PD patients trusted human faces significantly less than healthy controls. Our data suggests that PD patients’ interaction with avatars may constitute an effective way of communication in situations in which trust is required (e.g., a physician recommends intake of medication). We discuss the implications of these results for several areas of human-computer interaction and neurological research. PMID:27820864

Human-Computer Interaction in Smart Environments

PubMed Central

Paravati, Gianluca; Gatteschi, Valentina

2015-01-01

Here, we provide an overview of the content of the Special Issue on “Human-computer interaction in smart environments”. The aim of this Special Issue is to highlight technologies and solutions encompassing the use of mass-market sensors in current and emerging applications for interacting with Smart Environments. Selected papers address this topic by analyzing different interaction modalities, including hand/body gestures, face recognition, gaze/eye tracking, biosignal analysis, speech and activity recognition, and related issues.

Real-time non-invasive eyetracking and gaze-point determination for human-computer interaction and biomedicine

NASA Technical Reports Server (NTRS)

Talukder, Ashit; Morookian, John-Michael; Monacos, S.; Lam, R.; Lebaw, C.; Bond, A.

2004-01-01

Eyetracking is one of the latest technologies that has shown potential in several areas including human-computer interaction for people with and without disabilities, and for noninvasive monitoring, detection, and even diagnosis of physiological and neurological problems in individuals.

Computationally Guided Photothermal Tumor Therapy Using Long-Circulating Gold Nanorod Antennas

PubMed Central

Maltzahn, Geoffrey von; Park, Ji-Ho; Agrawal, Amit; Bandaru, Nanda Kishor; Das, Sarit K.; Sailor, Michael J.; Bhatia, Sangeeta N.

2009-01-01

Plasmonic nanomaterials have the opportunity to considerably improve the specificity of cancer ablation by i.v. homing to tumors and acting as antennas for accepting externally applied energy. Here, we describe an integrated approach to improved plasmonic therapy composed of multimodal nanomaterial optimization and computational irradiation protocol development. We synthesized polyethylene glycol (PEG)-protected gold nanorods (NR) that exhibit superior spectral bandwidth, photothermal heat generation per gram of gold, and circulation half-life in vivo (t1/2, ~17 hours) compared with the prototypical tunable plasmonic particles, gold nanoshells, as well as ~2-fold higher X-ray absorption than a clinical iodine contrast agent. After intratumoral or i.v. administration, we fuse PEG-NR biodistribution data derived via noninvasive X-ray computed tomography or ex vivo spectrometry, respectively, with four-dimensional computational heat transport modeling to predict photothermal heating during irradiation. In computationally driven pilot therapeutic studies, we show that a single i.v. injection of PEG-NRs enabled destruction of all irradiated human xenograft tumors in mice. These studies highlight the potential of integrating computational therapy design with nanotherapeutic development for ultraselective tumor ablation. PMID:19366797

Brief Survey of TSC Computing Facilities

DOT National Transportation Integrated Search

1972-05-01

The Transportation Systems Center (TSC) has four, essentially separate, in-house computing facilities. We shall call them Honeywell Facility, the Hybrid Facility, the Multimode Simulation Facility, and the Central Facility. In addition to these four,...

Performance Evaluation of Multimodal Multifeature Authentication System Using KNN Classification.

PubMed

Rajagopal, Gayathri; Palaniswamy, Ramamoorthy

2015-01-01

This research proposes a multimodal multifeature biometric system for human recognition using two traits, that is, palmprint and iris. The purpose of this research is to analyse integration of multimodal and multifeature biometric system using feature level fusion to achieve better performance. The main aim of the proposed system is to increase the recognition accuracy using feature level fusion. The features at the feature level fusion are raw biometric data which contains rich information when compared to decision and matching score level fusion. Hence information fused at the feature level is expected to obtain improved recognition accuracy. However, information fused at feature level has the problem of curse in dimensionality; here PCA (principal component analysis) is used to diminish the dimensionality of the feature sets as they are high dimensional. The proposed multimodal results were compared with other multimodal and monomodal approaches. Out of these comparisons, the multimodal multifeature palmprint iris fusion offers significant improvements in the accuracy of the suggested multimodal biometric system. The proposed algorithm is tested using created virtual multimodal database using UPOL iris database and PolyU palmprint database.

Performance Evaluation of Multimodal Multifeature Authentication System Using KNN Classification

PubMed Central

Rajagopal, Gayathri; Palaniswamy, Ramamoorthy

2015-01-01

This research proposes a multimodal multifeature biometric system for human recognition using two traits, that is, palmprint and iris. The purpose of this research is to analyse integration of multimodal and multifeature biometric system using feature level fusion to achieve better performance. The main aim of the proposed system is to increase the recognition accuracy using feature level fusion. The features at the feature level fusion are raw biometric data which contains rich information when compared to decision and matching score level fusion. Hence information fused at the feature level is expected to obtain improved recognition accuracy. However, information fused at feature level has the problem of curse in dimensionality; here PCA (principal component analysis) is used to diminish the dimensionality of the feature sets as they are high dimensional. The proposed multimodal results were compared with other multimodal and monomodal approaches. Out of these comparisons, the multimodal multifeature palmprint iris fusion offers significant improvements in the accuracy of the suggested multimodal biometric system. The proposed algorithm is tested using created virtual multimodal database using UPOL iris database and PolyU palmprint database. PMID:26640813

The experience of agency in human-computer interactions: a review

PubMed Central

Limerick, Hannah; Coyle, David; Moore, James W.

2014-01-01

The sense of agency is the experience of controlling both one’s body and the external environment. Although the sense of agency has been studied extensively, there is a paucity of studies in applied “real-life” situations. One applied domain that seems highly relevant is human-computer-interaction (HCI), as an increasing number of our everyday agentive interactions involve technology. Indeed, HCI has long recognized the feeling of control as a key factor in how people experience interactions with technology. The aim of this review is to summarize and examine the possible links between sense of agency and understanding control in HCI. We explore the overlap between HCI and sense of agency for computer input modalities and system feedback, computer assistance, and joint actions between humans and computers. An overarching consideration is how agency research can inform HCI and vice versa. Finally, we discuss the potential ethical implications of personal responsibility in an ever-increasing society of technology users and intelligent machine interfaces. PMID:25191256

SETI meets a social intelligence: Dolphins as a model for real-time interaction and communication with a sentient species

NASA Astrophysics Data System (ADS)

Herzing, Denise L.

2010-12-01

In the past SETI has focused on the reception and deciphering of radio signals from potential remote civilizations. It is conceivable that real-time contact and interaction with a social intelligence may occur in the future. A serious look at the development of relationship, and deciphering of communication signals within and between a non-terrestrial, non-primate sentient species is relevant. Since 1985 a resident community of free-ranging Atlantic spotted dolphins has been observed regularly in the Bahamas. Life history, relationships, regular interspecific interactions with bottlenose dolphins, and multi-modal underwater communication signals have been documented. Dolphins display social communication signals modified for water, their body types, and sensory systems. Like anthropologists, human researchers engage in benign observation in the water and interact with these dolphins to develop rapport and trust. Many individual dolphins have been known for over 20 years. Learning the culturally appropriate etiquette has been important in the relationship with this alien society. To engage humans in interaction the dolphins often initiate spontaneous displays, mimicry, imitation, and synchrony. These elements may be emergent/universal features of one intelligent species contacting another for the intention of initiating interaction. This should be a consideration for real-time contact and interaction for future SETI work.

Cognitive Architectures and Human-Computer Interaction. Introduction to Special Issue.

ERIC Educational Resources Information Center

Gray, Wayne D.; Young, Richard M.; Kirschenbaum, Susan S.

1997-01-01

In this introduction to a special issue on cognitive architectures and human-computer interaction (HCI), editors and contributors provide a brief overview of cognitive architectures. The following four architectures represented by articles in this issue are: Soar; LICAI (linked model of comprehension-based action planning and instruction taking);…

Factors Influencing Adoption of Ubiquitous Internet amongst Students

ERIC Educational Resources Information Center

Juned, Mohammad; Adil, Mohd

2015-01-01

Weiser's (1991) conceptualisation of a world wherein human's interaction with computer technology would no longer be limited to conventional input and output devices, has now been translated into a reality with human's constant interaction with multiple interconnected computers and sensors embedded in rooms, furniture, clothes, tools, and other…

New methods of multimode fiber interferometer signal processing

NASA Astrophysics Data System (ADS)

Vitrik, Oleg B.; Kulchin, Yuri N.; Maxaev, Oleg G.; Kirichenko, Oleg V.; Kamenev, Oleg T.; Petrov, Yuri S.

1995-06-01

New methods of multimode fiber interferometers signal processing are suggested. For scheme of single fiber multimode interferometers with two excited modes, the method based on using of special fiber unit is developed. This unit provides the modes interaction and further sum optical field filtering. As a result the amplitude of output signal is modulated by external influence on interferometer. The stabilization of interferometer sensitivity is achieved by using additional special modulation of output signal. For scheme of single fiber multimode interferometers with excitation of wide mode spectrum, the signal of intermode interference is registered by photodiode matrix and then special electronic unit performs correlation processing. For elimination of temperature destabilization, the registered signal is adopted to multimode interferometers optical signal temperature changes. The achieved parameters for double mode scheme: temporary stability--0.6% per hour, sensitivity to interferometer length deviations--3,2 nm; for multimode scheme: temperature stability--(0.5%)/(K), temporary nonstability--0.2% per hour, sensitivity to interferometer length deviations--20 nm, dynamic range--35 dB.

Using Interactive Computer to Communicate Scientific Information.

ERIC Educational Resources Information Center

Selnow, Gary W.

1988-01-01

Asks whether the computer is another channel of communication, if its interactive qualities make it an information source, or if it is an undefined hybrid. Concludes that computers are neither the medium nor the source but will in the future provide the possibility of a sophisticated interaction between human intelligence and artificial…

Effects of Webcams on Multimodal Interactive Learning

ERIC Educational Resources Information Center

Codreanu, Tatiana; Celik, Christelle Combe

2013-01-01

This paper describes the multimodal pedagogical communication of two groups of online teachers; trainee tutors (second year students of the Master of Arts in Teaching French as a Foreign Language at the University Lumiere-Lyon 2) and experienced teachers based in different locations (France, Spain and Finland). They all taught French as a Foreign…

Neurofunctional maps of the 'maternal brain' and the effects of oxytocin: a multimodal voxel-based meta-analysis.

PubMed

Rocchetti, Matteo; Radua, Joaquim; Paloyelis, Yannis; Xenaki, Lida-Alkisti; Frascarelli, Marianna; Caverzasi, Edgardo; Politi, Pierluigi; Fusar-Poli, Paolo

2014-10-01

Several studies have tried to understand the possible neurobiological basis of mothering. The putative involvement of oxytocin, in this regard, has been deeply investigated. Performing a voxel-based meta-analysis, we aimed at testing the hypothesis of overlapping brain activation in functional magnetic resonance imaging (fMRI) studies investigating the mother-infant interaction and the oxytocin modulation of emotional stimuli in humans. We performed two systematic literature searches: fMRI studies investigating the neurofunctional correlates of the 'maternal brain' by employing mother-infant paradigms; and fMRI studies employing oxytocin during emotional tasks. A unimodal voxel-based meta-analysis was performed on each database, whereas a multimodal voxel-based meta-analytical tool was adopted to assess the hypothesis that the neurofunctional effects of oxytocin are detected in brain areas implicated in the 'maternal brain.' We found greater activation in the bilateral insula extending to the inferior frontal gyrus, basal ganglia and thalamus during mother-infant interaction and greater left insular activation associated with oxytocin administration versus placebo. Left insula extending to basal ganglia and frontotemporal gyri as well as bilateral thalamus and amygdala showed consistent activation across the two paradigms. Right insula also showed activation across the two paradigms, and dorsomedial frontal cortex activation in mothers but deactivation with oxytocin. Significant activation in areas involved in empathy, emotion regulation, motivation, social cognition and theory of mind emerged from our multimodal meta-analysis, supporting the need for further studies directly investigating the neurobiology of oxytocin in the mother-infant relationship. © 2014 The Authors. Psychiatry and Clinical Neurosciences © 2014 Japanese Society of Psychiatry and Neurology.

The human factors of workstation telepresence

NASA Technical Reports Server (NTRS)

Smith, Thomas J.; Smith, Karl U.

1990-01-01

The term workstation telepresence has been introduced to describe human-telerobot compliance, which enables the human operator to effectively project his/her body image and behavioral skills to control of the telerobot itself. Major human-factors considerations for establishing high fidelity workstation telepresence during human-telerobot operation are discussed. Telerobot workstation telepresence is defined by the proficiency and skill with which the operator is able to control sensory feedback from direct interaction with the workstation itself, and from workstation-mediated interaction with the telerobot. Numerous conditions influencing such control have been identified. This raises the question as to what specific factors most critically influence the realization of high fidelity workstation telepresence. The thesis advanced here is that perturbations in sensory feedback represent a major source of variability in human performance during interactive telerobot operation. Perturbed sensory feedback research over the past three decades has established that spatial transformations or temporal delays in sensory feedback engender substantial decrements in interactive task performance, which training does not completely overcome. A recently developed social cybernetic model of human-computer interaction can be used to guide this approach, based on computer-mediated tracking and control of sensory feedback. How the social cybernetic model can be employed for evaluating the various modes, patterns, and integrations of interpersonal, team, and human-computer interactions which play a central role is workstation telepresence are discussed.

Combining kriging, multispectral and multimodal microscopy to resolve malaria-infected erythrocyte contents.

PubMed

Dabo-Niang, S; Zoueu, J T

2012-09-01

In this communication, we demonstrate how kriging, combine with multispectral and multimodal microscopy can enhance the resolution of malaria-infected images and provide more details on their composition, for analysis and diagnosis. The results of this interpolation applied to the two principal components of multispectral and multimodal images illustrate that the examination of the content of Plasmodium falciparum infected human erythrocyte is improved. © 2012 The Authors Journal of Microscopy © 2012 Royal Microscopical Society.

Choice of Human-Computer Interaction Mode in Stroke Rehabilitation.

PubMed

Mousavi Hondori, Hossein; Khademi, Maryam; Dodakian, Lucy; McKenzie, Alison; Lopes, Cristina V; Cramer, Steven C

2016-03-01

Advances in technology are providing new forms of human-computer interaction. The current study examined one form of human-computer interaction, augmented reality (AR), whereby subjects train in the real-world workspace with virtual objects projected by the computer. Motor performances were compared with those obtained while subjects used a traditional human-computer interaction, that is, a personal computer (PC) with a mouse. Patients used goal-directed arm movements to play AR and PC versions of the Fruit Ninja video game. The 2 versions required the same arm movements to control the game but had different cognitive demands. With AR, the game was projected onto the desktop, where subjects viewed the game plus their arm movements simultaneously, in the same visual coordinate space. In the PC version, subjects used the same arm movements but viewed the game by looking up at a computer monitor. Among 18 patients with chronic hemiparesis after stroke, the AR game was associated with 21% higher game scores (P = .0001), 19% faster reaching times (P = .0001), and 15% less movement variability (P = .0068), as compared to the PC game. Correlations between game score and arm motor status were stronger with the AR version. Motor performances during the AR game were superior to those during the PC game. This result is due in part to the greater cognitive demands imposed by the PC game, a feature problematic for some patients but clinically useful for others. Mode of human-computer interface influences rehabilitation therapy demands and can be individualized for patients. © The Author(s) 2015.

Computational Assessment of Blood Flow Heterogeneity in Peritoneal Dialysis Patients' Cardiac Ventricles

PubMed Central

Kharche, Sanjay R.; So, Aaron; Salerno, Fabio; Lee, Ting-Yim; Ellis, Chris; Goldman, Daniel; McIntyre, Christopher W.

2018-01-01

Dialysis prolongs life but augments cardiovascular mortality. Imaging data suggests that dialysis increases myocardial blood flow (BF) heterogeneity, but its causes remain poorly understood. A biophysical model of human coronary vasculature was used to explain the imaging observations, and highlight causes of coronary BF heterogeneity. Post-dialysis CT images from patients under control, pharmacological stress (adenosine), therapy (cooled dialysate), and adenosine and cooled dialysate conditions were obtained. The data presented disparate phenotypes. To dissect vascular mechanisms, a 3D human vasculature model based on known experimental coronary morphometry and a space filling algorithm was implemented. Steady state simulations were performed to investigate the effects of altered aortic pressure and blood vessel diameters on myocardial BF heterogeneity. Imaging showed that stress and therapy potentially increased mean and total BF, while reducing heterogeneity. BF histograms of one patient showed multi-modality. Using the model, it was found that total coronary BF increased as coronary perfusion pressure was increased. BF heterogeneity was differentially affected by large or small vessel blocking. BF heterogeneity was found to be inversely related to small blood vessel diameters. Simulation of large artery stenosis indicates that BF became heterogeneous (increase relative dispersion) and gave multi-modal histograms. The total transmural BF as well as transmural BF heterogeneity reduced due to large artery stenosis, generating large patches of very low BF regions downstream. Blocking of arteries at various orders showed that blocking larger arteries results in multi-modal BF histograms and large patches of low BF, whereas smaller artery blocking results in augmented relative dispersion and fractal dimension. Transmural heterogeneity was also affected. Finally, the effects of augmented aortic pressure in the presence of blood vessel blocking shows differential effects on BF heterogeneity as well as transmural BF. Improved aortic blood pressure may improve total BF. Stress and therapy may be effective if they dilate small vessels. A potential cause for the observed complex BF distributions (multi-modal BF histograms) may indicate existing large vessel stenosis. The intuitive BF heterogeneity methods used can be readily used in clinical studies. Further development of the model and methods will permit personalized assessment of patient BF status. PMID:29867555

Human-computer interface

DOEpatents

Anderson, Thomas G.

2004-12-21

The present invention provides a method of human-computer interfacing. Force feedback allows intuitive navigation and control near a boundary between regions in a computer-represented space. For example, the method allows a user to interact with a virtual craft, then push through the windshield of the craft to interact with the virtual world surrounding the craft. As another example, the method allows a user to feel transitions between different control domains of a computer representation of a space. The method can provide for force feedback that increases as a user's locus of interaction moves near a boundary, then perceptibly changes (e.g., abruptly drops or changes direction) when the boundary is traversed.

Multimode cavity-assisted quantum storage via continuous phase-matching control

NASA Astrophysics Data System (ADS)

Kalachev, Alexey; Kocharovskaya, Olga

2013-09-01

A scheme for spatial multimode quantum memory is developed such that spatial-temporal structure of a weak signal pulse can be stored and recalled via cavity-assisted off-resonant Raman interaction with a strong angular-modulated control field in an extended Λ-type atomic ensemble. It is shown that effective multimode storage is possible when the Raman coherence spatial grating involves wave vectors with different longitudinal components relative to the paraxial signal field. The possibilities of implementing the scheme in the solid-state materials are discussed.

Human agency beliefs influence behaviour during virtual social interactions.

PubMed

Caruana, Nathan; Spirou, Dean; Brock, Jon

2017-01-01

In recent years, with the emergence of relatively inexpensive and accessible virtual reality technologies, it is now possible to deliver compelling and realistic simulations of human-to-human interaction. Neuroimaging studies have shown that, when participants believe they are interacting via a virtual interface with another human agent, they show different patterns of brain activity compared to when they know that their virtual partner is computer-controlled. The suggestion is that users adopt an "intentional stance" by attributing mental states to their virtual partner. However, it remains unclear how beliefs in the agency of a virtual partner influence participants' behaviour and subjective experience of the interaction. We investigated this issue in the context of a cooperative "joint attention" game in which participants interacted via an eye tracker with a virtual onscreen partner, directing each other's eye gaze to different screen locations. Half of the participants were correctly informed that their partner was controlled by a computer algorithm ("Computer" condition). The other half were misled into believing that the virtual character was controlled by a second participant in another room ("Human" condition). Those in the "Human" condition were slower to make eye contact with their partner and more likely to try and guide their partner before they had established mutual eye contact than participants in the "Computer" condition. They also responded more rapidly when their partner was guiding them, although the same effect was also found for a control condition in which they responded to an arrow cue. Results confirm the influence of human agency beliefs on behaviour in this virtual social interaction context. They further suggest that researchers and developers attempting to simulate social interactions should consider the impact of agency beliefs on user experience in other social contexts, and their effect on the achievement of the application's goals.

High-Fidelity Design of Multimodal Restorative Interventions in Gulf War Illness

DTIC Science & Technology

2017-10-01

Bockmayr A, Klarner H, Siebert H. Time series dependent analysis of unparametrized Thomas networks. IEEE/ACM Transactions on Computational Biology and...Award Number: W81XWH-15-1-0582 TITLE:High-Fidelity Design of Multimodal Restorative Interventions in Gulf War Illness PRINCIPAL INVESTIGATOR...not be construed as an official Department of the Army position, policy or decision unless so designated by other documentation. REPORT

Piezoresistive pressure sensor array for robotic skin

NASA Astrophysics Data System (ADS)

Mirza, Fahad; Sahasrabuddhe, Ritvij R.; Baptist, Joshua R.; Wijesundara, Muthu B. J.; Lee, Woo H.; Popa, Dan O.

2016-05-01

Robots are starting to transition from the confines of the manufacturing floor to homes, schools, hospitals, and highly dynamic environments. As, a result, it is impossible to foresee all the probable operational situations of robots, and preprogram the robot behavior in those situations. Among human-robot interaction technologies, haptic communication is an intuitive physical interaction method that can help define operational behaviors for robots cooperating with humans. Multimodal robotic skin with distributed sensors can help robots increase perception capabilities of their surrounding environments. Electro-Hydro-Dynamic (EHD) printing is a flexible multi-modal sensor fabrication method because of its direct printing capability of a wide range of materials onto substrates with non-uniform topographies. In past work we designed interdigitated comb electrodes as a sensing element and printed piezoresistive strain sensors using customized EHD printable PEDOT:PSS based inks. We formulated a PEDOT:PSS derivative ink, by mixing PEDOT:PSS and DMSO. Bending induced characterization tests of prototyped sensors showed high sensitivity and sufficient stability. In this paper, we describe SkinCells, robot skin sensor arrays integrated with electronic modules. 4x4 EHD-printed arrays of strain sensors was packaged onto Kapton sheets and silicone encapsulant and interconnected to a custom electronic module that consists of a microcontroller, Wheatstone bridge with adjustable digital potentiometer, multiplexer, and serial communication unit. Thus, SkinCell's electronics can be used for signal acquisition, conditioning, and networking between sensor modules. Several SkinCells were loaded with controlled pressure, temperature and humidity testing apparatuses, and testing results are reported in this paper.

Harnessing the Power of Interactivity for Instruction.

ERIC Educational Resources Information Center

Borsook, Terry K.

Arguing that what sets the computer apart from all other teaching devices is its potential for interactivity, this paper examines the concept of interactivity and explores ways in which its power can be harnessed and put to work. A discussion of interactivity in human-to-human communication sets a context within which to view human/computer…

Interference of Multi-Mode Gaussian States and "non Appearance" of Quantum Correlations

NASA Astrophysics Data System (ADS)

Olivares, Stefano

2012-01-01

We theoretically investigate bilinear, mode-mixing interactions involving two modes of uncorrelated multi-mode Gaussian states. In particular, we introduce the notion of "locally the same states" (LSS) and prove that two uncorrelated LSS modes are invariant under the mode mixing, i.e. the interaction does not lead to the birth of correlations between the outgoing modes. We also study the interference of orthogonally polarized Gaussian states by means of an interferometric scheme based on a beam splitter, rotators of polarization and polarization filters.

A Framework and Implementation of User Interface and Human-Computer Interaction Instruction

ERIC Educational Resources Information Center

Peslak, Alan

2005-01-01

Researchers have suggested that up to 50 % of the effort in development of information systems is devoted to user interface development (Douglas, Tremaine, Leventhal, Wills, & Manaris, 2002; Myers & Rosson, 1992). Yet little study has been performed on the inclusion of important interface and human-computer interaction topics into a current…

A Project-Based Learning Setting to Human-Computer Interaction for Teenagers

ERIC Educational Resources Information Center

Geyer, Cornelia; Geisler, Stefan

2012-01-01

Knowledge of fundamentals of human-computer interaction resp. usability engineering is getting more and more important in technical domains. However this interdisciplinary field of work and corresponding degree programs are not broadly known. Therefore at the Hochschule Ruhr West, University of Applied Sciences, a program was developed to give…

Multimodality Instrument for Tissue Characterization

NASA Technical Reports Server (NTRS)

Mah, Robert W. (Inventor); Andrews, Russell J. (Inventor)

2000-01-01

A system with multimodality instrument for tissue identification includes a computer-controlled motor driven heuristic probe with a multisensory tip is discussed. For neurosurgical applications, the instrument is mounted on a stereotactic frame for the probe to penetrate the brain in a precisely controlled fashion. The resistance of the brain tissue being penetrated is continually monitored by a miniaturized strain gauge attached to the probe tip. Other modality sensors may be mounted near the probe tip to provide real-time tissue characterizations and the ability to detect the proximity of blood vessels, thus eliminating errors normally associated with registration of pre-operative scans, tissue swelling, elastic tissue deformation, human judgement, etc., and rendering surgical procedures safer, more accurate, and efficient. A neural network, program adaptively learns the information on resistance and other characteristic features of normal brain tissue during the surgery and provides near real-time modeling. A fuzzy logic interface to the neural network program incorporates expert medical knowledge in the learning process. Identification of abnormal brain tissue is determined by the detection of change and comparison with previously learned models of abnormal brain tissues. The operation of the instrument is controlled through a user friendly graphical interface. Patient data is presented in a 3D stereographics display. Acoustic feedback of selected information may optionally be provided. Upon detection of the close proximity to blood vessels or abnormal brain tissue, the computer-controlled motor immediately stops probe penetration.

Development and validation of technique for in-vivo 3D analysis of cranial bone graft survival

NASA Astrophysics Data System (ADS)

Bernstein, Mark P.; Caldwell, Curtis B.; Antonyshyn, Oleh M.; Ma, Karen; Cooper, Perry W.; Ehrlich, Lisa E.

1997-05-01

Bone autografts are routinely employed in the reconstruction of facial deformities resulting from trauma, tumor ablation or congenital malformations. The combined use of post- operative 3D CT and SPECT imaging provides a means for quantitative in vivo evaluation of bone graft volume and osteoblastic activity. The specific objectives of this study were: (1) Determine the reliability and accuracy of interactive computer-assisted analysis of bone graft volumes based on 3D CT scans; (2) Determine the error in CT/SPECT multimodality image registration; (3) Determine the error in SPECT/SPECT image registration; and (4) Determine the reliability and accuracy of CT-guided SPECT uptake measurements in cranial bone grafts. Five human cadaver heads served as anthropomorphic models for all experiments. Four cranial defects were created in each specimen with inlay and onlay split skull bone grafts and reconstructed to skull and malar recipient sites. To acquire all images, each specimen was CT scanned and coated with Technetium doped paint. For purposes of validation, skulls were landmarked with 1/16-inch ball-bearings and Indium. This study provides a new technique relating anatomy and physiology for the analysis of cranial bone graft survival.

Multimodal Image Registration through Simultaneous Segmentation.

PubMed

Aganj, Iman; Fischl, Bruce

2017-11-01

Multimodal image registration facilitates the combination of complementary information from images acquired with different modalities. Most existing methods require computation of the joint histogram of the images, while some perform joint segmentation and registration in alternate iterations. In this work, we introduce a new non-information-theoretical method for pairwise multimodal image registration, in which the error of segmentation - using both images - is considered as the registration cost function. We empirically evaluate our method via rigid registration of multi-contrast brain magnetic resonance images, and demonstrate an often higher registration accuracy in the results produced by the proposed technique, compared to those by several existing methods.

Introducing the Geneva Multimodal expression corpus for experimental research on emotion perception.

PubMed

Bänziger, Tanja; Mortillaro, Marcello; Scherer, Klaus R

2012-10-01

Research on the perception of emotional expressions in faces and voices is exploding in psychology, the neurosciences, and affective computing. This article provides an overview of some of the major emotion expression (EE) corpora currently available for empirical research and introduces a new, dynamic, multimodal corpus of emotion expressions, the Geneva Multimodal Emotion Portrayals Core Set (GEMEP-CS). The design features of the corpus are outlined and justified, and detailed validation data for the core set selection are presented and discussed. Finally, an associated database with microcoded facial, vocal, and body action elements, as well as observer ratings, is introduced.

Emerging CAE technologies and their role in Future Ambient Intelligence Environments

NASA Astrophysics Data System (ADS)

Noor, Ahmed K.

2011-03-01

Dramatic improvements are on the horizon in Computer Aided Engineering (CAE) and various simulation technologies. The improvements are due, in part, to the developments in a number of leading-edge technologies and their synergistic combinations/convergence. The technologies include ubiquitous, cloud, and petascale computing; ultra high-bandwidth networks, pervasive wireless communication; knowledge based engineering; networked immersive virtual environments and virtual worlds; novel human-computer interfaces; and powerful game engines and facilities. This paper describes the frontiers and emerging simulation technologies, and their role in the future virtual product creation and learning/training environments. The environments will be ambient intelligence environments, incorporating a synergistic combination of novel agent-supported visual simulations (with cognitive learning and understanding abilities); immersive 3D virtual world facilities; development chain management systems and facilities (incorporating a synergistic combination of intelligent engineering and management tools); nontraditional methods; intelligent, multimodal and human-like interfaces; and mobile wireless devices. The Virtual product creation environment will significantly enhance the productivity and will stimulate creativity and innovation in future global virtual collaborative enterprises. The facilities in the learning/training environment will provide timely, engaging, personalized/collaborative and tailored visual learning.

Soliton interactions and complexes for coupled nonlinear Schrödinger equations.

PubMed

Jiang, Yan; Tian, Bo; Liu, Wen-Jun; Sun, Kun; Li, Min; Wang, Pan

2012-03-01

Under investigation in this paper are the coupled nonlinear Schrödinger (CNLS) equations, which can be used to govern the optical-soliton propagation and interaction in such optical media as the multimode fibers, fiber arrays, and birefringent fibers. By taking the 3-CNLS equations as an example for the N-CNLS ones (N≥3), we derive the analytic mixed-type two- and three-soliton solutions in more general forms than those obtained in the previous studies with the Hirota method and symbolic computation. With the choice of parameters for those soliton solutions, soliton interactions and complexes are investigated through the asymptotic and graphic analysis. Soliton interactions and complexes with the bound dark solitons in a mode or two modes are observed, including that (i) the two bright solitons display the breatherlike structures while the two dark ones stay parallel, (ii) the two bright and dark solitons all stay parallel, and (iii) the states of the bound solitons change from the breatherlike structures to the parallel one even with the distance between those solitons smaller than that before the interaction with the regular one soliton. Asymptotic analysis is also used to investigate the elastic and inelastic interactions between the bound solitons and the regular one soliton. Furthermore, some discussions are extended to the N-CNLS equations (N>3). Our results might be helpful in such applications as the soliton switch, optical computing, and soliton amplification in the nonlinear optics.

Joint sparse representation for robust multimodal biometrics recognition.

PubMed

Shekhar, Sumit; Patel, Vishal M; Nasrabadi, Nasser M; Chellappa, Rama

2014-01-01

Traditional biometric recognition systems rely on a single biometric signature for authentication. While the advantage of using multiple sources of information for establishing the identity has been widely recognized, computational models for multimodal biometrics recognition have only recently received attention. We propose a multimodal sparse representation method, which represents the test data by a sparse linear combination of training data, while constraining the observations from different modalities of the test subject to share their sparse representations. Thus, we simultaneously take into account correlations as well as coupling information among biometric modalities. A multimodal quality measure is also proposed to weigh each modality as it gets fused. Furthermore, we also kernelize the algorithm to handle nonlinearity in data. The optimization problem is solved using an efficient alternative direction method. Various experiments show that the proposed method compares favorably with competing fusion-based methods.

Multimodality Image Fusion-Guided Procedures: Technique, Accuracy, and Applications

DOE Office of Scientific and Technical Information (OSTI.GOV)

Abi-Jaoudeh, Nadine, E-mail: naj@mail.nih.gov; Kruecker, Jochen, E-mail: jochen.kruecker@philips.com; Kadoury, Samuel, E-mail: samuel.kadoury@polymtl.ca

2012-10-15

Personalized therapies play an increasingly critical role in cancer care: Image guidance with multimodality image fusion facilitates the targeting of specific tissue for tissue characterization and plays a role in drug discovery and optimization of tailored therapies. Positron-emission tomography (PET), magnetic resonance imaging (MRI), and contrast-enhanced computed tomography (CT) may offer additional information not otherwise available to the operator during minimally invasive image-guided procedures, such as biopsy and ablation. With use of multimodality image fusion for image-guided interventions, navigation with advanced modalities does not require the physical presence of the PET, MRI, or CT imaging system. Several commercially available methodsmore » of image-fusion and device navigation are reviewed along with an explanation of common tracking hardware and software. An overview of current clinical applications for multimodality navigation is provided.« less

Effect of external index of refraction on multimode fiber couplers.

PubMed

Wang, G Z; Murphy, K A; Claus, R O

1995-12-20

The dependence of the performance of fused-taper multimode fiber couplers on the refractive index of the material surrounding the taper region has been investigated both theoretically and experimentally. It has been identified that for a 2 × 2 multimode fiber coupler there is a range of output-power-coupling ratios for which the effect of the external refractive index is negligible. When the coupler is tapered beyond this region, the performance becomes dependent on the external index of refraction and lossy. To analyze the multimode coupler-loss mechanism, we develop a two-dimensional ray-optics model that incorporates trapped cladding-mode loss and core-mode loss through frustrated total internal reflection.

Computer-simulation results support the experimental observations. Related issues such as coupler fabrication and packaging are also discussed.

Reciprocity in computer-human interaction: source-based, norm-based, and affect-based explanations.

PubMed

Lee, Seungcheol Austin; Liang, Yuhua Jake

2015-04-01

Individuals often apply social rules when they interact with computers, and this is known as the Computers Are Social Actors (CASA) effect. Following previous work, one approach to understand the mechanism responsible for CASA is to utilize computer agents and have the agents attempt to gain human compliance (e.g., completing a pattern recognition task). The current study focuses on three key factors frequently cited to influence traditional notions of compliance: evaluations toward the source (competence and warmth), normative influence (reciprocity), and affective influence (mood). Structural equation modeling assessed the effects of these factors on human compliance with computer request. The final model shows that norm-based influence (reciprocity) increased the likelihood of compliance, while evaluations toward the computer agent did not significantly influence compliance.

Peruvian Food Chain Jenga: Learning Ecosystems with an Interactive Model

ERIC Educational Resources Information Center

Hartweg, Beau; Biffi, Daniella; de la Fuente, Yohanis; Malkoc, Ummuhan; Patterson, Melissa E.; Pearce, Erin; Stewart, Morgan A.; Weinburgh, Molly

2017-01-01

A pilot study was conducted on a multimodal educational tool, Peruvian Food Chain Jenga (PFCJ), with 5th-grade students (N = 54) at a public charter school. The goal was to compare the effectiveness of the multimodal tool to a more traditional presentation of the same materials (food chain) using an experimental/control design. Data collection…

The Use of the Webcam for Teaching a Foreign Language in a Desktop Videoconferencing Environment

ERIC Educational Resources Information Center

Develotte, Christine; Guichon, Nicolas; Vincent, Caroline

2010-01-01

This paper explores how language teachers learn to teach with a synchronous multimodal setup ("Skype"), and it focuses on their use of the webcam during the pedagogical interaction. First, we analyze the ways that French graduate students learning to teach online use the multimodal resources available in a desktop videoconferencing (DVC)…

A Plurisemiotic Study of Multimodal Interactive Teaching through Videoconferencing

ERIC Educational Resources Information Center

Codreanu, Tatiana; Celik, Christelle Combe

2012-01-01

The aim of the study is to describe and analyze webcam pedagogical communication between a French Foreign Language tutor and two students during seven online classes. It tries to answer the following question: how does the tutor in a multimodal learning environment change her semio-discursive behavior from the first to the last session? We analyze…

Using a Multimodal Approach to Facilitate Articulation, Phonemic Awareness, and Literacy in Young Children

ERIC Educational Resources Information Center

Pieretti, Robert A.; Kaul, Sandra D.; Zarchy, Razi M.; O'Hanlon, Laureen M.

2015-01-01

The primary focus of this research study was to examine the benefit of a using a multimodal approach to speech sound correction with preschool children. The approach uses the auditory, tactile, and kinesthetic modalities and includes a unique, interactive visual focus that attempts to provide a visual representation of a phonemic category. The…

An In-Depth Exploration of the Effects of the Webcam on Multimodal Interactive Learning

ERIC Educational Resources Information Center

Codreanu, Tatiana; Celik, Christelle Combe

2012-01-01

Current research describes multimodal pedagogical communication of two populations of online teachers; trainee tutors (second year students of the Master of Arts in Teaching French as a Foreign Language at the university Lumiere-Lyon 2, France) and experienced teachers based in different locations (France, Spain and Finland). They all taught…

Linguistic Layering: Social Language Development in the Context of Multimodal Design and Digital Technologies

ERIC Educational Resources Information Center

Domingo, Myrrh

2012-01-01

In our contemporary society, digital texts circulate more readily and extend beyond page-bound formats to include interactive representations such as online newsprint with hyperlinks to audio and video files. This is to say that multimodality combined with digital technologies extends grammar to include voice, visual, and music, among other modes…

Are Children with Autism More Responsive to Animated Characters? A Study of Interactions with Humans and Human-Controlled Avatars

ERIC Educational Resources Information Center

Carter, Elizabeth J.; Williams, Diane L.; Hodgins, Jessica K.; Lehman, Jill F.

2014-01-01

Few direct comparisons have been made between the responsiveness of children with autism to computer-generated or animated characters and their responsiveness to humans. Twelve 4-to 8-year-old children with autism interacted with a human therapist; a human-controlled, interactive avatar in a theme park; a human actor speaking like the avatar; and…

Multimodal 2D Brain Computer Interface.

PubMed

Almajidy, Rand K; Boudria, Yacine; Hofmann, Ulrich G; Besio, Walter; Mankodiya, Kunal

2015-08-01

In this work we used multimodal, non-invasive brain signal recording systems, namely Near Infrared Spectroscopy (NIRS), disc electrode electroencephalography (EEG) and tripolar concentric ring electrodes (TCRE) electroencephalography (tEEG). 7 healthy subjects participated in our experiments to control a 2-D Brain Computer Interface (BCI). Four motor imagery task were performed, imagery motion of the left hand, the right hand, both hands and both feet. The signal slope (SS) of the change in oxygenated hemoglobin concentration measured by NIRS was used for feature extraction while the power spectrum density (PSD) of both EEG and tEEG in the frequency band 8-30Hz was used for feature extraction. Linear Discriminant Analysis (LDA) was used to classify different combinations of the aforementioned features. The highest classification accuracy (85.2%) was achieved by using features from all the three brain signals recording modules. The improvement in classification accuracy was highly significant (p = 0.0033) when using the multimodal signals features as compared to pure EEG features.

Almost human: Anthropomorphism increases trust resilience in cognitive agents.

PubMed

de Visser, Ewart J; Monfort, Samuel S; McKendrick, Ryan; Smith, Melissa A B; McKnight, Patrick E; Krueger, Frank; Parasuraman, Raja

2016-09-01

We interact daily with computers that appear and behave like humans. Some researchers propose that people apply the same social norms to computers as they do to humans, suggesting that social psychological knowledge can be applied to our interactions with computers. In contrast, theories of human–automation interaction postulate that humans respond to machines in unique and specific ways. We believe that anthropomorphism—the degree to which an agent exhibits human characteristics—is the critical variable that may resolve this apparent contradiction across the formation, violation, and repair stages of trust. Three experiments were designed to examine these opposing viewpoints by varying the appearance and behavior of automated agents. Participants received advice that deteriorated gradually in reliability from a computer, avatar, or human agent. Our results showed (a) that anthropomorphic agents were associated with greater trust resilience , a higher resistance to breakdowns in trust; (b) that these effects were magnified by greater uncertainty; and c) that incorporating human-like trust repair behavior largely erased differences between the agents. Automation anthropomorphism is therefore a critical variable that should be carefully incorporated into any general theory of human–agent trust as well as novel automation design. PsycINFO Database Record (c) 2016 APA, all rights reserved

Information visualization: Beyond traditional engineering

NASA Technical Reports Server (NTRS)

Thomas, James J.

1995-01-01

This presentation addresses a different aspect of the human-computer interface; specifically the human-information interface. This interface will be dominated by an emerging technology called Information Visualization (IV). IV goes beyond the traditional views of computer graphics, CADS, and enables new approaches for engineering. IV specifically must visualize text, documents, sound, images, and video in such a way that the human can rapidly interact with and understand the content structure of information entities. IV is the interactive visual interface between humans and their information resources.

"Look at what I am saying": Multimodal science teaching

NASA Astrophysics Data System (ADS)

Pozzer-Ardenghi, Lilian

Language constitutes the dominant representational mode in science teaching, and lectures are still the most prevalent of the teaching methods in school science. In this dissertation, I investigate lectures from a multimodal and communicative perspective to better understand how teaching as a cultural-historical and social activity unfolds; that is, I am concerned with teaching as a communicative event, where a variety of signs (or semiotic resources), expressed in diverse modalities (or modes of communication) are produced and reproduced while the teacher articulates very specific conceptual meanings for the students. Within a trans-disciplinary approach that merges theoretical and methodical frameworks of social and cultural studies of human activity and interaction, communicative and gestures studies, linguistics, semiotics, pragmatics, and studies on teaching and learning science, I investigate teaching as a communicative, dynamic, multimodal, and social activity. My research questions include: What are the resources produced and reproduced in the classroom when the teacher is lecturing? How do these resources interact with each other? What meanings do they carry and how are these associated to achieve the coherence necessary to accomplish the communication of complex and abstract scientific concepts, not only within one lecture, but also within an entire unit of the curricula encompassing various lectures? My results show that, when lecturing, the communication of scientific concepts occur along trajectories driven by the dialectical relation among the various semiotic resources a lecturer makes available that together constitute a unit---the idea. Speech, gestures, and other nonverbal resources are but one-sided expressions of a higher order communicative meaning unit. The iterable nature of the signs produced and reproduced during science lectures permits, supports, and encourages the repetition, variation, and translation of ideas, themes, and languages and therefore permits, supports, and encourages conceptual development at the boundary between the mundane and discipline-specific cultures that students (have to) traverse in learning. It is only within this multimodal and dialectical communicative meaning unit that we can understand and investigate science teaching and learning as these processes naturally occur.

Modal interactions between a large-wavelength inclined interface and small-wavelength multimode perturbations in a Richtmyer-Meshkov instability

NASA Astrophysics Data System (ADS)

McFarland, Jacob A.; Reilly, David; Black, Wolfgang; Greenough, Jeffrey A.; Ranjan, Devesh

2015-07-01

The interaction of a small-wavelength multimodal perturbation with a large-wavelength inclined interface perturbation is investigated for the reshocked Richtmyer-Meshkov instability using three-dimensional simulations. The ares code, developed at Lawrence Livermore National Laboratory, was used for these simulations and a detailed comparison of simulation results and experiments performed at the Georgia Tech Shock Tube facility is presented first for code validation. Simulation results are presented for four cases that vary in large-wavelength perturbation amplitude and the presence of secondary small-wavelength multimode perturbations. Previously developed measures of mixing and turbulence quantities are presented that highlight the large variation in perturbation length scales created by the inclined interface and the multimode complex perturbation. Measures are developed for entrainment, and turbulence anisotropy that help to identify the effects of and competition between each perturbations type. It is shown through multiple measures that before reshock the flow processes a distinct memory of the initial conditions that is present in both large-scale-driven entrainment measures and small-scale-driven mixing measures. After reshock the flow develops to a turbulentlike state that retains a memory of high-amplitude but not low-amplitude large-wavelength perturbations. It is also shown that the high-amplitude large-wavelength perturbation is capable of producing small-scale mixing and turbulent features similar to the small-wavelength multimode perturbations.

Human-computer interaction: psychological aspects of the human use of computing.

PubMed

Olson, Gary M; Olson, Judith S

2003-01-01

Human-computer interaction (HCI) is a multidisciplinary field in which psychology and other social sciences unite with computer science and related technical fields with the goal of making computing systems that are both useful and usable. It is a blend of applied and basic research, both drawing from psychological research and contributing new ideas to it. New technologies continuously challenge HCI researchers with new options, as do the demands of new audiences and uses. A variety of usability methods have been developed that draw upon psychological principles. HCI research has expanded beyond its roots in the cognitive processes of individual users to include social and organizational processes involved in computer usage in real environments as well as the use of computers in collaboration. HCI researchers need to be mindful of the longer-term changes brought about by the use of computing in a variety of venues.

Joint Sparse Representation for Robust Multimodal Biometrics Recognition

DTIC Science & Technology

2014-01-01

comprehensive multimodal dataset and a face database are described in section V. Finally, in section VI, we discuss the computational complexity of...fingerprint, iris, palmprint , hand geometry and voice from subjects of different age, gender and ethnicity as described in Table I. It is a...Taylor, “Constructing nonlinear discriminants from multiple data views,” Machine Learning and Knowl- edge Discovery in Databases , pp. 328–343, 2010

Applications of airborne ultrasound in human-computer interaction.

PubMed

Dahl, Tobias; Ealo, Joao L; Bang, Hans J; Holm, Sverre; Khuri-Yakub, Pierre

2014-09-01

Airborne ultrasound is a rapidly developing subfield within human-computer interaction (HCI). Touchless ultrasonic interfaces and pen tracking systems are part of recent trends in HCI and are gaining industry momentum. This paper aims to provide the background and overview necessary to understand the capabilities of ultrasound and its potential future in human-computer interaction. The latest developments on the ultrasound transducer side are presented, focusing on capacitive micro-machined ultrasonic transducers, or CMUTs. Their introduction is an important step toward providing real, low-cost multi-sensor array and beam-forming options. We also provide a unified mathematical framework for understanding and analyzing algorithms used for ultrasound detection and tracking for some of the most relevant applications. Copyright © 2014. Published by Elsevier B.V.

Portable tongue-supported human computer interaction system design and implementation.

PubMed

Quain, Rohan; Khan, Masood Mehmood

2014-01-01

Tongue supported human-computer interaction (TSHCI) systems can help critically ill patients interact with both computers and people. These systems can be particularly useful for patients suffering injuries above C7 on their spinal vertebrae. Despite recent successes in their application, several limitations restrict performance of existing TSHCI systems and discourage their use in real life situations. This paper proposes a low-cost, less-intrusive, portable and easy to use design for implementing a TSHCI system. Two applications of the proposed system are reported. Design considerations and performance of the proposed system are also presented.

Design and rationale of the Mechanical Retrieval and Recanalization of Stroke Clots Using Embolectomy (MR RESCUE) Trial.

PubMed

Kidwell, Chelsea S; Jahan, Reza; Alger, Jeffry R; Schaewe, Timothy J; Guzy, Judy; Starkman, Sidney; Elashoff, Robert; Gornbein, Jeffrey; Nenov, Val; Saver, Jeffrey L

2014-01-01

Multimodal imaging has the potential to identify acute ischaemic stroke patients most likely to benefit from late recanalization therapies. The general aim of the Mechanical Retrieval and Recanalization of Stroke Clots Using Embolectomy Trial is to investigate whether multimodal imaging can identify patients who will benefit substantially from mechanical embolectomy for the treatment of acute ischaemic stroke up to eight-hours from symptom onset. Mechanical Retrieval and Recanalization of Stroke Clots Using Embolectomy is a randomized, controlled, blinded-outcome clinical trial. Acute ischaemic stroke patients with large vessel intracranial internal carotid artery or middle cerebral artery M1 or M2 occlusion enrolled within eight-hours of symptom onset are eligible. The study sample size is 120 patients. Patients are randomized to endovascular embolectomy employing the Merci Retriever (Concentric Medical, Mountain View, CA) or the Penumbra System (Penumbra, Alameda, CA) vs. standard medical care, with randomization stratified by penumbral pattern. The primary aim of the trial is to test the hypothesis that the presence of substantial ischaemic penumbral tissue visualized on multimodal imaging (magnetic resonance imaging or computed tomography) predicts patients most likely to respond to mechanical embolectomy for treatment of acute ischaemic stroke due to a large vessel, intracranial occlusion up to eight-hours from symptom onset. This hypothesis will be tested by analysing whether pretreatment imaging pattern has a significant interaction with treatment as a determinant of functional outcome based on the distribution of scores on the modified Rankin Scale measure of global disability assessed 90 days post-stroke. Nested hypotheses test for (1) treatment efficacy in patients with a penumbral pattern pretreatment, and (2) absence of treatment benefit (equivalency) in patients without a penumbral pattern pretreatment. An additional aim will only be tested if the primary hypothesis of an interaction is negative: that patients treated with mechanical embolectomy have improved functional outcome vs. standard medical management. © 2012 The Authors. International Journal of Stroke © 2012 World Stroke Organization.

Multimodal Spatial Calibration for Accurately Registering EEG Sensor Positions

PubMed Central

Chen, Shengyong; Xiao, Gang; Li, Xiaoli

2014-01-01

This paper proposes a fast and accurate calibration method to calibrate multiple multimodal sensors using a novel photogrammetry system for fast localization of EEG sensors. The EEG sensors are placed on human head and multimodal sensors are installed around the head to simultaneously obtain all EEG sensor positions. A multiple views' calibration process is implemented to obtain the transformations of multiple views. We first develop an efficient local repair algorithm to improve the depth map, and then a special calibration body is designed. Based on them, accurate and robust calibration results can be achieved. We evaluate the proposed method by corners of a chessboard calibration plate. Experimental results demonstrate that the proposed method can achieve good performance, which can be further applied to EEG source localization applications on human brain. PMID:24803954

Real-time dynamic display of registered 4D cardiac MR and ultrasound images using a GPU

NASA Astrophysics Data System (ADS)

Zhang, Q.; Huang, X.; Eagleson, R.; Guiraudon, G.; Peters, T. M.

2007-03-01

In minimally invasive image-guided surgical interventions, different imaging modalities, such as magnetic resonance imaging (MRI), computed tomography (CT), and real-time three-dimensional (3D) ultrasound (US), can provide complementary, multi-spectral image information. Multimodality dynamic image registration is a well-established approach that permits real-time diagnostic information to be enhanced by placing lower-quality real-time images within a high quality anatomical context. For the guidance of cardiac procedures, it would be valuable to register dynamic MRI or CT with intraoperative US. However, in practice, either the high computational cost prohibits such real-time visualization of volumetric multimodal images in a real-world medical environment, or else the resulting image quality is not satisfactory for accurate guidance during the intervention. Modern graphics processing units (GPUs) provide the programmability, parallelism and increased computational precision to begin to address this problem. In this work, we first outline our research on dynamic 3D cardiac MR and US image acquisition, real-time dual-modality registration and US tracking. Then we describe image processing and optimization techniques for 4D (3D + time) cardiac image real-time rendering. We also present our multimodality 4D medical image visualization engine, which directly runs on a GPU in real-time by exploiting the advantages of the graphics hardware. In addition, techniques such as multiple transfer functions for different imaging modalities, dynamic texture binding, advanced texture sampling and multimodality image compositing are employed to facilitate the real-time display and manipulation of the registered dual-modality dynamic 3D MR and US cardiac datasets.

Brain-Computer Interfaces: A Neuroscience Paradigm of Social Interaction? A Matter of Perspective

PubMed Central

Mattout, Jérémie

2012-01-01

A number of recent studies have put human subjects in true social interactions, with the aim of better identifying the psychophysiological processes underlying social cognition. Interestingly, this emerging Neuroscience of Social Interactions (NSI) field brings up challenges which resemble important ones in the field of Brain-Computer Interfaces (BCI). Importantly, these challenges go beyond common objectives such as the eventual use of BCI and NSI protocols in the clinical domain or common interests pertaining to the use of online neurophysiological techniques and algorithms. Common fundamental challenges are now apparent and one can argue that a crucial one is to develop computational models of brain processes relevant to human interactions with an adaptive agent, whether human or artificial. Coupled with neuroimaging data, such models have proved promising in revealing the neural basis and mental processes behind social interactions. Similar models could help BCI to move from well-performing but offline static machines to reliable online adaptive agents. This emphasizes a social perspective to BCI, which is not limited to a computational challenge but extends to all questions that arise when studying the brain in interaction with its environment. PMID:22675291

Human computer confluence applied in healthcare and rehabilitation.

PubMed

Viaud-Delmon, Isabelle; Gaggioli, Andrea; Ferscha, Alois; Dunne, Stephen

2012-01-01

Human computer confluence (HCC) is an ambitious research program studying how the emerging symbiotic relation between humans and computing devices can enable radically new forms of sensing, perception, interaction, and understanding. It is an interdisciplinary field, bringing together researches from horizons as various as pervasive computing, bio-signals processing, neuroscience, electronics, robotics, virtual & augmented reality, and provides an amazing potential for applications in medicine and rehabilitation.

Iconic Gestures for Robot Avatars, Recognition and Integration with Speech.

PubMed

Bremner, Paul; Leonards, Ute

2016-01-01

Co-verbal gestures are an important part of human communication, improving its efficiency and efficacy for information conveyance. One possible means by which such multi-modal communication might be realized remotely is through the use of a tele-operated humanoid robot avatar. Such avatars have been previously shown to enhance social presence and operator salience. We present a motion tracking based tele-operation system for the NAO robot platform that allows direct transmission of speech and gestures produced by the operator. To assess the capabilities of this system for transmitting multi-modal communication, we have conducted a user study that investigated if robot-produced iconic gestures are comprehensible, and are integrated with speech. Robot performed gesture outcomes were compared directly to those for gestures produced by a human actor, using a within participant experimental design. We show that iconic gestures produced by a tele-operated robot are understood by participants when presented alone, almost as well as when produced by a human. More importantly, we show that gestures are integrated with speech when presented as part of a multi-modal communication equally well for human and robot performances.

Optimal Modality Selection for Cooperative Human-Robot Task Completion.

PubMed

Jacob, Mithun George; Wachs, Juan P

2016-12-01

Human-robot cooperation in complex environments must be fast, accurate, and resilient. This requires efficient communication channels where robots need to assimilate information using a plethora of verbal and nonverbal modalities such as hand gestures, speech, and gaze. However, even though hybrid human-robot communication frameworks and multimodal communication have been studied, a systematic methodology for designing multimodal interfaces does not exist. This paper addresses the gap by proposing a novel methodology to generate multimodal lexicons which maximizes multiple performance metrics over a wide range of communication modalities (i.e., lexicons). The metrics are obtained through a mixture of simulation and real-world experiments. The methodology is tested in a surgical setting where a robot cooperates with a surgeon to complete a mock abdominal incision and closure task by delivering surgical instruments. Experimental results show that predicted optimal lexicons significantly outperform predicted suboptimal lexicons (p <; 0.05) in all metrics validating the predictability of the methodology. The methodology is validated in two scenarios (with and without modeling the risk of a human-robot collision) and the differences in the lexicons are analyzed.

Multimodality cardiac imaging at IRCCS Policlinico San Donato: a new interdisciplinary vision.

PubMed

Lombardi, Massimo; Secchi, Francesco; Pluchinotta, Francesca R; Castelvecchio, Serenella; Montericcio, Vincenzo; Camporeale, Antonia; Bandera, Francesco

2016-04-28

Multimodality imaging is the efficient integration of various methods of cardiovascular imaging to improve the ability to diagnose, guide therapy, or predict outcome. This approach implies both the availability of different technologies in a single unit and the presence of dedicated staff with cardiologic and radiologic background and certified competence in more than one imaging technique. Interaction with clinical practice and existence of research programmes and educational activities are pivotal for the success of this model. The aim of this paper is to describe the multimodality cardiac imaging programme recently started at San Donato Hospital.

Mode-selective mapping and control of vectorial nonlinear-optical processes in multimode photonic-crystal fibers.

PubMed

Hu, Ming-Lie; Wang, Ching-Yue; Song, You-Jian; Li, Yan-Feng; Chai, Lu; Serebryannikov, Evgenii; Zheltikov, Aleksei

2006-02-06

We demonstrate an experimental technique that allows a mapping of vectorial nonlinear-optical processes in multimode photonic-crystal fibers (PCFs). Spatial and polarization modes of PCFs are selectively excited in this technique by varying the tilt angle of the input beam and rotating the polarization of the input field. Intensity spectra of the PCF output plotted as a function of the input field power and polarization then yield mode-resolved maps of nonlinear-optical interactions in multimode PCFs, facilitating the analysis and control of nonlinear-optical transformations of ultrashort laser pulses in such fibers.

Reframing clinical workplace learning using the theory of distributed cognition.

PubMed

Pimmer, Christoph; Pachler, Norbert; Genewein, Urs

2013-09-01

In medicine, knowledge is embodied and socially, temporally, spatially, and culturally distributed between actors and their environment. In addition, clinicians increasingly are using technology in their daily work to gain and share knowledge. Despite these characteristics, surprisingly few studies have incorporated the theory of distributed cognition (DCog), which emphasizes how cognition is distributed in a wider system in the form of multimodal representations (e.g., clinical images, speech, gazes, and gestures) between social actors (e.g., doctors and patients) in the physical environment (e.g., with technological instruments and computers). In this article, the authors provide an example of an interaction between medical actors. Using that example, they then introduce the important concepts of the DCog theory, identifying five characteristics of clinical representations-that they are interwoven, co-constructed, redundantly accessed, intersubjectively shared, and substantiated-and discuss their value for learning. By contrasting these DCog perspectives with studies from the field of medical education, the authors argue that researchers should focus future medical education scholarship on the ways in which medical actors use and connect speech, bodily movements (e.g., gestures), and the visual and haptic structures of their own bodies and of artifacts, such as technological instruments and computers, to construct complex, multimodal representations. They also argue that future scholarship should "zoom in" on detailed, moment-by-moment analysis and, at the same time, "zoom out" following the distribution of cognition through an overall system to develop a more integrated view of clinical workplace learning.

Localizing HIV/AIDS discourse in a rural Kenyan community.

PubMed

Banda, Felix; Oketch, Omondi

2011-01-01

This paper examines the effectiveness of multimodal texts used in HIV/AIDS campaigns in rural western Kenya using multimodal discourse analysis (Kress and Van Leeuwen, 2006; Martin and Rose, 2004). Twenty HIV/AIDS documents (posters, billboards and brochures) are analysed together with interview data (20 unstructured one-on-one interviews and six focus groups) from the target group to explore the effectiveness of the multimodal texts in engaging the target rural audience in meaningful interaction towards behavioural change. It is concluded that in some cases the HIV/AIDS messages are misinterpreted or lost as the multimodal texts used are unfamiliar and contradictory to the everyday life experiences of the rural folk. The paper suggests localization of HIV/AIDS discourse through use of local modes of communication and resources.

Using Vision and Speech Features for Automated Prediction of Performance Metrics in Multimodal Dialogs. Research Report. ETS RR-17-20

ERIC Educational Resources Information Center

Ramanarayanan, Vikram; Lange, Patrick; Evanini, Keelan; Molloy, Hillary; Tsuprun, Eugene; Qian, Yao; Suendermann-Oeft, David

2017-01-01

Predicting and analyzing multimodal dialog user experience (UX) metrics, such as overall call experience, caller engagement, and latency, among other metrics, in an ongoing manner is important for evaluating such systems. We investigate automated prediction of multiple such metrics collected from crowdsourced interactions with an open-source,…

Combined multimodal photoacoustic tomography, optical coherence tomography (OCT) and OCT based angiography system for in vivo imaging of multiple skin disorders in human(Conference Presentation)

NASA Astrophysics Data System (ADS)

Liu, Mengyang; Chen, Zhe; Sinz, Christoph; Rank, Elisabet; Zabihian, Behrooz; Zhang, Edward Z.; Beard, Paul C.; Kittler, Harald; Drexler, Wolfgang

2017-02-01

All optical photoacoustic tomography (PAT) using a planar Fabry-Perot interferometer polymer film sensor has been demonstrated for in vivo human palm imaging with an imaging penetration depth of 5 mm. The relatively larger vessels in the superficial plexus and the vessels in the dermal plexus are visible in PAT. However, due to both resolution and sensitivity limits, all optical PAT cannot reveal the smaller vessels such as capillary loops and venules. Melanin absorption also sometimes causes difficulties in PAT to resolve vessels. Optical coherence tomography (OCT) based angiography, on the other hand, has been proven suitable for microvasculature visualization in the first couple millimeters in human. In our work, we combine an all optical PAT system with an OCT system featuring a phase stable akinetic swept source. This multimodal PAT/OCT/OCT-angiography system provides us co-registered human skin vasculature information as well as the structural information of cutaneous. The scanning units of the sub-systems are assembled into one probe, which is then mounted onto a portable rack. The probe and rack design gives six degrees of freedom, allowing the multimodal optical imaging probe to access nearly all regions of human body. Utilizing this probe, we perform imaging on patients with various skin disorders as well as on healthy controls. Fused PAT/OCT-angiography volume shows the complete blood vessel network in human skin, which is further embedded in the morphology provided by OCT. A comparison between the results from the disordered regions and the normal regions demonstrates the clinical translational value of this multimodal optical imaging system in dermatology.

Flight test validation of a design procedure for digital autopilots

NASA Technical Reports Server (NTRS)

Bryant, W. H.

1983-01-01

Commercially available general aviation autopilots are currently in transition from an analogue circuit system to a computer implemented digital flight control system. Well known advantages of the digital autopilot include enhanced modes, self-test capacity, fault detection, and greater computational capacity. A digital autopilot's computational capacity can be used to full advantage by increasing the sophistication of the digital autopilot's chief function, stability and control. NASA's Langley Research Center has been pursuing the development of direct digital design tools for aircraft stabilization systems for several years. This effort has most recently been directed towards the development and realization of multi-mode digital autopilots for GA aircraft, conducted under a SPIFR-related program called the General Aviation Terminal Operations Research (GATOR) Program. This presentation focuses on the implementation and testing of a candidate multi-mode autopilot designed using these newly developed tools.

Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness

PubMed Central

Calhoun, Vince D; Sui, Jing

2016-01-01

It is becoming increasingly clear that combining multi-modal brain imaging data is able to provide more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e. capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multi-modal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this paper, we start by introducing the basic reasons why multimodal data fusion is important and what it can do, and importantly how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multi-modal fusion including deep learning and multimodal classification which show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential in order to mitigate against misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness. PMID:27347565

Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness.

PubMed

Calhoun, Vince D; Sui, Jing

2016-05-01

It is becoming increasingly clear that combining multi-modal brain imaging data is able to provide more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e. capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multi-modal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this paper, we start by introducing the basic reasons why multimodal data fusion is important and what it can do, and importantly how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multi-modal fusion including deep learning and multimodal classification which show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential in order to mitigate against misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness.

Object recognition through a multi-mode fiber

NASA Astrophysics Data System (ADS)

Takagi, Ryosuke; Horisaki, Ryoichi; Tanida, Jun

2017-04-01

We present a method of recognizing an object through a multi-mode fiber. A number of speckle patterns transmitted through a multi-mode fiber are provided to a classifier based on machine learning. We experimentally demonstrated binary classification of face and non-face targets based on the method. The measurement process of the experimental setup was random and nonlinear because a multi-mode fiber is a typical strongly scattering medium and any reference light was not used in our setup. Comparisons between three supervised learning methods, support vector machine, adaptive boosting, and neural network, are also provided. All of those learning methods achieved high accuracy rates at about 90% for the classification. The approach presented here can realize a compact and smart optical sensor. It is practically useful for medical applications, such as endoscopy. Also our study indicated a promising utilization of artificial intelligence, which has rapidly progressed, for reducing optical and computational costs in optical sensing systems.

Interactions among human behavior, social networks, and societal infrastructures: A Case Study in Computational Epidemiology

NASA Astrophysics Data System (ADS)

Barrett, Christopher L.; Bisset, Keith; Chen, Jiangzhuo; Eubank, Stephen; Lewis, Bryan; Kumar, V. S. Anil; Marathe, Madhav V.; Mortveit, Henning S.

Human behavior, social networks, and the civil infrastructures are closely intertwined. Understanding their co-evolution is critical for designing public policies and decision support for disaster planning. For example, human behaviors and day to day activities of individuals create dense social interactions that are characteristic of modern urban societies. These dense social networks provide a perfect fabric for fast, uncontrolled disease propagation. Conversely, people’s behavior in response to public policies and their perception of how the crisis is unfolding as a result of disease outbreak can dramatically alter the normally stable social interactions. Effective planning and response strategies must take these complicated interactions into account. In this chapter, we describe a computer simulation based approach to study these issues using public health and computational epidemiology as an illustrative example. We also formulate game-theoretic and stochastic optimization problems that capture many of the problems that we study empirically.

Social robots as embedded reinforcers of social behavior in children with autism.

PubMed

Kim, Elizabeth S; Berkovits, Lauren D; Bernier, Emily P; Leyzberg, Dan; Shic, Frederick; Paul, Rhea; Scassellati, Brian

2013-05-01

In this study we examined the social behaviors of 4- to 12-year-old children with autism spectrum disorders (ASD; N = 24) during three tradic interactions with an adult confederate and an interaction partner, where the interaction partner varied randomly among (1) another adult human, (2) a touchscreen computer game, and (3) a social dinosaur robot. Children spoke more in general, and directed more speech to the adult confederate, when the interaction partner was a robot, as compared to a human or computer game interaction partner. Children spoke as much to the robot as to the adult interaction partner. This study provides the largest demonstration of social human-robot interaction in children with autism to date. Our findings suggest that social robots may be developed into useful tools for social skills and communication therapies, specifically by embedding social interaction into intrinsic reinforcers and motivators.

DOE Office of Scientific and Technical Information (OSTI.GOV)

Ehlen, Mark A.; Sun, Amy C.; Pepple, Mark A.

The potential impacts of man-made and natural disasters on chemical plants, complexes, and supply chains are of great importance to homeland security. To be able to estimate these impacts, we developed an agent-based chemical supply chain model that includes: chemical plants with enterprise operations such as purchasing, production scheduling, and inventories; merchant chemical markets, and multi-modal chemical shipments. Large-scale simulations of chemical-plant activities and supply chain interactions, running on desktop computers, are used to estimate the scope and duration of disruptive-event impacts, and overall system resilience, based on the extent to which individual chemical plants can adjust their internal operationsmore » (e.g., production mixes and levels) versus their external interactions (market sales and purchases, and transportation routes and modes). As a result, to illustrate how the model estimates the impacts of a hurricane disruption, a simple example model centered on 1,4-butanediol is presented.« less

Chemical supply chain modeling for analysis of homeland security events

DOE PAGES

Ehlen, Mark A.; Sun, Amy C.; Pepple, Mark A.; ...

2013-09-06

The potential impacts of man-made and natural disasters on chemical plants, complexes, and supply chains are of great importance to homeland security. To be able to estimate these impacts, we developed an agent-based chemical supply chain model that includes: chemical plants with enterprise operations such as purchasing, production scheduling, and inventories; merchant chemical markets, and multi-modal chemical shipments. Large-scale simulations of chemical-plant activities and supply chain interactions, running on desktop computers, are used to estimate the scope and duration of disruptive-event impacts, and overall system resilience, based on the extent to which individual chemical plants can adjust their internal operationsmore » (e.g., production mixes and levels) versus their external interactions (market sales and purchases, and transportation routes and modes). As a result, to illustrate how the model estimates the impacts of a hurricane disruption, a simple example model centered on 1,4-butanediol is presented.« less

Nanoparticles as multimodal photon transducers of ionizing radiation

NASA Astrophysics Data System (ADS)

Pratt, Edwin C.; Shaffer, Travis M.; Zhang, Qize; Drain, Charles Michael; Grimm, Jan

2018-05-01

In biomedical imaging, nanoparticles combined with radionuclides that generate Cerenkov luminescence are used in diagnostic imaging, photon-induced therapies and as activatable probes. In these applications, the nanoparticle is often viewed as a carrier inert to ionizing radiation from the radionuclide. However, certain phenomena such as enhanced nanoparticle luminescence and generation of reactive oxygen species cannot be completely explained by Cerenkov luminescence interactions with nanoparticles. Herein, we report methods to examine the mechanisms of nanoparticle excitation by radionuclides, including interactions with Cerenkov luminescence, β particles and γ radiation. We demonstrate that β-scintillation contributes appreciably to excitation and reactivity in certain nanoparticle systems, and that excitation by radionuclides of nanoparticles composed of large atomic number atoms generates X-rays, enabling multiplexed imaging through single photon emission computed tomography. These findings demonstrate practical optical imaging and therapy using radionuclides with emission energies below the Cerenkov threshold, thereby expanding the list of applicable radionuclides.

Learning, Interactional, and Motivational Outcomes in One-to-One Synchronous Computer-Mediated versus Face-to-Face Tutoring

ERIC Educational Resources Information Center

Siler, Stephanie Ann; VanLehn, Kurt

2009-01-01

Face-to-face (FTF) human-human tutoring has ranked among the most effective forms of instruction. However, because computer-mediated (CM) tutoring is becoming increasingly common, it is instructive to evaluate its effectiveness relative to face-to-face tutoring. Does the lack of spoken, face-to-face interaction affect learning gains and…

Interacting with a Computer-Simulated Pet: Factors Influencing Children's Humane Attitudes and Empathy

ERIC Educational Resources Information Center

Tsai, Yueh-Feng; Kaufman, David

2014-01-01

Previous research by Tsai and Kaufman (2010a, 2010b) has suggested that computer-simulated virtual pet dogs can be used as a potential medium to enhance children's development of empathy and humane attitudes toward animals. To gain a deeper understanding of how and why interacting with a virtual pet dog might influence children's social and…

A new piezoelectric energy harvesting design concept: multimodal energy harvesting skin.

PubMed

Lee, Soobum; Youn, Byeng D

2011-03-01

This paper presents an advanced design concept for a piezoelectric energy harvesting (EH), referred to as multimodal EH skin. This EH design facilitates the use of multimodal vibration and enhances power harvesting efficiency. The multimodal EH skin is an extension of our previous work, EH skin, which was an innovative design paradigm for a piezoelectric energy harvester: a vibrating skin structure and an additional thin piezoelectric layer in one device. A computational (finite element) model of the multilayered assembly - the vibrating skin structure and piezoelectric layer - is constructed and the optimal topology and/or shape of the piezoelectric layer is found for maximum power generation from multiple vibration modes. A design rationale for the multimodal EH skin was proposed: designing a piezoelectric material distribution and external resistors. In the material design step, the piezoelectric material is segmented by inflection lines from multiple vibration modes of interests to minimize voltage cancellation. The inflection lines are detected using the voltage phase. In the external resistor design step, the resistor values are found for each segment to maximize power output. The presented design concept, which can be applied to any engineering system with multimodal harmonic-vibrating skins, was applied to two case studies: an aircraft skin and a power transformer panel. The excellent performance of multimodal EH skin was demonstrated, showing larger power generation than EH skin without segmentation or unimodal EH skin.

Readings and Experiences of Multimodality

ERIC Educational Resources Information Center

Leander, Kevin M.; Aziz, Seemi; Botzakis, Stergios; Ehret, Christian; Landry, David; Rowsell, Jennifer

2017-01-01

Our understanding of reading--including reading multimodal texts--is always constrained or opened up by what we consider to be a text, what aspects of a reader's embodied activity we focus on, and how we draw a boundary around a reading event. This article brings together five literacy researchers who respond to a human-scale graphic novel,…

The mechanical design of a humanoid robot with flexible skin sensor for use in psychiatric therapy

NASA Astrophysics Data System (ADS)

Burns, Alec; Tadesse, Yonas

2014-03-01

In this paper, a humanoid robot is presented for ultimate use in the rehabilitation of children with mental disorders, such as autism. Creating affordable and efficient humanoids could assist the therapy in psychiatric disability by offering multimodal communication between the humanoid and humans. Yet, the humanoid development needs a seamless integration of artificial muscles, sensors, controllers and structures. We have designed a human-like robot that has 15 DOF, 580 mm tall and 925 mm arm span using a rapid prototyping system. The robot has a human-like appearance and movement. Flexible sensors around the arm and hands for safe human-robot interactions, and a two-wheel mobile platform for maneuverability are incorporated in the design. The robot has facial features for illustrating human-friendly behavior. The mechanical design of the robot and the characterization of the flexible sensors are presented. Comprehensive study on the upper body design, mobile base, actuators selection, electronics, and performance evaluation are included in this paper.

The study of early human embryos using interactive 3-dimensional computer reconstructions.

PubMed

Scarborough, J; Aiton, J F; McLachlan, J C; Smart, S D; Whiten, S C

1997-07-01

Tracings of serial histological sections from 4 human embryos at different Carnegie stages were used to create 3-dimensional (3D) computer models of the developing heart. The models were constructed using commercially available software developed for graphic design and the production of computer generated virtual reality environments. They are available as interactive objects which can be downloaded via the World Wide Web. This simple method of 3D reconstruction offers significant advantages for understanding important events in morphological sciences.

Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery

PubMed Central

Fuerst, Bernhard; Tateno, Keisuke; Johnson, Alex; Fotouhi, Javad; Osgood, Greg; Tombari, Federico; Navab, Nassir

2017-01-01

Orthopaedic surgeons are still following the decades old workflow of using dozens of two-dimensional fluoroscopic images to drill through complex 3D structures, e.g. pelvis. This Letter presents a mixed reality support system, which incorporates multi-modal data fusion and model-based surgical tool tracking for creating a mixed reality environment supporting screw placement in orthopaedic surgery. A red–green–blue–depth camera is rigidly attached to a mobile C-arm and is calibrated to the cone-beam computed tomography (CBCT) imaging space via iterative closest point algorithm. This allows real-time automatic fusion of reconstructed surface and/or 3D point clouds and synthetic fluoroscopic images obtained through CBCT imaging. An adapted 3D model-based tracking algorithm with automatic tool segmentation allows for tracking of the surgical tools occluded by hand. This proposed interactive 3D mixed reality environment provides an intuitive understanding of the surgical site and supports surgeons in quickly localising the entry point and orienting the surgical tool during screw placement. The authors validate the augmentation by measuring target registration error and also evaluate the tracking accuracy in the presence of partial occlusion. PMID:29184659

Compact spectrometer for precision studies of multimode behavior in an extended-cavity diode laser

NASA Astrophysics Data System (ADS)

Roach, Timothy; Golemi, Josian; Krueger, Thomas

2016-05-01

We have built a compact, inexpensive, high-precision spectrometer and used it to investigate the tuning behavior of a grating stabilized extended-cavity diode laser (ECDL). A common ECDL design uses a laser chip with an uncoated (partially reflecting) front facet, and the laser output exhibits a complicated pattern of mode hops as the frequency is tuned, in some cases even showing chaotic dynamics. Our grating spectrometer (based on a design by White & Scholten) monitors a span of 4000 GHz (8 nm at 780 nm) with a linewidth of 3 GHz, which with line-splitting gives a precision of 0.02 GHz in determining the frequency of a laser mode. We have studied multimode operation of the ECDL, tracking two or three simultaneous chip cavity modes (spacing ~ 30 GHz) during tuning via current or piezo control of the external cavity. Simultaneous output on adjacent external cavity modes (spacing ~ 5 GHz) is monitored by measuring an increase in the spectral linewidth. Computer-control of the spectrometer (for line-fitting and averaging) and of the ECDL (electronic tuning) allows rapid collection of spectral data sets, which we will use to test mathematical simulation models of the non-linear laser cavity interactions.

How safe is gamete micromanipulation by laser tweezers?

NASA Astrophysics Data System (ADS)

Koenig, Karsten; Tromberg, Bruce J.; Tadir, Yona; Berns, Michael W.

1998-04-01

Laser tweezers, used as novel sterile micromanipulation tools of living cells, are employed in laser-assisted in vitro fertilization (IVF). For example, controlled spermatozoa transport with 1064 nm tweezers to human egg cells has been performed in European clinics in cases of male infertility. The interaction of approximately 100 mW near infrared (NIR) trapping beams at MW/cm2 intensity with human gametes results in low mean less than 2 K temperature increases and less than 100 pN trapping forces. Therefore, photothermal or photomechanical induced destructive effects appear unlikely. However, the high photon flux densities may induce simultaneous absorption of two NIR photons resulting in nonlinear interactions. These nonlinear interactions imply non-resonant two-photon excitation of endogenous cellular chromophores. In the case of less than 800 nm tweezers, UV- like damage effects may occur. The destructive effect is amplified when multimode cw lasers are used as tweezer sources due to longitudinal mode-beating effects and partial mode- locking. Spermatozoa damage within seconds using 760 nm traps due to formation of unstable ps pulses in a cw Ti:Sa ring laser is demonstrated. We recommend the use of greater than or equal to 800 nm traps for optical gamete micromanipulation. To our opinion, further basic studies on the influence of nonlinear effects of laser tweezers on human gamete are necessary.

Why E-Business Must Evolve beyond Market Orientation: Applying Human Interaction Models to Computer-Mediated Corporate Communications.

ERIC Educational Resources Information Center

Johnston, Kevin McCullough

2001-01-01

Considers the design of corporate communications for electronic business and discusses the increasing importance of corporate interaction as companies work in virtual environments. Compares sociological and psychological theories of human interaction and relationship formation with organizational interaction theories of corporate relationship…

A six-degree-of-freedom passive arm with dynamic constraints (PADyC) for cardiac surgery application: preliminary experiments.

PubMed

Schneider, O; Troccaz, J

2001-01-01

The purpose of Computer-Assisted Surgery (CAS) is to help physicians and surgeons plan and execute optimal strategies from multimodal image data. The execution of such planned strategies may be assisted by guidance systems. Some of these systems, called synergistic systems, are based on the cooperation of a robotic device with a human operator. We have developed such a synergistic device: PADyC (Passive Arm with Dynamic Constraints). The basic principle of PADyC is to have a manually actuated arm that dynamically constrains the authorized motions of the surgical tool held by the human operator during a planned task. Dynamic constraints are computed from the task definition, and are implemented by a patented mechanical system. In this paper, we first introduce synergistic systems and then focus on modeling and algorithmic issues related to the dynamic constraints. Finally, we describe a 6-degree-of-freedom prototype robot designed for a clinical application (cardiac surgery) and report on preliminary experiments to date. The experimental results are then discussed, and future work is proposed. Copyright 2002 Wiley-Liss, Inc.

U.S. Army Research Laboratory (ARL) multimodal signatures database

NASA Astrophysics Data System (ADS)

Bennett, Kelly

2008-04-01

The U.S. Army Research Laboratory (ARL) Multimodal Signatures Database (MMSDB) is a centralized collection of sensor data of various modalities that are co-located and co-registered. The signatures include ground and air vehicles, personnel, mortar, artillery, small arms gunfire from potential sniper weapons, explosives, and many other high value targets. This data is made available to Department of Defense (DoD) and DoD contractors, Intel agencies, other government agencies (OGA), and academia for use in developing target detection, tracking, and classification algorithms and systems to protect our Soldiers. A platform independent Web interface disseminates the signatures to researchers and engineers within the scientific community. Hierarchical Data Format 5 (HDF5) signature models provide an excellent solution for the sharing of complex multimodal signature data for algorithmic development and database requirements. Many open source tools for viewing and plotting HDF5 signatures are available over the Web. Seamless integration of HDF5 signatures is possible in both proprietary computational environments, such as MATLAB, and Free and Open Source Software (FOSS) computational environments, such as Octave and Python, for performing signal processing, analysis, and algorithm development. Future developments include extending the Web interface into a portal system for accessing ARL algorithms and signatures, High Performance Computing (HPC) resources, and integrating existing database and signature architectures into sensor networking environments.

An adaptive importance sampling algorithm for Bayesian inversion with multimodal distributions

DOE PAGES

Li, Weixuan; Lin, Guang

2015-03-21

Parametric uncertainties are encountered in the simulations of many physical systems, and may be reduced by an inverse modeling procedure that calibrates the simulation results to observations on the real system being simulated. Following Bayes’ rule, a general approach for inverse modeling problems is to sample from the posterior distribution of the uncertain model parameters given the observations. However, the large number of repetitive forward simulations required in the sampling process could pose a prohibitive computational burden. This difficulty is particularly challenging when the posterior is multimodal. We present in this paper an adaptive importance sampling algorithm to tackle thesemore » challenges. Two essential ingredients of the algorithm are: 1) a Gaussian mixture (GM) model adaptively constructed as the proposal distribution to approximate the possibly multimodal target posterior, and 2) a mixture of polynomial chaos (PC) expansions, built according to the GM proposal, as a surrogate model to alleviate the computational burden caused by computational-demanding forward model evaluations. In three illustrative examples, the proposed adaptive importance sampling algorithm demonstrates its capabilities of automatically finding a GM proposal with an appropriate number of modes for the specific problem under study, and obtaining a sample accurately and efficiently representing the posterior with limited number of forward simulations.« less

Rotational electrical impedance tomography using electrodes with limited surface coverage provides window for multimodal sensing

NASA Astrophysics Data System (ADS)

Lehti-Polojärvi, Mari; Koskela, Olli; Seppänen, Aku; Figueiras, Edite; Hyttinen, Jari

2018-02-01

Electrical impedance tomography (EIT) is an imaging method that could become a valuable tool in multimodal applications. One challenge in simultaneous multimodal imaging is that typically the EIT electrodes cover a large portion of the object surface. This paper investigates the feasibility of rotational EIT (rEIT) in applications where electrodes cover only a limited angle of the surface of the object. In the studied rEIT, the object is rotated a full 360° during a set of measurements to increase the information content of the data. We call this approach limited angle full revolution rEIT (LAFR-rEIT). We test LAFR-rEIT setups in two-dimensional geometries with computational and experimental data. We use up to 256 rotational measurement positions, which requires a new way to solve the forward and inverse problem of rEIT. For this, we provide a modification, available for EIDORS, in the supplementary material. The computational results demonstrate that LAFR-rEIT with eight electrodes produce the same image quality as conventional 16-electrode rEIT, when data from an adequate number of rotational measurement positions are used. Both computational and experimental results indicate that the novel LAFR-rEIT provides good EIT with setups with limited surface coverage and a small number of electrodes.

A new approach of building 3D visualization framework for multimodal medical images display and computed assisted diagnosis

NASA Astrophysics Data System (ADS)

Li, Zhenwei; Sun, Jianyong; Zhang, Jianguo

2012-02-01

As more and more CT/MR studies are scanning with larger volume of data sets, more and more radiologists and clinician would like using PACS WS to display and manipulate these larger data sets of images with 3D rendering features. In this paper, we proposed a design method and implantation strategy to develop 3D image display component not only with normal 3D display functions but also with multi-modal medical image fusion as well as compute-assisted diagnosis of coronary heart diseases. The 3D component has been integrated into the PACS display workstation of Shanghai Huadong Hospital, and the clinical practice showed that it is easy for radiologists and physicians to use these 3D functions such as multi-modalities' (e.g. CT, MRI, PET, SPECT) visualization, registration and fusion, and the lesion quantitative measurements. The users were satisfying with the rendering speeds and quality of 3D reconstruction. The advantages of the component include low requirements for computer hardware, easy integration, reliable performance and comfortable application experience. With this system, the radiologists and the clinicians can manipulate with 3D images easily, and use the advanced visualization tools to facilitate their work with a PACS display workstation at any time.

An adaptive importance sampling algorithm for Bayesian inversion with multimodal distributions

DOE Office of Scientific and Technical Information (OSTI.GOV)

Li, Weixuan; Lin, Guang, E-mail: guanglin@purdue.edu

2015-08-01

Parametric uncertainties are encountered in the simulations of many physical systems, and may be reduced by an inverse modeling procedure that calibrates the simulation results to observations on the real system being simulated. Following Bayes' rule, a general approach for inverse modeling problems is to sample from the posterior distribution of the uncertain model parameters given the observations. However, the large number of repetitive forward simulations required in the sampling process could pose a prohibitive computational burden. This difficulty is particularly challenging when the posterior is multimodal. We present in this paper an adaptive importance sampling algorithm to tackle thesemore » challenges. Two essential ingredients of the algorithm are: 1) a Gaussian mixture (GM) model adaptively constructed as the proposal distribution to approximate the possibly multimodal target posterior, and 2) a mixture of polynomial chaos (PC) expansions, built according to the GM proposal, as a surrogate model to alleviate the computational burden caused by computational-demanding forward model evaluations. In three illustrative examples, the proposed adaptive importance sampling algorithm demonstrates its capabilities of automatically finding a GM proposal with an appropriate number of modes for the specific problem under study, and obtaining a sample accurately and efficiently representing the posterior with limited number of forward simulations.« less

'Full dose' reirradiation of human cervical spinal cord.

PubMed

Ryu, S; Gorty, S; Kazee, A M; Bogart, J; Hahn, S S; Dalal, P S; Chung, C T; Sagerman, R H

2000-02-01

With the progress of modern multimodality cancer treatment, retreatment of late recurrences or second tumors became more commonly encountered in management of patients with cancer. Spinal cord retreatment with radiation is a common problem in this regard. Because radiation myelopathy may result in functional deficits, many oncologists are concerned about radiation-induced myelopathy when retreating tumors located within or immediately adjacent to the previous radiation portal. The treatment decision is complicated because it requires a pertinent assessment of prognostic factors with and without reirradiation, radiobiologic estimation of recovery of occult spinal cord damage from the previous treatment, as well as interactions because of multimodality treatment. Recent studies regarding reirradiation of spinal cord in animals using limb paralysis as an endpoint have shown substantial and almost complete recovery of spinal cord injury after a sufficient time after the initial radiotherapy. We report a case of "full" dose reirradiation of the entire cervical spinal cord in a patient who has not developed clinically detectable radiation-induced myelopathy on long-term follow-up of 17 years after the first radiotherapy and 5 years after the second radiotherapy.

Hybrid microscopy of human carotid atheroma by means of optical-resolution optoacoustic and non-linear optical microscopy

NASA Astrophysics Data System (ADS)

Seeger, Markus; Karlas, Angelos; Soliman, Dominik; Pelisek, Jaroslav; Ntziachristos, Vasilis

2017-03-01

Carotid atheromatosis is causally related to stroke, a leading cause of disability and death. We present the analysis of a human carotid atheroma using a novel hybrid microscopy system that combines optical-resolution optoacoustic (photoacoustic) microscopy and several non-linear optical microscopy modalities (second and third harmonic generation, as well as, two-photon excitation fluorescence) to achieve a multimodal examination of the extracted tissue within the same imaging framework. Our system enables the label-free investigation of atheromatous human carotid tissue with a resolution of about 1 μm and allows for the congruent interrogation of plaque morphology and clinically relevant constituents such as red blood cells, collagen, and elastin. Our data reveal mutual interactions between blood embeddings and connective tissue within the atheroma, offering comprehensive insights into its stage of evolution and severity, and potentially facilitating the further development of diagnostic tools, as well as treatment strategies.

Acute imaging does not improve ASTRAL score's accuracy despite having a prognostic value.

PubMed

Ntaios, George; Papavasileiou, Vasileios; Faouzi, Mohamed; Vanacker, Peter; Wintermark, Max; Michel, Patrik

2014-10-01

The ASTRAL score was recently shown to reliably predict three-month functional outcome in patients with acute ischemic stroke. The study aims to investigate whether information from multimodal imaging increases ASTRAL score's accuracy. All patients registered in the ASTRAL registry until March 2011 were included. In multivariate logistic-regression analyses, we added covariates derived from parenchymal, vascular, and perfusion imaging to the 6-parameter model of the ASTRAL score. If a specific imaging covariate remained an independent predictor of three-month modified Rankin score>2, the area-under-the-curve (AUC) of this new model was calculated and compared with ASTRAL score's AUC. We also performed similar logistic regression analyses in arbitrarily chosen patient subgroups. When added to the ASTRAL score, the following covariates on admission computed tomography/magnetic resonance imaging-based multimodal imaging were not significant predictors of outcome: any stroke-related acute lesion, any nonstroke-related lesions, chronic/subacute stroke, leukoaraiosis, significant arterial pathology in ischemic territory on computed tomography angiography/magnetic resonance angiography/Doppler, significant intracranial arterial pathology in ischemic territory, and focal hypoperfusion on perfusion-computed tomography. The Alberta Stroke Program Early CT score on plain imaging and any significant extracranial arterial pathology on computed tomography angiography/magnetic resonance angiography/Doppler were independent predictors of outcome (odds ratio: 0·93, 95% CI: 0·87-0·99 and odds ratio: 1·49, 95% CI: 1·08-2·05, respectively) but did not increase ASTRAL score's AUC (0·849 vs. 0·850, and 0·8563 vs. 0·8564, respectively). In exploratory analyses in subgroups of different prognosis, age or stroke severity, no covariate was found to increase ASTRAL score's AUC, either. The addition of information derived from multimodal imaging does not increase ASTRAL score's accuracy to predict functional outcome despite having an independent prognostic value. More selected radiological parameters applied in specific subgroups of stroke patients may add prognostic value of multimodal imaging. © 2014 World Stroke Organization.

A Closer Look at Bilingual Students' Use of Multimodality in the Context of an Area Comparison Problem from a Large-Scale Assessment

ERIC Educational Resources Information Center

Fernandes, Anthony; Kahn, Leslie H.; Civil, Marta

2017-01-01

In this article, we use multimodality to examine how bilingual students interact with an area task from the National Assessment of Educational Progress in task-based interviews. Using vignettes, we demonstrate how some of these students manipulate the concrete materials, and use gestures, as a primary form of structuring their explanations and…

Leveraging Human Insights by Combining Multi-Objective Optimization with Interactive Evolution

DTIC Science & Technology

2015-03-26

application, a program that used human selections to guide the evolution of insect -like images. He was able to demonstrate that humans provide key insights...LEVERAGING HUMAN INSIGHTS BY COMBINING MULTI-OBJECTIVE OPTIMIZATION WITH INTERACTIVE EVOLUTION THESIS Joshua R. Christman, Second Lieutenant, USAF...COMBINING MULTI-OBJECTIVE OPTIMIZATION WITH INTERACTIVE EVOLUTION THESIS Presented to the Faculty Department of Electrical and Computer Engineering

The multi-modal Australian ScienceS Imaging and Visualization Environment (MASSIVE) high performance computing infrastructure: applications in neuroscience and neuroinformatics research

PubMed Central

Goscinski, Wojtek J.; McIntosh, Paul; Felzmann, Ulrich; Maksimenko, Anton; Hall, Christopher J.; Gureyev, Timur; Thompson, Darren; Janke, Andrew; Galloway, Graham; Killeen, Neil E. B.; Raniga, Parnesh; Kaluza, Owen; Ng, Amanda; Poudel, Govinda; Barnes, David G.; Nguyen, Toan; Bonnington, Paul; Egan, Gary F.

2014-01-01

The Multi-modal Australian ScienceS Imaging and Visualization Environment (MASSIVE) is a national imaging and visualization facility established by Monash University, the Australian Synchrotron, the Commonwealth Scientific Industrial Research Organization (CSIRO), and the Victorian Partnership for Advanced Computing (VPAC), with funding from the National Computational Infrastructure and the Victorian Government. The MASSIVE facility provides hardware, software, and expertise to drive research in the biomedical sciences, particularly advanced brain imaging research using synchrotron x-ray and infrared imaging, functional and structural magnetic resonance imaging (MRI), x-ray computer tomography (CT), electron microscopy and optical microscopy. The development of MASSIVE has been based on best practice in system integration methodologies, frameworks, and architectures. The facility has: (i) integrated multiple different neuroimaging analysis software components, (ii) enabled cross-platform and cross-modality integration of neuroinformatics tools, and (iii) brought together neuroimaging databases and analysis workflows. MASSIVE is now operational as a nationally distributed and integrated facility for neuroinfomatics and brain imaging research. PMID:24734019

Rethinking Visual Analytics for Streaming Data Applications

DOE Office of Scientific and Technical Information (OSTI.GOV)

Crouser, R. Jordan; Franklin, Lyndsey; Cook, Kris

In the age of data science, the use of interactive information visualization techniques has become increasingly ubiquitous. From online scientific journals to the New York Times graphics desk, the utility of interactive visualization for both storytelling and analysis has become ever more apparent. As these techniques have become more readily accessible, the appeal of combining interactive visualization with computational analysis continues to grow. Arising out of a need for scalable, human-driven analysis, primary objective of visual analytics systems is to capitalize on the complementary strengths of human and machine analysis, using interactive visualization as a medium for communication between themore » two. These systems leverage developments from the fields of information visualization, computer graphics, machine learning, and human-computer interaction to support insight generation in areas where purely computational analyses fall short. Over the past decade, visual analytics systems have generated remarkable advances in many historically challenging analytical contexts. These include areas such as modeling political systems [Crouser et al. 2012], detecting financial fraud [Chang et al. 2008], and cybersecurity [Harrison et al. 2012]. In each of these contexts, domain expertise and human intuition is a necessary component of the analysis. This intuition is essential to building trust in the analytical products, as well as supporting the translation of evidence into actionable insight. In addition, each of these examples also highlights the need for scalable analysis. In each case, it is infeasible for a human analyst to manually assess the raw information unaided, and the communication overhead to divide the task between a large number of analysts makes simple parallelism intractable. Regardless of the domain, visual analytics tools strive to optimize the allocation of human analytical resources, and to streamline the sensemaking process on data that is massive, complex, incomplete, and uncertain in scenarios requiring human judgment.« less

Practice and Personhood in Professional Interaction: Social Identities and Information Needs.

ERIC Educational Resources Information Center

Mokros, Hartmut B.; And Others

1995-01-01

Explores the human aspect of information retrieval by examining the behavior and pronoun use of librarians in the course of communicating with patrons during online computer search interactions. Compares two studies on the conduct of librarians as intermediaries in naturally occurring online computer search interactions. (JMV)

Multimodality imaging of adult gastric emergencies: A pictorial review

PubMed Central

Sunnapwar, Abhijit; Ojili, Vijayanadh; Katre, Rashmi; Shah, Hardik; Nagar, Arpit

2017-01-01

Acute gastric emergencies require urgent surgical or nonsurgical intervention because they are associated with high morbidity and mortality. Imaging plays an important role in diagnosis since the clinical symptoms are often nonspecific and radiologist may be the first one to suggest a diagnosis as the imaging findings are often characteristic. The purpose of this article is to provide a comprehensive review of multimodality imaging (plain radiograph, fluoroscopy, and computed tomography) of various life threatening gastric emergencies. PMID:28515579

Pancreatic tissue assessment using fluorescence and reflectance spectroscopy

NASA Astrophysics Data System (ADS)

Chandra, Malavika; Heidt, David; Simeone, Diane; McKenna, Barbara; Scheiman, James; Mycek, Mary-Ann

2007-07-01

The ability of multi-modal optical spectroscopy to detect signals from pancreatic tissue was demonstrated by studying human pancreatic cancer xenografts in mice and freshly excised human pancreatic tumor tissue. Measured optical spectra and fluorescence decays were correlated with tissue morphological and biochemical properties. The measured spectral features and decay times correlated well with expected pathological differences in normal, pancreatitis and adenocarcinoma tissue states. The observed differences between the fluorescence and reflectance properties of normal, pancreatitis and adenocarcinoma tissue indicate a possible application of multi-modal optical spectroscopy to differentiating between the three tissue classifications.

Pilot interaction with automated airborne decision making systems

NASA Technical Reports Server (NTRS)

Rouse, W. B.; Chu, Y. Y.; Greenstein, J. S.; Walden, R. S.

1976-01-01

An investigation was made of interaction between a human pilot and automated on-board decision making systems. Research was initiated on the topic of pilot problem solving in automated and semi-automated flight management systems and attempts were made to develop a model of human decision making in a multi-task situation. A study was made of allocation of responsibility between human and computer, and discussed were various pilot performance parameters with varying degrees of automation. Optimal allocation of responsibility between human and computer was considered and some theoretical results found in the literature were presented. The pilot as a problem solver was discussed. Finally the design of displays, controls, procedures, and computer aids for problem solving tasks in automated and semi-automated systems was considered.

Overview Electrotactile Feedback for Enhancing Human Computer Interface

NASA Astrophysics Data System (ADS)

Pamungkas, Daniel S.; Caesarendra, Wahyu

2018-04-01

To achieve effective interaction between a human and a computing device or machine, adequate feedback from the computing device or machine is required. Recently, haptic feedback is increasingly being utilised to improve the interactivity of the Human Computer Interface (HCI). Most existing haptic feedback enhancements aim at producing forces or vibrations to enrich the user’s interactive experience. However, these force and/or vibration actuated haptic feedback systems can be bulky and uncomfortable to wear and only capable of delivering a limited amount of information to the user which can limit both their effectiveness and the applications they can be applied to. To address this deficiency, electrotactile feedback is used. This involves delivering haptic sensations to the user by electrically stimulating nerves in the skin via electrodes placed on the surface of the skin. This paper presents a review and explores the capability of electrotactile feedback for HCI applications. In addition, a description of the sensory receptors within the skin for sensing tactile stimulus and electric currents alsoseveral factors which influenced electric signal to transmit to the brain via human skinare explained.

SnapAnatomy, a computer-based interactive tool for independent learning of human anatomy.

PubMed

Yip, George W; Rajendran, Kanagasuntheram

2008-06-01

Computer-aided instruction materials are becoming increasing popular in medical education and particularly in the teaching of human anatomy. This paper describes SnapAnatomy, a new interactive program that the authors designed for independent learning of anatomy. SnapAnatomy is primarily tailored for the beginner student to encourage the learning of anatomy by developing a three-dimensional visualization of human structure that is essential to applications in clinical practice and the understanding of function. The program allows the student to take apart and to accurately put together body components in an interactive, self-paced and variable manner to achieve the learning outcome.

Human-Computer Interaction, Tourism and Cultural Heritage

NASA Astrophysics Data System (ADS)

Cipolla Ficarra, Francisco V.

We present a state of the art of the human-computer interaction aimed at tourism and cultural heritage in some cities of the European Mediterranean. In the work an analysis is made of the main problems deriving from training understood as business and which can derail the continuous growth of the HCI, the new technologies and tourism industry. Through a semiotic and epistemological study the current mistakes in the context of the interrelations of the formal and factual sciences will be detected and also the human factors that have an influence on the professionals devoted to the development of interactive systems in order to safeguard and boost cultural heritage.

Empowering Prospective Teachers to Become Active Sense-Makers: Multimodal Modeling of the Seasons

NASA Astrophysics Data System (ADS)

Kim, Mi Song

2015-10-01

Situating science concepts in concrete and authentic contexts, using information and communications technologies, including multimodal modeling tools, is important for promoting the development of higher-order thinking skills in learners. However, teachers often struggle to integrate emergent multimodal models into a technology-rich informal learning environment. Our design-based research co-designs and develops engaging, immersive, and interactive informal learning activities called "Embodied Modeling-Mediated Activities" (EMMA) to support not only Singaporean learners' deep learning of astronomy but also the capacity of teachers. As part of the research on EMMA, this case study describes two prospective teachers' co-design processes involving multimodal models for teaching and learning the concept of the seasons in a technology-rich informal learning setting. Our study uncovers four prominent themes emerging from our data concerning the contextualized nature of learning and teaching involving multimodal models in informal learning contexts: (1) promoting communication and emerging questions, (2) offering affordances through limitations, (3) explaining one concept involving multiple concepts, and (4) integrating teaching and learning experiences. This study has an implication for the development of a pedagogical framework for teaching and learning in technology-enhanced learning environments—that is empowering teachers to become active sense-makers using multimodal models.

Multimodal therapy of word retrieval disorder due to phonological encoding dysfunction.

PubMed

Weill-Chounlamountry, Agnès; Capelle, Nathalie; Tessier, Catherine; Pradat-Diehl, Pascale

2013-01-01

To determine whether phonological multimodal therapy can improve naming and communication in a patient showing a lexical phonological naming disorder. This study employed oral and written learning tasks, using an error reduction procedure. A single-case design computer-assisted treatment was used with a 52 year-old woman with fluent aphasia consecutive to a cerebral infarction. The cognitive analysis of her word retrieval disorder exhibited a phonological encoding dysfunction. Thus, a phonological procedure was designed addressing the output phonological lexicon using computer analysis of spoken and written words. The effects were tested for trained words, generalization to untrained words, maintenance and specificity. Transfer of improvement to daily life was also assessed. After therapy, the verbal naming of both trained and untrained words was improved at p < 0.001. The improvement was still maintained after 3 months without therapy. This treatment was specific since the word dictation task did not change. Communication in daily life was improved at p < 0.05. This study of a patient with word retrieval disorder due to phonological encoding dysfunction demonstrated the effectiveness of a phonological and multimodal therapeutic treatment.

Radionuclide Myocardial Perfusion Imaging for the Evaluation of Patients With Known or Suspected Coronary Artery Disease in the Era of Multimodality Cardiovascular Imaging

PubMed Central

Taqueti, Viviany R.; Di Carli, Marcelo F.

2018-01-01

Over the last several decades, radionuclide myocardial perfusion imaging (MPI) with single photon emission tomography and positron emission tomography has been a mainstay for the evaluation of patients with known or suspected coronary artery disease (CAD). More recently, technical advances in separate and complementary imaging modalities including coronary computed tomography angiography, computed tomography perfusion, cardiac magnetic resonance imaging, and contrast stress echocardiography have expanded the toolbox of diagnostic testing for cardiac patients. While the growth of available technologies has heralded an exciting era of multimodality cardiovascular imaging, coordinated and dispassionate utilization of these techniques is needed to implement the right test for the right patient at the right time, a promise of “precision medicine.” In this article, we review the maturing role of MPI in the current era of multimodality cardiovascular imaging, particularly in the context of recent advances in myocardial blood flow quantitation, and as applied to the evaluation of patients with known or suspected CAD. PMID:25770849

Dense depth maps from correspondences derived from perceived motion

NASA Astrophysics Data System (ADS)

Kirby, Richard; Whitaker, Ross

2017-01-01

Many computer vision applications require finding corresponding points between images and using the corresponding points to estimate disparity. Today's correspondence finding algorithms primarily use image features or pixel intensities common between image pairs. Some 3-D computer vision applications, however, do not produce the desired results using correspondences derived from image features or pixel intensities. Two examples are the multimodal camera rig and the center region of a coaxial camera rig. We present an image correspondence finding technique that aligns pairs of image sequences using optical flow fields. The optical flow fields provide information about the structure and motion of the scene, which are not available in still images but can be used in image alignment. We apply the technique to a dual focal length stereo camera rig consisting of a visible light-infrared camera pair and to a coaxial camera rig. We test our method on real image sequences and compare our results with the state-of-the-art multimodal and structure from motion (SfM) algorithms. Our method produces more accurate depth and scene velocity reconstruction estimates than the state-of-the-art multimodal and SfM algorithms.

What Machines Need to Learn to Support Human Problem-Solving

NASA Technical Reports Server (NTRS)

Vera, Alonso

2017-01-01

In the development of intelligent systems that interact with humans, there is often confusion between how the system functions with respect to the humans it interacts with and how it interfaces with those humans. The former is a much deeper challenge than the latter it requires a system-level understanding of evolving human roles as well as an understanding of what humans need to know (and when) in order to perform their tasks. This talk will focus on some of the challenges in getting this right as well as on the type of research and development that results in successful human-autonomy teaming. Brief Bio: Dr. Alonso Vera is Chief of the Human Systems Integration Division at NASA Ames Research Center. His expertise is in human-computer interaction, information systems, artificial intelligence, and computational human performance modeling. He has led the design, development and deployment of mission software systems across NASA robotic and human space flight missions, including Mars Exploration Rovers, Phoenix Mars Lander, ISS, Constellation, and Exploration Systems. Dr. Vera received a Bachelor of Science with First Class Honors from McGill University in 1985 and a Ph.D. from Cornell University in 1991. He went on to a Post-Doctoral Fellowship in the School of Computer Science at Carnegie Mellon University from 1990-93.

Signal and image processing algorithm performance in a virtual and elastic computing environment

NASA Astrophysics Data System (ADS)

Bennett, Kelly W.; Robertson, James

2013-05-01

The U.S. Army Research Laboratory (ARL) supports the development of classification, detection, tracking, and localization algorithms using multiple sensing modalities including acoustic, seismic, E-field, magnetic field, PIR, and visual and IR imaging. Multimodal sensors collect large amounts of data in support of algorithm development. The resulting large amount of data, and their associated high-performance computing needs, increases and challenges existing computing infrastructures. Purchasing computer power as a commodity using a Cloud service offers low-cost, pay-as-you-go pricing models, scalability, and elasticity that may provide solutions to develop and optimize algorithms without having to procure additional hardware and resources. This paper provides a detailed look at using a commercial cloud service provider, such as Amazon Web Services (AWS), to develop and deploy simple signal and image processing algorithms in a cloud and run the algorithms on a large set of data archived in the ARL Multimodal Signatures Database (MMSDB). Analytical results will provide performance comparisons with existing infrastructure. A discussion on using cloud computing with government data will discuss best security practices that exist within cloud services, such as AWS.

Intelligent Context-Aware and Adaptive Interface for Mobile LBS

PubMed Central

Liu, Yanhong

2015-01-01

Context-aware user interface plays an important role in many human-computer Interaction tasks of location based services. Although spatial models for context-aware systems have been studied extensively, how to locate specific spatial information for users is still not well resolved, which is important in the mobile environment where location based services users are impeded by device limitations. Better context-aware human-computer interaction models of mobile location based services are needed not just to predict performance outcomes, such as whether people will be able to find the information needed to complete a human-computer interaction task, but to understand human processes that interact in spatial query, which will in turn inform the detailed design of better user interfaces in mobile location based services. In this study, a context-aware adaptive model for mobile location based services interface is proposed, which contains three major sections: purpose, adjustment, and adaptation. Based on this model we try to describe the process of user operation and interface adaptation clearly through the dynamic interaction between users and the interface. Then we show how the model applies users' demands in a complicated environment and suggested the feasibility by the experimental results. PMID:26457077

Real-time multiple human perception with color-depth cameras on a mobile robot.

PubMed

Zhang, Hao; Reardon, Christopher; Parker, Lynne E

2013-10-01

The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

Using multimedia information and communication technology (ICT) to provide added value to reminiscence therapy for people with dementia : Lessons learned from three field studies.

PubMed

Bejan, Alexander; Gündogdu, Ramazan; Butz, Katherina; Müller, Nadine; Kunze, Christophe; König, Peter

2018-01-01

In the care of people with dementia (PwD), occupational therapies and activities aiming at maintaining the quality of life of PwD, such as reminiscence therapy (RT), are taking on a more and more important role. Information and communication technology (ICT) has the potential to improve and to facilitate RT by facilitating access to and selection of biographical information and related contents or by providing novel multimodal interaction forms to trigger memories; however, interactive multimedia technology is barely used in practice. This article presents three exploratory field studies that evaluated different aspects of RT technology use for PwD in care homes, including the utilization of online movie databases, interactive surface touch computers as well as natural user interfaces allowing gestures and haptic interaction. In these studies, the usage of prototype systems was observed in occupational sessions by 5, 12 and 16 PwD. The results indicate positive effects of technology use, e. g. in the form of verbally elicited reminiscence statements, expressed joy and playful interaction. Lessons learned for the design of technology-based RT interventions are presented and discussed.

Anthracenyl polar embedded stationary phases with enhanced aromatic selectivity. Part II: A density functional theory study.

PubMed

Mignot, Mélanie; Schammé, Benjamin; Tognetti, Vincent; Joubert, Laurent; Cardinael, Pascal; Peulon-Agasse, Valérie

2017-10-13

New polar embedded aromatic stationary phases (mono- and trifunctional versions) that contain an amide-embedded group coupled with a tricyclic aromatic moiety were developed for chromatographic applications and described in the first paper of this series. These phases offered better separation performance for PAHs than for alkylbenzene homologues, and an enhanced ability to differentiate aromatic planarity to aromatic tridimensional conformation, especially for the trifunctional version and when using methanol instead of acetonitrile. In this second paper, a density functional theory study of the retention process is reported. In particular, it was shown that the selection of the suitable computational protocol allowed for describing rigorously the interactions that could take place, the solvent effects, and the structural changes for the monofunctional and the trifunctional versions. For the first time, the experimental data coupled with these DFT results provided a better understanding of the interaction mechanisms and highlighted the importance of the multimodal character of the designed stationary phases: alkyl spacers for interactions with hydrophobic solutes, amide embedded groups for dipole-dipole and hydrogen-bond interactions, and aromatic terminal groups for π-π interactions. Copyright © 2017 Elsevier B.V. All rights reserved.

Iconic Gestures for Robot Avatars, Recognition and Integration with Speech

PubMed Central

Bremner, Paul; Leonards, Ute

2016-01-01

Co-verbal gestures are an important part of human communication, improving its efficiency and efficacy for information conveyance. One possible means by which such multi-modal communication might be realized remotely is through the use of a tele-operated humanoid robot avatar. Such avatars have been previously shown to enhance social presence and operator salience. We present a motion tracking based tele-operation system for the NAO robot platform that allows direct transmission of speech and gestures produced by the operator. To assess the capabilities of this system for transmitting multi-modal communication, we have conducted a user study that investigated if robot-produced iconic gestures are comprehensible, and are integrated with speech. Robot performed gesture outcomes were compared directly to those for gestures produced by a human actor, using a within participant experimental design. We show that iconic gestures produced by a tele-operated robot are understood by participants when presented alone, almost as well as when produced by a human. More importantly, we show that gestures are integrated with speech when presented as part of a multi-modal communication equally well for human and robot performances. PMID:26925010

Multimodal Imaging in Klippel-Trénaunay-Weber Syndrome: Clinical Photography, Computed Tomoangiography, Infrared Thermography, and 99mTc-Phytate Lymphoscintigraphy.

PubMed

Kim, Su Wan; Song, Heesung

2017-12-01

We report the case of a 19-year-old man who presented with a 12-year history of progressive fatigue, feeling hot, excessive sweating, and numbness in the left arm. He had undergone multimodal imaging and was diagnosed as having Klippel-Trénaunay-Weber syndrome (KTWS). This is a rare congenital disease, defined by combinations of nevus flammeus, venous and lymphatic malformation, and hypertrophy of the affected limbs. Lower extremities are affected mostly. Conventional modalities for evaluating KTWS are ultrasonography, CT, MRI, lymphoscintigraphy, and angiography. There are few reports on multimodal imaging of upper extremities of KTWS patients, and this is the first report of an infrared thermography in KTWS.

A Multimodal Mindfulness Training to Address Mental Health Symptoms in Providers Who Care for and Interact With Children in Relation to End-of-Life Care.

PubMed

O'Mahony, Sean; Gerhart, James; Abrams, Ira; Greene, Michelle; McFadden, Rory; Tamizuddin, Sara; Levy, Mitchell M

2017-11-01

Medical providers may face unique emotional challenges when confronted with the suffering of chronically ill, dying, and bereaved children. This study assessed the preliminary outcomes of participation in a group-based multimodal mindfulness training pilot designed to reduce symptoms of burnout and mental health symptoms in providers who interact with children in the context of end-of-life care. A total of 13 medical providers who care for children facing life-threatening illness or bereaved children participated in a 9-session multimodal mindfulness session. Mental health symptoms and burnout were assessed prior to the program, at the program midpoint, and at the conclusion of the program. Participation in the pilot was associated with significant reductions in depressive and posttraumatic stress disorder (PTSD) symptoms among providers ( P < .05). Mindfulness-based programs may help providers recognize and address symptoms of depression and PTSD. Additional research is needed to enhance access and uptake of programming among larger groups of participants.

The Design of Hand Gestures for Human-Computer Interaction: Lessons from Sign Language Interpreters.

PubMed

Rempel, David; Camilleri, Matt J; Lee, David L

2015-10-01

The design and selection of 3D modeled hand gestures for human-computer interaction should follow principles of natural language combined with the need to optimize gesture contrast and recognition. The selection should also consider the discomfort and fatigue associated with distinct hand postures and motions, especially for common commands. Sign language interpreters have extensive and unique experience forming hand gestures and many suffer from hand pain while gesturing. Professional sign language interpreters (N=24) rated discomfort for hand gestures associated with 47 characters and words and 33 hand postures. Clear associations of discomfort with hand postures were identified. In a nominal logistic regression model, high discomfort was associated with gestures requiring a flexed wrist, discordant adjacent fingers, or extended fingers. These and other findings should be considered in the design of hand gestures to optimize the relationship between human cognitive and physical processes and computer gesture recognition systems for human-computer input.

Simon Plays Simon Says: The Timing of Turn-Taking in an imitation Game

DTIC Science & Technology

2012-01-01

found in the linguistics literature as well. Some work focuses on the structure of syntax and semantics in language usage [3], and other work...components come from many different approaches. Turn- taking is a highly multimodal process, and prior work gives much in-depth analysis of specific...attractive as an initial domain of investigation for its multimodality , interactive symmetry, and relative simplicity, being isolated from such

Considering the Activity in Interactivity: A Multimodal Perspective

ERIC Educational Resources Information Center

Schwartz, Ruth N.

2010-01-01

What factors contribute to effective multimedia learning? Increasingly, interactivity is considered a critical component that can foster learning in multimedia environments, including simulations and games. Although a number of recent studies investigate interactivity as a factor in the effective design of multimedia instruction, most examine only…

Older users, multimodal reminders and assisted living technology.

PubMed

Warnock, David; McGee-Lennon, Marilyn; Brewster, Stephen

2012-09-01

The primary users of assisted living technology are older people who are likely to have one or more sensory impairments. Multimodal technology allows users to interact via non-impaired senses and provides alternative ways to interact if primary interaction methods fail. An empirical user study was carried out with older participants which evaluated the performance, disruptiveness and subjective workload of visual, audio, tactile and olfactory notifications then compared the results with earlier findings in younger participants. It was found that disruption and subjective workload were not affected by modality, although some modalities were more effective at delivering information accurately. It is concluded that although further studies need to be carried out in a real-world settings, the findings support the argument for multiple modalities in assisted living technology.

Sensor-Based Human Activity Recognition in a Multi-user Scenario

NASA Astrophysics Data System (ADS)

Wang, Liang; Gu, Tao; Tao, Xianping; Lu, Jian

Existing work on sensor-based activity recognition focuses mainly on single-user activities. However, in real life, activities are often performed by multiple users involving interactions between them. In this paper, we propose Coupled Hidden Markov Models (CHMMs) to recognize multi-user activities from sensor readings in a smart home environment. We develop a multimodal sensing platform and present a theoretical framework to recognize both single-user and multi-user activities. We conduct our trace collection done in a smart home, and evaluate our framework through experimental studies. Our experimental result shows that we achieve an average accuracy of 85.46% with CHMMs.

Top-down modulation: the crossroads of perception, attention and memory

NASA Astrophysics Data System (ADS)

Gazzaley, Adam

2010-02-01

Research in our laboratory focuses on understanding the neural mechanisms that serve at the crossroads of perception, memory and attention, specifically exploring how brain region interactions underlie these abilities. To accomplish this, we study top-down modulation, the process by which we enhance neural activity associated with relevant information and suppress activity for irrelevant information, thus establishing a neural basis for all higher-order cognitive operations. We also study alterations in top-down modulation that occur with normal aging. Our experiments are performed on human participants, using a multimodal approach that integrates functional MRI (fMRI), transcranial magnetic stimulation (TMS) and electroencephalography (EEG).

Integration of visual and non-visual self-motion cues during voluntary head movements in the human brain.

PubMed

Schindler, Andreas; Bartels, Andreas

2018-05-15

Our phenomenological experience of the stable world is maintained by continuous integration of visual self-motion with extra-retinal signals. However, due to conventional constraints of fMRI acquisition in humans, neural responses to visuo-vestibular integration have only been studied using artificial stimuli, in the absence of voluntary head-motion. We here circumvented these limitations and let participants to move their heads during scanning. The slow dynamics of the BOLD signal allowed us to acquire neural signal related to head motion after the observer's head was stabilized by inflatable aircushions. Visual stimuli were presented on head-fixed display goggles and updated in real time as a function of head-motion that was tracked using an external camera. Two conditions simulated forward translation of the participant. During physical head rotation, the congruent condition simulated a stable world, whereas the incongruent condition added arbitrary lateral motion. Importantly, both conditions were precisely matched in visual properties and head-rotation. By comparing congruent with incongruent conditions we found evidence consistent with the multi-modal integration of visual cues with head motion into a coherent "stable world" percept in the parietal operculum and in an anterior part of parieto-insular cortex (aPIC). In the visual motion network, human regions MST, a dorsal part of VIP, the cingulate sulcus visual area (CSv) and a region in precuneus (Pc) showed differential responses to the same contrast. The results demonstrate for the first time neural multimodal interactions between precisely matched congruent versus incongruent visual and non-visual cues during physical head-movement in the human brain. The methodological approach opens the path to a new class of fMRI studies with unprecedented temporal and spatial control over visuo-vestibular stimulation. Copyright © 2018 Elsevier Inc. All rights reserved.

Using Adaptive Automation to Increase Operator Performance and Decrease Stress in a Satellite Operations Environment

ERIC Educational Resources Information Center

Klein, David C.

2014-01-01

As advancements in automation continue to alter the systemic behavior of computer systems in a wide variety of industrial applications, human-machine interactions are increasingly becoming supervisory in nature, with less hands-on human involvement. This maturing of the human role within the human-computer relationship is relegating operations…

Enhancing Tele-robotics with Immersive Virtual Reality

DTIC Science & Technology

2017-11-03

graduate and undergraduate students within the Digital Gaming and Simulation, Computer Science, and psychology programs have actively collaborated...investigates the use of artificial intelligence and visual computing. Numerous fields across the human-computer interaction and gaming research areas...invested in digital gaming and simulation to cognitively stimulate humans by computers, forming a $10.5B industry [1]. On the other hand, cognitive

Social Network Extraction and Analysis Based on Multimodal Dyadic Interaction

PubMed Central

Escalera, Sergio; Baró, Xavier; Vitrià, Jordi; Radeva, Petia; Raducanu, Bogdan

2012-01-01

Social interactions are a very important component in people’s lives. Social network analysis has become a common technique used to model and quantify the properties of social interactions. In this paper, we propose an integrated framework to explore the characteristics of a social network extracted from multimodal dyadic interactions. For our study, we used a set of videos belonging to New York Times’ Blogging Heads opinion blog. The Social Network is represented as an oriented graph, whose directed links are determined by the Influence Model. The links’ weights are a measure of the “influence” a person has over the other. The states of the Influence Model encode automatically extracted audio/visual features from our videos using state-of-the art algorithms. Our results are reported in terms of accuracy of audio/visual data fusion for speaker segmentation and centrality measures used to characterize the extracted social network. PMID:22438733

Vestibular-somatosensory interactions: effects of passive whole-body rotation on somatosensory detection.

PubMed

Ferrè, Elisa Raffaella; Kaliuzhna, Mariia; Herbelin, Bruno; Haggard, Patrick; Blanke, Olaf

2014-01-01

Vestibular signals are strongly integrated with information from several other sensory modalities. For example, vestibular stimulation was reported to improve tactile detection. However, this improvement could reflect either a multimodal interaction or an indirect interaction driven by vestibular effects on spatial attention and orienting. Here we investigate whether natural vestibular activation induced by passive whole-body rotation influences tactile detection. In particular, we assessed the ability to detect faint tactile stimuli to the fingertips of the left and right hand during spatially congruent or incongruent rotations. We found that passive whole-body rotations significantly enhanced sensitivity to faint shocks, without affecting response bias. Critically, this enhancement of somatosensory sensitivity did not depend on the spatial congruency between the direction of rotation and the hand stimulated. Thus, our results support a multimodal interaction, likely in brain areas receiving both vestibular and somatosensory signals.

Participatory Classification in a System for Assessing Multimodal Transportation Patterns

DTIC Science & Technology

2015-02-17

Culler Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2015-8 http...California at Berkeley,Electrical Engineering and Computer Sciences,Berkeley,CA,94720 8. PERFORMING ORGANIZATION REPORT NUMBER 9. SPONSORING/MONITORING...confirmation screen This section sketches the characteristics of the data that was collected, computes the accuracy of the auto- mated inference algorithm

Multimodal instrument for high-sensitivity autofluorescence and spectral optical coherence tomography of the human eye fundus

PubMed Central

Komar, Katarzyna; Stremplewski, Patrycjusz; Motoczyńska, Marta; Szkulmowski, Maciej; Wojtkowski, Maciej

2013-01-01

In this paper we present a multimodal device for imaging fundus of human eye in vivo which combines functionality of autofluorescence by confocal SLO with Fourier domain OCT. Native fluorescence of human fundus was excited by modulated laser beam (λ = 473 nm, 20 MHz) and lock-in detection was applied resulting in improving sensitivity. The setup allows for acquisition of high resolution OCT and high contrast AF images using fluorescence excitation power of 50-65 μW without averaging consecutive images. Successful functioning of constructed device have been demonstrated for 8 healthy volunteers of different age ranging from 24 to 83 years old. PMID:24298426

Automatic segmentation of cerebral white matter hyperintensities using only 3D FLAIR images.

PubMed

Simões, Rita; Mönninghoff, Christoph; Dlugaj, Martha; Weimar, Christian; Wanke, Isabel; van Cappellen van Walsum, Anne-Marie; Slump, Cornelis

2013-09-01

Magnetic Resonance (MR) white matter hyperintensities have been shown to predict an increased risk of developing cognitive decline. However, their actual role in the conversion to dementia is still not fully understood. Automatic segmentation methods can help in the screening and monitoring of Mild Cognitive Impairment patients who take part in large population-based studies. Most existing segmentation approaches use multimodal MR images. However, multiple acquisitions represent a limitation in terms of both patient comfort and computational complexity of the algorithms. In this work, we propose an automatic lesion segmentation method that uses only three-dimensional fluid-attenuation inversion recovery (FLAIR) images. We use a modified context-sensitive Gaussian mixture model to determine voxel class probabilities, followed by correction of FLAIR artifacts. We evaluate the method against the manual segmentation performed by an experienced neuroradiologist and compare the results with other unimodal segmentation approaches. Finally, we apply our method to the segmentation of multiple sclerosis lesions by using a publicly available benchmark dataset. Results show a similar performance to other state-of-the-art multimodal methods, as well as to the human rater. Copyright © 2013 Elsevier Inc. All rights reserved.

Correcting Spellings in Second Language Learners' Computer-Assisted Collaborative Writing

ERIC Educational Resources Information Center

Musk, Nigel

2016-01-01

The present study uses multimodal conversation analysis to examine how pupils studying English as a foreign language make spelling corrections in real time while doing collaborative computer-assisted project work. Unlike most previous related investigations, this study focuses on the "process" rather than evaluating the final…

Fast and robust multimodal image registration using a local derivative pattern.

PubMed

Jiang, Dongsheng; Shi, Yonghong; Chen, Xinrong; Wang, Manning; Song, Zhijian

2017-02-01

Deformable multimodal image registration, which can benefit radiotherapy and image guided surgery by providing complementary information, remains a challenging task in the medical image analysis field due to the difficulty of defining a proper similarity measure. This article presents a novel, robust and fast binary descriptor, the discriminative local derivative pattern (dLDP), which is able to encode images of different modalities into similar image representations. dLDP calculates a binary string for each voxel according to the pattern of intensity derivatives in its neighborhood. The descriptor similarity is evaluated using the Hamming distance, which can be efficiently computed, instead of conventional L1 or L2 norms. For the first time, we validated the effectiveness and feasibility of the local derivative pattern for multimodal deformable image registration with several multi-modal registration applications. dLDP was compared with three state-of-the-art methods in artificial image and clinical settings. In the experiments of deformable registration between different magnetic resonance imaging (MRI) modalities from BrainWeb, between computed tomography and MRI images from patient data, and between MRI and ultrasound images from BITE database, we show our method outperforms localized mutual information and entropy images in terms of both accuracy and time efficiency. We have further validated dLDP for the deformable registration of preoperative MRI and three-dimensional intraoperative ultrasound images. Our results indicate that dLDP reduces the average mean target registration error from 4.12 mm to 2.30 mm. This accuracy is statistically equivalent to the accuracy of the state-of-the-art methods in the study; however, in terms of computational complexity, our method significantly outperforms other methods and is even comparable to the sum of the absolute difference. The results reveal that dLDP can achieve superior performance regarding both accuracy and time efficiency in general multimodal image registration. In addition, dLDP also indicates the potential for clinical ultrasound guided intervention. © 2016 The Authors. Medical Physics published by Wiley Periodicals, Inc. on behalf of American Association of Physicists in Medicine.

No Two Cues Are Alike: Depth of Learning during Infancy Is Dependent on What Orients Attention

ERIC Educational Resources Information Center

Wu, Rachel; Kirkham, Natasha Z.

2010-01-01

Human infants develop a variety of attentional mechanisms that allow them to extract relevant information from a cluttered multimodal world. We know that both social and nonsocial cues shift infants' attention, but not how these cues differentially affect learning of multimodal events. Experiment 1 used social cues to direct 8- and 4-month-olds'…

Sensor-based assessment of the in-situ quality of human computer interaction in the cars : final research report.

DOT National Transportation Integrated Search

2016-01-01

Human attention is a finite resource. When interrupted while performing a task, this : resource is split between two interactive tasks. People have to decide whether the benefits : from the interruptive interaction will be enough to offset the loss o...

Human-computer dialogue: Interaction tasks and techniques. Survey and categorization

NASA Technical Reports Server (NTRS)

Foley, J. D.

1983-01-01

Interaction techniques are described. Six basic interaction tasks, requirements for each task, requirements related to interaction techniques, and a technique's hardware prerequisites affective device selection are discussed.

Employing Textual and Facial Emotion Recognition to Design an Affective Tutoring System

ERIC Educational Resources Information Center

Lin, Hao-Chiang Koong; Wang, Cheng-Hung; Chao, Ching-Ju; Chien, Ming-Kuan

2012-01-01

Emotional expression in Artificial Intelligence has gained lots of attention in recent years, people applied its affective computing not only in enhancing and realizing the interaction between computers and human, it also makes computer more humane. In this study, emotional expressions were applied into intelligent tutoring system, where learners'…

Study of wavelength division multiplexing as a means of increasing the number of channels in multimode fiber optic communication links

NASA Technical Reports Server (NTRS)

Bates, Harry

1990-01-01

A number of optical communication lines are now in use at the Kennedy Space Center (KSC) for the transmission of voice, computer data, and video signals. Presently, all of these channels utilize a single carrier wavelength centered near 1300 nm. The theoretical bandwidth of the fiber far exceeds the utilized capacity. Yet, practical considerations limit the usable bandwidth. The fibers have the capability of transmitting a multiplicity of signals simultaneously in each of two separate bands (1300 and 1550 nm). Thus, in principle, the number of transmission channels can be increased without installing new cable if some means of wavelength division multiplexing (WDM) can be utilized. The main goal of these experiments was to demonstrate that a factor of 2 increase in bandwidth utilization can share the same fiber in both a unidirectional configuration and a bidirectional mode of operation. Both signal and multimode fiber are installed at KSC. The great majority is multimode; therefore, this effort concentrated on multimode systems.

End-to-End Multimodal Emotion Recognition Using Deep Neural Networks

NASA Astrophysics Data System (ADS)

Tzirakis, Panagiotis; Trigeorgis, George; Nicolaou, Mihalis A.; Schuller, Bjorn W.; Zafeiriou, Stefanos

2017-12-01

Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.

Toward Optimization of Gaze-Controlled Human-Computer Interaction: Application to Hindi Virtual Keyboard for Stroke Patients.

PubMed

Meena, Yogesh Kumar; Cecotti, Hubert; Wong-Lin, Kongfatt; Dutta, Ashish; Prasad, Girijesh

2018-04-01

Virtual keyboard applications and alternative communication devices provide new means of communication to assist disabled people. To date, virtual keyboard optimization schemes based on script-specific information, along with multimodal input access facility, are limited. In this paper, we propose a novel method for optimizing the position of the displayed items for gaze-controlled tree-based menu selection systems by considering a combination of letter frequency and command selection time. The optimized graphical user interface layout has been designed for a Hindi language virtual keyboard based on a menu wherein 10 commands provide access to type 88 different characters, along with additional text editing commands. The system can be controlled in two different modes: eye-tracking alone and eye-tracking with an access soft-switch. Five different keyboard layouts have been presented and evaluated with ten healthy participants. Furthermore, the two best performing keyboard layouts have been evaluated with eye-tracking alone on ten stroke patients. The overall performance analysis demonstrated significantly superior typing performance, high usability (87% SUS score), and low workload (NASA TLX with 17 scores) for the letter frequency and time-based organization with script specific arrangement design. This paper represents the first optimized gaze-controlled Hindi virtual keyboard, which can be extended to other languages.

Phonetic Variation and Interactional Contingencies in Simultaneous Responses

ERIC Educational Resources Information Center

Walker, Gareth

2016-01-01

An auspicious but unexplored environment for studying phonetic variation in naturalistic interaction is where two or more participants say the same thing at the same time. Working with a core dataset built from the multimodal Augmented Multi-party Interaction corpus, the principles of conversation analysis were followed to analyze the sequential…

The Semiotic Work of the Hands in Scientific Enquiry

ERIC Educational Resources Information Center

Sakr, Mona; Jewitt, Carey; Price, Sara

2014-01-01

This paper takes a multimodal approach to analysing embodied interaction and discourses of scientific investigation using an interactive tangible tabletop. It argues that embodied forms of interaction are central to science inquiry. More specifically, the paper examines the role of hand actions in the development of descriptions and explanations…

Interactive Learning System "VisMis" for Scientific Visualization Course

ERIC Educational Resources Information Center

Zhu, Xiaoming; Sun, Bo; Luo, Yanlin

2018-01-01

Now visualization courses have been taught at universities around the world. Keeping students motivated and actively engaged in this course can be a challenging task. In this paper we introduce our developed interactive learning system called VisMis (Visualization and Multi-modal Interaction System) for postgraduate scientific visualization course…

The Next Wave: Humans, Computers, and Redefining Reality

NASA Technical Reports Server (NTRS)

Little, William

2018-01-01

The Augmented/Virtual Reality (AVR) Lab at KSC is dedicated to " exploration into the growing computer fields of Extended Reality and the Natural User Interface (it is) a proving ground for new technologies that can be integrated into future NASA projects and programs." The topics of Human Computer Interface, Human Computer Interaction, Augmented Reality, Virtual Reality, and Mixed Reality are defined; examples of work being done in these fields in the AVR Lab are given. Current new and future work in Computer Vision, Speech Recognition, and Artificial Intelligence are also outlined.

IBM techexplorer and MathML: Interactive Multimodal Scientific Documents

NASA Astrophysics Data System (ADS)

Diaz, Angel

2001-06-01

The World Wide Web provides a standard publishing platform for disseminating scientific and technical articles, books, journals, courseware, or even homework on the internet; however, the transition from paper to web-based interactive content has brought new opportunities for creating interactive content. Students, scientists, and engineers are now faced with the task of rendering the 2D presentational structure of mathematics, harnessing the wealth of scientific and technical software, and creating truly accessible scientific portals across international boundaries and markets. The recent emergence of World Wide Web Consortium (W3C) standards such as the Mathematical Markup Language (MathML), Language (XSL), and Aural CSS (ACSS) provide a foundation whereby mathematics can be displayed, enlivened, computed, and audio formatted. With interoperability ensured by standards, software applications can be easily brought together to create extensible and interactive scientific content. In this presentation we will provide an overview of the IBM techexplorer Hypermedia Browser, a web browser plug-in and ActiveX control aimed at bringing interactive mathematics to the masses across platforms and applications. We will demonstrate "live" mathematics where documents that contain MathML expressions can be edited and computed right inside your favorite web browser. This demonstration will be generalized as we show how MathML can be used to enliven even PowerPoint presentations. Finally, we will close the loop by demonstrating a novel approach to spoken mathematics based on MathML, DOM, XSL, ACSS, techexplorer, and IBM ViaVoice. By making use of techexplorer as the glue that binds the rendered content to the web browser, the back-end computation software, the Java applets that augment the exposition, and voice-rendering systems such as ViaVoice, authors can indeed create truly extensible and interactive scientific content. For more information see: [http://www.software.ibm.com/techexplorer] [http://www.alphaworks.ibm.com] [http://www.w3.org

Generation of a suite of 3D computer-generated breast phantoms from a limited set of human subject data

DOE Office of Scientific and Technical Information (OSTI.GOV)

Hsu, Christina M. L.; Palmeri, Mark L.; Department of Anesthesiology, Duke University Medical Center, Durham, North Carolina 27710

2013-04-15

Purpose: The authors previously reported on a three-dimensional computer-generated breast phantom, based on empirical human image data, including a realistic finite-element based compression model that was capable of simulating multimodality imaging data. The computerized breast phantoms are a hybrid of two phantom generation techniques, combining empirical breast CT (bCT) data with flexible computer graphics techniques. However, to date, these phantoms have been based on single human subjects. In this paper, the authors report on a new method to generate multiple phantoms, simulating additional subjects from the limited set of original dedicated breast CT data. The authors developed an image morphingmore » technique to construct new phantoms by gradually transitioning between two human subject datasets, with the potential to generate hundreds of additional pseudoindependent phantoms from the limited bCT cases. The authors conducted a preliminary subjective assessment with a limited number of observers (n= 4) to illustrate how realistic the simulated images generated with the pseudoindependent phantoms appeared. Methods: Several mesh-based geometric transformations were developed to generate distorted breast datasets from the original human subject data. Segmented bCT data from two different human subjects were used as the 'base' and 'target' for morphing. Several combinations of transformations were applied to morph between the 'base' and 'target' datasets such as changing the breast shape, rotating the glandular data, and changing the distribution of the glandular tissue. Following the morphing, regions of skin and fat were assigned to the morphed dataset in order to appropriately assign mechanical properties during the compression simulation. The resulting morphed breast was compressed using a finite element algorithm and simulated mammograms were generated using techniques described previously. Sixty-two simulated mammograms, generated from morphing three human subject datasets, were used in a preliminary observer evaluation where four board certified breast radiologists with varying amounts of experience ranked the level of realism (from 1 ='fake' to 10 ='real') of the simulated images. Results: The morphing technique was able to successfully generate new and unique morphed datasets from the original human subject data. The radiologists evaluated the realism of simulated mammograms generated from the morphed and unmorphed human subject datasets and scored the realism with an average ranking of 5.87 {+-} 1.99, confirming that overall the phantom image datasets appeared more 'real' than 'fake.' Moreover, there was not a significant difference (p > 0.1) between the realism of the unmorphed datasets (6.0 {+-} 1.95) compared to the morphed datasets (5.86 {+-} 1.99). Three of the four observers had overall average rankings of 6.89 {+-} 0.89, 6.9 {+-} 1.24, 6.76 {+-} 1.22, whereas the fourth observer ranked them noticeably lower at 2.94 {+-} 0.7. Conclusions: This work presents a technique that can be used to generate a suite of realistic computerized breast phantoms from a limited number of human subjects. This suite of flexible breast phantoms can be used for multimodality imaging research to provide a known truth while concurrently producing realistic simulated imaging data.« less

Stability, structure and scale: improvements in multi-modal vessel extraction for SEEG trajectory planning.

PubMed

Zuluaga, Maria A; Rodionov, Roman; Nowell, Mark; Achhala, Sufyan; Zombori, Gergely; Mendelson, Alex F; Cardoso, M Jorge; Miserocchi, Anna; McEvoy, Andrew W; Duncan, John S; Ourselin, Sébastien

2015-08-01

Brain vessels are among the most critical landmarks that need to be assessed for mitigating surgical risks in stereo-electroencephalography (SEEG) implantation. Intracranial haemorrhage is the most common complication associated with implantation, carrying significantly associated morbidity. SEEG planning is done pre-operatively to identify avascular trajectories for the electrodes. In current practice, neurosurgeons have no assistance in the planning of electrode trajectories. There is great interest in developing computer-assisted planning systems that can optimise the safety profile of electrode trajectories, maximising the distance to critical structures. This paper presents a method that integrates the concepts of scale, neighbourhood structure and feature stability with the aim of improving robustness and accuracy of vessel extraction within a SEEG planning system. The developed method accounts for scale and vicinity of a voxel by formulating the problem within a multi-scale tensor voting framework. Feature stability is achieved through a similarity measure that evaluates the multi-modal consistency in vesselness responses. The proposed measurement allows the combination of multiple images modalities into a single image that is used within the planning system to visualise critical vessels. Twelve paired data sets from two image modalities available within the planning system were used for evaluation. The mean Dice similarity coefficient was 0.89 ± 0.04, representing a statistically significantly improvement when compared to a semi-automated single human rater, single-modality segmentation protocol used in clinical practice (0.80 ± 0.03). Multi-modal vessel extraction is superior to semi-automated single-modality segmentation, indicating the possibility of safer SEEG planning, with reduced patient morbidity.

Deep learning for hybrid EEG-fNIRS brain-computer interface: application to motor imagery classification.

PubMed

Chiarelli, Antonio Maria; Croce, Pierpaolo; Merla, Arcangelo; Zappasodi, Filippo

2018-06-01

Brain-computer interface (BCI) refers to procedures that link the central nervous system to a device. BCI was historically performed using electroencephalography (EEG). In the last years, encouraging results were obtained by combining EEG with other neuroimaging technologies, such as functional near infrared spectroscopy (fNIRS). A crucial step of BCI is brain state classification from recorded signal features. Deep artificial neural networks (DNNs) recently reached unprecedented complex classification outcomes. These performances were achieved through increased computational power, efficient learning algorithms, valuable activation functions, and restricted or back-fed neurons connections. By expecting significant overall BCI performances, we investigated the capabilities of combining EEG and fNIRS recordings with state-of-the-art deep learning procedures. We performed a guided left and right hand motor imagery task on 15 subjects with a fixed classification response time of 1 s and overall experiment length of 10 min. Left versus right classification accuracy of a DNN in the multi-modal recording modality was estimated and it was compared to standalone EEG and fNIRS and other classifiers. At a group level we obtained significant increase in performance when considering multi-modal recordings and DNN classifier with synergistic effect. BCI performances can be significantly improved by employing multi-modal recordings that provide electrical and hemodynamic brain activity information, in combination with advanced non-linear deep learning classification procedures.

Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification

NASA Astrophysics Data System (ADS)

Chiarelli, Antonio Maria; Croce, Pierpaolo; Merla, Arcangelo; Zappasodi, Filippo

2018-06-01

Objective. Brain–computer interface (BCI) refers to procedures that link the central nervous system to a device. BCI was historically performed using electroencephalography (EEG). In the last years, encouraging results were obtained by combining EEG with other neuroimaging technologies, such as functional near infrared spectroscopy (fNIRS). A crucial step of BCI is brain state classification from recorded signal features. Deep artificial neural networks (DNNs) recently reached unprecedented complex classification outcomes. These performances were achieved through increased computational power, efficient learning algorithms, valuable activation functions, and restricted or back-fed neurons connections. By expecting significant overall BCI performances, we investigated the capabilities of combining EEG and fNIRS recordings with state-of-the-art deep learning procedures. Approach. We performed a guided left and right hand motor imagery task on 15 subjects with a fixed classification response time of 1 s and overall experiment length of 10 min. Left versus right classification accuracy of a DNN in the multi-modal recording modality was estimated and it was compared to standalone EEG and fNIRS and other classifiers. Main results. At a group level we obtained significant increase in performance when considering multi-modal recordings and DNN classifier with synergistic effect. Significance. BCI performances can be significantly improved by employing multi-modal recordings that provide electrical and hemodynamic brain activity information, in combination with advanced non-linear deep learning classification procedures.

Human Pacman: A Mobile Augmented Reality Entertainment System Based on Physical, Social, and Ubiquitous Computing

NASA Astrophysics Data System (ADS)

Cheok, Adrian David

This chapter details the Human Pacman system to illuminate entertainment computing which ventures to embed the natural physical world seamlessly with a fantasy virtual playground by capitalizing on infrastructure provided by mobile computing, wireless LAN, and ubiquitous computing. With Human Pacman, we have a physical role-playing computer fantasy together with real human-social and mobile-gaming that emphasizes on collaboration and competition between players in a wide outdoor physical area that allows natural wide-area human-physical movements. Pacmen and Ghosts are now real human players in the real world experiencing mixed computer graphics fantasy-reality provided by using the wearable computers on them. Virtual cookies and actual tangible physical objects are incorporated into the game play to provide novel experiences of seamless transitions between the real and virtual worlds. This is an example of a new form of gaming that anchors on physicality, mobility, social interaction, and ubiquitous computing.

Seeking Synthesis: The Integrative Problem in Understanding Language and Its Evolution.

PubMed

Dale, Rick; Kello, Christopher T; Schoenemann, P Thomas

2016-04-01

We discuss two problems for a general scientific understanding of language, sequences and synergies: how language is an intricately sequenced behavior and how language is manifested as a multidimensionally structured behavior. Though both are central in our understanding, we observe that the former tends to be studied more than the latter. We consider very general conditions that hold in human brain evolution and its computational implications, and identify multimodal and multiscale organization as two key characteristics of emerging cognitive function in our species. This suggests that human brains, and cognitive function specifically, became more adept at integrating diverse information sources and operating at multiple levels for linguistic performance. We argue that framing language evolution, learning, and use in terms of synergies suggests new research questions, and it may be a fruitful direction for new developments in theory and modeling of language as an integrated system. Copyright © 2016 Cognitive Science Society, Inc.

Augmented Robotics Dialog System for Enhancing Human-Robot Interaction.

PubMed

Alonso-Martín, Fernando; Castro-González, Aĺvaro; Luengo, Francisco Javier Fernandez de Gorostiza; Salichs, Miguel Ángel

2015-07-03

Augmented reality, augmented television and second screen are cutting edge technologies that provide end users extra and enhanced information related to certain events in real time. This enriched information helps users better understand such events, at the same time providing a more satisfactory experience. In the present paper, we apply this main idea to human-robot interaction (HRI), to how users and robots interchange information. The ultimate goal of this paper is to improve the quality of HRI, developing a new dialog manager system that incorporates enriched information from the semantic web. This work presents the augmented robotic dialog system (ARDS), which uses natural language understanding mechanisms to provide two features: (i) a non-grammar multimodal input (verbal and/or written) text; and (ii) a contextualization of the information conveyed in the interaction. This contextualization is achieved by information enrichment techniques that link the extracted information from the dialog with extra information about the world available in semantic knowledge bases. This enriched or contextualized information (information enrichment, semantic enhancement or contextualized information are used interchangeably in the rest of this paper) offers many possibilities in terms of HRI. For instance, it can enhance the robot's pro-activeness during a human-robot dialog (the enriched information can be used to propose new topics during the dialog, while ensuring a coherent interaction). Another possibility is to display additional multimedia content related to the enriched information on a visual device. This paper describes the ARDS and shows a proof of concept of its applications.

Air Defense: A Computer Game for Research in Human Performance.

DTIC Science & Technology

1981-07-01

warfare (ANW) threat analysis. M’ajor elements of the threat analysis problem \\\\,erc eoibedded in an interactive air detoense game controlled by a...The game requires sustained attention to a complex and interactive "hostile" environment, provides proper experimental control of relevant variables...AD-A102 725 NAVY PERSONNEL RESEARCH AND DEVELOPMENT CENTER SAN DETC F/6 5/10 AIR DEFENSE: A COMPUTER GAME FOR RESEARCH IN HUMAN PERFORMANCE.(U) JUL

Intelligent Adaptive Systems: Literature Research of Design Guidance for Intelligent Adaptive Automation and Interfaces

DTIC Science & Technology

2007-09-01

behaviour based on past experience of interacting with the operator), and mobile (i.e., can move themselves from one machine to another). Edwards argues that...Sofge, D., Bugajska, M., Adams, W., Perzanowski, D., and Schultz, A. (2003). Agent-based Multimodal Interface for Dynamically Autonomous Mobile Robots...based architecture can provide a natural and scalable approach to implementing a multimodal interface to control mobile robots through dynamic

Open-Box Muscle-Computer Interface: Introduction to Human-Computer Interactions in Bioengineering, Physiology, and Neuroscience Courses

ERIC Educational Resources Information Center

Landa-Jiménez, M. A.; González-Gaspar, P.; Pérez-Estudillo, C.; López-Meraz, M. L.; Morgado-Valle, C.; Beltran-Parrazal, L.

2016-01-01

A Muscle-Computer Interface (muCI) is a human-machine system that uses electromyographic (EMG) signals to communicate with a computer. Surface EMG (sEMG) signals are currently used to command robotic devices, such as robotic arms and hands, and mobile robots, such as wheelchairs. These signals reflect the motor intention of a user before the…

[An interactive three-dimensional model of the human body].

PubMed

Liem, S L

2009-01-01

Driven by advanced computer technology, it is now possible to show the human anatomy on a computer. On the internet, the Visible Body programme makes it possible to navigate in all directions through the anatomical structures of the human body, using mouse and keyboard. Visible Body is a wonderful tool to give insight in the human structures, body functions and organs.

Recent technology products from Space Human Factors research

NASA Technical Reports Server (NTRS)

Jenkins, James P.

1991-01-01

The goals of the NASA Space Human Factors program and the research carried out concerning human factors are discussed with emphasis given to the development of human performance models, data, and tools. The major products from this program are described, which include the Laser Anthropometric Mapping System; a model of the human body for evaluating the kinematics and dynamics of human motion and strength in microgravity environment; an operational experience data base for verifying and validating the data repository of manned space flights; the Operational Experience Database Taxonomy; and a human-computer interaction laboratory whose products are the display softaware and requirements and the guideline documents and standards for applications on human-computer interaction. Special attention is given to the 'Convoltron', a prototype version of a signal processor for synthesizing the head-related transfer functions.

A Perspective on Computational Human Performance Models as Design Tools

NASA Technical Reports Server (NTRS)

Jones, Patricia M.

2010-01-01

The design of interactive systems, including levels of automation, displays, and controls, is usually based on design guidelines and iterative empirical prototyping. A complementary approach is to use computational human performance models to evaluate designs. An integrated strategy of model-based and empirical test and evaluation activities is particularly attractive as a methodology for verification and validation of human-rated systems for commercial space. This talk will review several computational human performance modeling approaches and their applicability to design of display and control requirements.

Motor priming in virtual reality can augment motor-imagery training efficacy in restorative brain-computer interaction: a within-subject analysis.

PubMed

Vourvopoulos, Athanasios; Bermúdez I Badia, Sergi

2016-08-09

The use of Brain-Computer Interface (BCI) technology in neurorehabilitation provides new strategies to overcome stroke-related motor limitations. Recent studies demonstrated the brain's capacity for functional and structural plasticity through BCI. However, it is not fully clear how we can take full advantage of the neurobiological mechanisms underlying recovery and how to maximize restoration through BCI. In this study we investigate the role of multimodal virtual reality (VR) simulations and motor priming (MP) in an upper limb motor-imagery BCI task in order to maximize the engagement of sensory-motor networks in a broad range of patients who can benefit from virtual rehabilitation training. In order to investigate how different BCI paradigms impact brain activation, we designed 3 experimental conditions in a within-subject design, including an immersive Multimodal Virtual Reality with Motor Priming (VRMP) condition where users had to perform motor-execution before BCI training, an immersive Multimodal VR condition, and a control condition with standard 2D feedback. Further, these were also compared to overt motor-execution. Finally, a set of questionnaires were used to gather subjective data on Workload, Kinesthetic Imagery and Presence. Our findings show increased capacity to modulate and enhance brain activity patterns in all extracted EEG rhythms matching more closely those present during motor-execution and also a strong relationship between electrophysiological data and subjective experience. Our data suggest that both VR and particularly MP can enhance the activation of brain patterns present during overt motor-execution. Further, we show changes in the interhemispheric EEG balance, which might play an important role in the promotion of neural activation and neuroplastic changes in stroke patients in a motor-imagery neurofeedback paradigm. In addition, electrophysiological correlates of psychophysiological responses provide us with valuable information about the motor and affective state of the user that has the potential to be used to predict MI-BCI training outcome based on user's profile. Finally, we propose a BCI paradigm in VR, which gives the possibility of motor priming for patients with low level of motor control.

Multimodal imaging of lung cancer and its microenvironment (Conference Presentation)

NASA Astrophysics Data System (ADS)

Hariri, Lida P.; Niederst, Matthew J.; Mulvey, Hillary; Adams, David C.; Hu, Haichuan; Chico Calero, Isabel; Szabari, Margit V.; Vakoc, Benjamin J.; Hasan, Tayyaba; Bouma, Brett E.; Engelman, Jeffrey A.; Suter, Melissa J.

2016-03-01

Despite significant advances in targeted therapies for lung cancer, nearly all patients develop drug resistance within 6-12 months and prognosis remains poor. Developing drug resistance is a progressive process that involves tumor cells and their microenvironment. We hypothesize that microenvironment factors alter tumor growth and response to targeted therapy. We conducted in vitro studies in human EGFR-mutant lung carcinoma cells, and demonstrated that factors secreted from lung fibroblasts results in increased tumor cell survival during targeted therapy with EGFR inhibitor, gefitinib. We also demonstrated that increased environment stiffness results in increased tumor survival during gefitinib therapy. In order to test our hypothesis in vivo, we developed a multimodal optical imaging protocol for preclinical intravital imaging in mouse models to assess tumor and its microenvironment over time. We have successfully conducted multimodal imaging of dorsal skinfold chamber (DSC) window mice implanted with GFP-labeled human EGFR mutant lung carcinoma cells and visualized changes in tumor development and microenvironment facets over time. Multimodal imaging included structural OCT to assess tumor viability and necrosis, polarization-sensitive OCT to measure tissue birefringence for collagen/fibroblast detection, and Doppler OCT to assess tumor vasculature. Confocal imaging was also performed for high-resolution visualization of EGFR-mutant lung cancer cells labeled with GFP, and was coregistered with OCT. Our results demonstrated that stromal support and vascular growth are essential to tumor progression. Multimodal imaging is a useful tool to assess tumor and its microenvironment over time.

Neurocognitive insights on conceptual knowledge and its breakdown

PubMed Central

Lambon Ralph, Matthew A.

2014-01-01

Conceptual knowledge reflects our multi-modal ‘semantic database’. As such, it brings meaning to all verbal and non-verbal stimuli, is the foundation for verbal and non-verbal expression and provides the basis for computing appropriate semantic generalizations. Multiple disciplines (e.g. philosophy, cognitive science, cognitive neuroscience and behavioural neurology) have striven to answer the questions of how concepts are formed, how they are represented in the brain and how they break down differentially in various neurological patient groups. A long-standing and prominent hypothesis is that concepts are distilled from our multi-modal verbal and non-verbal experience such that sensation in one modality (e.g. the smell of an apple) not only activates the intramodality long-term knowledge, but also reactivates the relevant intermodality information about that item (i.e. all the things you know about and can do with an apple). This multi-modal view of conceptualization fits with contemporary functional neuroimaging studies that observe systematic variation of activation across different modality-specific association regions dependent on the conceptual category or type of information. A second vein of interdisciplinary work argues, however, that even a smorgasbord of multi-modal features is insufficient to build coherent, generalizable concepts. Instead, an additional process or intermediate representation is required. Recent multidisciplinary work, which combines neuropsychology, neuroscience and computational models, offers evidence that conceptualization follows from a combination of modality-specific sources of information plus a transmodal ‘hub’ representational system that is supported primarily by regions within the anterior temporal lobe, bilaterally. PMID:24324236

Adaptive Annealed Importance Sampling for Multimodal Posterior Exploration and Model Selection with Application to Extrasolar Planet Detection

NASA Astrophysics Data System (ADS)

Liu, Bin

2014-07-01

We describe an algorithm that can adaptively provide mixture summaries of multimodal posterior distributions. The parameter space of the involved posteriors ranges in size from a few dimensions to dozens of dimensions. This work was motivated by an astrophysical problem called extrasolar planet (exoplanet) detection, wherein the computation of stochastic integrals that are required for Bayesian model comparison is challenging. The difficulty comes from the highly nonlinear models that lead to multimodal posterior distributions. We resort to importance sampling (IS) to estimate the integrals, and thus translate the problem to be how to find a parametric approximation of the posterior. To capture the multimodal structure in the posterior, we initialize a mixture proposal distribution and then tailor its parameters elaborately to make it resemble the posterior to the greatest extent possible. We use the effective sample size (ESS) calculated based on the IS draws to measure the degree of approximation. The bigger the ESS is, the better the proposal resembles the posterior. A difficulty within this tailoring operation lies in the adjustment of the number of mixing components in the mixture proposal. Brute force methods just preset it as a large constant, which leads to an increase in the required computational resources. We provide an iterative delete/merge/add process, which works in tandem with an expectation-maximization step to tailor such a number online. The efficiency of our proposed method is tested via both simulation studies and real exoplanet data analysis.

Bilinearization of the generalized coupled nonlinear Schrödinger equation with variable coefficients and gain and dark-bright pair soliton solutions.

PubMed

Chakraborty, Sushmita; Nandy, Sudipta; Barthakur, Abhijit

2015-02-01

We investigate coupled nonlinear Schrödinger equations (NLSEs) with variable coefficients and gain. The coupled NLSE is a model equation for optical soliton propagation and their interaction in a multimode fiber medium or in a fiber array. By using Hirota's bilinear method, we obtain the bright-bright, dark-bright combinations of a one-soliton solution (1SS) and two-soliton solutions (2SS) for an n-coupled NLSE with variable coefficients and gain. Crucial properties of two-soliton (dark-bright pair) interactions, such as elastic and inelastic interactions and the dynamics of soliton bound states, are studied using asymptotic analysis and graphical analysis. We show that a bright 2-soliton, in addition to elastic interactions, also exhibits multiple inelastic interactions. A dark 2-soliton, on the other hand, exhibits only elastic interactions. We also observe a breatherlike structure of a bright 2-soliton, a feature that become prominent with gain and disappears as the amplitude acquires a minimum value, and after that the solitons remain parallel. The dark 2-soliton, however, remains parallel irrespective of the gain. The results found by us might be useful for applications in soliton control, a fiber amplifier, all optical switching, and optical computing.

Multimodal Event Detection in Twitter Hashtag Networks

DOE PAGES

Yilmaz, Yasin; Hero, Alfred O.

2016-07-01

In this study, event detection in a multimodal Twitter dataset is considered. We treat the hashtags in the dataset as instances with two modes: text and geolocation features. The text feature consists of a bag-of-words representation. The geolocation feature consists of geotags (i.e., geographical coordinates) of the tweets. Fusing the multimodal data we aim to detect, in terms of topic and geolocation, the interesting events and the associated hashtags. To this end, a generative latent variable model is assumed, and a generalized expectation-maximization (EM) algorithm is derived to learn the model parameters. The proposed method is computationally efficient, and lendsmore » itself to big datasets. Lastly, experimental results on a Twitter dataset from August 2014 show the efficacy of the proposed method.« less

Grammar Is a System That Characterizes Talk in Interaction

PubMed Central

Ginzburg, Jonathan; Poesio, Massimo

2016-01-01

Much of contemporary mainstream formal grammar theory is unable to provide analyses for language as it occurs in actual spoken interaction. Its analyses are developed for a cleaned up version of language which omits the disfluencies, non-sentential utterances, gestures, and many other phenomena that are ubiquitous in spoken language. Using evidence from linguistics, conversation analysis, multimodal communication, psychology, language acquisition, and neuroscience, we show these aspects of language use are rule governed in much the same way as phenomena captured by conventional grammars. Furthermore, we argue that over the past few years some of the tools required to provide a precise characterizations of such phenomena have begun to emerge in theoretical and computational linguistics; hence, there is no reason for treating them as “second class citizens” other than pre-theoretical assumptions about what should fall under the purview of grammar. Finally, we suggest that grammar formalisms covering such phenomena would provide a better foundation not just for linguistic analysis of face-to-face interaction, but also for sister disciplines, such as research on spoken dialogue systems and/or psychological work on language acquisition. PMID:28066279

An Affordance-Based Framework for Human Computation and Human-Computer Collaboration.

PubMed

Crouser, R J; Chang, R

2012-12-01

Visual Analytics is "the science of analytical reasoning facilitated by visual interactive interfaces". The goal of this field is to develop tools and methodologies for approaching problems whose size and complexity render them intractable without the close coupling of both human and machine analysis. Researchers have explored this coupling in many venues: VAST, Vis, InfoVis, CHI, KDD, IUI, and more. While there have been myriad promising examples of human-computer collaboration, there exists no common language for comparing systems or describing the benefits afforded by designing for such collaboration. We argue that this area would benefit significantly from consensus about the design attributes that define and distinguish existing techniques. In this work, we have reviewed 1,271 papers from many of the top-ranking conferences in visual analytics, human-computer interaction, and visualization. From these, we have identified 49 papers that are representative of the study of human-computer collaborative problem-solving, and provide a thorough overview of the current state-of-the-art. Our analysis has uncovered key patterns of design hinging on human and machine-intelligence affordances, and also indicates unexplored avenues in the study of this area. The results of this analysis provide a common framework for understanding these seemingly disparate branches of inquiry, which we hope will motivate future work in the field.

Projecting Grammatical Features in Nominals: Cognitive Processing Theory & Computational Implementation

DTIC Science & Technology

2010-03-01

functionality and plausibility distinguishes this research from most research in computational linguistics and computational psycholinguistics . The... Psycholinguistic Theory There is extensive psycholinguistic evidence that human language processing is essentially incremental and interactive...challenges of psycholinguistic research is to explain how humans can process language effortlessly and accurately given the complexity and ambiguity that is

Assessing the Purpose and Importance University Students Attribute to Current ICT Applications

ERIC Educational Resources Information Center

DiGiuseppe, Maurice; Partosoedarso, Elita

2014-01-01

In this study we surveyed students in a mid-sized university in Ontario, Canada to explore various aspects associated with their use of computer-based applications. For the purpose of analysis, the computer applications under study were categorized according to the Human-Computer-Human Interaction (HCHI) model of Desjardins (2005) in which…

Graduate Education and Simulation Training for CBRNE Disasters Using a Multimodal Approach to Learning. Part 1: Education and Training from a Human-Performance Perspective

DTIC Science & Technology

2013-08-01

Learning Styles . Available at http://www2.umist.ac.uk.staff/talsc/TaLSC/VARK/default.htm; accessed Aug 9, 2010. 14. Joels M, Pu Z, Wiegert O, Oitzl M...and Simulation Training   for CBRNE Disasters Using a Multimodal  Approach to  Learning   Part 1: Education and Training from a Human‐ Performance...Graduate Education and Simulation Training for CBRNE Disasters Using a Multimodal 5a. CONTRACT NUMBER Approach to Learning . Part 1: Education and

Definition Of Touch-Sensitive Zones For Graphical Displays

NASA Technical Reports Server (NTRS)

Monroe, Burt L., III; Jones, Denise R.

1988-01-01

Touch zones defined simply by touching, while editing done automatically. Development of touch-screen interactive computing system, tedious task. Interactive Editor for Definition of Touch-Sensitive Zones computer program increases efficiency of human/machine communications by enabling user to define each zone interactively, minimizing redundancy in programming and eliminating need for manual computation of boundaries of touch areas. Information produced during editing process written to data file, to which access gained when needed by application program.

Individual differences in cognition, affect, and performance: Behavioral, neuroimaging, and molecular genetic approaches

PubMed Central

Parasuraman, Raja; Jiang, Yang

2012-01-01

We describe the use of behavioral, neuroimaging, and genetic methods to examine individual differences in cognition and affect, guided by three criteria: (1) relevance to human performance in work and everyday settings; (2) interactions between working memory, decision-making, and affective processing; and (3) examination of individual differences. The results of behavioral, functional MRI (fMRI), event-related potential (ERP), and molecular genetic studies show that analyses at the group level often mask important findings associated with sub-groups of individuals. Dopaminergic/noradrenergic genes influencing prefrontal cortex activity contribute to inter-individual variation in working memory and decision behavior, including performance in complex simulations of military decision-making. The interactive influences of individual differences in anxiety, sensation seeking, and boredom susceptibility on evaluative decision-making can be systematically described using ERP and fMRI methods. We conclude that a multi-modal neuroergonomic approach to examining brain function (using both neuroimaging and molecular genetics) can be usefully applied to understanding individual differences in cognition and affect and has implications for human performance at work. PMID:21569853

Virtual Reality for Enhanced Ecological Validity and Experimental Control in the Clinical, Affective and Social Neurosciences

PubMed Central

Parsons, Thomas D.

2015-01-01

An essential tension can be found between researchers interested in ecological validity and those concerned with maintaining experimental control. Research in the human neurosciences often involves the use of simple and static stimuli lacking many of the potentially important aspects of real world activities and interactions. While this research is valuable, there is a growing interest in the human neurosciences to use cues about target states in the real world via multimodal scenarios that involve visual, semantic, and prosodic information. These scenarios should include dynamic stimuli presented concurrently or serially in a manner that allows researchers to assess the integrative processes carried out by perceivers over time. Furthermore, there is growing interest in contextually embedded stimuli that can constrain participant interpretations of cues about a target’s internal states. Virtual reality environments proffer assessment paradigms that combine the experimental control of laboratory measures with emotionally engaging background narratives to enhance affective experience and social interactions. The present review highlights the potential of virtual reality environments for enhanced ecological validity in the clinical, affective, and social neurosciences. PMID:26696869

Multisensory integration of colors and scents: insights from bees and flowers.

PubMed

Leonard, Anne S; Masek, Pavel

2014-06-01

Karl von Frisch's studies of bees' color vision and chemical senses opened a window into the perceptual world of a species other than our own. A century of subsequent research on bees' visual and olfactory systems has developed along two productive but independent trajectories, leaving the questions of how and why bees use these two senses in concert largely unexplored. Given current interest in multimodal communication and recently discovered interplay between olfaction and vision in humans and Drosophila, understanding multisensory integration in bees is an opportunity to advance knowledge across fields. Using a classic ethological framework, we formulate proximate and ultimate perspectives on bees' use of multisensory stimuli. We discuss interactions between scent and color in the context of bee cognition and perception, focusing on mechanistic and functional approaches, and we highlight opportunities to further explore the development and evolution of multisensory integration. We argue that although the visual and olfactory worlds of bees are perhaps the best-studied of any non-human species, research focusing on the interactions between these two sensory modalities is vitally needed.

Virtual Reality for Enhanced Ecological Validity and Experimental Control in the Clinical, Affective and Social Neurosciences.

PubMed

Parsons, Thomas D

2015-01-01

An essential tension can be found between researchers interested in ecological validity and those concerned with maintaining experimental control. Research in the human neurosciences often involves the use of simple and static stimuli lacking many of the potentially important aspects of real world activities and interactions. While this research is valuable, there is a growing interest in the human neurosciences to use cues about target states in the real world via multimodal scenarios that involve visual, semantic, and prosodic information. These scenarios should include dynamic stimuli presented concurrently or serially in a manner that allows researchers to assess the integrative processes carried out by perceivers over time. Furthermore, there is growing interest in contextually embedded stimuli that can constrain participant interpretations of cues about a target's internal states. Virtual reality environments proffer assessment paradigms that combine the experimental control of laboratory measures with emotionally engaging background narratives to enhance affective experience and social interactions. The present review highlights the potential of virtual reality environments for enhanced ecological validity in the clinical, affective, and social neurosciences.

A Novel Experimental and Analytical Approach to the Multimodal Neural Decoding of Intent During Social Interaction in Freely-behaving Human Infants.

PubMed

Cruz-Garza, Jesus G; Hernandez, Zachery R; Tse, Teresa; Caducoy, Eunice; Abibullaev, Berdakh; Contreras-Vidal, Jose L

2015-10-04

Understanding typical and atypical development remains one of the fundamental questions in developmental human neuroscience. Traditionally, experimental paradigms and analysis tools have been limited to constrained laboratory tasks and contexts due to technical limitations imposed by the available set of measuring and analysis techniques and the age of the subjects. These limitations severely limit the study of developmental neural dynamics and associated neural networks engaged in cognition, perception and action in infants performing "in action and in context". This protocol presents a novel approach to study infants and young children as they freely organize their own behavior, and its consequences in a complex, partly unpredictable and highly dynamic environment. The proposed methodology integrates synchronized high-density active scalp electroencephalography (EEG), inertial measurement units (IMUs), video recording and behavioral analysis to capture brain activity and movement non-invasively in freely-behaving infants. This setup allows for the study of neural network dynamics in the developing brain, in action and context, as these networks are recruited during goal-oriented, exploration and social interaction tasks.

From Earth to Space--Advertising Films Created in a Computer-Based Primary School Task

ERIC Educational Resources Information Center

Öman, Anne

2017-01-01

Today, teachers orchestrate computer-based tasks in software applications in Swedish primary schools. Meaning is made through various modes, and multimodal perspectives on literacy have the basic assumption that meaning is made through many representational and communicational resources. The case study presented in this paper has analysed pupils'…

Learner Use of Holistic Language Units in Multimodal, Task-Based Synchronous Computer-Mediated Communication

ERIC Educational Resources Information Center

Collentine, Karina

2009-01-01

Second language acquisition (SLA) researchers strive to understand the language and exchanges that learners generate in synchronous computer-mediated communication (SCMC). Doughty and Long (2003) advocate replacing open-ended SCMC with task-based language teaching (TBLT) design principles. Since most task-based SCMC (TB-SCMC) research addresses an…

Human computer interface guide, revision A

NASA Technical Reports Server (NTRS)

1993-01-01

The Human Computer Interface Guide, SSP 30540, is a reference document for the information systems within the Space Station Freedom Program (SSFP). The Human Computer Interface Guide (HCIG) provides guidelines for the design of computer software that affects human performance, specifically, the human-computer interface. This document contains an introduction and subparagraphs on SSFP computer systems, users, and tasks; guidelines for interactions between users and the SSFP computer systems; human factors evaluation and testing of the user interface system; and example specifications. The contents of this document are intended to be consistent with the tasks and products to be prepared by NASA Work Package Centers and SSFP participants as defined in SSP 30000, Space Station Program Definition and Requirements Document. The Human Computer Interface Guide shall be implemented on all new SSFP contractual and internal activities and shall be included in any existing contracts through contract changes. This document is under the control of the Space Station Control Board, and any changes or revisions will be approved by the deputy director.

SIG -- The Role of Human-Computer Interaction in Next-Generation Control Rooms

DOE Office of Scientific and Technical Information (OSTI.GOV)

Ronald L. Boring; Jacques Hugo; Christian Richard

2005-04-01

The purpose of this CHI Special Interest Group (SIG) is to facilitate the convergence between human-computer interaction (HCI) and control room design. HCI researchers and practitioners actively need to infuse state-of-the-art interface technology into control rooms to meet usability, safety, and regulatory requirements. This SIG outlines potential HCI contributions to instrumentation and control (I&C) and automation in control rooms as well as to general control room design.

Interpersonal Biocybernetics: Connecting Through Social Psychophysiology

NASA Technical Reports Server (NTRS)

Pope, Alan T.; Stephens, Chad L.

2012-01-01

One embodiment of biocybernetic adaptation is a human-computer interaction system designed such that physiological signals modulate the effect that control of a task by other means, usually manual control, has on performance of the task. Such a modulation system enables a variety of human-human interactions based upon physiological self-regulation performance. These interpersonal interactions may be mixes of competition and cooperation for simulation training and/or videogame entertainment

An Empirical Study of User Experience on Touch Mice

ERIC Educational Resources Information Center

Chou, Jyh Rong

2016-01-01

The touch mouse is a new type of computer mouse that provides users with a new way of touch-based environment to interact with computers. For more than a decade, user experience (UX) has grown into a core concept of human-computer interaction (HCI), describing a user's perceptions and responses that result from the use of a product in a particular…

Trends in Human-Computer Interaction to Support Future Intelligence Analysis Capabilities

DTIC Science & Technology

2011-06-01

that allows data to be moved between different computing systems and displays. Figure 4- G-Speak gesture interaction (Oblong, 2011) 5.2 Multitouch ... Multitouch refers to a touchscreen interaction technique in which multiple simultaneous touchpoints and movements can be detected and used to...much of the style of interaction (such as rotate, pinch, zoom and flick movements) found in multitouch devices but can typically recognize more than

Getting seamless care right from the beginning - integrating computers into the human interaction.

PubMed

Pearce, Christopher; Kumarpeli, Pushpa; de Lusignan, Simon

2010-01-01

The digital age is coming to the health space, behind many other fields of society. In part this is because health remains heavily reliant on human interaction. The doctor-patient relationship remains a significant factor in determining patient outcomes. Whilst there are many benefits to E-Health, there are also significant risks if computers are not adequately integrated into this interaction and accurate data are consequently not available on the patient's journey through the health system. Video analysis of routine clinical consultations in Australian and UK primary care. We analyzed 308 consultations (141+167 respectively) from these systems, with an emphasis on how the consultation starts. Australian consultations have a mean duration of 12.7 mins, UK 11.8 mins. In both countries around 7% of consultations are computer initiated. Where doctors engaged with computer use the patient observed the computer screen much more and better records were produced. However, there was suboptimal engagement and poor records and no coding in around 20% of consultations. How the computer is used at the start of the consultation can set the scene for an effective interaction or reflect disengagement from technology and creation of poor records.

IMMERSE: Interactive Mentoring for Multimodal Experiences in Realistic Social Encounters

DTIC Science & Technology

2015-08-28

undergraduates funded by your agreement who graduated during this period and will receive scholarships or fellowships for further studies in science... Player Locomotion 9.2 Interacting with Real and Virtual Objects 9.3 Animation Combinations and Stage Management 10. Recommendations on the Way Ahead...Interaction with Virtual Characters ................................52! 9.1! Player Locomotion

A roadmap to computational social neuroscience.

PubMed

Tognoli, Emmanuelle; Dumas, Guillaume; Kelso, J A Scott

2018-02-01

To complement experimental efforts toward understanding human social interactions at both neural and behavioral levels, two computational approaches are presented: (1) a fully parameterizable mathematical model of a social partner, the Human Dynamic Clamp which, by virtue of experimentally controlled interactions between Virtual Partners and real people, allows for emergent behaviors to be studied; and (2) a multiscale neurocomputational model of social coordination that enables exploration of social self-organization at all levels-from neuronal patterns to people interacting with each other. These complementary frameworks and the cross product of their analysis aim at understanding the fundamental principles governing social behavior.

Optimality in mono- and multisensory map formation.

PubMed

Bürck, Moritz; Friedel, Paul; Sichert, Andreas B; Vossen, Christine; van Hemmen, J Leo

2010-07-01

In the struggle for survival in a complex and dynamic environment, nature has developed a multitude of sophisticated sensory systems. In order to exploit the information provided by these sensory systems, higher vertebrates reconstruct the spatio-temporal environment from each of the sensory systems they have at their disposal. That is, for each modality the animal computes a neuronal representation of the outside world, a monosensory neuronal map. Here we present a universal framework that allows to calculate the specific layout of the involved neuronal network by means of a general mathematical principle, viz., stochastic optimality. In order to illustrate the use of this theoretical framework, we provide a step-by-step tutorial of how to apply our model. In so doing, we present a spatial and a temporal example of optimal stimulus reconstruction which underline the advantages of our approach. That is, given a known physical signal transmission and rudimental knowledge of the detection process, our approach allows to estimate the possible performance and to predict neuronal properties of biological sensory systems. Finally, information from different sensory modalities has to be integrated so as to gain a unified perception of reality for further processing, e.g., for distinct motor commands. We briefly discuss concepts of multimodal interaction and how a multimodal space can evolve by alignment of monosensory maps.

Interaction between visual and chemical cues in a Liolaemus lizard: a multimodal approach.

PubMed

Vicente, Natalin S; Halloy, Monique

2017-12-01

Multimodal communication involves the use of signals and cues across two or more sensory modalities. The genus Liolaemus (Iguania: Liolaemidae) offers a great potential for studies on the ecology and evolution of multimodal communication, including visual and chemical signals. In this study, we analyzed the response of male and female Liolaemus pacha to chemical, visual and combined (multimodal) stimuli. Using cue-isolation tests, we registered the number of tongue flicks and headbob displays from exposure to signals in each modality. Number of tongue flicks was greater when a chemical stimulus was presented alone than in the presence of visual or multimodal stimuli. In contrast, headbob displays were fewer in number with visual and chemical stimuli alone, but significantly higher in number when combined. Female signallers triggered significantly more tongue flicks than male signallers, suggesting that chemical cues are involved in sexual recognition. We did not find an inhibition between chemical and visual cues. On the contrary, we observed a dominance of the chemical modality, because when presented with visual stimuli, lizards also responded with more tongue flicks than headbob displays. The total response produced by multimodal stimuli was similar to that of the chemical stimuli alone, possibly suggesting non-redundancy. We discuss whether the visual component of a multimodal signal could attract attention at a distance, increasing the effectiveness of transmission and reception of the information in chemical cues. Copyright © 2017 Elsevier GmbH. All rights reserved.

Emerging Themes in Image Informatics and Molecular Analysis for Digital Pathology.

PubMed

Bhargava, Rohit; Madabhushi, Anant

2016-07-11

Pathology is essential for research in disease and development, as well as for clinical decision making. For more than 100 years, pathology practice has involved analyzing images of stained, thin tissue sections by a trained human using an optical microscope. Technological advances are now driving major changes in this paradigm toward digital pathology (DP). The digital transformation of pathology goes beyond recording, archiving, and retrieving images, providing new computational tools to inform better decision making for precision medicine. First, we discuss some emerging innovations in both computational image analytics and imaging instrumentation in DP. Second, we discuss molecular contrast in pathology. Molecular DP has traditionally been an extension of pathology with molecularly specific dyes. Label-free, spectroscopic images are rapidly emerging as another important information source, and we describe the benefits and potential of this evolution. Third, we describe multimodal DP, which is enabled by computational algorithms and combines the best characteristics of structural and molecular pathology. Finally, we provide examples of application areas in telepathology, education, and precision medicine. We conclude by discussing challenges and emerging opportunities in this area.

Emerging Themes in Image Informatics and Molecular Analysis for Digital Pathology

PubMed Central

Bhargava, Rohit; Madabhushi, Anant

2017-01-01

Pathology is essential for research in disease and development, as well as for clinical decision making. For more than 100 years, pathology practice has involved analyzing images of stained, thin tissue sections by a trained human using an optical microscope. Technological advances are now driving major changes in this paradigm toward digital pathology (DP). The digital transformation of pathology goes beyond recording, archiving, and retrieving images, providing new computational tools to inform better decision making for precision medicine. First, we discuss some emerging innovations in both computational image analytics and imaging instrumentation in DP. Second, we discuss molecular contrast in pathology. Molecular DP has traditionally been an extension of pathology with molecularly specific dyes. Label-free, spectroscopic images are rapidly emerging as another important information source, and we describe the benefits and potential of this evolution. Third, we describe multimodal DP, which is enabled by computational algorithms and combines the best characteristics of structural and molecular pathology. Finally, we provide examples of application areas in telepathology, education, and precision medicine. We conclude by discussing challenges and emerging opportunities in this area. PMID:27420575

Using Multi-modal Sensing for Human Activity Modeling in the Real World

NASA Astrophysics Data System (ADS)

Harrison, Beverly L.; Consolvo, Sunny; Choudhury, Tanzeem

Traditionally smart environments have been understood to represent those (often physical) spaces where computation is embedded into the users' surrounding infrastructure, buildings, homes, and workplaces. Users of this "smartness" move in and out of these spaces. Ambient intelligence assumes that users are automatically and seamlessly provided with context-aware, adaptive information, applications and even sensing - though this remains a significant challenge even when limited to these specialized, instrumented locales. Since not all environments are "smart" the experience is not a pervasive one; rather, users move between these intelligent islands of computationally enhanced space while we still aspire to achieve a more ideal anytime, anywhere experience. Two key technological trends are helping to bridge the gap between these smart environments and make the associated experience more persistent and pervasive. Smaller and more computationally sophisticated mobile devices allow sensing, communication, and services to be more directly and continuously experienced by user. Improved infrastructure and the availability of uninterrupted data streams, for instance location-based data, enable new services and applications to persist across environments.

DOE Office of Scientific and Technical Information (OSTI.GOV)

Zhu, Yitan; Xu, Yanxun; Helseth, Donald L.

Background: Genetic interactions play a critical role in cancer development. Existing knowledge about cancer genetic interactions is incomplete, especially lacking evidences derived from large-scale cancer genomics data. The Cancer Genome Atlas (TCGA) produces multimodal measurements across genomics and features of thousands of tumors, which provide an unprecedented opportunity to investigate the interplays of genes in cancer. Methods: We introduce Zodiac, a computational tool and resource to integrate existing knowledge about cancer genetic interactions with new information contained in TCGA data. It is an evolution of existing knowledge by treating it as a prior graph, integrating it with a likelihood modelmore » derived by Bayesian graphical model based on TCGA data, and producing a posterior graph as updated and data-enhanced knowledge. In short, Zodiac realizes “Prior interaction map + TCGA data → Posterior interaction map.” Results: Zodiac provides molecular interactions for about 200 million pairs of genes. All the results are generated from a big-data analysis and organized into a comprehensive database allowing customized search. In addition, Zodiac provides data processing and analysis tools that allow users to customize the prior networks and update the genetic pathways of their interest. Zodiac is publicly available at www.compgenome.org/ZODIAC. Conclusions: Zodiac recapitulates and extends existing knowledge of molecular interactions in cancer. It can be used to explore novel gene-gene interactions, transcriptional regulation, and other types of molecular interplays in cancer.« less

Evolving technologies for Space Station Freedom computer-based workstations

NASA Technical Reports Server (NTRS)

Jensen, Dean G.; Rudisill, Marianne

1990-01-01

Viewgraphs on evolving technologies for Space Station Freedom computer-based workstations are presented. The human-computer computer software environment modules are described. The following topics are addressed: command and control workstation concept; cupola workstation concept; Japanese experiment module RMS workstation concept; remote devices controlled from workstations; orbital maneuvering vehicle free flyer; remote manipulator system; Japanese experiment module exposed facility; Japanese experiment module small fine arm; flight telerobotic servicer; human-computer interaction; and workstation/robotics related activities.

Computer-assisted visual interactive recognition and its prospects of implementation over the Internet

NASA Astrophysics Data System (ADS)

Zou, Jie; Gattani, Abhishek

2005-01-01

When completely automated systems don't yield acceptable accuracy, many practical pattern recognition systems involve the human either at the beginning (pre-processing) or towards the end (handling rejects). We believe that it may be more useful to involve the human throughout the recognition process rather than just at the beginning or end. We describe a methodology of interactive visual recognition for human-centered low-throughput applications, Computer Assisted Visual InterActive Recognition (CAVIAR), and discuss the prospects of implementing CAVIAR over the Internet. The novelty of CAVIAR is image-based interaction through a domain-specific parameterized geometrical model, which reduces the semantic gap between humans and computers. The user may interact with the computer anytime that she considers its response unsatisfactory. The interaction improves the accuracy of the classification features by improving the fit of the computer-proposed model. The computer makes subsequent use of the parameters of the improved model to refine not only its own statistical model-fitting process, but also its internal classifier. The CAVIAR methodology was applied to implement a flower recognition system. The principal conclusions from the evaluation of the system include: 1) the average recognition time of the CAVIAR system is significantly shorter than that of the unaided human; 2) its accuracy is significantly higher than that of the unaided machine; 3) it can be initialized with as few as one training sample per class and still achieve high accuracy; and 4) it demonstrates a self-learning ability. We have also implemented a Mobile CAVIAR system, where a pocket PC, as a client, connects to a server through wireless communication. The motivation behind a mobile platform for CAVIAR is to apply the methodology in a human-centered pervasive environment, where the user can seamlessly interact with the system for classifying field-data. Deploying CAVIAR to a networked mobile platform poses the challenge of classifying field images and programming under constraints of display size, network bandwidth, processor speed, and memory size. Editing of the computer-proposed model is performed on the handheld while statistical model fitting and classification take place on the server. The possibility that the user can easily take several photos of the object poses an interesting information fusion problem. The advantage of the Internet is that the patterns identified by different users can be pooled together to benefit all peer users. When users identify patterns with CAVIAR in a networked setting, they also collect training samples and provide opportunities for machine learning from their intervention. CAVIAR implemented over the Internet provides a perfect test bed for, and extends, the concept of Open Mind Initiative proposed by David Stork. Our experimental evaluation focuses on human time, machine and human accuracy, and machine learning. We devoted much effort to evaluating the use of our image-based user interface and on developing principles for the evaluation of interactive pattern recognition system. The Internet architecture and Mobile CAVIAR methodology have many applications. We are exploring in the directions of teledermatology, face recognition, and education.

A Human Factors Framework for Payload Display Design

NASA Technical Reports Server (NTRS)

Dunn, Mariea C.; Hutchinson, Sonya L.

1998-01-01

During missions to space, one charge of the astronaut crew is to conduct research experiments. These experiments, referred to as payloads, typically are controlled by computers. Crewmembers interact with payload computers by using visual interfaces or displays. To enhance the safety, productivity, and efficiency of crewmember interaction with payload displays, particular attention must be paid to the usability of these displays. Enhancing display usability requires adoption of a design process that incorporates human factors engineering principles at each stage. This paper presents a proposed framework for incorporating human factors engineering principles into the payload display design process.

deepNF: Deep network fusion for protein function prediction.

PubMed

Gligorijevic, Vladimir; Barot, Meet; Bonneau, Richard

2018-06-01

The prevalence of high-throughput experimental methods has resulted in an abundance of large-scale molecular and functional interaction networks. The connectivity of these networks provides a rich source of information for inferring functional annotations for genes and proteins. An important challenge has been to develop methods for combining these heterogeneous networks to extract useful protein feature representations for function prediction. Most of the existing approaches for network integration use shallow models that encounter difficulty in capturing complex and highly-nonlinear network structures. Thus, we propose deepNF, a network fusion method based on Multimodal Deep Autoencoders to extract high-level features of proteins from multiple heterogeneous interaction networks. We apply this method to combine STRING networks to construct a common low-dimensional representation containing high-level protein features. We use separate layers for different network types in the early stages of the multimodal autoencoder, later connecting all the layers into a single bottleneck layer from which we extract features to predict protein function. We compare the cross-validation and temporal holdout predictive performance of our method with state-of-the-art methods, including the recently proposed method Mashup. Our results show that our method outperforms previous methods for both human and yeast STRING networks. We also show substantial improvement in the performance of our method in predicting GO terms of varying type and specificity. deepNF is freely available at: https://github.com/VGligorijevic/deepNF. vgligorijevic@flatironinstitute.org, rb133@nyu.edu. Supplementary data are available at Bioinformatics online.

The Design of Hand Gestures for Human-Computer Interaction: Lessons from Sign Language Interpreters

PubMed Central

Rempel, David; Camilleri, Matt J.; Lee, David L.

2015-01-01

The design and selection of 3D modeled hand gestures for human-computer interaction should follow principles of natural language combined with the need to optimize gesture contrast and recognition. The selection should also consider the discomfort and fatigue associated with distinct hand postures and motions, especially for common commands. Sign language interpreters have extensive and unique experience forming hand gestures and many suffer from hand pain while gesturing. Professional sign language interpreters (N=24) rated discomfort for hand gestures associated with 47 characters and words and 33 hand postures. Clear associations of discomfort with hand postures were identified. In a nominal logistic regression model, high discomfort was associated with gestures requiring a flexed wrist, discordant adjacent fingers, or extended fingers. These and other findings should be considered in the design of hand gestures to optimize the relationship between human cognitive and physical processes and computer gesture recognition systems for human-computer input. PMID:26028955

Object and Facial Recognition in Augmented and Virtual Reality: Investigation into Software, Hardware and Potential Uses

NASA Technical Reports Server (NTRS)

Schulte, Erin

2017-01-01

As augmented and virtual reality grows in popularity, and more researchers focus on its development, other fields of technology have grown in the hopes of integrating with the up-and-coming hardware currently on the market. Namely, there has been a focus on how to make an intuitive, hands-free human-computer interaction (HCI) utilizing AR and VR that allows users to control their technology with little to no physical interaction with hardware. Computer vision, which is utilized in devices such as the Microsoft Kinect, webcams and other similar hardware has shown potential in assisting with the development of a HCI system that requires next to no human interaction with computing hardware and software. Object and facial recognition are two subsets of computer vision, both of which can be applied to HCI systems in the fields of medicine, security, industrial development and other similar areas.

Multimodal registration via spatial-context mutual information.

PubMed

Yi, Zhao; Soatto, Stefano

2011-01-01

We propose a method to efficiently compute mutual information between high-dimensional distributions of image patches. This in turn is used to perform accurate registration of images captured under different modalities, while exploiting their local structure otherwise missed in traditional mutual information definition. We achieve this by organizing the space of image patches into orbits under the action of Euclidean transformations of the image plane, and estimating the modes of a distribution in such an orbit space using affinity propagation. This way, large collections of patches that are equivalent up to translations and rotations are mapped to the same representative, or "dictionary element". We then show analytically that computing mutual information for a joint distribution in this space reduces to computing mutual information between the (scalar) label maps, and between the transformations mapping each patch into its closest dictionary element. We show that our approach improves registration performance compared with the state of the art in multimodal registration, using both synthetic and real images with quantitative ground truth.

Simulation of electrical and thermal fields in a multimode microwave oven using software written in C++

NASA Astrophysics Data System (ADS)

Abrudean, C.

2017-05-01

Due to multiple reflexions on walls, the electromagnetic field in a multimode microwave oven is difficult to estimate analytically. This paper presents a C++ program that calculates the electromagnetic field in a resonating cavity with an absorbing payload, uses the result to calculate heating in the payload taking its properties into account and then repeats. This results in a simulation of microwave heating, including phenomena like thermal runaway. The program is multithreaded to make use of today’s common multiprocessor/multicore computers.

Multimodal browsing using VoiceXML

NASA Astrophysics Data System (ADS)

Caccia, Giuseppe; Lancini, Rosa C.; Peschiera, Giuseppe

2003-06-01

With the increasing development of devices such as personal computers, WAP and personal digital assistants connected to the World Wide Web, end users feel the need to browse the Internet through multiple modalities. We intend to investigate on how to create a user interface and a service distribution platform granting the user access to the Internet through standard I/O modalities and voice simultaneously. Different architectures are evaluated suggesting the more suitable for each client terminal (PC o WAP). In particular the design of the multimodal usermachine interface considers the synchronization issue between graphical and voice contents.

Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection.

PubMed

Sarikaya, Duygu; Corso, Jason J; Guru, Khurshid A

2017-07-01

Video understanding of robot-assisted surgery (RAS) videos is an active research area. Modeling the gestures and skill level of surgeons presents an interesting problem. The insights drawn may be applied in effective skill acquisition, objective skill assessment, real-time feedback, and human-robot collaborative surgeries. We propose a solution to the tool detection and localization open problem in RAS video understanding, using a strictly computer vision approach and the recent advances of deep learning. We propose an architecture using multimodal convolutional neural networks for fast detection and localization of tools in RAS videos. To the best of our knowledge, this approach will be the first to incorporate deep neural networks for tool detection and localization in RAS videos. Our architecture applies a region proposal network (RPN) and a multimodal two stream convolutional network for object detection to jointly predict objectness and localization on a fusion of image and temporal motion cues. Our results with an average precision of 91% and a mean computation time of 0.1 s per test frame detection indicate that our study is superior to conventionally used methods for medical imaging while also emphasizing the benefits of using RPN for precision and efficiency. We also introduce a new data set, ATLAS Dione, for RAS video understanding. Our data set provides video data of ten surgeons from Roswell Park Cancer Institute, Buffalo, NY, USA, performing six different surgical tasks on the daVinci Surgical System (dVSS) with annotations of robotic tools per frame.

Investigations in Computer-Aided Instruction and Computer-Aided Controls. Final Report.

ERIC Educational Resources Information Center

Rosenberg, R.C.; And Others

These research projects, designed to delve into certain relationships between humans and computers, are focused on computer-assisted instruction and on man-computer interaction. One study demonstrates that within the limits of formal engineering theory, a computer simulated laboratory (Dynamic Systems Laboratory) can be built in which freshmen…

Proceedings from an International Conference on Computers and Philosophy, i-C&P 2006 held 3-5 May 2006 in Laval, France

DTIC Science & Technology

2008-10-20

embedded intelligence and cultural adaptations to the onslaught of robots in society. This volume constitutes a key contribution to the body of... Robotics , CNRS/Toulouse University, France Nathalie COLINEAU, Language & Multi-modality, CSIRO, Australia Roberto CORDESCHI, Computation & Communication...Intelligence, SONY CSL ­ Paris Nik KASABOV, Computer and Information Sciences, Auckland University, New Zealand Oussama KHATIB, Robotics & Artificial

Development of a Hybrid Optical Biopsy Probe to Improve Prostate Cancer Diagnosis

DTIC Science & Technology

2011-06-01

integrated needle probe can be developed for guiding needle biopsy for prostate cancer diagnosis. Multi-modal optical measurements to be utilized for... needle probe can be developed for guiding needle biopsy for prostate cancer diagnosis. Multi-modal optical measurements to be utilized for the study...tissue, into a transrectal- ultrasound , needle - biopsy probe. In the development phase, documentation to obtain IRB approval for ex vivo human prostate

Single-cell multimodal profiling reveals cellular epigenetic heterogeneity.

PubMed

Cheow, Lih Feng; Courtois, Elise T; Tan, Yuliana; Viswanathan, Ramya; Xing, Qiaorui; Tan, Rui Zhen; Tan, Daniel S W; Robson, Paul; Loh, Yuin-Han; Quake, Stephen R; Burkholder, William F

2016-10-01

Sample heterogeneity often masks DNA methylation signatures in subpopulations of cells. Here, we present a method to genotype single cells while simultaneously interrogating gene expression and DNA methylation at multiple loci. We used this targeted multimodal approach, implemented on an automated, high-throughput microfluidic platform, to assess primary lung adenocarcinomas and human fibroblasts undergoing reprogramming by profiling epigenetic variation among cell types identified through genotyping and transcriptional analysis.

Development of computational small animal models and their applications in preclinical imaging and therapy research.

PubMed

Xie, Tianwu; Zaidi, Habib

2016-01-01

The development of multimodality preclinical imaging techniques and the rapid growth of realistic computer simulation tools have promoted the construction and application of computational laboratory animal models in preclinical research. Since the early 1990s, over 120 realistic computational animal models have been reported in the literature and used as surrogates to characterize the anatomy of actual animals for the simulation of preclinical studies involving the use of bioluminescence tomography, fluorescence molecular tomography, positron emission tomography, single-photon emission computed tomography, microcomputed tomography, magnetic resonance imaging, and optical imaging. Other applications include electromagnetic field simulation, ionizing and nonionizing radiation dosimetry, and the development and evaluation of new methodologies for multimodality image coregistration, segmentation, and reconstruction of small animal images. This paper provides a comprehensive review of the history and fundamental technologies used for the development of computational small animal models with a particular focus on their application in preclinical imaging as well as nonionizing and ionizing radiation dosimetry calculations. An overview of the overall process involved in the design of these models, including the fundamental elements used for the construction of different types of computational models, the identification of original anatomical data, the simulation tools used for solving various computational problems, and the applications of computational animal models in preclinical research. The authors also analyze the characteristics of categories of computational models (stylized, voxel-based, and boundary representation) and discuss the technical challenges faced at the present time as well as research needs in the future.

Multimodal Deep Autoencoder for Human Pose Recovery.

PubMed

Hong, Chaoqun; Yu, Jun; Wan, Jian; Tao, Dacheng; Wang, Meng

2015-12-01

Video-based human pose recovery is usually conducted by retrieving relevant poses using image features. In the retrieving process, the mapping between 2D images and 3D poses is assumed to be linear in most of the traditional methods. However, their relationships are inherently non-linear, which limits recovery performance of these methods. In this paper, we propose a novel pose recovery method using non-linear mapping with multi-layered deep neural network. It is based on feature extraction with multimodal fusion and back-propagation deep learning. In multimodal fusion, we construct hypergraph Laplacian with low-rank representation. In this way, we obtain a unified feature description by standard eigen-decomposition of the hypergraph Laplacian matrix. In back-propagation deep learning, we learn a non-linear mapping from 2D images to 3D poses with parameter fine-tuning. The experimental results on three data sets show that the recovery error has been reduced by 20%-25%, which demonstrates the effectiveness of the proposed method.

'Tagger' - a Mac OS X Interactive Graphical Application for Data Inference and Analysis of N-Dimensional Datasets in the Natural Physical Sciences.

NASA Astrophysics Data System (ADS)

Morse, P. E.; Reading, A. M.; Lueg, C.

2014-12-01

Pattern-recognition in scientific data is not only a computational problem but a human-observer problem as well. Human observation of - and interaction with - data visualization software can augment, select, interrupt and modify computational routines and facilitate processes of pattern and significant feature recognition for subsequent human analysis, machine learning, expert and artificial intelligence systems.'Tagger' is a Mac OS X interactive data visualisation tool that facilitates Human-Computer interaction for the recognition of patterns and significant structures. It is a graphical application developed using the Quartz Composer framework. 'Tagger' follows a Model-View-Controller (MVC) software architecture: the application problem domain (the model) is to facilitate novel ways of abstractly representing data to a human interlocutor, presenting these via different viewer modalities (e.g. chart representations, particle systems, parametric geometry) to the user (View) and enabling interaction with the data (Controller) via a variety of Human Interface Devices (HID). The software enables the user to create an arbitrary array of tags that may be appended to the visualised data, which are then saved into output files as forms of semantic metadata. Three fundamental problems that are not strongly supported by conventional scientific visualisation software are addressed:1] How to visually animate data over time, 2] How to rapidly deploy unconventional parametrically driven data visualisations, 3] How to construct and explore novel interaction models that capture the activity of the end-user as semantic metadata that can be used to computationally enhance subsequent interrogation. Saved tagged data files may be loaded into Tagger, so that tags may be tagged, if desired. Recursion opens up the possibility of refining or overlapping different types of tags, tagging a variety of different POIs or types of events, and of capturing different types of specialist observations of important or noticeable events. Other visualisations and modes of interaction will also be demonstrated, with the aim of discovering knowledge in large datasets in the natural, physical sciences. Fig.1 Wave height data from an oceanographic Wave Rider Buoy. Colors/radii are driven by wave height data.

Rhesus macaques recognize unique multi-modal face-voice relations of familiar individuals and not of unfamiliar ones

PubMed Central

Habbershon, Holly M.; Ahmed, Sarah Z.; Cohen, Yale E.

2013-01-01

Communication signals in non-human primates are inherently multi-modal. However, for laboratory-housed monkeys, there is relatively little evidence in support of the use of multi-modal communication signals in individual recognition. Here, we used a preferential-looking paradigm to test whether laboratory-housed rhesus could “spontaneously” (i.e., in the absence of operant training) use multi-modal communication stimuli to discriminate between known conspecifics. The multi-modal stimulus was a silent movie of two monkeys vocalizing and an audio file of the vocalization from one of the monkeys in the movie. We found that the gaze patterns of those monkeys that knew the individuals in the movie were reliably biased toward the individual that did not produce the vocalization. In contrast, there was not a systematic gaze pattern for those monkeys that did not know the individuals in the movie. These data are consistent with the hypothesis that laboratory-housed rhesus can recognize and distinguish between conspecifics based on auditory and visual communication signals. PMID:23774779