Comparative Reliability of Structured Versus Unstructured Interviews in the Admission Process of a Residency Program

PubMed Central

Blouin, Danielle; Day, Andrew G.; Pavlov, Andrey

2011-01-01

Background Although never directly compared, structured interviews are reported as being more reliable than unstructured interviews. This study compared the reliability of both types of interview when applied to a common pool of applicants for positions in an emergency medicine residency program. Methods In 2008, one structured interview was added to the two unstructured interviews traditionally used in our resident selection process. A formal job analysis using the critical incident technique guided the development of the structured interview tool. This tool consisted of 7 scenarios assessing 4 of the domains deemed essential for success as a resident in this program. The traditional interview tool assessed 5 general criteria. In addition to these criteria, the unstructured panel members were asked to rate each candidate on the same 4 essential domains rated by the structured panel members. All 3 panels interviewed all candidates. Main outcomes were the overall, interitem, and interrater reliabilities, the correlations between interview panels, and the dimensionality of each interview tool. Results Thirty candidates were interviewed. The overall reliability reached 0.43 for the structured interview, and 0.81 and 0.71 for the unstructured interviews. Analyses of the variance components showed a high interrater, low interitem reliability for the structured interview, and a high interrater, high interitem reliability for the unstructured interviews. The summary measures from the 2 unstructured interviews were significantly correlated, but neither was correlated with the structured interview. Only the structured interview was multidimensional. Conclusions A structured interview did not yield a higher overall reliability than both unstructured interviews. The lower reliability is explained by a lower interitem reliability, which in turn is due to the multidimensionality of the interview tool. Both unstructured panels consistently rated a single dimension, even when prompted to assess the 4 specific domains established as essential to succeed in this residency program. PMID:23205201

Comparative reliability of structured versus unstructured interviews in the admission process of a residency program.

PubMed

Blouin, Danielle; Day, Andrew G; Pavlov, Andrey

2011-12-01

Although never directly compared, structured interviews are reported as being more reliable than unstructured interviews. This study compared the reliability of both types of interview when applied to a common pool of applicants for positions in an emergency medicine residency program. In 2008, one structured interview was added to the two unstructured interviews traditionally used in our resident selection process. A formal job analysis using the critical incident technique guided the development of the structured interview tool. This tool consisted of 7 scenarios assessing 4 of the domains deemed essential for success as a resident in this program. The traditional interview tool assessed 5 general criteria. In addition to these criteria, the unstructured panel members were asked to rate each candidate on the same 4 essential domains rated by the structured panel members. All 3 panels interviewed all candidates. Main outcomes were the overall, interitem, and interrater reliabilities, the correlations between interview panels, and the dimensionality of each interview tool. Thirty candidates were interviewed. The overall reliability reached 0.43 for the structured interview, and 0.81 and 0.71 for the unstructured interviews. Analyses of the variance components showed a high interrater, low interitem reliability for the structured interview, and a high interrater, high interitem reliability for the unstructured interviews. The summary measures from the 2 unstructured interviews were significantly correlated, but neither was correlated with the structured interview. Only the structured interview was multidimensional. A structured interview did not yield a higher overall reliability than both unstructured interviews. The lower reliability is explained by a lower interitem reliability, which in turn is due to the multidimensionality of the interview tool. Both unstructured panels consistently rated a single dimension, even when prompted to assess the 4 specific domains established as essential to succeed in this residency program.

Service quality assessment of workers compensation health care delivery programs in New York using SERVQUAL.

PubMed

Arunasalam, Mark; Paulson, Albert; Wallace, William

2003-01-01

Preferred provider organizations (PPOs) provide healthcare services to an expanding proportion of the U.S. population. This paper presents a programmatic assessment of service quality in the workers' compensation environment using two different models: the PPO program model and the fee-for-service (FFS) payor model. The methodology used here will augment currently available research in workers' compensation, which has been lacking in measuring service quality determinants and assessing programmatic success/failure of managed care type programs. Results indicated that the SERVQUAL tool provided a reliable and valid clinical quality assessment tool that ascertained that PPO marketers should focus on promoting physician outreach (to show empathy) and accessibility (to show reliability) for injured workers.

Thin-film reliability and engineering overview

NASA Technical Reports Server (NTRS)

Ross, R. G., Jr.

1984-01-01

The reliability and engineering technology base required for thin film solar energy conversions modules is discussed. The emphasis is on the integration of amorphous silicon cells into power modules. The effort is being coordinated with SERI's thin film cell research activities as part of DOE's Amorphous Silicon Program. Program concentration is on temperature humidity reliability research, glass breaking strength research, point defect system analysis, hot spot heating assessment, and electrical measurements technology.

Thin-film reliability and engineering overview

NASA Astrophysics Data System (ADS)

Ross, R. G., Jr.

1984-10-01

The reliability and engineering technology base required for thin film solar energy conversions modules is discussed. The emphasis is on the integration of amorphous silicon cells into power modules. The effort is being coordinated with SERI's thin film cell research activities as part of DOE's Amorphous Silicon Program. Program concentration is on temperature humidity reliability research, glass breaking strength research, point defect system analysis, hot spot heating assessment, and electrical measurements technology.

Validation of a method for assessing resident physicians' quality improvement proposals.

PubMed

Leenstra, James L; Beckman, Thomas J; Reed, Darcy A; Mundell, William C; Thomas, Kris G; Krajicek, Bryan J; Cha, Stephen S; Kolars, Joseph C; McDonald, Furman S

2007-09-01

Residency programs involve trainees in quality improvement (QI) projects to evaluate competency in systems-based practice and practice-based learning and improvement. Valid approaches to assess QI proposals are lacking. We developed an instrument for assessing resident QI proposals--the Quality Improvement Proposal Assessment Tool (QIPAT-7)-and determined its validity and reliability. QIPAT-7 content was initially obtained from a national panel of QI experts. Through an iterative process, the instrument was refined, pilot-tested, and revised. Seven raters used the instrument to assess 45 resident QI proposals. Principal factor analysis was used to explore the dimensionality of instrument scores. Cronbach's alpha and intraclass correlations were calculated to determine internal consistency and interrater reliability, respectively. QIPAT-7 items comprised a single factor (eigenvalue = 3.4) suggesting a single assessment dimension. Interrater reliability for each item (range 0.79 to 0.93) and internal consistency reliability among the items (Cronbach's alpha = 0.87) were high. This method for assessing resident physician QI proposals is supported by content and internal structure validity evidence. QIPAT-7 is a useful tool for assessing resident QI proposals. Future research should determine the reliability of QIPAT-7 scores in other residency and fellowship training programs. Correlations should also be made between assessment scores and criteria for QI proposal success such as implementation of QI proposals, resident scholarly productivity, and improved patient outcomes.

Ceramic component reliability with the restructured NASA/CARES computer program

NASA Technical Reports Server (NTRS)

Powers, Lynn M.; Starlinger, Alois; Gyekenyesi, John P.

1992-01-01

The Ceramics Analysis and Reliability Evaluation of Structures (CARES) integrated design program on statistical fast fracture reliability and monolithic ceramic components is enhanced to include the use of a neutral data base, two-dimensional modeling, and variable problem size. The data base allows for the efficient transfer of element stresses, temperatures, and volumes/areas from the finite element output to the reliability analysis program. Elements are divided to insure a direct correspondence between the subelements and the Gaussian integration points. Two-dimensional modeling is accomplished by assessing the volume flaw reliability with shell elements. To demonstrate the improvements in the algorithm, example problems are selected from a round-robin conducted by WELFEP (WEakest Link failure probability prediction by Finite Element Postprocessors).

The multiple mini-interview for selecting medical residents: first experience in the Middle East region.

PubMed

Ahmed, Ashraf; Qayed, Khalil Ibrahim; Abdulrahman, Mahera; Tavares, Walter; Rosenfeld, Jack

2014-08-01

Numerous studies have shown that multiple mini-interviews (MMI) provides a standard, fair, and more reliable method for assessing applicants. This article presents the first MMI experience for selection of medical residents in the Middle East culture and an Arab country. In 2012, we started using the MMI in interviewing applicants to the residency program of Dubai Health Authority. This interview process consisted of eight, eight-minute structured interview scenarios. Applicants rotated through the stations, each with its own interviewer and scenario. They read the scenario and were requested to discuss the issues with the interviewers. Sociodemographic and station assessment data provided for each applicant were analyzed to determine whether the MMI was a reliable assessment of the non-clinical attributes in the present setting of an Arab country. One hundred and eighty-seven candidates from 27 different countries were interviewed for Dubai Residency Training Program using MMI. They were graduates of 5 medical universities within United Arab Emirates (UAE) and 60 different universities outside UAE. With this applicant's pool, a MMI with eight stations, produced absolute and relative reliability of 0.8 and 0.81, respectively. The person × station interaction contributed 63% of the variance components, the person contributed 34% of the variance components, and the station contributed 2% of the variance components. The MMI has been used in numerous universities in English speaking countries. The MMI evaluates non-clinical attributes and this study provides further evidence for its reliability but in a different country and culture. The MMI offers a fair and more reliable assessment of applicants to medical residency programs. The present data show that this assessment technique applied in a non-western country and Arab culture still produced reliable results.

Investigating the Quality of the School Technology Needs Assessment (STNA) 3.0: A Validity and Reliability Study

ERIC Educational Resources Information Center

Corn, Jenifer O.

2010-01-01

Schools and districts should use a well-designed needs assessment to inform important decisions about a range of technology program areas. Presently, there is a lack of valid and reliable instruments available and accessible to schools to effectively assess their educational needs to better design and evaluate their projects and initiatives. The…

The probability estimation of the electronic lesson implementation taking into account software reliability

NASA Astrophysics Data System (ADS)

Gurov, V. V.

2017-01-01

Software tools for educational purposes, such as e-lessons, computer-based testing system, from the point of view of reliability, have a number of features. The main ones among them are the need to ensure a sufficiently high probability of their faultless operation for a specified time, as well as the impossibility of their rapid recovery by the way of replacing it with a similar running program during the classes. The article considers the peculiarities of reliability evaluation of programs in contrast to assessments of hardware reliability. The basic requirements to reliability of software used for carrying out practical and laboratory classes in the form of computer-based training programs are given. The essential requirements applicable to the reliability of software used for conducting the practical and laboratory studies in the form of computer-based teaching programs are also described. The mathematical tool based on Markov chains, which allows to determine the degree of debugging of the training program for use in the educational process by means of applying the graph of the software modules interaction, is presented.

Reliability and risk assessment of structures

NASA Technical Reports Server (NTRS)

Chamis, C. C.

1991-01-01

Development of reliability and risk assessment of structural components and structures is a major activity at Lewis Research Center. It consists of five program elements: (1) probabilistic loads; (2) probabilistic finite element analysis; (3) probabilistic material behavior; (4) assessment of reliability and risk; and (5) probabilistic structural performance evaluation. Recent progress includes: (1) the evaluation of the various uncertainties in terms of cumulative distribution functions for various structural response variables based on known or assumed uncertainties in primitive structural variables; (2) evaluation of the failure probability; (3) reliability and risk-cost assessment; and (4) an outline of an emerging approach for eventual certification of man-rated structures by computational methods. Collectively, the results demonstrate that the structural durability/reliability of man-rated structural components and structures can be effectively evaluated by using formal probabilistic methods.

John F. Kennedy Space Center, Safety, Reliability, Maintainability and Quality Assurance, Survey and Audit Program

NASA Technical Reports Server (NTRS)

1994-01-01

This document is the product of the KSC Survey and Audit Working Group composed of civil service and contractor Safety, Reliability, and Quality Assurance (SR&QA) personnel. The program described herein provides standardized terminology, uniformity of survey and audit operations, and emphasizes process assessments rather than a program based solely on compliance. The program establishes minimum training requirements, adopts an auditor certification methodology, and includes survey and audit metrics for the audited organizations as well as the auditing organization.

A rater training protocol to assess team performance.

PubMed

Eppich, Walter; Nannicelli, Anna P; Seivert, Nicholas P; Sohn, Min-Woong; Rozenfeld, Ranna; Woods, Donna M; Holl, Jane L

2015-01-01

Simulation-based methodologies are increasingly used to assess teamwork and communication skills and provide team training. Formative feedback regarding team performance is an essential component. While effective use of simulation for assessment or training requires accurate rating of team performance, examples of rater-training programs in health care are scarce. We describe our rater training program and report interrater reliability during phases of training and independent rating. We selected an assessment tool shown to yield valid and reliable results and developed a rater training protocol with an accompanying rater training handbook. The rater training program was modeled after previously described high-stakes assessments in the setting of 3 facilitated training sessions. Adjacent agreement was used to measure interrater reliability between raters. Nine raters with a background in health care and/or patient safety evaluated team performance of 42 in-situ simulations using post-hoc video review. Adjacent agreement increased from the second training session (83.6%) to the third training session (85.6%) when evaluating the same video segments. Adjacent agreement for the rating of overall team performance was 78.3%, which was added for the third training session. Adjacent agreement was 97% 4 weeks posttraining and 90.6% at the end of independent rating of all simulation videos. Rater training is an important element in team performance assessment, and providing examples of rater training programs is essential. Articulating key rating anchors promotes adequate interrater reliability. In addition, using adjacent agreement as a measure allows differentiation between high- and low-performing teams on video review. © 2015 The Alliance for Continuing Education in the Health Professions, the Society for Academic Continuing Medical Education, and the Council on Continuing Medical Education, Association for Hospital Medical Education.

Qualitative and Semiquantitative Assessment of Exposure to Engineered Nanomaterials within the French EpiNano Program: Inter- and Intramethod Reliability Study.

PubMed

Guseva Canu, Irina; Jezewski-Serra, Delphine; Delabre, Laurène; Ducamp, Stéphane; Iwatsubo, Yuriko; Audignon-Durand, Sabine; Ducros, Cécile; Radauceanu, Anca; Durand, Catherine; Witschger, Olivier; Flahaut, Emmanuel

2017-01-01

The relatively recent development of industries working with nanomaterials has created challenges for exposure assessment. In this article, we propose a relatively simple approach to assessing nanomaterial exposures for the purposes of epidemiological studies of workers in these industries. This method consists of an onsite industrial hygiene visit of facilities carried out individually and a description of workstations where nano-objects and their agglomerates and aggregates (NOAA) are present using a standardized tool, the Onsite technical logbook. To assess its reliability, we implemented this approach for assessing exposure to NOAA in workplaces at seven workstations which synthesize and functionalize carbon nanotubes. The prediction of exposure to NOAA using this method exhibited substantial agreement with that of the reference method, the latter being based on an onsite group visit, an expert's report and exposure measurements (Cohen kappa = 0.70, sensitivity = 0.88, specificity = 0.92). Intramethod comparison of results for exposure prediction showed moderate agreement between the three evaluators (two program team evaluators and one external evaluator) (weighted Fleiss kappa = 0.60, P = 0.003). Interevaluator reliability of the semiquantitative exposure characterization results was excellent between the two evaluators from the program team (Spearman rho = 0.93, P = 0.03) and fair when these two evaluators' results were compared with the external evaluator's results. The project was undertaken within the framework of the French epidemiological surveillance program EpiNano. This study allowed a first reliability assessment of the EpiNano method. However, to further validate this method a comparison with robust quantitative exposure measurement data is necessary. © The Author 2017. Published by Oxford University Press on behalf of the British Occupational Hygiene Society.

Predictive Validity of Measures of the Pathfinder Scaling Algorithm on Programming Performance: Alternative Assessment Strategy for Programming Education

ERIC Educational Resources Information Center

Lau, Wilfred W. F.; Yuen, Allan H. K.

2009-01-01

Recent years have seen a shift in focus from assessment of learning to assessment for learning and the emergence of alternative assessment methods. However, the reliability and validity of these methods as assessment tools are still questionable. In this article, we investigated the predictive validity of measures of the Pathfinder Scaling…

Initial Assessment for K-12 English Language Support in Six Countries: Revisiting the Validity-Reliability Paradox

ERIC Educational Resources Information Center

Sinclair, Jeanne; Lau, Clarissa

2018-01-01

It is common practice for K-12 schools to assess multilingual students' language proficiency to determine language support program placement. Because such programs can provide essential scaffolding, the policies guiding these assessments merit careful consideration. It is well accepted that quality assessments must be valid (representative of the…

Using Facility Condition Assessments to Identify Actions Related to Infrastructure

NASA Technical Reports Server (NTRS)

Rubert, Kennedy F.

2010-01-01

To support cost effective, quality research it is essential that laboratory and testing facilities are maintained in a continuous and reliable state of availability at all times. NASA Langley Research Center (LaRC) and its maintenance contractor, Jacobs Technology, Inc. Research Operations, Maintenance, and Engineering (ROME) group, are in the process of implementing a combined Facility Condition Assessment (FCA) and Reliability Centered Maintenance (RCM) program to improve asset management and overall reliability of testing equipment in facilities such as wind tunnels. Specific areas are being identified for improvement, the deferred maintenance cost is being estimated, and priority is being assigned against facilities where conditions have been allowed to deteriorate. This assessment serves to assist in determining where to commit available funds on the Center. RCM methodologies are being reviewed and enhanced to assure that appropriate preventive, predictive, and facilities/equipment acceptance techniques are incorporated to prolong lifecycle availability and assure reliability at minimum cost. The results from the program have been favorable, better enabling LaRC to manage assets prudently.

Standardization of the Functional Assessment and Intervention Program (FAIP) with Children Who Have Externalizing Behaviors

ERIC Educational Resources Information Center

Hartwig, Laurie; Heathfield, Lora Tuesday; Jenson, William R.

2004-01-01

The purpose of this study was to develop standardization data for the Functional Assessment Intervention Program (FAIP; University of Utah, Utah State University, & Utah State Office of Education, 1999), a computerized, functional behavioral assessment expert system. Reliability, validity, and utility analyses were conducted with students serving…

Assessment of Primary Representational Systems with Neurolinguistic Programming: Examination of Preliminary Literature.

ERIC Educational Resources Information Center

Dorn, Fred J.; And Others

1983-01-01

Reviews the inconsistent findings of studies on neurolinguistic programing and recommends some areas that should be examined to verify various claims. Discusses methods of assessing client's primary representational systems, including predicate usage and eye movements, and suggests that more reliable methods of assessing PRS must be found. (JAC)

NASA Aerospace Flight Battery Systems Program Update

NASA Technical Reports Server (NTRS)

Manzo, Michelle; ODonnell, Patricia

1997-01-01

The objectives of NASA's Aerospace Flight Battery Systems Program is to: develop, maintain and provide tools for the validation and assessment of aerospace battery technologies; accelerate the readiness of technology advances and provide infusion paths for emerging technologies; provide NASA projects with the required database and validation guidelines for technology selection of hardware and processes relating to aerospace batteries; disseminate validation and assessment tools, quality assurance, reliability, and availability information to the NASA and aerospace battery communities; and ensure that safe, reliable batteries are available for NASA's future missions.

Evaluating training of screening, brief intervention, and referral to treatment (SBIRT) for substance use: Reliability of the MD3 SBIRT Coding Scale.

PubMed

DiClemente, Carlo C; Crouch, Taylor Berens; Norwood, Amber E Q; Delahanty, Janine; Welsh, Christopher

2015-03-01

Screening, brief intervention, and referral to treatment (SBIRT) has become an empirically supported and widely implemented approach in primary and specialty care for addressing substance misuse. Accordingly, training of providers in SBIRT has increased exponentially in recent years. However, the quality and fidelity of training programs and subsequent interventions are largely unknown because of the lack of SBIRT-specific evaluation tools. The purpose of this study was to create a coding scale to assess quality and fidelity of SBIRT interactions addressing alcohol, tobacco, illicit drugs, and prescription medication misuse. The scale was developed to evaluate performance in an SBIRT residency training program. Scale development was based on training protocol and competencies with consultation from Motivational Interviewing coding experts. Trained medical residents practiced SBIRT with standardized patients during 10- to 15-min videotaped interactions. This study included 25 tapes from the Family Medicine program coded by 3 unique coder pairs with varying levels of coding experience. Interrater reliability was assessed for overall scale components and individual items via intraclass correlation coefficients. Coder pair-specific reliability was also assessed. Interrater reliability was excellent overall for the scale components (>.85) and nearly all items. Reliability was higher for more experienced coders, though still adequate for the trained coder pair. Descriptive data demonstrated a broad range of adherence and skills. Subscale correlations supported concurrent and discriminant validity. Data provide evidence that the MD3 SBIRT Coding Scale is a psychometrically reliable coding system for evaluating SBIRT interactions and can be used to evaluate implementation skills for fidelity, training, assessment, and research. Recommendations for refinement and further testing of the measure are discussed. (PsycINFO Database Record (c) 2015 APA, all rights reserved).

The implementation and use of Ada on distributed systems with high reliability requirements

NASA Technical Reports Server (NTRS)

Knight, J. C.

1988-01-01

The use and implementation of Ada were investigated in distributed environments in which reliability is the primary concern. In particular, the focus was on the possibility that a distributed system may be programmed entirely in Ada so that the individual tasks of the system are unconcerned with which processors are being executed, and that failures may occur in the software and underlying hardware. A secondary interest is in the performance of Ada systems and how that performance can be gauged reliably. Primary activities included: analysis of the original approach to recovery in distributed Ada programs using the Advanced Transport Operating System (ATOPS) example; review and assessment of the original approach which was found to be capable of improvement; development of a refined approach to recovery that was applied to the ATOPS example; and design and development of a performance assessment scheme for Ada programs based on a flexible user-driven benchmarking system.

Notes on numerical reliability of several statistical analysis programs

USGS Publications Warehouse

Landwehr, J.M.; Tasker, Gary D.

1999-01-01

This report presents a benchmark analysis of several statistical analysis programs currently in use in the USGS. The benchmark consists of a comparison between the values provided by a statistical analysis program for variables in the reference data set ANASTY and their known or calculated theoretical values. The ANASTY data set is an amendment of the Wilkinson NASTY data set that has been used in the statistical literature to assess the reliability (computational correctness) of calculated analytical results.

Reliability of the Colorado Family Support Assessment: A Self-Sufficiency Matrix for Families

ERIC Educational Resources Information Center

Richmond, Melissa K.; Pampel, Fred C.; Zarcula, Flavia; Howey, Virginia; McChesney, Brenda

2017-01-01

Purpose: Family support programs commonly use self-sufficiency matrices (SSMs) to measure family outcomes, however, validation research on SSMs is sparse. This study examined the reliability of the Colorado Family Support Assessment 2.0 (CFSA 2.0) to measure family self-reliance across 14 domains (e.g., employment). Methods: Ten written case…

Analysis of the Reliability and Validity of a Mentor's Assessment for Principal Internships

ERIC Educational Resources Information Center

Koonce, Glenn L.; Kelly, Michael D.

2014-01-01

In this study, researchers analyzed the reliability and validity of the mentor's assessment for principal internships at a university in the Southeast region of the United States. The results of the study yielded how trustworthy and dependable the instrument is and the effectiveness of the instrument in the current principal preparation program.…

Measuring Program Quality, Part 2: Addressing Potential Cultural Bias in a Rater Reliability Exam

ERIC Educational Resources Information Center

Richer, Amanda; Charmaraman, Linda; Ceder, Ineke

2018-01-01

Like instruments used in afterschool programs to assess children's social and emotional growth or to evaluate staff members' performance, instruments used to evaluate program quality should be free from bias. Practitioners and researchers alike want to know that assessment instruments, whatever their type or intent, treat all people fairly and do…

Development and Validity Testing of the Worksite Health Index: An Assessment Tool to Help and Improve Korean Employees' Health-Related Outcome.

PubMed

Yun, Young Ho; Sim, Jin Ah; Lim, Ye Jin; Lim, Cheol Il; Kang, Sung-Choon; Kang, Joon-Ho; Park, Jun Dong; Noh, Dong Young

2016-06-01

The objective of this study was to develop the Worksite Health Index (WHI) and validate its psychometric properties. The development of the WHI questionnaire included item generation, item construction, and field testing. To assess the instrument's reliability and validity, we recruited 30 different Korean worksites. We developed the WHI questionnaire of 136 items categorized into five domains, namely Governance and Infrastructure, Need Assessment and Planning, Health Prevention and Promotion Program, Occupational Safety, and Monitoring and Feedback. All WHI domains demonstrated a high reliability with good internal consistency. The total WHI scores differentiated worksite groups effectively according to firm size. Each domain was associated significantly with employees' health status, absence, and financial outcome. The WHI can assess comprehensive worksite health programs. This tool is publicly available for addressing the growing need for worksite health programs.

Bayesian Inference for NASA Probabilistic Risk and Reliability Analysis

NASA Technical Reports Server (NTRS)

Dezfuli, Homayoon; Kelly, Dana; Smith, Curtis; Vedros, Kurt; Galyean, William

2009-01-01

This document, Bayesian Inference for NASA Probabilistic Risk and Reliability Analysis, is intended to provide guidelines for the collection and evaluation of risk and reliability-related data. It is aimed at scientists and engineers familiar with risk and reliability methods and provides a hands-on approach to the investigation and application of a variety of risk and reliability data assessment methods, tools, and techniques. This document provides both: A broad perspective on data analysis collection and evaluation issues. A narrow focus on the methods to implement a comprehensive information repository. The topics addressed herein cover the fundamentals of how data and information are to be used in risk and reliability analysis models and their potential role in decision making. Understanding these topics is essential to attaining a risk informed decision making environment that is being sought by NASA requirements and procedures such as 8000.4 (Agency Risk Management Procedural Requirements), NPR 8705.05 (Probabilistic Risk Assessment Procedures for NASA Programs and Projects), and the System Safety requirements of NPR 8715.3 (NASA General Safety Program Requirements).

Design for Reliability and Safety Approach for the New NASA Launch Vehicle

NASA Technical Reports Server (NTRS)

Safie, Fayssal M.; Weldon, Danny M.

2007-01-01

The United States National Aeronautics and Space Administration (NASA) is in the midst of a space exploration program intended for sending crew and cargo to the international Space Station (ISS), to the moon, and beyond. This program is called Constellation. As part of the Constellation program, NASA is developing new launch vehicles aimed at significantly increase safety and reliability, reduce the cost of accessing space, and provide a growth path for manned space exploration. Achieving these goals requires a rigorous process that addresses reliability, safety, and cost upfront and throughout all the phases of the life cycle of the program. This paper discusses the "Design for Reliability and Safety" approach for the NASA new launch vehicles, the ARES I and ARES V. Specifically, the paper addresses the use of an integrated probabilistic functional analysis to support the design analysis cycle and a probabilistic risk assessment (PRA) to support the preliminary design and beyond.

Surgeons' attitude toward a competency-based training and assessment program: results of a multicenter survey.

PubMed

Hopmans, Cornelis J; den Hoed, Pieter T; Wallenburg, Iris; van der Laan, Lijkckle; van der Harst, Erwin; van der Elst, Maarten; Mannaerts, Guido H H; Dawson, Imro; van Lanschot, Jan J B; Ijzermans, Jan N M

2013-01-01

Currently, most surgical training programs are focused on the development and evaluation of professional competencies. Also in the Netherlands, competency-based training and assessment programs were introduced to restructure postgraduate medical training. The current surgical residency program is based on the Canadian Medical Education Directives for Specialists (CanMEDS) competencies and uses assessment tools to evaluate residents' competence progression. In this study, we examined the attitude of surgical residents and attending surgeons toward a competency-based training and assessment program used to restructure general surgical training in the Netherlands in 2009. In 2011, all residents (n = 51) and attending surgeons (n = 108) in 1 training region, consisting of 7 hospitals, were surveyed. Participants were asked to rate the importance of the CanMEDS competencies and the suitability of the adopted assessment tools. Items were rated on a 5-point Likert scale and considered relevant when at least 80% of the respondents rated an item with a score of 4 or 5 (indicating a positive attitude). Reliability was evaluated by calculating the Cronbach's α, and the Mann-Whitney test was applied to assess differences between groups. The response rate was 88% (n = 140). The CanMEDS framework demonstrated good reliability (Cronbach's α = 0.87). However, the importance of the competencies 'Manager' (78%) and 'Health Advocate' (70%) was undervalued. The assessment tools failed to achieve an acceptable reliability (Cronbach's α = 0.55), and individual tools were predominantly considered unsuitable for assessment. Exceptions were the tools 'in-training evaluation report' (91%) and 'objective structured assessment of technical skill' (82%). No significant differences were found between the residents and the attending surgeons. This study has demonstrated that, 2 years after the reform of the general surgical residency program, residents and attending surgeons in a large training region in the Netherlands do not acknowledge the importance of all CanMEDS competencies and consider the assessment tools generally unsuitable for competence evaluation. Copyright © 2013 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

Johnson Space Center's Risk and Reliability Analysis Group 2008 Annual Report

NASA Technical Reports Server (NTRS)

Valentine, Mark; Boyer, Roger; Cross, Bob; Hamlin, Teri; Roelant, Henk; Stewart, Mike; Bigler, Mark; Winter, Scott; Reistle, Bruce; Heydorn,Dick

2009-01-01

The Johnson Space Center (JSC) Safety & Mission Assurance (S&MA) Directorate s Risk and Reliability Analysis Group provides both mathematical and engineering analysis expertise in the areas of Probabilistic Risk Assessment (PRA), Reliability and Maintainability (R&M) analysis, and data collection and analysis. The fundamental goal of this group is to provide National Aeronautics and Space Administration (NASA) decisionmakers with the necessary information to make informed decisions when evaluating personnel, flight hardware, and public safety concerns associated with current operating systems as well as with any future systems. The Analysis Group includes a staff of statistical and reliability experts with valuable backgrounds in the statistical, reliability, and engineering fields. This group includes JSC S&MA Analysis Branch personnel as well as S&MA support services contractors, such as Science Applications International Corporation (SAIC) and SoHaR. The Analysis Group s experience base includes nuclear power (both commercial and navy), manufacturing, Department of Defense, chemical, and shipping industries, as well as significant aerospace experience specifically in the Shuttle, International Space Station (ISS), and Constellation Programs. The Analysis Group partners with project and program offices, other NASA centers, NASA contractors, and universities to provide additional resources or information to the group when performing various analysis tasks. The JSC S&MA Analysis Group is recognized as a leader in risk and reliability analysis within the NASA community. Therefore, the Analysis Group is in high demand to help the Space Shuttle Program (SSP) continue to fly safely, assist in designing the next generation spacecraft for the Constellation Program (CxP), and promote advanced analytical techniques. The Analysis Section s tasks include teaching classes and instituting personnel qualification processes to enhance the professional abilities of our analysts as well as performing major probabilistic assessments used to support flight rationale and help establish program requirements. During 2008, the Analysis Group performed more than 70 assessments. Although all these assessments were important, some were instrumental in the decisionmaking processes for the Shuttle and Constellation Programs. Two of the more significant tasks were the Space Transportation System (STS)-122 Low Level Cutoff PRA for the SSP and the Orion Pad Abort One (PA-1) PRA for the CxP. These two activities, along with the numerous other tasks the Analysis Group performed in 2008, are summarized in this report. This report also highlights several ongoing and upcoming efforts to provide crucial statistical and probabilistic assessments, such as the Extravehicular Activity (EVA) PRA for the Hubble Space Telescope service mission and the first fully integrated PRAs for the CxP's Lunar Sortie and ISS missions.

Psychometric Properties of the School Attitude Assessment Survey-Revised with International Baccalaureate High School Students

ERIC Educational Resources Information Center

Dedrick, Robert F.; Shaunessy-Dedrick, Elizabeth; Suldo, Shannon M.; Ferron, John M.

2015-01-01

In two studies (ns = 312 and 1,149) with 9- to 12-grade students in pre-International Baccalaureate (IB) and IB Diploma programs, we evaluated the reliability, factor structure, measurement invariance, and criterion-related validity of the scores from the School Attitude Assessment Survey-Revised (SAAS-R). Reliabilities of the five SAAS-R subscale…

Missile Systems Maintenance, AFSC 411XOB/C.

DTIC Science & Technology

1988-04-01

technician’s rating. A statistical measurement of their agreement, known as the interrater reliability (as assessed through components of variance of...senior technician’s ratings. A statistical measurement of their agreement, known as the interrater reliability (as assessed through components of...FABRICATION TRANSITORS *INPUT/OUTPUT (PERIPHERAL) DEVICES SOLID-STATE SPECIAL PURPOSE DEVICES COMPUTER MICRO PROCESSORS AND PROGRAMS POWER SUPPLIES

Timeline historical review of income and financial transactions: a reliable assessment of personal finances.

PubMed

Black, Anne C; Serowik, Kristin L; Ablondi, Karen M; Rosen, Marc I

2013-01-01

The need for accurate and reliable information about income and resources available to individuals with psychiatric disabilities is critical for the assessment of need and evaluation of programs designed to alleviate financial hardship or affect finance allocation. Measurement of finances is ubiquitous in studies of economics, poverty, and social services. However, evidence has demonstrated that these measures often contain error. We compare the 1-week test-retest reliability of income and finance data from 24 adult psychiatric outpatients using assessment-as-usual (AAU) and a new instrument, the Timeline Historical Review of Income and Financial Transactions (THRIFT). Reliability estimates obtained with the THRIFT for Income (0.77), Expenses (0.91), and Debt (0.99) domains were significantly better than those obtained with AAU. Reliability estimates for Balance did not differ. THRIFT reduced measurement error and provided more reliable information than AAU for assessment of personal finances in psychiatric patients receiving Social Security benefits. The instrument also may be useful with other low-income groups.

Evaluation Criteria for Micro-CAI: A Psychometric Approach

PubMed Central

Wallace, Douglas; Slichter, Mark; Bolwell, Christine

1985-01-01

The increased use of microcomputer-based instructional programs has resulted in a greater need for third-party evaluation of the software. This in turn has prompted the development of micro-CAI evaluation tools. The present project sought to develop a prototype instrument to assess the impact of CAI program presentation characteristics on students. Data analysis and scale construction was conducted using standard item reliability analyses and factor analytic techniques. Adequate subscale reliabilities and factor structures were found, suggesting that a psychometric approach to CAI evaluation may possess some merit. Efforts to assess the utility of the resultant instrument are currently underway.

The 747 primary flight control systems reliability and maintenance study

NASA Technical Reports Server (NTRS)

1979-01-01

The major operational characteristics of the 747 Primary Flight Control Systems (PFCS) are described. Results of reliability analysis for separate control functions are presented. The analysis makes use of a NASA computer program which calculates reliability of redundant systems. Costs for maintaining the 747 PFCS in airline service are assessed. The reliabilities and cost will provide a baseline for use in trade studies of future flight control system design.

Assessing performance outcomes of new graduates utilizing simulation in a military transition program.

PubMed

Hughes, Robie V; Smith, Sherrill J; Sheffield, Clair M; Wier, Grady

2013-01-01

This multi-site, quasi-experimental study examined the performance outcomes of nurses (n = 152) in a military nurse transition program. A modified-performance instrument was used to assess participants in two high-fidelity simulation scenarios. Although results indicated a significant increase in scores posttraining, only moderate interrater reliability results were found for the new instrument. These findings have implications for nurse educators assessing performance-based outcomes of new nurses completing transition programs.

The Growth of the International Baccalaureate[R] Diploma Program: Concerns about the Consistency and Reliability of the Assessments

ERIC Educational Resources Information Center

Bunnell, Tristan

2011-01-01

The International Baccalaureate[R] (IB) world, known as "the IB World," is doubling in size every five years. The IB has become a complex educational product, but offers high levels of consistency and reliability in terms of delivery and assessment. However, since late 2008, a number of concerns have been raised about the quality and manageability…

The value and limitations of global air-sampling networks for improving our understanding trace gas behavior

NASA Astrophysics Data System (ADS)

Montzka, S. A.

2016-12-01

Measurements from global surface-based air sampling networks provide a fundamental understanding of how and why concentrations of long-lived trace gases are changing over time. Results from these networks are used to quantify trace-gas concentrations and their time-dependent changes on global and smaller scales, and thus provide a means to quantify emission rates, loss frequencies, and mixing processes. Substantial advances in measurement and sampling technologies and the ability of these programs to create and maintain reliable gas standards mean that spatial concentration gradients and time-dependent changes are often very reliably measured. The presence of multiple independent networks allows an assessment of this reliability. Furthermore, recent global `snap-shot' surveys (e.g., HIPPO and ATom) and ongoing atmospheric profiling programs help us assess the ability of surface-based data to describe concentration distributions throughout most of the atmosphere ( 80% of its mass). In this overview talk, I'll explore the usefulness and limitations of existing long-term, ongoing sampling network programs and their advantages and disadvantages for characterizing concentrations on global and regional scales, and how recent advances (and short-term sampling programs) help us assess the accuracy of the surface networks to provide estimates of source and sink magnitudes, and inter-annual variability in both.

38 CFR 1.15 - Standards for program evaluation.

Code of Federal Regulations, 2010 CFR

2010-07-01

... program operates. (3) Validity. The degree of statistical validity should be assessed within the research... decisions. (4) Reliability. Use of the same research design by others should yield the same findings. (g...

Effect of formal specifications on program complexity and reliability: An experimental study

NASA Technical Reports Server (NTRS)

Goel, Amrit L.; Sahoo, Swarupa N.

1990-01-01

The results are presented of an experimental study undertaken to assess the improvement in program quality by using formal specifications. Specifications in the Z notation were developed for a simple but realistic antimissile system. These specifications were then used to develop 2 versions in C by 2 programmers. Another set of 3 versions in Ada were independently developed from informal specifications in English. A comparison of the reliability and complexity of the resulting programs suggests the advantages of using formal specifications in terms of number of errors detected and fault avoidance.

Human Reliability Program Workshop

DOE Office of Scientific and Technical Information (OSTI.GOV)

Landers, John; Rogers, Erin; Gerke, Gretchen

A Human Reliability Program (HRP) is designed to protect national security as well as worker and public safety by continuously evaluating the reliability of those who have access to sensitive materials, facilities, and programs. Some elements of a site HRP include systematic (1) supervisory reviews, (2) medical and psychological assessments, (3) management evaluations, (4) personnel security reviews, and (4) training of HRP staff and critical positions. Over the years of implementing an HRP, the Department of Energy (DOE) has faced various challenges and overcome obstacles. During this 4-day activity, participants will examine programs that mitigate threats to nuclear security andmore » the insider threat to include HRP, Nuclear Security Culture (NSC) Enhancement, and Employee Assistance Programs. The focus will be to develop an understanding of the need for a systematic HRP and to discuss challenges and best practices associated with mitigating the insider threat.« less

History of Reliability and Quality Assurance at Kennedy Space Center

NASA Technical Reports Server (NTRS)

Childers, Frank M.

2004-01-01

This Kennedy Historical Document (KHD) provides a unique historical perspective of the organizational and functional responsibilities for the manned and un-manned programs at Kennedy Space Center, Florida. As systems become more complex and hazardous, the attention to detailed planning and execution continues to be a challenge. The need for a robust reliability and quality assurance program will always be a necessity to ensure mission success. As new space missions are defined and technology allows for continued access to space, these programs cannot be compromised. The organizational structure that has provided the reliability and quality assurance functions for both the manned and unmanned programs has seen many changes since the first group came to Florida in the 1950's. The roles of government and contractor personnel have changed with each program and organizational alignment has changed based on that responsibility. The organizational alignment of the personnel performing these functions must ensure independent assessment of the processes.

Implementation of a personnel reliability program as a facilitator of biosafety and biosecurity culture in BSL-3 and BSL-4 laboratories.

PubMed

Higgins, Jacki J; Weaver, Patrick; Fitch, J Patrick; Johnson, Barbara; Pearl, R Marene

2013-06-01

In late 2010, the National Biodefense Analysis and Countermeasures Center (NBACC) implemented a Personnel Reliability Program (PRP) with the goal of enabling active participation by its staff to drive and improve the biosafety and biosecurity culture at the organization. A philosophical keystone for accomplishment of NBACC's scientific mission is simultaneous excellence in operations and outreach. Its personnel reliability program builds on this approach to: (1) enable and support a culture of responsibility based on human performance principles, (2) maintain compliance with regulations, and (3) address the risk associated with the insider threat. Recently, the Code of Federal Regulations (CFR) governing use and possession of biological select agents and toxins (BSAT) was amended to require a pre-access suitability assessment and ongoing evaluation for staff accessing Tier 1 BSAT. These 2 new requirements are in addition to the already required Federal Bureau of Investigation (FBI) Security Risk Assessment (SRA). Two years prior to the release of these guidelines, NBACC developed its PRP to supplement the SRA requirement as a means to empower personnel and foster an operational environment where any and all work with BSAT is conducted in a safe, secure, and reliable manner.

Reliability of questionnaires to assess the healthy eating and activity environment of a child's home and school.

PubMed

Wilson, Annabelle; Magarey, Anthea; Mastersson, Nadia

2013-01-01

Childhood overweight and obesity are a growing concern globally, and environments, including the home and school, can contribute to this epidemic. This paper assesses the reliability of two questionnaires (parent and teacher) used in the evaluation of a community-based childhood obesity prevention intervention, the eat well be active (ewba) Community Programs. Parents and teachers were recruited from two primary schools and they completed the same questionnaire twice in 2008 and 2009. Data from both questionnaires were classified into outcomes relevant to healthy eating and activity, and target outcomes, based on the goals of the ewba Community Programs, were identified. Fourteen and 12 outcomes were developed from the parent and teacher questionnaires, respectively. Sixty parents and 28 teachers participated in the reliability study. Intraclass correlation coefficients for outcomes ranged from 0.37 to 0.92 (parent) (P < 0.05) and from 0.42 to 0.86 (teacher) (P < 0.05). Internal consistency, measured by Cronbach's alpha, of teacher scores ranged from 0.11 to 0.91 and 0.13 to 0.78 for scores from the parent questionnaire. The parent and teacher questionnaires are moderately reliable tools for simultaneously assessing child intakes, environments, attitudes, and knowledge associated with healthy eating and physical activity in the home and school and may be useful for evaluation of similar programs.

Software analysis handbook: Software complexity analysis and software reliability estimation and prediction

NASA Technical Reports Server (NTRS)

Lee, Alice T.; Gunn, Todd; Pham, Tuan; Ricaldi, Ron

1994-01-01

This handbook documents the three software analysis processes the Space Station Software Analysis team uses to assess space station software, including their backgrounds, theories, tools, and analysis procedures. Potential applications of these analysis results are also presented. The first section describes how software complexity analysis provides quantitative information on code, such as code structure and risk areas, throughout the software life cycle. Software complexity analysis allows an analyst to understand the software structure, identify critical software components, assess risk areas within a software system, identify testing deficiencies, and recommend program improvements. Performing this type of analysis during the early design phases of software development can positively affect the process, and may prevent later, much larger, difficulties. The second section describes how software reliability estimation and prediction analysis, or software reliability, provides a quantitative means to measure the probability of failure-free operation of a computer program, and describes the two tools used by JSC to determine failure rates and design tradeoffs between reliability, costs, performance, and schedule.

Development, validation, and utility of an instrument to assess core competencies in the Leadership Education in Neurodevelopmental and Related Disabilities (LEND) program.

PubMed

Leff, Stephen S; Baum, Katherine T; Bevans, Katherine B; Blum, Nathan J

2015-02-01

To describe the development and psychometric evaluation of the Core Competency Measure (CCM), an instrument designed to assess professional competencies as defined by the Maternal Child Health Bureau (MCHB) and targeted by Leadership Education in Neurodevelopmental and Related Disabilities (LEND) programs. The CCM is a 44-item self-report measure comprised of six subscales to assess clinical, interdisciplinary, family-centered/cultural, community, research, and advocacy/policy competencies. The CCM was developed in an iterative fashion through participatory action research, and then nine cohorts of LEND trainees (N = 144) from 14 different disciplines completed the CCM during the first week of the training program. A 6-factor confirmatory factor analysis model was fit to data from the 44 original items. After three items were removed, the model adequately fit the data (comparative fit indices = .93, root mean error of approximation = .06) with all factor loadings exceeding .55. The measure was determined to be quite reliable as adequate internal consistency and test-retest reliability were found for each subscale. The instrument's construct validity was supported by expected differences in self-rated competencies among fellows representing various disciplines, and the convergent validity was supported by the pattern of inter-correlations between subscale scores. The CCM appears to be a reliable and valid measure of MCHB core competencies for our sample of LEND trainees. It provides an assessment of key training areas addressed by the LEND program. Although the measure was developed within only one LEND Program, with additional research it has the potential to serve as a standardized tool to evaluate the strengths and limitations of MCHB training, both within and between programs.

Training and Maintaining System-Wide Reliability in Outcome Management.

PubMed

Barwick, Melanie A; Urajnik, Diana J; Moore, Julia E

2014-01-01

The Child and Adolescent Functional Assessment Scale (CAFAS) is widely used for outcome management, for providing real time client and program level data, and the monitoring of evidence-based practices. Methods of reliability training and the assessment of rater drift are critical for service decision-making within organizations and systems of care. We assessed two approaches for CAFAS training: external technical assistance and internal technical assistance. To this end, we sampled 315 practitioners trained by external technical assistance approach from 2,344 Ontario practitioners who had achieved reliability on the CAFAS. To assess the internal technical assistance approach as a reliable alternative training method, 140 practitioners trained internally were selected from the same pool of certified raters. Reliabilities were high for both practitioners trained by external technical assistance and internal technical assistance approaches (.909-.995, .915-.997, respectively). 1 and 3-year estimates showed some drift on several scales. High and consistent reliabilities over time and training method has implications for CAFAS training of behavioral health care practitioners, and the maintenance of CAFAS as a global outcome management tool in systems of care.

Assessing the Quality of Mobile Exercise Apps Based on the American College of Sports Medicine Guidelines: A Reliable and Valid Scoring Instrument.

PubMed

Guo, Yi; Bian, Jiang; Leavitt, Trevor; Vincent, Heather K; Vander Zalm, Lindsey; Teurlings, Tyler L; Smith, Megan D; Modave, François

2017-03-07

Regular physical activity can not only help with weight management, but also lower cardiovascular risks, cancer rates, and chronic disease burden. Yet, only approximately 20% of Americans currently meet the physical activity guidelines recommended by the US Department of Health and Human Services. With the rapid development of mobile technologies, mobile apps have the potential to improve participation rates in exercise programs, particularly if they are evidence-based and are of sufficient content quality. The goal of this study was to develop and test an instrument, which was designed to score the content quality of exercise program apps with respect to the exercise guidelines set forth by the American College of Sports Medicine (ACSM). We conducted two focus groups (N=14) to elicit input for developing a preliminary 27-item scoring instruments based on the ACSM exercise prescription guidelines. Three reviewers who were no sports medicine experts independently scored 28 exercise program apps using the instrument. Inter- and intra-rater reliability was assessed among the 3 reviewers. An expert reviewer, a Fellow of the ACSM, also scored the 28 apps to create criterion scores. Criterion validity was assessed by comparing nonexpert reviewers' scores to the criterion scores. Overall, inter- and intra-rater reliability was high with most coefficients being greater than .7. Inter-rater reliability coefficients ranged from .59 to .99, and intra-rater reliability coefficients ranged from .47 to 1.00. All reliability coefficients were statistically significant. Criterion validity was found to be excellent, with the weighted kappa statistics ranging from .67 to .99, indicating a substantial agreement between the scores of expert and nonexpert reviewers. Finally, all apps scored poorly against the ACSM exercise prescription guidelines. None of the apps received a score greater than 35, out of a possible maximal score of 70. We have developed and presented valid and reliable scoring instruments for exercise program apps. Our instrument may be useful for consumers and health care providers who are looking for apps that provide safe, progressive general exercise programs for health and fitness. ©Yi Guo, Jiang Bian, Trevor Leavitt, Heather K Vincent, Lindsey Vander Zalm, Tyler L Teurlings, Megan D Smith, François Modave. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 07.03.2017.

Assessment of NDE reliability data

NASA Technical Reports Server (NTRS)

Yee, B. G. W.; Couchman, J. C.; Chang, F. H.; Packman, D. F.

1975-01-01

Twenty sets of relevant nondestructive test (NDT) reliability data were identified, collected, compiled, and categorized. A criterion for the selection of data for statistical analysis considerations was formulated, and a model to grade the quality and validity of the data sets was developed. Data input formats, which record the pertinent parameters of the defect/specimen and inspection procedures, were formulated for each NDE method. A comprehensive computer program was written and debugged to calculate the probability of flaw detection at several confidence limits by the binomial distribution. This program also selects the desired data sets for pooling and tests the statistical pooling criteria before calculating the composite detection reliability. An example of the calculated reliability of crack detection in bolt holes by an automatic eddy current method is presented.

Development of Reliable and Validated Tools to Evaluate Technical Resuscitation Skills in a Pediatric Simulation Setting: Resuscitation and Emergency Simulation Checklist for Assessment in Pediatrics.

PubMed

Faudeux, Camille; Tran, Antoine; Dupont, Audrey; Desmontils, Jonathan; Montaudié, Isabelle; Bréaud, Jean; Braun, Marc; Fournier, Jean-Paul; Bérard, Etienne; Berlengi, Noémie; Schweitzer, Cyril; Haas, Hervé; Caci, Hervé; Gatin, Amélie; Giovannini-Chami, Lisa

2017-09-01

To develop a reliable and validated tool to evaluate technical resuscitation skills in a pediatric simulation setting. Four Resuscitation and Emergency Simulation Checklist for Assessment in Pediatrics (RESCAPE) evaluation tools were created, following international guidelines: intraosseous needle insertion, bag mask ventilation, endotracheal intubation, and cardiac massage. We applied a modified Delphi methodology evaluation to binary rating items. Reliability was assessed comparing the ratings of 2 observers (1 in real time and 1 after a video-recorded review). The tools were assessed for content, construct, and criterion validity, and for sensitivity to change. Inter-rater reliability, evaluated with Cohen kappa coefficients, was perfect or near-perfect (>0.8) for 92.5% of items and each Cronbach alpha coefficient was ≥0.91. Principal component analyses showed that all 4 tools were unidimensional. Significant increases in median scores with increasing levels of medical expertise were demonstrated for RESCAPE-intraosseous needle insertion (P = .0002), RESCAPE-bag mask ventilation (P = .0002), RESCAPE-endotracheal intubation (P = .0001), and RESCAPE-cardiac massage (P = .0037). Significantly increased median scores over time were also demonstrated during a simulation-based educational program. RESCAPE tools are reliable and validated tools for the evaluation of technical resuscitation skills in pediatric settings during simulation-based educational programs. They might also be used for medical practice performance evaluations. Copyright © 2017 Elsevier Inc. All rights reserved.

The Validation of a Food Label Literacy Questionnaire for Elementary School Children

ERIC Educational Resources Information Center

Reynolds, Jesse S.; Treu, Judith A.; Njike, Valentine; Walker, Jennifer; Smith, Erica; Katz, Catherine S.; Katz, David L.

2012-01-01

Objective: To determine the reliability and validity of a 10-item questionnaire, the Food Label Literacy for Applied Nutrition Knowledge questionnaire. Methods: Participants were elementary school children exposed to a 90-minute school-based nutrition program. Reliability was assessed via Cronbach alpha and intraclass correlation coefficient…

Reliability assessment of Multichip Module technologies via the Triservice/NASA RELTECH program

NASA Astrophysics Data System (ADS)

Fayette, Daniel F.

1994-10-01

Multichip Module (MCM) packaging/interconnect technologies have seen increased emphasis from both the commercial and military communities as a means of increasing capability and performance while providing a vehicle for reducing cost, power and weight of the end item electronic application. This is accomplished through three basic Multichip module technologies, MCM-L that are laminates, MCM-C that are ceramic type substrates and MCM-D that are deposited substrates (e.g., polymer dielectric with thin film metals). Three types of interconnect structures are also used with these substrates and include, wire bond, Tape Automated Bonds (TAB) and flip chip ball bonds. Application, cost, producibility and reliability are the drivers that will determine which MCM technology will best fit a respective need or requirement. With all the benefits and technologies cited, it would be expected that the use of, or the planned use of, MCM's would be more extensive in both military and commercial applications. However, two significant roadblocks exist to implementation of these new technologies: the absence of reliability data and a single national standard for the procurement of reliable/quality MCM's. To address the preceding issues, the Reliability Technology to Achieve Insertion of Advanced Packaging (RELTECH) program has been established. This program, which began in May 1992, has endeavored to evaluate a cross section of MCM technologies covering all classes of MCM's previously cited. NASA and the Tri-Services (Air Force Rome Laboratory, Naval Surface Warfare Center, Crane IN and Army Research Laboratory) have teamed together with sponsorship from ARPA to evaluate the performance, reliability and producibility of MCM's for both military and commercial usage. This is done in close cooperation with our industry partners whose support is critical to the goals of the program. Several tasks are being performed by the RELTECH program and data from this effort, in conjunction with information from our industry partners as well as discussions with industry organizations (IPC, EIA, ISHM, etc.) are being used to develop the qualification and screening requirements for MCM's. Specific tasks being performed by the RELTECH program include technical assessments, product evaluations, reliability modeling, environmental testing, and failure analysis. This paper will describe the various tasks associated with the RELTECH program, status, progress and a description of the national dual use specification being developed for MCM technologies.

Reliability Technology to Achieve Insertion of Advanced Packaging (RELTECH) program

NASA Astrophysics Data System (ADS)

Fayette, Daniel F.; Speicher, Patricia; Stoklosa, Mark J.; Evans, Jillian V.; Evans, John W.; Gentile, Mike; Pagel, Chuck A.; Hakim, Edward

1993-08-01

A joint military-commercial effort to evaluate multichip module (MCM) structures is discussed. The program, Reliability Technology to Achieve Insertion of Advanced Packaging (RELTECH), has been designed to identify the failure mechanisms that are possible in MCM structures. The RELTECH test vehicles, technical assessment task, product evaluation plan, reliability modeling task, accelerated and environmental testing, and post-test physical analysis and failure analysis are described. The information obtained through RELTECH can be used to address standardization issues, through development of cost effective qualification and appropriate screening criteria, for inclusion into a commercial specification and the MIL-H-38534 general specification for hybrid microcircuits.

Reliability Technology to Achieve Insertion of Advanced Packaging (RELTECH) program

NASA Technical Reports Server (NTRS)

Fayette, Daniel F.; Speicher, Patricia; Stoklosa, Mark J.; Evans, Jillian V.; Evans, John W.; Gentile, Mike; Pagel, Chuck A.; Hakim, Edward

1993-01-01

A joint military-commercial effort to evaluate multichip module (MCM) structures is discussed. The program, Reliability Technology to Achieve Insertion of Advanced Packaging (RELTECH), has been designed to identify the failure mechanisms that are possible in MCM structures. The RELTECH test vehicles, technical assessment task, product evaluation plan, reliability modeling task, accelerated and environmental testing, and post-test physical analysis and failure analysis are described. The information obtained through RELTECH can be used to address standardization issues, through development of cost effective qualification and appropriate screening criteria, for inclusion into a commercial specification and the MIL-H-38534 general specification for hybrid microcircuits.

Development of the Systems Thinking Scale for Adolescent Behavior Change.

PubMed

Moore, Shirley M; Komton, Vilailert; Adegbite-Adeniyi, Clara; Dolansky, Mary A; Hardin, Heather K; Borawski, Elaine A

2018-03-01

This report describes the development and psychometric testing of the Systems Thinking Scale for Adolescent Behavior Change (STS-AB). Following item development, initial assessments of understandability and stability of the STS-AB were conducted in a sample of nine adolescents enrolled in a weight management program. Exploratory factor analysis of the 16-item STS-AB and internal consistency assessments were then done with 359 adolescents enrolled in a weight management program. Test-retest reliability of the STS-AB was .71, p = .03; internal consistency reliability was .87. Factor analysis of the 16-item STS-AB indicated a one-factor solution with good factor loadings, ranging from .40 to .67. Evidence of construct validity was supported by significant correlations with established measures of variables associated with health behavior change. We provide beginning evidence of the reliability and validity of the STS-AB to measure systems thinking for health behavior change in young adolescents.

Development of the Systems Thinking Scale for Adolescent Behavior Change

PubMed Central

Moore, Shirley M.; Komton, Vilailert; Adegbite-Adeniyi, Clara; Dolansky, Mary A.; Hardin, Heather K.; Borawski, Elaine A.

2017-01-01

This report describes the development and psychometric testing of the Systems Thinking Scale for Adolescent Behavior Change (STS-AB). Following item development, initial assessments of understandability and stability of the STS-AB were conducted in a sample of nine adolescents enrolled in a weight management program. Exploratory factor analysis of the 16-item STS-AB and internal consistency assessments were then done with 359 adolescents enrolled in a weight management program. Test–retest reliability of the STS-AB was .71, p = .03; internal consistency reliability was .87. Factor analysis of the 16-item STS-AB indicated a one-factor solution with good factor loadings, ranging from .40 to .67. Evidence of construct validity was supported by significant correlations with established measures of variables associated with health behavior change. We provide beginning evidence of the reliability and validity of the STS-AB to measure systems thinking for health behavior change in young adolescents. PMID:28303755

Reliability of risk assessment measures used in sexually violent predator proceedings.

PubMed

Miller, Cailey S; Kimonis, Eva R; Otto, Randy K; Kline, Suzonne M; Wasserman, Adam L

2012-12-01

The field interrater reliability of three assessment tools frequently used by mental health professionals when evaluating sex offenders' risk for reoffending--the Psychopathy Checklist-Revised (PCL-R), the Minnesota Sex Offender Screening Tool-Revised (MnSOST-R) and the Static-99-was examined within the context of sexually violent predator program proceedings. Rater agreement was highest for the Static--99 (intraclass correlation coefficient [ICC₁] = .78) and lowest for the PCL-R (ICC₁ = .60; MnSOST-R ICC₁ = .74), although all instruments demonstrated lower field reliability than that reported in their test manuals. Findings raise concerns about the reliability of risk assessment tools that are used to inform judgments of risk in high-stake sexually violent predator proceedings. Implications for future research and suggestions for improving evaluator training to increase accuracy when informing legal decision making are discussed.

10 CFR 712.36 - Medical assessment process.

Code of Federal Regulations, 2014 CFR

2014-01-01

... 10 Energy 4 2014-01-01 2014-01-01 false Medical assessment process. 712.36 Section 712.36 Energy DEPARTMENT OF ENERGY HUMAN RELIABILITY PROGRAM Medical Standards § 712.36 Medical assessment process. (a) The Designated Physician, under the supervision of the SOMD, is responsible for the medical assessment of HRP...

10 CFR 712.36 - Medical assessment process.

Code of Federal Regulations, 2012 CFR

2012-01-01

... 10 Energy 4 2012-01-01 2012-01-01 false Medical assessment process. 712.36 Section 712.36 Energy DEPARTMENT OF ENERGY HUMAN RELIABILITY PROGRAM Medical Standards § 712.36 Medical assessment process. (a) The Designated Physician, under the supervision of the SOMD, is responsible for the medical assessment of HRP...

The relationship between faculty performance assessment and results on the in-training examination for residents in an emergency medicine training program.

PubMed

Ryan, James G; Barlas, David; Pollack, Simcha

2013-12-01

Medical knowledge (MK) in residents is commonly assessed by the in-training examination (ITE) and faculty evaluations of resident performance. We assessed the reliability of clinical evaluations of residents by faculty and the relationship between faculty assessments of resident performance and ITE scores. We conducted a cross-sectional, observational study at an academic emergency department with a postgraduate year (PGY)-1 to PGY-3 emergency medicine residency program, comparing summative, quarterly, faculty evaluation data for MK and overall clinical competency (OC) with annual ITE scores, accounting for PGY level. We also assessed the reliability of faculty evaluations using a random effects, intraclass correlation analysis. We analyzed data for 59 emergency medicine residents during a 6-year period. Faculty evaluations of MK and OC were highly reliable (κ  =  0.99) and remained reliable after stratification by year of training (mean κ  =  0.68-0.84). Assessments of resident performance (MK and OC) and the ITE increased with PGY level. The MK and OC results had high correlations with PGY level, and ITE scores correlated moderately with PGY. The OC and MK results had a moderate correlation with ITE score. When residents were grouped by PGY level, there was no significant correlation between MK as assessed by the faculty and the ITE score. Resident clinical performance and ITE scores both increase with resident PGY level, but ITE scores do not predict resident clinical performance compared with peers at their PGY level.

The Relationship Between Faculty Performance Assessment and Results on the In-Training Examination for Residents in an Emergency Medicine Training Program

PubMed Central

Ryan, James G.; Barlas, David; Pollack, Simcha

2013-01-01

Background Medical knowledge (MK) in residents is commonly assessed by the in-training examination (ITE) and faculty evaluations of resident performance. Objective We assessed the reliability of clinical evaluations of residents by faculty and the relationship between faculty assessments of resident performance and ITE scores. Methods We conducted a cross-sectional, observational study at an academic emergency department with a postgraduate year (PGY)-1 to PGY-3 emergency medicine residency program, comparing summative, quarterly, faculty evaluation data for MK and overall clinical competency (OC) with annual ITE scores, accounting for PGY level. We also assessed the reliability of faculty evaluations using a random effects, intraclass correlation analysis. Results We analyzed data for 59 emergency medicine residents during a 6-year period. Faculty evaluations of MK and OC were highly reliable (κ  =  0.99) and remained reliable after stratification by year of training (mean κ  =  0.68–0.84). Assessments of resident performance (MK and OC) and the ITE increased with PGY level. The MK and OC results had high correlations with PGY level, and ITE scores correlated moderately with PGY. The OC and MK results had a moderate correlation with ITE score. When residents were grouped by PGY level, there was no significant correlation between MK as assessed by the faculty and the ITE score. Conclusions Resident clinical performance and ITE scores both increase with resident PGY level, but ITE scores do not predict resident clinical performance compared with peers at their PGY level. PMID:24455005

Medical student quality-of-life in the clerkships: a scale validation study.

PubMed

Brannick, Michael T; Horn, Gregory T; Schnaus, Michael J; Wahi, Monika M; Goldin, Steven B

2015-04-01

Many aspects of medical school are stressful for students. To empirically assess student reactions to clerkship programs, or to assess efforts to improve such programs, educators must measure the overall well-being of the students reliably and validly. The purpose of the study was to develop and validate a measure designed to achieve these goals. The authors developed a measure of quality of life for medical students by sampling (public domain) items tapping general happiness, fatigue, and anxiety. A quality-of-life scale was developed by factor analyzing responses to the items from students in two different clerkships from 2005 to 2008. Reliability was assessed using Cronbach's alpha. Validity was assessed by factor analysis, convergence with additional theoretically relevant scales, and sensitivity to change over time. The refined nine-item measure is a Likert scaled survey of quality-of-life items comprised of two domains: exhaustion and general happiness. The resulting scale demonstrated good reliability and factorial validity at two time points for each of the two samples. The quality-of-life measure also correlated with measures of depression and the amount of sleep reported during the clerkships. The quality-of-life measure appeared more sensitive to changes over time than did the depression measure. The measure is short and can be easily administered in a survey. The scale appears useful for program evaluation and more generally as an outcome variable in medical educational research.

Constellation Program (CxP) Crew Exploration Vehicle (CEV) Parachute Assembly System (CPAS) Independent Design Reliability Assessment. Volume 1

NASA Technical Reports Server (NTRS)

Kelly, Michael J.

2010-01-01

This report documents the activities, findings, and NASA Engineering and Safety Center (NESC) recommendations of a multidiscipline team to independently assess the Constellation Program (CxP) Crew Exploration Vehicle (CEV) Parachute Assembly System (CPAS). This assessment occurred during a period of 15 noncontiguous months between December 2008 and April 2010, prior to the CPAS Project's Preliminary Design Review (PDR) in August 2010.

10 CFR 712.36 - Medical assessment process.

Code of Federal Regulations, 2011 CFR

2011-01-01

... 10 Energy 4 2011-01-01 2011-01-01 false Medical assessment process. 712.36 Section 712.36 Energy DEPARTMENT OF ENERGY HUMAN RELIABILITY PROGRAM Medical Standards § 712.36 Medical assessment process. (a) The... the SOMD must integrate the medical evaluations, psychological evaluations, psychiatric evaluations...

Implementation of a Personnel Reliability Program as a Facilitator of Biosafety and Biosecurity Culture in BSL-3 and BSL-4 Laboratories

PubMed Central

Weaver, Patrick; Fitch, J. Patrick; Johnson, Barbara; Pearl, R. Marene

2013-01-01

In late 2010, the National Biodefense Analysis and Countermeasures Center (NBACC) implemented a Personnel Reliability Program (PRP) with the goal of enabling active participation by its staff to drive and improve the biosafety and biosecurity culture at the organization. A philosophical keystone for accomplishment of NBACC's scientific mission is simultaneous excellence in operations and outreach. Its personnel reliability program builds on this approach to: (1) enable and support a culture of responsibility based on human performance principles, (2) maintain compliance with regulations, and (3) address the risk associated with the insider threat. Recently, the Code of Federal Regulations (CFR) governing use and possession of biological select agents and toxins (BSAT) was amended to require a pre-access suitability assessment and ongoing evaluation for staff accessing Tier 1 BSAT. These 2 new requirements are in addition to the already required Federal Bureau of Investigation (FBI) Security Risk Assessment (SRA). Two years prior to the release of these guidelines, NBACC developed its PRP to supplement the SRA requirement as a means to empower personnel and foster an operational environment where any and all work with BSAT is conducted in a safe, secure, and reliable manner. PMID:23745523

Development and validation of a questionnaire to evaluate patient satisfaction with diabetes disease management.

PubMed

Paddock, L E; Veloski, J; Chatterton, M L; Gevirtz, F O; Nash, D B

2000-07-01

To develop a reliable and valid questionnaire to measure patient satisfaction with diabetes disease management programs. Questions related to structure, process, and outcomes were categorized into 14 domains defining the essential elements of diabetes disease management. Health professionals confirmed the content validity. Face validity was established by a patient focus group. The questionnaire was mailed to 711 patients with diabetes who participated in a disease management program. To reduce the number of questionnaire items, a principal components analysis was performed using a varimax rotation. The Scree test was used to select significant components. To further assess reliability and validity; Cronbach's alpha and product-moment correlations were calculated for components having > or =3 items with loadings >0.50. The validated 73-item mailed satisfaction survey had a 34.1% response rate. Principal components analysis yielded 13 components with eigenvalues > 1.0. The Scree test proposed a 6-component solution (39 items), which explained 59% of the total variation. Internal consistency reliabilities computed for the first 6 components (alpha = 0.79-0.95) were acceptable. The final questionnaire, the Diabetes Management Evaluation Tool (DMET), was designed to assess patient satisfaction with diabetes disease management programs. Although more extensive testing of the questionnaire is appropriate, preliminary reliability and validity of the DMET has been demonstrated.

Validity and reliability of an in-training evaluation report to measure the CanMEDS roles in emergency medicine residents.

PubMed

Kassam, Aliya; Donnon, Tyrone; Rigby, Ian

2014-03-01

There is a question of whether a single assessment tool can assess the key competencies of residents as mandated by the Royal College of Physicians and Surgeons of Canada CanMEDS roles framework. The objective of the present study was to investigate the reliability and validity of an emergency medicine (EM) in-training evaluation report (ITER). ITER data from 2009 to 2011 were combined for residents across the 5 years of the EM residency training program. An exploratory factor analysis with varimax rotation was used to explore the construct validity of the ITER. A total of 172 ITERs were completed on residents across their first to fifth year of training. A combined, 24-item ITER yielded a five-factor solution measuring the CanMEDs role Medical Expert/Scholar, Communicator/Collaborator, Professional, Health Advocate and Manager subscales. The factor solution accounted for 79% of the variance, and reliability coefficients (Cronbach alpha) ranged from α  =  0.90 to 0.95 for each subscale and α  =  0.97 overall. The combined, 24-item ITER used to assess residents' competencies in the EM residency program showed strong reliability and evidence of construct validity for assessment of the CanMEDS roles. Further research is needed to develop and test ITER items that will differentiate each CanMEDS role exclusively.

Methodology for Developing a New EFNEP Food and Physical Activity Behaviors Questionnaire.

PubMed

Murray, Erin K; Auld, Garry; Baker, Susan S; Barale, Karen; Franck, Karen; Khan, Tarana; Palmer-Keenan, Debra; Walsh, Jennifer

2017-10-01

Research methods are described for developing a food and physical activity behaviors questionnaire for the Expanded Food and Nutrition Education Program (EFNEP), a US Department of Agriculture nutrition education program serving low-income families. Mixed-methods observational study. The questionnaire will include 5 domains: (1) diet quality, (2) physical activity, (3) food safety, (4) food security, and (5) food resource management. A 5-stage process will be used to assess the questionnaire's test-retest reliability and content, face, and construct validity. Research teams across the US will coordinate questionnaire development and testing nationally. Convenience samples of low-income EFNEP, or EFNEP-eligible, adult participants across the US. A 5-stage process: (1) prioritize domain concepts to evaluate (2) question generation and content analysis panel, (3) question pretesting using cognitive interviews, (4) test-retest reliability assessment, and (5) construct validity testing. A nationally tested valid and reliable food and physical activity behaviors questionnaire for low-income adults to evaluate EFNEP's effectiveness. Cognitive interviews will be summarized to identify themes and dominant trends. Paired t tests (P ≤ .05) and Spearman and intra-class correlation coefficients (r > .5) will be conducted to assess reliability. Construct validity will be assessed using Wilcoxon t test (P ≤ .05), Spearman correlations, and Bland-Altman plots. Copyright © 2017 Society for Nutrition Education and Behavior. Published by Elsevier Inc. All rights reserved.

Interpretive Reliability of Six Computer-Based Test Interpretation Programs for the Minnesota Multiphasic Personality Inventory-2.

PubMed

Deskovitz, Mark A; Weed, Nathan C; McLaughlan, Joseph K; Williams, John E

2016-04-01

The reliability of six Minnesota Multiphasic Personality Inventory-Second edition (MMPI-2) computer-based test interpretation (CBTI) programs was evaluated across a set of 20 commonly appearing MMPI-2 profile codetypes in clinical settings. Evaluation of CBTI reliability comprised examination of (a) interrater reliability, the degree to which raters arrive at similar inferences based on the same CBTI profile and (b) interprogram reliability, the level of agreement across different CBTI systems. Profile inferences drawn by four raters were operationalized using q-sort methodology. Results revealed no significant differences overall with regard to interrater and interprogram reliability. Some specific CBTI/profile combinations (e.g., the CBTI by Automated Assessment Associates on a within normal limits profile) and specific profiles (e.g., the 4/9 profile displayed greater interprogram reliability than the 2/4 profile) were interpreted with variable consensus (α range = .21-.95). In practice, users should consider that certain MMPI-2 profiles are interpreted more or less consensually and that some CBTIs show variable reliability depending on the profile. © The Author(s) 2015.

Development and validation of an exercise performance support system for people with lower extremity impairment.

PubMed

Minor, M A; Reid, J C; Griffin, J Z; Pittman, C B; Patrick, T B; Cutts, J H

1998-02-01

To identify innovative strategies to support appropriate, self-directed exercise that increase physical activity levels of people with arthritis. This article reports on one interactive, multimedia exercise performance support system (PSS) for people with lower extremity impairments in strength or flexibility. An interdisciplinary team developed the PSS using self-report of lower extremity musculoskeletal impairments (flexibility and strength) to produce an individualized exercise program with video and print educational materials. Initial evaluation has investigated the validity and reliability of program assessments and recommendations. PSS self-report and professional assessments were similar, with more impairments indicated by self-report. PSS exercise recommendations were similar to those made by 3 expert physical therapists using the same exercise data base. Results of PSS impairment assessments were stable over a 1-week period. PSS exercise recommendations appear to be reliable and a valid reflection of current exercise knowledge in rheumatology. Furthermore, users were able to complete the computer-based program with minimal assistance and reported it to be enjoyable and informative.

Validating the Alcohol Use Disorders Identification Test with Persons Who Have a Serious Mental Illness

ERIC Educational Resources Information Center

O'Hare, Thomas; Sherrer, Margaret V.; LaButti, Annamaria; Emrick, Kelly

2004-01-01

Objective/Method: The use of brief, reliable, valid, and practical measures of substance use is critical for conducting individual assessments and program evaluation for integrated mental health-substance abuse services for persons with serious mental illness. This investigation examines the internal consistency reliability, concurrent validity,…

Final Report of the Vocational Assessment Project, 1979-80.

ERIC Educational Resources Information Center

Rutgers, The State Univ., New Brunswick, NJ. School of Medicine.

To improve vocational rehabilitation programs for schizophrenic persons, a project sought to design an effective assessment strategy. Inactive records of schizophrenic clients at New Jersey sheltered workshops were examined to determine validity and reliability of assessment instruments being used. General Aptitude Test Battery (GATB) profiles of…

The Diagnostic Validity and Reliability of an Internet-Based Clinical Assessment Program for Mental Disorders

PubMed Central

Klein, Britt; Meyer, Denny; Austin, David William; Abbott, Jo-Anne M

2015-01-01

Background Internet-based assessment has the potential to assist with the diagnosis of mental health disorders and overcome the barriers associated with traditional services (eg, cost, stigma, distance). Further to existing online screening programs available, there is an opportunity to deliver more comprehensive and accurate diagnostic tools to supplement the assessment and treatment of mental health disorders. Objective The aim was to evaluate the diagnostic criterion validity and test-retest reliability of the electronic Psychological Assessment System (e-PASS), an online, self-report, multidisorder, clinical assessment and referral system. Methods Participants were 616 adults residing in Australia, recruited online, and representing prospective e-PASS users. Following e-PASS completion, 158 participants underwent a telephone-administered structured clinical interview and 39 participants repeated the e-PASS within 25 days of initial completion. Results With structured clinical interview results serving as the gold standard, diagnostic agreement with the e-PASS varied considerably from fair (eg, generalized anxiety disorder: κ=.37) to strong (eg, panic disorder: κ=.62). Although the e-PASS’ sensitivity also varied (0.43-0.86) the specificity was generally high (0.68-1.00). The e-PASS sensitivity generally improved when reducing the e-PASS threshold to a subclinical result. Test-retest reliability ranged from moderate (eg, specific phobia: κ=.54) to substantial (eg, bulimia nervosa: κ=.87). Conclusions The e-PASS produces reliable diagnostic results and performs generally well in excluding mental disorders, although at the expense of sensitivity. For screening purposes, the e-PASS subclinical result generally appears better than a clinical result as a diagnostic indicator. Further development and evaluation is needed to support the use of online diagnostic assessment programs for mental disorders. Trial Registration Australian and New Zealand Clinical Trials Registry ACTRN121611000704998; http://www.anzctr.org.au/trial_view.aspx?ID=336143 (Archived by WebCite at http://www.webcitation.org/618r3wvOG). PMID:26392066

Development of KSC program for investigating and generating field failure rates. Reliability handbook for ground support equipment

NASA Technical Reports Server (NTRS)

Bloomquist, C. E.; Kallmeyer, R. H.

1972-01-01

Field failure rates and confidence factors are presented for 88 identifiable components of the ground support equipment at the John F. Kennedy Space Center. For most of these, supplementary information regarding failure mode and cause is tabulated. Complete reliability assessments are included for three systems, eight subsystems, and nine generic piece-part classifications. Procedures for updating or augmenting the reliability results are also included.

Predicting Vandalism in a General Youth Sample via the HEW Youth Development Model's Community Program Impact Scales, Age, and Sex.

ERIC Educational Resources Information Center

Truckenmiller, James L.

The former HEW National Strategy for Youth Development model was a community-based planning and procedural tool to enhance and to prevent delinquency through a process of youth needs assessments, needs targeted programs, and program impact evaluation. The program's 12 Impact Scales have been found to have acceptable reliabilities, substantial…

An instrument to characterize the environment for residents' evidence-based medicine learning and practice.

PubMed

Mi, Misa; Moseley, James L; Green, Michael L

2012-02-01

Many residency programs offer training in evidence-based medicine (EBM). However, these curricula often fail to achieve optimal learning outcomes, perhaps because they neglect various contextual factors in the learning environment. We developed and validated an instrument to characterize the environment for EBM learning and practice in residency programs. An EBM Environment Scale was developed following scale development principles. A survey was administered to residents across six programs in primary care specialties at four medical centers. Internal consistency reliability was analyzed with Cronbach's coefficient alpha. Validity was assessed by comparing predetermined subscales with the survey's internal structure as assessed via factor analysis. Scores were also compared for subgroups based on residency program affiliation and residency characteristics. Out of 262 eligible residents, 124 completed the survey (response rate 47%). The overall mean score was 3.89 (standard deviation=0.56). The initial reliability analysis of the 48-item scale had a high reliability coefficient (Cronbach α=.94). Factor analysis and further item analysis resulted in a shorter 36-item scale with a satisfactory reliability coefficient (Cronbach α=.86). Scores were higher for residents with prior EBM training in medical school (4.14 versus 3.62) and in residency (4.25 versus 3.69). If further testing confirms its properties, the EBM Environment Scale may be used to understand the influence of the learning environment on the effectiveness of EBM training. Additionally, it may detect changes in the EBM learning environment in response to programmatic or institutional interventions.

Psychometrics of the MHSIP Adult Consumer Survey.

PubMed

Jerrell, Jeanette M

2006-10-01

The reliability and validity of the Mental Health Statistics Improvement Program (MHSIP) Adult Consumer Survey were assessed in a statewide convenience sample of 459 persons with severe mental illness served through a public mental health system. Consistent with previous findings and the intent of its developers, three factors were identified that demonstrate good internal consistency, moderate test-retest reliability, and good convergent validity with consumer perceptions of other aspects of their care. The reliability and validity of the MHSIP Adult Consumer Survey documented in this study underscore its scientific and practical utility as an abbreviated tool for assessing access, quality and appropriateness, and outcome in mental health service systems.

Constellation Program (CxP) Crew Exploration Vehicle (CEV) Parachute Assembly System (CPAS) Independent Design Reliability Assessment. Volume 2; Appendices

NASA Technical Reports Server (NTRS)

Kelly, Michael J.

2010-01-01

This document contains the Appendices to the report documenting the activities, findings, and NASA Engineering and Safety Center (NESC) recommendations of a multidiscipline team to independently assess the Constellation Program (CxP) Crew Exploration Vehicle (CEV) Parachute Assembly System (CPAS). The assessment occurred during a period of 15 noncontiguous months between December 2008 and April 2010, prior to the CPAS Project's Preliminary Design Review (PDR) in August 2010.

Assessing impact of physical activity-based youth development programs: validation of the Life Skills Transfer Survey (LSTS).

PubMed

Weiss, Maureen R; Bolter, Nicole D; Kipp, Lindsay E

2014-09-01

A signature characteristic of positive youth development (PYD) programs is the opportunity to develop life skills, such as social, behavioral, and moral competencies, that can be generalized to domains beyond the immediate activity. Although context-specific instruments are available to assess developmental outcomes, a measure of life skills transfer would enable evaluation of PYD programs in successfully teaching skills that youth report using in other domains. The purpose of our studies was to develop and validate a measure of perceived life skills transfer, based on data collected with The First Tee, a physical activity-based PYD program. In 3 studies, we conducted a series of steps to provide content and construct validity and internal consistency reliability for the Life Skills Transfer Survey (LSTS), a measure of perceived life skills transfer. Study 1 provided content validity for the LSTS that included 8 life skills and 50 items. Study 2 revealed construct validity (structural validity) through a confirmatory factor analysis and convergent validity by correlating scores on the LSTS with scores on an assessment tool that measures a related construct. Study 3 offered additional construct validity by reassessing youth 1 year later and showing that scores during both time periods were invariant in factor pattern, loadings, and variances and covariances. Studies 2 and 3 demonstrated internal consistency reliability of the LSTS. RESULTS from 3 studies provide evidence of content and construct validity and internal consistency reliability for the LSTS, which can be used in evaluation research with youth development programs.

Development and Field-Test of an Instrument to Assess the Extent to Which a Vocational Educational Program Is Either Competency-Based or Conventional. Final Report from September 1, 1984 to August 31, 1985.

ERIC Educational Resources Information Center

University of Central Florida, Orlando. Coll. of Education.

This report describes the production and pilot test of an assessment instrument for vocational education programs. The instrument was designed to be used following a site visit that includes a 30- to 45-minute interview with the program instructor and a 30-minute interview with one small group of students. Reliability and validity information was…

Development and exploratory analysis of the Neurorehabilitation Program Styles Survey.

PubMed

McCorkel, Beth A; Glueckauf, Robert L; Ecklund-Johnson, Eric P; Tomusk, Allison B; Trexler, Lance E; Diller, Leonard

2003-01-01

To develop a survey instrument that assesses implementation of key components of outpatient neurorehabilitation programs and test the capacity of this instrument to differentiate between rehabilitation approaches. The Neurorehabilitation Program Styles Survey (NPSS) was administered to 18 outpatient facilities: 10 specialized and 8 discipline-specific outpatient neurorehabilitation programs. Scores were compared between types of programs using independent samples t tests. The NPSS showed good reliability and contrasted groups validity, significantly differentiating between types of programs. The NPSS holds considerable promise as a tool for distinguishing among different types of brain injury programs, and for assessing the differential effectiveness of specialized versus discipline-specific outpatient brain rehabilitation programs. Future research on the NPSS will assess the stability of the instrument over time, its content validity, and capacity to differentiate the full continuum of neurorehabilitation programs.

Assessing Reliability of Student Ratings of Advisor: A Comparison of Univariate and Multivariate Generalizability Approaches.

ERIC Educational Resources Information Center

Sun, Anji; Valiga, Michael J.

In this study, the reliability of the American College Testing (ACT) Program's "Survey of Academic Advising" (SAA) was examined using both univariate and multivariate generalizability theory approaches. The primary purpose of the study was to compare the results of three generalizability theory models (a random univariate model, a mixed…

Test-Retest Reliability of the Parent Behavior Importance Questionnaire-Revised and the Parent Behavior Frequency Questionnaire-Revised

ERIC Educational Resources Information Center

Mowder, Barbara A.; Shamah, Renee

2011-01-01

This study evaluated the test-retest reliability of two parenting measures: the Parent Behavior Importance Questionnaire-Revised (PBIQ-R) and Parent Behavior Frequency Questionnaire-Revised (PBFQ-R). These self-report parenting behavior assessment measures may be utilized as pre- and post-parent education program measures, with parents as well as…

Organizational readiness to change assessment (ORCA): Development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework

PubMed Central

Helfrich, Christian D; Li, Yu-Fang; Sharp, Nancy D; Sales, Anne E

2009-01-01

Background The Promoting Action on Research Implementation in Health Services, or PARIHS, framework is a theoretical framework widely promoted as a guide to implement evidence-based clinical practices. However, it has as yet no pool of validated measurement instruments that operationalize the constructs defined in the framework. The present article introduces an Organizational Readiness to Change Assessment instrument (ORCA), organized according to the core elements and sub-elements of the PARIHS framework, and reports on initial validation. Methods We conducted scale reliability and factor analyses on cross-sectional, secondary data from three quality improvement projects (n = 80) conducted in the Veterans Health Administration. In each project, identical 77-item ORCA instruments were administered to one or more staff from each facility involved in quality improvement projects. Items were organized into 19 subscales and three primary scales corresponding to the core elements of the PARIHS framework: (1) Strength and extent of evidence for the clinical practice changes represented by the QI program, assessed with four subscales, (2) Quality of the organizational context for the QI program, assessed with six subscales, and (3) Capacity for internal facilitation of the QI program, assessed with nine subscales. Results Cronbach's alpha for scale reliability were 0.74, 0.85 and 0.95 for the evidence, context and facilitation scales, respectively. The evidence scale and its three constituent subscales failed to meet the conventional threshold of 0.80 for reliability, and three individual items were eliminated from evidence subscales following reliability testing. In exploratory factor analysis, three factors were retained. Seven of the nine facilitation subscales loaded onto the first factor; five of the six context subscales loaded onto the second factor; and the three evidence subscales loaded on the third factor. Two subscales failed to load significantly on any factor. One measured resources in general (from the context scale), and one clinical champion role (from the facilitation scale). Conclusion We find general support for the reliability and factor structure of the ORCA. However, there was poor reliability among measures of evidence, and factor analysis results for measures of general resources and clinical champion role did not conform to the PARIHS framework. Additional validation is needed, including criterion validation. PMID:19594942

Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework.

PubMed

Helfrich, Christian D; Li, Yu-Fang; Sharp, Nancy D; Sales, Anne E

2009-07-14

The Promoting Action on Research Implementation in Health Services, or PARIHS, framework is a theoretical framework widely promoted as a guide to implement evidence-based clinical practices. However, it has as yet no pool of validated measurement instruments that operationalize the constructs defined in the framework. The present article introduces an Organizational Readiness to Change Assessment instrument (ORCA), organized according to the core elements and sub-elements of the PARIHS framework, and reports on initial validation. We conducted scale reliability and factor analyses on cross-sectional, secondary data from three quality improvement projects (n = 80) conducted in the Veterans Health Administration. In each project, identical 77-item ORCA instruments were administered to one or more staff from each facility involved in quality improvement projects. Items were organized into 19 subscales and three primary scales corresponding to the core elements of the PARIHS framework: (1) Strength and extent of evidence for the clinical practice changes represented by the QI program, assessed with four subscales, (2) Quality of the organizational context for the QI program, assessed with six subscales, and (3) Capacity for internal facilitation of the QI program, assessed with nine subscales. Cronbach's alpha for scale reliability were 0.74, 0.85 and 0.95 for the evidence, context and facilitation scales, respectively. The evidence scale and its three constituent subscales failed to meet the conventional threshold of 0.80 for reliability, and three individual items were eliminated from evidence subscales following reliability testing. In exploratory factor analysis, three factors were retained. Seven of the nine facilitation subscales loaded onto the first factor; five of the six context subscales loaded onto the second factor; and the three evidence subscales loaded on the third factor. Two subscales failed to load significantly on any factor. One measured resources in general (from the context scale), and one clinical champion role (from the facilitation scale). We find general support for the reliability and factor structure of the ORCA. However, there was poor reliability among measures of evidence, and factor analysis results for measures of general resources and clinical champion role did not conform to the PARIHS framework. Additional validation is needed, including criterion validation.

Development and Testing of a Nutrition, Food Safety, and Physical Activity Checklist for EFNEP and FSNE Adult Programs

ERIC Educational Resources Information Center

Bradford, Traliece; Serrano, Elena L.; Cox, Ruby H.; Lambur, Michael

2010-01-01

Objective: To develop and assess reliability and validity of the Nutrition, Food Safety, and Physical Activity Checklist to measure nutrition, food safety, and physical activity practices among adult Expanded Food and Nutrition Education Program (EFNEP) and Food Stamp Nutrition Education program (FSNE) participants. Methods: Test-retest…

Does a web-based feedback training program result in improved reliability in clinicians' ratings of the Global Assessment of Functioning (GAF) Scale?

PubMed

Støre-Valen, Jakob; Ryum, Truls; Pedersen, Geir A F; Pripp, Are H; Jose, Paul E; Karterud, Sigmund

2015-09-01

The Global Assessment of Functioning (GAF) Scale is used in routine clinical practice and research to estimate symptom and functional severity and longitudinal change. Concerns about poor interrater reliability have been raised, and the present study evaluated the effect of a Web-based GAF training program designed to improve interrater reliability in routine clinical practice. Clinicians rated up to 20 vignettes online, and received deviation scores as immediate feedback (i.e., own scores compared with expert raters) after each rating. Growth curves of absolute SD scores across the vignettes were modeled. A linear mixed effects model, using the clinician's deviation scores from expert raters as the dependent variable, indicated an improvement in reliability during training. Moderation by content of scale (symptoms; functioning), scale range (average; extreme), previous experience with GAF rating, profession, and postgraduate training were assessed. Training reduced deviation scores for inexperienced GAF raters, for individuals in clinical professions other than nursing and medicine, and for individuals with no postgraduate specialization. In addition, training was most beneficial for cases with average severity of symptoms compared with cases with extreme severity. The results support the use of Web-based training with feedback routines as a means to improve the reliability of GAF ratings performed by clinicians in mental health practice. These results especially pertain to clinicians in mental health practice who do not have a masters or doctoral degree. (c) 2015 APA, all rights reserved.

Towards an operational definition of pharmacy clinical competency

NASA Astrophysics Data System (ADS)

Douglas, Charles Allen

The scope of pharmacy practice and the training of future pharmacists have undergone a strategic shift over the last few decades. The pharmacy profession recognizes greater pharmacist involvement in patient care activities. Towards this strategic objective, pharmacy schools are training future pharmacists to meet these new clinical demands. Pharmacy students have clerkships called Advanced Pharmacy Practice Experiences (APPEs), and these clerkships account for 30% of the professional curriculum. APPEs provide the only opportunity for students to refine clinical skills under the guidance of an experienced pharmacist. Nationwide, schools of pharmacy need to evaluate whether students have successfully completed APPEs and are ready treat patients. Schools are left to their own devices to develop assessment programs that demonstrate to the public and regulatory agencies, students are clinically competent prior to graduation. There is no widely accepted method to evaluate whether these assessment programs actually discriminate between the competent and non-competent students. The central purpose of this study is to demonstrate a rigorous method to evaluate the validity and reliability of APPE assessment programs. The method introduced in this study is applicable to a wide variety of assessment programs. To illustrate this method, the study evaluated new performance criteria with a novel rating scale. The study had two main phases. In the first phase, a Delphi panel was created to bring together expert opinions. Pharmacy schools nominated exceptional preceptors to join a Delphi panel. Delphi is a method to achieve agreement of complex issues among experts. The principal researcher recruited preceptors representing a variety of practice settings and geographical regions. The Delphi panel evaluated and refined the new performance criteria. In the second phase, the study produced a novel set of video vignettes that portrayed student performances based on recommendations of an expert panel. Pharmacy preceptors assessed the performances with the new performance criteria. Estimates of reliability and accuracy from preceptors' assessments can be used to establish benchmarks for future comparisons. Findings from the first phase suggested preceptors held a unique perspective, where APPE assessments are based in relevance to clinical activities. The second phase analyzed assessment results from pharmacy preceptors who watched the video simulations. Reliability results were higher for non-randomized compared to randomized video simulations. Accuracy results showed preceptors more readily identified high and low student performances compared to average students. These results indicated the need for pharmacy preceptor training in performance assessment. The study illustrated a rigorous method to evaluate the validity and reliability of APPE assessment instruments.

NASA Electronic Parts and Packaging Program

NASA Technical Reports Server (NTRS)

Kayali, Sammy

2000-01-01

NEPP program objectives are to: (1) Access the reliability of newly available electronic parts and packaging technologies for usage on NASA projects through validations, assessments, and characterizations, and the development of test methods/tools; (2)Expedite infusion paths for advanced (emerging) electronic parts and packaging technologies by evaluations of readiness for manufacturability and project usage consideration; (3) Provide NASA projects with technology selection, application, and validation guidelines for electronic parts and packaging hardware and processes; nd (4) Retain and disseminate electronic parts and packaging quality assurance, reliability validations, tools, and availability information to the NASA community.

An Interprofessional Program Evaluation Case Study: Utilizing Multiple Measures To Assess What Matters. AIR 1997 Annual Forum Paper.

ERIC Educational Resources Information Center

Delaney, Anne Marie

This paper reviews the first two years of a model program-evaluation case study which is intended to show: (1) how program evaluation can contribute to academic and professional degree programs; (2) how qualitative and quantitative techniques can be used to produce reliable measures for evaluation studies; and (3) how the role of the institutional…

FY11 Facility Assessment Study for Aeronautics Test Program

NASA Technical Reports Server (NTRS)

Loboda, John A.; Sydnor, George H.

2013-01-01

This paper presents the approach and results for the Aeronautics Test Program (ATP) FY11 Facility Assessment Project. ATP commissioned assessments in FY07 and FY11 to aid in the understanding of the current condition and reliability of its facilities and their ability to meet current and future (five year horizon) test requirements. The principle output of the assessment was a database of facility unique, prioritized investments projects with budgetary cost estimates. This database was also used to identify trends for the condition of facility systems.

Exploring the Gap between Evidence and Judgement: Using Video Vignettes for Practice-based Assessment of Physiotherapy Undergraduates.

ERIC Educational Resources Information Center

Cross, Vinette; Hicks, Carolyn; Barwell, Fred

2001-01-01

Using videos of physiotherapy students, compared two assessment forms for validity and reliability (the first currently used by an academic program and the second developed from practitioners' perceptions of competence). Also investigated effects of training on assessment decisions. Found wide differences in individual ability to assess students…

The Future Value of Serious Games for Assessment: Where Do We Go Now?

ERIC Educational Resources Information Center

de Klerk, Sebastiaan; Kato, Pamela M.

2017-01-01

Game-based assessments will most likely be an increasing part of testing programs in future generations because they provide promising possibilities for more valid and reliable measurement of students' skills as compared to the traditional methods of assessment like paper-and-pencil tests or performance-based assessments. The current status of…

A Multi-Peer Assessment Platform for Programming Language Learning: Considering Group Non-Consensus and Personal Radicalness

ERIC Educational Resources Information Center

Wang, Yanqing; Liang, Yaowen; Liu, Luning; Liu, Ying

2016-01-01

Multi-peer assessment has often been used by teachers to reduce personal bias and make the assessment more reliable. This study reviews the design and development of multi-peer assessment systems that detect and solve two common issues in such systems: non-consensus among group members and personal radicalness in some assessments. A multi-peer…

Field Demonstration of Multi-Sensor Technology for Condition Assessment of Wastewater Collection Systems (Abstract)

EPA Science Inventory

The purpose of the field demonstration program is to gather technically reliable cost and performance information on selected condition assessment technologies under defined field conditions. The selected technologies include zoom camera, focused electrode leak location (FELL), ...

Automatic Fare Collection Equipment, Reliability and Maintainability Assessment Plan for Urban Rail Transit Properties

DOT National Transportation Integrated Search

1981-03-01

This project was conducted as part of UMTA's Rail Transit Fare Collection Program developed by the Transportation Systems Center of the U.S. Department of Transportation. The report presents a generalized survey methodology for conducting assessments...

New International Program to Asses the Reliability of Emerging Nondestructive Techniques (PARENT)

DOE Office of Scientific and Technical Information (OSTI.GOV)

Prokofiev, Iouri; Cumblidge, Stephen E.; Csontos, Aladar A.

2013-01-25

The Nuclear Regulatory Commission (NRC) established the Program to Assess the Reliability of Emerging Nondestructive Techniques (PARENT) to follow on from the successful Program for the Inspection of Nickel alloy Components (PINC). The goal of the PARENT is to conduct a confirmatory assessment of the reliability of nondestructive evaluation (NDE) techniques for detecting and sizing primary water stress corrosion cracks (PWSCC) and applying the lessons learned from PINC to a series of round-robin tests. These open and blind round-robin tests will comprise a new set of typical pressure boundary components including dissimilar metal welds (DMWs) and bottom-mounted instrumentation penetrations. Openmore » round-robin tests will engage research and industry teams worldwide to investigate and demonstrate the reliability of emerging NDE techniques to detect and size flaws with a wide range of lengths, depths, orientations, and locations. Blind round-robin tests will utilize various testing organizations, whose inspectors and procedures are certified by the standards for the nuclear industry in their respective countries, to investigate the ability of established NDE techniques to detect and size flaws whose characteristics range from relatively easy to very difficult for detection and sizing. Blind and open round-robin testing started in late 2011 and early 2012, respectively. This paper will present the work scope with reports on progress, NDE methods evaluated, and project timeline for PARENT.« less

A measurement tool to assess culture change regarding patient safety in hospital obstetrical units.

PubMed

Kenneth Milne, J; Bendaly, Nicole; Bendaly, Leslie; Worsley, Jill; FitzGerald, John; Nisker, Jeff

2010-06-01

Clinical error in acute care hospitals can only be addressed by developing a culture of safety. We sought to develop a cultural assessment survey (CAS) to assess patient safety culture change in obstetrical units. Interview prompts and a preliminary questionnaire were developed through a literature review of patient safety and "high reliability organizations," followed by interviews with members of the Managing Obstetrical Risk Efficiently (MOREOB) Program of the Society of Obstetricians and Gynaecologists of Canada. Three hundred preliminary questionnaires were mailed, and 21 interviews and 9 focus groups were conducted with the staff of 11 hospital sites participating in the program. To pilot test the CAS, 350 surveys were mailed to staff in participating hospitals, and interviews were conducted with seven nurses and five physicians who had completed the survey. Reliability analysis was conducted on four units that completed the CAS prior to and following the implementation of the first MOREOB module. Nineteen values and 105 behaviours, practices, and perceptions relating to patient safety were identified and included in the preliminary questionnaire, of which 143 of 300 (47.4%) were returned. Among the 220 cultural assessment surveys returned (62.9%), six cultural scales emerged: (1) patient safety as everyone's priority; (2) teamwork; (3) valuing individuals; (4) open communication; (5) learning; and (6) empowering individuals. The reliability analysis found all six scales to have internal reliability (Cronbach alpha), ranging from 0.72 (open communication) to 0.84 (valuing individuals). The CAS developed for this study may enable obstetrical units to assess change in patient safety culture.

Validity of instruments to assess students' travel and pedestrian safety.

PubMed

Mendoza, Jason A; Watson, Kathy; Baranowski, Tom; Nicklas, Theresa A; Uscanga, Doris K; Hanfling, Marcus J

2010-05-18

Safe Routes to School (SRTS) programs are designed to make walking and bicycling to school safe and accessible for children. Despite their growing popularity, few validated measures exist for assessing important outcomes such as type of student transport or pedestrian safety behaviors. This research validated the SRTS school travel survey and a pedestrian safety behavior checklist. Fourth grade students completed a brief written survey on how they got to school that day with set responses. Test-retest reliability was obtained 3-4 hours apart. Convergent validity of the SRTS travel survey was assessed by comparison to parents' report. For the measure of pedestrian safety behavior, 10 research assistants observed 29 students at a school intersection for completion of 8 selected pedestrian safety behaviors. Reliability was determined in two ways: correlations between the research assistants' ratings to that of the Principal Investigator (PI) and intraclass correlations (ICC) across research assistant ratings. The SRTS travel survey had high test-retest reliability (kappa = 0.97, n = 96, p < 0.001) and convergent validity (kappa = 0.87, n = 81, p < 0.001). The pedestrian safety behavior checklist had moderate reliability across research assistants' ratings (ICC = 0.48) and moderate correlation with the PI (r = 0.55, p = < 0.01). When two raters simultaneously used the instrument, the ICC increased to 0.65. Overall percent agreement (91%), sensitivity (85%) and specificity (83%) were acceptable. These validated instruments can be used to assess SRTS programs. The pedestrian safety behavior checklist may benefit from further formative work.

Assessment of communication, professionalism, and surgical skills in an objective structured performance-related examination (OSPRE): a psychometric study.

PubMed

Ponton-Carss, Alicia; Hutchison, Carol; Violato, Claudio

2011-10-01

The purpose of this study was to investigate the reliability and validity of a performance assessment of communication, professionalism, and surgical skills competencies for surgery residents. Fourteen residents from the general surgery program of the University of Calgary were assessed in 7 surgical simulation stations that included communication and professionalism skills. The internal consistency reliability of the checklists and global rating scales combined was adequate for communication (α = .75-.92) and surgical skills (α = .86-.96), but not for professionalism (α = 0). There was evidence of validity as surgical skills performance improved as a function of postgraduate year level but not for the professionalism checklist. Surgical skills and communication correlated in the 2 stations assessed (r = .55 and .57; P < .05). There is evidence for both reliability and validity for simultaneously assessing surgical skills and communication skills. Further instrument development is required to assess professionalism in a structured examination context. Copyright © 2011 Elsevier Inc. All rights reserved.

Design of fuel cell powered data centers for sufficient reliability and availability

NASA Astrophysics Data System (ADS)

Ritchie, Alexa J.; Brouwer, Jacob

2018-04-01

It is challenging to design a sufficiently reliable fuel cell electrical system for use in data centers, which require 99.9999% uptime. Such a system could lower emissions and increase data center efficiency, but the reliability and availability of such a system must be analyzed and understood. Currently, extensive backup equipment is used to ensure electricity availability. The proposed design alternative uses multiple fuel cell systems each supporting a small number of servers to eliminate backup power equipment provided the fuel cell design has sufficient reliability and availability. Potential system designs are explored for the entire data center and for individual fuel cells. Reliability block diagram analysis of the fuel cell systems was accomplished to understand the reliability of the systems without repair or redundant technologies. From this analysis, it was apparent that redundant components would be necessary. A program was written in MATLAB to show that the desired system reliability could be achieved by a combination of parallel components, regardless of the number of additional components needed. Having shown that the desired reliability was achievable through some combination of components, a dynamic programming analysis was undertaken to assess the ideal allocation of parallel components.

The Shuttle processing contractors (SPC) reliability program at the Kennedy Space Center - The real world

NASA Astrophysics Data System (ADS)

McCrea, Terry

The Shuttle Processing Contract (SPC) workforce consists of Lockheed Space Operations Co. as prime contractor, with Grumman, Thiokol Corporation, and Johnson Controls World Services as subcontractors. During the design phase, reliability engineering is instrumental in influencing the development of systems that meet the Shuttle fail-safe program requirements. Reliability engineers accomplish this objective by performing FMEA (failure modes and effects analysis) to identify potential single failure points. When technology, time, or resources do not permit a redesign to eliminate a single failure point, the single failure point information is formatted into a change request and presented to senior management of SPC and NASA for risk acceptance. In parallel with the FMEA, safety engineering conducts a hazard analysis to assure that potential hazards to personnel are assessed. The combined effort (FMEA and hazard analysis) is published as a system assurance analysis. Special ground rules and techniques are developed to perform and present the analysis. The reliability program at KSC is vigorously pursued, and has been extremely successful. The ground support equipment and facilities used to launch and land the Space Shuttle maintain an excellent reliability record.

Development of KSC program for investigating and generating field failure rates. Volume 2: Recommended format for reliability handbook for ground support equipment

NASA Technical Reports Server (NTRS)

Bloomquist, C. E.; Kallmeyer, R. H.

1972-01-01

Field failure rates and confidence factors are presented for 88 identifiable components of the ground support equipment at the John F. Kennedy Space Center. For most of these, supplementary information regarding failure mode and cause is tabulated. Complete reliability assessments are included for three systems, eight subsystems, and nine generic piece-part classifications. Procedures for updating or augmenting the reliability results presented in this handbook are also included.

The use and reliability of SymNose for quantitative measurement of the nose and lip in unilateral cleft lip and palate patients.

PubMed

Mosmuller, David; Tan, Robin; Mulder, Frans; Bachour, Yara; de Vet, Henrica; Don Griot, Peter

2016-10-01

It is essential to have a reliable assessment method in order to compare the results of cleft lip and palate surgery. In this study the computer-based program SymNose, a method for quantitative assessment of the nose and lip, will be assessed on usability and reliability. The symmetry of the nose and lip was measured twice in 50 six-year-old complete and incomplete unilateral cleft lip and palate patients by four observers. For the frontal view the asymmetry level of the nose and upper lip were evaluated and for the basal view the asymmetry level of the nose and nostrils were evaluated. A mean inter-observer reliability when tracing each image once or twice was 0.70 and 0.75, respectively. Tracing the photographs with 2 observers and 4 observers gave a mean inter-observer score of 0.86 and 0.92, respectively. The mean intra-observer reliability varied between 0.80 and 0.84. SymNose is a practical and reliable tool for the retrospective assessment of large caseloads of 2D photographs of cleft patients for research purposes. Moderate to high single inter-observer reliability was found. For future research with SymNose reliable outcomes can be achieved by using the average outcomes of single tracings of two observers. Copyright © 2016 European Association for Cranio-Maxillo-Facial Surgery. Published by Elsevier Ltd. All rights reserved.

Assessing I-Grid(TM) web-based monitoring for power quality and reliability benchmarking

DOE Office of Scientific and Technical Information (OSTI.GOV)

Divan, Deepak; Brumsickle, William; Eto, Joseph

2003-04-30

This paper presents preliminary findings from DOEs pilot program. The results show how a web-based monitoring system can form the basis for aggregation of data and correlation and benchmarking across broad geographical lines. A longer report describes additional findings from the pilot, including impacts of power quality and reliability on customers operations [Divan, Brumsickle, Eto 2003].

Training and quality assurance with the Structured Clinical Interview for DSM-IV (SCID-I/P).

PubMed

Ventura, J; Liberman, R P; Green, M F; Shaner, A; Mintz, J

1998-06-15

Accuracy in psychiatric diagnosis is critical for evaluating the suitability of the subjects for entry into research protocols and for establishing comparability of findings across study sites. However, training programs in the use of diagnostic instruments for research projects are not well systematized. Furthermore, little information has been published on the maintenance of interrater reliability of diagnostic assessments. At the UCLA Research Center for Major Mental Illnesses, a Training and Quality Assurance Program for SCID interviewers was used to evaluate interrater reliability and diagnostic accuracy. Although clinically experienced interviewers achieved better interrater reliability and overall diagnostic accuracy than neophyte interviewers, both groups were able to achieve and maintain high levels of interrater reliability, diagnostic accuracy, and interviewer skill. At the first quality assurance check after training, there were no significant differences between experienced and neophyte interviewers in interrater reliability or diagnostic accuracy. Standardization of training and quality assurance procedures within and across research projects may make research findings from study sites more comparable.

Development of an instrument to measure medical students' perceptions of the assessment environment: initial validation.

PubMed

Sim, Joong Hiong; Tong, Wen Ting; Hong, Wei-Han; Vadivelu, Jamuna; Hassan, Hamimah

2015-01-01

Assessment environment, synonymous with climate or atmosphere, is multifaceted. Although there are valid and reliable instruments for measuring the educational environment, there is no validated instrument for measuring the assessment environment in medical programs. This study aimed to develop an instrument for measuring students' perceptions of the assessment environment in an undergraduate medical program and to examine the psychometric properties of the new instrument. The Assessment Environment Questionnaire (AEQ), a 40-item, four-point (1=Strongly Disagree to 4=Strongly Agree) Likert scale instrument designed by the authors, was administered to medical undergraduates from the authors' institution. The response rate was 626/794 (78.84%). To establish construct validity, exploratory factor analysis (EFA) with principal component analysis and varimax rotation was conducted. To examine the internal consistency reliability of the instrument, Cronbach's α was computed. Mean scores for the entire AEQ and for each factor/subscale were calculated. Mean AEQ scores of students from different academic years and sex were examined. Six hundred and eleven completed questionnaires were analysed. EFA extracted four factors: feedback mechanism (seven items), learning and performance (five items), information on assessment (five items), and assessment system/procedure (three items), which together explained 56.72% of the variance. Based on the four extracted factors/subscales, the AEQ was reduced to 20 items. Cronbach's α for the 20-item AEQ was 0.89, whereas Cronbach's α for the four factors/subscales ranged from 0.71 to 0.87. Mean score for the AEQ was 2.68/4.00. The factor/subscale of 'feedback mechanism' recorded the lowest mean (2.39/4.00), whereas the factor/subscale of 'assessment system/procedure' scored the highest mean (2.92/4.00). Significant differences were found among the AEQ scores of students from different academic years. The AEQ is a valid and reliable instrument. Initial validation supports its use to measure students' perceptions of the assessment environment in an undergraduate medical program.

Reliability of a Market Basket Assessment Tool (MBAT) for Use in SNAP-Ed Healthy Retail Initiatives.

PubMed

Misyak, Sarah A; Hedrick, Valisa E; Pudney, Ellen; Serrano, Elena L; Farris, Alisha R

2018-05-01

To evaluate the reliability of the Market Basket Assessment Tool (MBAT) for assessing the availability of fruits and vegetables, low-fat or nonfat dairy and eggs, lean meats, whole-grain products, and seeds, beans, and nuts in Supplemental Nutrition Assistance Program-authorized retail environments. Different trained raters used the MBAT simultaneously at 14 retail environments to measure interrater reliability. Raters returned to 12 retail environments (85.7%) 1 week later to measure test-retest reliability. Data were analyzed using paired-sample t tests and correlations. No significant differences were found for interrater reliability or test-retest reliability for individual categories (mean differences, 0.0 to 0.3 ± 0.2 points) or total score (mean difference, 0.5 ± 0.4 points and (mean differences, 0.0 to 0.3 ± 0.3 points) or total score (mean difference, 0.8 ± 0.4 points), respectively. Future steps include validation of the MBAT. A low-burden tool can facilitate evaluation of efforts to promote healthful foods in retail environments. Copyright © 2018 Society for Nutrition Education and Behavior. Published by Elsevier Inc. All rights reserved.

Measuring the Process and Quality of Informed Consent for Clinical Research: Development and Testing

PubMed Central

Cohn, Elizabeth Gross; Jia, Haomiao; Smith, Winifred Chapman; Erwin, Katherine; Larson, Elaine L.

2013-01-01

Purpose/Objectives To develop and assess the reliability and validity of an observational instrument, the Process and Quality of Informed Consent (P-QIC). Design A pilot study of the psychometrics of a tool designed to measure the quality and process of the informed consent encounter in clinical research. The study used professionally filmed, simulated consent encounters designed to vary in process and quality. Setting A major urban teaching hospital in the northeastern region of the United States. Sample 63 students enrolled in health-related programs participated in psychometric testing, 16 students participated in test-retest reliability, and 5 investigator-participant dyads were observed for the actual consent encounters. Methods For reliability and validity testing, students watched and rated videotaped simulations of four consent encounters intentionally varied in process and content and rated them with the proposed instrument. Test-retest reliability was established by raters watching the videotaped simulations twice. Inter-rater reliability was demonstrated by two simultaneous but independent raters observing an actual consent encounter. Main Research Variables The essential elements of information and communication for informed consent. Findings The initial testing of the P-QIC demonstrated reliable and valid psychometric properties in both the simulated standardized consent encounters and actual consent encounters in the hospital setting. Conclusions The P-QIC is an easy-to-use observational tool that provides a quick assessment of the areas of strength and areas that need improvement in a consent encounter. It can be used in the initial trainings of new investigators or consent administrators and in ongoing programs of improvement for informed consent. Implications for Nursing The development of a validated observational instrument will allow investigators to assess the consent process more accurately and evaluate strategies designed to improve it. PMID:21708532

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention.

PubMed

Smith, Justin D; Dishion, Thomas J; Brown, Kimbree; Ramos, Karina; Knoble, Naomi B; Shaw, Daniel S; Wilson, Melvin N

2016-01-01

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist's delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist's selection of intervention strategies tailored to the family's context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

PubMed Central

Smith, Justin D.; Dishion, Thomas J.; Brown, Kimbree; Ramos, Karina; Knoble, Naomi B.; Shaw, Daniel S.; Wilson, Melvin N.

2015-01-01

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist's delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist's selection of intervention strategies tailored to the family's context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance. PMID:26271300

A Rapid Assessment Tool for affirming good practice in midwifery education programming.

PubMed

Fullerton, Judith T; Johnson, Peter; Lobe, Erika; Myint, Khine Haymar; Aung, Nan Nan; Moe, Thida; Linn, Nay Aung

2016-03-01

to design a criterion-referenced assessment tool that could be used globally in a rapid assessment of good practices and bottlenecks in midwifery education programs. a standard tool development process was followed, to generate standards and reference criteria; followed by external review and field testing to document psychometric properties. review of standards and scoring criteria were conducted by stakeholders around the globe. Field testing of the tool was conducted in Myanmar. eleven of Myanmar׳s 22 midwifery education programs participated in the assessment. the clinimetric tool was demonstrated to have content validity and high inter-rater reliability in use. a globally validated tool, and accompanying user guide and handbook are now available for conducting rapid assessments of compliance with good practice criteria in midwifery education programming. Copyright © 2016 The Authors. Published by Elsevier Ltd.. All rights reserved.

Automated Portable Test System (APTS) - A performance envelope assessment tool

NASA Technical Reports Server (NTRS)

Kennedy, R. S.; Dunlap, W. P.; Jones, M. B.; Wilkes, R. L.; Bittner, A. C., Jr.

1985-01-01

The reliability and stability of microcomputer-based psychological tests are evaluated. The hardware, test programs, and system control of the Automated Portable Test System, which assesses human performance and subjective status, are described. Subjects were administered 11 pen-and-pencil and microcomputer-based tests for 10 sessions. The data reveal that nine of the 10 tests stabilized by the third administration; inertial correlations were high and consistent. It is noted that the microcomputer-based tests display good psychometric properties in terms of differential stability and reliability.

Designing Computer-Based Assessments: Multidisciplinary Findings and Student Perspectives

ERIC Educational Resources Information Center

Dembitzer, Leah; Zelikovitz, Sarah; Kettler, Ryan J.

2017-01-01

A partnership was created between psychologists and computer programmers to develop a computer-based assessment program. Psychometric concerns of accessibility, reliability, and validity were juxtaposed with core development concepts of usability and user-centric design. Phases of development were iterative, with evaluation phases alternating with…

Sandia National Laboratories: Fabrication, Testing and Validation

Science.gov Websites

; Technology Defense Systems & Assessments About Defense Systems & Assessments Program Areas safe, secure, reliable, and can fully support the Nation's deterrence policy. Employing only the most support of this mission, Sandia National Laboratories has a significant role in advancing the "state

Field Demonstration of Electro-Scan Defect Location Technology for Condition Assessment of Wastewater Collection Systems

EPA Science Inventory

The purpose of the field demonstration program is to gather technically reliable cost and performance information on selected condition assessment technologies under defined field conditions. The selected technologies include zoom camera, electro-scan (FELL-41), and a multi-sens...

Assessing the Culture of Residency Using the C - Change Resident Survey: Validity Evidence in 34 U.S. Residency Programs.

PubMed

Pololi, Linda H; Evans, Arthur T; Civian, Janet T; Shea, Sandy; Brennan, Robert T

2017-07-01

A practical instrument is needed to reliably measure the clinical learning environment and professionalism for residents. To develop and present evidence of validity of an instrument to assess the culture of residency programs and the clinical learning environment. During 2014-2015, we surveyed residents using the C - Change Resident Survey to assess residents' perceptions of the culture in their programs. Residents in all years of training in 34 programs in internal medicine, pediatrics, and general surgery in 14 geographically diverse public and private academic health systems. The C - Change Resident Survey assessed residents' perceptions of 13 dimensions of the culture: Vitality, Self-Efficacy, Institutional Support, Relationships/Inclusion, Values Alignment, Ethical/Moral Distress, Respect, Mentoring, Work-Life Integration, Gender Equity, Racial/Ethnic Minority Equity, and self-assessed Competencies. We measured the internal reliability of each of the 13 dimensions and evaluated response process, content validity, and construct-related evidence validity by assessing relationships predicted by our conceptual model and prior research. We also assessed whether the measurements were sensitive to differences in specialty and across institutions. A total of 1708 residents completed the survey [internal medicine: n = 956, pediatrics: n = 411, general surgery: n = 311 (51% women; 16% underrepresented in medicine minority)], with a response rate of 70% (range across programs, 51-87%). Internal consistency of each dimension was high (Cronbach α: 0.73-0.90). The instrument was able to detect significant differences in the learning environment across programs and sites. Evidence of validity was supported by a good response process and the demonstration of several relationships predicted by our conceptual model. The C - Change Resident Survey assesses the clinical learning environment for residents, and we encourage further study of validity in different contexts. Results could be used to facilitate and monitor improvements in the clinical learning environment and resident well-being.

Performance Assessments in Science: Hands-On Tasks and Scoring Guides.

ERIC Educational Resources Information Center

Stecher, Brian M.; Klein, Stephen P.

In 1992, RAND received a grant from the National Science Foundation to study the technical quality of performance assessments in science and to evaluate their feasibility for use in large-scale testing programs. The specific goals of the project were to assess the reliability and validity of hands-on science testing and to investigate the cost and…

Interim reliability-evaluation program: analysis of the Browns Ferry, Unit 1, nuclear plant. Appendix C - sequence quantification

DOE Office of Scientific and Technical Information (OSTI.GOV)

Mays, S.E.; Poloski, J.P.; Sullivan, W.H.

1982-07-01

This report describes a risk study of the Browns Ferry, Unit 1, nuclear plant. The study is one of four such studies sponsored by the NRC Office of Research, Division of Risk Assessment, as part of its Interim Reliability Evaluation Program (IREP), Phase II. This report is contained in four volumes: a main report and three appendixes. Appendix C generally describes the methods used to estimate accident sequence frequency values. Information is presented concerning the approach, example collection, failure data, candidate dominant sequences, uncertainty analysis, and sensitivity analysis.

Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments.

PubMed

Ginsburg, Shiphra; Eva, Kevin; Regehr, Glenn

2013-10-01

Although scores on in-training evaluation reports (ITERs) are often criticized for poor reliability and validity, ITER comments may yield valuable information. The authors assessed across-rotation reliability of ITER scores in one internal medicine program, ability of ITER scores and comments to predict postgraduate year three (PGY3) performance, and reliability and incremental predictive validity of attendings' analysis of written comments. Numeric and narrative data from the first two years of ITERs for one cohort of residents at the University of Toronto Faculty of Medicine (2009-2011) were assessed for reliability and predictive validity of third-year performance. Twenty-four faculty attendings rank-ordered comments (without scores) such that each resident was ranked by three faculty. Mean ITER scores and comment rankings were submitted to regression analyses; dependent variables were PGY3 ITER scores and program directors' rankings. Reliabilities of ITER scores across nine rotations for 63 residents were 0.53 for both postgraduate year one (PGY1) and postgraduate year two (PGY2). Interrater reliabilities across three attendings' rankings were 0.83 for PGY1 and 0.79 for PGY2. There were strong correlations between ITER scores and comments within each year (0.72 and 0.70). Regressions revealed that PGY1 and PGY2 ITER scores collectively explained 25% of variance in PGY3 scores and 46% of variance in PGY3 rankings. Comment rankings did not improve predictions. ITER scores across multiple rotations showed decent reliability and predictive validity. Comment ranks did not add to the predictive ability, but correlation analyses suggest that trainee performance can be measured through these comments.

Reliability and Failure in NASA Missions: Blunders, Normal Accidents, High Reliability, Bad Luck

NASA Technical Reports Server (NTRS)

Jones, Harry W.

2015-01-01

NASA emphasizes crew safety and system reliability but several unfortunate failures have occurred. The Apollo 1 fire was mistakenly unanticipated. After that tragedy, the Apollo program gave much more attention to safety. The Challenger accident revealed that NASA had neglected safety and that management underestimated the high risk of shuttle. Probabilistic Risk Assessment was adopted to provide more accurate failure probabilities for shuttle and other missions. NASA's "faster, better, cheaper" initiative and government procurement reform led to deliberately dismantling traditional reliability engineering. The Columbia tragedy and Mars mission failures followed. Failures can be attributed to blunders, normal accidents, or bad luck. Achieving high reliability is difficult but possible.

Sustained Implementation Support Scale: Validation of a Measure of Program Characteristics and Workplace Functioning for Sustained Program Implementation.

PubMed

Hodge, Lauren M; Turner, Karen M T; Sanders, Matthew R; Filus, Ania

2017-07-01

An evaluation measure of enablers and inhibitors to sustained evidence-based program (EBP) implementation may provide a useful tool to enhance organizations' capacity. This paper outlines preliminary validation of such a measure. An expert informant and consumer feedback approach was used to tailor constructs from two existing measures assessing key domains associated with sustained implementation. Validity and reliability were evaluated for an inventory composed of five subscales: Program benefits, Program burden, Workplace support, Workplace cohesion, and Leadership style. Exploratory and confirmatory factor analysis with a sample of 593 Triple P-Positive Parenting Program-practitioners led to a 28-item scale with good reliability and good convergent, discriminant, and predictive validity. Practitioners sustaining implementation at least 3 years post-training were more likely to have supervision/peer support, reported higher levels of program benefit, workplace support, and positive leadership style, and lower program burden compared to practitioners who were non-sustainers.

Reliability analysis and initial requirements for FC systems and stacks

NASA Astrophysics Data System (ADS)

Åström, K.; Fontell, E.; Virtanen, S.

In the year 2000 Wärtsilä Corporation started an R&D program to develop SOFC systems for CHP applications. The program aims to bring to the market highly efficient, clean and cost competitive fuel cell systems with rated power output in the range of 50-250 kW for distributed generation and marine applications. In the program Wärtsilä focuses on system integration and development. System reliability and availability are key issues determining the competitiveness of the SOFC technology. In Wärtsilä, methods have been implemented for analysing the system in respect to reliability and safety as well as for defining reliability requirements for system components. A fault tree representation is used as the basis for reliability prediction analysis. A dynamic simulation technique has been developed to allow for non-static properties in the fault tree logic modelling. Special emphasis has been placed on reliability analysis of the fuel cell stacks in the system. A method for assessing reliability and critical failure predictability requirements for fuel cell stacks in a system consisting of several stacks has been developed. The method is based on a qualitative model of the stack configuration where each stack can be in a functional, partially failed or critically failed state, each of the states having different failure rates and effects on the system behaviour. The main purpose of the method is to understand the effect of stack reliability, critical failure predictability and operating strategy on the system reliability and availability. An example configuration, consisting of 5 × 5 stacks (series of 5 sets of 5 parallel stacks) is analysed in respect to stack reliability requirements as a function of predictability of critical failures and Weibull shape factor of failure rate distributions.

The usefulness and reliability of fitness testing protocols for ice hockey players: a literature review.

PubMed

Nightingale, Steven C; Miller, Stuart; Turner, Anthony

2013-06-01

Ice hockey, like most sports, uses fitness testing to assess athletes. This study reviews the current commonly used fitness testing protocols for ice hockey players, discussing their predictive values and reliability. It also discusses a range of less commonly used measures and limitations in current testing protocols. The article concludes with a proposed testing program suitable for ice hockey players.

Questions for Online Surveys

ERIC Educational Resources Information Center

Ritter, Lois A., Ed.; Sue, Valerie M., Ed.

2007-01-01

The primary function of an evaluation is often to assess the degree of success of a program or to collect information that may be used to improve a program, product, or service. To meet an evaluation's goals and objectives by using an online survey, it is imperative that the questionnaire contain valid and reliable items asked about specific…

Media's Moral Messages: Assessing Perceptions of Moral Content in Television Programming

ERIC Educational Resources Information Center

Glover, Rebecca J.; Garmon, Lance C.; Hull, Darrell M.

2011-01-01

This study extends the examination of moral content in the media by exploring moral messages in television programming and viewer characteristics predictive of the ability to perceive such messages. Generalisability analyses confirmed the reliability of the Media's Moral Messages (MMM) rating form for analysing programme content and the existence…

Reliability analysis for digital adolescent idiopathic scoliosis measurements.

PubMed

Kuklo, Timothy R; Potter, Benjamin K; O'Brien, Michael F; Schroeder, Teresa M; Lenke, Lawrence G; Polly, David W

2005-04-01

Analysis of adolescent idiopathic scoliosis (AIS) requires a thorough clinical and radiographic evaluation to completely assess the three-dimensional deformity. Recently, these radiographic parameters have been analyzed for reliability and reproducibility following manual measurements; however, most of these parameters have not been analyzed with regard to digital measurements. The purpose of this study is to determine the intra- and interobserver reliability of common scoliosis radiographic parameters using a digital software measurement program. Thirty sets of preoperative (posteroanterior [PA], lateral, and side-bending [SB]) and postoperative (PA and lateral) radiographs were analyzed by three independent observers on two separate occasions using a software measurement program (PhDx, Albuquerque, NM). Coronal measures included main thoracic (MT) and thoracolumbar-lumbar (TL/L) Cobb, SB MT Cobb, MT and TL/L apical vertical translation (AVT), C7 to center sacral vertical line (CSVL), T1 tilt, LIV tilt, disk below lowest instrumented vertebra (LIV), coronal balance, and Risser, whereas sagittal measures included T2-T5, T5-T12, T2-T12, T10-L2, T12-S1, and sagittal balance. Analysis of variance for repeated measures or Cohen three-way kappa correlation coefficient analysis was performed as appropriate to calculate the intra- and interobserver reliability for each parameter. The majority of the radiographic parameters assessed demonstrated good or excellent intra- and interobserver reliability. The relationship of the LIV to the CSVL (intraobserver kappaa = 0.48-0.78, fair to excellent; interobserver kappaa = 0.34-0.41, fair to poor), interobserver measurement of AVT (rho = 0.49-0.73, low to good), Risser grade (intraobserver rho = 0.41-0.97, low to excellent; interobserver rho = 0.60-0.70, fair to good), intraobserver measurement of the angulation of the disk inferior to the LIV (rho = 0.53-0.88, fair to good), apical Nash-Moe vertebral rotation (intraobserver rho = 0.50-0.85, fair to good; interobserver rho = 0.53-0.59, fair), and especially regional thoracic kyphosis from T2 to T5 (intraobserver rho = 0.22-0.65, poor to fair; interobserver rho = 0.33-0.47, low) demonstrated lesser reliability. In general, preoperative measures demonstrated greater reliability than postoperative measures, and coronal angular measures were more reliable than sagittal measures. Most common radiographic parameters for AIS assessment demonstrated good or excellent reliability for digital measurement and can be recommended for routine clinical and academic use. Preoperative assessments and coronal measures may be more reliable than postoperative and sagittal measurements. The reliability of digital measurements will be increasingly important as digital radiographic viewing becomes commonplace.

Field Demonstration of Electro-Scan Defect Location Technology for Condition Assessment of Wastewater Collection Systems - Paper

EPA Science Inventory

A USEPA-sponsored field demonstration program was conducted to gather technically reliable cost and performance information on the electro-scan (FELL -41) pipeline condition assessment technology. Electro-scan technology can be used to estimate the magnitude and location of pote...

Data Sufficiency Assessment and Pumping Test Design for Groundwater Prediction Using Decision Theory and Genetic Algorithms

NASA Astrophysics Data System (ADS)

McPhee, J.; William, Y. W.

2005-12-01

This work presents a methodology for pumping test design based on the reliability requirements of a groundwater model. Reliability requirements take into consideration the application of the model results in groundwater management, expressed in this case as a multiobjective management model. The pumping test design is formulated as a mixed-integer nonlinear programming (MINLP) problem and solved using a combination of genetic algorithm (GA) and gradient-based optimization. Bayesian decision theory provides a formal framework for assessing the influence of parameter uncertainty over the reliability of the proposed pumping test. The proposed methodology is useful for selecting a robust design that will outperform all other candidate designs under most potential 'true' states of the system

A prospective study assessing agreement and reliability of a geriatric evaluation.

PubMed

Locatelli, Isabella; Monod, Stéfanie; Cornuz, Jacques; Büla, Christophe J; Senn, Nicolas

2017-07-19

The present study takes place within a geriatric program, aiming at improving the diagnosis and management of geriatric syndromes in primary care. Within this program it was of prime importance to be able to rely on a robust and reproducible geriatric consultation to use as a gold standard for evaluating a primary care brief assessment tool. The specific objective of the present study was thus assessing the agreement and reliability of a comprehensive geriatric consultation. The study was conducted at the outpatient clinic of the Service of Geriatric Medicine, University of Lausanne, Switzerland. All community-dwelling older persons aged 70 years and above were eligible. Patients were excluded if they hadn't a primary care physician, they were unable to speak French, or they were already assessed by a geriatrician within the last 12 months. A set of 9 geriatricians evaluated 20 patients. Each patient was assessed twice within a 2-month delay. Geriatric consultations were based on a structured evaluation process, leading to rating the following geriatric conditions: functional, cognitive, visual, and hearing impairment, mood disorders, risk of fall, osteoporosis, malnutrition, and urinary incontinence. Reliability and agreement estimates on each of these items were obtained using a three-way Intraclass Correlation and a three-way Observed Disagreement index. The latter allowed a decomposition of overall disagreement into disagreements due to each source of error variability (visit, rater and random). Agreement ranged between 0.62 and 0.85. For most domains, geriatrician-related error variability explained an important proportion of disagreement. Reliability ranged between 0 and 0.8. It was poor/moderate for visual impairment, malnutrition and risk of fall, and good/excellent for functional/cognitive/hearing impairment, osteoporosis, incontinence and mood disorders. Six out of nine items of the geriatric consultation described in this study (functional/cognitive/hearing impairment, osteoporosis, incontinence and mood disorders) present a good to excellent reliability and can safely be used as a reference (gold standard) to evaluate the diagnostic performance of a primary care brief assessment tool. More objective/significant measures are needed to improve reliability of malnutrition, visual impairment, and risk of fall assessment before they can serve as a safe gold standard of a primary care tool.

Validity of instruments to assess students' travel and pedestrian safety

PubMed Central

2010-01-01

Background Safe Routes to School (SRTS) programs are designed to make walking and bicycling to school safe and accessible for children. Despite their growing popularity, few validated measures exist for assessing important outcomes such as type of student transport or pedestrian safety behaviors. This research validated the SRTS school travel survey and a pedestrian safety behavior checklist. Methods Fourth grade students completed a brief written survey on how they got to school that day with set responses. Test-retest reliability was obtained 3-4 hours apart. Convergent validity of the SRTS travel survey was assessed by comparison to parents' report. For the measure of pedestrian safety behavior, 10 research assistants observed 29 students at a school intersection for completion of 8 selected pedestrian safety behaviors. Reliability was determined in two ways: correlations between the research assistants' ratings to that of the Principal Investigator (PI) and intraclass correlations (ICC) across research assistant ratings. Results The SRTS travel survey had high test-retest reliability (κ = 0.97, n = 96, p < 0.001) and convergent validity (κ = 0.87, n = 81, p < 0.001). The pedestrian safety behavior checklist had moderate reliability across research assistants' ratings (ICC = 0.48) and moderate correlation with the PI (r = 0.55, p =< 0.01). When two raters simultaneously used the instrument, the ICC increased to 0.65. Overall percent agreement (91%), sensitivity (85%) and specificity (83%) were acceptable. Conclusions These validated instruments can be used to assess SRTS programs. The pedestrian safety behavior checklist may benefit from further formative work. PMID:20482778

How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: use of an objective structured clinical examination (OSCE)

PubMed Central

Dwyer, Tim; Takahashi, Susan Glover; Hynes, Melissa Kennedy; Herold, Jodi; Wasserstein, David; Nousiainen, Markku; Ferguson, Peter; Wadey, Veronica; Murnaghan, M. Lucas; Leroux, Tim; Semple, John; Hodges, Brian; Ogilvie-Harris, Darrell

2014-01-01

Background Assessing residents’ understanding and application of the 6 intrinsic CanMEDS roles (communicator, professional, manager, collaborator, health advocate, scholar) is challenging for postgraduate medical educators. We hypothesized that an objective structured clinical examination (OSCE) designed to assess multiple intrinsic CanMEDS roles would be sufficiently reliable and valid. Methods The OSCE comprised 6 10-minute stations, each testing 2 intrinsic roles using case-based scenarios (with or without the use of standardized patients). Residents were evaluated using 5-point scales and an overall performance rating at each station. Concurrent validity was sought by correlation with in-training evaluation reports (ITERs) from the last 12 months and an ordinal ranking created by program directors (PDs). Results Twenty-five residents from postgraduate years (PGY) 0, 3 and 5 participated. The interstation reliability for total test scores (percent) was 0.87, while reliability for each of the communicator, collaborator, manager and professional roles was greater than 0.8. Total test scores, individual station scores and individual CanMEDS role scores all showed a significant effect by PGY level. Analysis of the PD rankings of intrinsic roles demonstrated a high correlation with the OSCE role scores. A correlation was seen between ITER and OSCE for the communicator role, while the ITER medical expert and total scores highly correlated with the communicator, manager and professional OSCE scores. Conclusion An OSCE designed to assess the intrinsic CanMEDS roles was sufficiently valid and reliable for regular use in an orthopedic residency program. PMID:25078926

Improving Nutrition and Physical Activity Policies in Afterschool Programs: Results from a Group-Randomized Controlled Trial

PubMed Central

Kenney, Erica L.; Giles, Catherine M.; deBlois, Madeleine E.; Gortmaker, Steven L.; Chinfatt, Sherene; Cradock, Angie L.

2017-01-01

OBJECTIVE Afterschool programs can be health-promoting environments for children. Written policies positively influence nutrition and physical activity (PA) environments, but effective strategies for building staff capacity to write such policies have not been evaluated. This study measures the comprehensiveness of written nutrition, PA, and screen time policies in afterschool programs and assesses impact of the Out of School Nutrition and Physical Activity (OSNAP) intervention on key policies. METHODS Twenty afterschool programs in Boston, MA participated in a group-randomized, controlled trial from September 2010 to June 2011. Intervention program staff attended learning collaboratives focused on practice and policy change. The Out-of-School Time (OST) Policy Assessment Index evaluated written policies. Inter-rater reliability and construct validity of the measure and impact of the intervention on written policies were assessed. RESULTS The measure demonstrated moderate to excellent inter-rater reliability (Spearman’s r=0.53 to 0.97) and construct validity. OSNAP was associated with significant increases in standards-based policy statements surrounding snacks (+2.6, p=0.003), beverages (+2.3, p=0.008), screen time (+0.8, p=0.046), family communication (+2.2, p=0.002), and a summary index of OSNAP goals (+3.3, p=0.02). CONCLUSIONS OSNAP demonstrated success in building staff capacity to write health-promoting policy statements. Future research should focus on determining policy change impact on practices. PMID:24941286

Assessing Program Learning Objectives to Improve Undergraduate Physics Education

NASA Astrophysics Data System (ADS)

Menke, Carrie

2014-03-01

Our physics undergraduate program has five program learning objectives (PLOs) focusing on (1) physical principles, (2) mathematical expertise, (3) experimental technique, (4) communication and teamwork, and (5) research proficiency. One PLO is assessed each year, with the results guiding modifications in our curriculum and future assessment practices; we have just completed our first cycle of assessing all PLOs. Our approach strives to maximize the ease and applicability of our assessment practices while maintaining faculty's flexibility in course design and delivery. Objectives are mapped onto our core curriculum with identified coursework collected as direct evidence. We've utilized mostly descriptive rubrics, applying them at the course and program levels as well as sharing them with the students. This has resulted in more efficient assessment that is also applicable to reaccreditation efforts, higher inter-rater reliability than with other rubric types, and higher quality capstone projects. We've also found that the varied quality of student writing can interfere with our assessment of other objectives. This poster outlines our processes, resources, and how we have used PLO assessment to strengthen our undergraduate program.

Assessing the reliability and validity of anti-tobacco attitudes/beliefs in the context of a campaign strategy.

PubMed

Arheart, Kristopher L; Sly, David F; Trapido, Edward J; Rodriguez, Richard D; Ellestad, Amy J

2004-11-01

To identify multi-item attitude/belief scales associated with the theoretical foundations of an anti-tobacco counter-marketing campaign and assess their reliability and validity. The data analyzed are from two state-wide, random, cross-sectional telephone surveys [n(S1)=1,079, n(S2)=1,150]. Items forming attitude/belief scales are identified using factor analysis. Reliability is assessed with Chronbach's alpha. Relationships among scales are explored using Pearson correlation. Validity is assessed by testing associations derived from the Centers for Disease Control and Prevention's (CDC) logic model for tobacco control program development and evaluation linking media exposure to attitudes/beliefs, and attitudes/beliefs to smoking-related behaviors. Adjusted odds ratios are employed for these analyses. Three factors emerged: traditional attitudes/beliefs about tobacco and tobacco use, tobacco industry manipulation and anti-tobacco empowerment. Reliability coefficients are in the range of 0.70 and vary little between age groups. The factors are correlated with one-another as hypothesized. Associations between media exposure and the attitude/belief scales and between these scales and behaviors are consistent with the CDC logic model. Using reliable, valid multi-item scales is theoretically and methodologically more sound than employing single-item measures of attitudes/beliefs. Methodological, theoretical and practical implications are discussed.

NDE Techniques Used in PARENT Open Round Robin Testing

DOE Office of Scientific and Technical Information (OSTI.GOV)

Meyer, Ryan M.

2014-11-05

This is a draft technical letter report for NRC client describing the NDE techniques used in the open testing portion of the Program to Assess the Reliability of Emerging Nondestructive Techniques (PARENT).

Assessing the Kansas water-level monitoring program: An example of the application of classical statistics to a geological problem

USGS Publications Warehouse

Davis, J.C.

2000-01-01

Geologists may feel that geological data are not amenable to statistical analysis, or at best require specialized approaches such as nonparametric statistics and geostatistics. However, there are many circumstances, particularly in systematic studies conducted for environmental or regulatory purposes, where traditional parametric statistical procedures can be beneficial. An example is the application of analysis of variance to data collected in an annual program of measuring groundwater levels in Kansas. Influences such as well conditions, operator effects, and use of the water can be assessed and wells that yield less reliable measurements can be identified. Such statistical studies have resulted in yearly improvements in the quality and reliability of the collected hydrologic data. Similar benefits may be achieved in other geological studies by the appropriate use of classical statistical tools.

The admissions process of a bachelor of science in nursing program: initial reliability and validity of the personal interview.

PubMed

Carpio, B; Brown, B

1993-01-01

The undergraduate nursing degree program (B.Sc.N.) at McMaster University School of Nursing uses small groups, and is learner-centered and problem-based. A study was conducted during the 1991 admissions cycle to determine the initial reliability and validity of the semi-structured personal interview which constitutes the final component of candidate selection for this program. During the interview, three-member teams assess applicant suitability to the program based on six dimensions: applicant motivation, awareness of the program, problem-solving abilities, ability to relate to others, self-appraisal skills, and career goals. Each interviewer assigns the applicant a global rating using a seven-point scale. For the purposes of this study four interviewer teams were randomly selected from the pool of 31 teams to interview four simulated (preprogrammed) applicants. Using two-factor repeated-measures ANOVA to analyze interview ratings, inter-rater and inter-team intraclass correlation coefficients (ICC) were calculated. Inter-team reliability ranged from .64 to .97 for the individual dimensions, and .66 to .89 on global ratings. Inter-rater ICC for the six dimensions ranged from .81 to .99, and .96 to .99 for the global ratings. The item-to-total correlation coefficients between individual dimensions and global ratings ranged from .8 to 1.0. Pearson correlations between items ranged from .77 to 1.0. The ICC were then calculated for the interview scores of 108 actual applicants to the program. Inter-rater reliability based on global ratings was .79 for the single (1 rater) observation, and .91 for the multiple (3 rater) observation. These findings support the continued use of the interview as a reliable instrument with face validity. Studies of predictive validity will be undertaken.

Validation of a short questionnaire to assess mothers' perception of workplace breastfeeding support.

PubMed

Bai, Yeon; Peng, C-Y Joanne; Fly, Alyce D

2008-07-01

The purpose of this study was to create and establish the validity of a short questionnaire to measure mothers' perceived support for breastfeeding from the workplace. The items in the workplace breastfeeding support scale (WBSS) were derived from a literature review. The scale was self-administered in central Indiana during the fall of 2005 to a convenience sample of 66 volunteers who were primiparous, 6 to 12 months postpartum, worked outside home, and had initiated breastfeeding prior to the survey. Internal consistency (alpha) and split-half reliability (r) tests and a factor analysis were done to establish reliability and construct validity of the scale. The WBSS showed acceptable reliability (alpha=.77, r=0.86). Content validity was established by review using a panel of experts. Four distinct constructs of the scale were identified that accounted for 62.1% of the total variability of the scale: technical, environmental, facility, and peer support, thus establishing construct validity of the scale. Lactation consultants and worksite lactation program planners can use the WBSS to help mothers returning to work and to assess the needs for improvement of support programs.

Clinical audit project in undergraduate medical education curriculum: an assessment validation study

PubMed Central

Steketee, Carole; Mak, Donna

2016-01-01

Objectives To evaluate the merit of the Clinical Audit Project (CAP) in an assessment program for undergraduate medical education using a systematic assessment validation framework. Methods A cross-sectional assessment validation study at one medical school in Western Australia, with retrospective qualitative analysis of the design, development, implementation and outcomes of the CAP, and quantitative analysis of assessment data from four cohorts of medical students (2011- 2014). Results The CAP is fit for purpose with clear external and internal alignment to expected medical graduate outcomes.  Substantive validity in students’ and examiners’ response processes is ensured through relevant methodological and cognitive processes. Multiple validity features are built-in to the design, planning and implementation process of the CAP.  There is evidence of high internal consistency reliability of CAP scores (Cronbach’s alpha > 0.8) and inter-examiner consistency reliability (intra-class correlation>0.7). Aggregation of CAP scores is psychometrically sound, with high internal consistency indicating one common underlying construct.  Significant but moderate correlations between CAP scores and scores from other assessment modalities indicate validity of extrapolation and alignment between the CAP and the overall target outcomes of medical graduates.  Standard setting, score equating and fair decision rules justify consequential validity of CAP scores interpretation and use. Conclusions This study provides evidence demonstrating that the CAP is a meaningful and valid component in the assessment program. This systematic framework of validation can be adopted for all levels of assessment in medical education, from individual assessment modality, to the validation of an assessment program as a whole.  PMID:27716612

Clinical audit project in undergraduate medical education curriculum: an assessment validation study.

PubMed

Tor, Elina; Steketee, Carole; Mak, Donna

2016-09-24

To evaluate the merit of the Clinical Audit Project (CAP) in an assessment program for undergraduate medical education using a systematic assessment validation framework. A cross-sectional assessment validation study at one medical school in Western Australia, with retrospective qualitative analysis of the design, development, implementation and outcomes of the CAP, and quantitative analysis of assessment data from four cohorts of medical students (2011- 2014). The CAP is fit for purpose with clear external and internal alignment to expected medical graduate outcomes.  Substantive validity in students' and examiners' response processes is ensured through relevant methodological and cognitive processes. Multiple validity features are built-in to the design, planning and implementation process of the CAP.  There is evidence of high internal consistency reliability of CAP scores (Cronbach's alpha > 0.8) and inter-examiner consistency reliability (intra-class correlation>0.7). Aggregation of CAP scores is psychometrically sound, with high internal consistency indicating one common underlying construct.  Significant but moderate correlations between CAP scores and scores from other assessment modalities indicate validity of extrapolation and alignment between the CAP and the overall target outcomes of medical graduates.  Standard setting, score equating and fair decision rules justify consequential validity of CAP scores interpretation and use. This study provides evidence demonstrating that the CAP is a meaningful and valid component in the assessment program. This systematic framework of validation can be adopted for all levels of assessment in medical education, from individual assessment modality, to the validation of an assessment program as a whole.

Reliability of abstracting performance measures: results of the cardiac rehabilitation referral and reliability (CR3) project.

PubMed

Thomas, Randal J; Chiu, Jensen S; Goff, David C; King, Marjorie; Lahr, Brian; Lichtman, Steven W; Lui, Karen; Pack, Quinn R; Shahriary, Melanie

2014-01-01

Assessment of the reliability of performance measure (PM) abstraction is an important step in PM validation. Reliability has not been previously assessed for abstracting PMs for the referral of patients to cardiac rehabilitation (CR) and secondary prevention (SP) programs. To help validate these PMs, we carried out a multicenter assessment of their reliability. Hospitals and clinical practices from around the United States were invited to participate in the Cardiac Rehabilitation Referral Reliability (CR3) Project. Twenty-nine hospitals and 23 outpatient centers expressed interest in participating. Seven hospitals and 6 outpatient centers met participation criteria and submitted completed data. Site coordinators identified 35 patients whose charts were reviewed by 2 site abstractors twice, 1 week apart. Percent agreement and the Cohen κ statistic were used to describe intra- and interabstractor reliability for patient eligibility for CR/SP, patient exceptions for CR/SP referral, and documented referral to CR/SP. Results were obtained from within-site data, as well as from pooled data of all inpatient and all outpatient sites. We found that intra-abstractor reliability reflected excellent repeatability (≥ 90% agreement; κ ≥ 0.75) for ratings of CR/SP eligibility, exceptions, and referral, both from pooled and site-specific analyses of inpatient and outpatient data. Similarly, the interabstractor agreement from pooled analysis ranged from good to excellent for the 3 items, although with slightly lower measures of reliability. Abstraction of PMs for CR/SP referral has high reliability, supporting the use of these PMs in quality improvement initiatives aimed at increasing CR/SP delivery to patients with cardiovascular disease.

Program For Evaluation Of Reliability Of Ceramic Parts

NASA Technical Reports Server (NTRS)

Nemeth, N.; Janosik, L. A.; Gyekenyesi, J. P.; Powers, Lynn M.

1996-01-01

CARES/LIFE predicts probability of failure of monolithic ceramic component as function of service time. Assesses risk that component fractures prematurely as result of subcritical crack growth (SCG). Effect of proof testing of components prior to service also considered. Coupled to such commercially available finite-element programs as ANSYS, ABAQUS, MARC, MSC/NASTRAN, and COSMOS/M. Also retains all capabilities of previous CARES code, which includes estimation of fast-fracture component reliability and Weibull parameters from inert strength (without SCG contributing to failure) specimen data. Estimates parameters that characterize SCG from specimen data as well. Written in ANSI FORTRAN 77 to be machine-independent. Program runs on any computer in which sufficient addressable memory (at least 8MB) and FORTRAN 77 compiler available. For IBM-compatible personal computer with minimum 640K memory, limited program available (CARES/PC, COSMIC number LEW-15248).

Development and pilot-test of the Workplace Readiness Questionnaire, a theory-based instrument to measure small workplaces’ readiness to implement wellness programs

PubMed Central

Hannon, Peggy A.; Helfrich, Christian D.; Chan, K. Gary; Allen, Claire L.; Hammerback, Kristen; Kohn, Marlana J.; Parrish, Amanda T.; Weiner, Bryan J.; Harris, Jeffrey R.

2016-01-01

Purpose To develop a theory-based questionnaire to assess readiness for change in small workplaces adopting wellness programs. Design In developing our scale, we first tested items via “think-aloud” interviews. We tested the revised items in a cross-sectional quantitative telephone survey. Setting Small workplaces (20–250 employees) in low-wage industries. Subjects Decision-makers representing small workplaces in King County, Washington (think-aloud interviews, n=9) and the United States (telephone survey, n=201). Measures We generated items for each construct in Weiner’s theory of organizational readiness for change. We also measured workplace characteristics and current implementation of workplace wellness programs. Analysis We assessed reliability by coefficient alpha for each of the readiness questionnaire subscales. We tested the association of all subscales with employers’ current implementation of wellness policies, programs, and communications, and conducted a path analysis to test the associations in the theory of organizational readiness to change. Results Each of the readiness subscales exhibited acceptable internal reliability (coefficient alpha range = .75–.88) and was positively associated with wellness program implementation (p <.05). The path analysis was consistent with the theory of organizational readiness to change, except change efficacy did not predict change-related effort. Conclusion We developed a new questionnaire to assess small workplaces’ readiness to adopt and implement evidence-based wellness programs. Our findings also provide empirical validation of Weiner’s theory of readiness for change. PMID:26389975

Development and Pilot Test of the Workplace Readiness Questionnaire, a Theory-Based Instrument to Measure Small Workplaces' Readiness to Implement Wellness Programs.

PubMed

Hannon, Peggy A; Helfrich, Christian D; Chan, K Gary; Allen, Claire L; Hammerback, Kristen; Kohn, Marlana J; Parrish, Amanda T; Weiner, Bryan J; Harris, Jeffrey R

2017-01-01

To develop a theory-based questionnaire to assess readiness for change in small workplaces adopting wellness programs. In developing our scale, we first tested items via "think-aloud" interviews. We tested the revised items in a cross-sectional quantitative telephone survey. The study setting comprised small workplaces (20-250 employees) in low-wage industries. Decision-makers representing small workplaces in King County, Washington (think-aloud interviews, n = 9), and the United States (telephone survey, n = 201) served as study subjects. We generated items for each construct in Weiner's theory of organizational readiness for change. We also measured workplace characteristics and current implementation of workplace wellness programs. We assessed reliability by coefficient alpha for each of the readiness questionnaire subscales. We tested the association of all subscales with employers' current implementation of wellness policies, programs, and communications, and conducted a path analysis to test the associations in the theory of organizational readiness to change. Each of the readiness subscales exhibited acceptable internal reliability (coefficient alpha range, .75-.88) and was positively associated with wellness program implementation ( p < .05). The path analysis was consistent with the theory of organizational readiness to change, except change efficacy did not predict change-related effort. We developed a new questionnaire to assess small workplaces' readiness to adopt and implement evidence-based wellness programs. Our findings also provide empirical validation of Weiner's theory of readiness for change.

Translation and validation of the Malay version of the Stroke Knowledge Test.

PubMed

Sowtali, Siti Noorkhairina; Yusoff, Dariah Mohd; Harith, Sakinah; Mohamed, Monniaty

2016-04-01

To date, there is a lack of published studies on assessment tools to evaluate the effectiveness of stroke education programs. This study developed and validated the Malay language version of the Stroke Knowledge Test research instrument. This study involved translation, validity, and reliability phases. The instrument underwent backward and forward translation of the English version into the Malay language. Nine experts reviewed the content for consistency, clarity, difficulty, and suitability for inclusion. Perceived usefulness and utilization were obtained from experts' opinions. Later, face validity assessment was conducted with 10 stroke patients to determine appropriateness of sentences and grammar used. A pilot study was conducted with 41 stroke patients to determine the item analysis and reliability of the translated instrument using the Kuder Richardson 20 or Cronbach's alpha. The final Malay version Stroke Knowledge Test included 20 items with good content coverage, acceptable item properties, and positive expert review ratings. Psychometric investigations suggest that Malay version Stroke Knowledge Test had moderate reliability with Kuder Richardson 20 or Cronbach's alpha of 0.58. Improvement is required for Stroke Knowledge Test items with unacceptable difficulty indices. Overall, the average rating of perceived usefulness and perceived utility of the instruments were both 72.7%, suggesting that reviewers were likely to use the instruments in their facilities. Malay version Stroke Knowledge Test was a valid and reliable tool to assess educational needs and to evaluate stroke knowledge among participants of group-based stroke education programs in Malaysia.

Digital Avionics Information System (DAIS): Life Cycle Cost Impact Modeling System Reliability, Maintainability, and Cost Model (RMCM)--Description. Users Guide. Final Report.

ERIC Educational Resources Information Center

Goclowski, John C.; And Others

The Reliability, Maintainability, and Cost Model (RMCM) described in this report is an interactive mathematical model with a built-in sensitivity analysis capability. It is a major component of the Life Cycle Cost Impact Model (LCCIM), which was developed as part of the DAIS advanced development program to be used to assess the potential impacts…

How reliable is computerized assessment of readability?

PubMed

Mailloux, S L; Johnson, M E; Fisher, D G; Pettibone, T J

1995-01-01

To assess the consistency and comparability of readability software programs, four software programs (Corporate Voice, Grammatix IV, Microsoft Word for Windows, and RightWriter) were compared. Standard materials included 28 pieces of printed educational materials on human immunodeficiency virus/acquired immunodeficiency syndrome distributed nationally and the Gettysburg Address. Statistical analyses for the educational materials revealed that each of the three formulas assessed (Flesch-Kincaid, Flesch Reading Ease, and Gunning Fog Index) provided significantly different grade equivalent scores and that the Microsoft Word program provided significantly lower grade levels and was more inconsistent in the scores provided. For the Gettysburg Address, considerable variation was revealed among formulas, with the discrepancy being up to two grade levels. When averaging across formulas, there was a variation of 1.3 grade levels between the four software programs. Given the variation between formulas and programs, implications for decisions based on results of these software programs are provided.

Mathematical programming models for the economic design and assessment of wind energy conversion systems

NASA Astrophysics Data System (ADS)

Reinert, K. A.

The use of linear decision rules (LDR) and chance constrained programming (CCP) to optimize the performance of wind energy conversion clusters coupled to storage systems is described. Storage is modelled by LDR and output by CCP. The linear allocation rule and linear release rule prescribe the size and optimize a storage facility with a bypass. Chance constraints are introduced to explicitly treat reliability in terms of an appropriate value from an inverse cumulative distribution function. Details of deterministic programming structure and a sample problem involving a 500 kW and a 1.5 MW WECS are provided, considering an installed cost of $1/kW. Four demand patterns and three levels of reliability are analyzed for optimizing the generator choice and the storage configuration for base load and peak operating conditions. Deficiencies in ability to predict reliability and to account for serial correlations are noted in the model, which is concluded useful for narrowing WECS design options.

DOE Office of Scientific and Technical Information (OSTI.GOV)

Vanderwiel, Scott A; Wilson, Alyson G; Graves, Todd L

Both the U. S. Department of Defense (DoD) and Department of Energy (DOE) maintain weapons stockpiles: items like bullets, missiles and bombs that have already been produced and are being stored until needed. Ideally, these stockpiles maintain high reliability over time. To assess reliability, a surveillance program is implemented, where units are periodically removed from the stockpile and tested. The most definitive tests typically destroy the weapons so a given unit is tested only once. Surveillance managers need to decide how many units should be tested, how often they should be tested, what tests should be done, and how themore » resulting data are used to estimate the stockpile's current and future reliability. These issues are particularly critical from a planning perspective: given what has already been observed and our understanding of the mechanisms of stockpile aging, what is an appropriate and cost-effective surveillance program? Surveillance programs are costly, broad, and deep, especially in the DOE, where the US nuclear weapons surveillance program must 'ensure, through various tests, that the reliability of nuclear weapons is maintained' in the absence of full-system testing (General Accounting Office, 1996). The DOE program consists primarily of three types of tests: nonnuclear flight tests, that involve the actual dropping or launching of a weapon from which the nuclear components have been removed; and nonnuclear and nuclear systems laboratory tests, which detect defects due to aging, manufacturing, and design of the nonnuclear and nuclear portions of the weapons. Fully integrated analysis of the suite of nuclear weapons surveillance data is an ongoing area of research (Wilson et al., 2007). This paper introduces a simple model that captures high-level features of stockpile reliability over time and can be used to answer broad policy questions about surveillance programs. Our intention is to provide a framework that generates tractable answers that integrate expert knowledge and high-level summaries of surveillance data to allow decision-making about appropriate trade-offs between the cost of data and the precision of stockpile reliability estimates.« less

NASA EEE Parts and Advanced Interconnect Program (AIP)

NASA Technical Reports Server (NTRS)

Gindorf, T.; Garrison, A.

1996-01-01

none given From Program Objectives: I. Accelerate the readiness of new technologies through development of validation, assessment and test method/tools II. Provide NASA Projects infusion paths for emerging technologies III. Provide NASA Projects technology selection, application and validation guidelines for harware and processes IV. Disseminate quality assurance, reliability, validation, tools and availability information to the NASA community.

Design Development Test and Evaluation (DDT and E) Considerations for Safe and Reliable Human Rated Spacecraft Systems

NASA Technical Reports Server (NTRS)

Miller, James; Leggett, Jay; Kramer-White, Julie

2008-01-01

A team directed by the NASA Engineering and Safety Center (NESC) collected methodologies for how best to develop safe and reliable human rated systems and how to identify the drivers that provide the basis for assessing safety and reliability. The team also identified techniques, methodologies, and best practices to assure that NASA can develop safe and reliable human rated systems. The results are drawn from a wide variety of resources, from experts involved with the space program since its inception to the best-practices espoused in contemporary engineering doctrine. This report focuses on safety and reliability considerations and does not duplicate or update any existing references. Neither does it intend to replace existing standards and policy.

Development and reliability testing of the Worksite and Energy Balance Survey.

PubMed

Hoehner, Christine M; Budd, Elizabeth L; Marx, Christine M; Dodson, Elizabeth A; Brownson, Ross C

2013-01-01

Worksites represent important venues for health promotion. Development of psychometrically sound measures of worksite environments and policy supports for physical activity and healthy eating are needed for use in public health research and practice. Assess the test-retest reliability of the Worksite and Energy Balance Survey (WEBS), a self-report instrument for assessing perceptions of worksite supports for physical activity and healthy eating. The WEBS included items adapted from existing surveys or new items on the basis of a review of the literature and expert review. Cognitive interviews among 12 individuals were used to test the clarity of items and further refine the instrument. A targeted random-digit-dial telephone survey was administered on 2 occasions to assess test-retest reliability (mean days between time periods = 8; minimum = 5; maximum = 14). Five Missouri census tracts that varied by racial-ethnic composition and walkability. Respondents included 104 employed adults (67% white, 64% women, mean age = 48.6 years). Sixty-three percent were employed at worksites with less than 100 employees, approximately one-third supervised other people, and the majority worked a regular daytime shift (75%). Test-retest reliability was assessed using Spearman correlations for continuous variables, Cohen's κ statistics for nonordinal categorical variables, and 1-way random intraclass correlation coefficients for ordinal categorical variables. Test-retest coefficients ranged from 0.41 to 0.97, with 80% of items having reliability coefficients of more than 0.6. Items that assessed participation in or use of worksite programs/facilities tended to have lower reliability. Reliability of some items varied by gender, obesity status, and worksite size. Test-retest reliability and internal consistency for the 5 scales ranged from 0.84 to 0.94 and 0.63 to 0.84, respectively. The WEBS items and scales exhibited sound test-retest reliability and may be useful for research and surveillance. Further evaluation is needed to document the validity of the WEBS and associations with energy balance outcomes.

Assessment of NDE Reliability Data

NASA Technical Reports Server (NTRS)

Yee, B. G. W.; Chang, F. H.; Couchman, J. C.; Lemon, G. H.; Packman, P. F.

1976-01-01

Twenty sets of relevant Nondestructive Evaluation (NDE) reliability data have been identified, collected, compiled, and categorized. A criterion for the selection of data for statistical analysis considerations has been formulated. A model to grade the quality and validity of the data sets has been developed. Data input formats, which record the pertinent parameters of the defect/specimen and inspection procedures, have been formulated for each NDE method. A comprehensive computer program has been written to calculate the probability of flaw detection at several confidence levels by the binomial distribution. This program also selects the desired data sets for pooling and tests the statistical pooling criteria before calculating the composite detection reliability. Probability of detection curves at 95 and 50 percent confidence levels have been plotted for individual sets of relevant data as well as for several sets of merged data with common sets of NDE parameters.

Validation of the German revised version of the program in palliative care education and practice questionnaire (PCEP-GR).

PubMed

Fetz, Katharina; Wenzel-Meyburg, Ursula; Schulz-Quach, Christian

2017-12-28

The evaluation of the effectiveness of undergraduate palliative care education (UPCE) programs is an essential foundation to providing high-quality UPCE programs. Therefore, the implementation of valid evaluation tools is indispensable. Until today, there has been no general consensus regarding concrete outcome parameters and their accurate measurement. The Program in Palliative Care Education and Practice Questionnaire (German Revised Version; PCEP-GR) is a promising assessment tool for UPCE. The aim of the current study was to evaluate the psychometric properties of PCEP-GR and to demonstrate its feasibility for the evaluation of UPCE programs. The practical feasibility of the PCEP-GR and its acceptance in medical students were investigated in a pilot study with 24 undergraduate medical students at Heinrich Heine University Dusseldorf, Germany. Subsequently, the PCEP-GR was surveyed in a representative sample (N = 680) of medical students in order to investigate its psychometric properties. Factorial validity was investigated by means of principal component analysis (PCA). Reliability was examined by means of split-half-reliability analysis and analysis of internal consistency. After taking into consideration the PCA and distribution analysis results, an evaluation instruction for the PCEP-GR was developed. The PCEP-GR proved to be feasible and well-accepted in medical students. PCA revealed a four-factorial solution indicating four PCEP-GR subscales: preparation to provide palliative care, attitudes towards palliative care, self-estimation of competence in communication with dying patients and their relatives and self-estimation of knowledge and skills in palliative care. The PCEP-GR showed good split-half-reliability and acceptable to good internal consistency of subscales. Attitudes towards palliative care slightly missed the criterion of acceptable internal consistency. The evaluation instruction suggests a global PCEP-GR index and four subscales. The PCEP-GR has proven to be a feasible, economic, valid and reliable tool for the assessment of UPCE that comprises self-efficacy expectation and relevant attitudes towards palliative care.

Developing a tool for assessing competency in root cause analysis.

PubMed

Gupta, Priyanka; Varkey, Prathibha

2009-01-01

Root cause analysis (RCA) is a tool for identifying the key cause(s) contributing to a sentinel event or near miss. Although training in RCA is gaining popularity in medical education, there is no published literature on valid or reliable methods for assessing competency in the same. A tool for assessing competency in RCA was pilot tested as part of an eight-station Objective Structured Clinical Examination that was conducted at the completion of a three-week quality improvement (QI) curriculum for the Mayo Clinic Preventive Medicine and Endocrinology fellowship programs. As part of the curriculum, fellows completed a QI project to enhance physician communication of the diagnosis and treatment plan at the end of a patient visit. They had a didactic session on RCA, followed by process mapping of the information flow at the project clinic, after which fellows conducted an actual RCA using the Ishikawa fishbone diagram. For the RCA competency assessment, fellows performed an RCA regarding a scenario describing an adverse medication event and provided possible solutions to prevent such errors in the future. All faculty strongly agreed or agreed that they were able to accurately assess competency in RCA using the tool. Interrater reliability for the global competency rating and checklist scoring were 0.96 and 0.85, respectively. Internal consistency (Cronbach's alpha) was 0.76. Six of eight of the fellows found the difficulty level of the test to be optimal. Assessment methods must accompany education programs to ensure that graduates are competent in QI methodologies and are able to apply them effectively in the workplace. The RCA assessment tool was found to be a valid, reliable, feasible, and acceptable method for assessing competency in RCA. Further research is needed to examine its predictive validity and generalizability.

Object-oriented fault tree evaluation program for quantitative analyses

NASA Technical Reports Server (NTRS)

Patterson-Hine, F. A.; Koen, B. V.

1988-01-01

Object-oriented programming can be combined with fault free techniques to give a significantly improved environment for evaluating the safety and reliability of large complex systems for space missions. Deep knowledge about system components and interactions, available from reliability studies and other sources, can be described using objects that make up a knowledge base. This knowledge base can be interrogated throughout the design process, during system testing, and during operation, and can be easily modified to reflect design changes in order to maintain a consistent information source. An object-oriented environment for reliability assessment has been developed on a Texas Instrument (TI) Explorer LISP workstation. The program, which directly evaluates system fault trees, utilizes the object-oriented extension to LISP called Flavors that is available on the Explorer. The object representation of a fault tree facilitates the storage and retrieval of information associated with each event in the tree, including tree structural information and intermediate results obtained during the tree reduction process. Reliability data associated with each basic event are stored in the fault tree objects. The object-oriented environment on the Explorer also includes a graphical tree editor which was modified to display and edit the fault trees.

Inter-rater and test–retest reliability of quality assessments by novice student raters using the Jadad and Newcastle–Ottawa Scales

PubMed Central

Oremus, Carolina; Hall, Geoffrey B C; McKinnon, Margaret C

2012-01-01

Introduction Quality assessment of included studies is an important component of systematic reviews. Objective The authors investigated inter-rater and test–retest reliability for quality assessments conducted by inexperienced student raters. Design Student raters received a training session on quality assessment using the Jadad Scale for randomised controlled trials and the Newcastle–Ottawa Scale (NOS) for observational studies. Raters were randomly assigned into five pairs and they each independently rated the quality of 13–20 articles. These articles were drawn from a pool of 78 papers examining cognitive impairment following electroconvulsive therapy to treat major depressive disorder. The articles were randomly distributed to the raters. Two months later, each rater re-assessed the quality of half of their assigned articles. Setting McMaster Integrative Neuroscience Discovery and Study Program. Participants 10 students taking McMaster Integrative Neuroscience Discovery and Study Program courses. Main outcome measures The authors measured inter-rater reliability using κ and the intraclass correlation coefficient type 2,1 or ICC(2,1). The authors measured test–retest reliability using ICC(2,1). Results Inter-rater reliability varied by scale question. For the six-item Jadad Scale, question-specific κs ranged from 0.13 (95% CI −0.11 to 0.37) to 0.56 (95% CI 0.29 to 0.83). The ranges were −0.14 (95% CI −0.28 to 0.00) to 0.39 (95% CI −0.02 to 0.81) for the NOS cohort and −0.20 (95% CI −0.49 to 0.09) to 1.00 (95% CI 1.00 to 1.00) for the NOS case–control. For overall scores on the six-item Jadad Scale, ICC(2,1)s for inter-rater and test–retest reliability (accounting for systematic differences between raters) were 0.32 (95% CI 0.08 to 0.52) and 0.55 (95% CI 0.41 to 0.67), respectively. Corresponding ICC(2,1)s for the NOS cohort were −0.19 (95% CI −0.67 to 0.35) and 0.62 (95% CI 0.25 to 0.83), and for the NOS case–control, the ICC(2,1)s were 0.46 (95% CI −0.13 to 0.92) and 0.83 (95% CI 0.48 to 0.95). Conclusions Inter-rater reliability was generally poor to fair and test–retest reliability was fair to excellent. A pilot rating phase following rater training may be one way to improve agreement. PMID:22855629

Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales.

PubMed

Oremus, Mark; Oremus, Carolina; Hall, Geoffrey B C; McKinnon, Margaret C

2012-01-01

Quality assessment of included studies is an important component of systematic reviews. The authors investigated inter-rater and test-retest reliability for quality assessments conducted by inexperienced student raters. Student raters received a training session on quality assessment using the Jadad Scale for randomised controlled trials and the Newcastle-Ottawa Scale (NOS) for observational studies. Raters were randomly assigned into five pairs and they each independently rated the quality of 13-20 articles. These articles were drawn from a pool of 78 papers examining cognitive impairment following electroconvulsive therapy to treat major depressive disorder. The articles were randomly distributed to the raters. Two months later, each rater re-assessed the quality of half of their assigned articles. McMaster Integrative Neuroscience Discovery and Study Program. 10 students taking McMaster Integrative Neuroscience Discovery and Study Program courses. The authors measured inter-rater reliability using κ and the intraclass correlation coefficient type 2,1 or ICC(2,1). The authors measured test-retest reliability using ICC(2,1). Inter-rater reliability varied by scale question. For the six-item Jadad Scale, question-specific κs ranged from 0.13 (95% CI -0.11 to 0.37) to 0.56 (95% CI 0.29 to 0.83). The ranges were -0.14 (95% CI -0.28 to 0.00) to 0.39 (95% CI -0.02 to 0.81) for the NOS cohort and -0.20 (95% CI -0.49 to 0.09) to 1.00 (95% CI 1.00 to 1.00) for the NOS case-control. For overall scores on the six-item Jadad Scale, ICC(2,1)s for inter-rater and test-retest reliability (accounting for systematic differences between raters) were 0.32 (95% CI 0.08 to 0.52) and 0.55 (95% CI 0.41 to 0.67), respectively. Corresponding ICC(2,1)s for the NOS cohort were -0.19 (95% CI -0.67 to 0.35) and 0.62 (95% CI 0.25 to 0.83), and for the NOS case-control, the ICC(2,1)s were 0.46 (95% CI -0.13 to 0.92) and 0.83 (95% CI 0.48 to 0.95). Inter-rater reliability was generally poor to fair and test-retest reliability was fair to excellent. A pilot rating phase following rater training may be one way to improve agreement.

Validity and reliability of the robotic objective structured assessment of technical skills

PubMed Central

Siddiqui, Nazema Y.; Galloway, Michael L.; Geller, Elizabeth J.; Green, Isabel C.; Hur, Hye-Chun; Langston, Kyle; Pitter, Michael C.; Tarr, Megan E.; Martino, Martin A.

2015-01-01

Objective Objective structured assessments of technical skills (OSATS) have been developed to measure the skill of surgical trainees. Our aim was to develop an OSATS specifically for trainees learning robotic surgery. Study Design This is a multi-institutional study in eight academic training programs. We created an assessment form to evaluate robotic surgical skill through five inanimate exercises. Obstetrics/gynecology, general surgery, and urology residents, fellows, and faculty completed five robotic exercises on a standard training model. Study sessions were recorded and randomly assigned to three blinded judges who scored performance using the assessment form. Construct validity was evaluated by comparing scores between participants with different levels of surgical experience; inter- and intra-rater reliability were also assessed. Results We evaluated 83 residents, 9 fellows, and 13 faculty, totaling 105 participants; 88 (84%) were from obstetrics/gynecology. Our assessment form demonstrated construct validity, with faculty and fellows performing significantly better than residents (mean scores: 89 ± 8 faculty; 74 ± 17 fellows; 59 ± 22 residents, p<0.01). In addition, participants with more robotic console experience scored significantly higher than those with fewer prior console surgeries (p<0.01). R-OSATS demonstrated good inter-rater reliability across all five drills (mean Cronbach's α: 0.79 ± 0.02). Intra-rater reliability was also high (mean Spearman's correlation: 0.91 ± 0.11). Conclusions We developed an assessment form for robotic surgical skill that demonstrates construct validity, inter- and intra-rater reliability. When paired with standardized robotic skill drills this form may be useful to distinguish between levels of trainee performance. PMID:24807319

Overview of DOE-NE Proliferation and Terrorism Risk Assessment

DOE Office of Scientific and Technical Information (OSTI.GOV)

Sadasivan, Pratap

2012-08-24

Research objectives are: (1) Develop technologies and other solutions that can improve the reliability, sustain the safety, and extend the life of current reactors; (2) Develop improvements in the affordability of new reactors to enable nuclear energy; (3) Develop Sustainable Nuclear Fuel Cycles; and (4) Understand and minimize the risks of nuclear proliferation and terrorism. The goal is to enable the use of risk information to inform NE R&D program planning. The PTRA program supports DOE-NE's goal of using risk information to inform R&D program planning. The FY12 PTRA program is focused on terrorism risk. The program includes a mixmore » of innovative methods that support the general practice of risk assessments, and selected applications.« less

Assessing Reliability of Medical Record Reviews for the Detection of Hospital Adverse Events.

PubMed

Ock, Minsu; Lee, Sang-il; Jo, Min-Woo; Lee, Jin Yong; Kim, Seon-Ha

2015-09-01

The purpose of this study was to assess the inter-rater reliability and intra-rater reliability of medical record review for the detection of hospital adverse events. We conducted two stages retrospective medical records review of a random sample of 96 patients from one acute-care general hospital. The first stage was an explicit patient record review by two nurses to detect the presence of 41 screening criteria (SC). The second stage was an implicit structured review by two physicians to identify the occurrence of adverse events from the positive cases on the SC. The inter-rater reliability of two nurses and that of two physicians were assessed. The intra-rater reliability was also evaluated by using test-retest method at approximately two weeks later. In 84.2% of the patient medical records, the nurses agreed as to the necessity for the second stage review (kappa, 0.68; 95% confidence interval [CI], 0.54 to 0.83). In 93.0% of the patient medical records screened by nurses, the physicians agreed about the absence or presence of adverse events (kappa, 0.71; 95% CI, 0.44 to 0.97). When assessing intra-rater reliability, the kappa indices of two nurses were 0.54 (95% CI, 0.31 to 0.77) and 0.67 (95% CI, 0.47 to 0.87), whereas those of two physicians were 0.87 (95% CI, 0.62 to 1.00) and 0.37 (95% CI, -0.16 to 0.89). In this study, the medical record review for detecting adverse events showed intermediate to good level of inter-rater and intra-rater reliability. Well organized training program for reviewers and clearly defining SC are required to get more reliable results in the hospital adverse event study.

Suitability of the Literacy and Numeracy Screening (LINUS) 2.0 Programme in Assessing Children's Early Literacy

ERIC Educational Resources Information Center

Luyee, Eunice Ong; Roselan, Fauzan Izzati; Anwardeen, Nor Hafizah; Mustapa, Fatin Hazirah Mohd

2015-01-01

Early literacy skills are crucial in a child's learning process and awareness should be raised in order to ensure the quality of early literacy assessments. In this paper, the writers discuss the quality of early literacy assessment in Malaysia, LINUS 2.0 by looking at its validity and reliability. An established early literacy program is compared…

Quantitative Analysis of the Rubric as an Assessment Tool: An Empirical Study of Student Peer-Group Rating

ERIC Educational Resources Information Center

Hafner, John C.; Hafner, Patti M.

2003-01-01

Although the rubric has emerged as one of the most popular assessment tools in progressive educational programs, there is an unfortunate dearth of information in the literature quantifying the actual effectiveness of the rubric as an assessment tool "in the hands of the students." This study focuses on the validity and reliability of the rubric as…

Using a Scoring Rubric to Assess the Writing of Bioethics Students.

PubMed

Stoddard, Hugh A; Labrecque, Cory A; Schonfeld, Toby

2016-04-01

Educators in bioethics have struggled to find valid and reliable assessments that transcend the "reproduction of knowledge" to target more important skill sets. This manuscript reports on the process of developing and grading a minimal-competence comprehensive examination in a bioethics master's degree program. We describe educational theory and practice for the creation and deployment of scoring rubrics for high-stakes performance assessments that reduce scoring inconsistencies. The rubric development process can also benefit the program by building consensus among stakeholders regarding program goals and student outcomes. We describe the Structure of the Observed Learning Outcome taxonomy as a mechanism for rubric design and provide an example of how we applied that taxonomy to define pass/fail cut scores. Details about domains of assessment and writing descriptors of performance are also presented. Despite the laborious work required to create a scoring rubric, we found the effort to be worthwhile for our program.

Initial Development and Psychometric Properties of a New Measure of Substance Use Disorder "Recovery Progression": The Recovery Progression Measure (RPM).

PubMed

Elison, Sarah; Davies, Glyn; Ward, Jonathan

2016-07-28

There is a growing literature around substance use disorder treatment outcomes measures. Various constructs have been suggested as being appropriate for measuring recovery outcomes, including "recovery capital" and "treatment progression." However, these previously proposed constructs do not measure changes in psychosocial functioning during the recovery process. Therefore, a new psychometric assessment, the "Recovery Progression Measure" (RPM), has been developed to measure this recovery oriented psychosocial change. The aims of this study were to evaluate the reliability and factor structure of the RPM via data collected from 2218 service users being treated for their substance dependence. Data were collected from service users accessing the Breaking Free Online (BFO) substance use disorder treatment and recovery program, which has within its baseline assessment a 36-item psychometric measure previously developed by the authors to assess the six areas of functioning described in the RPM. Reliability analyses and exploratory factor analyses (EFA) were conducted to examine the underlying factor structure of the RPM measure. Internal reliability of the RPM measure was found to be excellent (α > .70) with the overall assessment to have reliability α = .89, with item-total correlations revealing moderate-excellent reliability of individual items. EFA revealed the RPM to contain an underlying factor structure of eight components. This study provides initial data to support the reliability of the RPM as a recovery measure. Further work is now underway to extend these findings, including convergent and predictive validity analyses.

Accuracy and reliability of peer assessment of athletic training psychomotor laboratory skills.

PubMed

Marty, Melissa C; Henning, Jolene M; Willse, John T

2010-01-01

Peer assessment is defined as students judging the level or quality of a fellow student's understanding. No researchers have yet demonstrated the accuracy or reliability of peer assessment in athletic training education. To determine the accuracy and reliability of peer assessment of athletic training students' psychomotor skills. Cross-sectional study. Entry-level master's athletic training education program. First-year (n  =  5) and second-year (n  =  8) students. Participants evaluated 10 videos of a peer performing 3 psychomotor skills (middle deltoid manual muscle test, Faber test, and Slocum drawer test) on 2 separate occasions using a valid assessment tool. Accuracy of each peer-assessment score was examined through percentage correct scores. We used a generalizability study to determine how reliable athletic training students were in assessing a peer performing the aforementioned skills. Decision studies using generalizability theory demonstrated how the peer-assessment scores were affected by the number of participants and number of occasions. Participants had a high percentage of correct scores: 96.84% for the middle deltoid manual muscle test, 94.83% for the Faber test, and 97.13% for the Slocum drawer test. They were not able to reliably assess a peer performing any of the psychomotor skills on only 1 occasion. However, the φ increased (exceeding the 0.70 minimal standard) when 2 participants assessed the skill on 3 occasions (φ  =  0.79) for the Faber test, with 1 participant on 2 occasions (φ  =  0.76) for the Slocum drawer test, and with 3 participants on 2 occasions for the middle deltoid manual muscle test (φ  =  0.72). Although students did not detect all errors, they assessed their peers with an average of 96% accuracy. Having only 1 student assess a peer performing certain psychomotor skills was less reliable than having more than 1 student assess those skills on more than 1 occasion. Peer assessment of psychomotor skills could be an important part of the learning process and a tool to supplement instructor assessment.

Reliability and Validity of 3 Methods of Assessing Orthopedic Resident Skill in Shoulder Surgery.

PubMed

Bernard, Johnathan A; Dattilo, Jonathan R; Srikumaran, Uma; Zikria, Bashir A; Jain, Amit; LaPorte, Dawn M

Traditional measures for evaluating resident surgical technical skills (e.g., case logs) assess operative volume but not level of surgical proficiency. Our goal was to compare the reliability and validity of 3 tools for measuring surgical skill among orthopedic residents when performing 3 open surgical approaches to the shoulder. A total of 23 residents at different stages of their surgical training were tested for technical skill pertaining to 3 shoulder surgical approaches using the following measures: Objective Structured Assessment of Technical Skills (OSATS) checklists, the Global Rating Scale (GRS), and a final pass/fail assessment determined by 3 upper extremity surgeons. Adverse events were recorded. The Cronbach α coefficient was used to assess reliability of the OSATS checklists and GRS scores. Interrater reliability was calculated with intraclass correlation coefficients. Correlations among OSATS checklist scores, GRS scores, and pass/fail assessment were calculated with Spearman ρ. Validity of OSATS checklists was determined using analysis of variance with postgraduate year (PGY) as a between-subjects factor. Significance was set at p < 0.05 for all tests. Criterion validity was shown between the OSATS checklists and GRS for the 3 open shoulder approaches. Checklist scores showed superior interrater reliability compared with GRS and subjective pass/fail measurements. GRS scores were positively correlated across training years. The incidence of adverse events was significantly higher among PGY-1 and PGY-2 residents compared with more experienced residents. OSATS checklists are a valid and reliable assessment of technical skills across 3 surgical shoulder approaches. However, checklist scores do not measure quality of technique. Documenting adverse events is necessary to assess quality of technique and ultimate pass/fail status. Multiple methods of assessing surgical skill should be considered when evaluating orthopedic resident surgical performance. Copyright © 2016 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

[Design of low-intermediate frequency electrotherapy and pain assessment system].

PubMed

Liang, Chunyan; Tian, Xuelong; Yu, Xuehong; Luo, Hongyan

2014-06-01

Aiming at the single treatment and the design separation between treatment and assessment in electrotherapy equipment, a kind of system including low-intermediate frequency treatment and efficacy evaluation was developed. With C8051F020 single-chip microcomputer as the core and the circuit design and software programming used, the system realized the random switch of therapeutic parameters, the collection, display and data storage of pressure pain threshold in the assessment. Experiment results showed that the stimulus waveform, current intensity, frequency, duty ratio of the system output were adjustable, accurate and reliable. The obtained pressure pain threshold had a higher accuracy (< 0.3 N) and better stability, guiding the parameter choice in the precise electrical stimulation. It, therefore, provides a reliable technical support for the treatment and curative effect assessment.

The NASA computer science research program plan

NASA Technical Reports Server (NTRS)

1983-01-01

A taxonomy of computer science is included, one state of the art of each of the major computer science categories is summarized. A functional breakdown of NASA programs under Aeronautics R and D, space R and T, and institutional support is also included. These areas were assessed against the computer science categories. Concurrent processing, highly reliable computing, and information management are identified.

Using Curriculum Based Measures To Identify and Monitor Progress in an Adult Basic Education Program. Final Report.

ERIC Educational Resources Information Center

Bean, Rita M.; And Others

The purpose of a project was to develop and test curriculum-based procedures and measures to monitor and assess the reading and writing progress of adults in a basic education program. The most efficient, reliable, and feasible measure of reading performance from beginning reading level through eighth-grade level was the repeated oral reading…

Enhancing nurses' ethical practice: development of a clinical ethics program.

PubMed

McDaniel, C

1998-06-01

There is increasing attention paid to ethics under managed care; however, few clinical-based ethics programs are reported. This paper reports the assessment and outcomes of one such program. A quasi-experimental research design with t-tests is used to assess the outcome differences between participants and control groups. There are twenty nurses in each; they are assessed for comparability. Differences are predicted on two outcomes using reliable and valid measures: nurses' time with their patients in ethics discussions, and nurses' opinions regarding their clinical ethics environments. Results reveal a statistically significant difference (p <.05) between the two groups, with modest positive change in the participants. Additional exploratory analyses are reported on variables influential in health care services.

Assessing the Generalizable Skills of Post-Secondary Vocational Students. A Validation Study.

ERIC Educational Resources Information Center

Greenan, James P.; Smith, Brandon B.

A study examined the feasibility, reliability, and validity of two instruments designed to assess the degree to which postsecondary vocational students possessed those generalizable skills that are believed to be functionally relevant to success in a vocational program. The instruments, a student self-rating and a teacher rating form, contained 81…

A Comparison of Reliability and Precision of Subscore Reporting Methods for a State English Language Proficiency Assessment

ERIC Educational Resources Information Center

Longabach, Tanya; Peyton, Vicki

2018-01-01

K-12 English language proficiency tests that assess multiple content domains (e.g., listening, speaking, reading, writing) often have subsections based on these content domains; scores assigned to these subsections are commonly known as subscores. Testing programs face increasing customer demands for the reporting of subscores in addition to the…

Automated Pilot Performance Assessment in the T-37: A Feasibility Study. Final Report (May 1968-April 1971).

ERIC Educational Resources Information Center

Knoop, Patricia A.; Welde, William L.

Air Force investigators conducted a three year program to develop a capability for automated quantification and assessment of in-flight pilot performance. Such a capability enhances pilot training by making ratings more objective, valid, reliable and sensitive, and by freeing instructors from rating responsibilities, allowing them to concentrate…

Large-Scale Multiobjective Static Test Generation for Web-Based Testing with Integer Programming

ERIC Educational Resources Information Center

Nguyen, M. L.; Hui, Siu Cheung; Fong, A. C. M.

2013-01-01

Web-based testing has become a ubiquitous self-assessment method for online learning. One useful feature that is missing from today's web-based testing systems is the reliable capability to fulfill different assessment requirements of students based on a large-scale question data set. A promising approach for supporting large-scale web-based…

Assessing the Culture and Climate for Quality Improvement in the Work Environment. AIR 1994 Annual Forum Paper.

ERIC Educational Resources Information Center

Cameron, Kim; And Others

This study attempted to develop a reliable and valid instrument for assessing work environment and continuous quality improvement efforts in the non-academic sectors of colleges and universities particularly those institutions who have adopted Total Quality Management programs. A model of a work environment for continuous quality improvement was…

Assessing the effects of employee assistance programs: a review of employee assistance program evaluations.

PubMed

Colantonio, A

1989-01-01

Employee assistance programs have grown at a dramatic rate, yet the effectiveness of these programs has been called into question. The purpose of this paper was to assess the effectiveness of employee assistance programs (EAPs) by reviewing recently published EAP evaluations. All studies evaluating EAPs published since 1975 from peer-reviewed journals in the English language were included in this analysis. Each of the articles was assessed in the following areas: (a) program description (subjects, setting, type of intervention, format), (b) evaluation design (research design, variables measured, operational methods), and (c) program outcomes. Results indicate numerous methodological and conceptual weaknesses and issues. These weaknesses included lack of controlled research designs and short time lags between pre- and post-test measures. Other problems identified are missing information regarding subjects, type of intervention, how variables are measured (operational methods), and reliability and validity of evaluation instruments. Due to the aforementioned weaknesses, positive outcomes could not be supported. Recommendations are made for future EAP evaluations.

Assessing the effects of employee assistance programs: a review of employee assistance program evaluations.

PubMed Central

Colantonio, A.

1989-01-01

Employee assistance programs have grown at a dramatic rate, yet the effectiveness of these programs has been called into question. The purpose of this paper was to assess the effectiveness of employee assistance programs (EAPs) by reviewing recently published EAP evaluations. All studies evaluating EAPs published since 1975 from peer-reviewed journals in the English language were included in this analysis. Each of the articles was assessed in the following areas: (a) program description (subjects, setting, type of intervention, format), (b) evaluation design (research design, variables measured, operational methods), and (c) program outcomes. Results indicate numerous methodological and conceptual weaknesses and issues. These weaknesses included lack of controlled research designs and short time lags between pre- and post-test measures. Other problems identified are missing information regarding subjects, type of intervention, how variables are measured (operational methods), and reliability and validity of evaluation instruments. Due to the aforementioned weaknesses, positive outcomes could not be supported. Recommendations are made for future EAP evaluations. PMID:2728498

JUPITER PROJECT - MERGING INVERSE PROBLEM FORMULATION TECHNOLOGIES

EPA Science Inventory

The JUPITER (Joint Universal Parameter IdenTification and Evaluation of Reliability) project seeks to enhance and build on the technology and momentum behind two of the most popular sensitivity analysis, data assessment, calibration, and uncertainty analysis programs used in envi...

Reliability and Validity of the World Health Organization Quality of Life: Brief Version (WHOQOL-BREF) in a Homeless Substance Dependent Veteran Population

ERIC Educational Resources Information Center

Garcia-Rea, Elizabeth A.; LePage, James P.

2010-01-01

With the high number of homeless, there is a critical need for rapid and accurate assessment of quality of life to assess program outcomes. The World Health Organization's WHOQOL-100 has demonstrated promise in accurately assessing quality-of-life in this population. However, its length may make large scale use impractical for working with a…

Reliability program requirements for aeronautical and space system contractors

NASA Technical Reports Server (NTRS)

1987-01-01

General reliability program requirements for NASA contracts involving the design, development, fabrication, test, and/or use of aeronautical and space systems including critical ground support equipment are prescribed. The reliability program requirements require (1) thorough planning and effective management of the reliability effort; (2) definition of the major reliability tasks and their place as an integral part of the design and development process; (3) planning and evaluating the reliability of the system and its elements (including effects of software interfaces) through a program of analysis, review, and test; and (4) timely status indication by formal documentation and other reporting to facilitate control of the reliability program.

System Architectural Considerations on Reliable Guidance, Navigation, and Control (GN and C) for Constellation Program (CxP) Spacecraft

NASA Technical Reports Server (NTRS)

Dennehy, Cornelius J.

2010-01-01

This final report summarizes the results of a comparative assessment of the fault tolerance and reliability of different Guidance, Navigation and Control (GN&C) architectural approaches. This study was proactively performed by a combined Massachusetts Institute of Technology (MIT) and Draper Laboratory team as a GN&C "Discipline-Advancing" activity sponsored by the NASA Engineering and Safety Center (NESC). This systematic comparative assessment of GN&C system architectural approaches was undertaken as a fundamental step towards understanding the opportunities for, and limitations of, architecting highly reliable and fault tolerant GN&C systems composed of common avionic components. The primary goal of this study was to obtain architectural 'rules of thumb' that could positively influence future designs in the direction of an optimized (i.e., most reliable and cost-efficient) GN&C system. A secondary goal was to demonstrate the application and the utility of a systematic modeling approach that maps the entire possible architecture solution space.

Reliability-Based Life Assessment of Stirling Convertor Heater Head

NASA Technical Reports Server (NTRS)

Shah, Ashwin R.; Halford, Gary R.; Korovaichuk, Igor

2004-01-01

Onboard radioisotope power systems being developed and planned for NASA's deep-space missions require reliable design lifetimes of up to 14 yr. The structurally critical heater head of the high-efficiency Stirling power convertor has undergone extensive computational analysis of operating temperatures, stresses, and creep resistance of the thin-walled Inconel 718 bill of material. A preliminary assessment of the effect of uncertainties in the material behavior was also performed. Creep failure resistance of the thin-walled heater head could show variation due to small deviations in the manufactured thickness and in uncertainties in operating temperature and pressure. Durability prediction and reliability of the heater head are affected by these deviations from nominal design conditions. Therefore, it is important to include the effects of these uncertainties in predicting the probability of survival of the heater head under mission loads. Furthermore, it may be possible for the heater head to experience rare incidences of small temperature excursions of short duration. These rare incidences would affect the creep strain rate and, therefore, the life. This paper addresses the effects of such rare incidences on the reliability. In addition, the sensitivities of variables affecting the reliability are quantified, and guidelines developed to improve the reliability are outlined. Heater head reliability is being quantified with data from NASA Glenn Research Center's accelerated benchmark testing program.

Reliability of a new test battery for fitness assessment of the European Astronaut corps.

PubMed

Petersen, Nora; Thieschäfer, Lutz; Ploutz-Snyder, Lori; Damann, Volker; Mester, Joachim

2015-01-01

To optimise health for space missions, European astronauts follow specific conditioning programs before, during and after their flights. To evaluate the effectiveness of these programs, the European Space Agency conducts an Astronaut Fitness Assessment (AFA), but the test-retest reliability of elements within it remains unexamined. The reliability study described here presents a scientific basis for implementing the AFA, but also highlights challenges faced by operational teams supporting humans in such unique environments, especially with respect to health and fitness monitoring of crew members travelling not only into space, but also across the world. The AFA tests assessed parameters known to be affected by prolonged exposure to microgravity: aerobic capacity (VO2max), muscular strength (one repetition max, 1 RM) and power (vertical jumps), core stability, flexibility and balance. Intraclass correlation coefficients (ICC3.1), standard error of measurement and coefficient of variation were used to assess relative and absolute test-retest reliability. Squat and bench 1 RM (ICC3.1 = 0.94-0.99), hip flexion (ICC3.1 = 0.99) and left and right handgrip strength (ICC3.1 = 0.95 and 0.97), showed the highest test-retest reliability, followed by VO2max (ICC3.1 = 0.91), core strength (ICC3.1 = 0.78-0.89), hip extension (ICC3.1 = 0.63), the countermeasure (ICC3.1 = 0.76) and squat (ICC3.1 = 0.63) jumps, and single right- and left-leg jump height (ICC3.1 = 0.51 and 0.14). For balance, relative reliability ranged from ICC3.1 = 0.78 for path length (two legs, head tilted back, eyes open) to ICC3.1 = 0.04 for average rotation velocity (one leg, eyes closed). In a small sample (n = 8) of young, healthy individuals, the AFA battery of tests demonstrated acceptable test-retest reliability for most parameters except some balance and single-leg jump tasks. These findings suggest that, for the application with astronauts, most AFA tests appear appropriate to be maintained in the test battery, but that some elements may be unreliable, and require either modification (duration, selection of task) or removal (single-leg jump, balance test on sphere) from the battery. The test battery is mobile and universally applicable for occupational and general fitness assessment by its comprehensive composition of tests covering many systems involved in whole body movement.

Report: Assessment of EPA’s Projected Pollutant Reductions Resulting from Enforcement Actions and Settlements

EPA Pesticide Factsheets

Report #2007-B-00002, July 24, 2007. The accuracy and reliability of EPA’s projected pollutant reductions for Fiscal Years 2003-2006 were dependent on the specific program in which the enforcement action took place.

Reliability analysis and utilization of PEMs in space application

NASA Astrophysics Data System (ADS)

Jiang, Xiujie; Wang, Zhihua; Sun, Huixian; Chen, Xiaomin; Zhao, Tianlin; Yu, Guanghua; Zhou, Changyi

2009-11-01

More and more plastic encapsulated microcircuits (PEMs) are used in space missions to achieve high performance. Since PEMs are designed for use in terrestrial operating conditions, the successful usage of PEMs in space harsh environment is closely related to reliability issues, which should be considered firstly. However, there is no ready-made methodology for PEMs in space applications. This paper discusses the reliability for the usage of PEMs in space. This reliability analysis can be divided into five categories: radiation test, radiation hardness, screening test, reliability calculation and reliability assessment. One case study is also presented to illuminate the details of the process, in which a PEM part is used in a joint space program Double-Star Project between the European Space Agency (ESA) and China. The influence of environmental constrains including radiation, humidity, temperature and mechanics on the PEM part has been considered. Both Double-Star Project satellites are still running well in space now.

CONTROL OF STREPTOCOCCAL THROAT INFECTIONS IN SCHOOLS—A Cooperative Program Followed in Orange County

PubMed Central

Russell, Edward Lee

1956-01-01

Attempts to identify streptococcal throat infections on clinical evidence alone do not provide an adequate or reliable index of the prevalence of these infections in the community. Epidemiologic information on streptococcal throat infections based on bacteriological identification permits a more accurate assessment of the situation and more logical and more effective control measures. Recent refinements in laboratory procedures have provided a simple, reliable and relatively inexpensive method for the identification of Group A beta hemolytic streptococci by public health or clinical laboratories. In Orange County a program for the identification of streptococcal throat infections by cooperative action of the medical profession, the health department and the school authorities greatly aided in control of the disease. A voluntary health agency (heart association) made an important contribution toward the success of the control program. PMID:13374555

Overview of the program to assess the reliability of emerging nondestructive techniques open testing and study of flaw type effect on NDE response

NASA Astrophysics Data System (ADS)

Meyer, Ryan M.; Komura, Ichiro; Kim, Kyung-cho; Zetterwall, Tommy; Cumblidge, Stephen E.; Prokofiev, Iouri

2016-02-01

In February 2012, the U.S. Nuclear Regulatory Commission (NRC) executed agreements with VTT Technical Research Centre of Finland, Nuclear Regulatory Authority of Japan (NRA, former JNES), Korea Institute of Nuclear Safety (KINS), Swedish Radiation Safety Authority (SSM), and Swiss Federal Nuclear Safety Inspectorate (ENSI) to establish the Program to Assess the Reliability of Emerging Nondestructive Techniques (PARENT). The goal of PARENT is to investigate the effectiveness of current emerging and perspective novel nondestructive examination procedures and techniques to find flaws in nickel-alloy welds and base materials. This is done by conducting a series of open and blind international round-robin tests on a set of large-bore dissimilar metal welds (LBDMW), small-bore dissimilar metal welds (SBDMW), and bottom-mounted instrumentation (BMI) penetration weld test blocks. The purpose of blind testing is to study the reliability of more established techniques and included only qualified teams and procedures. The purpose of open testing is aimed at a more basic capability assessment of emerging and novel technologies. The range of techniques applied in open testing varied with respect to maturity and performance uncertainty and were applied to a variety of simulated flaws. This paper will include a brief overview of the PARENT blind and open testing techniques and test blocks and present some of the blind testing results.

Psychometric characteristics of process evaluation measures for a rural school-based childhood obesity prevention study: Louisiana Health.

PubMed

Newton, Robert L; Thomson, Jessica L; Rau, Kristi K; Ragusa, Shelly A; Sample, Alicia D; Singleton, Nakisha N; Anton, Stephen D; Webber, Larry S; Williamson, Donald A

2011-01-01

To evaluate the implementation of intervention components of the Louisiana Health study, which was a multicomponent childhood obesity prevention program conducted in rural schools. Content analysis. Process evaluation assessed implementation in classrooms, gym classes, and cafeterias. Classroom teachers (n  =  232), physical education teachers (n  =  53), food service managers (n  =  33), and trained observers (n  =  9). Five process evaluation measures were created: Physical Education Questionnaire (PEQ), Intervention Questionnaire (IQ), Food Service Manager Questionnaire (FSMQ), Classroom Observation (CO), and School Nutrition Environment Observation (SNEO). Interrater reliability and internal consistency were assessed on all measures. Analysis of variance and χ(2) were used to compare differences across study groups on questionnaires and observations. The PEQ and one subscale from the FSMQ were eliminated because their reliability coefficients fell below acceptable standards. The subscale internal consistencies for the IQ, FSMQ, CO, and SNEO (all Cronbach α > .60) were acceptable. After the initial 4 months of intervention, there was evidence that the Louisiana Health intervention was being implemented as it was designed. In summary, four process evaluation measures were found to be sufficiently reliable and valid for assessing the delivery of various aspects of a school-based obesity prevention program. These process measures could be modified to evaluate the delivery of other similar school-based interventions.

Psychometric properties of the Peer Proficiency Assessment (PEPA): a tool for evaluation of undergraduate peer counselors' motivational interviewing fidelity.

PubMed

Mastroleo, Nadine R; Mallett, Kimberly A; Turrisi, Rob; Ray, Anne E

2009-09-01

Despite the expanding use of undergraduate student peer counseling interventions aimed at reducing college student drinking, few programs evaluate peer counselors' competency to conduct these interventions. The present research describes the development and psychometric assessments of the Peer Proficiency Assessment (PEPA), a new tool for examining Motivational Interviewing adherence in undergraduate student peer delivered interventions. Twenty peer delivered sessions were evaluated by master and undergraduate student coders using a cross-validation design to examine peer based alcohol intervention sessions. Assessments revealed high inter-rater reliability between student and master coders and good correlations between previously established fidelity tools. Findings lend support for the use of the PEPA to examine peer counselor competency. The PEPA, training for use, inter-rater reliability information, construct and predictive validity, and tool usefulness are described.

The Child Suicide Risk Assessment: A Screening Measure of Suicide Risk in Pre-Adolescents

ERIC Educational Resources Information Center

Larzelere, Robert E.; Andersen, Jamie J.; Ringle, Jay L.; Jorgensen, Dan D.

2004-01-01

This study documents the initial reliability and validity of the Child Suicide Risk Assessment (CSRA) for children under the age of 13. The revised CSRA retained 18 of 20 original items based on item-specific psychometric data from 140 pre-adolescents in out-of-home treatment programs. The CSRA demonstrated adequate internal consistency (alpha =…

The Development and Validation of an Alternative Assessment to Measure Changes in Understanding of the Longleaf Pine Ecosystem

ERIC Educational Resources Information Center

Dentzau, Michael W.; Martínez, Alejandro José Gallard

2016-01-01

A drawing assessment to gauge changes in fourth grade students' understanding of the essential components of the longleaf pine ecosystem was developed to support an out-of-school environmental education program. Pre- and post-attendance drawings were scored with a rubric that was determined to have content validity and reliability among users. In…

Turkish Teachers' and Students' Perceptions towards Computer Assisted Testing in Comparison with Spanish Teachers' and Students' Perceptions

ERIC Educational Resources Information Center

Berber, Aslihan; García Laborda, Jesús

2015-01-01

There are different opinions about using technology in the assessment field of education regarding computer-assisted assessments. People have some concerns such as its application, reliability and so on. It seems that those concerns may decrease with the developing technology in the following years since computer-based testing programs are…

Validity, Reliability, and Equity Issues in an Observational Talent Assessment Process in the Performing Arts

ERIC Educational Resources Information Center

Oreck, Barry A.; Owen, Steven V.; Baum, Susan M.

2003-01-01

The lack of valid, research-based methods to identify potential artistic talent hampers the inclusion of the arts in programs for the gifted and talented. The Talent Assessment Process in Dance, Music, and Theater (D/M/T TAP) was designed to identify potential performing arts talent in diverse populations, including bilingual and special education…

Development and Psychometric Testing of the Caregiver Communication Competence Scale in Patients With Dementia.

PubMed

Chao, Hui-Chen; Yang, Ya-Ping; Huang, Mei-Chih; Wang, Jing-Jy

2016-01-01

Appropriate communication skills are essential for understanding patient needs, particularly those of patients with dementia. Assessing health care providers' competence in communicating with patients with dementia is critical for planning a communication education program. However, no formally established scale can be used. The purpose of the current study was to develop a valid and reliable instrument for determining the communication competence of health care providers with patients with dementia. Through use of a literature review and previous clinical experience, an initial 28-item scale was developed to assess the frequency of use of each item by health care providers. Fourteen items were extracted and three factors were distinguished. Results indicated that the internal consistency reliability of the 14-item scale was 0.84. Favorable convergent and discriminant validities were reached. The communication competence scale provides administrators or educators with a useful tool for assessing communication competence of health care providers when interacting with patients with dementia so a suitable education program can be planned and implemented. Copyright 2016, SLACK Incorporated.

Risk management for the Space Exploration Initiative

NASA Technical Reports Server (NTRS)

Buchbinder, Ben

1993-01-01

Probabilistic Risk Assessment (PRA) is a quantitative engineering process that provides the analytic structure and decision-making framework for total programmatic risk management. Ideally, it is initiated in the conceptual design phase and used throughout the program life cycle. Although PRA was developed for assessment of safety, reliability, and availability risk, it has far greater application. Throughout the design phase, PRA can guide trade-off studies among system performance, safety, reliability, cost, and schedule. These studies are based on the assessment of the risk of meeting each parameter goal, with full consideration of the uncertainties. Quantitative trade-off studies are essential, but without full identification, propagation, and display of uncertainties, poor decisions may result. PRA also can focus attention on risk drivers in situations where risk is too high. For example, if safety risk is unacceptable, the PRA prioritizes the risk contributors to guide the use of resources for risk mitigation. PRA is used in the Space Exploration Initiative (SEI) Program. To meet the stringent requirements of the SEI mission, within strict budgetary constraints, the PRA structure supports informed and traceable decision-making. This paper briefly describes the SEI PRA process.

Towards an Operational Definition of Clinical Competency in Pharmacy

PubMed Central

2015-01-01

Objective. To estimate the inter-rater reliability and accuracy of ratings of competence in student pharmacist/patient clinical interactions as depicted in videotaped simulations and to compare expert panelist and typical preceptor ratings of those interactions. Methods. This study used a multifactorial experimental design to estimate inter-rater reliability and accuracy of preceptors’ assessment of student performance in clinical simulations. The study protocol used nine 5-10 minute video vignettes portraying different levels of competency in student performance in simulated clinical interactions. Intra-Class Correlation (ICC) was used to calculate inter-rater reliability and Fisher exact test was used to compare differences in distribution of scores between expert and nonexpert assessments. Results. Preceptors (n=42) across 5 states assessed the simulated performances. Intra-Class Correlation estimates were higher for 3 nonrandomized video simulations compared to the 6 randomized simulations. Preceptors more readily identified high and low student performances compared to satisfactory performances. In nearly two-thirds of the rating opportunities, a higher proportion of expert panelists than preceptors rated the student performance correctly (18 of 27 scenarios). Conclusion. Valid and reliable assessments are critically important because they affect student grades and formative student feedback. Study results indicate the need for pharmacy preceptor training in performance assessment. The process demonstrated in this study can be used to establish minimum preceptor benchmarks for future national training programs. PMID:26089563

A systematic review of evaluated suicide prevention programs targeting indigenous youth.

PubMed

Harlow, Alyssa F; Bohanna, India; Clough, Alan

2014-01-01

Indigenous young people have significantly higher suicide rates than their non-indigenous counterparts. There is a need for culturally appropriate and effective suicide prevention programs for this demographic. This review assesses suicide prevention programs that have been evaluated for indigenous youth in Australia, Canada, New Zealand, and the United States. The databases MEDLINE and PsycINFO were searched for publications on suicide prevention programs targeting indigenous youth that include reports on evaluations and outcomes. Program content, indigenous involvement, evaluation design, program implementation, and outcomes were assessed for each article. The search yielded 229 articles; 90 abstracts were assessed, and 11 articles describing nine programs were reviewed. Two Australian programs and seven American programs were included. Programs were culturally tailored, flexible, and incorporated multiple-levels of prevention. No randomized controlled trials were found, and many programs employed ad hoc evaluations, poor program description, and no process evaluation. Despite culturally appropriate content, the results of the review indicate that more controlled study designs using planned evaluations and valid outcome measures are needed in research on indigenous youth suicide prevention. Such changes may positively influence the future of research on indigenous youth suicide prevention as the outcomes and efficacy will be more reliable.

Transit Reliability Information Program : PATCO-WMATA Propulsion System Reliability/Productivity Analysis

DOT National Transportation Integrated Search

1984-10-01

The Transit Reliability Information Program (TRIP) is a government-initiated program to assist the transit industry in satisfying its need for transit reliability information. TRIP provides this assistance through the operation of a national data ban...

Shuttle Risk Progression by Flight

NASA Technical Reports Server (NTRS)

Hamlin, Teri; Kahn, Joe; Thigpen, Eric; Zhu, Tony; Lo, Yohon

2011-01-01

Understanding the early mission risk and progression of risk as a vehicle gains insights through flight is important: . a) To the Shuttle Program to understand the impact of re-designs and operational changes on risk. . b) To new programs to understand reliability growth and first flight risk. . Estimation of Shuttle Risk Progression by flight: . a) Uses Shuttle Probabilistic Risk Assessment (SPRA) and current knowledge to calculate early vehicle risk. . b) Shows impact of major Shuttle upgrades. . c) Can be used to understand first flight risk for new programs.

Study on application of aerospace technology to improve surgical implants

NASA Technical Reports Server (NTRS)

Johnson, R. E.; Youngblood, J. L.

1982-01-01

The areas where aerospace technology could be used to improve the reliability and performance of metallic, orthopedic implants was assessed. Specifically, comparisons were made of material controls, design approaches, analytical methods and inspection approaches being used in the implant industry with hardware for the aerospace industries. Several areas for possible improvement were noted such as increased use of finite element stress analysis and fracture control programs on devices where the needs exist for maximum reliability and high structural performance.

77 FR 71787 - Agency Information Collection Extension

Federal Register 2010, 2011, 2012, 2013, 2014

2012-12-04

... annual supervisory review, medical assessment, management evaluation, and a DOE personnel security review... explosive duties do not have emotional, mental, or physical conditions that could result in an accidental or.... 1910-5122; (2) Information Collection Request Title: Human Reliability Program; (3) Type of Review...

Transit Reliability Information Program : Reliability Verification Demonstration Plan for Rapid Rail Vehicles

DOT National Transportation Integrated Search

1981-08-01

The Transit Reliability Information Program (TRIP) is a government-initiated program to assist the transit industry in satisfying its need for transit reliability information. TRIP provides this assistance through the operation of a national Data Ban...

Probabilistic Structural Analysis Methods (PSAM) for select space propulsion system components, part 2

NASA Technical Reports Server (NTRS)

1991-01-01

The technical effort and computer code enhancements performed during the sixth year of the Probabilistic Structural Analysis Methods program are summarized. Various capabilities are described to probabilistically combine structural response and structural resistance to compute component reliability. A library of structural resistance models is implemented in the Numerical Evaluations of Stochastic Structures Under Stress (NESSUS) code that included fatigue, fracture, creep, multi-factor interaction, and other important effects. In addition, a user interface was developed for user-defined resistance models. An accurate and efficient reliability method was developed and was successfully implemented in the NESSUS code to compute component reliability based on user-selected response and resistance models. A risk module was developed to compute component risk with respect to cost, performance, or user-defined criteria. The new component risk assessment capabilities were validated and demonstrated using several examples. Various supporting methodologies were also developed in support of component risk assessment.

10 CFR 712.1 - Purpose.

Code of Federal Regulations, 2011 CFR

2011-01-01

... HUMAN RELIABILITY PROGRAM Establishment of and Procedures for the Human Reliability Program General Provisions § 712.1 Purpose. This part establishes the policies and procedures for a Human Reliability Program... judgment and reliability may be impaired by physical or mental/personality disorders, alcohol abuse, use of...

Management of the aging of critical safety-related concrete structures in light-water reactor plants

DOE Office of Scientific and Technical Information (OSTI.GOV)

Naus, D.J.; Oland, C.B.; Arndt, E.G.

1990-01-01

The Structural Aging Program has the overall objective of providing the USNRC with an improved basis for evaluating nuclear power plant safety-related structures for continued service. The program consists of a management task and three technical tasks: materials property data base, structural component assessment/repair technology, and quantitative methodology for continued-service determinations. Objectives, accomplishments, and planned activities under each of these tasks are presented. Major program accomplishments include development of a materials property data base for structural materials as well as an aging assessment methodology for concrete structures in nuclear power plants. Furthermore, a review and assessment of inservice inspection techniquesmore » for concrete materials and structures has been complete, and work on development of a methodology which can be used for performing current as well as reliability-based future condition assessment of concrete structures is well under way. 43 refs., 3 tabs.« less

Inter-operator and inter-device agreement and reliability of the SEM Scanner.

PubMed

Clendenin, Marta; Jaradeh, Kindah; Shamirian, Anasheh; Rhodes, Shannon L

2015-02-01

The SEM Scanner is a medical device designed for use by healthcare providers as part of pressure ulcer prevention programs. The objective of this study was to evaluate the inter-rater and inter-device agreement and reliability of the SEM Scanner. Thirty-one (31) volunteers free of pressure ulcers or broken skin at the sternum, sacrum, and heels were assessed with the SEM Scanner. Each of three operators utilized each of three devices to collect readings from four anatomical sites (sternum, sacrum, left and right heels) on each subject for a total of 108 readings per subject collected over approximately 30 min. For each combination of operator-device-anatomical site, three SEM readings were collected. Inter-operator and inter-device agreement and reliability were estimated. Over the course of this study, more than 3000 SEM Scanner readings were collected. Agreement between operators was good with mean differences ranging from -0.01 to 0.11. Inter-operator and inter-device reliability exceeded 0.80 at all anatomical sites assessed. The results of this study demonstrate the high reliability and good agreement of the SEM Scanner across different operators and different devices. Given the limitations of current methods to prevent and detect pressure ulcers, the SEM Scanner shows promise as an objective, reliable tool for assessing the presence or absence of pressure-induced tissue damage such as pressure ulcers. Copyright © 2015 Bruin Biometrics, LLC. Published by Elsevier Ltd.. All rights reserved.

A Psychometric Assessment of the "Businessweek," "U.S. News & World Report," and "Financial Times" Rankings of Business Schools' MBA Programs

ERIC Educational Resources Information Center

Iacobucci, Dawn

2013-01-01

This research investigates the reliability and validity of three major publications' rankings of MBA programs. Each set of rankings showed reasonable consistency over time, both at the level of the overall rankings and for most of the facets from which the rankings are derived. Each set of rankings also showed some levels of convergent and…

Candidate Technologies for the Integrated Health Management Program

NASA Technical Reports Server (NTRS)

Johnson, Neal F., Jr.; Martin, Fred H.

1993-01-01

The purpose of this report is to assess Vehicle Health Management (VHM) technologies for implementation as a demonstration. Extensive studies have been performed to determine technologies which could be implemented on the Atlas and Centaur vehicles as part of a bridging program. This paper discusses areas today where VHM can be implemented for benefits in reliability, performance, and cost reduction. VHM Options are identified and one demonstration is recommended for execution.

Interim reliability-evaluation program: analysis of the Browns Ferry, Unit 1, nuclear plant. Appendix B - system descriptions and fault trees

DOE Office of Scientific and Technical Information (OSTI.GOV)

Mays, S.E.; Poloski, J.P.; Sullivan, W.H.

1982-07-01

This report describes a risk study of the Browns Ferry, Unit 1, nuclear plant. The study is one of four such studies sponsored by the NRC Office of Research, Division of Risk Assessment, as part of its Interim Reliability Evaluation Program (IREP), Phase II. This report is contained in four volumes: a main report and three appendixes. Appendix B provides a description of Browns Ferry, Unit 1, plant systems and the failure evaluation of those systems as they apply to accidents at Browns Ferry. Information is presented concerning front-line system fault analysis; support system fault analysis; human error models andmore » probabilities; and generic control circuit analyses.« less

10 CFR 712.30 - Applicability.

Code of Federal Regulations, 2010 CFR

2010-01-01

... 10 Energy 4 2010-01-01 2010-01-01 false Applicability. 712.30 Section 712.30 Energy DEPARTMENT OF ENERGY HUMAN RELIABILITY PROGRAM Medical Standards § 712.30 Applicability. This subpart establishes standards and procedures for conducting medical assessments of DOE and DOE contractor individuals in HRP...

Issues in NASA Program and Project Management: Focus on Project Planning and Scheduling

NASA Technical Reports Server (NTRS)

Hoffman, Edward J. (Editor); Lawbaugh, William M. (Editor)

1997-01-01

Topics addressed include: Planning and scheduling training for working project teams at NASA, overview of project planning and scheduling workshops, project planning at NASA, new approaches to systems engineering, software reliability assessment, and software reuse in wind tunnel control systems.

Language-Specific Attention Treatment for Aphasia: Description and Preliminary Findings.

PubMed

Peach, Richard K; Nathan, Meghana R; Beck, Katherine M

2017-02-01

The need for a specific, language-based treatment approach to aphasic impairments associated with attentional deficits is well documented. We describe language-specific attention treatment, a specific skill-based approach for aphasia that exploits increasingly complex linguistic tasks that focus attention. The program consists of eight tasks, some with multiple phases, to assess and treat lexical and sentence processing. Validation results demonstrate that these tasks load on six attentional domains: (1) executive attention; (2) attentional switching; (3) visual selective attention/processing speed; (4) sustained attention; (5) auditory-verbal working memory; and (6) auditory processing speed. The program demonstrates excellent inter- and intrarater reliability and adequate test-retest reliability. Two of four people with aphasia exposed to this program demonstrated good language recovery whereas three of the four participants showed improvements in auditory-verbal working memory. The results provide support for this treatment program in patients with aphasia having no greater than a moderate degree of attentional impairment. Thieme Medical Publishers 333 Seventh Avenue, New York, NY 10001, USA.

Systems Analysis Programs for Hands-on Integrated Reliability Evaluations (SAPHIRE) Tutorial

DOE Office of Scientific and Technical Information (OSTI.GOV)

C. L. Smith; S. T. Beck; S. T. Wood

2008-08-01

The Systems Analysis Programs for Hands-on Integrated Reliability Evaluations (SAPHIRE) refers to a set of computer programs that were developed to create and analyze probabilistic risk assessment (PRAs). This volume is the tutorial manual for the SAPHIRE system. In this document, a series of lessons are provided that guide the user through basic steps common to most analyses preformed with SAPHIRE. The tutorial is divided into two major sections covering both basic and advanced features. The section covering basic topics contains lessons that lead the reader through development of a probabilistic hypothetical problem involving a vehicle accident, highlighting the program’smore » most fundamental features. The advanced features section contains additional lessons that expand on fundamental analysis features of SAPHIRE and provide insights into more complex analysis techniques. Together, these two elements provide an overview into the operation and capabilities of the SAPHIRE software.« less

Drug utilization review: mechanisms to improve its effectiveness and broaden its scope. The U.S. Pharmacopeia Drug Utilization Review Advisory Panel.

PubMed

2000-01-01

To address important problems and needed changes in online and retrospective drug utilization review (DUR) programs. Emphasis is placed on reliability of DUR criteria and the shift of traditional retrospective DUR programs toward disease management and health care outcomes. Published literature evaluating the role of online and retrospective DUR programs. Particular attention was given to studies assessing DUR criteria reliability and new interventions with retrospective DUR programs. A literature review was conducted along with an expert summary from the U.S. Pharmacopeia Drug Utilization Review Advisory Panel. Studies have revealed variations in DUR criteria that could be affecting clinical practice and patient care. Appropriate formal methodologies and use of consistent procedures in developing online prospective DUR programs and systems could help resolve these problems. Traditional retrospective DUR is also shifting to incorporate disease management and methodologies from health outcomes and pharmacoeconomics studies. Refinements are needed to improve the reliability and validity of online DUR criteria and to minimize false positive messages. Databases created as a result of DUR efforts have been used in new and innovative ways to incorporate health outcomes data and disease management interventions. Additional outcomes data, combined with quality assurance efforts, should increase the utility of DUR/disease management efforts in evaluating health systems while improving the effectiveness and efficiency of pharmacists' health care interventions.

The new GRID Hamilton Rating Scale for Depression demonstrates excellent inter-rater reliability for inexperienced and experienced raters before and after training.

PubMed

Tabuse, Hideaki; Kalali, Amir; Azuma, Hideki; Ozaki, Norio; Iwata, Nakao; Naitoh, Hiroshi; Higuchi, Teruhiko; Kanba, Shigenobu; Shioe, Kunihiko; Akechi, Tatsuo; Furukawa, Toshi A

2007-09-30

The Hamilton Rating Scale for Depression (HAMD) is the de facto international gold standard for the assessment of depression. There are some criticisms, however, especially with regard to its inter-rater reliability, due to the lack of standardized questions or explicit scoring procedures. The GRID-HAMD was developed to provide standardized explicit scoring conventions and a structured interview guide for administration and scoring of the HAMD. We developed the Japanese version of the GRID-HAMD and examined its inter-rater reliability among experienced and inexperienced clinicians (n=70), how rater characteristics may affect it, and how training can improve it in the course of a model training program using videotaped interviews. The results showed that the inter-rater reliability of the GRID-HAMD total score was excellent to almost perfect and those of most individual items were also satisfactory to excellent, both with experienced and inexperienced raters, and both before and after the training. With its standardized definitions, questions and detailed scoring conventions, the GRID-HAMD appears to be the best achievable set of interview guides for the HAMD and can provide a solid tool for highly reliable assessment of depression severity.

Physical activity and healthy eating environmental audit tools in youth care settings: A systematic review.

PubMed

Ajja, Rahma; Beets, Michael W; Chandler, Jessica; Kaczynski, Andrew T; Ward, Dianne S

2015-08-01

There is a growing interest in evaluating the physical activity (PA) and healthy eating (HE) policy and practice environment characteristics in settings frequented by youth (≤18years). This review evaluates the measurement properties of audit tools designed to assess PA and HE policy and practice environment characteristics in settings that care for youth (e.g., childcare, school, afterschool, summer camp). Three electronic databases, reference lists, educational department and national health organizations' web pages were searched between January 1980 and February 2014 to identify tools assessing PA and/or HE policy and practice environments in settings that care for youth (≤18years). Sixty-five audit tools were identified of which 53 individual tools met the inclusion criteria. Thirty-three tools assessed both the PA and HE domains, 6 assessed the PA domain and 14 assessed the HE domain solely. The majority of the tools were self-assessment tools (n=40), and were developed to assess the PA and/or HE environment in school settings (n=33), childcare (n=12), and after school programs (n=4). Four tools assessed the community at-large and had sections for assessing preschool, school and/or afterschool settings within the tool. The majority of audit tools lacked validity and/or reliability data (n=42). Inter-rater reliability and construct validity were the most frequently reported reliability (n=7) and validity types (n=5). Limited attention has been given to establishing the reliability and validity of audit tools for settings that care for youth. Future efforts should be directed towards establishing a strong measurement foundation for these important environmental audit tools. Published by Elsevier Inc.

Effects of back posture education on elementary schoolchildren's back function.

PubMed

Geldhof, Elisabeth; Cardon, Greet; De Bourdeaudhuij, Ilse; Danneels, Lieven; Coorevits, Pascal; Vanderstraeten, Guy; De Clercq, Dirk

2007-06-01

The possible effects of back education on children's back function were never evaluated. Therefore, main aim of the present study was to evaluate the effects of back education in elementary schoolchildren on back function parameters. Since the reliability of back function measurement in children is poorly defined, another objective was to test the selected instruments for reliability in 8-11-year olds. The multi-factorial intervention lasting two school-years consisted of a back education program and the stimulation of postural dynamism in the class. Trunk muscle endurance, leg muscle capacity and spinal curvature were evaluated in a pre-post design including 41 children who received the back education program (mean age at post-test: 11.2 +/- 0.9 years) and 28 controls (mean age at post-test: 11.4 +/- 0.6 years). Besides, test-retest reliability with a 1-week interval was investigated in a separate sample. Therefore, 47 children (mean age: 10.1 +/- 0.5 years) were tested for reliability of trunk muscle endurance and 40 children (mean age: 10.2 +/- 0.7 years) for the assessment of spinal curvatures. Reliability of endurance testing was very good to good for the trunk flexors (ICC = 0.82) and trunk extensors (ICC = 0.63). The assessment of the thoracic (ICC = 0.69) and the lumbar curvature (ICC = 0.52) in seating position showed good to acceptable reliability. Low ICCs were found for the assessment of the thoracic (ICC = 0.39) and the lumbar curvature (ICC = 0.37) in stance. The effects of 2 year back education showed an increase in trunk flexor endurance in the intervention group compared to a decrease in the controls and a trend towards significance for a higher increase in trunk extensor endurance in the intervention group. For leg muscle capacity and spinal curvature no intervention effects were found. The small samples recommend cautious interpretation of intervention effects. However, the present study's findings favor the implementation of back education with focus on postural dynamism in the class as an integral part of the elementary school curriculum in the scope of optimizing spinal loading through the school environment.

From a formal training program in musculoskeletal ultrasound (MSUS) to a high reproducibility for Doppler ultrasound in rheumatoid arthritis.

PubMed

Villota, Orlando; Diaz, Mario; Ceron, Carmen; Moller, Ingrid; Naredo, Esperanza; Saaibi, Diego Luis

2017-07-28

To assess the intra- and inter-observer reliability of ultrasound (US) in scoring B-mode, Doppler synovitis and combined B-mode and Doppler synovitis scores in different peripheral joints of rheumatoid arthritis (RA) patients. Four rheumatologists with a formal training in musculoskeletal US (MSKUS) particularly focus on definitions and scoring synovitis on B-mode and Doppler mode participated in a patient-based reliability exercise on 16 active RA patients. The four rheumatologists independently and consecutively performed a B-mode and power Doppler (PD) US assessment of 7 joints of each patient in two rounds in a blinded fashion. Each joint was semi quantitatively scored from 0 to 3 for B-mode synovitis (BS), Doppler synovitis (DS), and combined B-mode/Doppler synovitis (CS). Intraobserver reliability was assessed by Cohen's κ. Interobserver reliability was assessed by unweight Light's κ. The mean prevalence of synovitis on B-mode was 83% of joints; scores ranging from grade 1 in 18% of joints, to grade 3 in 33%. In 55% of joints synovial PD signal was detected and the distribution of scores range from 14% of joints for grade 3, to 26% for grade 2. After a total of 448 joints scanned with 896 adquired images our intraobserver and interobserver reliability was good to excellent for most of the joints. Formal, structured and continuous training in musculoskeletal ultrasound would bring a good to excellent reproducibility in rheumatological hands with a high reliability in real time acquisition BS, DS and CS modalities for scoring synovitis in patients with active rheumatoid arthritis. Copyright© Bentham Science Publishers; For any queries, please email at epub@benthamscience.org.

Reliability and validity of the Performance Recorder 1 for measuring isometric knee flexor and extensor strength.

PubMed

Neil, Sarah E; Myring, Alec; Peeters, Mon Jef; Pirie, Ian; Jacobs, Rachel; Hunt, Michael A; Garland, S Jayne; Campbell, Kristin L

2013-11-01

Muscular strength is a key parameter of rehabilitation programs and a strong predictor of functional capacity. Traditional methods to measure strength, such as manual muscle testing (MMT) and hand-held dynamometry (HHD), are limited by the strength and experience of the tester. The Performance Recorder 1 (PR1) is a strength assessment tool attached to resistance training equipment and may be a time- and cost-effective tool to measure strength in clinical practice that overcomes some limitations of MMT and HHD. However, reliability and validity of the PR1 have not been reported. Test-retest and inter-rater reliability was assessed using the PR1 in healthy adults (n  =  15) during isometric knee flexion and extension. Criterion-related validity was assessed through comparison of values obtained from the PR1 and Biodex® isokinetic dynamometer. Test-retest reliability was excellent for peak knee flexion (intra-class correlation coefficient [ICC] of 0.96, 95% CI: 0.85, 0.99) and knee extension (ICC  =  0.96, 95% CI: 0.87, 0.99). Inter-rater reliability was also excellent for peak knee flexion (ICC  =  0.95, 95% CI: 0.85, 0.99) and peak knee extension (ICC  =  0.97, 95% CI: 0.91, 0.99). Validity was moderate for peak knee flexion (ICC  =  0.75, 95% CI: 0.38, 0.92) but poor for peak knee extension (ICC  =  0.37, 95% CI: 0, 0.73). The PR1 provides a reliable measure of isometric knee flexor and extensor strength in healthy adults that could be used in the clinical setting, but absolute values may not be comparable to strength assessment by gold-standard measures.

[Evaluation of patient satisfaction after stroke rehabilitation program. Validation study for the Spanish version of the Satisfaction Pound Scale].

PubMed

Aguirrezabal Juaristi, Aizpea; Ferrer Fores, Montse; Marco Navarro, Ester; Mojal García, Sergi; Vilagut Saiz, Gemma; Duarte Oller, Esther

2016-11-18

The Satisfaction Pound Scale is a specific questionnaire to evaluate satisfaction with the rehabilitation program after a stroke. The aim of this study was to adapt this scale to Spanish and to evaluate its metric characteristics. The adaptation included translation and back-translation methods. Metric characteristics were evaluated in 74 patients, all of whom were administered the Satisfaction Pound Scale and the Short Form 36 (SF-36). The statistical model was tested by confirmatory factor analysis (CFA). Reliability was determined through Cronbach alpha coefficient and a test-retest procedure. Construct validity was assessed by means of correlations between the satisfaction scale and the SF-36. Adjustment indicators in the CFA were very good. Reproducibility test showed correlations higher than 0.85, and all correlations between SF-36 dimensions and the satisfaction scale were lower than 0.2, in accordance with the hypotheses raised. The Spanish version of the Satisfaction Pounds Scale is reliable and valid, therefore it is a useful tool to assess satisfaction with the post-stroke rehabilitation program in our area. Copyright © 2016 Elsevier España, S.L.U. All rights reserved.

Approach to developing reliable space reactor power systems

NASA Technical Reports Server (NTRS)

Mondt, Jack F.; Shinbrot, Charles H.

1991-01-01

During Phase II, the Engineering Development Phase, the SP-100 Project has defined and is pursuing a new approach to developing reliable power systems. The approach to developing such a system during the early technology phase is described along with some preliminary examples to help explain the approach. Developing reliable components to meet space reactor power system requirements is based on a top-down systems approach which includes a point design based on a detailed technical specification of a 100-kW power system. The SP-100 system requirements implicitly recognize the challenge of achieving a high system reliability for a ten-year lifetime, while at the same time using technologies that require very significant development efforts. A low-cost method for assessing reliability, based on an understanding of fundamental failure mechanisms and design margins for specific failure mechanisms, is being developed as part of the SP-100 Program.

Advanced Launch System Multi-Path Redundant Avionics Architecture Analysis and Characterization

NASA Technical Reports Server (NTRS)

Baker, Robert L.

1993-01-01

The objective of the Multi-Path Redundant Avionics Suite (MPRAS) program is the development of a set of avionic architectural modules which will be applicable to the family of launch vehicles required to support the Advanced Launch System (ALS). To enable ALS cost/performance requirements to be met, the MPRAS must support autonomy, maintenance, and testability capabilities which exceed those present in conventional launch vehicles. The multi-path redundant or fault tolerance characteristics of the MPRAS are necessary to offset a reduction in avionics reliability due to the increased complexity needed to support these new cost reduction and performance capabilities and to meet avionics reliability requirements which will provide cost-effective reductions in overall ALS recurring costs. A complex, real-time distributed computing system is needed to meet the ALS avionics system requirements. General Dynamics, Boeing Aerospace, and C.S. Draper Laboratory have proposed system architectures as candidates for the ALS MPRAS. The purpose of this document is to report the results of independent performance and reliability characterization and assessment analyses of each proposed candidate architecture and qualitative assessments of testability, maintainability, and fault tolerance mechanisms. These independent analyses were conducted as part of the MPRAS Part 2 program and were carried under NASA Langley Research Contract NAS1-17964, Task Assignment 28.

Assessment, development, and testing of glass for blast environments.

DOE Office of Scientific and Technical Information (OSTI.GOV)

Glass, Sarah Jill

2003-06-01

Glass can have lethal effects including fatalities and injuries when it breaks and then flies through the air under blast loading (''the glass problem''). One goal of this program was to assess the glass problem and solutions being pursued to mitigate it. One solution to the problem is the development of new glass technology that allows the strength and fragmentation to be controlled or selected depending on the blast performance specifications. For example the glass could be weak and fail, or it could be strong and survive, but it must perform reliably. Also, once it fails it should produce fragmentsmore » of a controlled size. Under certain circumstances it may be beneficial to have very small fragments, in others it may be beneficial to have large fragments that stay together. The second goal of this program was to evaluate the performance (strength, reliability, and fragmentation) of Engineered Stress Profile (ESP) glass under different loading conditions. These included pseudo-static strength and pressure tests and free-field blast tests. The ultimate goal was to provide engineers and architects with a glass whose behavior under blast loading is less lethal. A near-term benefit is a new approach for improving the reliability of glass and modifying its fracture behavior.« less

A Validation Study of the "School Leader Dispositions Inventory"[C

ERIC Educational Resources Information Center

Melton, Teri Denlea; Tysinger, Dawn; Mallory, Barbara; Green, James

2011-01-01

Although university-based school administrator preparation programs are required by accreditation agencies to assess the dispositions of candidates, valid and reliable methods for doing so remain scarce. "The School Leaders Disposition Inventory"[C] (SDLI) is proposed as an instrument that has promise for identifying leadership…

Development and validation of a 6-point grading scale in patients undergoing correction of nasolabial folds with a collagen implant.

PubMed

Monheit, Gary D; Gendler, Ellen C; Poff, Bradley; Fleming, Laura; Bachtell, Nathan; Garcia, Emily; Burkholder, David

2010-11-01

Various scoring techniques prone to subjective interpretation have been used to evaluate soft tissue augmentation of nasolabial folds (NLFs). To design and validate a reliable wrinkle assessment scoring scale. Six photographed wrinkles of varying severity were electronically copied onto the same facial image to become a 6-point grading scale (GGS). A pilot training program (13 investigators) determined reliability, and a 12-week multicenter survey study validated the GGS scoring method. Pilot study inter- and intrarater scoring reliability were high (weighted kappa scores of 0.85 and 0.86, respectively). Seventy-five percent of survey investigators and independent review panel (IRP) members considered a GGS score difference of 0.5 to be a minimally perceivable difference. Interrater weighted kappa scores were 0.91 for the IRP and 0.80 for investigators. Intrarater agreements after repeat testing were 0.91 and 0.89, respectively. The baseline "live" assessment GGS mean score was 3.34, and the baseline blinded photographic assessment GGS mean score was 2.00 for the IRP and 2.16 for the investigators. The GGS is a reproducible method of grading the severity of NLF wrinkles. Treatment effectiveness of a dermal filler can be reliably evaluated using the GGS by comparing "live" assessments with the standard GGS photographic panel. © 2010 by the American Society for Dermatologic Surgery, Inc.

Validity and reliability of a nutrition knowledge survey for assessment in elementary school children.

PubMed

Gower, Jared R; Moyer-Mileur, Laurie J; Wilkinson, Robert D; Slater, Hillarie; Jordan, Kristine C

2010-03-01

Limited surveys are available to assess the nutrition knowledge of children. The goals of this study were to test the validity and reliability of a computer nutrition knowledge survey for elementary school students and to evaluate the impact of the "Fit Kids 'r' Healthy Kids" nutrition intervention via the knowledge survey. During survey development, a sample (n=12) of health educators, elementary school teachers, and registered dietitians assessed the survey. The target population consisted of first- through fourth-grade students from Salt Lake City, UT, metropolitan area schools. Participants were divided into reliability (n=68), intervention (n=74), and control groups (n=59). The reliability group took the survey twice (2 weeks apart); the intervention and control groups also took the survey twice, but at pre- and post-intervention (4 weeks later). Only students from the intervention group participated in four weekly nutrition classes. Reliability was assessed by Pearson's correlation coefficients for knowledge scores. Results demonstrated appropriate content validity, as indicated by expert peer ratings. Test-retest reliability correlations were found to be significant for the overall survey (r=0.54; P<0.001) and for all subscales: food groups, healthful foods, and food functions (r=0.51, 0.65, and 0.49, respectively; P<0.001). Nutrition knowledge was assessed upon program completion with paired samples t tests. Students from the intervention group demonstrated improvement in nutrition knowledge (12.2+/-1.9 to 13.5+/-1.6; P<0.001), while scores for the control group remained unchanged. The difference in total scores from pre- to post-intervention between the two groups was significant (P<0.001). These results suggest that the computerized nutrition survey demonstrated content validity and test-retest reliability for first- through fourth-grade elementary school children. Also, the study results imply that the Fit Kids 'r' Healthy Kids intervention promoted gains in nutrition knowledge. Overall, the computer survey shows promise as an appealing medium for assessing nutrition knowledge in children. Copyright 2010 American Dietetic Association. Published by Elsevier Inc. All rights reserved.

Reliability and Probabilistic Risk Assessment - How They Play Together

NASA Technical Reports Server (NTRS)

Safie, Fayssal M.; Stutts, Richard G.; Zhaofeng, Huang

2015-01-01

PRA methodology is one of the probabilistic analysis methods that NASA brought from the nuclear industry to assess the risk of LOM, LOV and LOC for launch vehicles. PRA is a system scenario based risk assessment that uses a combination of fault trees, event trees, event sequence diagrams, and probability and statistical data to analyze the risk of a system, a process, or an activity. It is a process designed to answer three basic questions: What can go wrong? How likely is it? What is the severity of the degradation? Since 1986, NASA, along with industry partners, has conducted a number of PRA studies to predict the overall launch vehicles risks. Planning Research Corporation conducted the first of these studies in 1988. In 1995, Science Applications International Corporation (SAIC) conducted a comprehensive PRA study. In July 1996, NASA conducted a two-year study (October 1996 - September 1998) to develop a model that provided the overall Space Shuttle risk and estimates of risk changes due to proposed Space Shuttle upgrades. After the Columbia accident, NASA conducted a PRA on the Shuttle External Tank (ET) foam. This study was the most focused and extensive risk assessment that NASA has conducted in recent years. It used a dynamic, physics-based, integrated system analysis approach to understand the integrated system risk due to ET foam loss in flight. Most recently, a PRA for Ares I launch vehicle has been performed in support of the Constellation program. Reliability, on the other hand, addresses the loss of functions. In a broader sense, reliability engineering is a discipline that involves the application of engineering principles to the design and processing of products, both hardware and software, for meeting product reliability requirements or goals. It is a very broad design-support discipline. It has important interfaces with many other engineering disciplines. Reliability as a figure of merit (i.e. the metric) is the probability that an item will perform its intended function(s) for a specified mission profile. In general, the reliability metric can be calculated through the analyses using reliability demonstration and reliability prediction methodologies. Reliability analysis is very critical for understanding component failure mechanisms and in identifying reliability critical design and process drivers. The following sections discuss the PRA process and reliability engineering in detail and provide an application where reliability analysis and PRA were jointly used in a complementary manner to support a Space Shuttle flight risk assessment.

Selenide isotope generator for the Galileo mission. Reliability program plan

DOE Office of Scientific and Technical Information (OSTI.GOV)

Not Available

1978-10-01

The reliability program plan for the Selenide Isotope Generator (SIG) program is presented. It delineates the specific tasks that will be accomplished by Teledyne Energy Systems and its suppliers during design, development, fabrication and test of deliverable Radioisotopic Thermoelectric Generators (RTG), Electrical Heated Thermoelectric Generators (ETG) and associated Ground Support Equipment (GSE). The Plan is formulated in general accordance with procedures specified in DOE Reliability Engineering Program Requirements Publication No. SNS-2, dated June 17, 1974. The Reliability Program Plan presented herein defines the total reliability effort without further reference to Government Specifications. The reliability tasks to be accomplished are delineatedmore » herein and become the basis for contract compliance to the extent specified in the SIG contract Statement of Work.« less

78 FR 49595 - Aviation Rulemaking Advisory Committee-New Task

Federal Register 2010, 2011, 2012, 2013, 2014

2013-08-14

... the new ARAC activity and solicits membership for the Maintenance Reliability Program Working Group... establish the Maintenance Reliability Program Working Group. The working group will serve as staff to ARAC... programs. The Maintenance Reliability Program Working Group will provide advice and recommendations on the...

Transit Reliability Information Program Participants Guidelines

DOT National Transportation Integrated Search

1981-03-01

The document provides guidelines for participation in the Transit Reliability Information Program (TRIP). TRIP is a government-initiated program designed to assist the transit industry in satisfying its need for transit equipment reliability data. TR...

Evaluation of audit-based performance measures for dental care plans.

PubMed

Bader, J D; Shugars, D A; White, B A; Rindal, D B

1999-01-01

Although a set of clinical performance measures, i.e., a report card for dental plans, has been designed for use with administrative data, most plans do not have administrative data systems containing the data needed to calculate the measures. Therefore, we evaluated the use of a set of proxy clinical performance measures calculated from data obtained through chart audits. Chart audits were conducted in seven dental programs--three public health clinics, two dental health maintenance organizations (DHMO), and two preferred provider organizations (PPO). In all instances audits were completed by clinical staff who had been trained using telephone consultation and a self-instructional audit manual. The performance measures were calculated for the seven programs, audit reliability was assessed in four programs, and for one program the audit-based proxy measures were compared to the measures calculated using administrative data. The audit-based measures were sensitive to known differences in program performance. The chart audit procedures yielded reasonably reliable data. However, missing data in patient charts rendered the calculation of some measures problematic--namely, caries and periodontal disease assessment and experience. Agreement between administrative and audit-based measures was good for most, but not all, measures in one program. The audit-based proxy measures represent a complex but feasible approach to the calculation of performance measures for those programs lacking robust administrative data systems. However, until charts contain more complete diagnostic information (i.e., periodontal charting and diagnostic codes or reason-for-treatment codes), accurate determination of these aspects of clinical performance will be difficult.

RELAV - RELIABILITY/AVAILABILITY ANALYSIS PROGRAM

NASA Technical Reports Server (NTRS)

Bowerman, P. N.

1994-01-01

RELAV (Reliability/Availability Analysis Program) is a comprehensive analytical tool to determine the reliability or availability of any general system which can be modeled as embedded k-out-of-n groups of items (components) and/or subgroups. Both ground and flight systems at NASA's Jet Propulsion Laboratory have utilized this program. RELAV can assess current system performance during the later testing phases of a system design, as well as model candidate designs/architectures or validate and form predictions during the early phases of a design. Systems are commonly modeled as System Block Diagrams (SBDs). RELAV calculates the success probability of each group of items and/or subgroups within the system assuming k-out-of-n operating rules apply for each group. The program operates on a folding basis; i.e. it works its way towards the system level from the most embedded level by folding related groups into single components. The entire folding process involves probabilities; therefore, availability problems are performed in terms of the probability of success, and reliability problems are performed for specific mission lengths. An enhanced cumulative binomial algorithm is used for groups where all probabilities are equal, while a fast algorithm based upon "Computing k-out-of-n System Reliability", Barlow & Heidtmann, IEEE TRANSACTIONS ON RELIABILITY, October 1984, is used for groups with unequal probabilities. Inputs to the program include a description of the system and any one of the following: 1) availabilities of the items, 2) mean time between failures and mean time to repairs for the items from which availabilities are calculated, 3) mean time between failures and mission length(s) from which reliabilities are calculated, or 4) failure rates and mission length(s) from which reliabilities are calculated. The results are probabilities of success of each group and the system in the given configuration. RELAV assumes exponential failure distributions for reliability calculations and infinite repair resources for availability calculations. No more than 967 items or groups can be modeled by RELAV. If larger problems can be broken into subsystems of 967 items or less, the subsystem results can be used as item inputs to a system problem. The calculated availabilities are steady-state values. Group results are presented in the order in which they were calculated (from the most embedded level out to the system level). This provides a good mechanism to perform trade studies. Starting from the system result and working backwards, the granularity gets finer; therefore, system elements that contribute most to system degradation are detected quickly. RELAV is a C-language program originally developed under the UNIX operating system on a MASSCOMP MC500 computer. It has been modified, as necessary, and ported to an IBM PC compatible with a math coprocessor. The current version of the program runs in the DOS environment and requires a Turbo C vers. 2.0 compiler. RELAV has a memory requirement of 103 KB and was developed in 1989. RELAV is a copyrighted work with all copyright vested in NASA.

Validity and reliability of global operative assessment of laparoscopic skills (GOALS) in novice trainees performing a laparoscopic cholecystectomy.

PubMed

Kramp, Kelvin H; van Det, Marc J; Hoff, Christiaan; Lamme, Bas; Veeger, Nic J G M; Pierie, Jean-Pierre E N

2015-01-01

Global Operative Assessment of Laparoscopic Skills (GOALS) assessment has been designed to evaluate skills in laparoscopic surgery. A longitudinal blinded study of randomized video fragments was conducted to estimate the validity and reliability of GOALS in novice trainees. In total, 10 trainees each performed 6 consecutive laparoscopic cholecystectomies. Sixty procedures were recorded on video. Video fragments of (1) opening of the peritoneum; (2) dissection of Calot's triangle and achievement of critical view of safety; and (3) dissection of the gallbladder from the liver bed were blinded, randomized, and rated by 2 consultant surgeons using GOALS. Also, a grade was given for overall competence. The correlation of GOALS with live observation Objective Structured Assessment of Technical Skills (OSATS) scores was calculated. Construct validity was estimated using the Friedman 2-way analysis of variance by ranks and the Wilcoxon signed-rank test. The interrater reliability was calculated using the absolute and consistency agreement 2-way random-effects model intraclass correlation coefficient. A high correlation was found between mean GOALS score (r = 0.879, p = 0.021) and mean OSATS score. The GOALS score increased significantly across the 6 procedures (p = 0.002). The trainees performed significantly better on their sixth when compared with their first cholecystectomy (p = 0.004). The consistency agreement interrater reliability was 0.37 for the mean GOALS score (p = 0.002) and 0.55 for overall competence (p < 0.001) of the 3 video fragments. The validity observed in this randomized blinded longitudinal study supports the existing evidence that GOALS is a valid tool for assessment of novice trainees. A relatively low reliability was found in this study. Copyright © 2014 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

Milestones: Critical Elements in Clinical Informatics Fellowship Programs

PubMed Central

Lehmann, Christoph U.; Munger, Benson

2016-01-01

Summary Background Milestones refer to points along a continuum of a competency from novice to expert. Resident and fellow assessment and program evaluation processes adopted by the ACGME include the mandate that programs report the educational progress of residents and fellows twice annually utilizing Milestones developed by a specialty specific ACGME working group of experts. Milestones in clinical training programs are largely unmapped to specific assessment tools. Residents and fellows are mainly assessed using locally derived assessment instruments. These assessments are then reviewed by the Clinical Competency Committee which assigns and reports trainee ratings using the specialty specific reporting Milestones. Methods and Results The challenge and opportunity facing the nascent specialty of Clinical Informatics is how to optimally utilize this framework across a growing number of accredited fellowships. The authors review how a mapped milestone framework, in which each required sub-competency is mapped to a single milestone assessment grid, can enable the use of milestones for multiple uses including individualized learning plans, fellow assessments, and program evaluation. Furthermore, such a mapped strategy will foster the ability to compare fellow progress within and between Clinical Informatics Fellowships in a structured and reliable fashion. Clinical Informatics currently has far less variability across programs and thus could easily utilize a more tightly defined set of milestones with a clear mapping to sub-competencies. This approach would enable greater standardization of assessment instruments and processes across programs while allowing for variability in how those sub-competencies are taught. Conclusions A mapped strategy for Milestones offers significant advantages for Clinical Informatics programs. PMID:27081414

Measurement characteristics of the levels of institutionalization scales: examining reliability and validity.

PubMed

Barab, S A; Redman, B K; Froman, R D

1998-01-01

The Level of Institutionalization (LoIn) scales were developed to assess the extent to which a health promotion program has become integrated into a health care organization. The instrument was designed specifically to measure the amount of routinization and niche saturation of four subsystems (production, maintenance, supportive, and managerial) believed to make up an organization. In this study, the LoIn scales were completed for diabetes programs in 102 general hospitals and 30 home health agencies in Maryland and Pennsylvania. Reliability estimates across the four subsystems for routines (alpha = .61) and for niche saturation (alpha = .44) were substandard. Average correlation among the four subsystems for routines was .67, and among the four subsystems for niche saturation was .38, indicating moderate to large amounts of shared variance among subsystems and challenging claims of discriminant validity. Given these large correlations and a poor fit when testing the eight-factor model, higher-order confirmatory factor analyses were carried out. Results supported the existence of two second-order factors. When collapsed into two factors, the reliabilities were adequate (routines alpha = .90; niche saturation alpha = .80). Criterion-related validity also was found between length of program existence and the routine factor.

Assessing reliability and validity measures in managed care studies.

PubMed

Montoya, Isaac D

2003-01-01

To review the reliability and validity literature and develop an understanding of these concepts as applied to managed care studies. Reliability is a test of how well an instrument measures the same input at varying times and under varying conditions. Validity is a test of how accurately an instrument measures what one believes is being measured. A review of reliability and validity instructional material was conducted. Studies of managed care practices and programs abound. However, many of these studies utilize measurement instruments that were developed for other purposes or for a population other than the one being sampled. In other cases, instruments have been developed without any testing of the instrument's performance. The lack of reliability and validity information may limit the value of these studies. This is particularly true when data are collected for one purpose and used for another. The usefulness of certain studies without reliability and validity measures is questionable, especially in cases where the literature contradicts itself

Transit Reliability Information Program (TRIP) : Final Technical Report

DOT National Transportation Integrated Search

1984-05-01

The Transit Reliability Information Program (TRIP) is a government-initiated program to assist the transit industry in satisfying its need for rail transit car subsystem reliability information. TRIP provided this assistance through the operation of ...

Transit Reliability Information Program (TRIP) Phase I Report

DOT National Transportation Integrated Search

1981-06-01

The Transit Reliability Information Program (TRIP) is a government initiated program to assist the transit industry in satisfying its need for transit reliability information. TRIP provides this assistance through the operation of a national reliabil...

Comprehensive clinical assessment in community setting: applicability of the MDS-HC.

PubMed

Morris, J N; Fries, B E; Steel, K; Ikegami, N; Bernabei, R; Carpenter, G I; Gilgen, R; Hirdes, J P; Topinková, E

1997-08-01

To describe the results of an international trial of the home care version of the MDS assessment and problem identification system (the MDS-HC), including reliability estimates, a comparison of MDS-HC reliabilities with reliabilities of the same items in the MDS 2.0 nursing home assessment instrument, and an examination of the types of problems found in home care clients using the MDS-HC. Independent, dual assessment of clients of home-care agencies by trained clinicians using a draft of the MDS-HC, with additional descriptive data regarding problem profiles for home care clients. Reliability data from dual assessments of 241 randomly selected clients of home care agencies in five countries, all of whom volunteered to test the MDS-HC. Also included are an expanded sample of 780 home care assessments from these countries and 187 dually assessed residents from 21 nursing homes in the United States. The array of MDS-HC assessment items included measures in the following areas: personal items, cognitive patterns, communication/hearing, vision, mood and behavior, social functioning, informal support services, physical functioning, continence, disease diagnoses health conditions and preventive health measures, nutrition/hydration, dental status, skin condition, environmental assessment, service utilization, and medications. Forty-seven percent of the functional, health status, social environment, and service items in the MDS-HC were taken from the MDS 2.0 for nursing homes. For this item set, it is estimated that the average weighted Kappa is .74 for the MDS-HC and .75 for the MDS 2.0. Similarly, high reliability values were found for items newly introduced in the MDS-HC (weighted Kappa = .70). Descriptive findings also characterize the problems of home care clients, with subanalyses within cognitive performance levels. Findings indicate that the core set of items in the MDS 2.0 work equally well in community and nursing home settings. New items are highly reliable. In tandem, these instruments can be used within the international community, assisting and planning care for older adults within a broad spectrum of service settings, including nursing homes and home care programs. With this community-based, second-generation problem and care plan-driven assessment instrument, disability assessment can be performed consistently across the world.

Development of a Self-Rated Mixed Methods Skills Assessment: The National Institutes of Health Mixed Methods Research Training Program for the Health Sciences.

PubMed

Guetterman, Timothy C; Creswell, John W; Wittink, Marsha; Barg, Fran K; Castro, Felipe G; Dahlberg, Britt; Watkins, Daphne C; Deutsch, Charles; Gallo, Joseph J

2017-01-01

Demand for training in mixed methods is high, with little research on faculty development or assessment in mixed methods. We describe the development of a self-rated mixed methods skills assessment and provide validity evidence. The instrument taps six research domains: "Research question," "Design/approach," "Sampling," "Data collection," "Analysis," and "Dissemination." Respondents are asked to rate their ability to define or explain concepts of mixed methods under each domain, their ability to apply the concepts to problems, and the extent to which they need to improve. We administered the questionnaire to 145 faculty and students using an internet survey. We analyzed descriptive statistics and performance characteristics of the questionnaire using the Cronbach alpha to assess reliability and an analysis of variance that compared a mixed methods experience index with assessment scores to assess criterion relatedness. Internal consistency reliability was high for the total set of items (0.95) and adequate (≥0.71) for all but one subscale. Consistent with establishing criterion validity, respondents who had more professional experiences with mixed methods (eg, published a mixed methods article) rated themselves as more skilled, which was statistically significant across the research domains. This self-rated mixed methods assessment instrument may be a useful tool to assess skills in mixed methods for training programs. It can be applied widely at the graduate and faculty level. For the learner, assessment may lead to enhanced motivation to learn and training focused on self-identified needs. For faculty, the assessment may improve curriculum and course content planning.

Assessing the validity and reliability of three indicators self-reported on the pregnancy risk assessment monitoring system survey.

PubMed

Ahluwalia, Indu B; Helms, Kristen; Morrow, Brian

2013-01-01

We investigated the reliability and validity of three self-reported indicators from the Pregnancy Risk Assessment Monitoring System (PRAMS) survey. We used 2008 PRAMS (n=15,646) data from 12 states that had implemented the 2003 revised U.S. Certificate of Live Birth. We estimated reliability by kappa coefficient and validity by sensitivity and specificity using the birth certificate data as the reference for the following: prenatal participation in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC); Medicaid payment for delivery; and breastfeeding initiation. These indicators were examined across several demographic subgroups. The reliability was high for all three measures: 0.81 for WIC participation, 0.67 for Medicaid payment of delivery, and 0.72 for breastfeeding initiation. The validity of PRAMS indicators was also high: WIC participation (sensitivity = 90.8%, specificity = 90.6%), Medicaid payment for delivery (sensitivity = 82.4%, specificity = 85.6%), and breastfeeding initiation (sensitivity = 94.3%, specificity = 76.0%). The prevalence estimates were higher on PRAMS than the birth certificate for each of the indicators except Medicaid-paid delivery among non-Hispanic black women. Kappa values within most subgroups remained in the moderate range (0.40-0.80). Sensitivity and specificity values were lower for Hispanic women who responded to the PRAMS survey in Spanish and for breastfeeding initiation among women who delivered very low birthweight and very preterm infants. The validity and reliability of the PRAMS data for measures assessed were high. Our findings support the use of PRAMS data for epidemiological surveillance, research, and planning.

Cyber Security: Assessing Our Vulnerabilities and Developing an Effective Defense

NASA Astrophysics Data System (ADS)

Spafford, Eugene H.

The number and sophistication of cyberattacks continues to increase, but no national policy is in place to confront them. Critical systems need to be built on secure foundations, rather than the cheapest general-purpose platform. A program that combines education in cyber security, increasing resources for law enforcement, development of reliable systems for critical applications, and expanding research support in multiple areas of security and reliability is essential to combat risks that are far beyond the nuisances of spam email and viruses, and involve widespread espionage, theft, and attacks on essential services.

Risk-Based Probabilistic Approach to Aeropropulsion System Assessment

NASA Technical Reports Server (NTRS)

Tong, Michael T.

2002-01-01

In an era of shrinking development budgets and resources, where there is also an emphasis on reducing the product development cycle, the role of system assessment, performed in the early stages of an engine development program, becomes very critical to the successful development of new aeropropulsion systems. A reliable system assessment not only helps to identify the best propulsion system concept among several candidates, it can also identify which technologies are worth pursuing. This is particularly important for advanced aeropropulsion technology development programs, which require an enormous amount of resources. In the current practice of deterministic, or point-design, approaches, the uncertainties of design variables are either unaccounted for or accounted for by safety factors. This could often result in an assessment with unknown and unquantifiable reliability. Consequently, it would fail to provide additional insight into the risks associated with the new technologies, which are often needed by decision makers to determine the feasibility and return-on-investment of a new aircraft engine. In this work, an alternative approach based on the probabilistic method was described for a comprehensive assessment of an aeropropulsion system. The statistical approach quantifies the design uncertainties inherent in a new aeropropulsion system and their influences on engine performance. Because of this, it enhances the reliability of a system assessment. A technical assessment of a wave-rotor-enhanced gas turbine engine was performed to demonstrate the methodology. The assessment used probability distributions to account for the uncertainties that occur in component efficiencies and flows and in mechanical design variables. The approach taken in this effort was to integrate the thermodynamic cycle analysis embedded in the computer code NEPP (NASA Engine Performance Program) and the engine weight analysis embedded in the computer code WATE (Weight Analysis of Turbine Engines) with the fast probability integration technique (FPI). FPI was developed by Southwest Research Institute under contract with the NASA Glenn Research Center. The results were plotted in the form of cumulative distribution functions and sensitivity analyses and were compared with results from the traditional deterministic approach. The comparison showed that the probabilistic approach provides a more realistic and systematic way to assess an aeropropulsion system. The current work addressed the application of the probabilistic approach to assess specific fuel consumption, engine thrust, and weight. Similarly, the approach can be used to assess other aspects of aeropropulsion system performance, such as cost, acoustic noise, and emissions. Additional information is included in the original extended abstract.

Adapting the helpful responses questionnaire to assess communication skills involved in delivering contingency management: preliminary psychometrics.

PubMed

Hartzler, Bryan

2015-08-01

A paper/pencil instrument, adapted from Miller and colleagues' (1991) Helpful Responses Questionnaire (HRQ), was developed to assess clinician skill with core communicative aspects involved in delivering contingency management (CM). The instrument presents a single vignette consisting of six points of client dialogue to which respondents write 'what they would say next.' In the context of an implementation/effectiveness hybrid trial, 19 staff clinicians at an opiate treatment program completed serial training outcome assessments before, following, and three months after CM training. Assessments included this adaptation of the HRQ, a multiple-choice CM knowledge test, and a recorded standardized patient encounter scored for CM skillfulness. Study results reveal promising psychometric properties for the instrument, including strong scoring reliability, internal consistency, concurrent and predictive validity, test-retest reliability and sensitivity to training effects. These preliminary findings suggest the instrument is a viable, practical method to assess clinician skill in communicative aspects of CM delivery. Copyright © 2015 Elsevier Inc. All rights reserved.

Validation of the process criteria for assessment of a hospital nursing service.

PubMed

Feldman, Liliane Bauer; Cunha, Isabel Cristina Kowal Olm; D'Innocenzo, Maria

2013-01-01

to validate an instrument containing process criteria for assessment of a hospital nursing service based on the National Accreditation Organization program. a descriptive, quantitative methodological study performed in stages. An instrument constructed with 69 process criteria was assessed by 49 nurses from accredited hospitals in 2009, according to a Likert scale, and validated by 16 judges through Delphi rounds in 2010. the original instrument assessed by nurses with 69 process criteria was judged by the degree of importance, and changed to 39 criteria. In the first Delphi round, the 39 criteria reached consensus among the 19 judges, with a medium reliability by Cronbach's alpha. In the second round, 40 converging criteria were validated by 16 judges, with high reliability. The criteria addressed management, costs, teaching, education, indicators, protocols, human resources, communication, among others. the 40 process criteria formed a validated instrument to assess the hospital nursing service which, when measured, can better direct interventions by nurses in reaching and strengthening outcomes.

Holistic Assessment and the Study Abroad Experience

ERIC Educational Resources Information Center

Doyle, Dennis

2009-01-01

While many educators who work closely with study abroad programs could conjure up a litany of testimonials about the dramatic impact of study abroad, it is often difficult to move beyond vaguely descriptive accounts to reliable data showing how this experience influenced a student's growth in intercultural sensitivity and awareness. King and…

Instruments and Scoring Guide of the Experiential Education Evaluation Project.

ERIC Educational Resources Information Center

Conrad, Dan; Hedin, Diane

As a result of the Experiential Education Evaluation Project the publication identifies instruments used to measure and assess experiential learning programs. The following information is given for each instrument: rationale for its inclusion in the study; precise issues or outcomes designed to measure, validity and reliability data; and…

75 FR 13515 - Office of Innovation and Improvement (OII); Overview Information; Ready-to-Learn Television...

Federal Register 2010, 2011, 2012, 2013, 2014

2010-03-22

... on rigorous scientifically based research methods to assess the effectiveness of a particular... activities and programs; and (B) Includes research that-- (i) Employs systematic, empirical methods that draw... or observational methods that provide reliable and valid data across evaluators and observers, across...

An evaluation of the success of a surgical resident learning portfolio.

PubMed

Webb, Travis P; Merkley, Taylor R

2012-01-01

Learning portfolios have gained modest acceptance in graduate medical education because of challenges related to user satisfaction, time and resource commitment, and quality assessment. In 2001, the Department of Surgery implemented the Surgical Learning and Instructional Portfolio (SLIP) to help residents develop a case-based portfolio demonstrating practice-based learning. In 2008, the format was changed to a Web-based platform with open viewing of portfolios for all learners. This study was performed to evaluate the SLIP program using resident and faculty perspectives in the domains of satisfaction, compliance, and educational value. Likert scale surveys were distributed to residents to assess satisfaction. Using a semistructured format with subsequent qualitative analysis of the meeting transcript, a focus group discussion was held with the SLIP director, SLIP facilitator, and program coordinator. An analysis of the program compliance was performed by review of SLIP entry dates. Finally, the quality of the SLIP entries (n = 420) was analyzed in a blinded manner using a locally developed standardized SLIP assessment tool. Data analysis was performed using Pearson's correlation and Cronbach's alpha. Residents were satisfied with the program and felt the Web-based format promoted self-reflection. They perceived that time spent was appropriate. Residents also believed they gained medical knowledge of their own specific entry topics but did not learn routinely from others' entries. Faculty asserted that the Web-based platform eased the administrative burden but did not necessarily alter the quality of the SLIP entries. Compliance with the assignment was 100%. SLIP entry analysis demonstrated the reflection and understanding of the topics chosen. However, the overall quality assessment of entries was hindered by suboptimal interrater reliability (inter-rater reliability (IR) = 0.636). The SLIP program allows residents to demonstrate practice-based learning and improvement of medical knowledge. The Web-based format provides transparency and ease of administration. Quality assessment of individual portfolio entries remains a challenge to the widespread adoption of portfolios. Copyright © 2012 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

NASA Applications and Lessons Learned in Reliability Engineering

NASA Technical Reports Server (NTRS)

Safie, Fayssal M.; Fuller, Raymond P.

2011-01-01

Since the Shuttle Challenger accident in 1986, communities across NASA have been developing and extensively using quantitative reliability and risk assessment methods in their decision making process. This paper discusses several reliability engineering applications that NASA has used over the year to support the design, development, and operation of critical space flight hardware. Specifically, the paper discusses several reliability engineering applications used by NASA in areas such as risk management, inspection policies, components upgrades, reliability growth, integrated failure analysis, and physics based probabilistic engineering analysis. In each of these areas, the paper provides a brief discussion of a case study to demonstrate the value added and the criticality of reliability engineering in supporting NASA project and program decisions to fly safely. Examples of these case studies discussed are reliability based life limit extension of Shuttle Space Main Engine (SSME) hardware, Reliability based inspection policies for Auxiliary Power Unit (APU) turbine disc, probabilistic structural engineering analysis for reliability prediction of the SSME alternate turbo-pump development, impact of ET foam reliability on the Space Shuttle System risk, and reliability based Space Shuttle upgrade for safety. Special attention is given in this paper to the physics based probabilistic engineering analysis applications and their critical role in evaluating the reliability of NASA development hardware including their potential use in a research and technology development environment.

HiRel: Hybrid Automated Reliability Predictor (HARP) integrated reliability tool system, (version 7.0). Volume 1: HARP introduction and user's guide

NASA Technical Reports Server (NTRS)

Bavuso, Salvatore J.; Rothmann, Elizabeth; Dugan, Joanne Bechta; Trivedi, Kishor S.; Mittal, Nitin; Boyd, Mark A.; Geist, Robert M.; Smotherman, Mark D.

1994-01-01

The Hybrid Automated Reliability Predictor (HARP) integrated Reliability (HiRel) tool system for reliability/availability prediction offers a toolbox of integrated reliability/availability programs that can be used to customize the user's application in a workstation or nonworkstation environment. HiRel consists of interactive graphical input/output programs and four reliability/availability modeling engines that provide analytical and simulative solutions to a wide host of reliable fault-tolerant system architectures and is also applicable to electronic systems in general. The tool system was designed to be compatible with most computing platforms and operating systems, and some programs have been beta tested, within the aerospace community for over 8 years. Volume 1 provides an introduction to the HARP program. Comprehensive information on HARP mathematical models can be found in the references.

Measurement properties of a novel survey to assess stages of organizational readiness for evidence-based interventions in community chronic disease prevention settings

PubMed Central

2012-01-01

Background There is a great deal of variation in the existing capacity of primary prevention programs and policies addressing chronic disease to deliver evidence-based interventions (EBIs). In order to develop and evaluate implementation strategies that are tailored to the appropriate level of capacity, there is a need for an easy-to-administer tool to stage organizational readiness for EBIs. Methods Based on theoretical frameworks, including Rogers’ Diffusion of Innovations, we developed a survey instrument to measure four domains representing stages of readiness for EBI: awareness, adoption, implementation, and maintenance. A separate scale representing organizational climate as a potential mediator of readiness for EBIs was also included in the survey. Twenty-three questions comprised the four domains, with four to nine items each, using a seven-point response scale. Representatives from obesity, asthma, diabetes, and tobacco prevention programs serving diverse populations in the United States were surveyed (N = 243); test-retest reliability was assessed with 92 respondents. Results Confirmatory factor analysis (CFA) was used to test and refine readiness scales. Test-retest reliability of the readiness scales, as measured by intraclass correlation, ranged from 0.47–0.71. CFA found good fit for the five-item adoption and implementation scales and resulted in revisions of the awareness and maintenance scales. The awareness scale was split into two two-item scales, representing community and agency awareness. The maintenance scale was split into five- and four-item scales, representing infrastructural maintenance and evaluation maintenance, respectively. Internal reliability of scales (Cronbach’s α) ranged from 0.66–0.78. The model for the final revised scales approached good fit, with most factor loadings >0.6 and all >0.4. Conclusions The lack of adequate measurement tools hinders progress in dissemination and implementation research. These preliminary results help fill this gap by describing the reliability and measurement properties of a theory-based tool; the short, user-friendly instrument may be useful to researchers and practitioners seeking to assess organizational readiness for EBIs across a variety of chronic disease prevention programs and settings. PMID:22800294

Measurement properties of a novel survey to assess stages of organizational readiness for evidence-based interventions in community chronic disease prevention settings.

PubMed

Stamatakis, Katherine A; McQueen, Amy; Filler, Carl; Boland, Elizabeth; Dreisinger, Mariah; Brownson, Ross C; Luke, Douglas A

2012-07-16

There is a great deal of variation in the existing capacity of primary prevention programs and policies addressing chronic disease to deliver evidence-based interventions (EBIs). In order to develop and evaluate implementation strategies that are tailored to the appropriate level of capacity, there is a need for an easy-to-administer tool to stage organizational readiness for EBIs. Based on theoretical frameworks, including Rogers' Diffusion of Innovations, we developed a survey instrument to measure four domains representing stages of readiness for EBI: awareness, adoption, implementation, and maintenance. A separate scale representing organizational climate as a potential mediator of readiness for EBIs was also included in the survey. Twenty-three questions comprised the four domains, with four to nine items each, using a seven-point response scale. Representatives from obesity, asthma, diabetes, and tobacco prevention programs serving diverse populations in the United States were surveyed (N=243); test-retest reliability was assessed with 92 respondents. Confirmatory factor analysis (CFA) was used to test and refine readiness scales. Test-retest reliability of the readiness scales, as measured by intraclass correlation, ranged from 0.47-0.71. CFA found good fit for the five-item adoption and implementation scales and resulted in revisions of the awareness and maintenance scales. The awareness scale was split into two two-item scales, representing community and agency awareness. The maintenance scale was split into five- and four-item scales, representing infrastructural maintenance and evaluation maintenance, respectively. Internal reliability of scales (Cronbach's α) ranged from 0.66-0.78. The model for the final revised scales approached good fit, with most factor loadings >0.6 and all >0.4. The lack of adequate measurement tools hinders progress in dissemination and implementation research. These preliminary results help fill this gap by describing the reliability and measurement properties of a theory-based tool; the short, user-friendly instrument may be useful to researchers and practitioners seeking to assess organizational readiness for EBIs across a variety of chronic disease prevention programs and settings.

Undergraduate study in psychology: Curriculum and assessment.

PubMed

Norcross, John C; Hailstorks, Robin; Aiken, Leona S; Pfund, Rory A; Stamm, Karen E; Christidis, Peggy

2016-01-01

The undergraduate curriculum in psychology profoundly reflects and shapes the discipline. Yet, reliable information on the undergraduate psychology curriculum has been difficult to acquire due to insufficient research carried out on unrepresentative program samples with disparate methods. In 2014, APA launched the first systematic effort in a decade to gather national data on the psychology major and program outcomes. We surveyed a stratified random sample of department chairs/coordinators of accredited colleges and universities in the United States that offer undergraduate courses and programs in psychology. A total of 439 undergraduate psychology programs (45.2%) completed the survey. This article summarizes, for both associate and baccalaureate programs, the results of the Undergraduate Study in Psychology. Current practices concerning the introductory course, the courses offered, core requirements, the psychology minor, and tracks/concentrations are presented. The frequency of formal program reviews and program-level assessment methods are also addressed. By extending prior research on the undergraduate curriculum, we chronicle longitudinal changes in the psychology major over the past 20 years. (c) 2016 APA, all rights reserved).

Further empirical data on the psychoeducational profile-revised (PEP-R): reliability and validation with the Vineland adaptive behavior scales.

PubMed

Villa, Susanna; Micheli, Enrico; Villa, Laura; Pastore, Valentina; Crippa, Alessandro; Molteni, Massimo

2010-03-01

The PEP-R (psychoeducational profile revised) is an instrument that has been used in many countries to assess abilities and formulate treatment programs for children with autism and related developmental disorders. To the end to provide further information on the PEP-R's psychometric properties, a large sample (N = 137) of children presenting Autistic Disorder symptoms under the age of 12 years, including low-functioning individuals, was examined. Results yielded data of interest especially in terms of: Cronbach's alpha, interrater reliability, and validation with the Vineland Adaptive Behavior Scales. These findings help complete the instrument's statistical description and augment its usefulness, not only in designing treatment programs for these individuals, but also as an instrument for verifying the efficacy of intervention.

Safety, reliability, maintainability and quality provisions for the Space Shuttle program

NASA Technical Reports Server (NTRS)

1990-01-01

This publication establishes common safety, reliability, maintainability and quality provisions for the Space Shuttle Program. NASA Centers shall use this publication both as the basis for negotiating safety, reliability, maintainability and quality requirements with Shuttle Program contractors and as the guideline for conduct of program safety, reliability, maintainability and quality activities at the Centers. Centers shall assure that applicable provisions of the publication are imposed in lower tier contracts. Centers shall give due regard to other Space Shuttle Program planning in order to provide an integrated total Space Shuttle Program activity. In the implementation of safety, reliability, maintainability and quality activities, consideration shall be given to hardware complexity, supplier experience, state of hardware development, unit cost, and hardware use. The approach and methods for contractor implementation shall be described in the contractors safety, reliability, maintainability and quality plans. This publication incorporates provisions of NASA documents: NHB 1700.1 'NASA Safety Manual, Vol. 1'; NHB 5300.4(IA), 'Reliability Program Provisions for Aeronautical and Space System Contractors'; and NHB 5300.4(1B), 'Quality Program Provisions for Aeronautical and Space System Contractors'. It has been tailored from the above documents based on experience in other programs. It is intended that this publication be reviewed and revised, as appropriate, to reflect new experience and to assure continuing viability.

The Healthy Afterschool Activity and Nutrition Documentation Instrument

PubMed Central

Ajja, Rahma; Beets, Michael W.; Huberty, Jennifer; Kaczynski, Andrew T.; Ward, Dianne S.

2012-01-01

Background Policies call on afterschool programs to improve the physical activity and nutrition habits of youth attending. No tool exists to assess the extent to which the afterschool program environment meets physical activity and nutrition policies. Purpose To describe the development of the Healthy Afterschool Activity and Nutrition Documentation (HAAND) instrument, which consists of two subscales: Healthy Afterschool Program Index for Physical Activity (HAPI-PA) and the HAPI-Nutrition (HAPI-N). Methods Thirty-nine afterschool programs took part in the HAAND evaluation during fall/spring 2010–2011. Inter-rater reliability data were collected at 20 afterschool programs during a single site visit via direct observation, personal interview and written document review. Validity of the HAPI-PA was established by comparing HAPI-PA scores to pedometer steps collected in a subsample of 934 children attending 25 of the afterschool programs. Validity of the HAPI-N scores was compared against the mean number of times/week that fruits/vegetables (FV) and whole grains were served in the program. Results Data were analyzed in June/July 2011. Inter-rater percent agreement was 85%–100% across all items. Increased pedometer steps were associated with the presence of a written policy related to physical activity, amount/quality of staff training, use of a physical activity curriculum, and offering activities that appeal to both genders. Higher servings of FV and whole grains per week were associated with the presence of a written policy regarding the nutritional quality of snacks. Conclusions The HAAND instrument is a reliable and valid measurement tool that can be used to assess the physical activity and nutritional environment of afterschool programs. PMID:22898119

The healthy afterschool activity and nutrition documentation instrument.

PubMed

Ajja, Rahma; Beets, Michael W; Huberty, Jennifer; Kaczynski, Andrew T; Ward, Dianne S

2012-09-01

Policies call on afterschool programs to improve the physical activity and nutrition habits of youth attending. No tool exists to assess the extent to which the afterschool program environment meets physical activity and nutrition policies. To describe the development of the Healthy Afterschool Activity and Nutrition Documentation (HAAND) instrument, which consists of two subscales: Healthy Afterschool Program Index for Physical Activity (HAPI-PA) and the HAPI-Nutrition (HAPI-N). Thirty-nine afterschool programs took part in the HAAND evaluation during fall/spring 2010-2011. Inter-rater reliability data were collected at 20 afterschool programs during a single site visit via direct observation, personal interview, and written document review. Validity of the HAPI-PA was established by comparing HAPI-PA scores to pedometer steps collected in a subsample of 934 children attending 25 of the afterschool programs. Validity of the HAPI-N scores was compared against the mean number of times/week that fruits and vegetables (FV) and whole grains were served in the program. Data were analyzed in June/July 2011. Inter-rater percent agreement was 85%-100% across all items. Increased pedometer steps were associated with the presence of a written policy related to physical activity, amount/quality of staff training, use of a physical activity curriculum, and offering activities that appeal to both genders. Higher servings of FV and whole grains per week were associated with the presence of a written policy regarding the nutritional quality of snacks. The HAAND instrument is a reliable and valid measurement tool that can be used to assess the physical activity and nutritional environment of afterschool programs. Copyright © 2012 American Journal of Preventive Medicine. Published by Elsevier Inc. All rights reserved.

External quality-assurance programs managed by the U.S. Geological Survey in support of the National Atmospheric Deposition Program/National Trends Network

USGS Publications Warehouse

Latysh, Natalie E.; Wetherbee, Gregory A.

2005-01-01

The U.S. Geological Survey, Branch of Quality Systems, operates the external quality-assurance programs for the National Atmospheric Deposition Program/National Trends Network (NADP/NTN). Beginning in 1978, six different programs have been implemented?the intersite-comparison program, the blind-audit program, the sample-handling evaluation program, the field-audit program, the interlaboratory-comparison program, and the collocated-sampler program. Each program was designed to measure error contributed by specific components in the data-collection process. The intersite-comparison program, which was discontinued in 2004, was designed to assess the accuracy and reliability of field pH and specific-conductance measurements made by site operators. The blind-audit and sample-handling evaluation programs, which also were discontinued in 2002 and 2004, respectively, assessed contamination that may result from sampling equipment and routine handling and processing of the wet-deposition samples. The field-audit program assesses the effects of sample handling, processing, and field exposure. The interlaboratory-comparison program evaluates bias and precision of analytical results produced by the contract laboratory for NADP, the Illinois State Water Survey, Central Analytical Laboratory, and compares its performance with the performance of international laboratories. The collocated-sampler program assesses the overall precision of wet-deposition data collected by NADP/NTN. This report documents historical operations and the operating procedures for each of these external quality-assurance programs. USGS quality-assurance information allows NADP/NTN data users to discern between actual environmental trends and inherent measurement variability.

DOE-NE Proliferation and Terrorism Risk Assessment: FY12 Plans Update

DOE Office of Scientific and Technical Information (OSTI.GOV)

Sadasivan, Pratap

2012-06-21

This presentation provides background information on FY12 plans for the DOE Office of Nuclear Energy Proliferation and Terrorism Risk Assessment program. Program plans, organization, and individual project elements are described. Research objectives are: (1) Develop technologies and other solutions that can improve the reliability, sustain the safety, and extend the life of current reactors; (2) Develop improvements in the affordability of new reactors to enable nuclear energy; (3) Develop Sustainable Nuclear Fuel Cycles; and (4) Understand and minimize the risks of nuclear proliferation and terrorism - Goal is to enable the use of risk information to inform NE R&D programmore » planning.« less

Reliability-Growth Assessment, Prediction, and Control for Electronic Engine Control (GAPCEEC)

DTIC Science & Technology

1984-04-01

COMPEtTIN FOR Control (GAPCEEC) 4. PERFORMING ORG. REPORT NUMUR ______________________________ P&W/GPD/FR-17847 7. AUTI4OR(e) Michael E. McGlone...Control for Electronic Engine Control (GAPCEEC) program was performed under contract F33615-81 -C-2015. This 22-month program I. was formulated to study and...and (3) that data should be tracked c-ontinuously on an indiidual and fleet basis. UNCLASSIFIED 86CURITY CLASSIICATION OF THIS PAGErWmm Doe Rateio

Digital electronic engine control history

NASA Technical Reports Server (NTRS)

Putnam, T. W.

1984-01-01

Full authority digital electronic engine controls (DEECs) were studied, developed, and ground tested because of projected benefits in operability, improved performance, reduced maintenance, improved reliability, and lower life cycle costs. The issues of operability and improved performance, however, are assessed in a flight test program. The DEEC on a F100 engine in an F-15 aircraft was demonstrated and evaluated. The events leading to the flight test program are chronicled and important management and technical results are identified.

Computer Aided Method for System Safety and Reliability Assessments

DTIC Science & Technology

2008-09-01

program between 1998 and 2003. This tool was not marketed in the public domain after the CRV program ended. The other tool is called eXpress, and it...support Government reviewed and approved analyses methodologies which can 5 then be shared with other government agencies and industry partners...Documented for B&R, UP&L, EPRI 30 DEC 80 GO IBM Version Enhanced at UCC , Dallas, Descriptors, Facility to Alter Array Sizes, Explanation of Use 1 SEP 82

Technology readiness levels and technology status for selected long term/high payoff technologies on the RLV program

NASA Technical Reports Server (NTRS)

Rosmait, Russell L.

1996-01-01

The development of a new space transportation system in a climate of constant budget cuts and staff reductions can be and is a difficult task. It is no secret that NASA's current launching system consumes a very large portion of NASA funding and requires a large army of people to operate & maintain the system. The new Reusable Launch Vehicle (RLV) project and it's programs are faced with a monumental task of making the cost of access to space dramatically lower and more efficient than NASA's current system. With pressures from congressional budget cutters and also increased competition and loss of market share from international agencies RLV's first priority is to develop a 'low-cost, reliable transportation to earth orbit.' One of the RLV's major focus in achieving low-cost, reliable transportation to earth orbit is to rely on the maturing of advanced technologies. The technologies for the RLV are numerous and varied. Trying to assess their current status, within the RLV development program is paramount. There are several ways to assess these technologies. One way is through the use of Technology Readiness Levels (TRL's). This project focused on establishing current (summer 95) 'worst case' TRL's for six selected technologies that are under consideration for use within the RLV program. The six technologies evaluated were Concurrent Engineering, Embedded Sensor Technology, Rapid Prototyping, Friction Stir Welding, Thermal Spray Coatings, and VPPA Welding.

Excellent reliability of the Hamilton Depression Rating Scale (HDRS-21) in Indonesia after training.

PubMed

Istriana, Erita; Kurnia, Ade; Weijers, Annelies; Hidayat, Teddy; Pinxten, Lucas; de Jong, Cor; Schellekens, Arnt

2013-09-01

The Hamilton Depression Rating Scale (HDRS) is the most widely used depression rating scale worldwide. Reliability of HDRS has been reported mainly from Western countries. The current study tested the reliability of HDRS ratings among psychiatric residents in Indonesia, before and after HDRS training. The hypotheses were that: (i) prior to the training reliability of HDRS ratings is poor; and (ii) HDRS training can improve reliability of HDRS ratings to excellent levels. Furthermore, we explored cultural validity at item level. Videotaped HDRS interviews were rated by 30 psychiatric residents before and after 1 day of HDRS training. Based on a gold standard rating, percentage correct ratings and deviation from the standard were calculated. Correct ratings increased from 83% to 99% at item level and from 70% to 100% for the total rating. The average deviation from the gold standard rating improved from 0.07 to 0.02 at item level and from 2.97 to 0.46 for the total rating. HDRS assessment by psychiatric trainees in Indonesia without prior training is unreliable. A short, evidence-based HDRS training improves reliability to near perfect levels. The outlined training program could serve as a template for HDRS trainings. HDRS items that may be less valid for assessment of depression severity in Indonesia are discussed. Copyright © 2013 Wiley Publishing Asia Pty Ltd.

Software reliability experiments data analysis and investigation

NASA Technical Reports Server (NTRS)

Walker, J. Leslie; Caglayan, Alper K.

1991-01-01

The objectives are to investigate the fundamental reasons which cause independently developed software programs to fail dependently, and to examine fault tolerant software structures which maximize reliability gain in the presence of such dependent failure behavior. The authors used 20 redundant programs from a software reliability experiment to analyze the software errors causing coincident failures, to compare the reliability of N-version and recovery block structures composed of these programs, and to examine the impact of diversity on software reliability using subpopulations of these programs. The results indicate that both conceptually related and unrelated errors can cause coincident failures and that recovery block structures offer more reliability gain than N-version structures if acceptance checks that fail independently from the software components are available. The authors present a theory of general program checkers that have potential application for acceptance tests.

Maximizing Energy Savings Reliability in BC Hydro Industrial Demand-side Management Programs: An Assessment of Performance Incentive Models

NASA Astrophysics Data System (ADS)

Gosman, Nathaniel

For energy utilities faced with expanded jurisdictional energy efficiency requirements and pursuing demand-side management (DSM) incentive programs in the large industrial sector, performance incentive programs can be an effective means to maximize the reliability of planned energy savings. Performance incentive programs balance the objectives of high participation rates with persistent energy savings by: (1) providing financial incentives and resources to minimize constraints to investment in energy efficiency, and (2) requiring that incentive payments be dependent on measured energy savings over time. As BC Hydro increases its DSM initiatives to meet the Clean Energy Act objective to reduce at least 66 per cent of new electricity demand with DSM by 2020, the utility is faced with a higher level of DSM risk, or uncertainties that impact the costeffective acquisition of planned energy savings. For industrial DSM incentive programs, DSM risk can be broken down into project development and project performance risks. Development risk represents the project ramp-up phase and is the risk that planned energy savings do not materialize due to low customer response to program incentives. Performance risk represents the operational phase and is the risk that planned energy savings do not persist over the effective measure life. DSM project development and performance risks are, in turn, a result of industrial economic, technological and organizational conditions, or DSM risk factors. In the BC large industrial sector, and characteristic of large industrial sectors in general, these DSM risk factors include: (1) capital constraints to investment in energy efficiency, (2) commodity price volatility, (3) limited internal staffing resources to deploy towards energy efficiency, (4) variable load, process-based energy saving potential, and (5) a lack of organizational awareness of an operation's energy efficiency over time (energy performance). This research assessed the capacity of alternative performance incentive program models to manage DSM risk in BC. Three performance incentive program models were assessed and compared to BC Hydro's current large industrial DSM incentive program, Power Smart Partners -- Transmission Project Incentives, itself a performance incentive-based program. Together, the selected program models represent a continuum of program design and implementation in terms of the schedule and level of incentives provided, the duration and rigour of measurement and verification (M&V), energy efficiency measures targeted and involvement of the private sector. A multi criteria assessment framework was developed to rank the capacity of each program model to manage BC large industrial DSM risk factors. DSM risk management rankings were then compared to program costeffectiveness, targeted energy savings potential in BC and survey results from BC industrial firms on the program models. The findings indicate that the reliability of DSM energy savings in the BC large industrial sector can be maximized through performance incentive program models that: (1) offer incentives jointly for capital and low-cost operations and maintenance (O&M) measures, (2) allow flexible lead times for project development, (3) utilize rigorous M&V methods capable of measuring variable load, process-based energy savings, (4) use moderate contract lengths that align with effective measure life, and (5) integrate energy management software tools capable of providing energy performance feedback to customers to maximize the persistence of energy savings. While this study focuses exclusively on the BC large industrial sector, the findings of this research have applicability to all energy utilities serving large, energy intensive industrial sectors.

Test-Retest Reliability of Innovated Strength Tests for Hip Muscles

PubMed Central

Meyer, Christophe; Corten, Kristoff; Wesseling, Mariska; Peers, Koen; Simon, Jean-Pierre; Jonkers, Ilse; Desloovere, Kaat

2013-01-01

The burden of hip muscles weakness and its relation to other impairments has been well documented. It is therefore a pre-requisite to have a reliable method for clinical assessment of hip muscles function allowing the design and implementation of a proper strengthening program. Motor-driven dynamometry has been widely accepted as the gold-standard for lower limb muscle strength assessment but is mainly related to the knee joint. Studies focusing on the hip joint are less exhaustive and somewhat discrepant with regard to optimal participants position, consequently influencing outcome measures. Thus, we aimed to develop a standardized test setup for the assessment of hip muscles strength, i.e. flexors/extensors and abductors/adductors, with improved participant stability and to define its psychometric characteristics. Eighteen participants performed unilateral isokinetic and isometric contractions of the hip muscles in the sagittal and coronal plane at two separate occasions. Peak torque and normalized peak torque were measured for each contraction. Relative and absolute measures of reliability were calculated using the intraclass correlation coefficient and standard error of measurement, respectively. Results from this study revealed higher levels of between-day reliability of isokinetic/isometric hip abduction/flexion peak torque compared to existing literature. The least reliable measures were found for hip extension and adduction, which could be explained by a less efficient stabilization technique. Our study additionally provided a first set of reference normalized data which can be used in future research. PMID:24260550

Using the Knowledge, Process, Practice (KPP) model for driving the design and development of online postgraduate medical education.

PubMed

Shaw, Tim; Barnet, Stewart; Mcgregor, Deborah; Avery, Jennifer

2015-01-01

Online learning is a primary delivery method for continuing health education programs. It is critical that programs have curricula objectives linked to educational models that support learning. Using a proven educational modelling process ensures that curricula objectives are met and a solid basis for learning and assessment is achieved. To develop an educational design model that produces an educationally sound program development plan for use by anyone involved in online course development. We have described the development of a generic educational model designed for continuing health education programs. The Knowledge, Process, Practice (KPP) model is founded on recognised educational theory and online education practice. This paper presents a step-by-step guide on using this model for program development that encases reliable learning and evaluation. The model supports a three-step approach, KPP, based on learning outcomes and supporting appropriate assessment activities. It provides a program structure for online or blended learning that is explicit, educationally defensible, and supports multiple assessment points for health professionals. The KPP model is based on best practice educational design using a structure that can be adapted for a variety of online or flexibly delivered postgraduate medical education programs.

Defining and assessing professional competence.

PubMed

Epstein, Ronald M; Hundert, Edward M

2002-01-09

Current assessment formats for physicians and trainees reliably test core knowledge and basic skills. However, they may underemphasize some important domains of professional medical practice, including interpersonal skills, lifelong learning, professionalism, and integration of core knowledge into clinical practice. To propose a definition of professional competence, to review current means for assessing it, and to suggest new approaches to assessment. We searched the MEDLINE database from 1966 to 2001 and reference lists of relevant articles for English-language studies of reliability or validity of measures of competence of physicians, medical students, and residents. We excluded articles of a purely descriptive nature, duplicate reports, reviews, and opinions and position statements, which yielded 195 relevant citations. Data were abstracted by 1 of us (R.M.E.). Quality criteria for inclusion were broad, given the heterogeneity of interventions, complexity of outcome measures, and paucity of randomized or longitudinal study designs. We generated an inclusive definition of competence: the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served. Aside from protecting the public and limiting access to advanced training, assessments should foster habits of learning and self-reflection and drive institutional change. Subjective, multiple-choice, and standardized patient assessments, although reliable, underemphasize important domains of professional competence: integration of knowledge and skills, context of care, information management, teamwork, health systems, and patient-physician relationships. Few assessments observe trainees in real-life situations, incorporate the perspectives of peers and patients, or use measures that predict clinical outcomes. In addition to assessments of basic skills, new formats that assess clinical reasoning, expert judgment, management of ambiguity, professionalism, time management, learning strategies, and teamwork promise a multidimensional assessment while maintaining adequate reliability and validity. Institutional support, reflection, and mentoring must accompany the development of assessment programs.

Proceedings of the international meeting on thermal nuclear reactor safety. Vol. 1

DOE Office of Scientific and Technical Information (OSTI.GOV)

None

Separate abstracts are included for each of the papers presented concerning current issues in nuclear power plant safety; national programs in nuclear power plant safety; radiological source terms; probabilistic risk assessment methods and techniques; non LOCA and small-break-LOCA transients; safety goals; pressurized thermal shocks; applications of reliability and risk methods to probabilistic risk assessment; human factors and man-machine interface; and data bases and special applications.

Bulk electric system reliability evaluation incorporating wind power and demand side management

NASA Astrophysics Data System (ADS)

Huang, Dange

Electric power systems are experiencing dramatic changes with respect to structure, operation and regulation and are facing increasing pressure due to environmental and societal constraints. Bulk electric system reliability is an important consideration in power system planning, design and operation particularly in the new competitive environment. A wide range of methods have been developed to perform bulk electric system reliability evaluation. Theoretically, sequential Monte Carlo simulation can include all aspects and contingencies in a power system and can be used to produce an informative set of reliability indices. It has become a practical and viable tool for large system reliability assessment technique due to the development of computing power and is used in the studies described in this thesis. The well-being approach used in this research provides the opportunity to integrate an accepted deterministic criterion into a probabilistic framework. This research work includes the investigation of important factors that impact bulk electric system adequacy evaluation and security constrained adequacy assessment using the well-being analysis framework. Load forecast uncertainty is an important consideration in an electrical power system. This research includes load forecast uncertainty considerations in bulk electric system reliability assessment and the effects on system, load point and well-being indices and reliability index probability distributions are examined. There has been increasing worldwide interest in the utilization of wind power as a renewable energy source over the last two decades due to enhanced public awareness of the environment. Increasing penetration of wind power has significant impacts on power system reliability, and security analyses become more uncertain due to the unpredictable nature of wind power. The effects of wind power additions in generating and bulk electric system reliability assessment considering site wind speed correlations and the interactive effects of wind power and load forecast uncertainty on system reliability are examined. The concept of the security cost associated with operating in the marginal state in the well-being framework is incorporated in the economic analyses associated with system expansion planning including wind power and load forecast uncertainty. Overall reliability cost/worth analyses including security cost concepts are applied to select an optimal wind power injection strategy in a bulk electric system. The effects of the various demand side management measures on system reliability are illustrated using the system, load point, and well-being indices, and the reliability index probability distributions. The reliability effects of demand side management procedures in a bulk electric system including wind power and load forecast uncertainty considerations are also investigated. The system reliability effects due to specific demand side management programs are quantified and examined in terms of their reliability benefits.

Probabilistic simulation of uncertainties in thermal structures

NASA Technical Reports Server (NTRS)

Chamis, Christos C.; Shiao, Michael

1990-01-01

Development of probabilistic structural analysis methods for hot structures is a major activity at Lewis Research Center. It consists of five program elements: (1) probabilistic loads; (2) probabilistic finite element analysis; (3) probabilistic material behavior; (4) assessment of reliability and risk; and (5) probabilistic structural performance evaluation. Recent progress includes: (1) quantification of the effects of uncertainties for several variables on high pressure fuel turbopump (HPFT) blade temperature, pressure, and torque of the Space Shuttle Main Engine (SSME); (2) the evaluation of the cumulative distribution function for various structural response variables based on assumed uncertainties in primitive structural variables; (3) evaluation of the failure probability; (4) reliability and risk-cost assessment, and (5) an outline of an emerging approach for eventual hot structures certification. Collectively, the results demonstrate that the structural durability/reliability of hot structural components can be effectively evaluated in a formal probabilistic framework. In addition, the approach can be readily extended to computationally simulate certification of hot structures for aerospace environments.

Changes in J-SOAP-II and SAVRY Scores Over the Course of Residential, Cognitive-Behavioral Treatment for Adolescent Sexual Offending

PubMed Central

Viljoen, Jodi L.; Gray, Andrew L.; Shaffer, Catherine; Latzman, Natasha E.; Scalora, Mario J.; Ullman, Daniel

2018-01-01

Although the Juvenile Sex Offender Assessment Protocol–II (J-SOAP-II) and the Structured Assessment of Violence Risk in Youth (SAVRY) include an emphasis on dynamic, or modifiable factors, there has been little research on dynamic changes on these tools. To help address this gap, we compared admission and discharge scores of 163 adolescents who attended a residential, cognitive-behavioral treatment program for sexual offending. Based on reliable change indices, one half of youth showed a reliable decrease on the J-SOAP-II Dynamic Risk Total Score and one third of youth showed a reliable decrease on the SAVRY Dynamic Risk Total Score. Contrary to expectations, decreases in risk factors and increases in protective factors did not predict reduced sexual, violent nonsexual, or any reoffending. In addition, no associations were found between scores on the Psychopathy Checklist:Youth Version and levels of change. Overall, the J-SOAP-II and the SAVRY hold promise in measuring change, but further research is needed. PMID:26199271

Changes in J-SOAP-II and SAVRY Scores Over the Course of Residential, Cognitive-Behavioral Treatment for Adolescent Sexual Offending.

PubMed

Viljoen, Jodi L; Gray, Andrew L; Shaffer, Catherine; Latzman, Natasha E; Scalora, Mario J; Ullman, Daniel

2017-06-01

Although the Juvenile Sex Offender Assessment Protocol-II (J-SOAP-II) and the Structured Assessment of Violence Risk in Youth (SAVRY) include an emphasis on dynamic, or modifiable factors, there has been little research on dynamic changes on these tools. To help address this gap, we compared admission and discharge scores of 163 adolescents who attended a residential, cognitive-behavioral treatment program for sexual offending. Based on reliable change indices, one half of youth showed a reliable decrease on the J-SOAP-II Dynamic Risk Total Score and one third of youth showed a reliable decrease on the SAVRY Dynamic Risk Total Score. Contrary to expectations, decreases in risk factors and increases in protective factors did not predict reduced sexual, violent nonsexual, or any reoffending. In addition, no associations were found between scores on the Psychopathy Checklist:Youth Version and levels of change. Overall, the J-SOAP-II and the SAVRY hold promise in measuring change, but further research is needed.

[Knowledge of university students in Szeged, Hungary about reliable contraceptive methods and sexually transmitted diseases].

PubMed

Devosa, Iván; Kozinszky, Zoltán; Vanya, Melinda; Szili, Károly; Fáyné Dombi, Alice; Barabás, Katalin

2016-04-03

Promiscuity and lack of use of reliable contraceptive methods increase the probability of sexually transmitted diseases and the risk of unwanted pregnancies, which are quite common among university students. The aim of the study was to assess the knowledge of university students about reliable contraceptive methods and sexually transmitted diseases, and to assess the effectiveness of the sexual health education in secondary schools, with specific focus on the education held by peers. An anonymous, self-administered questionnaire survey was carried out in a randomized sample of students at the University of Szeged (n = 472, 298 women and 174 men, average age 21 years) between 2009 and 2011. 62.1% of the respondents declared that reproductive health education lessons in high schools held by peers were reliable and authentic source of information, 12.3% considered as a less reliable source, and 25.6% defined the school health education as irrelevant source. Among those, who considered the health education held by peers as a reliable source, there were significantly more females (69.3% vs. 46.6%, p = 0.001), significantly fewer lived in cities (83.6% vs. 94.8%, p = 0.025), and significantly more responders knew that Candida infection can be transmitted through sexual intercourse (79.5% versus 63.9%, p = 0.02) as compared to those who did not consider health education held by peers as a reliable source. The majority of respondents obtained knowledge about sexual issues from the mass media. Young people who considered health educating programs reliable were significantly better informed about Candida disease.

HiRel: Hybrid Automated Reliability Predictor (HARP) integrated reliability tool system, (version 7.0). Volume 4: HARP Output (HARPO) graphics display user's guide

NASA Technical Reports Server (NTRS)

Sproles, Darrell W.; Bavuso, Salvatore J.

1994-01-01

The Hybrid Automated Reliability Predictor (HARP) integrated Reliability (HiRel) tool system for reliability/availability prediction offers a toolbox of integrated reliability/availability programs that can be used to customize the user's application in a workstation or nonworkstation environment. HiRel consists of interactive graphical input/output programs and four reliability/availability modeling engines that provide analytical and simulative solutions to a wide host of highly reliable fault-tolerant system architectures and is also applicable to electronic systems in general. The tool system was designed at the outset to be compatible with most computing platforms and operating systems and some programs have been beta tested within the aerospace community for over 8 years. This document is a user's guide for the HiRel graphical postprocessor program HARPO (HARP Output). HARPO reads ASCII files generated by HARP. It provides an interactive plotting capability that can be used to display alternate model data for trade-off analyses. File data can also be imported to other commercial software programs.

Application of reliability-centered-maintenance to BWR ECCS motor operator valve performance

DOE Office of Scientific and Technical Information (OSTI.GOV)

Feltus, M.A.; Choi, Y.A.

1993-01-01

This paper describes the application of reliability-centered maintenance (RCM) methods to plant probabilistic risk assessment (PRA) and safety analyses for four boiling water reactor emergency core cooling systems (ECCSs): (1) high-pressure coolant injection (HPCI); (2) reactor core isolation cooling (RCIC); (3) residual heat removal (RHR); and (4) core spray systems. Reliability-centered maintenance is a system function-based technique for improving a preventive maintenance program that is applied on a component basis. Those components that truly affect plant function are identified, and maintenance tasks are focused on preventing their failures. The RCM evaluation establishes the relevant criteria that preserve system function somore » that an RCM-focused approach can be flexible and dynamic.« less

NASA's computer science research program

NASA Technical Reports Server (NTRS)

Larsen, R. L.

1983-01-01

Following a major assessment of NASA's computing technology needs, a new program of computer science research has been initiated by the Agency. The program includes work in concurrent processing, management of large scale scientific databases, software engineering, reliable computing, and artificial intelligence. The program is driven by applications requirements in computational fluid dynamics, image processing, sensor data management, real-time mission control and autonomous systems. It consists of university research, in-house NASA research, and NASA's Research Institute for Advanced Computer Science (RIACS) and Institute for Computer Applications in Science and Engineering (ICASE). The overall goal is to provide the technical foundation within NASA to exploit advancing computing technology in aerospace applications.

The Faculty Self-Reported Assessment Survey (FRAS): Differentiating Faculty Knowledge and Experience in Assessment

PubMed Central

Hanauer, David I.; Bauerle, Cynthia

2015-01-01

Science, technology, engineering, and mathematics education reform efforts have called for widespread adoption of evidence-based teaching in which faculty members attend to student outcomes through assessment practice. Awareness about the importance of assessment has illuminated the need to understand what faculty members know and how they engage with assessment knowledge and practice. The Faculty Self-Reported Assessment Survey (FRAS) is a new instrument for evaluating science faculty assessment knowledge and experience. Instrument validation was composed of two distinct studies: an empirical evaluation of the psychometric properties of the FRAS and a comparative known-groups validation to explore the ability of the FRAS to differentiate levels of faculty assessment experience. The FRAS was found to be highly reliable (α = 0.96). The dimensionality of the instrument enabled distinction of assessment knowledge into categories of program design, instrumentation, and validation. In the known-groups validation, the FRAS distinguished between faculty groups with differing levels of assessment experience. Faculty members with formal assessment experience self-reported higher levels of familiarity with assessment terms, higher frequencies of assessment activity, increased confidence in conducting assessment, and more positive attitudes toward assessment than faculty members who were novices in assessment. These results suggest that the FRAS can reliably and validly differentiate levels of expertise in faculty knowledge of assessment. PMID:25976653

NDE detectability of fatigue-type cracks in high-strength alloys: NDI reliability assessments

NASA Technical Reports Server (NTRS)

Christner, Brent K.; Long, Donald L.; Rummel, Ward D.

1988-01-01

This program was conducted to generate quantitative flaw detection capability data for the nondestructive evaluation (NDE) techniques typically practiced by aerospace contractors. Inconel 718 and Haynes 188 alloy test specimens containing fatigue flaws with a wide distribution of sizes were used to assess the flaw detection capabilities at a number of contractor and government facilities. During this program 85 inspection sequences were completed presenting a total of 20,994 fatigue cracks to 53 different inspectors. The inspection sequences completed included 78 liquid penetrant, 4 eddy current, and 3 ultrasonic evaluations. The results of the assessment inspections are presented and discussed. In generating the flaw detection capability data base, procedures for data collection, data analysis, and specimen care and maintenance were developed, demonstrated, and validated. The data collection procedures and methods that evolved during this program for the measurement of flaw detection capabilities and the effects of inspection variables on performance are discussed. The Inconel 718 and Haynes 188 test specimens that were used in conducting this program and the NDE assessment procedures that were demonstrated, provide NASA with the capability to accurately assess the flaw detection capabilities of specific inspection procedures being applied or proposed for use on current and future fracture control hardware program.

FLiGS Score: A New Method of Outcome Assessment for Lip Carcinoma–Treated Patients

PubMed Central

Grassi, Rita; Toia, Francesca; Di Rosa, Luigi; Cordova, Adriana

2015-01-01

Background: Lip cancer and its treatment have considerable functional and cosmetic effects with resultant nutritional and physical detriments. As we continue to investigate new treatment regimens, we are simultaneously required to assess postoperative outcomes to design interventions that lessen the adverse impact of this disease process. We wish to introduce Functional Lip Glasgow Scale (FLiGS) score as a new method of outcome assessment to measure the effect of lip cancer and its treatment on patients’ daily functioning. Methods: Fifty patients affected by lip squamous cell carcinoma were recruited between 2009 and 2013. Patients were asked to fill the FLiGS questionnaire before surgery, 1 month, 6 months, and 1 year after surgery. The subscores were used to calculate a total FLiGS score of global oral disability. Statistical analysis was performed to test validity and reliability. Results: FLiGS scores improved significantly from preoperative to 12 months postoperative values (P = 0.000). Statistical evidence of validity was provided through rs (Spearman correlation coefficient) that resulted >0.30 for all surveys and for which P < 0.001. FLiGS score reliability was shown through examination of internal consistency and test-retest reliability. Conclusions: FLiGS score is a simple way of assessing functional impairment related to lip cancer before and after surgery; it is sensitive, valid, reliable, and clinically relevant: it provides useful information to orient the physician in the postoperative management and in the rehabilitation program. PMID:26034652

Development of a self-report questionnaire designed for population-based surveillance of gingivitis in adolescents: assessment of content validity and reliability

PubMed Central

QUIROZ, Viviana; REINERO, Daniela; HERNÁNDEZ, Patricia; CONTRERAS, Johanna; VERNAL, Rolando; CARVAJAL, Paola

2017-01-01

Abstract The major infectious diseases in Chile encompass the periodontal diseases, with a combined prevalence that rises up to 90% of the population. Thus, the population-based surveillance of periodontal diseases plays a central role for assessing their prevalence and for planning, implementing, and evaluating preventive and control programs. Self-report questionnaires have been proposed for the surveillance of periodontal diseases in adult populations world-wide. Objective This study aimed to develop and assess the content validity and reliability of a cognitively adapted self-report questionnaire designed for surveillance of gingivitis in adolescents. Material and Methods Ten predetermined self-report questions evaluating early signs and symptoms of gingivitis were preliminary assessed by a panel of clinical experts. Eight questions were selected and cognitively tested in 20 adolescents aged 12 to 18 years from Santiago de Chile. The questionnaire was then conducted and answered by 178 Chilean adolescents. Internal consistency was measured using the Cronbach’s alpha and temporal stability was calculated using the Kappa-index. Results A reliable final self-report questionnaire consisting of 5 questions was obtained, with a total Cronbach’s alpha of 0.73 and a Kappa-index ranging from 0.41 to 0.77 between the different questions. Conclusions The proposed questionnaire is reliable, with an acceptable internal consistency and a temporal stability from moderate to substantial, and it is promising for estimating the prevalence of gingivitis in adolescents. PMID:28877279

CLASS Reliability Training as Professional Development for Preschool Teachers

ERIC Educational Resources Information Center

Casbergue, Renée M.; Bedford, April Whatley; Burstein, Karen

2014-01-01

Use of the Classroom Assessment Scoring System (CLASS) is increasing across the United States as an important indicator of the quality of programs for young children. Professional development is required to facilitate teachers' understanding of the instructional behaviors upon which they will be judged. This study investigated the use of the…

An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator: Evidence from Mexico's PROGRESA Program

ERIC Educational Resources Information Center

Diaz, Juan Jose; Handa, Sudhanshu

2006-01-01

Not all policy questions can be addressed by social experiments. Nonexperimental evaluation methods provide an alternative to experimental designs but their results depend on untestable assumptions. This paper presents evidence on the reliability of propensity score matching (PSM), which estimates treatment effects under the assumption of…

Prediction of School Performance from the Minnesota Child Development Inventory: Implications for Preschool Screening.

ERIC Educational Resources Information Center

Colligan, Robert C.

Almost all preschool screening programs depend entirely on information and observations obtained during a brief evaluative session with the child. However, the logistics involved in managing large numbers of parents and children, the use of volunteers having varying degrees of sophistication or competency in assessment, the reliability and…

Using Microcomputer-Based Logistics Models to Enhance Supportability Assessment for the USAF Productivity, Reliability, Availability and Maintainability (PRAM) Program Office: A Tailored Approach

DTIC Science & Technology

1989-09-01

goes on to discuss how the innovation process should function within an organization, including five specific phases for successfully managing ... innovation : the recognition of opportunity; idea formulation; product defin- ition; prototype solution; and finally, technology utiliza- tion and diffusion

SHEDS-Multimedia Model Version 3 (a) Technical Manual; (b) User Guide; and (c) Executable File to Launch SAS Program and Install Model

EPA Science Inventory

Reliable models for assessing human exposures are important for understanding health risks from chemicals. The Stochastic Human Exposure and Dose Simulation model for multimedia, multi-route/pathway chemicals (SHEDS-Multimedia), developed by EPA’s Office of Research and Developm...

Vegetable behavioral tool demonstrates validity with MyPlate vegetable cups and carotenoid and inflammatory biomarkers

USDA-ARS?s Scientific Manuscript database

Young children are not meeting recommendations for vegetable intake. Our objective is to provide evidence of validity and reliability for a pictorial vegetable behavioral assessment for use by federally funded community nutrition programs. Parent/child pairs (n=133) from Head Start and the Special S...

Assessment Practices of School Psychologists When Identifying Children for SED Classes.

ERIC Educational Resources Information Center

Strelnieks, Maija; Wessel, Joan

This study investigated the procedures used by psychologists in a large midwestern urban area for the initial diagnosis and placement of elementary children with severe emotional disturbance (SED) in educational programs in light of the widespread criticism of the use of projective tests due to the questionable reliability of the tests and…

Market study phase 2 follow-up activity. The Baylor Mark 3 Haploscope

NASA Technical Reports Server (NTRS)

1977-01-01

Efforts to accelerate commercialization of the haploscope, and to determine quickly and reliably the level of manufacturer interest in the product are presented. The nature of the decision making process within firms as it concerns project selection and new product evaluation is discussed. Implications for the NASA marketing program were assessed.

Measuring the Quality of Inclusive Practices: Findings from the Inclusive Classroom Profile Pilot

ERIC Educational Resources Information Center

Soukakou, Elena P.; Winton, Pam J.; West, Tracey A.; Sideris, John H.; Rucker, Lia M.

2014-01-01

The purpose of this study was to test the reliability and validity of the Inclusive Classroom Profile (ICP), an observation measure designed to assess the quality of classroom practices in inclusive preschool programs. The measure was field tested in 51 inclusive classrooms. Results confirmed and extended previous research findings, providing…

Assessment of first-year veterinary students' communication skills using an objective structured clinical examination: the importance of context.

PubMed

Hecker, Kent G; Adams, Cindy L; Coe, Jason B

2012-01-01

Communication skills are considered to be a core clinical skill in veterinary medicine and essential for practice success, including outcomes of care for patients and clients. While veterinary schools include communication skills training in their programs, there is minimal knowledge on how best to assess communication competence throughout the undergraduate program. The purpose of this study was to further our understanding of the reliability, utility, and suitability of a communication skills Objective Structured Clinical Examination (OSCE). Specifically we wanted to (1) identify the greatest source of variability (student, rater, station, and track) within a first-year, four station OSCE using exam scores and scores from videotape review by two trained raters, and (2) determine the effect of different stations on students' communication skills performance. Reliability of the scores from both the exam data and the two expert raters was 0.50 and 0.46 respectively, with the greatest amount of variance attributable to student by station. The percentage of variance due to raters in the exam data was 16.35%, whereas the percentage of variance for the two expert raters was 0%. These results have three important implications. First, the results reinforce the need for communication educators to emphasize that use of communication skills is moderated by the context of the clinical interaction. Second, by increasing rater training the amount of error in the scores due to raters can be reduced and inter-rater reliability increases. Third, the communication assessment method (in this case the OSCE checklist) should be built purposefully, taking into consideration the context of the case.

Validation of Satellite Aerosol Retrievals from AERONET Ground-Based Measurements

NASA Technical Reports Server (NTRS)

Holben, Brent; Remer, Lorraine; Torres, Omar; Zhao, Tom; Smith, David E. (Technical Monitor)

2001-01-01

Accurate and comprehensive assessment of the parameters that control key atmospheric and biospheric processes including assessment of anthropogenic effects on climate change is a fundamental measurement objective of NASA's EOS program (King and Greenstone, 1999). Satellite assessment programs and associated global climate models require validation and additional parameterization with frequent reliable ground-based observations. A critical and highly uncertain element of the measurement program is characterization of tropospheric aerosols requiring basic observations of aerosols optical and microphysical properties. Unfortunately as yet we do not know the aerosol burden man is contributing to the atmosphere and thus we will have no definitive measure of change for the future. This lack of aerosol assessment is the impetus for some of the EOS measurement activities (Kaufman et al., 1997; King et al., 1999) and the formation of the AERONET program (Holben et al., 1998). The goals of the AERONET program are to develop long term monitoring at globally distributed sites providing critical data for multiannual trend changes in aerosol loading and optical properties with the specific goal of providing a data base for validation of satellite derived aerosol optical properties. The AERONET program has evolved into an international federated network of approximately 100 ground-based remote sensing monitoring stations to characterize the optical and microphysical properties of aerosols.

Development of a scale to measure individuals’ ratings of peace

PubMed Central

2014-01-01

Background The evolving concept of peace-building and the interplay between peace and health is examined in many venues, including at the World Health Assembly. However, without a metric to determine effectiveness of intervention programs all efforts are prone to subjective assessment. This paper develops a psychometric index that lays the foundation for measuring community peace stemming from intervention programs. Methods After developing a working definition of ‘peace’ and delineating a Peace Evaluation Across Cultures and Environments (PEACE) scale with seven constructs comprised of 71 items, a beta version of the index was pilot-tested. Two hundred and fifty subjects in three sites in the U.S. were studied using a five-point Likert scale to evaluate the psychometric functioning of the PEACE scale. Known groups validation was performed using the SOS-10. In addition, test-retest reliability was performed on 20 subjects. Results The preliminary data demonstrated that the scale has acceptable psychometric properties for measuring an individual’s level of peacefulness. The study also provides reliability and validity data for the scale. The data demonstrated internal consistency, correlation between data and psychological well-being, and test-retest reliability. Conclusions The PEACE scale may serve as a novel assessment tool in the health sector and be valuable in monitoring and evaluating the peace-building impact of health initiatives in conflict-affected regions. PMID:25298781

Psychometric Characteristics of Process Evaluation Measures for a Rural School-based Childhood Obesity Prevention Study: Louisiana Health

PubMed Central

Newton, R. L.; Thomson, J. L.; Rau, K.; Duhe’, S.; Sample, A.; Singleton, N.; Anton, S. D.; Webber, L. S.; Williamson, D. A.

2011-01-01

Purpose To evaluate the implementation of intervention components of the Louisiana Health study, which was a multi-component childhood obesity prevention program conducted in rural schools. Design Content analysis. Setting Process evaluation assessed implementation in the classrooms, gym classes, and cafeterias. Subjects Classroom teachers (n = 232), physical education teachers (n = 53), food service managers (n = 33), and trained observers (n = 9). Measures Five process evaluation measures were created: Physical Education Questionnaire (PEQ), Intervention Questionnaire (IQ), Food Service Manager Questionnaire (FSMQ), Classroom Observation (CO) and School Nutrition Environment Observation (SNEO). Analysis Inter-rater reliability and internal consistency were conducted on all measures. ANOVA and Chi-square were used to compare differences across study groups on questionnaires and observations. Results The PEQ and one sub-scale from the FSMQ were eliminated because their reliability coefficients fell below acceptable standards. The sub-scale internal consistencies for the IQ, FSMQ, CO, and SNEO (all Cronbach’s α > .60) were acceptable. Conclusions After the initial 4 months of intervention, there was evidence that the Louisiana Health intervention was being implemented as it was designed. In summary, four process evaluation measures were found to be sufficiently reliable and valid for assessing the delivery of various aspects of a school-based obesity prevention program. These process measures could be modified to evaluate the delivery of other similar school-based interventions. PMID:21721969

Quality management for space systems in ISRO

NASA Astrophysics Data System (ADS)

Satish, S.; Selva Raju, S.; Nanjunda Swamy, T. S.; Kulkarni, P. L.

2009-11-01

In a little over four decades, the Indian Space Program has carved a niche for itself with the unique application driven program oriented towards National development. The end-to-end capability approach of the space projects in the country call for innovative practices and procedures in assuring the quality and reliability of space systems. The System Reliability (SR) efforts initiated at the start of the projects continue during the entire life cycle of the project encompassing design, development, realisation, assembly, testing and integration and during launch. Even after the launch, SR groups participate in the on-orbit evaluation of transponders in communication satellites and camera systems in remote sensing satellites. SR groups play a major role in identification, evaluation and inculcating quality practices in work centres involved in the fabrication of mechanical, electronics and propulsion systems required for Indian Space Research Organization's (ISRO's) launch vehicle and spacecraft projects. Also the reliability analysis activities like prediction, assessment and demonstration as well as de-rating analysis, Failure Mode Effects and Criticality Analysis (FMECA) and worst-case analysis are carried out by SR groups during various stages of project realisation. These activities provide the basis for project management to take appropriate techno-managerial decisions to ensure that the required reliability goals are met. Extensive test facilities catering to the needs of the space program has been set up. A system for consolidating the experience and expertise gained for issue of standards called product assurance specifications to be used in all ISRO centres has also been established.

Building model analysis applications with the Joint Universal Parameter IdenTification and Evaluation of Reliability (JUPITER) API

USGS Publications Warehouse

Banta, E.R.; Hill, M.C.; Poeter, E.; Doherty, J.E.; Babendreier, J.

2008-01-01

The open-source, public domain JUPITER (Joint Universal Parameter IdenTification and Evaluation of Reliability) API (Application Programming Interface) provides conventions and Fortran-90 modules to develop applications (computer programs) for analyzing process models. The input and output conventions allow application users to access various applications and the analysis methods they embody with a minimum of time and effort. Process models simulate, for example, physical, chemical, and (or) biological systems of interest using phenomenological, theoretical, or heuristic approaches. The types of model analyses supported by the JUPITER API include, but are not limited to, sensitivity analysis, data needs assessment, calibration, uncertainty analysis, model discrimination, and optimization. The advantages provided by the JUPITER API for users and programmers allow for rapid programming and testing of new ideas. Application-specific coding can be in languages other than the Fortran-90 of the API. This article briefly describes the capabilities and utility of the JUPITER API, lists existing applications, and uses UCODE_2005 as an example.

10 CFR 712.19 - Removal from HRP.

Code of Federal Regulations, 2010 CFR

2010-01-01

... OF ENERGY HUMAN RELIABILITY PROGRAM Establishment of and Procedures for the Human Reliability Program... immediately remove that individual from HRP duties pending a determination of the individual's reliability. A... HRP duties pending a determination of the individual's reliability is an interim, precautionary action...

Development of a Self-Rated Mixed Methods Skills Assessment: The NIH Mixed Methods Research Training Program for the Health Sciences

PubMed Central

Guetterman, Timothy C.; Creswell, John W.; Wittink, Marsha; Barg, Fran K.; Castro, Felipe G.; Dahlberg, Britt; Watkins, Daphne C.; Deutsch, Charles; Gallo, Joseph J.

2017-01-01

Introduction Demand for training in mixed methods is high, with little research on faculty development or assessment in mixed methods. We describe the development of a Self-Rated Mixed Methods Skills Assessment and provide validity evidence. The instrument taps six research domains: “Research question,” “Design/approach,” “Sampling,” “Data collection,” “Analysis,” and “Dissemination.” Respondents are asked to rate their ability to define or explain concepts of mixed methods under each domain, their ability to apply the concepts to problems, and the extent to which they need to improve. Methods We administered the questionnaire to 145 faculty and students using an internet survey. We analyzed descriptive statistics and performance characteristics of the questionnaire using Cronbach’s alpha to assess reliability and an ANOVA that compared a mixed methods experience index with assessment scores to assess criterion-relatedness. Results Internal consistency reliability was high for the total set of items (.95) and adequate (>=.71) for all but one subscale. Consistent with establishing criterion validity, respondents who had more professional experiences with mixed methods (e.g., published a mixed methods paper) rated themselves as more skilled, which was statistically significant across the research domains. Discussion This Self-Rated Mixed Methods Assessment instrument may be a useful tool to assess skills in mixed methods for training programs. It can be applied widely at the graduate and faculty level. For the learner, assessment may lead to enhanced motivation to learn and training focused on self-identified needs. For faculty, the assessment may improve curriculum and course content planning. PMID:28562495

Software reliability: Application of a reliability model to requirements error analysis

NASA Technical Reports Server (NTRS)

Logan, J.

1980-01-01

The application of a software reliability model having a well defined correspondence of computer program properties to requirements error analysis is described. Requirements error categories which can be related to program structural elements are identified and their effect on program execution considered. The model is applied to a hypothetical B-5 requirement specification for a program module.

Superconducting magnet development for tokamaks and mirrors: a technical assessment

DOE Office of Scientific and Technical Information (OSTI.GOV)

Laverick, C.; Jacobs, R. B.; Boom, R. W.

1977-11-01

The role of superconducting magnets in Magnetic Fusion Energy Research and Development is assessed from a consideration of program plans and schedules, the present status of the programs and the research and development suggestions arising from recent studies and workshops. A principal conclusion is that the large superconducting magnet systems needed for commercial magnetic fusion reactors can be constructed. However such magnets working under severe conditions, with increasingly stringent reliability, safety and cost restrictions can never be built unless experience is first gained in a number of important installations designed to prove physics and technology steps on the way tomore » commercial power demonstration. The immediate problem is to design a technology program in the absence of definite device needs and specifications, giving a priority weighting to the multiplicity of good, high quality development program suggestions when all proposals cannot be supported.« less

Assess program: Interactive data management systems for airborne research

NASA Technical Reports Server (NTRS)

Munoz, R. M.; Reller, J. O., Jr.

1974-01-01

Two data systems were developed for use in airborne research. Both have distributed intelligence and are programmed for interactive support among computers and with human operators. The C-141 system (ADAMS) performs flight planning and telescope control functions in addition to its primary role of data acquisition; the CV-990 system (ADDAS) performs data management functions in support of many research experiments operating concurrently. Each system is arranged for maximum reliability in the first priority function, precision data acquisition.

An assessment of laser velocimetry in hypersonic flow

NASA Technical Reports Server (NTRS)

1992-01-01

Although extensive progress has been made in computational fluid mechanics, reliable flight vehicle designs and modifications still cannot be made without recourse to extensive wind tunnel testing. Future progress in the computation of hypersonic flow fields is restricted by the need for a reliable mean flow and turbulence modeling data base which could be used to aid in the development of improved empirical models for use in numerical codes. Currently, there are few compressible flow measurements which could be used for this purpose. In this report, the results of experiments designed to assess the potential for laser velocimeter measurements of mean flow and turbulent fluctuations in hypersonic flow fields are presented. Details of a new laser velocimeter system which was designed and built for this test program are described.

Self-audit of lockout/tagout in manufacturing workplaces: A pilot study.

PubMed

Yamin, Samuel C; Parker, David L; Xi, Min; Stanley, Rodney

2017-05-01

Occupational health and safety (OHS) self-auditing is a common practice in industrial workplaces. However, few audit instruments have been tested for inter-rater reliability and accuracy. A lockout/tagout (LOTO) self-audit checklist was developed for use in manufacturing enterprises. It was tested for inter-rater reliability and accuracy using responses of business self-auditors and external auditors. Inter-rater reliability at ten businesses was excellent (κ = 0.84). Business self-auditors had high (100%) accuracy in identifying elements of LOTO practice that were present as well those that were absent (81% accuracy). Reliability and accuracy increased further when problematic checklist questions were removed from the analysis. Results indicate that the LOTO self-audit checklist would be useful in manufacturing firms' efforts to assess and improve their LOTO programs. In addition, a reliable self-audit instrument removes the need for external auditors to visit worksites, thereby expanding capacity for outreach and intervention while minimizing costs. © 2017 Wiley Periodicals, Inc.

New Brunswick Laboratory: Progress report, October 1987--September 1988

DOE Office of Scientific and Technical Information (OSTI.GOV)

Not Available

NBL has been tasked by the DOE Office of Safeguards and Security, Defense Programs (OSS/DP) to assure the application of accurate and reliable measurement technology for the safeguarding of special nuclear materials. NBL is fulfilling its mission responsibilities by identifying and addressing the measurement and measurement-related needs of the nuclear material safeguards community. These responsibilities are being addressed by activities in the following program areas: (1) reference and calibration materials, (2) measurement development, (3) measurement services, (4) measurement evaluation, (5) safeguards assessment, and (6) site-specific assistance. Highlights of each of these programs areas are provided in this summary.

Reliability, precision, and measurement in the context of data from ability tests, surveys, and assessments

NASA Astrophysics Data System (ADS)

Fisher, W. P., Jr.; Elbaum, B.; Coulter, A.

2010-07-01

Reliability coefficients indicate the proportion of total variance attributable to differences among measures separated along a quantitative continuum by a testing, survey, or assessment instrument. Reliability is usually considered to be influenced by both the internal consistency of a data set and the number of items, though textbooks and research papers rarely evaluate the extent to which these factors independently affect the data in question. Probabilistic formulations of the requirements for unidimensional measurement separate consistency from error by modelling individual response processes instead of group-level variation. The utility of this separation is illustrated via analyses of small sets of simulated data, and of subsets of data from a 78-item survey of over 2,500 parents of children with disabilities. Measurement reliability ultimately concerns the structural invariance specified in models requiring sufficient statistics, parameter separation, unidimensionality, and other qualities that historically have made quantification simple, practical, and convenient for end users. The paper concludes with suggestions for a research program aimed at focusing measurement research more on the calibration and wide dissemination of tools applicable to individuals, and less on the statistical study of inter-variable relations in large data sets.

Psychosocial Adjustment to Illness Scale: Factor structure, reliability, and validity assessment in a sample of Greek breast cancer patients.

PubMed

Kolokotroni, Philippa; Anagnostopoulos, Fotios; Missitzis, Ioannis

2017-07-01

The study and measurement of psychosocial adjustment is important for evaluating patients' well-being, and assessing the illness's course, treatment's success, and patients' recovery. In this study, internal consistency reliability and construct validity of the Greek version of the Psychosocial Adjustment to Illness Scale-Self-Report (PAIS-SR) were examined. Demographic and psychosocial data were collected from a sample of 243 women with breast cancer, recruited from September 2011 to December 2012. With some exceptions in specific items, the original conceptually-derived PAIS-SR subscales emerged in a seven-factor solution. Social Environment, Job and Household Duties, and Psychological Distress accounted for more of the total variance than other subscales. PAIS-SR showed good internal consistency reliability, with Cronbach's alpha coefficients >0.62. Correlations of PAIS-SR domains with measures of quality of life and posttraumatic stress symptoms supported the convergent validity of the PAIS-SR and its significance for cancer research. The Greek version of the PAIS-SR has acceptable internal consistency reliability and construct validity, as well as satisfactory convergent validity. Results provide some suggestions for the development of programs to evaluate adjustment status and implement psychosocial interventions among breast cancer survivors.

Psychometric considerations in the measurement of event-related brain potentials: Guidelines for measurement and reporting.

PubMed

Clayson, Peter E; Miller, Gregory A

2017-01-01

Failing to consider psychometric issues related to reliability and validity, differential deficits, and statistical power potentially undermines the conclusions of a study. In research using event-related brain potentials (ERPs), numerous contextual factors (population sampled, task, data recording, analysis pipeline, etc.) can impact the reliability of ERP scores. The present review considers the contextual factors that influence ERP score reliability and the downstream effects that reliability has on statistical analyses. Given the context-dependent nature of ERPs, it is recommended that ERP score reliability be formally assessed on a study-by-study basis. Recommended guidelines for ERP studies include 1) reporting the threshold of acceptable reliability and reliability estimates for observed scores, 2) specifying the approach used to estimate reliability, and 3) justifying how trial-count minima were chosen. A reliability threshold for internal consistency of at least 0.70 is recommended, and a threshold of 0.80 is preferred. The review also advocates the use of generalizability theory for estimating score dependability (the generalizability theory analog to reliability) as an improvement on classical test theory reliability estimates, suggesting that the latter is less well suited to ERP research. To facilitate the calculation and reporting of dependability estimates, an open-source Matlab program, the ERP Reliability Analysis Toolbox, is presented. Copyright © 2016 Elsevier B.V. All rights reserved.

Test-Retest Reliability, Agreement and Responsiveness of Productivity Loss (iPCQ-VR) and Healthcare Utilization (TiCP-VR) Questionnaires for Sick Workers with Chronic Musculoskeletal Pain.

PubMed

Beemster, Timo T; van Velzen, Judith M; van Bennekom, Coen A M; Reneman, Michiel F; Frings-Dresen, Monique H W

2018-03-16

The purpose of this study was to assess test-retest reliability, agreement, and responsiveness of questionnaires on productivity loss (iPCQ-VR) and healthcare utilization (TiCP-VR) for sick-listed workers with chronic musculoskeletal pain who were referred to vocational rehabilitation. Methods Test-retest reliability and agreement was assessed with a 2-week interval. Responsiveness was assessed at discharge after a 15-week vocational rehabilitation (VR) program. Data was obtained from six Dutch VR centers. Test-retest reliability was determined with intraclass correlation coefficient (ICC) and Cohen's kappa. Agreement was determined by Standard Error of Measurement (SEM), smallest detectable changes (on group and individual level), and percentage observed, positive and negative agreement. Responsiveness was determined with area under the curve (AUC) obtained from receiver operation characteristic (ROC). Results A sample of 52 participants on test-retest reliability and agreement, and a sample of 223 on responsiveness were included in the analysis. Productivity loss (iPCQ-VR): ICCs ranged from 0.52 to 0.90, kappa ranged from 0.42 to 0.96, and AUC ranged from 0.55 to 0.86. Healthcare utilization (TiCP-VR): ICC was 0.81, and kappa values of the single healthcare utilization items ranged from 0.11 to 1.00. Conclusions The iPCQ-VR showed good measurement properties on working status, number of hours working per week and long-term sick leave, and low measurement properties on short-term sick leave and presenteeism. The TiCP-VR showed adequate reliability on all healthcare utilization items together and medication use, but showed low measurement properties on the single healthcare utilization items.

Absolute and relative reliability of isokinetic and isometric trunk strength testing using the IsoMed-2000 dynamometer.

PubMed

Roth, Ralf; Donath, Lars; Kurz, Eduard; Zahner, Lukas; Faude, Oliver

2017-03-01

The present study aimed to assess the between day reliability of isokinetic and isometric peak torque (PT) during trunk measurement on an isokinetic device (IsoMed 2000). Test-retest-protocol on five separate days. Fifteen healthy sport students (8 female and 7 male) aged 21 to 26. PT was assessed in isometric back extension and flexion as well as right and left rotation. Isokinetic strength was captured at a speed of 60°/s and 150°/s for all tasks. For none of the assessed parameters a meaningful variation in PT during test days was observed. Relative reliability (ICC = 0.85-0.96) was excellent for all tasks. Estimates of absolute reliability as Coefficient of Variation (CoV) and Standard Error of Measurement (SEM in Nm/kg lean body mass) remained stable for isometric (6.9% < CoV < 9.4%; 0.15 < SEM < 0.23) and isokinetic mode (60°/s: 3.7% < CoV < 8.6%; 0.08 < SEM < 0.24; 150°/s: 6.9% < CoV < 12.4%; 0.10 < SEM < 0.31). In contrast, reliability between familiarization day and day 1 was lower (6.6% < CoV < 26.2%; 0.10 < SEM < 0.65). Trunk strength measurement in flexion and extension or trunk rotation in either isometric or isokinetic condition is highly reliable. Therefore, it seems possible to elucidate changes which are smaller than 10% due to intervention programs when a preceding familiarization condition was applied. Copyright © 2016 Elsevier Ltd. All rights reserved.

Cultural competence in mental health nursing: validity and internal consistency of the Portuguese version of the multicultural mental health awareness scale-MMHAS.

PubMed

de Almeida Vieira Monteiro, Ana Paula Teixeira; Fernandes, Alexandre Bastos

2016-05-17

Cultural competence is an essential component in rendering effective and culturally responsive services to culturally and ethnically diverse clients. Still, great difficulty exists in assessing the cultural competence of mental health nurses. There are no Portuguese validated measurement instruments to assess cultural competence in mental health nurses. This paper reports a study testing the reliability and validity of the Portuguese version of the Multicultural Mental Health Awareness Scale-MMHAS in a sample of Portuguese nurses. Following a standard forward/backward translation into Portuguese, the adapted version of MMHAS, along with a sociodemographic questionnaire, were applied to a sample of 306 Portuguese nurses (299 males, 77 females; ages 21-68 years, M = 35.43, SD = 9.85 years). A psychometric research design was used with content and construct validity and reliability. Reliability was assessed using internal consistency and item-total correlations. Construct validity was determined using factor analysis. The factor analysis confirmed that the Portuguese version of MMHAS has a three-factor structure of multicultural competencies (Awareness, Knowledge, and Skills) explaining 59.51% of the total variance. Strong content validity and reliability correlations were demonstrated. The Portuguese version of MMHAS has a strong internal consistency, with a Cronbach's alpha of 0.958 for the total scale. The results supported the construct validity and reliability of the Portuguese version of MMHAS, proving that is a reliable and valid measure of multicultural counselling competencies in mental health nursing. The MMHAS Portuguese version can be used to evaluate the effectiveness of multicultural competency training programs in Portuguese-speaking mental health nurses. The scale can also be a useful in future studies of multicultural competencies in Portuguese-speaking nurses.

Validation of an iPad activity to measure preschool children's food and physical activity knowledge and preferences.

PubMed

Wiseman, Nicola; Harris, Neil; Downes, Martin

2017-02-01

Preschool children's knowledge of, and preference for food and physical activity play an important role in the development of lifestyle behaviors throughout childhood. Valid and reliable instruments that are interactive and appealing to preschool children are needed, to obtain quality information in a way that actively engages children and encourages willing participation. The purpose of the current research is to assess the reliability and validity of an adapted computerized (iPad) version of the photo-pair food and exercise questionnaire (PPFEQ). The adaptation of the PPFEQ involved generating the questionnaire as an iPad-based tool, updating the photo-pairs within the questionnaire and testing for validity and reliability. This involved four phases of investigation to assess test-retest reliability, internal consistency, sensitivity to change and percent agreement of the questionnaire. The adaption of the PPFEQ resulted in an 18-item questionnaire, titled the preschool food and play questionnaire (Pre-FPQ). The Pre-FPQ demonstrated acceptable reliability and sensitivity to change. Test-retest reliability and internal consistency improved with age, however, it was evident that the tool was not suitable for children younger than 4 years of age. Children encounter a dynamic world that shapes their knowledge, preferences, choices and behaviors. The Pre-FPQ is an innovative tool to measure preschool children's knowledge of and preference for food and physical activity. The questionnaire offers the advantage of being presented in a well-received modality for preschool children as well as being easy and inexpensive to administer. This new tool is likely to be useful for the assessment of the effectiveness of healthy lifestyle programs implemented in the childcare setting. Future work is needed to refine and improve measures of physical activity preference in preschool children.

Reliability analysis of laminated CMC components through shell subelement techniques

NASA Technical Reports Server (NTRS)

Starlinger, A.; Duffy, S. F.; Gyekenyesi, J. P.

1992-01-01

An updated version of the integrated design program C/CARES (composite ceramic analysis and reliability evaluation of structures) was developed for the reliability evaluation of CMC laminated shell components. The algorithm is now split in two modules: a finite-element data interface program and a reliability evaluation algorithm. More flexibility is achieved, allowing for easy implementation with various finite-element programs. The new interface program from the finite-element code MARC also includes the option of using hybrid laminates and allows for variations in temperature fields throughout the component.

The implementation and use of Ada on distributed systems with high reliability requirements

NASA Technical Reports Server (NTRS)

Knight, J. C.

1986-01-01

The use and implementation of Ada in distributed environments in which reliability is the primary concern were investigted. A distributed system, programmed entirely in Ada, was studied to assess the use of individual tasks without concern for the processor used. Continued development and testing of the fault tolerant Ada testbed; development of suggested changes to Ada to cope with the failures of interest; design of approaches to fault tolerant software in real time systems, and the integration of these ideas into Ada; and the preparation of various papers and presentations were discussed.

Intelligence Assessment Instruments in Adult Prison Populations: A Systematic Review.

PubMed

van Esch, A Y M; Denzel, A D; Scherder, E J A; Masthoff, E D M

2017-10-01

Detection of intellectual disability (ID) in the penitentiary system is important for the following reasons: (a) to provide assistance to people with ID in understanding their legal rights and court proceedings; (b) to facilitate rehabilitation programs tailored to ID patients, which improves the enhancement of their quality of life and reduces their risk of reoffending; and (c) to provide a reliable estimate of the risk of offence recidivism. It requires a short assessment instrument that provides a reliable estimation of a person's intellectual functioning at the earliest possible stage of this process. The aim of this systematic review is (a) to provide an overview of recent short assessment instruments that provide a full-scale IQ score in adult prison populations and (b) to achieve a quality measurement of the validation studies regarding these instruments to determine which tests are most feasible in this target population. The Preferred Reporting Items for Systematic reviews and Meta-Analyses Statement is used to ensure reliability. The Satz-Mögel, an item-reduction short form of the Wechsler Adult Intelligence Scale, shows the highest correlation with the golden standard and is described to be most reliable. Nevertheless, when it comes to applicability in prison populations, the shorter and less verbal Quick Test can be preferred over others. Without affecting these conclusions, major limitations emerge from the present systematic review, which give rise to several important recommendations for further research.

Measuring assessment standards in undergraduate medical programs: Development and validation of AIM tool.

PubMed

Sajjad, Madiha; Khan, Rehan Ahmed; Yasmeen, Rahila

2018-01-01

To develop a tool to evaluate faculty perceptions of assessment quality in an undergraduate medical program. The Assessment Implementation Measure (AIM) tool was developed by a mixed method approach. A preliminary questionnaire developed through literature review was submitted to a panel of 10 medical education experts for a three-round 'Modified Delphi technique'. Panel agreement of > 75% was considered the criterion for inclusion of items in the questionnaire. Cognitive pre-testing of five faculty members was conducted. Pilot study was done with 30 randomly selected faculty members. Content validity index (CVI) was calculated for individual items (I-CVI) and composite scale (S-CVI). Cronbach's alpha was calculated to determine the internal consistency reliability of the tool. The final AIM tool had 30 items after the Delphi process. S-CVI was 0.98 with the S-CVI/Avg method and 0.86 by S-CVI/UA method, suggesting good content validity. Cut-off value of < 0.9 I-CVI was taken as criterion for item deletion. Cognitive pre-testing revealed good item interpretation. Cronbach's alpha calculated for the AIM was 0.9, whereas Cronbach's alpha for the four domains ranged from 0.67 to 0.80. 'AIM' is a relevant and useful instrument with good content validity and reliability of results, and may be used to evaluate the teachers´ perceptions about assessment quality.

NASA trend analysis procedures

NASA Technical Reports Server (NTRS)

1993-01-01

This publication is primarily intended for use by NASA personnel engaged in managing or implementing trend analysis programs. 'Trend analysis' refers to the observation of current activity in the context of the past in order to infer the expected level of future activity. NASA trend analysis was divided into 5 categories: problem, performance, supportability, programmatic, and reliability. Problem trend analysis uncovers multiple occurrences of historical hardware or software problems or failures in order to focus future corrective action. Performance trend analysis observes changing levels of real-time or historical flight vehicle performance parameters such as temperatures, pressures, and flow rates as compared to specification or 'safe' limits. Supportability trend analysis assesses the adequacy of the spaceflight logistics system; example indicators are repair-turn-around time and parts stockage levels. Programmatic trend analysis uses quantitative indicators to evaluate the 'health' of NASA programs of all types. Finally, reliability trend analysis attempts to evaluate the growth of system reliability based on a decreasing rate of occurrence of hardware problems over time. Procedures for conducting all five types of trend analysis are provided in this publication, prepared through the joint efforts of the NASA Trend Analysis Working Group.

Trial application of reliability technology to emergency diesel generators at the Trojan Nuclear Power Plant

DOE Office of Scientific and Technical Information (OSTI.GOV)

Wong, S.M.; Boccio, J.L.; Karimian, S.

1986-01-01

In this paper, a trial application of reliability technology to the emergency diesel generator system at the Trojan Nuclear Power Plant is presented. An approach for formulating a reliability program plan for this system is being developed. The trial application has shown that a reliability program process, using risk- and reliability-based techniques, can be interwoven into current plant operational activities to help in controlling, analyzing, and predicting faults that can challenge safety systems. With the cooperation of the utility, Portland General Electric Co., this reliability program can eventually be implemented at Trojan to track its effectiveness.

HiRel: Hybrid Automated Reliability Predictor (HARP) integrated reliability tool system, (version 7.0). Volume 3: HARP Graphics Oriented (GO) input user's guide

NASA Technical Reports Server (NTRS)

Bavuso, Salvatore J.; Rothmann, Elizabeth; Mittal, Nitin; Koppen, Sandra Howell

1994-01-01

The Hybrid Automated Reliability Predictor (HARP) integrated Reliability (HiRel) tool system for reliability/availability prediction offers a toolbox of integrated reliability/availability programs that can be used to customize the user's application in a workstation or nonworkstation environment. HiRel consists of interactive graphical input/output programs and four reliability/availability modeling engines that provide analytical and simulative solutions to a wide host of highly reliable fault-tolerant system architectures and is also applicable to electronic systems in general. The tool system was designed at the outset to be compatible with most computing platforms and operating systems, and some programs have been beta tested within the aerospace community for over 8 years. This document is a user's guide for the HiRel graphical preprocessor Graphics Oriented (GO) program. GO is a graphical user interface for the HARP engine that enables the drawing of reliability/availability models on a monitor. A mouse is used to select fault tree gates or Markov graphical symbols from a menu for drawing.

Trunk postural adjustments: Medium-term reliability and correlation with changes of clinical outcomes following an 8-week lumbar stabilization exercise program.

PubMed

Boucher, Jean-Alexandre; Preuss, Richard; Henry, Sharon M; Nugent, Marilee; Larivière, Christian

2018-04-22

Low back pain (LBP) has been previously associated with delayed anticipatory postural adjustments (APAs) determined by trunk muscle activation. Lumbar stabilization exercise programs (LSEP) for patients with LBP may restore the trunk neuromuscular control of the lumbar spine, and normalize APAs. This exploratory study aimed at testing the reliability of EMG and kinematics-based postural adjustment measures over an 8-week interval, assessing their sensitivity to LBP status and treatment and examining their relationship with clinical outcomes. Muscle activation of 10 trunk muscles, using surface electromyography (EMG), and lumbar angular kinematics were recorded during a rapid arm-raising/lowering task. Patients with LBP were tested before and after an 8-week LSEP. Healthy controls receiving no treatment were assessed over the same interval to determine the reliability of the measures and act as a control group at baseline. Muscle activation onsets and reactive range of motion, range of velocities and accelerations were assessed for between group differences at baseline and pre- to post-treatment effects within patients with LBP using t-tests. Correlations between these dependent variables and the change of clinical outcomes (pain, disability) over treatment were also explored. Kinematic-based measures showed comparable reliability to EMG-based measures. Between-group differences were found in lumbar lateral flexion ROM at baseline (patients < controls). In the patients with LBP, lateral flexion velocity and acceleration significantly increased following the LSEP. Correlational analyses revealed that lumbar angular kinematics were more sensitive to changes in pain intensity following the LSEP compared to EMG measures. These findings are interpreted in from the perspective of guarding behaviors and lumbar stability hypotheses. Future clinical trials are needed to target patients with and without delayed APAs at baseline and to explore the sensitivity of different outcome measures related to APAs. Different tasks more challenging to postural stability may need to be explored to more effectively reveal APA dysfunction. Copyright © 2018. Published by Elsevier Ltd.

Quantitative comparison and evaluation of software packages for assessment of abdominal adipose tissue distribution by magnetic resonance imaging.

PubMed

Bonekamp, S; Ghosh, P; Crawford, S; Solga, S F; Horska, A; Brancati, F L; Diehl, A M; Smith, S; Clark, J M

2008-01-01

To examine five available software packages for the assessment of abdominal adipose tissue with magnetic resonance imaging, compare their features and assess the reliability of measurement results. Feature evaluation and test-retest reliability of softwares (NIHImage, SliceOmatic, Analyze, HippoFat and EasyVision) used in manual, semi-automated or automated segmentation of abdominal adipose tissue. A random sample of 15 obese adults with type 2 diabetes. Axial T1-weighted spin echo images centered at vertebral bodies of L2-L3 were acquired at 1.5 T. Five software packages were evaluated (NIHImage, SliceOmatic, Analyze, HippoFat and EasyVision), comparing manual, semi-automated and automated segmentation approaches. Images were segmented into cross-sectional area (CSA), and the areas of visceral (VAT) and subcutaneous adipose tissue (SAT). Ease of learning and use and the design of the graphical user interface (GUI) were rated. Intra-observer accuracy and agreement between the software packages were calculated using intra-class correlation. Intra-class correlation coefficient was used to obtain test-retest reliability. Three of the five evaluated programs offered a semi-automated technique to segment the images based on histogram values or a user-defined threshold. One software package allowed manual delineation only. One fully automated program demonstrated the drawbacks of uncritical automated processing. The semi-automated approaches reduced variability and measurement error, and improved reproducibility. There was no significant difference in the intra-observer agreement in SAT and CSA. The VAT measurements showed significantly lower test-retest reliability. There were some differences between the software packages in qualitative aspects, such as user friendliness. Four out of five packages provided essentially the same results with respect to the inter- and intra-rater reproducibility. Our results using SliceOmatic, Analyze or NIHImage were comparable and could be used interchangeably. Newly developed fully automated approaches should be compared to one of the examined software packages.

Quantitative comparison and evaluation of software packages for assessment of abdominal adipose tissue distribution by magnetic resonance imaging

PubMed Central

Bonekamp, S; Ghosh, P; Crawford, S; Solga, SF; Horska, A; Brancati, FL; Diehl, AM; Smith, S; Clark, JM

2009-01-01

Objective To examine five available software packages for the assessment of abdominal adipose tissue with magnetic resonance imaging, compare their features and assess the reliability of measurement results. Design Feature evaluation and test–retest reliability of softwares (NIHImage, SliceOmatic, Analyze, HippoFat and EasyVision) used in manual, semi-automated or automated segmentation of abdominal adipose tissue. Subjects A random sample of 15 obese adults with type 2 diabetes. Measurements Axial T1-weighted spin echo images centered at vertebral bodies of L2–L3 were acquired at 1.5 T. Five software packages were evaluated (NIHImage, SliceOmatic, Analyze, HippoFat and EasyVision), comparing manual, semi-automated and automated segmentation approaches. Images were segmented into cross-sectional area (CSA), and the areas of visceral (VAT) and subcutaneous adipose tissue (SAT). Ease of learning and use and the design of the graphical user interface (GUI) were rated. Intra-observer accuracy and agreement between the software packages were calculated using intra-class correlation. Intra-class correlation coefficient was used to obtain test–retest reliability. Results Three of the five evaluated programs offered a semi-automated technique to segment the images based on histogram values or a user-defined threshold. One software package allowed manual delineation only. One fully automated program demonstrated the drawbacks of uncritical automated processing. The semi-automated approaches reduced variability and measurement error, and improved reproducibility. There was no significant difference in the intra-observer agreement in SAT and CSA. The VAT measurements showed significantly lower test–retest reliability. There were some differences between the software packages in qualitative aspects, such as user friendliness. Conclusion Four out of five packages provided essentially the same results with respect to the inter- and intra-rater reproducibility. Our results using SliceOmatic, Analyze or NIHImage were comparable and could be used interchangeably. Newly developed fully automated approaches should be compared to one of the examined software packages. PMID:17700582

Putting the pediatrics milestones into practice: a consensus roadmap and resource analysis.

PubMed

Schumacher, Daniel J; Spector, Nancy D; Calaman, Sharon; West, Daniel C; Cruz, Mario; Frohna, John G; Gonzalez Del Rey, Javier; Gustafson, Kristina K; Poynter, Sue Ellen; Rosenbluth, Glenn; Southgate, W Michael; Vinci, Robert J; Sectish, Theodore C

2014-05-01

The Accreditation Council for Graduate Medical Education has partnered with member boards of the American Board of Medical Specialties to initiate the next steps in advancing competency-based assessment in residency programs. This initiative, known as the Milestone Project, is a paradigm shift from traditional assessment efforts and requires all pediatrics residency programs to report individual resident progression along a series of 4 to 5 developmental levels of performance, or milestones, for individual competencies every 6 months beginning in June 2014. The effort required to successfully make this shift is tremendous given the number of training programs, training institutions, and trainees. However, it holds great promise for achieving training outcomes that align with patient needs; developing a valid, reliable, and meaningful way to track residents' development; and providing trainees with a roadmap for learning. Recognizing the resources needed to implement this new system, the authors, all residency program leaders, provide their consensus view of the components necessary for implementing and sustaining this effort, including resource estimates for completing this work. The authors have identified 4 domains: (1) Program Review and Development of Stakeholders and Participants, (2) Assessment Methods and Validation, (3) Data and Assessment System Development, and (4) Summative Assessment and Feedback. This work can serve as a starting point and framework for collaboration with program, department, and institutional leaders to identify and garner necessary resources and plan for local and national efforts that will ensure successful transition to milestones-based assessment. Copyright © 2014 by the American Academy of Pediatrics.

The need for harmonisation and innovation of neuropsychological assessment in neurodegenerative dementias in Europe: consensus document of the Joint Program for Neurodegenerative Diseases Working Group.

PubMed

Costa, Alberto; Bak, Thomas; Caffarra, Paolo; Caltagirone, Carlo; Ceccaldi, Mathieu; Collette, Fabienne; Crutch, Sebastian; Della Sala, Sergio; Démonet, Jean François; Dubois, Bruno; Duzel, Emrah; Nestor, Peter; Papageorgiou, Sokratis G; Salmon, Eric; Sikkes, Sietske; Tiraboschi, Pietro; van der Flier, Wiesje M; Visser, Pieter Jelle; Cappa, Stefano F

2017-04-17

Cognitive, behavioural, and functional assessment is crucial in longitudinal studies of neurodegenerative dementias (NDD). Central issues, such as the definition of the study population (asymptomatic, at risk, or individuals with dementia), the detection of change/decline, and the assessment of relevant outcomes depend on quantitative measures of cognitive, behavioural, and functional status.Currently, we are far from having available reliable protocols and tools for the assessment of dementias in Europe. The main problems are the heterogeneity of the tools used across different European countries, the lack of standardisation of administration and scoring methods across centres, and the limited information available about the psychometric properties of many tests currently in widespread use. This situation makes it hard to compare results across studies carried out in different centres, thus hampering research progress, in particular towards the contribution to a "big data" common data set.We present here the results of a project funded by the Joint Program for Neurodegenerative Diseases (JPND) and by the Italian Ministry of Health. The project aimed at providing a consensus framework for the harmonisation of assessment tools to be applied to research in neurodegenerative disorders affecting cognition across Europe. A panel of European experts reviewed the current methods of neuropsychological assessment, identified pending issues, and made recommendations for the harmonisation of neuropsychological assessment of neurodegenerative dementias in Europe.A consensus was achieved on the general recommendations to be followed in developing procedures and tools for neuropsychological assessment, with the aim of harmonising tools and procedures to achieve more reliable data on the cognitive-behavioural examination. The results of this study should be considered as a first step to enhancing a common view and practise on NDD assessment across European countries.

Ecologically relevant outcome measure for post-inpatient rehabilitation.

PubMed

Marquez de la Plata, Carlos; Qualls, Devin; Plenger, Patrick; Malec, James F; Hayden, Mary Ellen

2017-01-01

Transfer of skills learned within the clinic environment to patients' home or community is important in post-inpatient brain injury rehabilitation (PBIR). Outcome measures used in PBIR assess level of independence during functional tasks; however, available functional instruments do not quantitate the environment in which the behaviors occur. To examine the reliability and validity of an instrument used to assess patients' functional abilities while quantifying the amount of structure and distractions in the environment. 2501 patients who sustained a traumatic brain injury (TBI) or cerebrovascular accident (CVA) and participated in a multidisciplinary PBIR program between 2006 and 2014 were identified retrospectively for this study. The PERPOS and MPAI-4 were used to assess functional abilities at admission and at discharge. Construct validity was assessed using a bivariate Spearman rho analysis A subsample of 56 consecutive admissions during 2014 were examined to determine inter-rater reliability. Intra-class correlation coefficient (ICC) and Kappa coefficients assessed inter-rater agreement of the total PERPOS and PERPOS subscales respectively. The PERPOS and MPAI-4 demonstrated a strong negative association among both TBI and CVA patients. Kappa scores for the three PERPOS scales each demonstrated good to excellent inter-rater agreement. The ICC for overall PERPOS scores fell in the good agreement range. The PERPOS can be used reliably in PBIR to quantify patients' functional abilities within the context of environmental demands.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

PubMed

Blum, Richard H; Muret-Wagstaff, Sharon L; Boulet, John R; Cooper, Jeffrey B; Petrusa, Emil R; Baker, Keith H; Davidyuk, Galina; Dearden, Jennifer L; Feinstein, David M; Jones, Stephanie B; Kimball, William R; Mitchell, John D; Nadelberg, Robert L; Wiser, Sarah H; Albrecht, Meredith A; Anastasi, Amanda K; Bose, Ruma R; Chang, Laura Y; Culley, Deborah J; Fisher, Lauren J; Grover, Meera; Klainer, Suzanne B; Kveraga, Rikante; Martel, Jeffrey P; McKenna, Shannon S; Minehart, Rebecca D; Mitchell, John D; Mountjoy, Jeremi R; Pawlowski, John B; Pilon, Robert N; Shook, Douglas C; Silver, David A; Warfield, Carol A; Zaleski, Katherine L

2018-04-01

Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Probabilistic Risk Assessment (PRA): A Practical and Cost Effective Approach

NASA Technical Reports Server (NTRS)

Lee, Lydia L.; Ingegneri, Antonino J.; Djam, Melody

2006-01-01

The Lunar Reconnaissance Orbiter (LRO) is the first mission of the Robotic Lunar Exploration Program (RLEP), a space exploration venture to the Moon, Mars and beyond. The LRO mission includes spacecraft developed by NASA Goddard Space Flight Center (GSFC) and seven instruments built by GSFC, Russia, and contractors across the nation. LRO is defined as a measurement mission, not a science mission. It emphasizes the overall objectives of obtaining data to facilitate returning mankind safely to the Moon in preparation for an eventual manned mission to Mars. As the first mission in response to the President's commitment of the journey of exploring the solar system and beyond: returning to the Moon in the next decade, then venturing further into the solar system, ultimately sending humans to Mars and beyond, LRO has high-visibility to the public but limited resources and a tight schedule. This paper demonstrates how NASA's Lunar Reconnaissance Orbiter Mission project office incorporated reliability analyses in assessing risks and performing design tradeoffs to ensure mission success. Risk assessment is performed using NASA Procedural Requirements (NPR) 8705.5 - Probabilistic Risk Assessment (PRA) Procedures for NASA Programs and Projects to formulate probabilistic risk assessment (PRA). As required, a limited scope PRA is being performed for the LRO project. The PRA is used to optimize the mission design within mandated budget, manpower, and schedule constraints. The technique that LRO project office uses to perform PRA relies on the application of a component failure database to quantify the potential mission success risks. To ensure mission success in an efficient manner, low cost and tight schedule, the traditional reliability analyses, such as reliability predictions, Failure Modes and Effects Analysis (FMEA), and Fault Tree Analysis (FTA), are used to perform PRA for the large system of LRO with more than 14,000 piece parts and over 120 purchased or contractor built components.

Space Shuttle Main Engine - The Relentless Pursuit of Improvement

NASA Technical Reports Server (NTRS)

VanHooser, Katherine P.; Bradley, Douglas P.

2011-01-01

The Space Shuttle Main Engine (SSME) is the only reusable large liquid rocket engine ever developed. The specific impulse delivered by the staged combustion cycle, substantially higher than previous rocket engines, minimized volume and weight for the integrated vehicle. The dual pre-burner configuration permitted precise mixture ratio and thrust control while the fully redundant controller and avionics provided a very high degree of system reliability and health diagnosis. The main engine controller design was the first rocket engine application to incorporate digital processing. The engine was required to operate at a high chamber pressure to minimize engine volume and weight. Power level throttling was required to minimize structural loads on the vehicle early in flight and acceleration levels on the crew late in ascent. Fatigue capability, strength, ease of assembly and disassembly, inspectability, and materials compatibility were all major considerations in achieving a fully reusable design. During the multi-decade program the design evolved substantially using a series of block upgrades. A number of materials and manufacturing challenges were encountered throughout SSME s history. Significant development was required for the final configuration of the high pressure turbopumps. Fracture control was implemented to assess life limits of critical materials and components. Survival in the hydrogen environment required assessment of hydrogen embrittlement. Instrumentation systems were a challenge due to the harsh thermal and dynamic environments within the engine. Extensive inspection procedures were developed to assess the engine components between flights. The Space Shuttle Main Engine achieved a remarkable flight performance record. All flights were successful with only one mission requiring an ascent abort condition, which still resulted in an acceptable orbit and mission. This was achieved in large part via extensive ground testing to fully characterize performance and to establish acceptable life limits. During the program over a million seconds of accumulated test and flight time was achieved. Post flight inspection and assessment was a key part of assuring proper performance of the flight hardware. By the end of the program the predicted reliability had improved by a factor of four. These unique challenges, evolution of the design, and the resulting reliability will be discussed in this paper.

External quality assessment programs in the context of ISO 15189 accreditation.

PubMed

Sciacovelli, Laura; Secchiero, Sandra; Padoan, Andrea; Plebani, Mario

2018-05-23

Effective management of clinical laboratories participating in external quality assessment schemes (EQAS) is of fundamental importance in ensuring reliable analytical results. The International Standard ISO 15189:2012 requires participation in interlaboratory comparison [e.g. external quality assessment (EQA)] for all tests provided by an individual laboratory. If EQAS is not commercially available, alternative approaches should be identified, although clinical laboratories may find it challenging to choose the EQAS that comply with the international standards and approved guidelines. Great competence is therefore required, as well as knowledge of the characteristics and key elements affecting the reliability of an EQAS, and the analytical quality specifications stated in approved documents. Another skill of fundamental importance is the ability to identify an alternative approach when the available EQAS are inadequate or missing. Yet the choice of the right EQA program alone does not guarantee its effectiveness. In fact, the fundamental steps of analysis of the information provided in EQA reports and the ability to identify improvement actions to be undertaken call for the involvement of all laboratory staff playing a role in the specific activity. The aim of this paper was to describe the critical aspects that EQA providers and laboratory professionals should control in order to guarantee effective EQAS management and compliance with ISO 15189 accreditation requirements.

Measurement of stable changes of self-management skills after rehabilitation: a latent state-trait analysis of the Health Education Impact Questionnaire (heiQ™).

PubMed

Schuler, M; Musekamp, G; Bengel, J; Schwarze, M; Spanier, K; Gutenbrunner, Chr; Ehlebracht-König, I; Nolte, S; Osborne, R H; Faller, H

2014-11-01

To assess stable effects of self-management programs, measurement instruments should primarily capture the attributes of interest, for example, the self-management skills of the measured persons. However, measurements of psychological constructs are always influenced by both aspects of the situation (states) and aspects of the person (traits). This study tests whether the Health Education Impact Questionnaire (heiQ™), an instrument assessing a wide range of proximal outcomes of self-management programs, is primarily influenced by person factors instead of situational factors. Furthermore, measurement invariance over time, changes in traits and predictors of change for each heiQ™ scale were examined. Subjects were N = 580 patients with rheumatism, asthma, orthopedic conditions or inflammatory bowel disease, who filled out the heiQ™ at the beginning, the end of and 3 months after a disease-specific inpatient rehabilitation program in Germany. Structural equation modeling techniques were used to estimate latent trait-change models and test for measurement invariance in each heiQ™ scale. Coefficients of consistency, occasion specificity and reliability were computed. All scales showed scalar invariance over time. Reliability coefficients were high (0.80-0.94), and consistency coefficients (0.49-0.79) were always substantially higher than occasion specificity coefficients (0.14-0.38), indicating that the heiQ™ scales primarily capture person factors. Trait-changes with small to medium effect sizes were shown in five scales and were affected by sex, age and diagnostic group. The heiQ™ can be used to assess stable effects in important outcomes of self-management programs over time, e.g., changes in self-management skills or emotional well-being.

Reliability of an interactive computer program for advance care planning.

PubMed

Schubart, Jane R; Levi, Benjamin H; Camacho, Fabian; Whitehead, Megan; Farace, Elana; Green, Michael J

2012-06-01

Despite widespread efforts to promote advance directives (ADs), completion rates remain low. Making Your Wishes Known: Planning Your Medical Future (MYWK) is an interactive computer program that guides individuals through the process of advance care planning, explaining health conditions and interventions that commonly involve life or death decisions, helps them articulate their values/goals, and translates users' preferences into a detailed AD document. The purpose of this study was to demonstrate that (in the absence of major life changes) the AD generated by MYWK reliably reflects an individual's values/preferences. English speakers ≥30 years old completed MYWK twice, 4 to 6 weeks apart. Reliability indices were assessed for three AD components: General Wishes; Specific Wishes for treatment; and Quality-of-Life values (QoL). Twenty-four participants completed the study. Both the Specific Wishes and QoL scales had high internal consistency in both time periods (Knuder Richardson formula 20 [KR-20]=0.83-0.95, and 0.86-0.89). Test-retest reliability was perfect for General Wishes (κ=1), high for QoL (Pearson's correlation coefficient=0.83), but lower for Specific Wishes (Pearson's correlation coefficient=0.57). MYWK generates an AD where General Wishes and QoL (but not Specific Wishes) statements remain consistent over time.

Reliability of an Interactive Computer Program for Advance Care Planning

PubMed Central

Levi, Benjamin H.; Camacho, Fabian; Whitehead, Megan; Farace, Elana; Green, Michael J

2012-01-01

Abstract Despite widespread efforts to promote advance directives (ADs), completion rates remain low. Making Your Wishes Known: Planning Your Medical Future (MYWK) is an interactive computer program that guides individuals through the process of advance care planning, explaining health conditions and interventions that commonly involve life or death decisions, helps them articulate their values/goals, and translates users' preferences into a detailed AD document. The purpose of this study was to demonstrate that (in the absence of major life changes) the AD generated by MYWK reliably reflects an individual's values/preferences. English speakers ≥30 years old completed MYWK twice, 4 to 6 weeks apart. Reliability indices were assessed for three AD components: General Wishes; Specific Wishes for treatment; and Quality-of-Life values (QoL). Twenty-four participants completed the study. Both the Specific Wishes and QoL scales had high internal consistency in both time periods (Knuder Richardson formula 20 [KR-20]=0.83–0.95, and 0.86–0.89). Test-retest reliability was perfect for General Wishes (κ=1), high for QoL (Pearson's correlation coefficient=0.83), but lower for Specific Wishes (Pearson's correlation coefficient=0.57). MYWK generates an AD where General Wishes and QoL (but not Specific Wishes) statements remain consistent over time. PMID:22512830

76 FR 17159 - Office of New Reactors; Final Interim Staff Guidance on Standard Review Plan, Section 17.4...

Federal Register 2010, 2011, 2012, 2013, 2014

2011-03-28

... Interim Staff Guidance on Standard Review Plan, Section 17.4, ``Reliability Assurance Program'' AGENCY... design reliability assurance program (RAP). This ISG updates the guidance provided to the staff in Standard Review Plan (SRP), Section 17.4, ``Reliability Assurance Program,'' of NUREG-0800, ``Standard...

The psychometric properties of the cervical nonorganic signs in patients with neck pain: an assessment of pain expression.

PubMed

Lue, Yi-Jing; Chang, Jyh-Jong; Wu, Yuh-Yih; Lin, Rong-Fong; Lu, Yen-Mou

2018-04-01

Neck pain is a common cause of disability. This study investigated the psychometric properties of the cervical nonorganic signs (CNOS), a tool for assessing abnormal illness behaviors in patients with neck pain. The CNOS was administered on patients with neck pain. Reliability and validity analyses were used to evaluate the psychometric properties. Exploratory factor analysis was used to investigate the dimensionality. Correlations with the Short Form-36 were used to investigate the convergent validity. The results supported the reliability (inter-rater reliability intra-class correlation: 0.920), validity (correlated with body pain (|ρ|=0.31) and vitality (|ρ| =0.30), and two-factor dimensionality (χ 2 =   5.904, p= 0.66; χ 2 /df = 0.738; RMSEA< 0.001; CFI = 1.000; TLI = 1.024; SRMR = 0.047) of the scale. The two factors were pain (severe pain) and vitality (poor vitality) expressed by the patients. The CNOS is a reliable and valid instrument for assessing pain and vitality problems. It helps patients to express severe pain and lack of vitality. The rehabilitation discipline could use the scale to understand pain expression and to design proper rehabilitation programs. Implications for Rehabilitation The cervical nonorganic signs has two domains (pain and vitality). The scale is reliable and valid for patients with neck pain. Patients with high scores on the pain domain have severe body pain that may interfere with normal social activities. Clinicians should understand their suffering and try to help them to alleviate the pain.

Mass and Reliability System (MaRS)

NASA Technical Reports Server (NTRS)

Barnes, Sarah

2016-01-01

The Safety and Mission Assurance (S&MA) Directorate is responsible for mitigating risk, providing system safety, and lowering risk for space programs from ground to space. The S&MA is divided into 4 divisions: The Space Exploration Division (NC), the International Space Station Division (NE), the Safety & Test Operations Division (NS), and the Quality and Flight Equipment Division (NT). The interns, myself and Arun Aruljothi, will be working with the Risk & Reliability Analysis Branch under the NC Division's. The mission of this division is to identify, characterize, diminish, and communicate risk by implementing an efficient and effective assurance model. The team utilizes Reliability and Maintainability (R&M) and Probabilistic Risk Assessment (PRA) to ensure decisions concerning risks are informed, vehicles are safe and reliable, and program/project requirements are realistic and realized. This project pertains to the Orion mission, so it is geared toward a long duration Human Space Flight Program(s). For space missions, payload is a critical concept; balancing what hardware can be replaced by components verse by Orbital Replacement Units (ORU) or subassemblies is key. For this effort a database was created that combines mass and reliability data, called Mass and Reliability System or MaRS. The U.S. International Space Station (ISS) components are used as reference parts in the MaRS database. Using ISS components as a platform is beneficial because of the historical context and the environment similarities to a space flight mission. MaRS uses a combination of systems: International Space Station PART for failure data, Vehicle Master Database (VMDB) for ORU & components, Maintenance & Analysis Data Set (MADS) for operation hours and other pertinent data, & Hardware History Retrieval System (HHRS) for unit weights. MaRS is populated using a Visual Basic Application. Once populated, the excel spreadsheet is comprised of information on ISS components including: operation hours, random/nonrandom failures, software/hardware failures, quantity, orbital replaceable units (ORU), date of placement, unit weight, frequency of part, etc. The motivation for creating such a database will be the development of a mass/reliability parametric model to estimate mass required for replacement parts. Once complete, engineers working on future space flight missions will have access a mean time to failures and on parts along with their mass, this will be used to make proper decisions for long duration space flight missions

Measuring stakeholder participation in evaluation: an empirical validation of the Participatory Evaluation Measurement Instrument (PEMI).

PubMed

Daigneault, Pierre-Marc; Jacob, Steve; Tremblay, Joël

2012-08-01

Stakeholder participation is an important trend in the field of program evaluation. Although a few measurement instruments have been proposed, they either have not been empirically validated or do not cover the full content of the concept. This study consists of a first empirical validation of a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument (PEMI). It specifically examines (1) the intercoder reliability of scores derived by two research assistants on published evaluation cases; (2) the convergence between the scores of coders and those of key respondents (i.e., authors); and (3) the convergence between the authors' scores on the PEMI and the Evaluation Involvement Scale (EIS). A purposive sample of 40 cases drawn from the evaluation literature was used to assess reliability. One author per case in this sample was then invited to participate in a survey; 25 fully usable questionnaires were received. Stakeholder participation was measured on nominal and ordinal scales. Cohen's κ, the intraclass correlation coefficient, and Spearman's ρ were used to assess reliability and convergence. Reliability results ranged from fair to excellent. Convergence between coders' and authors' scores ranged from poor to good. Scores derived from the PEMI and the EIS were moderately associated. Evidence from this study is strong in the case of intercoder reliability and ranges from weak to strong in the case of convergent validation. Globally, this suggests that the PEMI can produce scores that are both reliable and valid.

North American Science Symposium: Toward a unified framework for inventorying and monitoring forest ecosystem resources

Treesearch

Celedonio Aguirre-Bravo; Carlos Rodriguez Franco

1999-01-01

The general objective of this Symposium was to build on the best science and technology available to assure that the data and information produced in future inventory and monitoring programs are comparable, quality assured, available, and adequate for their intended purposes, thereby providing a reliable framework for characterization, assessment, and management of...

Comparisons of Observed Process Quality in German and American Infant/Toddler Programs

ERIC Educational Resources Information Center

Tietze, Wolfgang; Cryer, Debby

2004-01-01

Observed process quality in infant/toddler classrooms was compared in Germany (n = 75) and the USA (n = 219). Process quality was assessed with the Infant/Toddler Environment Rating Scale(ITERS) and parent attitudes about ITERS content with the ITERS Parent Questionnaire (ITERSPQ). The ITERS had comparable reliabilities in the two countries and…

Estimated Student Score Gain on the ACT COMP Exam: Valid Tool for Institutional Assessment?

ERIC Educational Resources Information Center

Banta, Trudy W.; And Others

1987-01-01

An institution can test seniors with the ACT College Outcome Measures Project (COMP) exam, then subtract from the senior score an estimated freshman score. Studies at the University of Tennessee, Knoxville, indicate that this method is not reliable to make judgments about the quality of general education programs. (Author/MLW)

What the Research Tells Us about the Impact of Induction and Mentoring Programs for Beginning Teachers

ERIC Educational Resources Information Center

Ingersoll, Richard; Strong, Michael

2012-01-01

This chapter summarizes a comprehensive and critical review that the authors recently completed of empirical studies that evaluate the effects of induction on various outcomes. The review's objective was to provide researchers, policy makers, and educators with a reliable and current assessment of what is known and not known about the…

Development and validity of a scale to measure workplace culture of health.

PubMed

Kwon, Youngbum; Marzec, Mary L; Edington, Dee W

2015-05-01

To describe the development of and test the validity and reliability of the Workplace Culture of Health (COH) scale. Exploratory factor analysis and confirmatory factor analysis were performed on data from a health care organization (N = 627). To verify the factor structure, confirmatory factor analysis was performed on a second data set from a medical equipment manufacturer (N = 226). The COH scale included a structure of five orthogonal factors: senior leadership and polices, programs and rewards, quality assurance, supervisor support, and coworker support. With regard to construct validity (convergent and discriminant) and reliability, two different US companies showed the same factorial structure, satisfactory fit statistics, and suitable internal and external consistency. The COH scale represents a reliable and valid scale to assess the workplace environment and culture for supporting health.

Examples of Nonconservatism in the CARE 3 Program

NASA Technical Reports Server (NTRS)

Dotson, Kelly J.

1988-01-01

This paper presents parameter regions in the CARE 3 (Computer-Aided Reliability Estimation version 3) computer program where the program overestimates the reliability of a modeled system without warning the user. Five simple models of fault-tolerant computer systems are analyzed; and, the parameter regions where reliability is overestimated are given. The source of the error in the reliability estimates for models which incorporate transient fault occurrences was not readily apparent. However, the source of much of the error for models with permanent and intermittent faults can be attributed to the choice of values for the run-time parameters of the program.

Older adults' drug benefit beliefs: construct definition and measure development.

PubMed

Cline, Richard R; Gupta, Kiran; Singh, Reshmi L

2008-03-01

The Medicare Prescription Drug, Improvement and Modernization Act of 2003 provides coverage of outpatient prescription drugs for Medicare beneficiaries. Although much has been learned since the program's implementation, a context within which this information can be understood is lacking. The purpose of this study was to develop a reliable and valid multi-item instrument measuring beliefs about Medicare prescription drug benefits. Survey items were generated using focus group transcripts, other surveys on the Medicare Part "D" program, and past studies of choice and satisfaction in drug insurance programs. Using data from the survey pilot test, item and reliability analyses were used to reduce and refine an initial pool of items. Data then were collected from a cross-sectional, mail survey of older adults living in Minnesota. Data were analyzed using exploratory factor analysis. Summated rating scales then were constructed and assessed further using reliability analyses. Construct validity of summated scales was examined by comparing scale scores across response categories of survey items that collected information on general political attitudes, perceptions of the Medicare Part "D" program, health status, and health care utilization and demographics. The adjusted response rate for the main survey was 55.98% (744/1329). Iterative factor analysis produced 2 interpretable scales. The first, termed "access/equity" (13 items, Cronbach's alpha=0.89) measures beliefs that a Medicare drug benefit should both provide affordable prescription drugs for beneficiaries and do this in a manner that is equitable for all participants. The second, termed "comprehensibility" (6 items, Cronbach's alpha=0.80) assesses beliefs that regulations governing a Medicare drug benefit should be easily understood. Discriminant validity tests suggest that these measures behave in a manner consistent with related research in these areas. Measures of 2 facets of older adults' drug benefit beliefs were developed using a multiple step procedure. Future research could focus on developing a better understanding of other facets of these beliefs and sound methods of measurement.

Quantitative analysis of the rubric as an assessment tool: an empirical study of student peer-group rating

NASA Astrophysics Data System (ADS)

Hafner, John C.; Hafner, Patti M.

2003-12-01

Although the rubric has emerged as one of the most popular assessment tools in progressive educational programs, there is an unfortunate dearth of information in the literature quantifying the actual effectiveness of the rubric as an assessment tool in the hands of the students. This study focuses on the validity and reliability of the rubric as an assessment tool for student peer-group evaluation in an effort to further explore the use and effectiveness of the rubric. A total of 1577 peer-group ratings using a rubric for an oral presentation was used in this 3-year study involving 107 college biology students. A quantitative analysis of the rubric used in this study shows that it is used consistently by both students and the instructor across the study years. Moreover, the rubric appears to be 'gender neutral' and the students' academic strength has no significant bearing on the way that they employ the rubric. A significant, one-to-one relationship (slope = 1.0) between the instructor's assessment and the students' rating is seen across all years using the rubric. A generalizability study yields estimates of inter-rater reliability of moderate values across all years and allows for the estimation of variance components. Taken together, these data indicate that the general form and evaluative criteria of the rubric are clear and that the rubric is a useful assessment tool for peer-group (and self-) assessment by students. To our knowledge, these data provide the first statistical documentation of the validity and reliability of the rubric for student peer-group assessment.

Reliability and Validity of Digital Imagery Methodology for Measuring Starting Portions and Plate Waste from School Salad Bars.

PubMed

Bean, Melanie K; Raynor, Hollie A; Thornton, Laura M; Sova, Alexandra; Dunne Stewart, Mary; Mazzeo, Suzanne E

2018-04-12

Scientifically sound methods for investigating dietary consumption patterns from self-serve salad bars are needed to inform school policies and programs. To examine the reliability and validity of digital imagery for determining starting portions and plate waste of self-serve salad bar vegetables (which have variable starting portions) compared with manual weights. In a laboratory setting, 30 mock salads with 73 vegetables were made, and consumption was simulated. Each component (initial and removed portion) was weighed; photographs of weighed reference portions and pre- and post-consumption mock salads were taken. Seven trained independent raters visually assessed images to estimate starting portions to the nearest ¼ cup and percentage consumed in 20% increments. These values were converted to grams for comparison with weighed values. Intraclass correlations between weighed and digital imagery-assessed portions and plate waste were used to assess interrater reliability and validity. Pearson's correlations between weights and digital imagery assessments were also examined. Paired samples t tests were used to evaluate mean differences (in grams) between digital imagery-assessed portions and measured weights. Interrater reliabilities were excellent for starting portions and plate waste with digital imagery. For accuracy, intraclass correlations were moderate, with lower accuracy for determining starting portions of leafy greens compared with other vegetables. However, accuracy of digital imagery-assessed plate waste was excellent. Digital imagery assessments were not significantly different from measured weights for estimating overall vegetable starting portions or waste; however, digital imagery assessments slightly underestimated starting portions (by 3.5 g) and waste (by 2.1 g) of leafy greens. This investigation provides preliminary support for use of digital imagery in estimating starting portions and plate waste from school salad bars. Results might inform methods used in empirical investigations of dietary intake in schools with self-serve salad bars. Copyright © 2018 Academy of Nutrition and Dietetics. Published by Elsevier Inc. All rights reserved.

Software For Computing Reliability Of Other Software

NASA Technical Reports Server (NTRS)

Nikora, Allen; Antczak, Thomas M.; Lyu, Michael

1995-01-01

Computer Aided Software Reliability Estimation (CASRE) computer program developed for use in measuring reliability of other software. Easier for non-specialists in reliability to use than many other currently available programs developed for same purpose. CASRE incorporates mathematical modeling capabilities of public-domain Statistical Modeling and Estimation of Reliability Functions for Software (SMERFS) computer program and runs in Windows software environment. Provides menu-driven command interface; enabling and disabling of menu options guides user through (1) selection of set of failure data, (2) execution of mathematical model, and (3) analysis of results from model. Written in C language.

Performance of a quality assurance program for assessing dental health in methamphetamine users.

PubMed

Dye, Bruce A; Harrell, Lauren; Murphy, Debra A; Belin, Thomas; Shetty, Vivek

2015-07-05

Systematic characterization of the dental consequences of methamphetamine (MA) abuse presupposes a rigorous quality assurance (QA) program to ensure the credibility of the data collected and the scientific integrity and validity of the clinical study. In this report we describe and evaluate the performance of a quality assurance program implemented in a large cross-sectional study of the dental consequences of MA use. A large community sample of MA users was recruited over a 30 month period during 2011-13 and received comprehensive oral examinations and psychosocial assessments by site examiners based at two large community health centers in Los Angeles. National Health and Nutrition Examination Survey (NHANES) protocols for oral health assessments were utilized to characterize dental disease. Using NHANES oral health quality assurance guidelines, examiner reliability statistics such as Cohen's Kappa coefficients and inter-class correlation coefficients were calculated to assess the magnitude of agreement between the site examiners and a reference examiner to ensure conformance and comparability with NHANES practices. Approximately 9% (n = 49) of the enrolled 574 MA users received a repeat dental caries and periodontal examination conducted by the reference examiner. There was high concordance between the reference examiner and the site examiners for identification of untreated dental disease (Kappa statistic values: 0.57-0.75, percent agreement 83-88%). For identification of untreated caries on at least 5 surfaces of anterior teeth, the Kappas ranged from 0.77 to 0.87, and percent agreement from 94 to 97%. The intra-class coefficients (ICCs) ranged from 0.87 to 89 for attachment loss across all periodontal sites assessed and the ICCs ranged from 0.79 to 0.81 for pocket depth. For overall gingival recession, the ICCs ranged from 0.88 to 0.91. When Kappa was calculated based on the CDC/AAP case definitions for severe periodontitis, inter-examiner reliability for site examiners was low (Kappa 0.27-0.67). Overall, the quality assurance program confirmed the procedural adherence of the quality of the data collected on the distribution of dental caries and periodontal disease in MA-users. Examiner concordance was higher for dental caries but lower for specific periodontal assessments.

Longitudinal Improvement in Balance Error Scoring System Scores among NCAA Division-I Football Athletes.

PubMed

Mathiasen, Ross; Hogrefe, Christopher; Harland, Kari; Peterson, Andrew; Smoot, M Kyle

2018-02-15

The Balance Error Scoring System (BESS) is a commonly used concussion assessment tool. Recent studies have questioned the stability and reliability of baseline BESS scores. The purpose of this longitudinal prospective cohort study is to examine differences in yearly baseline BESS scores in athletes participating on an NCAA Division-I football team. NCAA Division-I freshman football athletes were videotaped performing the BESS test at matriculation and after 1 year of participation in the football program. Twenty-three athletes were enrolled in year 1 of the study, and 25 athletes were enrolled in year 2. Those athletes enrolled in year 1 were again videotaped after year 2 of the study. The paired t-test was used to assess for change in score over time for the firm surface, foam surface, and the cumulative BESS score. Additionally, inter- and intrarater reliability values were calculated. Cumulative errors on the BESS significantly decreased from a mean of 20.3 at baseline to 16.8 after 1 year of participation. The mean number of errors following the second year of participation was 15.0. Inter-rater reliability for the cumulative score ranged from 0.65 to 0.75. Intrarater reliability was 0.81. After 1 year of participation, there is a statistically and clinically significant improvement in BESS scores in an NCAA Division-I football program. Although additional improvement in BESS scores was noted after a second year of participation, it did not reach statistical significance. Football athletes should undergo baseline BESS testing at least yearly if the BESS is to be optimally useful as a diagnostic test for concussion.

Design for Reliability and Safety Approach for the NASA New Launch Vehicle

NASA Technical Reports Server (NTRS)

Safie, Fayssal, M.; Weldon, Danny M.

2007-01-01

The United States National Aeronautics and Space Administration (NASA) is in the midst of a space exploration program intended for sending crew and cargo to the international Space Station (ISS), to the moon, and beyond. This program is called Constellation. As part of the Constellation program, NASA is developing new launch vehicles aimed at significantly increase safety and reliability, reduce the cost of accessing space, and provide a growth path for manned space exploration. Achieving these goals requires a rigorous process that addresses reliability, safety, and cost upfront and throughout all the phases of the life cycle of the program. This paper discusses the "Design for Reliability and Safety" approach for the NASA new crew launch vehicle called ARES I. The ARES I is being developed by NASA Marshall Space Flight Center (MSFC) in support of the Constellation program. The ARES I consists of three major Elements: A solid First Stage (FS), an Upper Stage (US), and liquid Upper Stage Engine (USE). Stacked on top of the ARES I is the Crew exploration vehicle (CEV). The CEV consists of a Launch Abort System (LAS), Crew Module (CM), Service Module (SM), and a Spacecraft Adapter (SA). The CEV development is being led by NASA Johnson Space Center (JSC). Designing for high reliability and safety require a good integrated working environment and a sound technical design approach. The "Design for Reliability and Safety" approach addressed in this paper discusses both the environment and the technical process put in place to support the ARES I design. To address the integrated working environment, the ARES I project office has established a risk based design group called "Operability Design and Analysis" (OD&A) group. This group is an integrated group intended to bring together the engineering, design, and safety organizations together to optimize the system design for safety, reliability, and cost. On the technical side, the ARES I project has, through the OD&A environment, implemented a probabilistic approach to analyze and evaluate design uncertainties and understand their impact on safety, reliability, and cost. This paper focuses on the use of the various probabilistic approaches that have been pursued by the ARES I project. Specifically, the paper discusses an integrated functional probabilistic analysis approach that addresses upffont some key areas to support the ARES I Design Analysis Cycle (DAC) pre Preliminary Design (PD) Phase. This functional approach is a probabilistic physics based approach that combines failure probabilities with system dynamics and engineering failure impact models to identify key system risk drivers and potential system design requirements. The paper also discusses other probabilistic risk assessment approaches planned by the ARES I project to support the PD phase and beyond.

Reliability and Validity of the Pediatric Palliative Care Questionnaire for Measuring Self-Efficacy, Knowledge, and Adequacy of Prior Medical Education among Pediatric Fellows

PubMed Central

Cohen, Harvey J.; Popat, Rita A.; Halamek, Louis P.

2015-01-01

Abstract Background: Interventions to improve pediatric trainee education in palliative care have been limited by a lack of reliable and valid tools for measuring effectiveness. Objective: We developed a questionnaire to measure pediatric fellows' self-efficacy (comfort), knowledge, and perceived adequacy of prior medical education. We measured the questionnaire's reliability and validity. Methods: The questionnaire contains questions regarding self-efficacy (23), knowledge (10), fellow's perceived adequacy of prior medical education (6), and demographics. The survey was developed with palliative care experts, and sent to fellows in U.S. pediatric cardiology, critical care, hematology/ oncology, and neonatal-perinatal medicine programs. Measures of reliability, internal consistency, and validity were calculated. Results: One hundred forty-seven fellows completed the survey at test and retest. The self-efficacy and medical education questionnaires showed high internal consistency of 0.95 and 0.84. The test-retest reliability for the Self-Efficacy Summary Score, measured by intraclass correlation coefficient (ICC) and weighted kappa, was 0.78 (item range 0.44–0.81) and 0.61 (item range 0.36–0.70), respectively. For the Adequacy of Medical Education Summary Score, ICC was 0.85 (item range 0.6–0.78) and weighted kappa was 0.63 (item range 0.47–0.62). Validity coefficients for these two questionnaires were 0.88 and 0.92. Fellows answered a mean of 8.8/10 knowledge questions correctly; percentage agreement ranged from 65% to 99%. Conclusions: This questionnaire is capable of assessing self-efficacy and fellow-perceived adequacy of their prior palliative care training. We recommend use of this tool for fellowship programs seeking to evaluate fellow education in palliative care, or for research studies assessing the effectiveness of a palliative care educational intervention. PMID:26185912

Investigating the reliability and validity of the Dutch versions of the illness management and recovery scales among clients with mental disorders.

PubMed

Goossens, Peter J J; Beentjes, Titus A A; Knol, Suzanne; Salyers, Michelle P; de Vries, Sjoerd J

2017-12-01

The Illness Management and Recovery scales (IMRS) can measure the progress of clients' illness self-management and recovery. Previous studies have examined the psychometric properties of the IMRS. This study examined the reliability and validity of the Dutch version of the IMRS. Clients (n = 111) and clinicians (n = 40) completed the client and clinician versions of the IMRS, respectively. The scales were administered again 2 weeks later to assess stability over time. Validity was assessed with the Utrecht Coping List (UCL), Dutch Empowerment Scale (DES), and Brief Symptom Inventory (BSI). The client and clinician versions of the IMRS had moderate internal reliability, with α = 0.69 and 0.71, respectively. The scales showed strong test-retest reliability, r = 0.79, for the client version and r = 0.86 for the clinician version. Correlations between client and clinician versions ranged from r = 0.37 to 0.69 for the total and subscales. We also found relationships in expected directions between the client IMRS and UCL, DES and BSI, which supports validity of the Dutch version of the IMRS. The Dutch version of the IMRS demonstrated good reliability and validity. The IMRS could be useful for Dutch-speaking programs interested in evaluating client progress on illness self-management and recovery.

Development and Initial Reliability Testing of NAK-50+: A Nutrition Attitude and Knowledge Questionnaire for Adults 50+ Years of Age.

PubMed

Ducak, Kate; Keller, Heather

2016-03-01

Few questionnaires to test nutrition knowledge and attitudes of older adults living independently in the community have been developed and tested to assess self-management tools such as Nutri-eSCREEN and other education programs. This study is a first step in the development of a questionnaire designed to evaluate the nutrition knowledge and attitudes of independent older adults (NAK-50+). The steps involved in this study were: (i) drafting initial questions based on the content of the Nutri-eSCREEN education material, (ii) using cognitive interviewing to determine if these questions were understandable and relevant (n = 9 adults ≥50 years of age), and (iii) completing test-retest reliability in a convenient community sample (n = 60 adults ≥50 years of age). Intra-class coefficients (ICC) and kappa were used to determine reliability. A 33-item questionnaire resulted from this development and analysis. ICC for the total score was 0.68 indicating good agreement and thus initial reliability. NAK-50+ is a face valid and reliable questionnaire that assesses nutrition knowledge and attitudes in independent adults aged ≥50 years. Further work to determine construct validity and to refine the questionnaire is warranted. Availability of the questionnaire for this age group will support rigorous evaluation of education and self-management interventions for this segment of the population.

Organizational readiness for implementing change: a psychometric assessment of a new measure.

PubMed

Shea, Christopher M; Jacobs, Sara R; Esserman, Denise A; Bruce, Kerry; Weiner, Bryan J

2014-01-10

Organizational readiness for change in healthcare settings is an important factor in successful implementation of new policies, programs, and practices. However, research on the topic is hindered by the absence of a brief, reliable, and valid measure. Until such a measure is developed, we cannot advance scientific knowledge about readiness or provide evidence-based guidance to organizational leaders about how to increase readiness. This article presents results of a psychometric assessment of a new measure called Organizational Readiness for Implementing Change (ORIC), which we developed based on Weiner's theory of organizational readiness for change. We conducted four studies to assess the psychometric properties of ORIC. In study one, we assessed the content adequacy of the new measure using quantitative methods. In study two, we examined the measure's factor structure and reliability in a laboratory simulation. In study three, we assessed the reliability and validity of an organization-level measure of readiness based on aggregated individual-level data from study two. In study four, we conducted a small field study utilizing the same analytic methods as in study three. Content adequacy assessment indicated that the items developed to measure change commitment and change efficacy reflected the theoretical content of these two facets of organizational readiness and distinguished the facets from hypothesized determinants of readiness. Exploratory and confirmatory factor analysis in the lab and field studies revealed two correlated factors, as expected, with good model fit and high item loadings. Reliability analysis in the lab and field studies showed high inter-item consistency for the resulting individual-level scales for change commitment and change efficacy. Inter-rater reliability and inter-rater agreement statistics supported the aggregation of individual level readiness perceptions to the organizational level of analysis. This article provides evidence in support of the ORIC measure. We believe this measure will enable testing of theories about determinants and consequences of organizational readiness and, ultimately, assist healthcare leaders to reduce the number of health organization change efforts that do not achieve desired benefits. Although ORIC shows promise, further assessment is needed to test for convergent, discriminant, and predictive validity.

Organizational readiness for implementing change: a psychometric assessment of a new measure

PubMed Central

2014-01-01

Background Organizational readiness for change in healthcare settings is an important factor in successful implementation of new policies, programs, and practices. However, research on the topic is hindered by the absence of a brief, reliable, and valid measure. Until such a measure is developed, we cannot advance scientific knowledge about readiness or provide evidence-based guidance to organizational leaders about how to increase readiness. This article presents results of a psychometric assessment of a new measure called Organizational Readiness for Implementing Change (ORIC), which we developed based on Weiner’s theory of organizational readiness for change. Methods We conducted four studies to assess the psychometric properties of ORIC. In study one, we assessed the content adequacy of the new measure using quantitative methods. In study two, we examined the measure’s factor structure and reliability in a laboratory simulation. In study three, we assessed the reliability and validity of an organization-level measure of readiness based on aggregated individual-level data from study two. In study four, we conducted a small field study utilizing the same analytic methods as in study three. Results Content adequacy assessment indicated that the items developed to measure change commitment and change efficacy reflected the theoretical content of these two facets of organizational readiness and distinguished the facets from hypothesized determinants of readiness. Exploratory and confirmatory factor analysis in the lab and field studies revealed two correlated factors, as expected, with good model fit and high item loadings. Reliability analysis in the lab and field studies showed high inter-item consistency for the resulting individual-level scales for change commitment and change efficacy. Inter-rater reliability and inter-rater agreement statistics supported the aggregation of individual level readiness perceptions to the organizational level of analysis. Conclusions This article provides evidence in support of the ORIC measure. We believe this measure will enable testing of theories about determinants and consequences of organizational readiness and, ultimately, assist healthcare leaders to reduce the number of health organization change efforts that do not achieve desired benefits. Although ORIC shows promise, further assessment is needed to test for convergent, discriminant, and predictive validity. PMID:24410955

Center to Advance Palliative Care palliative care clinical care and customer satisfaction metrics consensus recommendations.

PubMed

Weissman, David E; Morrison, R Sean; Meier, Diane E

2010-02-01

Data collection and analysis are vital for strategic planning, quality improvement, and demonstration of palliative care program impact to hospital administrators, private funders and policymakers. Since 2000, the Center to Advance Palliative Care (CAPC) has provided technical assistance to hospitals, health systems and hospices working to start, sustain, and grow nonhospice palliative care programs. CAPC convened a consensus panel in 2008 to develop recommendations for specific clinical and customer metrics that programs should track. The panel agreed on four key domains of clinical metrics and two domains of customer metrics. Clinical metrics include: daily assessment of physical/psychological/spiritual symptoms by a symptom assessment tool; establishment of patient-centered goals of care; support to patient/family caregivers; and management of transitions across care sites. For customer metrics, consensus was reached on two domains that should be tracked to assess satisfaction: patient/family satisfaction, and referring clinician satisfaction. In an effort to ensure access to reliably high-quality palliative care data throughout the nation, hospital palliative care programs are encouraged to collect and report outcomes for each of the metric domains described here.

Simplified Asset Indices to Measure Wealth and Equity in Health Programs: A Reliability and Validity Analysis Using Survey Data From 16 Countries

PubMed Central

Chakraborty, Nirali M; Fry, Kenzo; Behl, Rasika; Longfield, Kim

2016-01-01

ABSTRACT Background: Social franchising programs in low- and middle-income countries have tried using the standard wealth index, based on the Demographic and Health Survey (DHS) questionnaire, in client exit interviews to assess clients’ relative wealth compared with the national wealth distribution to ensure equity in service delivery. The large number of survey questions required to capture the wealth index variables have proved cumbersome for programs. Methods: Using an adaptation of the Delphi method, we developed shortened wealth indices and in February 2015 consulted 15 stakeholders in equity measurement. Together, we selected the best of 5 alternative indices, accompanied by 2 measures of agreement (percent agreement and Cohen’s kappa statistic) comparing wealth quintile assignment in the new indices to the full DHS index. The panel agreed that reducing the number of assets was more important than standardization across countries because a short index would provide strong indication of client wealth and be easier to collect and use in the field. Additionally, the panel agreed that the simplified index should be highly correlated with the DHS for each country (kappa ≥ 0.75) for both national and urban-specific samples. We then revised indices for 16 countries and selected the minimum number of questions and question options required to achieve a kappa statistic ≥ 0.75 for both national and urban populations. Findings: After combining the 5 wealth quintiles into 3 groups, which the expert panel deemed more programmatically meaningful, reliability between the standard DHS wealth index and each of 3 simplified indices was high (median kappa = 0.81, 086, and 0.77, respectively, for index B that included only the common questions from the DHS VI questionnaire, index D that included the common questions plus country-specific questions, and index E that found the shortest list of common and country-specific questions that met the minimum reliability criteria of kappa ≥ 0.75). Index E was the simplified index of choice because it was reliable in national and urban contexts while requiring the fewest number of survey questions—6 to 18 per country compared with 25 to 47 in the original DHS wealth index (a 66% average reduction). Conclusion: Social franchise clinics and other types of service delivery programs that want to assess client wealth in relation to a national or urban population can do so with high reliability using a short questionnaire. Future uses of the simplified asset questionnaire include a mobile application for rapid data collection and analysis. PMID:27016550

Simplified Asset Indices to Measure Wealth and Equity in Health Programs: A Reliability and Validity Analysis Using Survey Data From 16 Countries.

PubMed

Chakraborty, Nirali M; Fry, Kenzo; Behl, Rasika; Longfield, Kim

2016-03-01

Social franchising programs in low- and middle-income countries have tried using the standard wealth index, based on the Demographic and Health Survey (DHS) questionnaire, in client exit interviews to assess clients' relative wealth compared with the national wealth distribution to ensure equity in service delivery. The large number of survey questions required to capture the wealth index variables have proved cumbersome for programs. Using an adaptation of the Delphi method, we developed shortened wealth indices and in February 2015 consulted 15 stakeholders in equity measurement. Together, we selected the best of 5 alternative indices, accompanied by 2 measures of agreement (percent agreement and Cohen's kappa statistic) comparing wealth quintile assignment in the new indices to the full DHS index. The panel agreed that reducing the number of assets was more important than standardization across countries because a short index would provide strong indication of client wealth and be easier to collect and use in the field. Additionally, the panel agreed that the simplified index should be highly correlated with the DHS for each country (kappa ≥ 0.75) for both national and urban-specific samples. We then revised indices for 16 countries and selected the minimum number of questions and question options required to achieve a kappa statistic ≥ 0.75 for both national and urban populations. After combining the 5 wealth quintiles into 3 groups, which the expert panel deemed more programmatically meaningful, reliability between the standard DHS wealth index and each of 3 simplified indices was high (median kappa = 0.81, 086, and 0.77, respectively, for index B that included only the common questions from the DHS VI questionnaire, index D that included the common questions plus country-specific questions, and index E that found the shortest list of common and country-specific questions that met the minimum reliability criteria of kappa ≥ 0.75). Index E was the simplified index of choice because it was reliable in national and urban contexts while requiring the fewest number of survey questions-6 to 18 per country compared with 25 to 47 in the original DHS wealth index (a 66% average reduction). Social franchise clinics and other types of service delivery programs that want to assess client wealth in relation to a national or urban population can do so with high reliability using a short questionnaire. Future uses of the simplified asset questionnaire include a mobile application for rapid data collection and analysis. © Chakraborty et al.

Psychometric Properties of the Deep Muscle Contraction Scale for Assessment of the Drawing-in Maneuver in Patients With Chronic Nonspecific Low Back Pain.

PubMed

Oliveira, Crystian B; Negrão Filho, Ruben F; Franco, Márcia R; Morelhão, Priscila K; Araujo, Amanda C; Pinto, Rafael Z

2017-06-01

Study Design A prospective cohort study. Background Motor control dysfunctions have been commonly reported in patients with chronic nonspecific low back pain (LBP). Physical therapists need clinical tools with adequate psychometric properties to assess such patients in clinical practice. The deep muscle contraction (DMC) scale is a clinical rating scale for assessing patients' ability to voluntarily contract deep abdominal muscles. Objectives To investigate the intrarater reliability, floor and ceiling effects, internal and external responsiveness, and correlation analysis (with ultrasound measures) of the DMC scale in patients with chronic nonspecific LBP undergoing a lumbar stabilization exercise program. Methods Sixty-two patients with chronic nonspecific LBP were included. At baseline, self-report questionnaires were administered to patients and a trained assessor evaluated abdominal muscle recruitment with the DMC scale and ultrasound imaging. Four ratios of the change in abdominal muscle thickness between the resting and contracted states were calculated through the ultrasound measures. After 1 week, the same ultrasound measures and DMC scale were collected again for the reliability analysis. The proportions of patients with the lowest and highest scores on the DMC scale were calculated to investigate floor and ceiling effects. All patients underwent a lumbar stabilization program, administered twice a week for 8 weeks. After the treatment period, all measures were collected again, with the addition of the global perceived effect scale, to assess the internal and external responsiveness of the measures. Correlation coefficients between ultrasound ratios and DMC scale total and subscale scores were also calculated. Results The intrarater reliability of the DMC scale and the 4 ratios of abdominal muscle thickness varied from moderate to excellent. The DMC scale showed no floor or ceiling effects. Results for internal responsiveness of the DMC scale showed large effect sizes (2.26; 84% confidence interval [CI]: 2.06, 2.45), whereas the external responsiveness was below the proposed threshold (area under the curve = 0.54; 95% CI: 0.39, 0.68). Fair and significant correlations between some ultrasound ratios and DMC subscales were found. Conclusion The DMC scale was demonstrated to be a reliable tool, with no ceiling and floor effects, and to detect change in the ability to contract the deep abdominal muscles after a lumbar stabilization exercise program, but with low accuracy for estimating patient-perceived clinical outcome. J Orthop Sports Phys Ther 2017;47(6):432-441. doi:10.2519/jospt.2017.7140.

Digital avionics systems - Overview of FAA/NASA/industry-wide briefing

NASA Technical Reports Server (NTRS)

Larsen, William E.; Carro, Anthony

1986-01-01

The effects of incorporating digital technology into the design of aircraft on the airworthiness criteria and certification procedures for aircraft are investigated. FAA research programs aimed at providing data for the functional assessment of aircraft which use digital systems for avionics and flight control functions are discussed. The need to establish testing, assurance assessment, and configuration management technologies to insure the reliability of digital systems is discussed; consideration is given to design verification, system performance/robustness, and validation technology.

[Environmental Hazards Assessment Program annual report, June 1992--June 1993]. Use of diatom distributions to monitor environmental health

DOE Office of Scientific and Technical Information (OSTI.GOV)

Levine, R.H.

1993-12-01

A variety of approaches has been used in the past to assess the environmental impact of anthropogenic contaminants. One reliable index for aquatic environments is the analysis of diatom species distribution; the focus in this case being on the Savannah River. The completed objectives of this study were: (A) the development and use of procedures for measuring diatom distribution in the water column and (B) the development and evaluation of sediment sampling methods for retrospective analysis.

SEASAT economic assessment. Volume 10: The SATIL 2 program (a program for the evaluation of the costs of an operational SEASAT system as a function of operational requirements and reliability. [computer programs for economic analysis and systems analysis of SEASAT satellite systems

NASA Technical Reports Server (NTRS)

1975-01-01

The SATIL 2 computer program was developed to assist with the programmatic evaluation of alternative approaches to establishing and maintaining a specified mix of operational sensors on spacecraft in an operational SEASAT system. The program computes the probability distributions of events (i.e., number of launch attempts, number of spacecraft purchased, etc.), annual recurring cost, and present value of recurring cost. This is accomplished for the specific task of placing a desired mix of sensors in orbit in an optimal fashion in order to satisfy a specified sensor demand function. Flow charts are shown, and printouts of the programs are given.

Validity, Reliability and Acceptability of the Team Standardized Assessment of Clinical Encounter Report*

PubMed Central

Wong, Camilla L.; Norris, Mireille; Sinha, Samir S.; Zorzitto, Maria L.; Madala, Sushma; Hamid, Jemila S.

2016-01-01

Background The Team Standardized Assessment of a Clinical Encounter Report (StACER) was designed for use in Geriatric Medicine residency programs to evaluate Communicator and Collaborator competencies. Methods The Team StACER was completed by two geriatricians and interdisciplinary team members based on observations during a geriatric medicine team meeting. Postgraduate trainees were recruited from July 2010–November 2013. Inter-rater reliability between two geriatricians and between all team members was determined. Internal consistency of items for the constructs Communicator and Collaborator competencies was calculated. Raters completed a survey previously administered to Canadian geriatricians to assess face validity. Trainees completed a survey to determine the usefulness of this instrument as a feedback tool. Results Thirty postgraduate trainees participated. The prevalence-adjusted bias-adjusted kappa range inter-rater reliability for Communicator and Collaborator items were 0.87–1.00 and 0.86–1.00, respectively. The Cronbach’s alpha coefficient for Communicator and Collaborator items was 0.997 (95% CI: 0.993–1.00) and 0.997 (95% CI: 0.997–1.00), respectively. The instrument lacked discriminatory power, as all trainees scored “meets requirements” in the overall assessment. Niney-three per cent and 86% of trainees found feedback useful for developing Communicator and Collaborator competencies, respectively. Conclusions The Team StACER has adequate inter-rater reliability and internal consistency. Poor discriminatory power and face validity challenge the merit of using this evaluation tool. Trainees felt the tool provided useful feedback on Collaborator and Communicator competencies. PMID:28050222

Provider Attitudes toward Pay-for-Performance Programs: Development and Validation of a Measurement Instrument

PubMed Central

Meterko, Mark; Young, Gary J; White, Bert; Bokhour, Barbara G; Burgess, James F; Berlowitz, Dan; Guldin, Matthew R; Nealon Seibert, Marjorie

2006-01-01

Objective To develop an instrument for assessing physician attitudes toward quality incentive programs, and to assess its reliability and validity. Data Sources Study involved primary data collection. A 40-item paper and pencil survey of primary care physicians in Rochester, New York, and Massachusetts was conducted between May 2004 and December 2004. Seven-hundred and ninety-eight completed questionnaires were received, representing a response rate of 32 percent (798/2,497). Study Design Based on an extensive review of the literature and discussions with experts in the field, we developed a conceptual framework representing the features of pay-for-performance (P4P) programs hypothesized to affect physician behavior in that context. A draft questionnaire was developed based on that conceptual model and pilot tested in three groups of physicians. The questionnaire was modified based on the physician feedback, and the revised version was distributed to 2,497 primary care physicians affiliated with two of the seven sites participating in Rewarding Results, a national evaluation of quality target and financial incentive programs. Data Collection Respondents were randomly divided into a derivation and a validation sample. Exploratory factor analysis was applied to the responses of the derivation sample. Those results were used to create scales in the validation sample, and these were then subjected to multitrait analysis (MTA). One scale representing physicians' perception of the impact of P4P on their clinical practice was regressed on the other scales as a test of construct validity. Principal Findings Seven constructs were identified and demonstrated substantial convergent and discriminant validity in the MTA: awareness and understanding, clinical relevance, cooperation, unintended consequences, control, financial salience, and impact. Internal consistency reliabilities (Cronbach's α coefficients) ranged from 0.50 to 0.80. A statistically significant 25 percent of the variation in perceived impact was accounted for by physician perceptions of the other six characteristics of P4P programs. Conclusions It is possible to identify and measure the key salient features of P4P programs using a valid and reliable 26-item survey. This instrument may now be used in further studies to better understand the impact of P4P programs on physician behavior. PMID:16987311

Groundwater studies: principal aquifer surveys

USGS Publications Warehouse

Burow, Karen R.; Belitz, Kenneth

2014-01-01

In 1991, the U.S. Congress established the National Water-Quality Assessment (NAWQA) program within the U.S. Geological Survey (USGS) to develop nationally consistent long-term datasets and provide information about the quality of the Nation’s streams and groundwater. The USGS uses objective and reliable data, water-quality models, and systematic scientific studies to assess current water-quality conditions, to identify changes in water quality over time, and to determine how natural factors and human activities affect the quality of streams and groundwater. NAWQA is the only non-regulatory Federal program to perform these types of studies; participation is voluntary. In the third decade (Cycle 3) of the NAWQA program (2013–2023), the USGS will evaluate the quality and availability of groundwater for drinking supply, improve our understanding of where and why water quality is degraded, and assess how groundwater quality could respond to changes in climate and land use. These goals will be addressed through the implementation of a new monitoring component in Cycle 3: Principal Aquifer Surveys.

The Americleft Speech Project: A Training and Reliability Study.

PubMed

Chapman, Kathy L; Baylis, Adriane; Trost-Cardamone, Judith; Cordero, Kelly Nett; Dixon, Angela; Dobbelsteyn, Cindy; Thurmes, Anna; Wilson, Kristina; Harding-Bell, Anne; Sweeney, Triona; Stoddard, Gregory; Sell, Debbie

2016-01-01

To describe the results of two reliability studies and to assess the effect of training on interrater reliability scores. The first study (1) examined interrater and intrarater reliability scores (weighted and unweighted kappas) and (2) compared interrater reliability scores before and after training on the use of the Cleft Audit Protocol for Speech-Augmented (CAPS-A) with British English-speaking children. The second study examined interrater and intrarater reliability on a modified version of the CAPS-A (CAPS-A Americleft Modification) with American and Canadian English-speaking children. Finally, comparisons were made between the interrater and intrarater reliability scores obtained for Study 1 and Study 2. The participants were speech-language pathologists from the Americleft Speech Project. In Study 1, interrater reliability scores improved for 6 of the 13 parameters following training on the CAPS-A protocol. Comparison of the reliability results for the two studies indicated lower scores for Study 2 compared with Study 1. However, this appeared to be an artifact of the kappa statistic that occurred due to insufficient variability in the reliability samples for Study 2. When percent agreement scores were also calculated, the ratings appeared similar across Study 1 and Study 2. The findings of this study suggested that improvements in interrater reliability could be obtained following a program of systematic training. However, improvements were not uniform across all parameters. Acceptable levels of reliability were achieved for those parameters most important for evaluation of velopharyngeal function.

The Americleft Speech Project: A Training and Reliability Study

PubMed Central

Chapman, Kathy L.; Baylis, Adriane; Trost-Cardamone, Judith; Cordero, Kelly Nett; Dixon, Angela; Dobbelsteyn, Cindy; Thurmes, Anna; Wilson, Kristina; Harding-Bell, Anne; Sweeney, Triona; Stoddard, Gregory; Sell, Debbie

2017-01-01

Objective To describe the results of two reliability studies and to assess the effect of training on interrater reliability scores. Design The first study (1) examined interrater and intrarater reliability scores (weighted and unweighted kappas) and (2) compared interrater reliability scores before and after training on the use of the Cleft Audit Protocol for Speech–Augmented (CAPS-A) with British English-speaking children. The second study examined interrater and intrarater reliability on a modified version of the CAPS-A (CAPS-A Americleft Modification) with American and Canadian English-speaking children. Finally, comparisons were made between the interrater and intrarater reliability scores obtained for Study 1 and Study 2. Participants The participants were speech-language pathologists from the Americleft Speech Project. Results In Study 1, interrater reliability scores improved for 6 of the 13 parameters following training on the CAPS-A protocol. Comparison of the reliability results for the two studies indicated lower scores for Study 2 compared with Study 1. However, this appeared to be an artifact of the kappa statistic that occurred due to insufficient variability in the reliability samples for Study 2. When percent agreement scores were also calculated, the ratings appeared similar across Study 1 and Study 2. Conclusion The findings of this study suggested that improvements in interrater reliability could be obtained following a program of systematic training. However, improvements were not uniform across all parameters. Acceptable levels of reliability were achieved for those parameters most important for evaluation of velopharyngeal function. PMID:25531738

Can local staff reliably assess their own programs? A confirmatory test-retest study of Lot Quality Assurance Sampling data collectors in Uganda.

PubMed

Beckworth, Colin A; Anguyo, Robert; Kyakulaga, Francis Cranmer; Lwanga, Stephen K; Valadez, Joseph J

2016-08-17

Data collection techniques that routinely provide health system information at the local level are in demand and needed. LQAS is intended for use by local health teams to collect data at the district and sub-district levels. Our question is whether local health staff produce biased results as they are responsible for implementing the programs they also assess. This test-retest study replicates on a larger scale an earlier LQAS reliability assessment in Uganda. We conducted in two districts an LQAS survey using 15 local health staff as data collectors. A week later, the data collectors swapped districts, where they acted as disinterested non-local data collectors, repeating the LQAS survey with the same respondents. We analysed the resulting two data sets for agreement using Cohens' Kappa. The average Kappa score for the knowledge indicators was k = 0.43 (SD = 0.16) and for practice indicators k = 0.63 (SD = 0.17). These scores show moderate agreement for knowledge indicators and substantial agreement for practice indicators. Analyses confirm that respondents were more knowledgeable on retest; no evidence of bias was found for practice indicators. The findings of this study are remarkably similar to those produced in the first reliability study. There is no evidence that using local healthcare staff to collect LQAS data biases data collection in an LQAS study. The bias observed in the knowledge indicators was most likely due to a 'practice effect', whereby respondents increased their knowledge as a result of completing the first survey; no corresponding effect was seen in the practice indicators.

Reliability analysis of laminated CMC components through shell subelement techniques

NASA Technical Reports Server (NTRS)

Starlinger, Alois; Duffy, Stephen F.; Gyekenyesi, John P.

1992-01-01

An updated version of the integrated design program Composite Ceramics Analysis and Reliability Evaluation of Structures (C/CARES) was developed for the reliability evaluation of ceramic matrix composites (CMC) laminated shell components. The algorithm is now split into two modules: a finite-element data interface program and a reliability evaluation algorithm. More flexibility is achieved, allowing for easy implementation with various finite-element programs. The interface program creates a neutral data base which is then read by the reliability module. This neutral data base concept allows easy data transfer between different computer systems. The new interface program from the finite-element code Matrix Automated Reduction and Coupling (MARC) also includes the option of using hybrid laminates (a combination of plies of different materials or different layups) and allows for variations in temperature fields throughout the component. In the current version of C/CARES, a subelement technique was implemented, enabling stress gradients within an element to be taken into account. The noninteractive reliability function is now evaluated at each Gaussian integration point instead of using averaging techniques. As a result of the increased number of stress evaluation points, considerable improvements in the accuracy of reliability analyses were realized.

The Cost of Saving Electricity Through Energy Efficiency Programs Funded by Utility Customers: 2009–2015

DOE Office of Scientific and Technical Information (OSTI.GOV)

Hoffman, Ian M.; Goldman, Charles A.; Murphy, Sean

The average cost to utilities to save a kilowatt-hour (kWh) in the United States is 2.5 cents, according to the most comprehensive assessment to date of the cost performance of energy efficiency programs funded by electricity customers. These costs are similar to those documented earlier. Cost-effective efficiency programs help ensure electricity system reliability at the most affordable cost as part of utility planning and implementation activities for resource adequacy. Building on prior studies, Berkeley Lab analyzed the cost performance of 8,790 electricity efficiency programs between 2009 and 2015 for 116 investor-owned utilities and other program administrators in 41 states. Themore » Berkeley Lab database includes programs representing about three-quarters of total spending on electricity efficiency programs in the United States.« less

Development and practical implications of the Exercise Resourcefulness Inventory.

PubMed

Fast, Hilary V; Kennett, Deborah J

2015-05-01

To determine the validity and reliability of the Exercise Resourcefulness Inventory (ERI) designed to assess the self-regulatory strategies used to promote regular exercise. In Study 1, the inventory's relationship with other established scales in the exercise behavior change field was examined. In Study 2, the test-retest reliability and predictive validity of the ERI was established by having participants from Study 1 complete the inventory a second time. Internal consistency, and convergent, discriminant, and concurrent validity were supported in both studies. The test-retest correlation of the ERI was .80. As well, participants scoring higher on the ERI in Study 1 were more likely to be at a higher stage of change in Study 2, and greater increases in exercise resourcefulness over time were predictive of advancement to higher stages of change. ERI is a reliable and valid measure to assess the self-regulatory strategies used to promote regular exercise. Facilitators may want to tailor exercise programs for individuals scoring lower in resourcefulness to prevent them from relapsing. Copyright © 2015 Elsevier Ireland Ltd. All rights reserved.

Progressive brain atrophy in patients with chronic neuropsychiatric symptoms after mild traumatic brain injury: a preliminary study.

PubMed

Ross, David E; Ochs, Alfred L; Seabaugh, Jan M; Demark, Michael F; Shrader, Carole R; Marwitz, Jennifer H; Havranek, Michael D

2012-01-01

NeuroQuant® is a recently developed, FDA-approved software program for measuring brain MRI volume in clinical settings. The aims of this study were as follows: (1) to examine the test-retest reliability of NeuroQuant®; (2) to test the hypothesis that patients with mild traumatic brain injury (TBI) would have abnormally rapid progressive brain atrophy; and (3) to test the hypothesis that progressive brain atrophy in patients with mild TBI would be associated with vocational outcome. Sixteen patients with mild TBI were compared to 20 normal controls. Vocational outcome was assessed with the Glasgow Outcome Scale-Extended (GOSE) and Disability Rating Scale (DRS). NeuroQuant® showed high test-re-test reliability. Patients had abnormally rapid progressive atrophy in several brain regions and the rate of atrophy was associated with inability to return to work. NeuroQuant®, is a reliable and valid method for assessing the anatomic effects of TBI. Progression of atrophy may continue for years after injury, even in patients with mild TBI.

Reliability of an Automated High-Resolution Manometry Analysis Program across Expert Users, Novice Users, and Speech-Language Pathologists

ERIC Educational Resources Information Center

Jones, Corinne A.; Hoffman, Matthew R.; Geng, Zhixian; Abdelhalim, Suzan M.; Jiang, Jack J.; McCulloch, Timothy M.

2014-01-01

Purpose: The purpose of this study was to investigate inter- and intrarater reliability among expert users, novice users, and speech-language pathologists with a semiautomated high-resolution manometry analysis program. We hypothesized that all users would have high intrarater reliability and high interrater reliability. Method: Three expert…

Reusable Solid Rocket Motor - Accomplishment, Lessons, and a Culture of Success

NASA Technical Reports Server (NTRS)

Moore, D. R.; Phelps, W. J.

2011-01-01

The Reusable Solid Rocket Motor (RSRM) represents the largest solid rocket motor (SRM) ever flown and the only human-rated solid motor. High reliability of the RSRM has been the result of challenges addressed and lessons learned. Advancements have resulted by applying attention to process control, testing, and postflight through timely and thorough communication in dealing with all issues. A structured and disciplined approach was taken to identify and disposition all concerns. Careful consideration and application of alternate opinions was embraced. Focus was placed on process control, ground test programs, and postflight assessment. Process control is mandatory for an SRM, because an acceptance test of the delivered product is not feasible. The RSRM maintained both full-scale and subscale test articles, which enabled continuous improvement of design and evaluation of process control and material behavior. Additionally RSRM reliability was achieved through attention to detail in post flight assessment to observe any shift in performance. The postflight analysis and inspections provided invaluable reliability data as it enables observation of actual flight performance, most of which would not be available if the motors were not recovered. RSRM reusability offered unique opportunities to learn about the hardware. NASA is moving forward with the Space Launch System that incorporates propulsion systems that takes advantage of the heritage Shuttle and Ares solid motor programs. These unique challenges, features of the RSRM, materials and manufacturing issues, and design improvements will be discussed in the paper.

Reusable Solid Rocket Motor - Accomplishments, Lessons, and a Culture of Success

NASA Technical Reports Server (NTRS)

Moore, Dennis R.; Phelps, Willie J.

2011-01-01

The Reusable Solid Rocket Motor represents the largest solid rocket motor ever flown and the only human rated solid motor. Each Reusable Solid Rocket Motor (RSRM) provides approximately 3-million lb of thrust to lift the integrated Space Shuttle vehicle from the launch pad. The motors burn out approximately 2 minutes later, separate from the vehicle and are recovered and refurbished. The size of the motor and the need for high reliability were challenges. Thrust shaping, via shaping of the propellant grain, was needed to limit structural loads during ascent. The motor design evolved through several block upgrades to increase performance and to increase safety and reliability. A major redesign occurred after STS-51L with the Redesigned Solid Rocket Motor. Significant improvements in the joint sealing systems were added. Design improvements continued throughout the Program via block changes with a number of innovations including development of low temperature o-ring materials and incorporation of a unique carbon fiber rope thermal barrier material. Recovery of the motors and post flight inspection improved understanding of hardware performance, and led to key design improvements. Because of the multidecade program duration material obsolescence was addressed, and requalification of materials and vendors was sometimes needed. Thermal protection systems and ablatives were used to protect the motor cases and nozzle structures. Significant understanding of design and manufacturing features of the ablatives was developed during the program resulting in optimization of design features and processing parameters. The project advanced technology in eliminating ozone-depleting materials in manufacturing processes and the development of an asbestos-free case insulation. Manufacturing processes for the large motor components were unique and safety in the manufacturing environment was a special concern. Transportation and handling approaches were also needed for the large hardware segments. The reusable solid rocket motor achieved significant reliability via process control, ground test programs, and postflight assessment. Process control is mandatory for a solid rocket motor as an acceptance test of the delivered product is not feasible. Process control included process failure modes and effects analysis, statistical process control, witness panels, and process product integrity audits. Material controls and inspections were maintained throughout the sub tier vendors. Material fingerprinting was employed to assess any drift in delivered material properties. The RSRM maintained both full scale and sub-scale test articles. These enabled continuous improvement of design and evaluation of process control and material behavior. Additionally RSRM reliability was achieved through attention to detail in post flight assessment to observe any shift in performance. The postflight analysis and inspections provided invaluable reliability data as it enables observation of actual flight performance, most of which would not be available if the motors were not recovered. These unique challenges, features of the reusable solid rocket motor, materials and manufacturing issues, and design improvements will be discussed in the paper.

Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review.

PubMed

Zendejas, Benjamin; Ruparel, Raaj K; Cook, David A

2016-02-01

The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment. We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences). From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert-novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability. Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.

Assembly-line Simulation Program

NASA Technical Reports Server (NTRS)

Chamberlain, Robert G.; Zendejas, Silvino; Malhotra, Shan

1987-01-01

Costs and profits estimated for models based on user inputs. Standard Assembly-line Manufacturing Industry Simulation (SAMIS) program generalized so useful for production-line manufacturing companies. Provides accurate and reliable means of comparing alternative manufacturing processes. Used to assess impact of changes in financial parameters as cost of resources and services, inflation rates, interest rates, tax policies, and required rate of return of equity. Most important capability is ability to estimate prices manufacturer would have to receive for its products to recover all of costs of production and make specified profit. Written in TURBO PASCAL.

U.S. Geological Survey Mineral Resources Program—Mineral resource science supporting informed decisionmaking

USGS Publications Warehouse

Wilkins, Aleeza M.; Doebrich, Jeff L.

2016-09-19

The USGS Mineral Resources Program (MRP) delivers unbiased science and information to increase understanding of mineral resource potential, production, and consumption, and how mineral resources interact with the environment. The MRP is the Federal Government’s sole source for this mineral resource science and information. Program goals are to (1) increase understanding of mineral resource formation, (2) provide mineral resource inventories and assessments, (3) broaden knowledge of the effects of mineral resources on the environment and society, and (4) provide analysis on the availability and reliability of mineral supplies.

[Process design in high-reliability organizations].

PubMed

Sommer, K-J; Kranz, J; Steffens, J

2014-05-01

Modern medicine is a highly complex service industry in which individual care providers are linked in a complicated network. The complexity and interlinkedness is associated with risks concerning patient safety. Other highly complex industries like commercial aviation have succeeded in maintaining or even increasing its safety levels despite rapidly increasing passenger figures. Standard operating procedures (SOPs), crew resource management (CRM), as well as operational risk evaluation (ORE) are historically developed and trusted parts of a comprehensive and systemic safety program. If medicine wants to follow this quantum leap towards increased patient safety, it must intensively evaluate the results of other high-reliability industries and seek step-by-step implementation after a critical assessment.

Effectiveness of different approaches to disseminating traveler information on travel time reliability.

DOT National Transportation Integrated Search

2014-01-01

The second Strategic Highway Research Program (SHRP 2) Reliability program aims to improve trip time reliability by reducing the frequency and effects of events that cause travel times to fluctuate unpredictably. Congestion caused by unreliable, or n...

Basis And Application Of The CARES/LIFE Computer Program

NASA Technical Reports Server (NTRS)

Nemeth, Noel N.; Janosik, Lesley A.; Gyekenyesi, John P.; Powers, Lynn M.

1996-01-01

Report discusses physical and mathematical basis of Ceramics Analysis and Reliability Evaluation of Structures LIFE prediction (CARES/LIFE) computer program, described in "Program for Evaluation of Reliability of Ceramic Parts" (LEW-16018).

Accuracy of remotely sensed data: Sampling and analysis procedures

NASA Technical Reports Server (NTRS)

Congalton, R. G.; Oderwald, R. G.; Mead, R. A.

1982-01-01

A review and update of the discrete multivariate analysis techniques used for accuracy assessment is given. A listing of the computer program written to implement these techniques is given. New work on evaluating accuracy assessment using Monte Carlo simulation with different sampling schemes is given. The results of matrices from the mapping effort of the San Juan National Forest is given. A method for estimating the sample size requirements for implementing the accuracy assessment procedures is given. A proposed method for determining the reliability of change detection between two maps of the same area produced at different times is given.

Implications of DSM-5 for the diagnosis of pediatric eating disorders.

PubMed

Limburg, Karina; Shu, Chloe Y; Watson, Hunna J; Hoiles, Kimberley J; Egan, Sarah J

2018-05-01

The aim of the study was to compare the DSM-IV, DSM-5, and ICD-10 eating disorders (ED) nomenclatures to assess their value in the classification of pediatric eating disorders. We investigated the prevalence of the disorders in accordance with each system's diagnostic criteria, diagnostic concordance between the systems, and interrater reliability. Participants were 1062 children and adolescents assessed at intake to a specialist Eating Disorders Program (91.6% female, mean age 14.5 years, SD = 1.75). Measures were collected from routine intake assessments. DSM-5 categorization led to a lower prevalence of unspecified EDs when compared with DSM-IV. There was almost complete overlap for specified EDs. Kappa values indicated almost excellent agreement between the two coders on all three diagnostic systems, although there was higher interrater reliability for DSM-5 and ICD-10 when compared with DSM-IV. DSM-5 nomenclature is useful in classifying eating disorders in pediatric clinical samples. © 2018 Wiley Periodicals, Inc.

Assessment of physical server reliability in multi cloud computing system

NASA Astrophysics Data System (ADS)

Kalyani, B. J. D.; Rao, Kolasani Ramchand H.

2018-04-01

Business organizations nowadays functioning with more than one cloud provider. By spreading cloud deployment across multiple service providers, it creates space for competitive prices that minimize the burden on enterprises spending budget. To assess the software reliability of multi cloud application layered software reliability assessment paradigm is considered with three levels of abstractions application layer, virtualization layer, and server layer. The reliability of each layer is assessed separately and is combined to get the reliability of multi-cloud computing application. In this paper, we focused on how to assess the reliability of server layer with required algorithms and explore the steps in the assessment of server reliability.

Enhanced Damage-Resistant Optics for Spaceflight Laser Systems: Workshop findings and recommendations

NASA Technical Reports Server (NTRS)

Schulze, Norman; Cimolino, Marc; Guenther, Arthur; Mcminn, Ted; Rainer, Frank; Schmid, Ansgar; Seitel, Steven C.; Soileau, M. J.; Theon, John S.; Walz, William

1991-01-01

NASA has defined a program to address critical laser-induced damage issues peculiar to its remote sensing systems. The Langley Research Center (LaRC), with input from the Goddard Space Flight Center (GSFC), has developed a program plan focusing on the certification of optical materials for spaceflight applications and the development of techniques to determine the reliability of such materials under extended laser exposures. This plan involves cooperative efforts between NASA and optics manufacturers to quantify the performance of optical materials for NASA systems and to ensure NASA's continued application of the highest quality optics possible for enhanced system reliability. A review panel was organized to assess NASA's optical damage concerns and to evaluate the effectiveness of the LaRC proposed program plan. This panel consisted of experts in the areas of laser-induced damage, optical coating manufacture, and the design and development of laser systems for space. The panel was presented information on NASA's current and planned laser remote sensing programs, laser-induced damage problems already encountered in NASA systems, and the proposed program plan to address these issues. Additionally, technical presentations were made on the state of the art in damage mechanisms, optical materials testing, and issues of coating manufacture germane to laser damage.

Further Empirical Data on the Psychoeducational Profile-Revised (PEP-R): Reliability and Validation with the Vineland Adaptive Behavior Scales

ERIC Educational Resources Information Center

Villa, Susanna; Micheli, Enrico; Villa, Laura; Pastore, Valentina; Crippa, Alessandro; Molteni, Massimo

2010-01-01

The PEP-R (psychoeducational profile revised) is an instrument that has been used in many countries to assess abilities and formulate treatment programs for children with autism and related developmental disorders. To the end to provide further information on the PEP-R's psychometric properties, a large sample (N = 137) of children presenting…

Reliability and Validity of a Student Scale for Assessing the Quality of Internet-Based Distance Learning

ERIC Educational Resources Information Center

Scanlan, Craig L.

2003-01-01

U.S. universities and colleges offering distance education courses have increased immensely since 1998, and by 2004 it was expected that distance learners will constitute about 14% of all those enrolled in degree programs. In its preliminary review of distance learning, the Institute for Higher Education Policy (1998) emphasized the need for…

Validation of the Italian Version of the Dizziness Handicap Inventory, the Situational Vertigo Questionnaire, and the Activity-Specific Balance Confidence Scale for Peripheral and Central Vestibular Symptoms.

PubMed

Colnaghi, Silvia; Rezzani, Cristiana; Gnesi, Marco; Manfrin, Marco; Quaglieri, Silvia; Nuti, Daniele; Mandalà, Marco; Monti, Maria Cristina; Versino, Maurizio

2017-01-01

Neurophysiological measurements of the vestibular function for diagnosis and follow-up evaluations provide an objective assessment, which, unfortunately, does not necessarily correlate with the patients' self-feeling. The literature provides many questionnaires to assess the outcome of rehabilitation programs for disequilibrium, but only for the Dizziness Handicap Inventory (DHI) is an Italian translation available, validated on a small group of patients suffering from a peripheral acute vertigo. We translated and validated the reliability and validity of the DHI, the Situational Vertigo Questionnaire (SVQ), and the Activities-Specific Balance Confidence Scale (ABC) in 316 Italian patients complaining of dizziness due either to a peripheral or to a central vestibular deficit, or in whom vestibular signs were undetectable by means of instrumental testing or clinical evaluation. Cronbach's coefficient alpha, the homogeneity index, and test-retest reproducibility, confirmed reliability of the Italian version of the three questionnaires. Validity was confirmed by correlation test between questionnaire scores. Correlations with clinical variables suggested that they can be used as a complementary tool for the assessment of vestibular symptoms. In conclusion, the Italian versions of DHI, SVQ, and ABC are reliable and valid questionnaires for assessing the impact of dizziness on the quality of life of Italian patients with peripheral or central vestibular deficit.

Validation of the Italian Version of the Dizziness Handicap Inventory, the Situational Vertigo Questionnaire, and the Activity-Specific Balance Confidence Scale for Peripheral and Central Vestibular Symptoms

PubMed Central

Colnaghi, Silvia; Rezzani, Cristiana; Gnesi, Marco; Manfrin, Marco; Quaglieri, Silvia; Nuti, Daniele; Mandalà, Marco; Monti, Maria Cristina; Versino, Maurizio

2017-01-01

Neurophysiological measurements of the vestibular function for diagnosis and follow-up evaluations provide an objective assessment, which, unfortunately, does not necessarily correlate with the patients’ self-feeling. The literature provides many questionnaires to assess the outcome of rehabilitation programs for disequilibrium, but only for the Dizziness Handicap Inventory (DHI) is an Italian translation available, validated on a small group of patients suffering from a peripheral acute vertigo. We translated and validated the reliability and validity of the DHI, the Situational Vertigo Questionnaire (SVQ), and the Activities-Specific Balance Confidence Scale (ABC) in 316 Italian patients complaining of dizziness due either to a peripheral or to a central vestibular deficit, or in whom vestibular signs were undetectable by means of instrumental testing or clinical evaluation. Cronbach’s coefficient alpha, the homogeneity index, and test–retest reproducibility, confirmed reliability of the Italian version of the three questionnaires. Validity was confirmed by correlation test between questionnaire scores. Correlations with clinical variables suggested that they can be used as a complementary tool for the assessment of vestibular symptoms. In conclusion, the Italian versions of DHI, SVQ, and ABC are reliable and valid questionnaires for assessing the impact of dizziness on the quality of life of Italian patients with peripheral or central vestibular deficit. PMID:29066999

Assessing the predictive value of the American Board of Family Practice In-training Examination.

PubMed

Replogle, William H; Johnson, William D

2004-03-01

The American Board of Family Practice In-training Examination (ABFP ITE) is a cognitive examination similar in content to the ABFP Certification Examination (CE). The ABFP ITE is widely used in family medicine residency programs. It was originally developed and intended to be used for assessment of groups of residents. Despite lack of empirical support, however, some residency programs are using ABFP ITE scores as individual resident performance indicators. This study's objective was to estimate the positive predictive value of the ABFP ITE for identifying residents at risk for poor performance on the ABFP CE or a subsequent ABFP ITE. We used a normal distribution model for correlated test scores and Monte Carlo simulation to investigate the effect of test reliability (measurement errors) on the positive predictive value of the ABFP ITE. The positive predictive value of the composite score was .72. The positive predictive value of the eight specialty subscales ranged from .26 to .57. Only the composite score of the ABFP ITE has acceptable positive predictive value to be used as part of a comprehension resident evaluation system. The ABFP ITE specialty subscales do not have sufficient positive predictive value or reliability to warrant use as performance indicators.

[Development and validity of workplace bullying in nursing-type inventory (WPBN-TI)].

PubMed

Lee, Younju; Lee, Mihyoung

2014-04-01

The purpose of this study was to develop an instrument to assess bullying of nurses, and test the validity and reliability of the instrument. The initial thirty items of WPBN-TI were identified through a review of the literature on types bullying related to nursing and in-depth interviews with 14 nurses who experienced bullying at work. Sixteen items were developed through 2 content validity tests by 9 experts and 10 nurses. The final WPBN-TI instrument was evaluated by 458 nurses from five general hospitals in the Incheon metropolitan area. SPSS 18.0 program was used to assess the instrument based on internal consistency reliability, construct validity, and criterion validity. WPBN-TI consisted of 16 items with three distinct factors (verbal and nonverbal bullying, work-related bullying, and external threats), which explained 60.3% of the total variance. The convergent validity and determinant validity for WPBN-TI were 100.0%, 89.7%, respectively. Known-groups validity of WPBN-TI was proven through the mean difference between subjective perception of bullying. The satisfied criterion validity for WPBN-TI was more than .70. The reliability of WPBN-TI was Cronbach's α of .91. WPBN-TI with high validity and reliability is suitable to determine types of bullying in nursing workplace.

Reliability and Validity of 2 Self-Report Measures to Assess Sedentary Behavior in Older Adults.

PubMed

Gennuso, Keith P; Matthews, Charles E; Colbert, Lisa H

2015-05-01

The purpose of this study was to examine the reliability and validity of 2 currently available physical activity surveys for assessing time spent in sedentary behavior (SB) in older adults. Fifty-eight adults (≥65 years) completed the Yale Physical Activity Survey for Older Adults (YPAS) and Community Health Activities Model Program for Seniors (CHAMPS) before and after a 10-day period during which they wore an ActiGraph accelerometer (ACC). Intraclass correlation coefficients (ICC) examined test-retest reliability. Overall percent agreement and a kappa statistic examined YPAS validity. Lin's concordance correlation, Pearson correlation, and Bland-Altman analysis examined CHAMPS validity. Both surveys had moderate test-retest reliability (ICC: YPAS = 0.59 (P < .001), CHAMPS = 0.64 (P < .001)) and significantly underestimated SB time. Agreement between YPAS and ACC was low (κ = -0.0003); however, there was a linear increase (P < .01) in ACC-derived SB time across YPAS response categories. There was poor agreement between ACC-derived SB and CHAMPS (Lin's r = .005; 95% CI, -0.010 to 0.020), and no linear trend across CHAMPS quartiles (P = .53). Neither of the surveys should be used as the sole measure of SB in a study; though the YPAS has the ability to rank individuals, providing it with some merit for use in correlational SB research.

[New questionnaire to assess self-efficacy toward physical activity in children].

PubMed

Aedo, Angeles; Avila, Héctor

2009-10-01

To design a questionnaire for assessment of self-efficacy toward physical activity in school children, as well as to measure its construct validity, test-retest reliability, and internal consistency. A four-stage multimethod approach was used: (1) bibliographic research followed by exploratory study and the formulation of questions and responses based on a dichotomous scale of 14 items; (2) validation of the content by a panel of experts; (3) application of the preliminary version of the questionnaire to a sample of 900 school-aged children in Mexico City; and (4) determination of the construct validity, test-retest reliability, and internal consistency (Cronbach's alpha). Three factors were identified that explain 64.15% of the variance: the search for positive alternatives to physical activity, ability to deal with possible barriers to exercising, and expectations of skill or competence. The model was validated using the goodness of fit, and the result of 65% less than 0.05 indicated that the estimated factor model fit the data. Cronbach's consistency alpha was 0.733; test-retest reliability was 0.867. The scale designed has adequate reliability and validity. These results are a good indicator of self-efficacy toward physical activity in school children, which is important when developing programs intended to promote such behavior in this age group.

Bayesian Chance-Constrained Hydraulic Barrier Design under Geological Structure Uncertainty.

PubMed

Chitsazan, Nima; Pham, Hai V; Tsai, Frank T-C

2015-01-01

The groundwater community has widely recognized geological structure uncertainty as a major source of model structure uncertainty. Previous studies in aquifer remediation design, however, rarely discuss the impact of geological structure uncertainty. This study combines chance-constrained (CC) programming with Bayesian model averaging (BMA) as a BMA-CC framework to assess the impact of geological structure uncertainty in remediation design. To pursue this goal, the BMA-CC method is compared with traditional CC programming that only considers model parameter uncertainty. The BMA-CC method is employed to design a hydraulic barrier to protect public supply wells of the Government St. pump station from salt water intrusion in the "1500-foot" sand and the "1700-foot" sand of the Baton Rouge area, southeastern Louisiana. To address geological structure uncertainty, three groundwater models based on three different hydrostratigraphic architectures are developed. The results show that using traditional CC programming overestimates design reliability. The results also show that at least five additional connector wells are needed to achieve more than 90% design reliability level. The total amount of injected water from the connector wells is higher than the total pumpage of the protected public supply wells. While reducing the injection rate can be achieved by reducing the reliability level, the study finds that the hydraulic barrier design to protect the Government St. pump station may not be economically attractive. © 2014, National Ground Water Association.

Analysis of whisker-toughened CMC structural components using an interactive reliability model

NASA Technical Reports Server (NTRS)

Duffy, Stephen F.; Palko, Joseph L.

1992-01-01

Realizing wider utilization of ceramic matrix composites (CMC) requires the development of advanced structural analysis technologies. This article focuses on the use of interactive reliability models to predict component probability of failure. The deterministic William-Warnke failure criterion serves as theoretical basis for the reliability model presented here. The model has been implemented into a test-bed software program. This computer program has been coupled to a general-purpose finite element program. A simple structural problem is presented to illustrate the reliability model and the computer algorithm.

Stirling engine - Approach for long-term durability assessment

NASA Technical Reports Server (NTRS)

Tong, Michael T.; Bartolotta, Paul A.; Halford, Gary R.; Freed, Alan D.

1992-01-01

The approach employed by NASA Lewis for the long-term durability assessment of the Stirling engine hot-section components is summarized. The approach consists of: preliminary structural assessment; development of a viscoplastic constitutive model to accurately determine material behavior under high-temperature thermomechanical loads; an experimental program to characterize material constants for the viscoplastic constitutive model; finite-element thermal analysis and structural analysis using a viscoplastic constitutive model to obtain stress/strain/temperature at the critical location of the hot-section components for life assessment; and development of a life prediction model applicable for long-term durability assessment at high temperatures. The approach should aid in the provision of long-term structural durability and reliability of Stirling engines.

A preliminary evaluation of sediment quality assessment values for freshwater ecosystems

USGS Publications Warehouse

Smith, Sherri L.; MacDonald, Donald D.; Keenleyside, Karen A.; Ingersoll, Christopher G.; Field, L. Jay

1996-01-01

Sediment quality assessment values were developed using a weight of evidence approach in which matching biological and chemical data from numerous modelling, laboratory, and field studies performed on freshwater sediments were compiled and analyzed. Two assessment values (a threshold effect level (TEL) and a probable effect level(PEL)) were derived for 23 substances, including eight trace metals, six individual polycyclic aromatic hydrocarbons (PAHs), total polychlorinated biphenyls (PCBs), and eight pesticides. The two values defined three ranges of chemical concentrations; those that were (1) rarely, (2) occasionally, and (3) frequently associated with adverse biological effects. An evaluation of the percent incidence of adverse biological effects within the three concentration ranges indicated that the reliability of the TELs (i.e., the degree to which the TELs represent concentrations within the data set below which adverse effects rarely occur) was consistently good. However, this preliminary evaluation indicated that most of the PELs were less reliable (i.e., they did not adequately represent concentrations within the data set above which adverse effects frequently occur). Nonetheless, these values were often comparable to other biological effects-based assessment values (which were themselves reliable), which increased the level of confidence that could be placed in our values. This method is being used as a basis for developing national sediment quality guidelines for freshwater systems in Canada and sediment effect concentrations as part of the Assessment and Remediation of Contaminated Sediments (ARCS) program in the Great Lakes.

A Guide to the Application of Probability Risk Assessment Methodology and Hazard Risk Frequency Criteria as a Hazard Control for the Use of the Mobile Servicing System on the International Space Station

NASA Astrophysics Data System (ADS)

D'silva, Oneil; Kerrison, Roger

2013-09-01

A key feature for the increased utilization of space robotics is to automate Extra-Vehicular manned space activities and thus significantly reduce the potential for catastrophic hazards while simultaneously minimizing the overall costs associated with manned space. The principal scope of the paper is to evaluate the use of industry standard accepted Probability risk/safety assessment (PRA/PSA) methodologies and Hazard Risk frequency Criteria as a hazard control. This paper illustrates the applicability of combining the selected Probability risk assessment methodology and hazard risk frequency criteria, in order to apply the necessary safety controls that allow for the increased use of the Mobile Servicing system (MSS) robotic system on the International Space Station. This document will consider factors such as component failure rate reliability, software reliability, and periods of operation and dormancy, fault tree analyses and their effects on the probability risk assessments. The paper concludes with suggestions for the incorporation of existing industry Risk/Safety plans to create an applicable safety process for future activities/programs

What should students learn about complementary and alternative medicine?

PubMed

Gaster, Barak; Unterborn, John N; Scott, Richard B; Schneeweiss, Ronald

2007-10-01

With thousands of complementary and alternative medicine (CAM) treatments currently being used in the United States today, it is challenging to design a concise body of CAM content which will fit into already overly full curricula for health care students. The purpose of this article is to outline key principles which 15 National Center for Complementary and Alternative Medicine-funded education programs found useful when developing CAM course-work and selecting CAM content. Three key guiding principles are discussed: teach foundational CAM competencies to give students a framework for learning about CAM; choose specific content on the basis of evidence, demographics and condition (what conditions are most appropriate for CAM therapies?); and finally, provide students with skills for future learning, including where to find reliable information about CAM and how to search the scientific literature and assess the results of CAM research. Most of the programs developed evidence-based guides to help students find reliable CAM resources. The cumulative experiences of the 15 programs have been compiled, and an annotated table outlining the most highly recommended resources about CAM is presented.

Reliability and validity of a novel Kinect-based software program for measuring posture, balance and side-bending.

PubMed

Grooten, Wilhelmus Johannes Andreas; Sandberg, Lisa; Ressman, John; Diamantoglou, Nicolas; Johansson, Elin; Rasmussen-Barr, Eva

2018-01-08

Clinical examinations are subjective and often show a low validity and reliability. Objective and highly reliable quantitative assessments are available in laboratory settings using 3D motion analysis, but these systems are too expensive to use for simple clinical examinations. Qinematic™ is an interactive movement analyses system based on the Kinect camera and is an easy-to-use clinical measurement system for assessing posture, balance and side-bending. The aim of the study was to test the test-retest the reliability and construct validity of Qinematic™ in a healthy population, and to calculate the minimal clinical differences for the variables of interest. A further aim was to identify the discriminative validity of Qinematic™ in people with low-back pain (LBP). We performed a test-retest reliability study (n = 37) with around 1 week between the occasions, a construct validity study (n = 30) in which Qinematic™ was tested against a 3D motion capture system, and a discriminative validity study, in which a group of people with LBP (n = 20) was compared to healthy controls (n = 17). We tested a large range of psychometric properties of 18 variables in three sections: posture (head and pelvic position, weight distribution), balance (sway area and velocity in single- and double-leg stance), and side-bending. The majority of the variables in the posture and balance sections, showed poor/fair reliability (ICC < 0.4) and poor/fair validity (Spearman <0.4), with significant differences between occasions, between Qinematic™ and the 3D-motion capture system. In the clinical study, Qinematic™ did not differ between people with LPB and healthy for these variables. For one variable, side-bending to the left, there was excellent reliability (ICC =0.898), excellent validity (r = 0.943), and Qinematic™ could differentiate between LPB and healthy individuals (p = 0.012). This paper shows that a novel software program (Qinematic™) based on the Kinect camera for measuring balance, posture and side-bending has poor psychometric properties, indicating that the variables on balance and posture should not be used for monitoring individual changes over time or in research. Future research on the dynamic tasks of Qinematic™ is warranted.

Reliability of a quantitative clinical posture assessment tool among persons with idiopathic scoliosis.

PubMed

Fortin, Carole; Feldman, Debbie Ehrmann; Cheriet, Farida; Gravel, Denis; Gauthier, Frédérique; Labelle, Hubert

2012-03-01

To determine overall, test-retest and inter-rater reliability of posture indices among persons with idiopathic scoliosis. A reliability study using two raters and two test sessions. Tertiary care paediatric centre. Seventy participants aged between 10 and 20 years with different types of idiopathic scoliosis (Cobb angle 15 to 60°) were recruited from the scoliosis clinic. Based on the XY co-ordinates of natural reference points (e.g., eyes) as well as markers placed on several anatomical landmarks, 32 angular and linear posture indices taken from digital photographs in the standing position were calculated from a specially developed software program. Generalisability theory served to estimate the reliability and standard error of measurement (SEM) for the overall, test-retest and inter-rater designs. Bland and Altman's method was also used to document agreement between sessions and raters. In the random design, dependability coefficients demonstrated a moderate level of reliability for six posture indices (ϕ=0.51 to 0.72) and a good level of reliability for 26 posture indices out of 32 (ϕ≥0.79). Error attributable to marker placement was negligible for most indices. Limits of agreement and SEM values were larger for shoulder protraction, trunk list, Q angle, cervical lordosis and scoliosis angles. The most reproducible indices were waist angles and knee valgus and varus. Posture can be assessed in a global fashion from photographs in persons with idiopathic scoliosis. Despite the good reliability of marker placement, other studies are needed to minimise measurement errors in order to provide a suitable tool for monitoring change in posture over time. Copyright © 2011 Chartered Society of Physiotherapy. Published by Elsevier Ltd. All rights reserved.

Digital tooth-based superimposition method for assessment of alveolar bone levels on cone-beam computed tomography images.

PubMed

Romero-Delmastro, Alejandro; Kadioglu, Onur; Currier, G Frans; Cook, Tanner

2014-08-01

Cone-beam computed tomography images have been previously used for evaluation of alveolar bone levels around teeth before, during, and after orthodontic treatment. Protocols described in the literature have been vague, have used unstable landmarks, or have required several software programs, file conversions, or hand tracings, among other factors that could compromise the precision of the measurements. The purposes of this article are to describe a totally digital tooth-based superimposition method for the quantitative assessment of alveolar bone levels and to evaluate its reliability. Ultra cone-beam computed tomography images (0.1-mm reconstruction) from 10 subjects were obtained from the data pool of the University of Oklahoma; 80 premolars were measured twice by the same examiner and a third time by a second examiner to determine alveolar bone heights and thicknesses before and more than 6 months after orthodontic treatment using OsiriX (version 3.5.1; Pixeo, Geneva, Switzerland). Intraexaminer and interexaminer reliabilities were evaluated, and Dahlberg's formula was used to calculate the error of the measurements. Cross-sectional and longitudinal evaluations of alveolar bone levels were possible using a digital tooth-based superimposition method. The mean differences for buccal alveolar crest heights and thicknesses were below 0.10 mm for the same examiner and below 0.17 mm for all examiners. The ranges of errors for any measurement were between 0.02 and 0.23 mm for intraexaminer errors, and between 0.06 and 0.29 mm for interexaminer errors. This protocol can be used for cross-sectional or longitudinal assessment of alveolar bone levels with low interexaminer and intraexaminer errors, and it eliminates the use of less reliable or less stable landmarks and the need for multiple software programs and image printouts. Standardization of the methods for bone assessment in orthodontics is necessary; this method could be the answer to this need. Copyright © 2014 American Association of Orthodontists. Published by Mosby, Inc. All rights reserved.

Health and Safety Checklist for Early Care and Education Programs to Assess Key National Health and Safety Standards.

PubMed

Alkon, Abbey; Rose, Roberta; Wolff, Mimi; Kotch, Jonathan B; Aronson, Susan S

2016-01-01

The project aims were to (1) develop an observational Health and Safety Checklist to assess health and safety practices and conditions in early care and education (ECE) programs using Stepping Stones To Caring For Our Children, 3rd Edition national standards, (2) pilot test the Checklist, completed by nurse child care health consultants, to assess feasibility, ease of completion, objectivity, validity, and reliability, and (3) revise the Checklist based on the qualitative and quantitative results of the pilot study. The observable national health and safety standards were identified and then rated by health, safety, and child care experts using a Delphi technique to validate the standards as essential to prevent harm and promote health. Then, child care health consultants recruited ECE centers and pilot tested the 124-item Checklist. The pilot study was conducted in Arizona, California and North Carolina. The psychometric properties of the Checklist were assessed. The 37 participating ECE centers had 2627 children from ethnically-diverse backgrounds and primarily low-income families. The child care health consultants found the Checklist easy to complete, objective, and useful for planning health and safety interventions. The Checklist had content and face validity, inter-rater reliability, internal consistency, and concurrent validity. Based on the child care health consultant feedback and psychometric properties of the Checklist, the Checklist was revised and re-written at an 8th grade literacy level. The Health and Safety Checklist provides a standardized instrument of observable, selected national standards to assess the quality of health and safety in ECE centers.

A Novel Scenario-Based Interview Tool to Evaluate Nontechnical Skills and Competencies in Global Health Delivery.

PubMed

Wroe, Emily B; McBain, Ryan K; Michaelis, Annie; Dunbar, Elizabeth L; Hirschhorn, Lisa R; Cancedda, Corrado

2017-08-01

Despite rapid growth in the number of physicians and academic institutions entering the field of global health, there are few tools that inform global health curricula and assess physician readiness for this field. To address this gap, we describe the development and pilot testing of a new tool to assess nontechnical competencies and values in global health. Competencies assessed include systems-based practice, interpersonal and cross-cultural communication, professionalism and self-care, patient care, mentoring, teaching, management, and personal motivation and experience. The Global Health Delivery Competency Assessment Tool presents 15 case vignettes and open-ended questions related to situations a global health practitioner might encounter, and grades the quality of responses on a 6-point ordinal scale. We interviewed 17 of 18 possible global health residents (94%), matched with 17 residents not training in global health, for a total of 34 interviews. A second reviewer independently scored recordings of 13 interviews for reliability. Pilot testing indicated a high degree of discriminant validity, as measured by the instrument's ability to distinguish between residents who were and were not enrolled in a global health program ( P  < .001). It also demonstrated acceptable consistency, as assessed by interrater reliability (κ = 0.53), with a range of item-level agreement from 84%-96%. The tool has potential applicability to a variety of academic and programmatic activities, including evaluation of candidates for global health positions and evaluating the success of training programs in equipping practitioners for entry into this field.

Component technology for stirling power converters

NASA Technical Reports Server (NTRS)

Thieme, Lanny G.

1991-01-01

NASA Lewis Research Center has organized a component technology program as part of the efforts to develop Stirling converter technology for space power applications. The Stirling Space Power Program is part of the NASA High Capacity Power Project of the Civil Space Technology Initiative (CSTI). NASA Lewis is also providing technical management for the DOE/Sandia program to develop Stirling converters for solar terrestrial power producing electricity for the utility grid. The primary contractors for the space power and solar terrestrial programs develop component technologies directly related to their goals. This Lewis component technology effort, while coordinated with the main programs, aims at longer term issues, advanced technologies, and independent assessments. An overview of work on linear alternators, engine/alternator/load interactions and controls, heat exchangers, materials, life and reliability, and bearings is presented.

A plan for the North American Bat Monitoring Program (NABat)

USGS Publications Warehouse

Loeb, Susan C.; Rodhouse, Thomas J.; Ellison, Laura E.; Lausen, Cori L.; Reichard, Jonathan D.; Irvine, Kathryn M.; Ingersoll, Thomas E.; Coleman, Jeremy; Thogmartin, Wayne E.; Sauer, John R.; Francis, Charles M.; Bayless, Mylea L.; Stanley, Thomas R.; Johnson, Douglas H.

2015-01-01

The purpose of the North American Bat Monitoring Program (NABat) is to create a continent-wide program to monitor bats at local to rangewide scales that will provide reliable data to promote effective conservation decisionmaking and the long-term viability of bat populations across the continent. This is an international, multiagency program. Four approaches will be used to gather monitoring data to assess changes in bat distributions and abundances: winter hibernaculum counts, maternity colony counts, mobile acoustic surveys along road transects, and acoustic surveys at stationary points. These monitoring approaches are described along with methods for identifying species recorded by acoustic detectors. Other chapters describe the sampling design, the database management system (Bat Population Database), and statistical approaches that can be used to analyze data collected through this program.

Reliability Measure of a Clinical Test: Appreciation of Music in Cochlear Implantees (AMICI)

PubMed Central

Cheng, Min-Yu; Spitzer, Jaclyn B.; Shafiro, Valeriy; Sheft, Stanley; Mancuso, Dean

2014-01-01

Purpose The goals of this study were (1) to investigate the reliability of a clinical music perception test, Appreciation of Music in Cochlear Implantees (AMICI), and (2) examine associations between the perception of music and speech. AMICI was developed as a clinical instrument for assessing music perception in persons with cochlear implants (CIs). The test consists of four subtests: (1) music versus environmental noise discrimination, (2) musical instrument identification (closed-set), (3) musical style identification (closed-set), and (4) identification of musical pieces (open-set). To be clinically useful, it is crucial for AMICI to demonstrate high test-retest reliability, so that CI users can be assessed and retested after changes in maps or programming strategies. Research Design Thirteen CI subjects were tested with AMICI for the initial visit and retested again 10–14 days later. Two speech perception tests (consonant-nucleus-consonant [CNC] and Bamford-Kowal-Bench Speech-in-Noise [BKB-SIN]) were also administered. Data Analysis Test-retest reliability and equivalence of the test’s three forms were analyzed using paired t-tests and correlation coefficients, respectively. Correlation analysis was also conducted between results from the music and speech perception tests. Results Results showed no significant difference between test and retest (p > 0.05) with adequate power (0.9) as well as high correlations between the three forms (Forms A and B, r = 0.91; Forms A and C, r = 0.91; Forms B and C, r = 0.95). Correlation analysis showed high correlation between AMICI and BKB-SIN (r = −0.71), and moderate correlation between AMICI and CNC (r = 0.4). Conclusions The study showed AMICI is highly reliable for assessing musical perception in CI users. PMID:24384082

[An instrument in Spanish to evaluate the performance of clinical teachers by students].

PubMed

Bitran, Marcela; Mena, Beltrán; Riquelme, Arnoldo; Padilla, Oslando; Sánchez, Ignacio; Moreno, Rodrigo

2010-06-01

The modernization of clinical teaching has called for the creation of faculty development programs, and the design of suitable instruments to evaluate clinical teachers' performance. To report the development and validation of an instrument in Spanish designed to measure the students' perceptions of their clinical teachers' performance and to provide them with feedback to improve their teaching practices. In a process that included the active participation of authorities, professors in charge of courses and internships, clinical teachers, students and medical education experts, we developed a 30-item questionnaire called MEDUC30 to evaluate the performance of clinical teachers by their students. The internal validity was assessed by factor analysis of 5214 evaluations of 265 teachers, gathered from 2004 to 2007. The reliability was measured with the Cronbach's alpha coefficient and the generalizability coefficient (g). MEDUC30 had good content and construct validity. Its internal structure was compatible with four factors: patient-centered teaching, teaching skills, assessment skills and learning climate, and it proved to be consistent with the structure anticipated by the theory. The scores were highly reliable (Cronbach's alpha: 0.97); five evaluations per teacher were sufficient to reach a reliability coefficient (g) of 0.8. MEDUC30 is a valid, reliable and useful instrument to evaluate the performance of clinical teachers. To our knowledge, this is the first instrument in Spanish for which solid validity and reliability evidences have been reported. We hope that MEDUC30 will be used to improve medical education in Spanish-speaking medical schools, providing teachers a specific feedback upon which to improve their pedagogical practice, and authorities with valuable information for the assessment of their faculty.

Teaching program for the Unified Dyskinesia Rating Scale.

PubMed

Goetz, Christopher G; Nutt, John G; Stebbins, Glenn T; Chmura, Teresa A

2009-07-15

The Unified Dyskinesia Rating Scale (UDysRS) has been introduced as a comprehensive rating tool for the evaluation of dyskinesias in Parkinson's disease (PD). To enhance a uniform application, we developed a DVD-based training program with instructions, patient examples, and a certification exercise. For training on the objective assessment of dyskinesia, seventy PD patients spanning the gamut of dyskinesias (none to severe) were videotaped during four tasks of daily living (speaking, drinking from a cup, putting on a coat, and walking). Dyskinesia severity in seven body parts was rated by 20 international movement disorder specialists using the UDysRS for impairment. Each task was also rated for disability. Inter-rater reliability was assessed with generalized weighted kappa and intraclass correlation coefficients. For the teaching program, examples of each severity level and each body part were selected based on the criterion that they received a uniform rating (+/- 1 point) by at least 75% of the raters. For the certification exercise, four cases were selected to represent the four quartiles of overall objective UDysRS scores to reflect slight, mild, moderate, and severe dyskinesia. Each selection was based on the highest inter-rater reliability score for that quartile (minimum kappa or intraclass correlation coefficient = 0.6). UDysRS ranges for certification were calculated based on the 95% confidence interval. The teaching program lasts 41 min, and the certification exercise requires 10 min (total 51 min). This training program, based on visual examples of dyskinesia and anchored in scores generated by movement disorder experts is aimed at increasing homogeneity of ratings among and within raters and centers. Large-scale multicenter randomized clinical trials of dyskinesia treatment are strengthened by a uniform standard of scale application. 2009 Movement Disorder Society.

Reliability of cervical vertebral maturation staging.

PubMed

Rainey, Billie-Jean; Burnside, Girvan; Harrison, Jayne E

2016-07-01

Growth and its prediction are important for the success of many orthodontic treatments. The aim of this study was to determine the reliability of the cervical vertebral maturation (CVM) method for the assessment of mandibular growth. A group of 20 orthodontic clinicians, inexperienced in CVM staging, was trained to use the improved version of the CVM method for the assessment of mandibular growth with a teaching program. They independently assessed 72 consecutive lateral cephalograms, taken at Liverpool University Dental Hospital, on 2 occasions. The cephalograms were presented in 2 different random orders and interspersed with 11 additional images for standardization. The intraobserver and interobserver agreement values were evaluated using the weighted kappa statistic. The intraobserver and interobserver agreement values were substantial (weighted kappa, 0.6-0.8). The overall intraobserver agreement was 0.70 (SE, 0.01), with average agreement of 89%. The interobserver agreement values were 0.68 (SE, 0.03) for phase 1 and 0.66 (SE, 0.03) for phase 2, with average interobserver agreement of 88%. The intraobserver and interobserver agreement values of classifying the vertebral stages with the CVM method were substantial. These findings demonstrate that this method of CVM classification is reproducible and reliable. Copyright © 2016 American Association of Orthodontists. Published by Elsevier Inc. All rights reserved.

Video training and certification program improves reliability of postischemic neurologic deficit measurement in the rat.

PubMed

Taninishi, Hideki; Pearlstein, Molly; Sheng, Huaxin; Izutsu, Miwa; Chaparro, Rafael E; Goldstein, Larry B; Warner, David S

2016-12-01

Scoring systems are used to measure behavioral deficits in stroke research. Video-assisted training is used to standardize stroke-related neurologic deficit scoring in humans. We hypothesized that a video-assisted training and certification program can improve inter-rater reliability in assessing neurologic function after middle cerebral artery occlusion in rats. Three expert raters scored neurologic deficits in post-middle cerebral artery occlusion rats using three published systems having different complexity levels (3, 18, or 48 points). The system having the highest point estimate for the correlation between neurologic score and infarct size was selected to create a video-assisted training and certification program. Eight trainee raters completed the video-assisted training and certification program. Inter-rater agreement ( Κ: score) and agreement with expert consensus scores were measured before and after video-assisted training and certification program completion. The 48-point system correlated best with infarct size. Video-assisted training and certification improved agreement with expert consensus scores (pretraining = 65 ± 10, posttraining = 87 ± 14, 112 possible scores, P < 0.0001), median number of trainee raters with scores within ±2 points of the expert consensus score (pretraining = 4, posttraining = 6.5, P < 0.01), categories with Κ:  > 0.4 (pretraining = 4, posttraining = 9), and number of categories with an improvement in the Κ: score from pretraining to posttraining (n = 6). Video-assisted training and certification improved trainee inter-rater reliability and agreement with expert consensus behavioral scores in rats after middle cerebral artery occlusion. Video-assisted training and certification may be useful in multilaboratory preclinical studies. © The Author(s) 2015.

A reliability as an independent variable (RAIV) methodology for optimizing test planning for liquid rocket engines

NASA Astrophysics Data System (ADS)

Strunz, Richard; Herrmann, Jeffrey W.

2011-12-01

The hot fire test strategy for liquid rocket engines has always been a concern of space industry and agency alike because no recognized standard exists. Previous hot fire test plans focused on the verification of performance requirements but did not explicitly include reliability as a dimensioning variable. The stakeholders are, however, concerned about a hot fire test strategy that balances reliability, schedule, and affordability. A multiple criteria test planning model is presented that provides a framework to optimize the hot fire test strategy with respect to stakeholder concerns. The Staged Combustion Rocket Engine Demonstrator, a program of the European Space Agency, is used as example to provide the quantitative answer to the claim that a reduced thrust scale demonstrator is cost beneficial for a subsequent flight engine development. Scalability aspects of major subsystems are considered in the prior information definition inside the Bayesian framework. The model is also applied to assess the impact of an increase of the demonstrated reliability level on schedule and affordability.

Monolithic ceramic analysis using the SCARE program

NASA Technical Reports Server (NTRS)

Manderscheid, Jane M.

1988-01-01

The Structural Ceramics Analysis and Reliability Evaluation (SCARE) computer program calculates the fast fracture reliability of monolithic ceramic components. The code is a post-processor to the MSC/NASTRAN general purpose finite element program. The SCARE program automatically accepts the MSC/NASTRAN output necessary to compute reliability. This includes element stresses, temperatures, volumes, and areas. The SCARE program computes two-parameter Weibull strength distributions from input fracture data for both volume and surface flaws. The distributions can then be used to calculate the reliability of geometrically complex components subjected to multiaxial stress states. Several fracture criteria and flaw types are available for selection by the user, including out-of-plane crack extension theories. The theoretical basis for the reliability calculations was proposed by Batdorf. These models combine linear elastic fracture mechanics (LEFM) with Weibull statistics to provide a mechanistic failure criterion. Other fracture theories included in SCARE are the normal stress averaging technique and the principle of independent action. The objective of this presentation is to summarize these theories, including their limitations and advantages, and to provide a general description of the SCARE program, along with example problems.

Design and development of food safety knowledge and attitude scales for consumer food safety education.

PubMed

Medeiros, Lydia C; Hillers, Virginia N; Chen, Gang; Bergmann, Verna; Kendall, Patricia; Schroeder, Mary

2004-11-01

The objective of this study was to design and develop food safety knowledge and attitude scales based on food-handling guidelines developed by a national panel of food safety experts. Knowledge (n=43) and attitude (n=49) questions were developed and pilot-tested with a variety of consumer groups. Final questions were selected based on item analysis and on validity and reliability statistical tests. Knowledge questions were tested in Washington State with participants in low-income nutrition education programs (pretest/posttest n=58, test/retest n=19) and college students (pretest/posttest n=34). Attitude questions were tested in Ohio with nutrition education program participants (n=30) and college students (non-nutrition majors n=138, nutrition majors n=57). Item analysis, paired sample t tests, Pearson's correlation coefficients, and Cronbach's alpha were used. Reliability and validity tests of individual items and the question sets were used to reduce the scales to 18 knowledge questions and 10 attitude questions. The knowledge and attitude scales covered topics ranked as important by a national panel of experts and met most validity and reliability standards. The 18-item knowledge questionnaire had instructional sensitivity (mean score increase of more than three points after instruction), internal reliability (Cronbach's alpha >.75), and produced similar results in test-retest without intervention (coefficient of stability=.81). Knowledge of correct procedures for hand washing and avoiding cross-contamination was widespread before instruction. Knowledge was limited regarding avoiding food preparation while ill, cooking hamburgers, high-risk foods, and whether cooked rice and potatoes could be stored at room temperature. The 10-item attitude scale had an appropriate range of responses (item difficulty) and produced similar results in test-retest ( P

An enhanced reliability-oriented workforce planning model for process industry using combined fuzzy goal programming and differential evolution approach

NASA Astrophysics Data System (ADS)

Ighravwe, D. E.; Oke, S. A.; Adebiyi, K. A.

2018-03-01

This paper draws on the "human reliability" concept as a structure for gaining insight into the maintenance workforce assessment in a process industry. Human reliability hinges on developing the reliability of humans to a threshold that guides the maintenance workforce to execute accurate decisions within the limits of resources and time allocations. This concept offers a worthwhile point of deviation to encompass three elegant adjustments to literature model in terms of maintenance time, workforce performance and return-on-workforce investments. These fully explain the results of our influence. The presented structure breaks new grounds in maintenance workforce theory and practice from a number of perspectives. First, we have successfully implemented fuzzy goal programming (FGP) and differential evolution (DE) techniques for the solution of optimisation problem in maintenance of a process plant for the first time. The results obtained in this work showed better quality of solution from the DE algorithm compared with those of genetic algorithm and particle swarm optimisation algorithm, thus expressing superiority of the proposed procedure over them. Second, the analytical discourse, which was framed on stochastic theory, focusing on specific application to a process plant in Nigeria is a novelty. The work provides more insights into maintenance workforce planning during overhaul rework and overtime maintenance activities in manufacturing systems and demonstrated capacity in generating substantially helpful information for practice.

CARES/LIFE Ceramics Analysis and Reliability Evaluation of Structures Life Prediction Program

NASA Technical Reports Server (NTRS)

Nemeth, Noel N.; Powers, Lynn M.; Janosik, Lesley A.; Gyekenyesi, John P.

2003-01-01

This manual describes the Ceramics Analysis and Reliability Evaluation of Structures Life Prediction (CARES/LIFE) computer program. The program calculates the time-dependent reliability of monolithic ceramic components subjected to thermomechanical and/or proof test loading. CARES/LIFE is an extension of the CARES (Ceramic Analysis and Reliability Evaluation of Structures) computer program. The program uses results from MSC/NASTRAN, ABAQUS, and ANSYS finite element analysis programs to evaluate component reliability due to inherent surface and/or volume type flaws. CARES/LIFE accounts for the phenomenon of subcritical crack growth (SCG) by utilizing the power law, Paris law, or Walker law. The two-parameter Weibull cumulative distribution function is used to characterize the variation in component strength. The effects of multiaxial stresses are modeled by using either the principle of independent action (PIA), the Weibull normal stress averaging method (NSA), or the Batdorf theory. Inert strength and fatigue parameters are estimated from rupture strength data of naturally flawed specimens loaded in static, dynamic, or cyclic fatigue. The probabilistic time-dependent theories used in CARES/LIFE, along with the input and output for CARES/LIFE, are described. Example problems to demonstrate various features of the program are also included.

Assessing reflective thinking and approaches to learning.

PubMed

Dunn, Louise; Musolino, Gina M

2011-01-01

Facilitation of reflective practice is critical for the ongoing demands of health care practitioners. Reflective thinking concepts, grounded in the work of Dewey and Schön, emphasize critical reflection to promote transformation in beliefs and learning necessary for reflective practice. The Reflective Thinking Questionnaire (QRT) and Revised Study Process Questionnaire (RSPQ-2F) assess skill aspects of professional reasoning, with promise for measuring changes over time. The purpose of this study was to examine the reliability and responsiveness and the model validity of reflective thinking and approaches to learning measures for U.S. health professions students enrolled in entry-level occupational (MOT) and physical therapy (DPT) programs. This measurement study addressed reliability and responsiveness of two measures, the QRT and RSPQ-2F, for graduate health professionals. A convenience sample of 125 MOT and DPT students participated in the two-measure, test-retest investigation, with electronic data collection. Outcomes support the stability of the four-scale QRT (ICC 0.63 to 0.82) and the two-scale RSPQ-2F (ICC 0.91 and 0.87). Descriptive data supporting responsiveness are presented. With noted limitations, the results support the use of the QRT and RSPQ-2F measures to assess changes in reflective thinking and approaches to learning. Measurement of these learning outcomes furthers our understanding and knowledge about instructional strategies, development of professional reasoning, and fostering of self-directed learning within MOT and DPT programs.

Design for reliability: NASA reliability preferred practices for design and test

NASA Technical Reports Server (NTRS)

Lalli, Vincent R.

1994-01-01

This tutorial summarizes reliability experience from both NASA and industry and reflects engineering practices that support current and future civil space programs. These practices were collected from various NASA field centers and were reviewed by a committee of senior technical representatives from the participating centers (members are listed at the end). The material for this tutorial was taken from the publication issued by the NASA Reliability and Maintainability Steering Committee (NASA Reliability Preferred Practices for Design and Test. NASA TM-4322, 1991). Reliability must be an integral part of the systems engineering process. Although both disciplines must be weighed equally with other technical and programmatic demands, the application of sound reliability principles will be the key to the effectiveness and affordability of America's space program. Our space programs have shown that reliability efforts must focus on the design characteristics that affect the frequency of failure. Herein, we emphasize that these identified design characteristics must be controlled by applying conservative engineering principles.

A Web-Based System for Bayesian Benchmark Dose Estimation.

PubMed

Shao, Kan; Shapiro, Andrew J

2018-01-11

Benchmark dose (BMD) modeling is an important step in human health risk assessment and is used as the default approach to identify the point of departure for risk assessment. A probabilistic framework for dose-response assessment has been proposed and advocated by various institutions and organizations; therefore, a reliable tool is needed to provide distributional estimates for BMD and other important quantities in dose-response assessment. We developed an online system for Bayesian BMD (BBMD) estimation and compared results from this software with U.S. Environmental Protection Agency's (EPA's) Benchmark Dose Software (BMDS). The system is built on a Bayesian framework featuring the application of Markov chain Monte Carlo (MCMC) sampling for model parameter estimation and BMD calculation, which makes the BBMD system fundamentally different from the currently prevailing BMD software packages. In addition to estimating the traditional BMDs for dichotomous and continuous data, the developed system is also capable of computing model-averaged BMD estimates. A total of 518 dichotomous and 108 continuous data sets extracted from the U.S. EPA's Integrated Risk Information System (IRIS) database (and similar databases) were used as testing data to compare the estimates from the BBMD and BMDS programs. The results suggest that the BBMD system may outperform the BMDS program in a number of aspects, including fewer failed BMD and BMDL calculations and estimates. The BBMD system is a useful alternative tool for estimating BMD with additional functionalities for BMD analysis based on most recent research. Most importantly, the BBMD has the potential to incorporate prior information to make dose-response modeling more reliable and can provide distributional estimates for important quantities in dose-response assessment, which greatly facilitates the current trend for probabilistic risk assessment. https://doi.org/10.1289/EHP1289.

The Chinese version of Instrument of Professional Attitude for Student Nurses (IPASN): Assessment of reliability and validity.

PubMed

Xiao, Yu-Ying; Li, Ting; Xiao, Lin; Wang, Su-Wei; Wang, Si-Qi; Wang, Han-Xiao; Wang, Bei-Bei; Gao, Yu-Lin

2017-02-01

Professional attitude is of great importance for nursing talents in the modern society. To develop an effective educational program for student nurses in China, an appropriate instrument is required for the assessment of their professional attitude. To assess the validity and reliability of the Instrument of Professional Attitude for Student Nurses (IPASN) in Chinese version. The original version of IPASN was translated through Brislin model (translation, back translation, culture adaption and pilot study) with the authorization from the developer. A total of 681 nursing students were chosen by stratified convenience sampling to assess construct validity using exploratory factor analysis (EFA). Besides, item analysis, Cronbach's alpha coefficients, test-retest reliability were conducted to test the psychometric properties in this part. A total of 204 nursing undergraduate trainees were selected by cluster convenience sampling to confirm the structure using confirmatory factor analysis (CFA) in another time. Corrected item-total correlations, alpha if item deleted were between 0.33 and 0.69, 0.906 and 0.913, respectively, indicating no item should be deleted. Cronbach alpha value was 0.91 for the total scale and Cronbach alpha coefficient for subscales ranged from 0.67 to 0.89. Test-retest reliability estimated from intraclass correlation coefficient (ICC) was 0.74 (P<0.05). Differences in item scores between the high-score group (the first 27%) and low-score group (the last 27%) were significant (P<0.001), indicating that the item discrimination ability was good. Seven subscales (contribution to increase of scientific information load, autonomy, community service, continuous education, to promote professional development, cooperation and theory guiding practice) were identified in EFA and confirmed in CFA, and explained 65.5% of the total variance. It indicated that the Chinese version of IPASN was valid and reliable for the evaluation of nursing students' professional attitude. Copyright © 2016 Elsevier Ltd. All rights reserved.

Software reliability models for critical applications

DOE Office of Scientific and Technical Information (OSTI.GOV)

Pham, H.; Pham, M.

This report presents the results of the first phase of the ongoing EG G Idaho, Inc. Software Reliability Research Program. The program is studying the existing software reliability models and proposes a state-of-the-art software reliability model that is relevant to the nuclear reactor control environment. This report consists of three parts: (1) summaries of the literature review of existing software reliability and fault tolerant software reliability models and their related issues, (2) proposed technique for software reliability enhancement, and (3) general discussion and future research. The development of this proposed state-of-the-art software reliability model will be performed in the secondmore » place. 407 refs., 4 figs., 2 tabs.« less

Software reliability models for critical applications

DOE Office of Scientific and Technical Information (OSTI.GOV)

Pham, H.; Pham, M.

This report presents the results of the first phase of the ongoing EG&G Idaho, Inc. Software Reliability Research Program. The program is studying the existing software reliability models and proposes a state-of-the-art software reliability model that is relevant to the nuclear reactor control environment. This report consists of three parts: (1) summaries of the literature review of existing software reliability and fault tolerant software reliability models and their related issues, (2) proposed technique for software reliability enhancement, and (3) general discussion and future research. The development of this proposed state-of-the-art software reliability model will be performed in the second place.more » 407 refs., 4 figs., 2 tabs.« less

Reliability and responsiveness of the Self-Efficacy in Assessing, Training and Spotting wheelchair skills (SEATS) outcome measure.

PubMed

Rushton, Paula W; Smith, Emma M; Miller, William C; Kirby, R Lee; Daoust, Geneviève

2018-01-31

The aim of this study was to evaluate the internal consistency, test-retest reliability and responsiveness of the Self-Efficacy in Assessing, Training and Spotting manual wheelchair skills (SEATS-M) and Self-Efficacy in Assessing, Training and Spotting power wheelchair skills (SEATS-P). A 2-week test-retest design was used with a convenience sample of occupational and physical therapists who worked at a provincial rehabilitation centre (inpatient and outpatient services). Sixteen participants completed the SEATS-M and 18 participants completed the SEATS-P. For the SEATS-M assessment, training, spotting and documentation sections, Cronbach's alpha coefficients ranged from 0.90 to 0.97, the 2-week intraclass correlation coefficients (ICC 1,1 ) ranged from 0.81 to 0.95, the standard error of measurements (SEM) ranged from 5.06 to 8.70 and the smallest real differences (SRD) ranged from 6.24 to 8.18. For the SEATS-P assessment, training, spotting and documentation sections, Cronbach's alpha coefficients ranged from 0.83 to 0.92, the ICCs ranged from 0.72 to 0.86, the SEMs ranged from 4.54 to 8.91 and the SRDs ranged from 5.90 to 8.27. There is preliminary evidence that both the SEATS-M and the SEATS-P have high internal consistency, good test-retest reliability and support for responsiveness. These tools can be used in evaluating clinician self-efficacy with assessing, training, spotting and documenting wheelchair skills included on the Wheelchair Skills Test. Implications for Rehabilitation There is preliminary evidence that the SEATS-M and SEATS-P are reliable and responsive outcome measures that can be used to evaluate the self-efficacy of clinicians to administer the Wheelchair Skills Program. Measurement of clinicians' self-efficacy in this area of practice may enable an enhanced understanding of the areas in which clinicians lack self-efficacy, thereby informing the development of improved knowledge translation interventions.

Reliability assessments in qualitative health promotion research.

PubMed

Cook, Kay E

2012-03-01

This article contributes to the debate about the use of reliability assessments in qualitative research in general, and health promotion research in particular. In this article, I examine the use of reliability assessments in qualitative health promotion research in response to health promotion researchers' commonly held misconception that reliability assessments improve the rigor of qualitative research. All qualitative articles published in the journal Health Promotion International from 2003 to 2009 employing reliability assessments were examined. In total, 31.3% (20/64) articles employed some form of reliability assessment. The use of reliability assessments increased over the study period, ranging from <20% in 2003/2004 to 50% and above in 2008/2009, while at the same time the total number of qualitative articles decreased. The articles were then classified into four types of reliability assessments, including the verification of thematic codes, the use of inter-rater reliability statistics, congruence in team coding and congruence in coding across sites. The merits of each type were discussed, with the subsequent discussion focusing on the deductive nature of reliable thematic coding, the limited depth of immediately verifiable data and the usefulness of such studies to health promotion and the advancement of the qualitative paradigm.

General Monte Carlo reliability simulation code including common mode failures and HARP fault/error-handling

NASA Technical Reports Server (NTRS)

Platt, M. E.; Lewis, E. E.; Boehm, F.

1991-01-01

A Monte Carlo Fortran computer program was developed that uses two variance reduction techniques for computing system reliability applicable to solving very large highly reliable fault-tolerant systems. The program is consistent with the hybrid automated reliability predictor (HARP) code which employs behavioral decomposition and complex fault-error handling models. This new capability is called MC-HARP which efficiently solves reliability models with non-constant failures rates (Weibull). Common mode failure modeling is also a specialty.

Invited review: Animal-based indicators for on-farm welfare assessment for dairy goats.

PubMed

Battini, M; Vieira, A; Barbieri, S; Ajuda, I; Stilwell, G; Mattiello, S

2014-11-01

This paper reviews animal-based welfare indicators to develop a valid, reliable, and feasible on-farm welfare assessment protocol for dairy goats. The indicators were considered in the light of the 4 accepted principles (good feeding, good housing, good health, appropriate behavior) subdivided into 12 criteria developed by the European Welfare Quality program. We will only examine the practical indicators to be used on-farm, excluding those requiring the use of specific instruments or laboratory analysis and those that are recorded at the slaughterhouse. Body condition score, hair coat condition, and queuing at the feed barrier or at the drinker seem the most promising indicators for the assessment of the "good feeding" principle. As to "good housing," some indicators were considered promising for assessing "comfort around resting" (e.g., resting in contact with a wall) or "thermal comfort" (e.g., panting score for the detection of heat stress and shivering score for the detection of cold stress). Several indicators related to "good health," such as lameness, claw overgrowth, presence of external abscesses, and hair coat condition, were identified. As to the "appropriate behavior" principle, different criteria have been identified: agonistic behavior is largely used as the "expression of social behavior" criterion, but it is often not feasible for on-farm assessment. Latency to first contact and the avoidance distance test can be used as criteria for assessing the quality of the human-animal relationship. Qualitative behavior assessment seems to be a promising indicator for addressing the "positive emotional state" criterion. Promising indicators were identified for most of the considered criteria; however, no valid indicator has been identified for "expression of other behaviors." Interobserver reliability has rarely been assessed and warrants further attention; in contrast, short-term intraobserver reliability is frequently assessed and some studies consider mid- and long-term reliability. The feasibility of most of the reviewed indicators in commercial farms still needs to be carefully evaluated, as several studies were performed under experimental conditions. Our review highlights some aspects of goat welfare that have been widely studied, but some indicators need to be investigated further and drafted before being included in a valid, reliable, and feasible welfare assessment protocol. The indicators selected and examined may be an invaluable starting point for the development of an on-farm welfare assessment protocol for dairy goats. Copyright © 2014 American Dairy Science Association. Published by Elsevier Inc. All rights reserved.

Automation of reliability evaluation procedures through CARE - The computer-aided reliability estimation program.

NASA Technical Reports Server (NTRS)

Mathur, F. P.

1972-01-01

Description of an on-line interactive computer program called CARE (Computer-Aided Reliability Estimation) which can model self-repair and fault-tolerant organizations and perform certain other functions. Essentially CARE consists of a repository of mathematical equations defining the various basic redundancy schemes. These equations, under program control, are then interrelated to generate the desired mathematical model to fit the architecture of the system under evaluation. The mathematical model is then supplied with ground instances of its variables and is then evaluated to generate values for the reliability-theoretic functions applied to the model.

Fundamentals of endoscopic surgery: creation and validation of the hands-on test.

PubMed

Vassiliou, Melina C; Dunkin, Brian J; Fried, Gerald M; Mellinger, John D; Trus, Thadeus; Kaneva, Pepa; Lyons, Calvin; Korndorffer, James R; Ujiki, Michael; Velanovich, Vic; Kochman, Michael L; Tsuda, Shawn; Martinez, Jose; Scott, Daniel J; Korus, Gary; Park, Adrian; Marks, Jeffrey M

2014-03-01

The Fundamentals of Endoscopic Surgery™ (FES) program consists of online materials and didactic and skills-based tests. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study was to evaluate the reliability and validity of the hands-on component of the FES examination, and to establish the pass score. Expert endoscopists identified the critical skill set required for flexible endoscopy. They were then modeled in a virtual reality simulator (GI Mentor™ II, Simbionix™ Ltd., Airport City, Israel) to create five tasks and metrics. Scores were designed to measure both speed and precision. Validity evidence was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists [GIs]). Internal consistency of each test task was assessed using Cronbach's alpha. Test-retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting groups methodology and use of receiver operating characteristic curves. A total of 160 participants (17 % GIs) performed the simulator test. Scores on the five tasks showed good internal consistency reliability and all had significant correlations with endoscopic experience. Total FES scores correlated 0.73, with participants' level of endoscopic experience providing evidence of their validity, and their internal consistency reliability (Cronbach's alpha) was 0.82. Test-retest reliability was assessed in 11 participants, and the intraclass correlation was 0.85. The passing score was determined and is estimated to have a sensitivity (true positive rate) of 0.81 and a 1-specificity (false positive rate) of 0.21. The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high-stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.

Education Research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination

PubMed Central

Schuh, L A.; London, Z; Neel, R; Brock, C; Kissela, B M.; Schultz, L; Gelb, D J.

2009-01-01

Objective: The American Board of Psychiatry and Neurology (ABPN) has recently replaced the traditional, centralized oral examination with the locally administered Neurology Clinical Skills Examination (NEX). The ABPN postulated the experience with the NEX would be similar to the Mini-Clinical Evaluation Exercise, a reliable and valid assessment tool. The reliability and validity of the NEX has not been established. Methods: NEX encounters were videotaped at 4 neurology programs. Local faculty and ABPN examiners graded the encounters using 2 different evaluation forms: an ABPN form and one with a contracted rating scale. Some NEX encounters were purposely failed by residents. Cohen’s kappa and intraclass correlation coefficients (ICC) were calculated for local vs ABPN examiners. Results: Ninety-eight videotaped NEX encounters of 32 residents were evaluated by 20 local faculty evaluators and 18 ABPN examiners. The interrater reliability for a determination of pass vs fail for each encounter was poor (kappa 0.32; 95% confidence interval [CI] = 0.11, 0.53). ICC between local faculty and ABPN examiners for each performance rating on the ABPN NEX form was poor to moderate (ICC range 0.14-0.44), and did not improve with the contracted rating form (ICC range 0.09-0.36). ABPN examiners were more likely than local examiners to fail residents. Conclusions: There is poor interrater reliability between local faculty and American Board of Psychiatry and Neurology examiners. A bias was detected for favorable assessment locally, which is concerning for the validity of the examination. Further study is needed to assess whether training can improve interrater reliability and offset bias. GLOSSARY ABIM = American Board of Internal Medicine; ABPN = American Board of Psychiatry and Neurology; CI = confidence interval; HFH = Henry Ford Hospital; ICC = intraclass correlation coefficients; IM = internal medicine; mini-CEX = Mini-Clinical Evaluation Exercise; NEX = Neurology Clinical Skills Examination; RITE = residency inservice training examination; UC = University of Cincinnati; UM = University of Michigan; USF = University of South Florida. PMID:19605769

Reliability and validity of assessing subspecialty level of faculty anesthesiologists' supervision of anesthesiology residents.

PubMed

De Oliveira, Gildasio S; Dexter, Franklin; Bialek, Jane M; McCarthy, Robert J

2015-01-01

Supervision of anesthesiology residents is a major responsibility of faculty (academic) anesthesiologists. Supervision can be evaluated daily for individual anesthesiologists using a 9-question instrument. Faculty anesthesiologists with lesser individual scores contribute to lesser departmental (global) scores. Low (<3, "frequent") department-wide evaluations of supervision are associated with more mistakes with negative consequences to patients. With the long-term aim for residency programs to be evaluated partly based on the quality of their resident supervision, we assessed the 9-item instrument's reliability and validity when used to compare anesthesia programs' rotations nationwide. One thousand five hundred residents in the American Society of Anesthesiologists' directory of anesthesia trainees were randomly selected to be participants. Residents were contacted via e-mail and requested to complete a Web-based survey. Nonrespondents were mailed a paper version of the survey. Internal consistency of the supervision scale was excellent, with Cronbach's α = 0.909 (95% CI, 0.896-0.922, n = 641 respondents). Discriminant validity was found based on absence of rank correlation of supervision score with characteristics of the respondents and programs (all P > 0.10): age, hours worked per week, female, year of anesthesia training, weeks in the current rotation, sequence of survey response, size of residency class, and number of survey respondents from the current rotation and program. Convergent validity was found based on significant positive correlation between supervision score and variables related to safety culture (all P < 0.0001): "Overall perceptions of patient safety," "Teamwork within units," "Nonpunitive response to errors," "Handoffs and transitions," "Feedback and communication about error," "Communication openness," and rotation's "overall grade on patient safety." Convergent validity was found also based on significant negative correlation with variables related to the individual resident's burnout (all P < 0.0001): "I feel burnout from my work," "I have become more callous toward people since I took this job," and numbers of "errors with potential negative consequences to patients [that you have] made and/or witnessed." Usefulness was shown by supervision being predicted by the same 1 variable for each of 3 regression tree criteria: "Teamwork within [the rotation]" (e.g., "When one area in this rotation gets busy, others help out"). Evaluation of the overall quality of supervision of residents by faculty anesthesiologists depends on the reliability and validity of the instrument. Our results show that the 9-item de Oliveira Filho et al. supervision scale can be applied for overall (department, rotation) assessment of anesthesia training programs.

Parts and Components Reliability Assessment: A Cost Effective Approach

NASA Technical Reports Server (NTRS)

Lee, Lydia

2009-01-01

System reliability assessment is a methodology which incorporates reliability analyses performed at parts and components level such as Reliability Prediction, Failure Modes and Effects Analysis (FMEA) and Fault Tree Analysis (FTA) to assess risks, perform design tradeoffs, and therefore, to ensure effective productivity and/or mission success. The system reliability is used to optimize the product design to accommodate today?s mandated budget, manpower, and schedule constraints. Stand ard based reliability assessment is an effective approach consisting of reliability predictions together with other reliability analyses for electronic, electrical, and electro-mechanical (EEE) complex parts and components of large systems based on failure rate estimates published by the United States (U.S.) military or commercial standards and handbooks. Many of these standards are globally accepted and recognized. The reliability assessment is especially useful during the initial stages when the system design is still in the development and hard failure data is not yet available or manufacturers are not contractually obliged by their customers to publish the reliability estimates/predictions for their parts and components. This paper presents a methodology to assess system reliability using parts and components reliability estimates to ensure effective productivity and/or mission success in an efficient manner, low cost, and tight schedule.

A job-satisfaction measure for internal medicine residency program directors.

PubMed

Beasley, B W; Kern, D E; Howard, D M; Kolodner, K

1999-03-01

To develop a job-satisfaction measure that encompasses the multifaceted job of internal medicine residency program directors. Questions were devised to measure program directors satisfaction with various facets of their jobs. In 1996, the authors surveyed all non-military internal medicine program directors in the United States. Of the program directors surveyed, 301 (78%) responded. More respondents than non-respondents held the title of department chairperson in addition to the title of program director (22% vs 7%). Factor analysis and correlation analysis yielded a multifaceted measure (termed PD-Sat) composed of 20 questions and six facets (work with residents, colleague relationships, resources, patient care, pay, and promotion) that made sense based on literature review and discussions with program directors (face validity). The PD-Sat had good internal reliability (Cronbach's alpha = .88), as had each of its six facets (Cronbach's alphas = .60-.90). The six facets correlated modestly with one another (Pearson's r2 = .12-.67), suggesting they were measuring different aspects of a common concept. The PD-Sat correlated significantly with an established four-question global job-satisfaction scale used in previous studies (Pearson's r2 = .33) demonstrating concurrent validity. Scores on the PD-Sat predicted whether program directors were considering, seeking, or making a job change (predictive validity). The PD-Sat performed comparably well in subsets of program directors who were and were not department chairs, suggesting that it might be applicable to different populations of program directors. The authors have developed a new facet-specific job-satisfaction measure that is reliable and valid for assessing the job satisfaction of internal medicine program directors. Because job descriptions for program directors in other specialties are similar, it may also be useful in these populations.

Do respiratory therapists receive training and education in smoking cessation? A national study of post-secondary training programs.

PubMed

Jordan, Timothy R; Khubchandani, Jagdish; Wiblishauser, Michael; Glassman, Tavis; Thompson, Amy

2011-10-01

To assess the tobacco-related education provided by post-secondary respiratory therapy training programs in the United States. A cross-sectional research design was used to survey the entire population of program directors of post-secondary, respiratory therapy training programs in the United States. A valid and reliable questionnaire was developed and mailed using a 2-wave mailing technique (73% return rate). Internal reliability coefficients (Cronbach alpha) for the various components of the questionnaire ranged from 0.78 to 0.91. More than half of programs (56%) offered no teaching on the 5R's. Nearly half (47%) offered no teaching on the 5A's. Of the 13 tobacco-related topics listed in the basic science and clinical science sections of the questionnaire, only one topic (i.e., diseases linked to tobacco use) received 3h or more of instruction by approximately a third of programs (35.8%). The majority of programs (>90%) spent no time teaching students about the socio-political aspects of tobacco use cessation. Moreover, 41% of programs did not formally evaluate students' competence in providing smoking cessation counseling to patients. Tobacco-related education is a very minor component of the education and training received by respiratory therapy students in the United States. Respiratory therapy training programs in the United States have great potential to strengthen the tobacco-related education that they provide to students. Practicing respiratory therapists would likely benefit from continuing medical education focused on how to use evidence-based smoking cessation counseling techniques with patients. Copyright © 2010 Elsevier Ireland Ltd. All rights reserved.

SPSS and SAS programs for generalizability theory analyses.

PubMed

Mushquash, Christopher; O'Connor, Brian P

2006-08-01

The identification and reduction of measurement errors is a major challenge in psychological testing. Most investigators rely solely on classical test theory for assessing reliability, whereas most experts have long recommended using generalizability theory instead. One reason for the common neglect of generalizability theory is the absence of analytic facilities for this purpose in popular statistical software packages. This article provides a brief introduction to generalizability theory, describes easy to use SPSS, SAS, and MATLAB programs for conducting the recommended analyses, and provides an illustrative example, using data (N = 329) for the Rosenberg Self-Esteem Scale. Program output includes variance components, relative and absolute errors and generalizability coefficients, coefficients for D studies, and graphs of D study results.

Toronto Bariatric Interprofessional Psychosocial Assessment Suitability Scale: Evaluating A New Clinical Assessment Tool for Bariatric Surgery Candidates.

PubMed

Thiara, Gurneet; Yanofksy, Richard; Abdul-Kader, Sayed; Santiago, Vincent A; Cassin, Stephanie; Okrainec, Allan; Jackson, Timothy; Hawa, Raed; Sockalingam, Sanjeev

2016-01-01

Patients who are referred for possible bariatric surgery (BS) intervention undergo a series of assessments conducted by an interdisciplinary health care team to determine suitability for surgery. Herein, we report the initial validation and reliability studies of the Bariatric Interprofessional Psychosocial Assessment Suitability Scale (BIPASS) and its relationship to interdisciplinary psychosocial assessment practices for BS. This study was conducted at the Toronto Western Hospital, a Level 1A BS center of excellence accredited by the American College of Surgeons. Phase I: a total of 4 blinded raters applied the BIPASS to 31 randomly selected BS cases referred to our program to establish interrater reliability. Phase II: in all, 3 raters with clinical experience in bariatric psychosocial care applied the BIPASS to 54 randomly selected BS cases. In total, 46 of 54 (85.1%) patients were women. The median age of all patient cases was 49 years (range: 21-74). Raters׳ BIPASS scores ranged from 4-52 (median = 19.24, standard deviation =10.38). BIPASS scores were highly predictive of the BS psychosocial outcome (area under curve = 0.915; 95% CI: 0.844-0.985; p < 0.001). A BIPASS score of ≥16 was chosen as the cutoff score for further clinical assessment before proceeding with surgical evaluation based on a receiver operating characteristic curve analysis (sensitivity = 0.839; specificity = 0.783). The instrument has very good interrater reliability (Pearson correlation coefficient = 0.847) even among novice raters. The findings show that the BIPASS is a comprehensive screening tool in the psychosocial assessment of BS candidates, which standardizes the evaluation process and systematically identify at-risk patients for negative outcomes after BS. Copyright © 2016 The Academy of Psychosomatic Medicine. Published by Elsevier Inc. All rights reserved.

Guiding dental student learning and assessing performance in critical thinking with analysis of emerging strategies.

PubMed

Johnsen, David C; Lipp, Mitchell J; Finkelstein, Michael W; Cunningham-Ford, Marsha A

2012-12-01

Patient-centered care involves an inseparable set of knowledge, abilities, and professional traits on the part of the health care provider. For practical reasons, health professions education is segmented into disciplines or domains like knowledge, technical skills, and critical thinking, and the culture of dental education is weighted toward knowledge and technical skills. Critical thinking, however, has become a growing presence in dental curricula. To guide student learning and assess performance in critical thinking, guidelines have been developed over the past several decades in the educational literature. Prominent among these guidelines are the following: engage the student in multiple situations/exercises reflecting critical thinking; for each exercise, emulate the intended activity for validity; gain agreement of faculty members across disciplines and curriculum years on the learning construct, application, and performance assessment protocol for reliability; and use the same instrument to guide learning and assess performance. The purposes of this article are 1) to offer a set of concepts from the education literature potentially helpful to guide program design or corroborate existing programs in dental education; 2) to offer an implementation model consolidating these concepts as a guide for program design and execution; 3) to cite specific examples of exercises and programs in critical thinking in the dental education literature analyzed against these concepts; and 4) to discuss opportunities and challenges in guiding student learning and assessing performance in critical thinking for dentistry.

Application of SAW method for multiple-criteria comparative analysis of the reliability of heat supply organizations

NASA Astrophysics Data System (ADS)

Akhmetova, I. G.; Chichirova, N. D.

2016-12-01

Heat supply is the most energy-consuming sector of the economy. Approximately 30% of all used primary fuel-and-energy resources is spent on municipal heat-supply needs. One of the key indicators of activity of heat-supply organizations is the reliability of an energy facility. The reliability index of a heat supply organization is of interest to potential investors for assessing risks when investing in projects. The reliability indices established by the federal legislation are actually reduced to a single numerical factor, which depends on the number of heat-supply outages in connection with disturbances in operation of heat networks and the volume of their resource recovery in the calculation year. This factor is rather subjective and may change in a wide range during several years. A technique is proposed for evaluating the reliability of heat-supply organizations with the use of the simple additive weighting (SAW) method. The technique for integrated-index determination satisfies the following conditions: the reliability level of the evaluated heat-supply system is represented maximum fully and objectively; the information used for the reliability-index evaluation is easily available (is located on the Internet in accordance with demands of data-disclosure standards). For reliability estimation of heat-supply organizations, the following indicators were selected: the wear of equipment of thermal energy sources, the wear of heat networks, the number of outages of supply of thermal energy (heat carrier due to technological disturbances on heat networks per 1 km of heat networks), the number of outages of supply of thermal energy (heat carrier due to technologic disturbances on thermal energy sources per 1 Gcal/h of installed power), the share of expenditures in the cost of thermal energy aimed at recovery of the resource (renewal of fixed assets), coefficient of renewal of fixed assets, and a coefficient of fixed asset retirement. A versatile program is developed and the analysis of heat-supply organizations is performed by the example of the Republic of Tatarstan. The assessment system is based on construction of comparative ratings of heat-supply organizations. A rating is the assessment of reliability of the organization, is characterized by a numerical value, and makes it possible to compare organizations engaged in the same kind of activity between each other.

Summary of NASA Aerospace Flight Battery Systems Program activities

NASA Technical Reports Server (NTRS)

Manzo, Michelle; Odonnell, Patricia

1994-01-01

A summary of NASA Aerospace Flight Battery Systems Program Activities is presented. The NASA Aerospace Flight Battery Systems Program represents a unified NASA wide effort with the overall objective of providing NASA with the policy and posture which will increase the safety, performance, and reliability of space power systems. The specific objectives of the program are to: enhance cell/battery safety and reliability; maintain current battery technology; increase fundamental understanding of primary and secondary cells; provide a means to bring forth advanced technology for flight use; assist flight programs in minimizing battery technology related flight risks; and ensure that safe, reliable batteries are available for NASA's future missions.

Structural Analyses of Stirling Power Convertor Heater Head for Long-Term Reliability, Durability, and Performance

NASA Technical Reports Server (NTRS)

Halford, Gary R.; Shah, Ashwin; Arya, Vinod K.; Krause, David L.; Bartolotta, Paul A.

2002-01-01

Deep-space missions require onboard electric power systems with reliable design lifetimes of up to 10 yr and beyond. A high-efficiency Stirling radioisotope power system is a likely candidate for future deep-space missions and Mars rover applications. To ensure ample durability, the structurally critical heater head of the Stirling power convertor has undergone extensive computational analyses of operating temperatures (up to 650 C), stresses, and creep resistance of the thin-walled Inconel 718 bill of material. Durability predictions are presented in terms of the probability of survival. A benchmark structural testing program has commenced to support the analyses. This report presents the current status of durability assessments.

The brief multidimensional students' life satisfaction scale-college version.

PubMed

Zullig, Keith J; Huebner, E Scott; Patton, Jon M; Murray, Karen A

2009-01-01

To investigate the psychometric properties of the BMSLSS-College among 723 college students. Internal consistency estimates explored scale reliability, factor analysis explored construct validity, and known-groups validity was assessed using the National College Youth Risk Behavior Survey and Harvard School of Public Health College Alcohol Study. Criterion-related validity was explored through analyses with the CDC's health-related quality of life scale and a social isolation scale. Acceptable internal consistency reliability, construct, known-groups, and criterion-related validity were established. Findings offer preliminary support for the BMSLSS-C; it could be useful in large-scale research studies, applied screening contexts, and for program evaluation purposes toward achieving Healthy People 2010 objectives.

The implementation and use of Ada on distributed systems with high reliability requirements

NASA Technical Reports Server (NTRS)

Knight, J. C.

1987-01-01

Performance analysis was begin on the Ada implementations. The goal is to supply the system designer with tools that will allow a rational decision to be made about whether a particular implementation can support a given application early in the design cycle. Primary activities were: analysis of the original approach to recovery in distributed Ada programs using the Advanced Transport Operating System (ATOPS) example; review and assessment of the original approach which was found to be capable of improvement; preparation and presentation of a paper at the 1987 Washington DC Ada Symposium; development of a refined approach to recovery that is presently being applied to the ATOPS example; and design and development of a performance assessment scheme for Ada programs based on a flexible user-driven benchmarking system.

Interrater Reliability and Discriminative Validity of the Structural Elements of the Ayres Sensory Integration® Fidelity Measure©

PubMed Central

Roley, Susanne Smith; Mailloux, Zoe; Parham, L. Diane; Koomar, Jane; Schaaf, Roseann C.; Van Jaarsveld, Annamarie; Cohn, Ellen

2014-01-01

This study examined the reliability and validity of the structural section of the Ayres Sensory Integration® Fidelity Measure© (ASIFM), which provides a method for monitoring the extent to which an intervention was implemented as conceptualized in studies of occupational therapy using sensory integration intervention methods (OT–SI). We examined the structural elements of the measure, including content of assessment reports, availability of specific equipment and adequate space, safety monitoring, and integration of communication with parents and other team members, such as collaborative goal setting with parents or family and teacher education, into the intervention program. Analysis of self-report ratings by 259 occupational therapists from 185 different facilities indicated that the structural section of the ASIFM has acceptable interrater reliability (r ≥ .82) and significantly differentiates between settings in which therapists reportedly do and do not practice OT–SI (p < .001). PMID:25184462

Developing an Objective Structured Assessment of Technical Skills for Laparoscopic Suturing and Intracorporeal Knot Tying.

PubMed

Chang, Olivia H; King, Louise P; Modest, Anna M; Hur, Hye-Chun

2016-01-01

To develop a teaching and assessment tool for laparoscopic suturing and intracorporeal knot tying. We designed an Objective Structured Assessment of Technical Skills (OSATS) tool that includes a procedure-specific checklist (PSC) and global rating scale (GRS) to assess laparoscopic suturing and intracorporeal knot-tying performance. Obstetrics and Gynecology residents at our institution were videotaped while performing a laparoscopic suturing and intracorporeal knot-tying task at a surgical simulation workshop. A total of 2 expert reviewers assessed resident performance using the OSATS tool during live performance and 1 month later using the videotaped recordings. OSATS scores were analyzed using the Wilcoxon rank-sum test. Data are presented as median scores (interquartile range [IQR]). Intrarater and interrater reliabilities were assessed using a Spearman correlation and are presented as an r correlation coefficient and p value. An r ≥ 0.8 was considered as a high correlation. After testing, we received feedback from residents and faculty to improve the OSATS tool as part of an iterative design process. In all, 14 of 21 residents (66.7%) completed the study, with 9 junior residents and 5 senior residents. Junior residents had a lower score on the PSC than senior residents did; however, this was not statistically significant (median = 6.0 [IQR: 4.0-10.0] and median = 13.0 [IQR: 10.0-13.0]; p = 0.09). There was excellent intrarater reliability with our OSATS tool (for PSC component, r = 0.88 for Rater 1 and 0.93 for Rater 2, both p < 0.0001; for GRS component, r = 0.85 for Rater 1 and 0.88 for Rater 2, both p ≤ 0.0002). The PSC also has high interrater reliability during live evaluation (r = 0.92; p < 0.0001), and during the videotape scoring with r = 0.77 (p = 0.001). Our OSATS tool may be a useful assessment and teaching tool for laparoscopic suturing and intracorporeal knot-tying skills. Overall, good intrarater reliability was demonstrated, suggesting that this tool may be useful for longitudinal assessment of surgical skills. Copyright © 2015 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

Validation of a novel venous duplex ultrasound objective structured assessment of technical skills for the assessment of venous reflux.

PubMed

Jaffer, Usman; Normahani, Pasha; Lackenby, Kimberly; Aslam, Mohammed; Standfield, Nigel J

2015-01-01

Duplex ultrasound measurement of reflux time is central to the diagnosis of venous incompetence. We have developed an assessment tool for Duplex measurement of venous reflux for both simulator and patient-based training. A novel assessment tool, Venous Duplex Ultrasound Assessment of Technical Skills (V-DUOSATS), was developed. A modified DUOSATS was used for simulator training. Participants of varying skill level were invited to viewed an instructional video and were allowed ample time to familiarize with the Duplex equipment. Attempts made by the participants were recorded and independently assessed by 3 expert assessors and 5 novice assessors using the modified V-DUOSATS. "Global" assessment was also done by expert assessors on a 4-point Likert scale. Content, construct, and concurrent validities as well as reliability were evaluated. Content and construct validity as well as reliability were demonstrated. Receiver operator characteristic analysis-established cut points of 19/22 and 21/30 were most appropriate for simulator and patient-based assessment, respectively. We have validated a novel assessment tool for Duplex venous reflux measurement. Further work is required to establish transference validity of simulator training to improve skill in scanning patients. We have developed and validated V-DUOSATS for simulator training. Copyright © 2015 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

Development of a Digital Image Measurement System

NASA Technical Reports Server (NTRS)

2004-01-01

An unexpected tragedy took place on April 28, 1988, when the roof of an Aloha Airlines 737 aircraft ripped open at 24,000 feet, killing a flight attendant and injuring eight people. The in-flight structural failure of Aloha Flight 243 s 19-year-old aircraft prompted NASA Langley Research Center to join with colleagues at the U.S. Federal Aviation Administration and the U.S. Air Force to initiate the Nation's first Aging Aircraft Research program. One of the program's essential goals was to develop reliable, predictive methods for assessing the residual strength of aging aerospace structures.

49 CFR Appendix E to Part 238 - General Principles of Reliability-Based Maintenance Programs

Code of Federal Regulations, 2010 CFR

2010-10-01

... 49 Transportation 4 2010-10-01 2010-10-01 false General Principles of Reliability-Based... STANDARDS Pt. 238, App. E Appendix E to Part 238—General Principles of Reliability-Based Maintenance... maintenance programs are based on the following general principles. A failure is an unsatisfactory condition...

Use of the Environment and Policy Evaluation and Observation as a Self-Report Instrument (EPAO-SR) to measure nutrition and physical activity environments in child care settings: validity and reliability evidence.

PubMed

Ward, Dianne S; Mazzucca, Stephanie; McWilliams, Christina; Hales, Derek

2015-09-26

Early care and education (ECE) centers are important settings influencing young children's diet and physical activity (PA) behaviors. To better understand their impact on diet and PA behaviors as well as to evaluate public health programs aimed at ECE settings, we developed and tested the Environment and Policy Assessment and Observation - Self-Report (EPAO-SR), a self-administered version of the previously validated, researcher-administered EPAO. Development of the EPAO-SR instrument included modification of items from the EPAO, community advisory group and expert review, and cognitive interviews with center directors and classroom teachers. Reliability and validity data were collected across 4 days in 3-5 year old classrooms in 50 ECE centers in North Carolina. Center teachers and directors completed relevant portions of the EPAO-SR on multiple days according to a standardized protocol, and trained data collectors completed the EPAO for 4 days in the centers. Reliability and validity statistics calculated included percent agreement, kappa, correlation coefficients, coefficients of variation, deviations, mean differences, and intraclass correlation coefficients (ICC), depending on the response option of the item. Data demonstrated a range of reliability and validity evidence for the EPAO-SR instrument. Reporting from directors and classroom teachers was consistent and similar to the observational data. Items that produced strongest reliability and validity estimates included beverages served, outside time, and physical activity equipment, while items such as whole grains served and amount of teacher-led PA had lower reliability (observation and self-report) and validity estimates. To overcome lower reliability and validity estimates, some items need administration on multiple days. This study demonstrated appropriate reliability and validity evidence for use of the EPAO-SR in the field. The self-administered EPAO-SR is an advancement of the measurement of ECE settings and can be used by researchers and practitioners to assess the nutrition and physical activity environments of ECE settings.

Reliability and validity of the Brief Pain Inventory in individuals with chronic obstructive pulmonary disease.

PubMed

Chen, Y-W; HajGhanbari, B; Road, J D; Coxson, H O; Camp, P G; Reid, W D

2018-06-08

Pain is prevalent in chronic obstructive pulmonary disease (COPD) and the Brief Pain Inventory (BPI) appears to be a feasible questionnaire to assess this symptom. However, the reliability and validity of the BPI have not been determined in individuals with COPD. This study aimed to determine the internal consistency, test-retest reliability and validity (construct, convergent, divergent and discriminant) of the BPI in individuals with COPD. In order to examine the test-retest reliability, individuals with COPD were recruited from pulmonary rehabilitation programmes to complete the BPI twice 1 week apart. In order to investigate validity, de-identified data was retrieved from two previous studies, including forced expiratory volume in 1-s, age, sex and data from four questionnaires: the BPI, short-form McGill Pain Questionnaire (SF-MPQ), 36-Item Short Form Survey (SF-36) and Community Health Activities Model Program for Seniors (CHAMPS) questionnaire. In total, 123 participants were included in the analyses (eligible data were retrieved from 86 participants and additional 37 participants were recruited). The BPI demonstrated excellent internal consistency and test-retest reliability. It also showed convergent validity with the SF-MPQ and divergent validity with the SF-36. The factor analysis yielded two factors of the BPI, which demonstrated that the two domains of the BPI measure the intended constructs. The BPI can also discriminate pain levels among COPD patients with varied levels of quality of life (SF-36) and physical activity (CHAMPS). The BPI is a reliable and valid pain questionnaire that can be used to evaluate pain in COPD. This study formally established the reliability and validity of the BPI in individuals with COPD, which have not been determined in this patient group. The results of this study provide strong evidence that assessment results from this pain questionnaire are reliable and valid. © 2018 European Pain Federation - EFIC®.

[Validation of the Spanish parent satisfaction questionnaire with neonatal hearing screening programs].

PubMed

Núñez-Batalla, Faustino; Antuña-León, Eva; González-Trelles, Teresa; Carro-Fernández, Pilar

2009-01-01

Although measuring parent satisfaction has been recommended as one of the important outcome measures in assessing the effectiveness of neonatal hearing screening programs, there are few published studies investigating this issue. To validate the Spanish version of the Parent Satisfaction Questionnaire with Neonatal Hearing Screening Program (PSQ-NHSP). 112 parents whose children had received hearing screening participated in this study. High levels of satisfaction were reported with more than 90% of parents satisfied with all aspects of the program. The psychometric properties of the Spanish version of the PSQ-NHSP were analyzed and demonstrated good internal consistency (alpha=0.75). Construct validity was indicated by a significant positive relationship between overall satisfaction and the three specific dimensions in the questionnaire. The development of a valid and reliable parent satisfaction questionnaire is important for improving hearing screening programs.

Computing Reliabilities Of Ceramic Components Subject To Fracture

NASA Technical Reports Server (NTRS)

Nemeth, N. N.; Gyekenyesi, J. P.; Manderscheid, J. M.

1992-01-01

CARES calculates fast-fracture reliability or failure probability of macroscopically isotropic ceramic components. Program uses results from commercial structural-analysis program (MSC/NASTRAN or ANSYS) to evaluate reliability of component in presence of inherent surface- and/or volume-type flaws. Computes measure of reliability by use of finite-element mathematical model applicable to multiple materials in sense model made function of statistical characterizations of many ceramic materials. Reliability analysis uses element stress, temperature, area, and volume outputs, obtained from two-dimensional shell and three-dimensional solid isoparametric or axisymmetric finite elements. Written in FORTRAN 77.

[Validation and reliability study of the parent concerns about surgery questionnaire: What worries parents?

PubMed

Gironés Muriel, Alberto; Campos Segovia, Ana; Ríos Gómez, Patricia

2018-01-01

The study of mediating variables and psychological responses to child surgery involves the evaluation of both the patient and the parents as regards different stressors. To have a reliable and reproducible valid evaluation tool that assesses the level of paternal involvement in relation to different stressors in the setting of surgery. A self-report questionnaire study was completed by 123 subjects of both sexes, subdivided into 2populations, due to their relationship with the hospital setting. The items were determined by a group of experts and analysed using the Lawshe validity index to determine a first validity of content. Subsequently, the reliability of the tool was determined by an item-re-item analysis of the 2sub-populations. A factorial analysis was performed to analyse the construct validity with the maximum likelihood and rotation of varimax type factors. A questionnaire of paternal concern was offered, consisting of 21 items with a Cronbach coefficient of 0.97, giving good precision and stability. The posterior factor analysis gives an adequate validity to the questionnaire, with the determination of 10 common stressors that cover 74.08% of the common and non-common variance of the questionnaire. The proposed questionnaire is reliable, valid and easy-to-apply and is developed to assess the level of paternal concern about the surgery of a child and to be able to apply measures and programs through the prior assessment of these elements. Copyright © 2016 Asociación Española de Pediatría. Publicado por Elsevier España, S.L.U. All rights reserved.

Modern psychometrics for assessing achievement goal orientation: a Rasch analysis.

PubMed

Muis, Krista R; Winne, Philip H; Edwards, Ordene V

2009-09-01

A program of research is needed that assesses the psychometric properties of instruments designed to quantify students' achievement goal orientations to clarify inconsistencies across previous studies and to provide a stronger basis for future research. We conducted traditional psychometric and modern Rasch-model analyses of the Achievement Goals Questionnaire (AGQ, Elliot & McGregor, 2001) and the Patterns of Adaptive Learning Scale (PALS, Midgley et al., 2000) to provide an in-depth analysis of the two most popular instruments in educational psychology. For Study 1, 217 undergraduate students enrolled in educational psychology courses participated. Thirty-four were male and 181 were female (two did not respond). Participants completed the AGQ in the context of their educational psychology class. For Study 2, 126 undergraduate students enrolled in educational psychology courses participated. Thirty were male and 95 were female (one did not respond). Participants completed the PALS in the context of their educational psychology class. Traditional psychometric assessments of the AGQ and PALS replicated previous studies. For both, reliability estimates ranged from good to very good for raw subscale scores and fit for the models of goal orientations were good. Based on traditional psychometrics, the AGQ and PALS are valid and reliable indicators of achievement goals. Rasch analyses revealed that estimates of reliability for items were very good but respondent ability estimates varied from poor to good for both the AGQ and PALS. These findings indicate that items validly and reliably reflect a group's aggregate goal orientation, but using either instrument to characterize an individual's goal orientation is hazardous.

Test effectiveness study report: An analytical study of system test effectiveness and reliability growth of three commercial spacecraft programs

NASA Technical Reports Server (NTRS)

Feldstein, J. F.

1977-01-01

Failure data from 16 commercial spacecraft were analyzed to evaluate failure trends, reliability growth, and effectiveness of tests. It was shown that the test programs were highly effective in ensuring a high level of in-orbit reliability. There was only a single catastrophic problem in 44 years of in-orbit operation on 12 spacecraft. The results also indicate that in-orbit failure rates are highly correlated with unit and systems test failure rates. The data suggest that test effectiveness estimates can be used to guide the content of a test program to ensure that in-orbit reliability goals are achieved.

Morphology delimits more species than molecular genetic clusters of invasive Pilosella.

PubMed

Moffat, Chandra E; Ensing, David J; Gaskin, John F; De Clerck-Floate, Rosemarie A; Pither, Jason

2015-07-01

• Accurate assessments of biodiversity are paramount for understanding ecosystem processes and adaptation to change. Invasive species often contribute substantially to local biodiversity; correctly identifying and distinguishing invaders is thus necessary to assess their potential impacts. We compared the reliability of morphology and molecular sequences to discriminate six putative species of invasive Pilosella hawkweeds (syn. Hieracium, Asteraceae), known for unreliable identifications and historical introgression. We asked (1) which morphological traits dependably discriminate putative species, (2) if genetic clusters supported morphological species, and (3) if novel hybridizations occur in the invaded range.• We assessed 33 morphometric characters for their discriminatory power using the randomForest classifier and, using AFLPs, evaluated genetic clustering with the program structure and subsequently with an AMOVA. The strength of the association between morphological and genotypic dissimilarity was assessed with a Mantel test.• Morphometric analyses delimited six species while genetic analyses defined only four clusters. Specifically, we found (1) eight morphological traits could reliably distinguish species, (2) structure suggested strong genetic differentiation but for only four putative species clusters, and (3) genetic data suggest both novel hybridizations and multiple introductions have occurred.• (1) Traditional floristic techniques may resolve more species than molecular analyses in taxonomic groups subject to introgression. (2) Even within complexes of closely related species, relatively few but highly discerning morphological characters can reliably discriminate species. (3) By clarifying patterns of morphological and genotypic variation of invasive Pilosella, we lay foundations for further ecological study and mitigation. © 2015 Botanical Society of America, Inc.

NASA-Ames workload research program

NASA Technical Reports Server (NTRS)

Hart, Sandra

1988-01-01

Research has been underway for several years to develop valid and reliable measures and predictors of workload as a function of operator state, task requirements, and system resources. Although the initial focus of this research was on aeronautics, the underlying principles and methodologies are equally applicable to space, and provide a set of tools that NASA and its contractors can use to evaluate design alternatives from the perspective of the astronauts. Objectives and approach of the research program are described, as well as the resources used in conducting research and the conceptual framework around which the program evolved. Next, standardized tasks are described, in addition to predictive models and assessment techniques and their application to the space program. Finally, some of the operational applications of these tasks and measures are reviewed.

The development and validation of The Inquiry Science Observation Coding Sheet.

PubMed

Brandon, P R; Taum, A K H; Young, D B; Pottenger, F M

2008-08-01

Evaluation reports increasingly document the degree of program implementation, particularly the extent to which programs adhere to prescribed steps and procedures. Many reports are cursory, however, and few, if any, fully portray the long and winding path taken when developing evaluation instruments, particularly observation instruments. In this article, we describe the development of an observational method for evaluating the degree to which K-12 inquiry science programs are implemented, including the many steps and decisions that occurred during the development, and present evidence for the reliability and validity of the data that we collected with the instrument. The article introduces a method for measuring the adherence of inquiry science implementation and gives evaluators a full picture of what they might expect when developing observation instruments for assessing the degree of program implementation.

The assessment of emergency physicians by a regulatory authority.

PubMed

Lockyer, Jocelyn M; Violato, Claudio; Fidler, Herta

2006-12-01

To determine whether it is possible to develop a feasible, valid, and reliable multisource feedback program (360 degree evaluation) for emergency physicians. Surveys with 16, 20, 30, and 31 items were developed to assess emergency physicians by 25 patients, eight coworkers, eight medical colleagues, and self, respectively, using five-point scales along with an "unable to assess" category. Items addressed key competencies related to communication skills, professionalism, collegiality, and self-management. Data from 187 physicians who identified themselves as emergency physicians were available. The mean number of respondents per physician was 21.6 (SD +/- 3.87) (93%) for patients, 7.6 (SD +/- 0.89) (96%) for coworkers, and 7.7 (SD +/- 0.61) (95%) for medical colleagues, suggesting it was a feasible tool. Only the patient survey had four items with "unable to assess" percentages > or = 15%. The factor analysis indicated there were two factors on the patient questionnaire (communication/professionalism and patient education), two on the coworker survey (communication/collegiality and professionalism), and four on the medical colleague questionnaire (clinical performance, professionalism, self-management, and record management) that accounted for 80.0%, 62.5%, and 71.9% of the variance on the surveys, respectively. The factors were consistent with the intent of the instruments, providing empirical evidence of validity for the instruments. Reliability was established for the instruments (Cronbach's alpha > 0.94) and for each physician (generalizability coefficients were 0.68 for patients, 0.85 for coworkers, and 0.84 for medical colleagues). The psychometric examination of the data suggests that the instruments developed to assess emergency physicians were feasible and provide evidence for validity and reliability.

Flash Memory Reliability: Read, Program, and Erase Latency Versus Endurance Cycling

NASA Technical Reports Server (NTRS)

Heidecker, Jason

2010-01-01

This report documents the efforts and results of the fiscal year (FY) 2010 NASA Electronic Parts and Packaging Program (NEPP) task for nonvolatile memory (NVM) reliability. This year's focus was to measure latency (read, program, and erase) of NAND Flash memories and determine how these parameters drift with erase/program/read endurance cycling.

Assessing physical activity during youth sport: the Observational System for Recording Activity in Children: Youth Sports.

PubMed

Cohen, Alysia; McDonald, Samantha; McIver, Kerry; Pate, Russell; Trost, Stewart

2014-05-01

The purpose of this study was to evaluate the validity and interrater reliability of the Observational System for Recording Activity in Children: Youth Sports (OSRAC:YS). Children (N = 29) participating in a parks and recreation soccer program were observed during regularly scheduled practices. Physical activity (PA) intensity and contextual factors were recorded by momentary time-sampling procedures (10-second observe, 20-second record). Two observers simultaneously observed and recorded children's PA intensity, practice context, social context, coach behavior, and coach proximity. Interrater reliability was based on agreement (Kappa) between the observer's coding for each category, and the Intraclass Correlation Coefficient (ICC) for percent of time spent in MVPA. Validity was assessed by calculating the correlation between OSRAC:YS estimated and objectively measured MVPA. Kappa statistics for each category demonstrated substantial to almost perfect interobserver agreement (Kappa = 0.67-0.93). The ICC for percent time in MVPA was 0.76 (95% C.I. = 0.49-0.90). A significant correlation (r = .73) was observed for MVPA recorded by observation and MVPA measured via accelerometry. The results indicate the OSRAC:YS is a reliable and valid tool for measuring children's PA and contextual factors during a youth soccer practice.

Reliability and Validity of Two Self-report Measures to Assess Sedentary Behavior in Older Adults

PubMed Central

Gennuso, Keith P.; Matthews, Charles E.; Colbert, Lisa H.

2015-01-01

Background The purpose of this study was to examine the reliability and validity of two currently available physical activity surveys for assessing time spent in sedentary behavior (SB) in older adults. Methods Fifty-eight adults (≥65 years) completed the Yale Physical Activity Survey for Older Adults (YPAS) and Community Health Activities Model Program for Seniors (CHAMPS) before and after a 10-day period during which they wore an ActiGraph accelerometer (ACC). Intraclass correlation coefficients (ICC) examined test-retest reliability. Overall percent agreement and a kappa statistic examined YPAS validity. Lin’s concordance correlation, Pearson correlation, and Bland-Altman analysis examined CHAMPS validity. Results Both surveys had moderate test-retest reliability (ICC: YPAS=0.59 (P<0.001), CHAMPS=0.64 (P<0.001)) and significantly underestimated SB time. Agreement between YPAS and ACC was low (κ=−0.0003); however, there was a linear increase (P< 0.01) in ACC-derived SB time across YPAS response categories. There was poor agreement between ACC-derived SB and CHAMPS (Lin’s r=0.005; 95% CI, −0.010 to 0.020), and no linear trend across CHAMPS quartiles (p=0.53). Conclusions Neither of the surveys should be used as the sole measure of SB in a study; though the YPAS has the ability to rank individuals, providing it with some merit for use in correlational SB research. PMID:25110344

Development of a direct observation Measure of Environmental Qualities of Activity Settings.

PubMed

King, Gillian; Rigby, Patty; Batorowicz, Beata; McMain-Klein, Margot; Petrenchik, Theresa; Thompson, Laura; Gibson, Michelle

2014-08-01

The aim of this study was to develop an observer-rated measure of aesthetic, physical, social, and opportunity-related qualities of leisure activity settings for young people (with or without disabilities). Eighty questionnaires were completed by sets of raters who independently rated 22 community/home activity settings. The scales of the 32-item Measure of Environmental Qualities of Activity Settings (MEQAS; Opportunities for Social Activities, Opportunities for Physical Activities, Pleasant Physical Environment, Opportunities for Choice, Opportunities for Personal Growth, and Opportunities to Interact with Adults) were determined using principal components analyses. Test-retest reliability was determined for eight activity settings, rated twice (4-6wk interval) by a trained rater. The factor structure accounted for 80% of the variance. The Kaiser-Meyer-Olkin Measure of Sampling Adequacy was 0.73. Cronbach's alphas for the scales ranged from 0.76 to 0.96, and interrater reliabilities (ICCs) ranged from 0.60 to 0.93. Test-retest reliabilities ranged from 0.70 to 0.90. Results suggest that the MEQAS has a sound factor structure and preliminary evidence of internal consistency, interrater, and test-retest reliability. The MEQAS is the first observer-completed measure of environmental qualities of activity settings. The MEQAS allows researchers to assess comprehensively qualities and affordances of activity settings, and can be used to design and assess environmental qualities of programs for young people. © 2014 Mac Keith Press.

Development of a questionnaire for assessing factors predicting blood donation among university students: a pilot study.

PubMed

Jalalian, Mehrdad; Latiff, Latiffah; Hassan, Syed Tajuddin Syed; Hanachi, Parichehr; Othman, Mohamed

2010-05-01

University students are a target group for blood donor programs. To develop a blood donation culture among university students, it is important to identify factors used to predict their intent to donate blood. This study attempted to develop a valid and reliable measurement tool to be employed in assessing variables in a blood donation behavior model based on the Theory of Planned Behavior (TPB), a commonly used theoretical foundation for social psychology studies. We employed an elicitation study, in which we determined the commonly held behavioral and normative beliefs about blood donation. We used the results of the elicitation study and a standard format for creating questionnaire items for all constructs of the TPB model to prepare the first draft of the measurement tool. After piloting the questionnaire, we prepared the final draft of the questionnaire to be used in our main study. Examination of internal consistency using Chronbach's alpha coefficient and item-total statistics indicated the constructs "Intention" and "Self efficacy" had the highest reliability. Removing one item from each of the constructs, "Attitude," "Subjective norm," "Self efficacy," or "Behavioral beliefs", can considerably increase the reliability of the measurement tool, however, such action is controversial, especially for the variables "attitude" and "subjective norm." We consider all the items of our first draft questionnaire in our main study to make it a reliable measurement tool.

Feasibility, Validity, and Reliability of the Italian Pediatric Quality of Life Inventory Multidimensional Fatigue Scale for Adults in Inpatients with Severe Obesity

PubMed Central

Manzoni, Gian Mauro; Rossi, Alessandro; Marazzi, Nicoletta; Agosti, Fiorenza; De Col, Alessandra; Pietrabissa, Giada; Castelnuovo, Gianluca; Molinari, Enrico; Sartorio, Allessandro

2018-01-01

Objective This study was aimed to examine the feasibility, validity, and reliability of the Italian Pediatric Quality of Life Inventory Multidimensional Fatigue Scale (PedsQL™ MFS) for adult inpatients with severe obesity. Methods 200 inpatients (81% females) with severe obesity (BMI ≥ 35 kg/m2) completed the PedsQL MFS (General Fatigue, Sleep/Rest Fatigue and Cognitive Fatigue domains), the Fatigue Severity Scale, and the Center for Epidemiologic Studies Depression Scale immediately after admission to a 3-week residential body weight reduction program. A randomized subsample of 48 patients re-completed the PedsQL MFS after 3 days. Results Confirmatory factor analysis showed that a modified hierarchical model with two items moved from the Sleep/Rest Fatigue domain to the General Fatigue domain and a second-order latent factor best fitted the data. Internal consistency and test-retest reliabilities were acceptable to high in all scales, and small to high statistically significant correlations were found with all convergent measures, with the exception of BMI. Significant floor effects were found in two scales (Cognitive Fatigue and Sleep/Rest Fatigue). Conclusion The Italian modified PedsQL MFS for adults showed to be a valid and reliable tool for the assessment of fatigue in inpatients with severe obesity. Future studies should assess its discriminant validity as well as its responsiveness to weight reduction. PMID:29402854

Does Changing Examiner Stations During UK Postgraduate Surgery Objective Structured Clinical Examinations Influence Examination Reliability and Candidates' Scores?

PubMed

Brennan, Peter A; Croke, David T; Reed, Malcolm; Smith, Lee; Munro, Euan; Foulkes, John; Arnett, Richard

2016-01-01

Objective structured clinical examinations (OSCE) are widely used for summative assessment in surgery. Despite standardizing these as much as possible, variation, including examiner scoring, can occur which may affect reliability. In study of a high-stakes UK postgraduate surgical OSCE, we investigated whether examiners changing stations once during a long examining day affected marking, reliability, and overall candidates' scores compared with examiners who examined the same scenario all day. An observational study of 18,262 examiner-candidate interactions from the UK Membership of the Royal College of Surgeons examination was carried at 3 Surgical Colleges across the United Kingdom. Scores between examiners were compared using analysis of variance. Examination reliability was assessed with Cronbach's alpha, and the comparative distribution of total candidates' scores for each day was evaluated using t-tests of unit-weighted z scores. A significant difference was found in absolute scores differences awarded in the morning and afternoon sessions between examiners who changed stations at lunchtime and those who did not (p < 0.001). No significant differences were found for the main effects of either broad content area (p = 0.290) or station content area (p = 0.450). The reliability of each day was not affected by examiner switching (p = 0.280). Overall, no difference was found in z-score distribution of total candidate scores and categories of examiner switching. This large study has found that although the range of marks awarded varied when examiners change OSCE stations, examination reliability and the likely candidate outcome were not affected. These results may have implications for examination design and examiner experience in surgical OSCEs and beyond. Copyright © 2016 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

An investigation of a sterile access technique for the repair and adjustment of sterile spacecraft

NASA Technical Reports Server (NTRS)

Farmer, F. H.; Fuller, H. V.; Hueschen, R. M.

1973-01-01

A description is presented of a unique system for the sterilization and sterile repair of spacecraft and the results of a test program designed to assess the biological integrity and engineering reliability of the system. This trailer-mounted system, designated the model assembly sterilizer for testing (MAST), is capable of the dry-heat sterilization of spacecraft and/or components less than 2.3 meters in diameter at temperatures up to 433 K and the steam sterilization of components less than 0.724 meter in diameter. Sterile access to spacecraft is provided by two tunnel suits, called the bioisolator suit systems (BISS), which are contiguous with the walls of the sterilization chambers. The test program was designed primarily to verify the biological and engineering reliability of the MAST system by processing simulated space hardware. Each test cycle simulated the initial sterilization of a spacecraft, sterile repair of a failed component, removal of the spacecraft from the MAST for mating with the bus, and a sterile recycle repair.

Psychometric properties of the Postgraduate Hospital Educational Environment Measure in an Iranian hospital setting.

PubMed

Shokoohi, Shahrzad; Emami, Amir Hossein; Mohammadi, Aeen; Ahmadi, Soleiman; Mojtahedzadeh, Rita

2014-01-01

Background Students' perceptions of the educational environment are an important construct in assessing and enhancing the quality of medical training programs. Reliable and valid measurement, however, can be problematic - especially as instruments developed and tested in one culture are translated for use in another. Materials and method This study sought to explore the psychometric properties of the Postgraduate Hospital Educational Environment Measure (PHEEM) for use in an Iranian hospital training setting. We translated the instrument into Persian and ensured its content validity by back translation and expert review prior to administering it to 127 residents of Urmia University of Medical Science. Results Overall internal consistency of the translated measure was good (a=0.94). Principal components analysis revealed five factors accounting for 52.8% of the variance. Conclusion The Persian version of the PHEEM appears to be a reliable and potentially valid instrument for use in Iranian medical schools and may find favor in evaluating the educational environments of residency programs nationwide.

Psychometric properties of the postgraduate hospital educational environment measure in an Iranian hospital setting.

PubMed

Shokoohi, Shahrzad; Hossein Emami, Amir; Mohammadi, Aeen; Ahmadi, Soleiman; Mojtahedzadeh, Rita

2014-01-01

Students' perceptions of the educational environment are an important construct in assessing and enhancing the quality of medical training programs. Reliable and valid measurement, however, can be problematic - especially as instruments developed and tested in one culture are translated for use in another. This study sought to explore the psychometric properties of the Postgraduate Hospital Educational Environment Measure (PHEEM) for use in an Iranian hospital training setting. We translated the instrument into Persian and ensured its content validity by back translation and expert review prior to administering it to 127 residents of Urmia University of Medical Science. Overall internal consistency of the translated measure was good (a=0.94). Principal components analysis revealed five factors accounting for 52.8% of the variance. The Persian version of the PHEEM appears to be a reliable and potentially valid instrument for use in Iranian medical schools and may find favor in evaluating the educational environments of residency programs nationwide.

Evaluation of urology residents' perception of surgical theater educational environment.

PubMed

Binsaleh, Saleh; Babaeer, Abdulrahman; Rabah, Danny; Madbouly, Khaled

2015-01-01

To evaluate surgical theater learning environment perception in urology residents in Saudi Arabia and to investigate association of learning environment perception and stages of residency program, sectors of health care system, and regions of Saudi Arabia. A cross-sectional survey using the surgical theater educational environment measure (STEEM) inventory. The STEEM inventory was used to measure theater learning environment perception of urology residents in Saudi Arabia. Respondents' perception was compared regarding different residency stages, sectors of the health care system, and regions of Saudi Arabia. Internal reliability of the inventory was assessed using the Cronbach α coefficient. Correlation analysis was done using the Spearman ρ coefficient. Of 72 registered residents, 33 (45.8%) completed the questionnaire. The residents perceived their environment less than acceptable (135.9 ± 16.7, 67.95%). No significant differences in perception were found among residents of different program stages, different sectors of health care system, or different regions in Saudi Arabia. Residents from the eastern region perceived the training and teaching domain better (p = 0.025). The inventory showed a high internal consistency with a Cronbach α of 0.862. STEEM survey is an applicable and reliable instrument for assessing the learning environment and training skills of urology residency program in Saudi Arabia. Urology residents in Saudi Arabia perceived the theater learning environment as less than ideal. The perceptions of theater learning environment did not change significantly among different stages of the program, different sectors of health care system, or different training regions of Saudi Arabia assuring the uniformity of urology training all over Saudi Arabia. The training programs should address significant concerns and pay close attention to areas in surgical theater educational environment, which need development and enhancement, mainly planned fashion of training, supportive supervision and hospital environment, and proper coverage and management of workloads. Copyright © 2014 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.

Hand Society and Matching Program Web Sites Provide Poor Access to Information Regarding Hand Surgery Fellowship.

PubMed

Hinds, Richard M; Klifto, Christopher S; Naik, Amish A; Sapienza, Anthony; Capo, John T

2016-08-01

The Internet is a common resource for applicants of hand surgery fellowships, however, the quality and accessibility of fellowship online information is unknown. The objectives of this study were to evaluate the accessibility of hand surgery fellowship Web sites and to assess the quality of information provided via program Web sites. Hand fellowship Web site accessibility was evaluated by reviewing the American Society for Surgery of the Hand (ASSH) on November 16, 2014 and the National Resident Matching Program (NRMP) fellowship directories on February 12, 2015, and performing an independent Google search on November 25, 2014. Accessible Web sites were then assessed for quality of the presented information. A total of 81 programs were identified with the ASSH directory featuring direct links to 32% of program Web sites and the NRMP directory directly linking to 0%. A Google search yielded direct links to 86% of program Web sites. The quality of presented information varied greatly among the 72 accessible Web sites. Program description (100%), fellowship application requirements (97%), program contact email address (85%), and research requirements (75%) were the most commonly presented components of fellowship information. Hand fellowship program Web sites can be accessed from the ASSH directory and, to a lesser extent, the NRMP directory. However, a Google search is the most reliable method to access online fellowship information. Of assessable programs, all featured a program description though the quality of the remaining information was variable. Hand surgery fellowship applicants may face some difficulties when attempting to gather program information online. Future efforts should focus on improving the accessibility and content quality on hand surgery fellowship program Web sites.

Using qualitative methods to improve questionnaires for Spanish speakers: assessing face validity of a food behavior checklist.

PubMed

Banna, Jinan C; Vera Becerra, Luz E; Kaiser, Lucia L; Townsend, Marilyn S

2010-01-01

Development of outcome measures relevant to health nutrition behaviors requires a rigorous process of testing and revision. Whereas researchers often report performance of quantitative data collection to assess questionnaire validity and reliability, qualitative testing procedures are often overlooked. This report outlines a procedure for assessing face validity of a Spanish-language dietary assessment tool. Reviewing the literature produced no rigorously validated Spanish-language food behavior assessment tools for the US Department of Agriculture's food assistance and education programs. In response to this need, this study evaluated the face validity of a Spanish-language food behavior checklist adapted from a 16-item English version of a food behavior checklist shown to be valid and reliable for limited-resource English speakers. The English version was translated using rigorous methods involving initial translation by one party and creation of five possible versions. Photos were modified based on client input and new photos were taken as necessary. A sample of low-income, Spanish-speaking women completed cognitive interviews (n=20). Spanish translation experts (n=7) fluent in both languages and familiar with both cultures made minor modifications but essentially approved client preferences. The resulting checklist generated a readability score of 93, indicating low reading difficulty. The Spanish-language checklist has adequate face validity in the target population and is ready for further validation using convergent measures. At the conclusion of testing, this instrument may be used to evaluate nutrition education interventions in California. These qualitative procedures provide a framework for designing evaluation tools for low-literate audiences participating in the US Department of Agriculture food assistance and education programs. Copyright 2010 American Dietetic Association. Published by Elsevier Inc. All rights reserved.

Using Qualitative Methods to Improve Questionnaires for Spanish Speakers: Assessing Face Validity of a Food Behavior Checklist

PubMed Central

BANNA, JINAN C.; VERA BECERRA, LUZ E.; KAISER, LUCIA L.; TOWNSEND, MARILYN S.

2015-01-01

Development of outcome measures relevant to health nutrition behaviors requires a rigorous process of testing and revision. Whereas researchers often report performance of quantitative data collection to assess questionnaire validity and reliability, qualitative testing procedures are often overlooked. This report outlines a procedure for assessing face validity of a Spanish-language dietary assessment tool. Reviewing the literature produced no rigorously validated Spanish-language food behavior assessment tools for the US Department of Agriculture’s food assistance and education programs. In response to this need, this study evaluated the face validity of a Spanish-language food behavior checklist adapted from a 16-item English version of a food behavior checklist shown to be valid and reliable for limited-resource English speakers. The English version was translated using rigorous methods involving initial translation by one party and creation of five possible versions. Photos were modified based on client input and new photos were taken as necessary. A sample of low-income, Spanish-speaking women completed cognitive interviews (n=20). Spanish translation experts (n=7) fluent in both languages and familiar with both cultures made minor modifications but essentially approved client preferences. The resulting checklist generated a readability score of 93, indicating low reading difficulty. The Spanish-language checklist has adequate face validity in the target population and is ready for further validation using convergent measures. At the conclusion of testing, this instrument may be used to evaluate nutrition education interventions in California. These qualitative procedures provide a framework for designing evaluation tools for low-literate audiences participating in the US Department of Agriculture food assistance and education programs. PMID:20102831

Using Multivariate Generalizability Theory to Assess the Effect of Content Stratification on the Reliability of a Performance Assessment

ERIC Educational Resources Information Center

Keller, Lisa A.; Clauser, Brian E.; Swanson, David B.

2010-01-01

In recent years, demand for performance assessments has continued to grow. However, performance assessments are notorious for lower reliability, and in particular, low reliability resulting from task specificity. Since reliability analyses typically treat the performance tasks as randomly sampled from an infinite universe of tasks, these estimates…

DOE Office of Scientific and Technical Information (OSTI.GOV)

Goodwin, Malik

Reliable public lighting remains a critically important and valuable public service in Detroit, Michigan. The Downtown Detroit Energy Efficiency Lighting Program (the, “Program”) was designed and implemented to bring the latest advancements in lighting technology, energy efficiency, public safety and reliability to Detroit’s Central Business District, and the Program accomplished those goals successfully. Downtown’s nighttime atmosphere has been upgraded as a result of the installation of over 1000 new LED roadway lighting fixtures that were installed as part of the Program. The reliability of the lighting system has also improved.

Update on MTTF figures for linear and rotary coolers of Thales Cryogenics

NASA Astrophysics Data System (ADS)

van de Groep, W.; van der Weijden, H.; van Leeuwen, R.; Benschop, T.; Cauquil, J. M.; Griot, R.

2012-06-01

Thales Cryogenics has an extensive background in delivering linear and rotary coolers for military, civil and space programs. During the last years several technical improvements have increased the lifetime of all Thales coolers resulting in significantly higher Mean Time To Failure (MTTF) figures. In this paper not only updated MTTF values for most of the products in our portfolio will be presented but also the methodology used to come to these reliability figures will be explained. The differences between rotary and linear coolers will be highlighted including the different failure modes influencing the lifetime under operational conditions. These updated reliability figures are based on extensive test results for both rotary and linear coolers as well as Weibull analysis, failure mode identifications, various types of lifetime testing and field results of operational coolers. The impact of the cooler selection for typical applications will be outlined. This updated reliability approach will enable an improved tradeoff for cooler selection in applications where MTTF and a correct reliability assessment is key. Improbing on cooler selection and an increased insight in cooler reliability will result in a higher uptime and operability of equipment, less risk on unexpected failures and lower costs of ownership.

Liquid Rocket Engine Turbopump Rotating-shaft Seals

NASA Technical Reports Server (NTRS)

Burcham, R. E.; Keller, R. B., Jr. (Editor)

1978-01-01

A monograph is organized and presents, for effective use in design, the significant experience and knowledge accumulated in development and operational programs to date. It reviews and assesses current practices, and from them establishes firm guidance for achieving greater consistency in design, increased reliability in the end product, and greater efficiency in the design effort. The monograph is divided into two major sections: state of the art and design criteria.

Nuclear Weapons: Comprehensive Test Ban Treaty

DTIC Science & Technology

2006-07-10

continued...) The complex could contain explosions up to 500 pounds of explosive and associated plutonium. Another SCE, “ Unicorn ,” is to be conducted...scheduled for FY2006, as noted below. SCEs try to determine if radioactive decay of aged plutonium would degrade weapon performance. Several SCEs...Richardson called SCEs “a key part of our scientific program to provide new tools and data that assess age -related complications and maintain the reliability

Evaluation of the Military Functional Assessment Program: Inter rater Reliability of Task Scores

DTIC Science & Technology

2017-09-19

return-to-duty. Performance on the tasks is rated by a non-commissioned officer (NCO), occupational therapist, physical therapist, and mental health ...and additional ratings are provided on a subset of the tasks by an occupational therapist (OT), physical therapist (PT), and mental health (MH...3National Intrepid Center of Excellence United States Army Aeromedical Research Laboratory Aircrew Health and Performance Division September 2017