If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Boards of nursing (BON) approval of nurse education programs is an integral part of their mission of public protection. In the United States , nursing education programs are required to be approved by the BON
In Mississippi, the registered nurse programs are approved by the Mississippi Institutions of Higher Learning and the practical nursing programs are approved by the BON. In New York, the programs are approved by the New York Board of Regents. In Idaho, programs are approved as long as they are accredited by a national nursing accrediting agency recognized by the U.S. Department of Education, though the BON takes over if that accreditation is lost.
in the state where the program is officially located. The purpose of program approval is to ensure the program comprehensively covers the knowledge and skills students need to be licensed as a nurse and to practice safely as new graduate nurses, thereby providing society a competent nurse workforce.
To obtain BON nursing education program approval, nursing programs must meet the nursing education standards established by their BON. Only students graduating from officially recognized and approved programs are permitted to take the the NCLEX, the official nursing licensure exam in the US and Canada. ((
). To determine whether graduates are eligible to take the NCLEX, BONs rely on verification from the nursing education program that each student has successfully completed all program requirements, including successfully meeting clinical learning objectives.
BONs offer two types of nursing education program approval: initial approval of new programs before they open for enrollment and ongoing monitoring and continued approval of programs. For a new program, the approval process begins with an initial application and proposal to the BON. The BON conducts an extensive evaluation to ensure that the program has the proper facilities, resources, administration and faculty, curriculum, clinical agreements, policies, and procedures, among other requirements set forth in state regulations. The process for continued approval of established programs is based upon monitoring the programs’ performance outcomes and compliance with BON rules over time (
BONs use different models for approving nursing programs, and nursing education rules and regulations are not always consistent across all jurisdictions. Most BONs hire graduate-prepared education consultants with experience in nursing education to make recommendations on the approval status of the nursing programs in their state. In a few states, the BON’s executive officer and board members from the BON’s education committee (or educators on the board) may make these recommendations. About half of the BONs make site visits as needed, while the other half make regular visits (National Council of State Boards of Nursing [NCSBN], 2016a). In most states, the approval status will be designated as (a) full approval when all requirements are met; (b) conditional or probationary when some, but not all, of the requirements are met; or (c) approval removal when programs fail to meet requirements of correct cited deficiencies (
The three most common performance outcome measures used by BONs and other health profession accreditors are employment rates, graduation rates, and NCLEX pass rates (
). Although BONs use different models for approving nursing programs, the approval process is well recognized overall. Questions have arisen from nursing education experts regarding valid measures of nursing program quality. One is the use of NCLEX pass rates (
Fifteen states require the national pass rate or a percentage thereof.
* 64% of them require an 80% pass rate (NSCBN, 2019). BONs are also questioning whether this method of measuring program performance is appropriate and asking whether other metrics exist that should amend or replace the current regulatory standards, which are set by each state.
A large mixed-method study was conducted by NCSBN starting in 2017 to answer these questions and, more specifically, to identify the quality indicators of approved nursing education programs and the warning signs indicating a program may be falling below required standards for approval.
Methods and Selected Findings
The study consists of a comprehensive literature review; a national Delphi study providing data on consensus of experts in nursing education, regulation, and practice; a study analysis of 5 years’ worth of BON annual reports of nursing programs; and a study analysis of 5 years’ worth of BON site visit documents.
Literature Review
The literature review yielded 65 relevant published articles that were reviewed and graded using the Johns Hopkins Levels of Evidence and Quality Guide. Overall, the literature review revealed a number of quality indicators and warning signs that may serve as metrics for evaluating higher education programs, although the identified articles did not provide the levels of evidence needed for making policy decisions (
National Delphi Study Identifying Quality Indicators and Warning Signs of Nursing Education Program Performance
For our national Delphi study, data were provided on consensus from experts in nursing education, regulation, and practice on nursing education quality indicators, warning signs when programs are beginning to fall below standards, and performance outcome measures of nursing education programs. Consensus among the experts was reached after 2 rounds of discussion. This Delphi study identified 18 quality indicators (characteristics of nursing programs that graduate safe and competent students), 11 warning signs when nursing programs begin to fall below standards, and eight program performance outcomes that nursing regulatory bodies could measure. The quality indicators fall into the categories of (a) school leadership and faculty support; (b) consistent and competent faculty; (c) quality, hands-on clinical experiences with meaningful collaboration with clinical partners; and (d) an evidence-based curriculum emphasizing quality and safety and critical thinking/clinical reasoning. Although the warning signs are similar to the quality indicators (only the opposite), there are additional ones that are of interest, including over-reliance on simulation to replace clinical experiences and refusal of clinical facilities to host clinical experiences. There were few surprises with the outcomes that were identified (NCLEX pass rates, graduation rates, employment rates, etc.).
A Quantitative Analysis of 5 Years of BONs Annual Report Document
This study was a retrospective cohort study of 11,378 annual report data collected over a 5-year period (2012-2017) by 43 BONs. This quantitative analysis examined data contained within the BONs’ annual reports to learn about indicators associated with full approval of nursing education program performance and those associated with programs that have lost approval. Statistically significant characteristics of approved programs and those with ≥ 80% NCLEX pass rates included (a) national accreditation, (b) traditional or hybrid modalities, (c) longer-standing programs, (d) higher enrollment capacity, (e) multiple program sites, (f) private nonprofit or public institutions, (g) program director with a PhD, (h) licensed practical nurse/licensed vocational nurse (LPN/LVN) and bachelor of science in nursing (BSN) programs, and (i) no more than three program directors in 5 years. A marginally significant finding was that programs with more than 35% full-time faculty had ≥ 80% first-time NCLEX pass rates and full approval. Importantly, the NCLEX was viewed as a lagging indicator in this study; meaning, lower licensure examination performance was considered indicative of other program deficiencies, not vice versa.
A Quantitative Analysis of 5 Years of BONs Annual Report Documents
The qualitative study of 5 years’ of BONs’ site visit documents was conducted to better understand the qualifiable descriptors of why programs either become at risk for failing or do fail. After the inclusion/exclusion criteria were applied, there were 1,278 site visit reports for LPN/LVN and registered nurse (RN) programs eligible for the analysis, which included documents from programs that were on probation, under review, or did not have full approval. Two researchers used MaxQDA qualitative data analysis software to analyze the documents. Considerable, specific data on what causes nursing programs to begin to fail or fail, were found in the site visit documents. The main signal for a “site visit trigger” was NCLEX pass rates ≤80% for four or more quarters. The length of time it took to trigger a site visit related to NCLEX performance concerns varied by state regulations. Administrative processes, such as a lack of policies and procedures, were found to be problematic for nursing programs. High faculty turnover and the inability to recruit qualified faculty were linked to poor NCLEX performance. Faculty with little training in basic pedagogies was a persistent theme found in failing programs. Similarly, heavy faculty workloads and limited faculty development opportunities were also identified. Many failing programs had no overarching philosophy and curricular framework that tied the curriculum together. This gap resulted in curricula that were task-oriented, which masked the curricula as being “competency-based.” The issues identified in this study coalesce nicely with the data found in the literature, our Delphi study, and our quantitative study of annual reports.
Guideline Development
After all the evidence was collected, NCSBN invited experts from nursing regulation, education, research, and law to review the data and findings and to develop guidelines for BONs to use when approving nursing education programs that include evidence-based criteria, quality indicators, and warning signs.
Conclusion
This study provides substantial evidence-based criteria for identifying quality indicators of successful nursing education programs as well as warning signs for high-risk programs. The quality indicators and warning signs can serve as the basis for legally defensible and evidence-based guidelines for nursing education approval.
It is hoped that these guidelines will enhance collaboration between educators and regulators. Together, they will be able to use the quality indicators to guide nursing programs to approval and to identify warning signs when the nursing program is beginning to fall below standards. This early intervention will assist nursing programs to act before BON sanctions or program closures, thus continuing to graduate safe and competent nurses, in adequate numbers, to care for patients.
NCSBN Regulatory Guidelines and Evidence-Based Quality Indicators for Nursing Education Programs
In the United States, prelicensure nursing education programs are required to be approved by the BON in the state where the program is officially located.
In Mississippi, the registered nurse programs are approved by the Mississippi Institutions of Higher Learning and the practical nurse programs are approved by the BON. In New York, the programs are approved by the New York Board of Regents. In Idaho, programs are approved as long as they are accredited by a national nursing accrediting agency recognized by the U.S. Department of Education, though the BON takes over if that accreditation is lost.
This approval process begins with an initial application and extensive proposal to the BON, which performs an extensive evaluation ensuring the program has the proper facilities, resources, administration and faculty, curriculum, clinical agreements, policies, and procedures, among many other requirements set forth in state regulations. Once the program is approved, the BON continually monitors the program. The monitoring process consists of overseeing NCLEX pass rates and may include other metrics such as student retention and/or graduation rates. Additionally, many BONs periodically conduct formal site visits to the program.
A common cause for conditional or loss of approval is a drop in the required NCLEX pass rate. Although the approval process is well recognized at the state level, there are questions regarding the prelicensure nursing program approval criteria, particularly using the NCLEX pass rates as the sole criteria for ongoing approval. Of the 57 BONs that regulate nursing education programs surveyed in NCSBN’s Member Board Profiles, 36 use first-time NCLEX pass rates; of those, 64% use the 80% pass rate as their standard. Six do not use pass rates as a performance measure, and another 15 use the national pass rate or a percentage thereof (NCSBN, 2019).
Questions by nurse educators remain as to whether first-time NCLEX pass rates are valid measures, when used alone, of nursing education program quality (
). BONs also are asking whether other evidence-based metrics exist that should amend or replace the current regulatory standards set by each state. Therefore, in 2017, NCSBN embarked on a 3-year journey to identify the evidence needed to answer these questions.
This report presents a literature review that found there is currently little evidence with the rigor needed to support quality indicators of nursing education. Additionally, it details a three-part, mixed-methods national study that NCSBN conducted to identify quality indicators of nursing education programs, as well as warning signs when programs begin to fall below standards. From this large study, consisting of three national studies using very different research methodologies (Delphi, quantitative, and qualitative studies), nursing education approval guidelines were developed.
This groundbreaking work provides nursing regulators with evidence-based and legally defensible tools for approving programs. In their missions of public protection, regulators will be able to first identify warning signs when nursing programs are beginning to fall below standards. They can then use the quality indicators to guide programs before sanctions or program closures occur, thus continuing to graduate safe and competent nurses, in adequate numbers, to care for patients. Additionally, nurse educators will find this evidence valuable as they plan for and evaluate their programs.
Definitions of terms used in this study are provided in Appendix A.
Literature Review
The following three criteria relative to nursing program approval formed the basis of the literature review:
•
Use of NCLEX pass rates as a performance measure of prelicensure nursing programs.
•
Additional metrics used to measure performance of higher education programs and the supporting evidence.
•
Warning signs indicating a nursing program is falling below standards and at risk of losing BON approval.
Medline, PsychInfo, ERIC (Education Resources Information Center), and CINAHL (Cumulative Index of Nursing and Allied Health Literature) Complete were queried using the following keywords: (a) nursing education outcomes (and higher education outcomes); (b) nursing education (and higher education) quality indicators; and (c) predictors of nursing education (and higher education) quality. Because of the lack of literature in these areas, we also searched gray literature, which included literature and publications related to nationally recognized expert reports from organizations, governmental agencies (eg, U.S. Department of Education [USDE]), international nursing regulatory bodies, national healthcare regulatory and accreditation bodies, and national education workshops. Additionally, literature reviews, case reports, and opinions of nationally recognized experts based on experiential evidence were included.
Our literature search also included U.S. and international databases from education and other related fields. Citations from 65 relevant articles and reports were retrieved and reviewed. These articles and reports are summarized citing the type of publication, its purpose, key findings (Appendix B2), using the level of evidence according to Johns Hopkins Levels of Evidence and Quality Guide (
) (Appendix B1). For the levels of evidence, two researchers (one external to NCSBN) rated the evidence levels separately and then came to consensus on the final rating.
Use of NCLEX Pass Rates as a Performance Measure of Prelicensure Nursing Programs
Regulatory bodies in other professions use licensure or certification pass rates to assess program performance, though not as the sole measure (
The National Academies of Sciences, Engineering, and Medicine. (2018). Graduate medical education outcomes and metrics: Proceedings of a workshop. The National Academies Press.
stated that the Fundamentals of Engineering licensing examination should not be used to determine program content because the examination is meant to measure competency for licensure and the criterion is too broad to be effective in program improvement. They instead assert that more specific measures are needed (
surveyed 45 specialized and professional accreditors and found that not all professions use licensure or certification examinations to measure education outcomes. They reported that while 84% of the professions have certification or licensure examinations, 64% of those accreditors require education programs to use pass rates as part of their self-assessments.
In 2018, NASEM held a workshop on graduate medical education outcomes and metrics (
The National Academies of Sciences, Engineering, and Medicine. (2018). Graduate medical education outcomes and metrics: Proceedings of a workshop. The National Academies Press.
). The workshop participants agreed that test results are being used as outcomes to assess graduate medical education. However, they also acknowledged that measuring outcomes in graduate medical programs is complex and encouraged institutions to pilot other criteria and innovative ways to provide feedback to the programs and trainees.
Similarly, how does program use of standardized examinations such as admission and course-related progression tests relate to pass rates?
, in a large national study of 832 nursing programs, found that higher NCLEX pass rates were associated with programs that did not use standardized examinations for either admission or progression. The investigators caution that they did not assess overall performance of the programs and the standardized examinations may have been implemented by low-performing schools and thus had nothing to do with the influence of standardized tests on NCLEX pass rates. Similarly,
statewide study of 34 nursing programs found that when programs required a set score on an NCLEX predictor examination for graduation, NCLEX pass rates and on-time graduation rates were statistically significantly lower than those programs that did not have cut scores on predictor examinations. Randolph theorized that the low-performing programs use predictor examinations to eliminate students who would fail the NCLEX and lower their pass rates. She further concluded that if BONs use NCLEX pass rates as the only metric, nursing programs that use predictor examinations as exit examinations could be falsely elevating their NCLEX pass rates because they are preventing lower performing students from taking the examination.
Many questions from faculty exist regarding the use of first-time pass rates as the primary metric.
, in a study of NCLEX pass rates from 2010–2014 of 1,792 programs across the United States, found that 28% of the programs that failed to meet states’ pass rate standards were within the 95% confidence interval (CI), meaning that 28% of the programs that failed to meet their respective states’ pass rates had a 95% CI that included and at times surpassed the passing threshold. He concluded that it was perhaps by chance these programs fell below the pass rate standard. For this reason, most BONs take action after 2 or more years of below-standard pass rates.
The USDE recommends higher education use employment rates and graduation rates in addition to licensure or certification examination pass rates because these metrics are necessary for graduates to enter the workforce (
; National League for Nursing, 2016), and other U.S. healthcare accreditors use the USDE’s requirements for their outcome metrics. While these outcomes have face validity, as new nurses must graduate, pass the NCLEX, and become employed in order to enter the workforce, there is little evidence these outcomes are indicators of program quality (
≥80% first-time pass rate on NCLEX over 3-y period
Programs set target rates based on unit, demographics, etc.
Students express satisfaction with program effectiveness
Programs set target rates based on unit, demographics, etc.
Employers express satisfaction with program effectiveness
Initial approval is 6 y with mid-cycle report due after first 3 y; continuing accreditation is granted for up to 10 y; mid-cycle report due at 5 y; annual reports and substantive change reports must be filed
Liaison Committee on Medical Education. (2019). Functions and structure of a medical school: Standards for accreditation of medical education programs leading to the MD degree. https://lcme.org/publications/#Standards
Commission of Accreditation in Physical Therapy Education. (2017). Standards and required elements for accreditation of physical therapist education programs. http://www.capteonline.org/AccreditationHandbook/
Accreditation Council for Pharmacy Education. (2015). Accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree. https://www.acpe-accredit.org/pdf/Standards2016FINAL.pdf
First-time performance on NAPLEX compared to national average first-time pass rate
Required but no rate specified
AACP Graduating Student Survey
Not addressed
Not addressed
Initial approval is 2 y followed by 8-y cycle
Note. USMLE = United States Medical Licensing Examination; AAMC = Association of American Medical Colleges; NRMP = National Resident Matching Program; NBCOT = National Board for Certification of Occupational Therapy; NPTE = National Physical Therapy Examination; NAPLEX = North American Pharmacist Licensure Examination; AACP = American Association of Colleges of Pharmacy.
Additional Metrics Used to Measure Performance of Higher Education Programs
Academia is changing what it considers performance metrics in undergraduate education. The NASEM workshop also explored the quality of undergraduate education and found student performance outcomes were the most appropriate set of indicators determining education quality over program inputs (i.e., student, faculty, and program characteristics) (
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
The National Academies of Sciences, Engineering, and Medicine. (2018). Graduate medical education outcomes and metrics: Proceedings of a workshop. The National Academies Press.
). In the past, input measures such as faculty-student ratios, expenditures, or student test scores were used as metrics of education; however, many data elements measuring performance outcomes are not comparable across institutions due to different conceptual definitions and populations (
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
). Personal communications with representatives of national accreditors (February 28 and March 1, 2017) confirmed that employment rates are viewed as the least valid measure of quality, despite widespread use. While it may be presumed that a higher quality program produces a higher number of employed graduates, there is no sound method of using employment rates as a proxy for program performance without accounting for geographic differences in labor markets (
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
In a descriptive study of 5,182 engineering students in Italy, researchers measured both incoming quality and outgoing performance and determined that if employment rates are used to measure institutional quality, then assessors need to collect data on the average regional unemployment rate for the age group, in both the region of student residence and the region of the institution (
). Ferrante suggested that such data collection requires a significant investment of time and resources, one that yields little return because employment rates are mostly used with other quality measures.
provided evidence that the changing job market, which nursing programs cannot control, reflected in various regions and the U.S. economy has a large impact on employment rates. For example, the U.S. economy negatively affected the job market from 2009 to 2012. The decreasing rates of new graduate employment from 2008 to 2010 (
) reflected the economy and not the quality of the nursing program. Similarly, new graduate employment rates in the West and Northeast parts of the United States tend to be significantly lower than those in the South and Midwest parts of the United States (
). These variables need to be accounted for if nursing relies on employment rates as a measure of quality.
Currently, only seven U.S. nursing regulatory bodies report using employment data for their approval processes (Nursing Education Outcomes and Metrics Committee, 2017). Given the difficulty obtaining the data and that employment rates reflect regional economics and job availability, using employment rates as a metric for BON approval is not recommended because it is burdensome and without evidence of validity as a measure of education performance (
Another common metric used to measure institutional quality is graduation rates, which is considered a more valid assessment tool than employment rates (
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
allows the faculty to establish an expected level of completion that reflects student demographics (Table 1). According to an NCSBN survey of the education consultants (Nursing Education Outcomes and Metrics Committee, 2017), 17 U.S. nursing regulatory bodies use graduation rates for their approval requirements.
How graduation rates are calculated is a point of debate. A common calculation is the Integrated Postsecondary Education Data System (IPEDS) calculation, which is also used to measure the quality of an institution. The IPEDS calculation counts only those students who enroll in an institution as full-time degree-seekers and finish a degree at the same institution within a prescribed period. IPEDS ignores certain students such as nontraditional (i.e., older) students, transfer students, part-time students, and students who enroll mid-year (
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
). This calculation gap risks incentivizing programs to implement more selective admissions policies, prioritizing students who are more likely to maintain the full-time, 6-year-or-less graduation track. Conversely, institutions that admit student populations not captured by the current calculation are at a disadvantage because the system does not count all graduates. Graduation rates should account for all students if they are going to be used as a metric of program performance.
Another method for calculating graduation rates proposed by
is the graduation efficiency metric. This metric emerged out of the perceived problems with the standard graduation rates and attempted to capture more of the student population, including part-time students and transfer students. To capture these students in the calculation, the graduation efficiency metric measures an institution’s “production of graduates” in relation to the size of its full-time equivalent undergraduate student body adjusted for the balance of beginning and transfer students. Graduation rate calculations that account for different types of students (e.g., students who leave programs for maternity leave, illness, family issues, etc.) quickly become much more complex than the standard calculation. The graduation efficiency metric, however, is relatively straightforward and can be calculated from existing data that schools already collect. Moreover, this metric does not encourage institutions to turn away part-time students and transfer students in order to increase raw graduation rates in accordance with the traditional graduation rate calculation.
With a goal of holding institutions accountable for their graduation rates,
studied 210,056 students in 356 nonprofit, 4-year-degree institutions, merging data from the National Student Clearinghouse and freshman surveys. They found student characteristics, rather than the institution’s characteristics, had an impact on student outcomes. Using multiple regression models, they could predict graduation rates based on student high school grades, SAT scores, race/ethnicity, and gender, as well as a variety of other indicators as obtained from the freshmen surveys (i.e., volunteer work in high school, student finances, parental background, working full-time, etc.). Students living off campus during their first year had 35% lower odds of finishing their degree than students living in campus residence halls. Additionally, they found that the 4-year degree attainment for public universities would increase to 140% if they admitted students with the same characteristics as those admitted to private universities. They concluded that differences in graduation rates among higher education institutions are largely attributable to the profiles and characteristics of their incoming students. This study demonstrates that graduation rates may have little to do with the school’s performance but rather are impacted by student characteristics.
Retention
Another way of measuring program performance is to shift the focus from raw rates of completion (using the number of degrees awarded to the population of first-time, full-time students who graduate from the institution that admitted them) to rates of student retention (
) as the number and percentage of entering undergraduate students who enroll consecutively from fall to spring and fall to fall at an institution of higher education. Retention rates are also known as persistence rates (
suggested, rates of retention should be approached by asking the general question, “What proportion of a university’s nursing students graduate with a nursing degree within the typical time frame plus 50%?” The additional 50% is added to account for students who take 6 years to graduate. The typical timeframe can be adjusted depending on the type of nursing program and track option (e.g., baccalaureate, accelerated). Furthermore, the timeframe can be set according to the date a given student took their first nursing course at the institution and the date the degree was awarded. Students who do not graduate should also be counted. How to measure the standard for each nursing program option requires discussion and modification among stakeholders until the data accurately report retention/persistence. Papes and Lopez also suggested that, from an assessment perspective, low persistence rates may be considered an aberration, but a declining rate should be taken seriously and considered a warning sign.
In a study of 489 public and 820 private nonprofit universities, researchers investigated retention rates from 2003 to 2013 (
) and found that universities with students of similar profiles had differing retention rates. The researchers concluded that what universities do above and beyond traditional teaching methods can influence retention rates. For example, San Diego State University employed several strategies to improve retention rates. They partnered with local junior high and high schools to connect students to college earlier, they pushed all students to carry a minimum of 15 credit hours, and they instituted proactive advising and degree planning, fostered communities for first year students, and used data to improve curricula. As a result, San Diego State University vastly improved retention of underrepresented students (Eberle-Sudre et al., 2015). These results add another perspective to
graduation rate findings previously discussed. Student profiles and characteristics, as well as strategies that supplement traditional teaching methods, influence retention and graduation rates.
also found multiple factors not directly related to teaching that influenced graduate nurse outcomes on the NCLEX. They conducted a national study of 832 nursing programs in the United States and compared program, faculty, and curriculum characteristics to NCLEX pass rates. Using multilevel modeling and regression analyses and controlling for variables, they found a statistically significant increase in NCLEX first-time pass rates with public schools, semester schedules, larger admission cohorts, more students per didactic faculty, and a higher percentage of full-time faculty. There were no statistically significant findings associated with the use of simulation, integrated curricula (i.e., specialties are not offered as separate courses), online learning environments, individual course grades, minimal course grades, clinical evaluations, or allowing students to repeat courses (
Studies and experts have examined program quality in terms of passage on the licensure examination, employment, graduation, and retention rates of students up to this point. However, the literature examining clinical experiences shifts the quality discussion to production of graduates that are prepared to safely care for patients.
Quality Clinical Experiences
There is consensus in the international literature that quality, direct-care clinical experiences with actual patients are the foundation of quality nursing education (
). Similarly, nurse regulators recognize clinical experiences with actual patients are an integral part of the program approval process in the United States and Canada (
A suggested quality indicator for nursing education is the number of clinical hours required in a nursing curriculum, although this indicator has not been extensively studied. In an integrative literature review of 50 articles,
conducted a scoping review of four countries’ clinical hours and found disparity among clinical hour requirements with little evidence to support any of them.
Australia requires 800 hours at the baccalaureate level, not including simulation. New Zealand requires 1,100 hours of clinical experience, with 360 hours in the final semester. The United Kingdom requires 2,300 hours of clinical experience. The United States has no required hours nationally (although a few states have requirements), but the national median is 712 hours for baccalaureate programs; 683 for diploma programs; and 573 for ADN programs (
They suggest further research to determine the number of practice experience hours needed and how to ensure the practice hours experienced are of a high quality (
recommends integrating clinical and classroom experiences. As part of a series of studies known as the Carnegie Foundation’s Preparation for the Professions Program, Benner et al. used an ethnographic, interpretive, and evaluative design to study all aspects of nursing education. They researched nine programs they deemed to be excellent*
Excellence was defined as follows: (a) reputation for teaching and learning; (b) high NCLEX pass rates; (c) recommended by either the NRB or the accrediting body; and (d) additional consideration given to geographic sampling and accommodating the school’s calendar (
* at all prelicensure levels. The findings demonstrated that nursing programs that provided hands-on, interactive clinical experiences and integrated those experiences into the classroom had higher ratings of student satisfaction. A fragmented system where clinical and classroom learning are not linked may not provide for comprehensive understanding and does not allow for students to make astute clinical judgments (
Researchers in a statewide survey of 352 nurses (12% response rate) found the majority of respondents reported that they needed more clinical hours in the nursing program to enhance their readiness for practice (
study to evaluate student outcomes using two different models of clinical immersion with senior nursing students had different results. Kumm et al. evaluated the difference between the original 16-week clinical immersion experience and a revised 8-week experience in preparing senior nursing students for practice by using the New Graduate Nurse Performance Survey (later changed to the Nursing Practice Readiness Tool) developed by the Nursing Executive Center. The survey evaluates new graduates in six distinct competency areas (i.e., clinical knowledge, technical skills, critical thinking, communication, professionalism, and management of responsibilities) using 36 items. The researchers found no statistically significant differences in the new graduate nurses’ performance between the two clinical immersion models, suggesting that it is the quality of experience that is important rather than the length of time. Similarly,
) asserting hospital-trained graduates in Australia had too much practice and not enough theory in the 1970s. Today, the argument related to the education-practice gap seems to be the opposite. This suggests the discourse should change from quantity of hours to quality of the direct care clinical experiences.
The NCSBN national simulation study: A longitudinal, randomized, controlled study replacing clinical hours with simulation in prelicensure nursing education.
Qualified nurses’ perceptions of nursing graduates’ abilities vary according to specific demographic and clinical characteristics: A descriptive quantitative study.
) have addressed preparation for practice by obtaining the input of practicing professionals and nursing graduates. The need for quality clinical hours, either with supervised clinical experiences with actual patients or with simulation, is a major research finding (
The NCSBN national simulation study: A longitudinal, randomized, controlled study replacing clinical hours with simulation in prelicensure nursing education.
). However, what are quality clinical hours, and how can BONs be sure that clinical experiences are providing the needed knowledge to prepare students for entry to practice? The following are cited in the literature as elements integral to a quality clinical experience:
conducted a descriptive exploratory study of 569 nursing student preceptors to determine student readiness for practice. Their results, along with those of
, suggest nursing programs need to place more emphasis on patient safety and integrating pharmacology more meaningfully throughout the program to have a truly high-quality program.
Developed by consensus of national nursing and healthcare experts, including input from nurse regulators, the Quality and Safety Education for Nurses (QSEN) competencies (
) have been integrated into many U.S. nursing education programs as a foundation for professional competence and patient safety. No studies, however, have been conducted to determine whether these are quality indicators or whether programs that embed these competencies into their curriculum have better prepared students than schools that do not. Similarly, the provincial and territorial nurse regulators in Canada have developed the Jurisdictional Competency Process Entry-Level Registered Nurse Competencies (
Professional responsibility and accountability, knowledge-based practice, ethical practice, service to the public and self-regulation.
While the domains have different labels, the QSEN and the jurisdictional competency process for entry-level competencies are similar, including patient safety, professional responsibility, evidence-based practice, and knowledge-based care. This suggests there is some regulatory consistency for nursing program quality between the United States and Canada. Furthermore, the College of Nurses of Ontario finds these competencies so essential that they are incorporating them into their nursing education approval process.
Faculty
It is presumed that faculty play a leading role in the overall performance of a nursing education program, yet actual evidence for this assumption is limited.
recommended strategies for Illinois programs to improve their NCLEX pass rates based on their observations as regulation board members who conducted site visits to prelicensure programs. Their suggestions included focusing on the appropriate ratio of full-time versus part-time faculty (no recommendation given) and whether faculty professional development needs were being met. Additionally, they suggested evaluating the program’s administrative support, including leadership of the program and financing, to support ongoing program improvement. Other areas they found important to assess included evaluation tools, teaching/learning methodologies, admission policies, faculty-student ratios, and academic support.
examined faculty qualifications such as whether a higher percentage of doctoral faculty teaching didactic courses and certification in specialty fields or in nursing education resulted in higher NCLEX pass rates. Data were not statistically significant. Only one study (an unpublished master’s thesis) of 92 nursing programs in Kansas and Missouri found a positive trend between NCLEX pass rates and faculty with doctoral degrees (
Number of clinical hours in the nursing programs and National Council Licensure Examination for Registered Nurses (NCLEX-RN) passing rate [Unpublished master’s thesis].
did find a statistically significant difference in NCLEX pass rates when a program had a higher percentage of full-time faculty versus part-time or adjunct faculty.
Systematic Program Evaluation
The need for a program evaluation system has been cited as a crucial element for assessing a program by regulators, accreditors, and educators (
). Oermann (2017, p. 1) defines program evaluation as a systematic process for collecting data for making decisions about the nursing program and assessing its value. This process is also foundational to the national nursing accreditors as they evaluate programs for accreditation (Accreditation Commission for Education in Nursing, 2017;
; National League for Nursing, 2016). No actual studies have been conducted or data collected as to the most important elements of a program evaluation.
Institution Type
Although there is no specific explanation as to why, there is evidence that the type of institution (i.e., public, nonprofit, for profit) affects program and student outcomes. As cited previously,
found that public schools would outperform private schools in terms of graduation rates if the schools had similar characteristics. For-profit institutions were not included in DeAngelo’s sample.
studied 5 years’ worth of data from 13,745 nursing programs in 41 states and the District of Columbia using multivariable linear and regression models and controlling for variables. Public schools outperformed (using NCLEX first-time pass rates as the measure) the nonprofit and for-profit schools, though the margin of effect was much higher for the for-profit schools. Similarly,
did not find a statistically significant difference between accredited schools versus unaccredited schools; however, only 43 (6%) of the programs they studied were not accredited
It should be noted that while almost 89% of BSN nursing programs are accredited, only about 53% of associate degree programs and 11% of practical nursing programs are accredited (
studied all RN nursing programs in 2016, comparing first-time NCLEX pass rates with accreditation status (i.e., Yes/No), and found a statistically significant difference in NCLEX pass rates between accredited programs versus unaccredited programs. Specifically, 741 programs (ADN and BSN) were not accredited and had pass rates of 72%, whereas 1,531 programs (ADN and BSN) were accredited and had pass rates of 87%. More research is needed on the relationship between national nursing accreditation and program outcomes.
In summary, during a national workshop, higher education experts reviewed the literature and found that there are no magical solutions to the long-standing issue of performance measurement in higher education institutions (
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
), which is similar to the findings of this literature review of higher education outcomes and metrics. In fact, none of these components alone may be indicative of a quality program. It may require a combination of factors that lead to producing competent graduates.
Warning Signs Indicating a Nursing Program Is Falling Below Standards and At Risk of Losing BON Approval
While the literature does not address warning signs, per se, it does address observations when nursing programs begin to experience difficulties.
Failing to address unsafe students in the clinical area could be a warning sign for a nursing program.
surveyed 84 faculty in one Western state and found that faculty frequently do not fail nursing students, neither in didactic nor clinical areas, who demonstrate unsatisfactory progression in the program. This poses safety issues for patients and challenges for the practice setting and nurse regulators, especially when the school does not have a remediation process. Furthermore,
conducted a qualitative descriptive design with nursing, education, and social work students to learn why faculty have difficulty failing students in clinical experiences. They suggest student failure is a difficult experience for the student and faculty and there are consequences for the program as well.
, nurse regulators from Texas, documented some of the weaknesses they encounter when they evaluate nursing programs and provided ideas for early interventions. Areas of weakness include (a) lack of early recognition of at-risk students, (b) inconsistent use of grading policies, (c) insufficient numbers of faculty, (d) lack of faculty development, (e) lack of rigor across the curriculum, (f) inadequacies with the testing processes, (g) difficulty in locating clinical placements, and (h) an ineffective program evaluation plan.
editorialized about what nurse regulators observe when a program abruptly shuts its doors. The warning signs that nursing regulatory bodies observe when programs experience difficulties include:
•
Rapid growth in admissions
•
High faculty turnover
•
Unclear workload policies
•
High administrator turnover
•
High rate of complaints
•
Weak admissions policies
•
Old-fashioned skills lab with high student ratios
•
Poor clinical placements.
Conclusion
There is an overall lack of evidence regarding the existence of validated metrics that could be used to evaluate a nursing education program, although the number of articles suggest there is a growing body of evidence that is defining what constitutes a quality education in nursing. We did not critique the quality of the research studies within the text of this report since our goal was to determine the state of the science so we could answer our research questions. However, we did rate the level and quality of the research and reports using Johns Hopkins evidence levels and quality ratings, which can be found in Appendix B1 and B2.
While many studies examined different components of nursing education, there is not one quality indicator or one warning sign that indicates a program’s quality. Rather, this literature review points to several factors that in combination may serve as metrics for evaluating a program. These vary from components a program may have little to no control over, such as the type of institutional ownership, to the selected clinical experiences the school is able to obtain and afford to students. The evidence is insufficient to lead us to any conclusion. More research is needed in this area. To this end, NCSBN embarked upon a three-part national study to further examine the three topics studied within this review. Those data, along with the work herein, may provide evidence into the development of a guidance document for program approval.
A National Mixed-Methods Study to Identify Quality Indicators and Warning Signs of Nursing Education Program Performance
NCSBN conducted a groundbreaking, national, mixed-methods study to identify evidence-based quality indicators and warning signs of nursing program performance. This comprehensive study comprises three national studies using different methodologies: (1) a national Delphi study, (2) a quantitative 5-year annual report study, and (3) a qualitative 5-year site visit study.
A National Delphi Study to Determine Quality Indicators and Warning Signs of Nursing Education Program Performance
The objective of this Delphi study was to provide data on consensus from experts in nursing education, regulation, and practice regarding nursing education quality indicators, warning signs when programs are beginning to fall below standards, and performance of nursing education programs. Specifically, we aimed to answer the following research questions:
•
What are characteristics/quality indicators of nursing education programs that graduate safe and competent nurses?
•
What are warning signs that indicate a nursing program is falling below the standard of graduating safe and competent nurses?
•
What outcome measures could BONs use to determine whether nursing programs are graduating safe and competent nurses?
Institutional review board approval was obtained from the Western Institutional Review Board.
Methods
The Delphi method assumes group opinion is more valid than individual opinion (
). In this method, generally there are two to four rounds of surveys, with the goal being that the group comes to consensus on issues. Round one is a qualitative round where the participants are asked to provide their views on issues. It is imperative that the questions are clear and understandable by the participants. To this end, it is recommended to pilot the questions with a small group of experts first (
). In round two, the participants rate the factors identified in round one. If there are areas of disagreement, rounds three and four will allow participants to change their minds based on the findings of the group.
indicated that the advantages of this method are (a) gathering expert opinion while providing anonymity to the participants, (b) providing for a controlled and structured process, and (c) allowing for relatively simple statistics to interpret the results. The Delphi method has been used successfully for answering policy questions (
explains that particularly with policy Delphi studies, a 67% agreement is in concert with the threshold vote for Robert’s Rules Online (n.d., Art. VII. Debate.) when the vote addresses an important policy issue. Therefore, for this study, a 67% agreement was set for the threshold. Additionally, the interquartile range, which measures the dispersion of the data and, therefore, the collective judgments of the respondents, was set at 1.0 or below, which is also supported in the literature (
The research team used NCLEX program code data to obtain a list of email addresses for nurse educators of RN-MSN, RN-BSN, RN-ADN, RN-diploma and licensed practical nurse/licensed vocational nurse (LPN/LVN) programs in the United States. A list of email addresses for clinical nurse educators was purchased from the Association for Nursing Professional Development. The list of experts in nursing regulation was obtained by using the education consultant distribution list from the NCSBN membership email address list. Inclusion criteria for educators were as follows
•
Taught master’s entry, BSN, ADN, diploma, or LPN/LVN for at least 2 years
•
If an LPN/LVN educator, must have at least a BSN
•
If an RN educator, must have at least a master’s degree.
Clinical educators were required to have worked with new graduate LPN/LVNs or RNs for at least 2 years. Education consultants were required to have been hired by the BON to regulate nursing programs. In addition, all participants were required to be willing to complete three rounds of surveys about nursing education programs that graduate students who are competent and safe to practice.
By including regulators, educators, and those who supervise new graduates in practice, we were able to include diverse perspectives in this Delphi study. Additionally, with practice readiness being addressed in the literature related to performance outcomes, we wanted the practice perspective. Thus clinical nurse educators who work with new graduates in hospitals were included.
The demographics of the Delphi study participants are presented in Table 2. The demographics were balanced across the sample, except for the highest level of education attained. Whereas 51% of educators and 50% of the regulators had doctorates, only 19% of the clinical nurse educators who work in hospitals did.
Table 2Demographics of Survey Participants in the Delphi Study
Years of Experience Working With New Graduate Nurses
0–2
3
4
3–5
16
23
6–10
12
17
> 10
40
56
Note. ADN = associate degree in nursing; BSN = bachelor of science in nursing; MS = master of science; MSN = master of science in nursing; DNP = doctor of nursing practice; LPN = licensed practical nurse; LVN = licensed vocational nurse.
a Percentages may not total 100 due to rounding.
b Educators with less than 2 years’ experience were excluded and skipped to the end of the survey.
Ten experts in regulation, education, and clinical education (in hospitals) piloted the surveys for clarity, and revisions were made based on their feedback. For example, we originally used the phrase “regulatory quality indicators,” and although the educators and education consultants understood the term, the clinical educators did not. Therefore, we changed it to “characteristics of nursing programs that graduate safe and competent nurses,” which was universally understood.
An introductory email describing the Delphi study was sent to the entire list of clinical nurse educators, educators in nursing programs, and education consultants inviting them to participate. If they met the inclusion criteria and were interested in participating, they were directed via hyperlink to the Qualtrics (Utah) online survey platform asking for demographics and related experience.
The educators in nursing programs (n = 293), clinical nurse educators (n = 125), and education consultants (n = 62) who agreed to participate and completed the demographic survey were sent the first round of Delphi surveys (round one).
In round one, the participants were asked a series of open-ended questions. These included asking the participants to list up to 15 variables they would consider for each of the following:
•
Characteristics/quality indicators of nursing education programs that graduate safe and competent nurses
•
Warning signs that indicate a nursing program is falling below the standard of graduating safe and competent nurses
•
Outcome measures BONs could use to determine whether nursing programs are graduating safe and competent nurses.
All participants had the opportunity to provide qualitative feedback during this round and were allowed 15 responses for each question (
). Participants were allowed 2 weeks to respond to the survey. Reminder emails were sent at specific intervals to those who had not completed the surveys.
Major recurring themes from the first round were initially identified using NVivo 12 Plus software (QRS International). Text search and word frequency queries were run within each of the quality indicators, warning signs, and outcomes categories, and then across all three categories to determine possible themes. Text searches were directed by using the top word frequencies and the predominant themes that came out of the published literature. The possible themes were further content-analyzed by the research team.
R software (R Foundation for Statistical Computing) was used to help validate the themes obtained manually by the research team and by using NVivo. Latent Dirichlet allocation with R (
) was used to categorize comments into naturally occurring themes by examining word frequency. Latent Dirichlet allocation is a statistical unsupervised learning technique that categorizes comments by assigning them to topics where the comments within a topic share more words in common than those in other topics.
A set of major themes emerged from round one (Table 3) for use in round two. In round two, each participant from round one was administered another survey that included the major themes identified. The participants were asked to rate the importance of each theme or variable using a 4-point Likert scale (1 = unimportant, 2 = not too important, 3 = important, and 4 = very important). The 4-point scale is particularly suited to a policy Delphi because it forces the respondents to take a position (
). Participants were again allowed 2 weeks to respond to the survey and reminder emails were sent at specific intervals to those who had not completed the surveys.
Table 3Major Themes Emerging From Delphi Round One
Quality Indicators
1.Evidence-based curriculum that emphasizes quality and safety standards for patient care
2. Evidence-based curriculum that emphasizes critical thinking and clinical reasoning skills
3. Faculty are able to role model professional behaviors
4. Clinical experiences with actual patients that prepare students for the reality of clinical practice
5. Systematic process is in place to address and remediate student practice errors
6. Faculty teaching clinical courses demonstrate current clinical competence
7. Consistent administrative leadership in the nursing program
8. Collaboration between education and practice to enhance readiness for practice
9. Ongoing systematic evaluation of the nursing program
10. Institutional administrative support of the nursing program
11. Consistently has a pattern of NCLEX pass rates that meet set standards
12. Administrative support for ongoing faculty development
13. Significant opportunities for a variety of clinical experiences with diverse populations
14. Consistent full-time faculty, as opposed to reliance on adjunct faculty
15. Quality simulation is used to augment clinical experiences
16. Comprehensive student support services
17. National nursing accreditation
18. Admission criteria emphasize a background in the sciences
Warning Signs
1. Lack of consistent and prepared clinical faculty
2. Limited clinical experiences that do not prepare the students for practice
3. Poor leadership in the nursing program
4. Trend of NCLEX pass rates is inconsistent or decreasing
5. Complaints to the nursing program or board of nursing from employers, students, or faculty
6. Pattern of faculty attrition
7. Pattern of nursing program administrator attrition
8. Unwillingness of health care institutions to host clinical experiences for the nursing program’s students
9. Pattern of student attrition
10. Curriculum is based on “teaching to the NCLEX”
11. Over-reliance on simulation to replace clinical experiences with actual patients
Program Outcome Measures
1. NCLEX pass rates of the nursing program
2. Relationship of the nursing program with its clinical partners
3. Employer satisfaction with the graduates’ readiness for practice
4. Graduate preparedness to practice for an interprofessional environment
5. Graduates’ satisfaction with the nursing program
6. Graduation rates of students in the nursing program
7. Consistency of graduate employment rates with regional data on nurse employment rates
8. History of board of nursing discipline with the graduates of the nursing program
A third round of Delphi surveys was planned but round two yielded such high agreement (see Results) across all groups for all variables that it was deemed unnecessary.
Statistical Analysis
Statistical analysis was conducted using SPSS version 22.0. Simple descriptive statistics were estimated for each item and agreement was estimated by looking at the percentage of respondents who agreed that an item was either important or very important (a Likert rating of 3 or 4). Group differences were examined using a one-way analysis of variance, where post hoc comparisons were used to determine which group or groups differed on rating the item.
Results
Of the 293 educators, 125 clinical educators, and 62 education consultants, 174 educators (59% response rate); 71 clinical nurse educators who work with new graduates (57% response rate); and 50 education consultants, who are hired by BONs and approve nursing programs (81% response rate), completed both rounds of the study.
Results from the second round of the Delphi analysis found excellent agreement, and relatively little dispersion of ratings of importance (Table 4). Percent agreement ranged from 78% to 100%. None of the items had a median rating below 3.00 (important). All of the interquartile ranges (IQRs) were either zero or one. These results met the criteria established for adequate agreement with percent agreement above 67% and all IQR’s at one or below.
Table 4Agreement With Regulatory Quality Indicators Among Participants in the Second Round of the Delphi Analysis
There were some statistical differences between the clinical nurse educators, academic nurse educators, and educational consultants on mean importance for 15 of the 37 characteristics. Most of these findings were because the clinical nurse educators rated items as more or less important than the other two groups. For only one item was the mean score for the differing group below 2.90. This was for the item stating, “Admission criteria that emphasize a background in the sciences,” and even in this case, the clinical nurse educators rated the item as 2.83 in importance, which was still in the “important” range (from 2.5 to 3.5) of the Likert scale.
Discussion
This Delphi study identified 18 quality indicators (characteristics of nursing programs that graduate safe and competent students), 11 warning signs when nursing programs begin to fall below standards, and eight performance outcomes nursing regulatory bodies could measure. The quality indicators fall into the following categories: (a) school leadership and faculty support, (b) consistent and competent faculty, (c) providing quality, hands-on clinical experiences with meaningful collaboration with clinical partners, and (d) having an evidence-based curriculum emphasizing quality and safety and critical thinking/clinical reasoning. While the warning signs are similar to the quality indicators (only the opposite), there are some additional intriguing ones. There were few surprises with the performance outcomes (i.e, NCLEX pass rates, graduation rates, employment rates, etc.), although there were a few that are new.
Quality Indicators
While many of the quality indicators identified are supported by the literature, this national Delphi study lends further credence to the previous findings. Additionally, under each category are some possible indicators that could be used by regulators and educators when evaluating nursing programs. The general category of leadership and faculty support was identified (Table 4, quality indicators 3, 6, 7, 10, and 12).
, in their observations when approving nursing programs in Texas, found faculty development is an important factor for quality programs. Nurse regulators in Illinois (
The highest level of evidence is a randomized controlled trial, followed by a quasi-experimental study, a nonexperimental study, expert opinion, and lastly experiential and nonresearch evidence (
, based on their observations regulating nursing programs, found that a focus on appropriate full-time faculty ratios was important when remediating nursing programs that were falling below state standards.
Related to the curriculum, the Delphi participants strongly agreed on an evidence-based curriculum that emphasizes critical thinking and clinical reasoning skills, as well as one that emphasizes quality and safety standards for patient care (Table 4, quality indicators 1 and 2).
in their mixed-methods study of nine prelicensure RN nursing programs (all levels) with excellent reputations for teaching and learning, concluded that to shift to integrating clinical experience into the classroom, faculty should place more emphasis on clinical reasoning. Others have provided evidence to support clinical reasoning as a quality indicator (
; QSEN, 2019), which is being integrated in undergraduate nursing education.
The Delphi participants identified four indicators for quality clinical experiences in the nursing program (Table 4, quality indicators 4, 8, 13, and 15).
, in their national study of nursing education, highlighted the importance of quality, hands-on clinical experiences as being a strength of the nursing programs in their study.
both called for more hours in clinical experiences, although there have been no studies linking increased numbers of clinical hours to improved educational outcomes. The evidence does, however, support clinical experiences with actual patients that mirror the reality of practice and opportunities for a variety of clinical experiences (
The NCSBN national simulation study: A longitudinal, randomized, controlled study replacing clinical hours with simulation in prelicensure nursing education.
calls for practice to become more innovative in clinical experience, planning for more opportunities in primary and ambulatory care settings, and thus preparing new graduates for cross-continuum practice of the future.
Warning Signs
Some of the warning signs the participants identified were the opposite of the quality indicators (Table 4, warning signs 1, 2, 6, 7, 9, and 10). However, participants identified specific ones, such as an unwillingness for the healthcare institutions to host clinical experiences. If an institution hosts other programs, but refuses one program, it likely is a problem with that program. The literature only alludes to this problem (
in an editorial reporting on observations from U.S. nursing regulatory bodies when nursing programs are beginning to fall below standards. The Delphi participants identified the curriculum being heavily based on the NCLEX as a warning sign, which also has been addressed in the nursing literature as a problem (
Additionally, an interesting warning sign was identified that has not been reported in the literature: over-reliance on simulation to replace clinical experiences with actual patients. This item could be affected by a number of intervening variables, such as the program has a lack of sufficient hands-on clinical experiences or it is increasing its simulation percentage without adhering to accepted simulation guidelines (
The participants were asked to identify outcomes that nursing regulatory bodies could measure. There were a few new ideas, although some of the items addressed those metrics that are already being used and have little evidence to support their being related to the quality of the nursing programs, such as NCLEX pass rates (
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
The National Academies of Sciences, Engineering, and Medicine. (2016). Quality in the undergraduate experience: What is it? How is it measured? Who decides? The National Academies Press.
). History of the U.S. nursing regulatory body’s discipline with the graduates was another metric cited by the participants; however, there would be many intervening variables, such as the practice environment.
Two other metrics identified were the graduate’s satisfaction with the nursing program and employer satisfaction with the graduate. Although some healthcare accreditors evaluate these metrics, nursing accreditors report that these data are often incomplete and difficult for programs to obtain. Two outcomes were reported that have not been reported in the literature: (1) the relationship the nursing program has with its clinical partners, and (2) the graduates’ preparedness to practice for an interprofessional environment. These outcomes relate to the quality of clinical experiences, which was identified as an important quality indicator.
Achieving Consensus
We reached consensus on two rounds with this Delphi study. The piloting of the questions was very important for ensuring the questions were understood uniformly across our sample. We piloted the survey to all three groups in our sample and made many revisions based on the feedback.
Some of the educators and education consultants may have resorted to metrics they commonly use related to either the accreditation or regulatory standards. We particularly saw that with the outcomes that were identified. This likely was not a major factor because the clinical nurse educators, who work with new graduates in practice, are not tied to the national accreditation standards or to state requirements. Therefore, they were more apt to come up with innovative factors that have not been used when assessing programs. Some ideas, not previously cited, did come from the practice educators, and those were then selected as important or very important by the educators and regulators.
Limitations
While these quality indicators, warning signs, and outcomes were identified by experts, it should be noted the metrics are the opinions of experts in the field, which is the lowest level of evidence. Additionally, while our response rate across the two rounds was good (61% overall), a 70% response rate is recommended by some researchers (
). Our response rate, therefore, was acceptable given our large sample, and it probably benefitted by our sending out reminders every 2 days.
Conclusions
NCSBN conducted this Delphi study to learn about expert consensus of quality indicators, warning signs, and performance outcomes. The diverse group of educators, regulators, and clinical educators who work with new graduates agreed on 18 quality indicators, 11 warning signs, and eight outcome measures. While this study lends more support to those metrics that have already been studied, some newer ones have also been identified (such as collecting evidence on the relationship the nursing program has with the facilities they use for clinical experiences or the graduates’ preparedness to work in an interprofessional environment).
Some highlights of this study are that we used three separate methods of qualitative analysis (content analysis done by hand as well as verifying the findings with NVivo and R [Latent Dirichlet Allocation] software), thereby providing a comprehensive and reliable list of quality indicators, warning signs, and performance outcomes. Additionally, by including regulators (education consultants), educators, and those who work with new graduates in practice, our experts provided diverse perspectives and therefore enhanced the breadth of findings.
Acknowledgment
The researchers would like to acknowledge Lou Fogg, PhD, Associate Professor in the Department of Community, Systems and Mental Health Nursing, at Rush College of Nursing in Chicago, for his direction and mentorship in conducting this study and his expertise in statistically analyzing the results.
A Quantitative Analysis of 5 years of BONs Annual Report Documents
This second national study to identify evidence-based quality indicators, warning signs, and performance outcomes is a quantitative analysis examining 5 years’ worth of data from BON annual reports. Specifically, we aimed to answer the following research questions:
•
What nursing education program performance indicators are associated with full approval of a prelicensure nursing education program?
•
What additional factors exist that are associated with prelicensure nursing programs that have lost full program approval?
In a post hoc analysis, in order to uncover all the possible evidence, we asked the same questions, using 80% or higher NCLEX pass rates as the outcome.
Methods
This is a quantitative retrospective cohort study examining 5 years’ worth of data (2012-2017) from U.S. BONs’ annual reports to identify QIs and thereby systematically evaluate nursing education program performance. All 55 U.S. BONs that approve nursing programs, as well as the Board of Regents in New York and the Mississippi Institutions of Higher Learning, were invited to participate. In total, 43 U.S. BONs, including one U.S. territory and geographically diverse states of all sizes and demographics (Figure 1), provided 11,378 annual report documents for inclusion in the study. Institutional review board approval was obtained from the American Institutes for Research (AIR) because they assisted with collecting the data.
Figure 1Geographic Representation of U.S. Boards of Nursing That Participated in the Study
NCSBN staff sent a detailed request to the U.S. BONs that approve nursing programs in February 2018, and a webinar for the BONs was held on February 20, 2018, to answer any questions and provide more information. Annual report and site visit data were requested for the academic years of 2012-2013, 2013-2014, 2014-2015, 2015-2016, and 2016-2017 from all BONs. Data submissions began after the webinar with an April 15, 2018 deadline for data collection. Many BONs needed permission from other entities or their attorneys before submitting their data. Some BONs requested NCSBN send a request to their state agencies. As a result, many BONs could not meet the April 15 deadline, so it was extended to September 24, 2018.
Because of the large quantity of submitted data, an outside vendor, AIR, collected it on a secure database. Each state and territory had its own password-protected link for sending the data. In a few cases, the BONs sent in boxes of their documents for NCSBN or AIR to scan and enter into the database. When all the data were collected, AIR transferred the information to a secure database at NCSBN for data analysis and storage.
Variables
More than 40 variables were reviewed to determine initial eligibility for analysis. Of the more than 40 variables, 25 factors had sufficient sample size for inclusion (a priori threshold of ≥ 1,000 records). Upon further review, 17 variables were selected for the study based on a range of criteria, including valid response values, sufficiently common or similar tracking procedures across boards, etc. Still, these variables presented in varying levels of completeness (some factors had only ≤ 5% missing data, others had as much as ≥ 50%). Valid N totals provide important context when interpreting the results presented in the accompanying descriptive table. Thus, each median estimate and proportion is reported only out of the total number of program entries for which the information could be verified.
Extensive recoding was also applied to a majority of the 17 variables given the disparate means of data collection and coding across all participating BONs. Several quality assurance measures were implemented to ensure accuracy, including redundant coding procedures and substantial peer review. All variable transformations and recode parameters are specified below.
Student Age
Student age was initially tracked as a multi-level ordinal variable. The proportion of enrolled students was reported for each of the following age categories, in years: (a) 17-20, (b) 21-25, (c) 26-30, (d) 31-40, (e) 41-50, (f) 51-60, and (g) ≥ 61. Initial steps were taken to validate the original tracking by ensuring that no single cell value fell outside a predetermined maximum acceptable range, 0.99–1.01, to account for rounding error. Any record that did not meet this eligibility criterion was reviewed and, if it could not be reconciled, was omitted. In addition, to facilitate further modeling, a single response for each program record was selected. The age category that a majority or plurality (if no majority existed) of students fell was chosen for analysis. Still, due to low observed cell counts, each band was further collapsed for analysis until only two age categories remained with sufficient numbers for modeling (17–25 and ≥ 26).
Student Race
Student race was initially tracked as a multi-level categorical variable. The proportion of enrolled students was reported or each of the following categories: (a) Asian, Black, (b) Caucasian, (c) Hispanic, (d) Native American, and (e) Native Hawaiian-Pacific Islander. Initial steps were taken to validate the original tracking by ensuring that no single cell value fell outside a predetermined maximum acceptable range, 0.99–1.01, to account for rounding error. Any record that did not meet this eligibility criterion was reviewed and, if it could not be reconciled, was omitted. In addition, to facilitate further modeling, a single response for each program record was selected to indicate a plurality of Caucasian students (≥ 40%).
Program Director Credentials
Program director credentials were initially tracked as a multi-level categorical variable, which included BA, BS, MSN, DNP, PhD, and various other fields (master of education [MEd], doctorate of education [EdD], “non-nursing master’s”, etc.). To facilitate the analysis, broader “baccalaureate” and “other graduate” fields were created to further collapse otherwise related fields with low observed cell counts. The bulk of the other graduate field included MEd and EdD recipients.
Faculty Qualifications
Faculty qualifications were initially tracked as a multi-level categorical variable. For each of the following categories, the proportion of faculty members holding these education credentials was reported: (a) associate degree, (b) baccalaureate nursing, (c) baccalaureate non-nursing, (d) master’s nursing, (e) master’s non-nursing, (f) DNP, (g) PhD nursing, and (h) PhD non-nursing. Initial steps were taken to validate the original tracking by ensuring that no single row fell outside a predetermined maximum acceptable range, 0.99–1.01, to account for rounding error. Any record that did not meet this eligibility criterion was reviewed and, if it could not be reconciled, was omitted. In addition, to facilitate further modeling, a single response for each program record was selected. The binned category in which a majority or plurality (if no majority) of faculty members fell was chosen for analysis. Due to low observed cell counts, each band was further collapsed for analysis until only two categories remained: “baccalaureate or lower” and “master’s or higher”.
Proportion of Full-Time Faculty
The proportion of full-time faculty was originally scored on both a 0–1 and 0–100 scale. When possible, valid cell values were rescaled (e.g., 34 to 0.34) to a 0–1 scale. Steps were then taken to validate the original tracking by ensuring that no single cell value fell outside a predetermined acceptable range, 0 to 0.99–1.01 (accounting for rounding error). Non-possible values (e.g., > 101) were reviewed and, if they could not be reconciled, were recoded as missing. Given how skewed the raw scores remained, the variable was further collapsed into quartiles.
Student-to-Clinical Faculty Ratio
Raw scores were highly skewed and grouped around a single common value (8); thus, the variable was collapsed into a binary predictor (e.g. ≤ 8 vs. ≥ 9).
Program Age
Raw scores were highly skewed; thus, the variable was further collapsed into quartiles.
Total Enrollment/Maximum Capacity
Separate variables were present for total student enrollment and maximum program capacity. Most of the programs did not have data for both variables. The 28 programs with data for both variables were analyzed, and it was determined that the data were sufficiently comparable and could be combined into a single variable. For the 28 programs containing both values, the data on total enrollment was used. Any record containing a zero-cell value for both criteria was recoded as missing. Given how skewed the raw scores were, the data were then binned into quartiles.
Estimated Graduation Rate
Raw scores were highly skewed; thus, the variable was collapsed into quartiles.
NCLEX Pass Rates
NCLEX pass rates were originally tracked separately from all other data elements. As a result, this information was matched to program data for analysis using unique program codes as the primary key. As ≥ 80% is currently the passing standard used by the majority of U.S. BONs, as well as by the national nursing accreditors, that cut point was selected for analysis purposes.
Number of Program Sites
Given low observed cell counts for higher raw values and the truncated nature of the range, this variable was further collapsed into a binary predictor (e.g., 1 vs. ≥ 2).
Program Status
This variable was not originally embedded in the supporting program documentation. Supplementary secondary searches were conducted to ascertain if the programs under review were either public, private nonprofit, or private for-profit.
Statistical Methods
A descriptive summary of the data included frequencies and proportions for all categorical characteristics, whereas continuous variables were reported using median and IQR estimates (Table 5). Generalized linear mixed-effects models were used to estimate the odds of full approval as a function of univariable faculty demographic and program characteristics. Post-hoc analyses assessing NCLEX pass rates at or above 80% were also investigated. In both instances, full multivariable modeling was pursued only if two criteria were met: (1) specific evidence-driven hypotheses guiding the inquiry and (2) sufficiently robust model samples. In all models, a binomial distribution was specified for the outcome, and logit links were used to estimate the odds ratio for each predictor. To account for the longitudinal structure of the data, random intercepts were allowed for each program and state to account for within-program correlation and the possible influence of common state-level regulations. An alpha error rate of p ≤ .05 was considered statistically significant and all analyses were conducted using SAS 9.4.
Table 5Program Characteristics and Student Demographics as Provided in Annual Report Documents
A total of 11,378 annual reports from the 43 BONs that participated in the study were analyzed by researchers at NCSBN.
Program Demographics Characteristics
The median program age in the sample was 23 years (IQR = 7–33 years) with a median enrollment capacity of approximately 66 students (IQR = 32–123). Summary outcome measures were strong, with a median graduation rate of 70% (IQR = 51–85) and NCLEX pass rate of 87% (IQR = 77–94). Approximately 90% of all programs (n = 9,168) received full program approval during the study period.
A majority of the programs included in the analysis reported more traditional (17-25 years old; n = 497, 61.7%) and largely Caucasian (≥ 40% Caucasian; n = 1,621, 87.3%) student populations. Approximately 90% of programs had directors with evidence of graduate training in place, led by MSN (n = 1,128, 46.2%) and PhD (n = 495, 20.3%) degrees. Similarly, nearly three-quarters of programs (n = 1,115, 72.8%) also reported a majority or, at minimum, a plurality of faculty with an academic credential of MSN or higher.
The median rate of full-time faculty across all programs was 50% (IQR = 34–75%). Between 30% and 40% of all programs reported a student-to–clinical faculty ratio of 8, so data were binned accordingly, resulting in a near even split between those programs with a ratio ≤ 8 (n = 682, 46.8%) and those with ≥ 9 (n = 776, 53.2%). A majority of the programs were accredited (n = 4,738, 68.4%).
Overall, there were fairly even distributions of programs by learning modality (in-person n = 1,004, 43.9%; hybrid n = 750, 32.8%; online n = 534, 23.3%), as well as degree type (LPN/LVN n = 2,556, 42.7%; ADN n = 2,077, 34.7%; BSN n = 1,354, 22.6%). Most programs in the sample were public institutions (n = 5,878, 61.7%), and the median number of program directors was one, with a range of one to seven.
Faculty Characteristics Related to Full Approval
Programs with a majority of graduate-educated faculty were marginally more likely (odds ratio [OR] = 1.82, 95% CI = 0.89–3.73, p = .10) to receive full approval compared to programs with a majority of faculty with a bachelor’s or lower degree. Similarly, programs with a larger proportion of full-time faculty were marginally more likely to receive full approval (p = .08) (Table 6).
Table 6Univariable Binary Logistic Regression Results Examining Faculty Characteristics Related to Program Full Approval Status
Faculty Characteristics
n
OR (95% CI)
p
Program Director Credentials
3,353
.39
Baccalaureate
1.63 (0.89–2.99)
.11
MSN (Ref)
-
DNP
1.17 (0.53–2.51)
.70
PhD
1.19 (0.78–1.83)
.42
Other graduate
0.88 (0.56–1.37)
.56
Faculty Qualifications
1,421
Baccalaureate or lower (Ref)
-
MSN or higher
1.82 (0.89–3.73)
.10
% Full-Time Faculty
4,353
.08
≤ 34 (Ref)
-
35–50
1.46 (1.06–2.01)
.02
51–75
1.46 (1.03–2.06)
.03
> 76
1.34 (0.93–1.92)
.11
Student-to-Clinical Faculty Ratio
879
≤ 8 (Ref)
-
> 9
1.51 (0.76–2.99)
.24
Note. OR = odds rate; CI = confidence interval; MSN = master of science in nursing; DNP = doctor of nursing practice.
After adjusting for degree type, programs with a majority graduate educated faculty were found to be 2.80 times more likely (95% CI = 1.22–6.39, p = .003) to receive full approval compared to programs with a majority of bachelor’s or lower educated faculty (Table 6).
Program Characteristics Related to Full Approval
Programs that are accredited by a national nursing accreditation body were 2.03 times (95% CI = 1.44–2.87) more likely to receive full approval compared to non-accredited programs (p < .001) (Table 7).
Table 7Univariable Binary Logistic Regression Results Examining Program Characteristics Related to Program Full Approval Status
Program Characteristics
n
OR (95% CI)
p
Accreditation
5,913
Not accredited (Ref)
-
Accredited
2.03 (1.44–2.87)
< .001
Learning Modality
2,156
.01
In-person only (Ref)
-
Hybrid
0.92 (0.62–1.35)
.66
Online
0.45 (0.27–0.73)
.001
Degree Type
4,928
.13
LPN/LVN
0.69 (0.46–1.04)
.08
RN – ADN
0.67 (0.44–1.01)
.06
RN – BSN (Ref)
-
Program Age, in y
9,224
< .001
≤ 7 (Ref)
-
8–23
1.66 (1.30–2.12)
< .001
24–32
2.92 (2.24–3.79)
< .001
> 33
2.79 (2.05–3.79)
< .001
Enrollment Capacity
3,371
.01
1–32
0.39 (0.22–0.68)
< .001
33–66
0.66 (0.38–1.14)
.14
67–123
0.58 (0.34–0.99)
.04
> 123 (Ref)
-
Estimated Graduation Rate
1,466
.62
≤ 50% (Ref)
-
51%–70%
0.84 (0.48–1.46)
.54
71%–85%
1.06 (0.58–1.93)
.86
> 85%+
1.23 (0.67–2.28)
.51
NCLEX Pass Rate
8,035
≤ 79% (Ref)
-
> 80%
5.34 (4.36–6.54)
< .001
Number of Program Sites
1,172
1 (Ref)
-
> 2
1.70 (1.04–2.77)
.03
Program Type
8,028
< .001
Private nonprofit
0.73 (0.55–0.96)
.03
Private for-profit
0.29 (0.22–0.38)
< .001
Public (Ref)
-
Number of Program Directors
2,879
0.86 (0.69–1.07)
.17
Note. OR = odds ratio; CI = confidence interval; LPN = licensed practical nurse; LVN = licensed vocational nurse; RN = registered nurse ADN = associate degree in nursing; BSN = bachelor of science in nursing.
Online programs were also 55% (OR = 0.45, 95% CI = 0.27–0.73, p = .001) and 51% (OR = 0.49, 95% CI = 0.27–0.73, p = .001) less likely to receive full approval compared to in-person and hybrid programs, respectively.
Longer standing and larger enrollment/capacity programs were both more likely to receive full approval (p < .001) compared to new programs (both p < .001). Similarly, programs with more than one site were 70% more likely (OR = 1.70, 95% CI = 1.04–2.77) to receive full approval compared to programs with only a single site (p = .03).
Private for-profit programs were 71% (OR = 0.29, 95% CI = 0.22–0.38, p < .001) less likely to receive full approval compared to public programs and 60% (OR = 0.40, 95% CI = 0.28–0.56, p < .001) less likely to receive full approval compared to private nonprofit programs. Private nonprofit programs were also 27% (OR = 0.73, 95% CI = 0.55–0.96, p = .03) less likely to receive full approval compared to public programs. Programs with NCLEX pass rates at or above 80% were 5.34 times more likely (95% CI = 4.36–6.54) to receive full approval compared to programs that fell below that passing threshold (p < .001).
While less pronounced, we also observed several other noteworthy trends. Those included an inverse relationship between higher rates of program director attrition and full program approval (OR = 0.86, 95% CI = 0.69–1.07, p = .17). Similarly, there was an observed trend of BSN programs receiving full program approval at higher rates than both LPN/LVN (OR = 0.69, 95% CI = 0.46–1.04, p = 0.08) and ADN (OR = 0.67, 95% CI = 0.44–1.01, p = .06) programs. These relationships by degree type (LPN/LVN vs. BSN – adjusted OR [AOR] = 0.83, 95% CI = 0.54–1.29, p = .40 and ADN vs. BSN – AOR = 0.71, 95% CI = 0.48–1.07, p = .10) held even after controlling for national accreditation.
Relationship Between NCLEX Pass Rates and Faculty Characteristics
The relationship between ≥ 80% NCLEX pass rates and faculty characteristics is illustrated in Table 8. This was a post hoc analysis carried out to uncover all possible faculty characteristics related to program outcomes.
Table 8Univariable Binary Logistic Regression Results Examining Faculty Characteristics Related to NCLEX Pass Rates
Faculty Characteristics
n
OR (95% CI)
p
Program Director Credentials
2,864
.08
Baccalaureate
1.12 (0.75–1.68)
.57
MSN (Ref)
-
DNP
1.35 (0.82–2.22)
.24
PhD
1.56 (1.14–2.13)
.01
Other graduate
1.21 (0.86–1.70)
.27
Faculty Qualifications
604
Baccalaureate or lower (Ref)
-
MSN or higher
1.26 (0.72–2.20)
.41
% Full-Time Faculty
3,287
.11
≤ 34 (Ref)
-
35–50
1.24 (0.93–1.66)
.14
51–75
1.17 (0.85–1.60)
.33
> 76
1.49 (1.07–2.08)
.02
Student-to-Clinical Faculty Ratio
1,357
≤ 8 (Ref)
-
> 9
0.96 (0.60–1.52)
.85
Note. OR = odds ratio; CI = confidence interval; MSN = master of science in nursing; DNP = doctor of nursing practice.
Programs whose director had a PhD were marginally more likely to have NCLEX pass rates ≥ 80% (p = .08). In addition, there was a trend toward programs with a greater proportion of full-time faculty having NCLEX pass rates ≥ 80% compared to programs with a smaller proportion full-time faculty (p = .11).
Program Characteristics Related to NCLEX Pass Rates
Hybrid programs were 51% (OR = 1.51, 95% CI = 1.09–2.10, p = .01) and 64% (OR = 1.64, 95% CI = 1.03–2.56, p = .03) more likely to have a NCLEX pass rate ≥ 80% compared to in-person and online programs, respectively (Table 9). ADN programs were 45% less likely (OR = 0.55, 95% CI = 0.39–0.78, p < .001) to have an NCLEX pass rate ≥ 80% of 80% or above compared to BSN programs. When the outcome was full approval, the BSN programs trended toward (though it was not statistically significant) being more likely to have full approval than ADN or LPN/LVN programs.
Table 9Univariable Binary Logistic Regression Results Examining Program Characteristics Related to NCLEX Pass Rates
Program Characteristics
n
OR (95% CI)
p
Accreditation
5,148
Not accredited (Ref)
-
Accredited
1.12 (0.89–1.41)
.32
Learning Modality
1,808
.01
In-person only (Ref)
-
Hybrid
1.51 (1.09–2.10)
.01
Online
0.93 (0.60–1.42)
.72
Degree Type
3,902
.003
LPN/LVN
0.75 (0.53–1.06)
.11
RN – ADN
0.55 (0.39-0.78)
< .001
RN – BSN (Ref)
-
Program Age, in y
9,060
< .001
≤ 7 (Ref)
-
8–23
1.83 (1.51–2.21)
< .001
24–32
3.07 (2.52–3.75)
< .001
> 33
3.83 (3.04–4.82)
< .001
Enrollment Capacity
2,221
.20
1–32
0.97 (0.61–1.55)
.90
33–66
0.73 (0.47–1.12)
.15
67–123
0.69 (0.45–1.07)
.10
> 123 (Ref)
-
Estimated Graduation Rate
1,958
.54
≤ 50% (Ref)
-
51–70%
1.26 (0.86–1.86)
.24
71–85%
1.01 (0.67–1.53)
.97
> 85%
1.16 (0.76–1.77)
.50
Number of Program Sites
1,758
1 (Ref)
-
> 2
1.50 (1.04–2.16)
.03
Program Type
8,762
< .001
Private nonprofit
0.67 (0.54–0.84)
< .001
Private for-profit
0.18 (0.15–0.23)
< .001
Public (Ref)
-
Number of Program Directors
2,198
0.77 (0.66–0.91)
.002
Note. OR = odds ratio; CI = confidence interval; LPN = licensed practical nurse; LVN = licensed vocational nurse; RN = registered nurse; ADN = associate degree in nursing; BSN = bachelor of science in nursing.
As when the outcome was full approval, longer-standing programs were more likely to have a NCLEX pass rate ≥ 80% of 80% or above (p < .001) compared to new programs (≤ 7 years).
Also, similar to the outcome of full approval, programs with more than one site were also 50% more likely (OR = 1.50, 95% CI = 1.04–2.16) to have a NCLEX pass rate of ≥ 80% compared to programs with only a single site (p = .03).
Private for-profit programs were 82% (OR = 0.18, 95% CI = 0.15–0.23, p < .001) and 73% (OR = 0.27, 95% CI = 0.21-0.36, p < .001) less likely to have a NCLEX pass rate ≥ 80% compared to public and private nonprofit programs, respectively. Private nonprofit programs were also 33% (OR = 0.67, 95% CI = 0.54–0.84, p < .001) less likely to have a NCLEX pass rate ≥ 80% compared to public programs. As director attrition increases, programs were 23% less likely (OR = 0.77, 95% CI = 0.66–0.91) to have a NCLEX pass rate ≥ 80% (p = .002).
Discussion
A profile of the nursing programs most likely to secure full approval status emerged in the analysis. Specifically, shared characteristics of approved programs included (a) national accreditation, (b) traditional or hybrid modalities, (c) longer-standing programs, (d) higher enrollment capacity, (e) ≥ 80% or above first-time NCLEX pass rates, (f) multiple program sites, and (g) private nonprofit or public institutions. While not statistically significant, there were some observed trends that may also prove noteworthy for BONs. Related to faculty, programs with a majority of graduate-educated faculty were marginally more likely to receive full approval (p = .10), as were those with a greater proportion of full-time faculty (p = .08). Importantly, adjusting for degree type, programs with a majority graduate-educated faculty were ultimately found to more likely to receive full approval (p = .003). There was also a marginal, inverse relationship between program director turnover and full program approval (p = .17), which underscores the potentially important role of administrative stability. Finally, there was evidence that BSN programs received full approval more frequently than either LPN/LVN (p = 0.08) or ADN programs (p = .06).
When using first-time NCLEX pass rates of ≥ 80% as the outcome, many of the findings associated with full program approval were replicated, though there were a few notable differences. For example, in the post-hoc NCLEX analysis, programs were marginally (p = .08) more likely to have NCLEX pass rates ≥ 80% when the program director was PhD educated, as compared to program directors with other graduate and undergraduate credentials. Online programs were significantly less likely to be approved than programs using traditional or hybrid modalities, whereas programs incorporating hybrid learning strategies were significantly more likely to have higher NCLEX pass rates. ADN programs were also significantly less likely to have higher NCLEX pass rates compared to BSN and LPN/LVN programs, as were all programs that experienced greater director attrition (> 3 directors in 5 years). In both instances, degree type and program director attrition were only marginally associated with full program approval. By contrast, enrollment capacity was a significant finding when examining approval status, but not NCLEX pass rates (p = .20). For both full approval and NCLEX pass rates, the percent of full-time faculty (> 35%) was a marginal finding (p = .08; p = .11).
This quantitative study found that programs using hybrid-learning modalities were significantly more likely to have ≥ 80% first-time NCLEX pass rates. Similarly, a seminal, multi-year USDE meta-analysis of more than 1,000 studies with measured student outcomes found that hybrid (or blended) approaches had significantly better outcomes (
). A rigorous national study of nursing education characteristics as compared to NCLEX outcomes found the percentage of full-time faculty was a predictor of NCLEX success. Our quantitative study found that trend with both the full approval and NCLEX pass rates outcomes as well (
), as well as our quantitative study, was that public nursing programs have significantly better outcomes than private nonprofit or private for-profit programs. Both in the
study and our study, the private for-profit programs had significantly poorer outcomes. What is it about the for-profit programs that puts their students at risk? This finding needs to be studied with more depth in the future. Somewhat related, our study found that long-standing programs with more than one site and with higher capacities have significantly better outcomes.
The evidence for quality clinical experiences and simulation was strong in the Delphi study and literature.
, in a seminal mixed-methods, longitudinal study, found that highly performing programs provided quality clinical experiences, emphasizing clinical judgment and reasoning. Similarly,
, in their national study of more than 5,000 new graduates, found that high-performing nursing programs had competent clinical faculty who focused on the development of clinical judgment. However, these factors were not assessed in our quantitative study as those data are not consistently collected in annual reports and therefore are not available for analysis.
Limitations
While a high number of annual and site reports were obtained from diverse U.S. BONs, the data collected were not consistent across all BONs. For example, many BONs did not report estimated graduation rates, numbers of hours for clinical experiences, simulation percentages, use of simulation guidelines, etc. Similarly, there were missing or incomplete data for those characteristics the BONs did track. This gap in data significantly limited the extent of the multivariable modeling possible. Guided by specific hypotheses, targeted multivariable models were generated when sufficiently robust model samples could be confirmed. However, full models assessing all potential factors simultaneously were not possible given the current sample limitations.
Conclusion
This quantitative study of 5-years of BON annual reports provides us with a profile of those nursing programs that meet state approval requirements. Statistically significant characteristics of approved programs and those with ≥ 80% NCLEX pass rates included (a) national accreditation, (b) traditional or hybrid modalities, (c) long-standing programs, (d) higher enrollment capacity, (e) multiple program sites, (f) private nonprofit or public institutions, (g) program director with a PhD, (h) LPN/LVN and BSN programs (as opposed to ADN programs), and (i) no more than three program directors in 5 years. A marginally significant finding was that programs with more than 35% full-time faculty had ≥ 80% first-time NCLEX pass rates and full approval.
A Qualitative Analysis of 5 years of BONs Site Visit Documents
A qualitative study of 5 years’ worth of BONs’ site visit documents was conducted to better understand the qualifiable descriptors of why programs become at risk for failing or do fail. Specifically, we aimed to answer the following research question:
What are the warning signs when programs become at risk for failing or do fail?
Methods
This qualitative descriptive design blended directed-content analysis techniques to generate the findings for this report. Qualitative descriptive designs are the most basic of all approaches to data analysis and seek to identify and describe a phenomenon that is not well understood (
). Considering that what leads to program failures in nursing is not well defined, a descriptive approach was the best methodological match. NCSBN and AIR researchers collected the data while external experts in qualitative research analyzed the data.
Document Sample Inclusion and Exclusion Criteria
The analytical sample provided to the team included 2,853 eligible documents from 40 states (Table 10). For each state, first the number of documents per state was counted. Next, the researchers checked whether files were “readable” (in a compatible file format) according to the MaxQDA software. Documents were then reviewed and sorted according to the inclusion and exclusion criteria, adhering to the principles of best practices of systematic reviews. Documents were included for analysis if they were classified as “site” or “survey” visits. Exclusion criteria were as follows:
•
Self-study reports/plans
•
Letters (e.g., letters of intent, approval letters, etc.)
California has two nursing boards: California Board of Registered Nursing (CA-RN) and California Board of Vocational Nursing and Psychiatric Technicians (CA-VN). Both boards submitted documents.
a Documents were not site visit or survey reports.
b California has two nursing boards: California Board of Registered Nursing (CA-RN) and California Board of Vocational Nursing and Psychiatric Technicians (CA-VN). Both boards submitted documents.
Of the documentation from 40 states in the sample, nine states had no documents that met the inclusion criteria; therefore, this analysis represents data from 31 states. After the inclusion/exclusion criteria were applied, there were 1,278 site visit reports for all RN and LPN/LVN programs eligible for the analysis. The final step in the process was to determine which reports were for programs that were on probation, under review, or did not have full approval. Only these reports would form the dataset for analysis.
Data Analysis
Two researchers used MaxQDA qualitative data analysis software to analyze the documents. Coding occurred through content analysis and context analysis. Content analysis is an iterative coding process whereby codes emerge naturally from the data using a specific intent to guide the process (
). In this case, the intent was issues contributing to low performing schools (poor NCLEX pass rates and approval downgrades). All documents from fully approved programs were randomly reviewed to be sure they didn’t have the same issues, and these issues were unique to the low performing schools.
Researchers also used context analysis that considered the geographic location of the school, the state regulatory context (as specified through websites), and whether the school was classified as urban, suburban, rural, or virtual (
). The team developed a codebook to harmonize their coding (Appendix C). RN and LPN/LVN programs were analyzed separately, and then coding was harmonized and tracked for similarities and distinctions based on entry-level program type.
Figure 2 illustrates how codes emerged during the data analysis process. Researchers tracked the number of codes generated with each document to determine when coding saturation would occur, according to the methods recommended by
. Our process results were similar to those of the study by Hennink et al, which demonstrated that coding saturation occurs between interviews/data sources nine to 11 and minimal codes generated after interview 16. Ours differed only slightly, with saturation occurring later in the coding process, which may reflect the different reporting formats of the states. Themes and categories emerged iteratively from the coding process, and finalized themes and categories represent consensus agreement by the analysts.
Figure 2Number of New Codes Generated by Site Visit Documents
The findings here represent data from 139 survey or site visit reports that formed the final sample for analysis, with 52 reports from RN programs and 87 from LPN/LVN programs. Eighteen states had both RN and LPN/LVN programs represented, and 15 states had only RN programs.
There were several notable observations in the analysis. First, a large number of for-profit programs received citations. This may merit further exploration. Second, “younger” programs (< 10 years in operation) appeared to be at higher risk for failure. Third, more LPN/LVN programs received citations compared to RN programs.
The data comprising the sample allowed us to achieve a level of data saturation that generated themes and categories. Through a summative content analysis, which examined the frequency with which codes were used to analyze the data (
), all codes appeared in the documents with a frequency of 50% or greater. The themes and categories aligned well with Bronfenbrenner’s socioecological model as an organizing framework for why programs become at risk or fail (
In the case of nursing education, the student is the center of development as they are socialized and educated to become nurses within a specific context. Their responses to the environment where they are shaped affect their developmental processes and outcomes. With NCLEX as the final quantitative developmental indicator of an entry-level educational program in nursing, a series of failures on the NCLEX suggests there may be contextual issues affecting students’ performance. The right educational context should be able to ameliorate individual level issues with students who would otherwise confound the outcomes. Effective educational environments are “person centered” with teaching strategies, student services and support, and faculty qualifications aligned to optimize student success. As such, themes and associated categories are presented in alignment with this model in Figure 3. All schools had at least two of these areas where problems had occurred, with faculty and leadership issues being the most common. Failing programs usually had problems at every level, with the state regulatory context dictating the severity of those issues based on local laws..
Figure 3A Socioecological Perspective on Factors Contributing to At-Risk Status of NursingEducation Programs
The findings are presented by the three overarching themes that emerged from the analysis followed by the categorical presentation of findings associated with the theoretical framework.
Emerging Themes
Theme 1: Site Visit Triggers
Site visit triggers are defined as the issue or issues that triggered a review of the program via a site or survey visit. The main signal for a “site visit trigger” was NCLEX pass rates ≤ 80% for 4 or more quarters. The length of time it took to trigger a site visit related to NCLEX performance concerns varied by state regulations. Other site visit triggers were associated with student complaints about the program, clinical site complaints about the program or students, and/or public formal complaints about a program or its graduates.
What triggered the site visit appeared to set the tone of the visit for the surveyor. Reports that involved student or public complaints were more detailed, and assessments appeared more thorough than with only a trend in NCLEX failures.
Theme 2: Administrative Processes Are a Primary Source of Program Vulnerability
Administrative processes are defined as the necessary operational procedures, policies, and resources needed to ensure adequate record keeping for students and faculty, support faculty productivity, and facilitate program leadership. When programs had failing NCLEX rates for more than a year, it was clear that administrative processes had either (a) never existed, (b) not been revised in more than 5 years, (c) were ignored altogether, or (d) had been cut during a reorganization or as part of “efficiency” measures.
Notable hallmarks of poor administrative processes leading to citations by the state include:
•
Poor record keeping of faculty credentials, course evaluations, and student records
•
A general lack of policies and procedures
•
A lack of quality improvement processes around program and curricular evaluations
•
A lack of faculty and student input into policies, procedures, and processes
•
Students failing to receive educational materials (e.g., books, uniforms, software, internet access, syllabi, etc.) at the beginning of the semester.
Most failing programs had at least two of these factors involved when receiving a violation.
Theme 3: Use of Data
Programs that failed to use data to set admission, progression, and student performance standards appeared to have consistent problems. These issues were also persistent over time when programs received a citation for a deficiency and data were not used to address it. It was also a persistent issue when programs failed to conduct adequate self-evaluations. The potential reasons for a lack of use of data include:
•
Lack of administrative competence with interpreting and using data to guide decision making (e.g., statistical process control)
•
Lack of faculty competence with interpreting and using data to set standards
•
No internal statistical support to conduct predictive analyses about what factors influence student performance and may predict NCLEX first-time pass success
•
No resources to contract out data for analysis to inform decision making.
Key areas where data need to be used to facilitate student success include:
•
Student demographics
o
Socioeconomic status
o
English
−
First language
−
Other languages
o
Presence of children younger than 18 years in the home
o
Need to work while attending the program
•
Program admission
o
High school or previous coursework cumulative grade point average
o
SAT or ACT scores (when applicable)
o
Secondary education in the United States
•
Presence of a remediation program prior to start of coursework
These were all indicated in survey reports captured by the surveyor as factors that were influencing the risk for program success or failure.
Categorical Findings Associated With the Theoretical Framework
Student Feedback
Student feedback reported in the summaries provided useful perspectives on sources of problems from a non-faculty viewpoint. From the student perspective, they were readily able to:
•
Differentiate between well-managed versus poorly managed schools
•
Identify lack of program director support
•
Verbalize fear of retaliation from faculty for discussing program concerns
•
Identify a lack of student input into program decision making
•
Highlight school-student communication issues.
Student demographics may also be factors. The most consistent faculty-reported student issues associated with NCLEX performance problems included status of English as a first or other languages speaker and undiagnosed learning disabilities. Poverty and family issues were the next most attributed factors affecting students’ academic performance and NCLEX pass rates. The faculty-cited student issues were reported almost entirely in LPN/LVN program reports.
Faculty Feedback
A persistent theme throughout all reports and the most consistent characteristic of programs becoming at risk or failing was that faculty do not have training in basic pedagogical methods. Reasons why faculty lacked training included (a) lack of bachelor’s or master’s degree–prepared faculty who had undergone any kind of teaching training; (b) faculty who transitioned directly from clinical practice roles and who had little to no experience with precepting; (c) heavy workloads; (d) a lack of ongoing faculty development of substance; and (e) limited new faculty mentorship. Regarding the lack of ongoing faculty development, such development opportunities focused on what leadership deemed important rather than preparing faculty to become teachers or improve skills.
Another persistent faculty theme was “a lack of recent clinical practice experience.” Schools with faculty who had not completed direct patient care within the past 5 years appeared to have outdated teaching approaches, were not in tune with the latest in clinical practice, and often relied less on the use of evidence in their curricula. Given the increased use of technology in the workplace and the increased use of electronic health records that have fundamentally changed the implementation of the hospital nursing role, faculty without recent clinical experience appear to be a liability that places programs at risk.
Curriculum development skills by faculty were also lacking in many programs and appears associated with the level of curricular control faculty were perceived to have by the surveyor. Many failing schools had no overarching philosophy that tied the curriculum together. This resulted in curricular plans that were task centered, often in ways that masked themselves as being competency based.
High faculty turnover or an inability to recruit qualified personnel was also a factor in many schools. An extensive reliance on adjunct faculty to teach all classes is a known issue in higher education quality and nursing programs. Programs that lacked full-time faculty saw problems with NCLEX performance as their quality indicator. Poor compensation (in comparison to full- time clinical nursing positions) and a lack of incentives were both contributors to turnover and recruitment issues. Faculty in for-profit schools often observed that while administrators often had incentives like stock options in the parent company who owned the program, faculty were rarely offered such incentives.
Teaching and Learning Resources
Teaching and learning resources were a critical subtheme for faculty. Even qualified faculty would have trouble doing their jobs if teaching and learning resources were not available or poorly managed. Teaching and learning resources were also tied directly to the leader’s ability to procure them resources for faculty and the organization’s management.
Key teaching and learning resources that appear tied to a program’s risk for failure fell into three categories: (a) teaching resources, (b) physical instructional resources, and (c) quality of materials. Using NCLEX test preparation materials and online supplemental instructional resources with classroom and clinical instruction appeared linked to satisfactory NCLEX pass rates. The brand of these materials did not matter. A survey of programs may produce insights as to which brands are most effective, but it also may be linked to student demographic data.
Physical Instructional Resources
Physical resources include the quality of materials in the simulation laboratory, the quality of other physical instructional resources for teaching and learning, and whether full- or part-time faculty had private office space for student meetings or their own work. Office space for adjunct faculty did not appear significant, but the ability to reserve a conference room to meet with students was important for them. Programs that lacked simulation laboratory accreditation appeared at higher risk for failure. Broken mannequins or equipment, out-of-date materials, and a lack of equipment for medication administration were common issues cited.
The quality of materials is defined as whether teaching materials were prepared and managed according to the course outcomes in the syllabi and were consistent in their design with internal policies. It was not uncommon for site visitors to find that the content of a class did not match the approved course description or outcomes. The more classes with issues, the more likely the program was to have prolonged performance issues on NCLEX. However, it was not always possible to follow up on NCLEX performance at a school level after a probationary citation because of variations in state transparency around problem program reporting.
Leadership of the Nursing Program
Nursing program leadership had three dimensions that appeared to affect the risk of a program failing or falling under review. The first was when the director of the nursing program, through organizational consolidation, was placed in charge of other allied health and/or vocational programs. These added responsibilities often came without the addition of an assistant director who could manage the day- to-day operations of the nursing program. The additional responsibilities detracted from program quality, a factor that was also reflected in student feedback. It is another “symptom” of potentially problematic program management practices.
The second leadership dimension appears to be tied to the degree qualifications of a director. Doctoral-level education appeared to mitigate against a lack of academic administrative experience, though the exact effect of why this level of education appeared to be protective against program failure is not yet clear and would merit further exploration. It may be that individuals with doctorates have more diversity of work experience in general and that the training provided additional skills that facilitated program management. It was clear that directors in charge of programs that did not have a college or university affiliation and whose leaders were only prepared at the master’s level were at greater risk for failure. Because demographic data about these individuals were not available, it was difficult to determine why this finding occurred in the reports; however, it was consistently observed.
The final nursing program leadership issue that arose frequently was when a nurse was not in charge of the program. This could be either due to the position being vacant for a long period or higher administration not thinking a nurse needed to oversee the program despite the regulatory context dictating otherwise. Both factors were more common in for- profit programs than other types.
Educational Organizations
Educational organizations had other specific issues that emerged as distinct categories in the analysis—namely, organizational changes and resources, which could influence program success or failure.
Organizational changes are changes in schools with other degree-granting programs where administration decided to make changes based on economic efficiencies. Sometimes, these changes masked broader financial problems for the parent institution overall. Also, changes could add or decrease responsibilities for nursing faculty. From the reports, it appears that 1 to 3 years after these changes, programs are at risk for changes in NCLEX performance, which increase the risk of probation. The longer performance issues persist after these changes, the more likely the program would transition from probation to failing. These trends likely reflected the nursing program leadership’s ability to navigate existing faculty through the changes or how they managed higher faculty turnover rates that are often associated with organizational change of any kind.
Resources provided by organizations to facilitate nursing education were another factor that was often missing. While often monetary in nature, resources include (but are not limited to) student affairs support, administrative support, libraries, and information technology. How resources were allocated toward clinical learning experiences and clinical sites, including simulation and laboratory supplies, were also important. Problematic programs usually lacked in at least one but sometimes all of these areas.
It is important to note that we observed that for-profit schools appeared to trigger more site visits than nonprofit or state schools. This was especially true for LPN/LVN programs.
State Regulatory Context
It was clear the regulatory context of the program approval of nursing education had a positive effect in terms of holding the programs accountable for standards. This was particularly true related to the minimum requirements for faculty. It held whether or not the program was accredited by a national accrediting agency.
Probationary and failure consequences varied in the length of time schools had to address their deficiencies. Unsurprisingly, shorter periods usually meant increased chance of failure. A shorter period also meant that schools had to rely on obtaining resources to hire consultants to help them address deficiencies.
Without a standardized chart to compare regulatory contexts for nursing education, our ability to compare between states to determine the associations with geographic, socioeconomic, and other factors was limited.
Limitations
Despite the volume of documents that served as the initial sample size, there were a number of problems with file management that may have precluded a larger sample size or fully complete analysis of all site or survey visits. These include:
•
An inconsistent and unstructured file naming and management system
•
Incompatible file types with qualitative data analysis software
•
Improper file format for analysis or optical character recognition
•
Files that were not able to be downloaded for analysis
•
Missing reports for RN or LPN/LVN programs from 10 states.
While every effort was made to ensure that all site visit or survey report documents were included, some may have not been included due to how files were named or classified.
Nonetheless, because of the number of documents included, the researchers are confident of the results because of coder consensus, because only one new code appeared after the 20th document, and because both coders believed they had achieved saturation around 40 to 45 coded interviews.
Conclusion
Considerable and specific data on what happens when nursing programs begin to fail or do fail were found in the site visit documents. Three overarching themes (site visit triggers, administration processes, and the nursing school’s use of data in continuous improvement) were found. Specific findings in the areas of student, faculty, leadership, organization, and state regulatory context were also presented. The issues coalesce nicely with the data found in the literature and our Delphi study.
The site visit study was the first to find that BON approval of nursing education programs is an essential process for protecting the public and maintaining nursing education program standards. This was true regardless of whether the programs were accredited by a national nursing accreditation agency.
SUMMARY
This comprehensive literature review and three-part national study provides substantial evidence-based criteria for identifying quality indicators of successful and high-risk nursing education programs to effectively recommend guidelines for nursing education approval. These criteria include quality indicators and warning signs related to: (a) organizational requirements and processes, (b) program leadership, (c) faculty quality and requirements, and (d) curriculum and clinical program components.
Organizational Requirements, Policies, and Processes
Administrative processes, such as a lack of policies and procedures, were found in both the site visit study and the literature review as being problematic for nursing programs. The literature review, Delphi study, and site visit study all emphasized the importance of collecting data to establish policies and procedures and to evaluate the nursing program based on the data collected. A major theme identified in the site visit study was that programs that fail to collect data to set admission, progression, and student performance standards received downgraded approval statuses.
The site visit study cited key areas where these data could facilitate student success: (a) student demographics, (b) program admission standards, (c) remediation program, and (d) program progression. The literature review and Delphi study supported this finding as well. BONs should expect these data to be collected and acted upon. For example, if the program has a high rate of students with English as a foreign language, they should have resources in place to assist these students. Likewise, if the socioeconomic status of the students means they come from disadvantaged homes where the school systems are not up to par, there should be remediation programs in place. Additionally, the site visit study found that students in underperforming schools often verbalized communication issues and lack of input into program decision making. These findings illustrate the importance of regulators meeting with the students when making a site visit.
The failure of nursing programs to collect data also meant that they were unable to evaluate their programs. The literature review, Delphi study, and site visit study all emphasized the importance of a systematic evaluation of the nursing program based on data collection. Yet, one problem identified in the site visit study was that in underperforming programs, even when data were collected, faculty often are not able to interpret or use data for setting standards. Likewise, faculty may not have access to statistical support when analyzing the data.
In the site visit study, researchers found that in programs that lost approval, policies and procedures had not been revised in more than 5 years or they had been ignored (because of “efficiency” measures). Furthermore, there was no faculty or student input into the policies and procedures.
In programs that lose full board approval, organizational changes made for “economic efficiencies” sometimes mask larger financial problems. BONs will want to find out more about what is behind organizational changes because they can affect faculty workload and responsibilities. As noted in the site visit study, when organizations begin to make organizational changes because of financial problems, they often lose board approval within 1 to 3 years. This is an important consideration for BONs when evaluating programs. Similarly, when the parent organization does not provide the program with sufficient resources, such as student services, libraries, information technology, and adequate clinical facilities and simulation, the programs and students struggle.
In the site visit study, another issue that often triggered further review by a BON was associated with student, clinical site, or formal public complaints. This finding was further substantiated by the Delphi study, which cited complaints to the BON as a warning sign for the program. Given the issues that were uncovered in these site visits, BONs should continue the practice of site visits to programs with a substantial number of complaints.
The annual report and site visit studies and some of the literature demonstrated that nursing program ownership may impact outcomes. Negative outcomes can range from low NCLEX pass rates to the program unexpectedly shutting down while students are enrolled. Data indicate that for-profit schools are at the greatest risk. Public schools and those that are well-established (≥10 years) are the most likely to maintain educational standards set forth by the state. Therefore, new programs require more oversight, as do online programs.
Leadership
Consistent leadership in a nursing program was found to be crucial. The literature review and Delphi study cited program director attrition as a warning sign, whereas the annual report study found frequent director turnover resulted in significant statistical differences in the NCLEX pass rates and differences in program approval status.
Both the site visit study and the annual report study found that when the program director was doctorally prepared, the programs had higher NCLEX pass rates and were more likely to have full approval. In the site visit study, it appears as though the doctoral degree may make up for a lack of academic administrative experience due to the other valuable experiences the individual may bring to the role.
The site visit study was able to delve deeper into leadership of the nursing program and found that when a director is placed in charge of other allied health and/or vocational programs, usually for financial reasons, the program is more at risk due to having the director’s attention diverted by multiple programs. Additionally, the site visit study discovered that program issues arose when a nurse was not in charge of the program. This often happened when the position had been vacant for some time or when administration did not think it necessary. These situations happened more often in for-profit institutions.
Faculty Quality and Requirements
The quality of faculty is at the core of a successful nursing program. Having consistent, full-time faculty (at least 35% full-time faculty, as opposed to adjunct or part-time faculty) in a nursing program predicts full approval and higher NCLEX pass rates according to the annual report study. The literature review also found that the full-time faculty percentage was linked to higher NCLEX pass rates, and the Delphi study reported consistent, full-time faculty as an essential element in a nursing program. The site visit study found high faculty turnover and the inability to recruit qualified faculty were linked to poor NCLEX performance.
Both the annual report and site visit studies demonstrated that a lack of a graduate degree for faculty was linked to less than full approval status. Additionally, as seen in the site visit study, faculty with little training in basic pedagogies was a persistent theme in failing programs. Faculty in programs that were failing often had no training in teaching, having transitioned directly from clinical practice to education. Likewise, they had heavy workloads and limited new faculty mentorship opportunities. The site visit study cited the lack of substantive and ongoing faculty development opportunities as an important element of failing nursing programs. The literature review and Delphi study also cited faculty development as important factors in successful nursing programs.
The literature review, Delphi study, and site visit study all identified current clinical experiences as a critical element of successful nursing programs. The site visit study found that schools where faculty had not provided direct patient care in the past 5 years appeared to have outdated teaching experiences and were not teaching the latest technological advances. There are many ways a program could provide their students with faculty who are clinically competent. They might, for example, develop partnerships with practice, such as dedicated education units, in which the faculty lead the clinical experiences but experienced nurses work directly with their students.
The site visit study also found that in programs that lost approval, faculty did not have the resources needed to teach. For example, faculty lacked the ability to reserve a conference room to meet with students or equipment in their learning and simulation laboratories was missing or broken. Likewise, the quality of the syllabi was often questionable in underperforming programs; for example, it was typical that the content of the classes did not match the course descriptions or outcomes.
Curriculum and Clinical Experiences
The annual report study found hybrid education was a predictor of 80% or higher NCLEX pass rates and that online education predicted the program was less likely to be approved.
Quality and safety concepts, such as the QSEN competencies, were identified in the literature review and Delphi study as important elements of nursing curricula. However, more research on whether integrating QSEN into the curriculum is associated with better outcomes is needed.
According to the site visit study, many failing schools had no overarching philosophy and curricular framework that tied the curriculum together. This resulted in curricula that were task-oriented, masking themselves as being “competency-based.” The literature review and Delphi study highlighted that clinical judgment is critical to thread throughout the curricula but provided little detail on specifically how to do that, though that literature is growing.
The literature review, Delphi study, and site visit study all found quality clinical experiences and simulation to be critical for successful nursing programs. Clinical experiences with actual patients in a variety of clinical settings were found to be important. BONs should evaluate the relationship the program has with its clinical partners, looking for collaboration between the nursing program and practice sites. Programs that lost BON approval often had a limited number of clinical sites, and their parent organizations did not allocate enough resources (such as clinical faculty) toward clinical learning experiences according to the site visit study. Likewise, in weaker programs, supplemental instructional resources (such as videos and online modules) were lacking. The literature review found the following to be important areas to include in clinical experiences: (a) clinical reasoning, (b) delegation, (c) electronic data management, (d) emergency procedures, (e) interprofessional communication, (f) knowledge of pharmacology, (g) leadership, (h) time management, and (i) understanding pathophysiology.
As documented in the literature review, Delphi study, and site visit study, quality simulation is an important element of a successful nursing program and is an important curricular component for BONs to evaluate. The site visit study found the quality of the materials in the simulation laboratory was poor with broken or out-of-date materials in failing programs. Often there was a lack of equipment for teaching medication administration, a critical curricular element. Simulation laboratory accreditation should be mandated for all programs substituting simulation for direct care clinical experiences.
National nursing accreditation of the nursing program is associated with higher NCLEX pass rates, as seen in the literature review, Delphi study, and annual report study, although we are not sure why. It may be that the more seasoned and successful programs seek national nursing accreditation. More research should be conducted to clarify the reasons. While most BSN programs are nationally accredited, only about 53% of ADN programs and 11% of LPN/LVN programs are accredited (
). Currently, about half the BONs require programs to be nationally nursing accredited.
Conclusion
In their missions of public protection, the BONs have called for nursing education quality indicators and warning signs as they approve nursing programs. This literature review and three-part mixed-methods study have provided robust and specific data for developing evidence-based and legally-defensible approval guidelines. From this evidence, a site visit template (Appendix D) was developed for BONs to use when making site visits, and an annual report template (Appendix E) was developed for collecting core data on an annual basis. The annual report template will enable the collection of core, consistent data across the BONs, thus allowing for continuing data analysis and making the guidelines a living document that will change based on new data. Part III presents the approval guidelines.
NCSBN Guidelines for NursingEducation Program Approval
Introductionss
Considering the literature and study evidence presented, NCSBN invited a group of research, education, regulatory, and legal experts (Table 11) to analyze the data together and make recommendations for evidence-based, legally defensible guidelines for nursing regulatory bodies (NRBs) and nursing education programs (Figure 4). It is hoped that these guidelines will increase collaboration between regulators and educators, allow for transparency in the approval process, help NRBs avoid antitrust issues, and provide criteria that allow NRBs to intervene prior to programs falling below standards.
The guidelines allow NRBs to use the evidence-based quality indicators to provide guidance on where the nursing program needs to act. NRBs will also be able to identify warning signs and high-risk programs, from either site visits (Appendix D) or annual reports (Appendix E), and to take action before a program falls below standards. This will enable the BONs to be proactive rather than reactive. The evidence for the quality indicators and warning signs can be found in Table 12. The site visit template (Appendix D) was developed from the evidence and can be used by NRBs during site visits. Additionally, the annual report core data template (Appendix E) was devised from the quantitative data and can be used by BONs to collect critical nursing education data.
Table 12Evidence Supporting Guidelines for Quality Indicators and Warning Signs
Quality Indicators
Evidence
Administrative Requirements
1. The program can provide evidence that their admission, progression, and student performance standards are based on data.
Literature review, qualitative 5-year site visit study
2. Policies and procedures are in place and based on data that have been vetted by faculty and students.
Literature review, qualitative 5-year site visit study
Program Director
1. The program director of an RN program has a doctorate and a degree in nursing.
Literature review, qualitative 5-year site visit study, quantitative 5-year annual report study
2. The program director of a LPN/LVN program has a graduate degree and a degree in nursing.
Literature review, quantitative 5-year annual report study
Faculty
1. At a minimum, 35% of the total faculty (including all clinical adjunct, part-time, or other faculty) are employed at the institution as full-time faculty.
Literature review, Delphi study, qualitative 5-year site visit study, quantitative 5-year annual report study
2. In RN programs, faculty hold a graduate degree.
Literature review, qualitative 5-year site visit study, quantitative 5-year annual report study
3. In LPN/LVN programs, faculty hold a BSN degree.
Literature review, quantitative 5-year annual report study
4. Faculty can demonstrate they have been educated in basic instruction of teaching and adult learning principles and curriculum development. This may include the following: Methods of instruction Teaching in clinical practice settings Teaching in simulation settings How to conduct assessments, including test item writing Managing “difficult” students.
Literature review, qualitative 5-year site visit study
5. Faculty can demonstrate participation in continuing education related to nursing education and adult learning pedagogies.
Literature review, qualitative 5-year site visit study
6. The school provides substantive and periodic workshops and presentations devoted to faculty development.
Literature review, Delphi study, qualitative 5-year site visit study
7. Formal mentoring of new full-time and part-time faculty takes place by established peers.
Literature review, Delphi study, qualitative 5-year site visit study
8. Formal orientation of adjunct clinical faculty.
Literature review, Delphi study, qualitative 5-year site visit Study
9. Clinical faculty have up-to-date clinical skills and have had experience in direct patient care in the past 5 years.
Literature review, Delphi study, qualitative 5-year site visit Study
10. Simulation faculty are certified.
Literature review, qualitative 5-year site visit study
Students
The nursing program should ensure the following are in place to assist students: English as a second language assistance Assistance for students with learning disabilities Necessary books and resources available throughout the program, as well as strategies to help students who cannot afford books and resources Remediation strategies are in place at the beginning of each course and students are aware of how to seek help. This should include processes to remediate errors and near misses in the clinical setting.
Literature review, qualitative 5-year site visit study
Curriculum and Clinical Experiences
1. 50% or more of clinical experience in each clinical course is direct care with patients.
Literature review, Delphi study
2. Variety of clinical settings with diverse patients.
Literature review, Delphi study, qualitative 5-year site visit study
3. Opportunities for quality and safety education integrated into the curriculum, including delegating effectively, emergency procedures, interprofessional communication, and time management.
Literature review, Delphi study
4. Systematic evaluation plan of the curriculum is in place.
Literature review, Delphi study, qualitative 5-year site visit study
Teaching and Learning Resources
1. The simulation laboratory is accredited.
Literature review, qualitative 5-year site visit study
2. Students have access to a library, technology, and other resources.
Literature review, qualitative 5-year site visit study
3. Programs are able to assess students with learning disabilities and tailor the curriculum to meet their needs.
Literature review, qualitative 5-year site visit study
Warning Signs
Evidence
1. Complaints to boards of nursing or other nursing regulatory boards from students, faculty, clinical sites, or the public.
Literature review, Delphi study, qualitative 5-year site visit study
2. Turnover of program directors; more than three directors in a 5-year period.
Literature review, Delphi study, qualitative 5-year site visit study, quantitative 5-year annual report study
3. Frequent faculty turnover or cuts in number of faculty members.
Literature review, Delphi study, qualitative 5-year site visit study, quantitative 5-year annual report study
4. Trend of decreasing NCLEX pass rates.
Delphi, qualitative 5-year site visit study, quantitative 5-year annual report study
5. High-Risk Programs needing additional oversight, such as Prelicensure programs younger than 7 years.
Literature review, qualitative 5-year site visit study, quantitative 5-year annual report study
The quality indicators are categorized into administrative requirements, program director, faculty, students, curriculum and clinical experiences, and teaching and learning resources. They were developed by the expert panel based on the literature review and the Delphi, annual report, and site visit studies (Table 12).
Administrative Requirements
1.
The program has criteria for admission, progression, and student performance.
2.
Written policies and procedures are in place and have been vetted by faculty and students.
Program Director
1.
The program director of an RN program is doctorally prepared and has a degree in nursing.
2.
The program director of an LPN/LVN program has a graduate degree and a degree in nursing.
Faculty
1.
At a minimum, 35% of the total faculty (including all clinical adjunct, part-time, or other faculty) are employed at the institution as full-time faculty.
2.
In RN programs, faculty hold a graduate degree.
3.
In LPN/LVN programs, faculty hold a BSN degree.
4.
Faculty can demonstrate they have been educated in basic instruction of teaching and adult learning principles and curriculum development. This may include the following:
5.
Methods of instruction
a.
Teaching in clinical practice settings
b.
Teaching in simulation settings
c.
How to conduct assessments, including test item writing
d.
Managing “difficult” students.
6.
Faculty can demonstrate participation in continuing education related to nursing education and adult learning pedagogies.
7.
The school provides substantive and periodic workshops and presentations devoted to faculty development.
8.
Formal mentoring of new full-time and part-time faculty takes place by established peers.
9.
Formal orientation of adjunct clinical faculty occurs.
10.
Clinical faculty have up-to-date clinical skills and have had experience in direct patient care in the past 5 years.
11.
Simulation faculty are certified.
Students
1.
The nursing program should ensure the following are in place to assist students:
a.
English as a second language assistance is provided.
b.
Assistance is available for students with learning disabilities.
c.
All students have books and resources necessary throughout the program and strategies are in place to help students who can’t afford books and resources.
d.
Remediation strategies are in place at the beginning of each course and students are aware of how to seek help. This should include processes to remediate errors and near misses in the clinical setting.
Curriculum and Clinical Experiences
1.
At least 50% or more of clinical experience in each clinical course is direct care with patients.
2.
Variety of clinical settings with diverse patients.
3.
Opportunities for quality and safety education integrated into the curriculum, including delegating effectively, emergency procedures, interprofessional communication, and time management.
4.
Systematic evaluation plan of the curriculum is in place.
Teaching and Learning Resources
1.
The simulation laboratory is accredited.
2.
Students have access to a library, technology, and other resources.
3.
Programs are able to assess students with learning disabilities and tailor the curriculum to meet their needs.
Warning Signs
NRBs should intervene early when programs experience the following warning signs. The evidence indicates these programs could be identified either from site visits or annual reports (Table 12). The warning signs include:
1.
Complaints to BONs or other NRBs from students, faculty, clinical sites, or the public.
2.
Turnover of program directors (more than three directors in a 5-year period).
3.
Frequent faculty turnover/cuts in numbers of faculty.
4.
Trend of decreasing NCLEX pass rates.
High-Risk Programs That May Need Additional Oversight
If a program has been in operation for 7 years or fewer, it may need additional oversight because the NRB does not have a history with that program. This recommendation is supported by the literature review, the annual report study, and the site visit study. Additional oversight may include more frequent progress reports related to the number of students, faculty qualifications, stability of the program director, and NCLEX pass rates, in addition to the regularly collected annual reports. If there is concern, the BON may make a focused visit to the program to further assess and possibly make recommendations.
Supportive Evidence for the Approval Guidelines
Evidence supporting how each warning sign and quality indicator is linked to the evidence is presented in Table 12.
APPENDIX A Definition Of Terms
Tabled
1
Annual Report
Contains data the NRBs require from the nursing programs on a yearly basis. These data are not consistent among the NRBs but often consist of faculty, student, and program demographic data; program resources; student outcomes; clinical experiences; curriculum; etc.
Approval of nursing education programs
Official recognition of nursing education programs that go through a systematic approval process implemented by U.S. BONs, thus meeting regulatory standards and being able to make their students eligible to take the nursing licensure examination. In most states, the approval process will be designated as full approval when all requirements are met; conditional or probationary or other designations when some of the requirements are met; or approval removal when programs fail to correct cited deficiencies (adapted from
Number and percentage of degree-seeking students who graduate within the normal program time.
Hybrid learning modality
Blended elements of face-to-face and online instruction.
Metrics
For the purposes of this report, those measures that are used when evaluating the outcomes, quality, and warning signs of nursing programs.
NCLEX-RN predictor examinations
Examinations developed by proprietary companies external (not related) to NCSBN. The examinations are intended to measure the readiness of a graduating nursing student to take the NCLEX-RN. They are also termed exit examinations.
Outcomes
The behaviors, characteristics, qualities, or attributes that learners display at the end of an educational program (
Newly licensed nurses being able to deliver consistent, competent, and safe care in predictable situations, with some guidance in more complex situations (adapted from
Either in face-to-face clinical experiences or in simulation, under the oversight of an experienced clinical instructor, the intentional integration of knowledge, clinical reasoning, skilled know-how, and ethical comportment across the lifespan (adapted from
As adapted from the Agency for Healthcare Research and Quality (https://www.qualityindicators.ahrq.gov/), quality indicators are evidence-based measures of nursing education outcomes that are readily available data to track program performance.
Site visit documents
Documented findings from an NRB’s face-to-face visits with the program, often obtained from interviewing faculty, students, nursing program directors, administrators, and clinical facilities.
Warning signs
Negative indicators when a program is beginning to fall below the standards of graduating safe and competent students.
Note. NRB = nursing regulatory board; BON = board of nursing; NCSBN = National Council of State Boards of Nursing.
APPENDIX B1 The Johns Hopkins Evidence Levels and Quality Ratings
Tabled
1
Source: Dang, D., & Dearholt, S. (2017). Johns Hopkins nursing evidence-based practice: Models and guidelines (3rd ed.). Sigma Theta Tau International. pp. 278-279.
Level and Quality Rating
Description
Level I
Experimental study, RCT, systematic review of RCTs; explanatory mixed-method design with only level I quantitative studies.
Level II
Quasi-experimental study; systematic review of a combination of RCTs and quasi-experimental studies or quasi-experimental studies alone; explanatory mixed-methods with only level II quantitative study
Level III
Nonexperimental study; systematic reviews of combination of RCTs, quasi-experimental, and nonexperimental studies or nonexperimental studies alone; qualitative study; meta-synthesis; exploratory, convergent, or multiphasic mixed-methods studies; explanatory mixed-method design that includes only level III quantitative study.
Quality Rating
For Level I-III Evidence – Quantitative Studies
A
High – Consistent generalizable results; sufficient sample size; adequate control; definitive conclusions; consistent recommendations based on reference to scientific evidence.
B
Good – Reasonably consistent results; sample size sufficient; fairly definitive conclusions; reasonable recommendations based on a fairly comprehensive literature review.
C
Low – Little evidence with inconsistent results; insufficient sample size; conclusions can’t be drawn.
Quality Rating
For Level I-III Evidence – Qualitative Studies
A/B
High/Good Transparency – Documentation justifying decisions; how data were reviewed by others; how themes and categories were formulated. Diligence – Reads and rereads data to check interpretations; seeks opportunity to find multiple sources to corroborate evidence. Verification – The process of checking, confirming, and ensuring methodological coherence. Self-reflection and scrutiny – Being continuously aware of how a researcher’s experiences, background, or prejudices might shape and bias analysis and interpretations. Participant-driven inquiry – Participants shape the scope and breadth of questions; analysis and interpretation give voice to those who participated. Insightful interpretation – Data and knowledge are linked in meaningful ways to relevant literature.
C
Low – Study contributes little to the overall review of findings and have few, if any, of the features listed above.
Level IV
Opinion of respected authorities and/or nationally recognized expert committees/consensus panels. Includes consensus panels and clinical practice guidelines based on scientific evidence.
Quality Rating
For Level IV Evidence
A
High – Officially sponsored by a professional, public, private organization or government agency; documented systematic search strategy; consistent results with sufficient numbers of well-designed studies; criteria-based evaluation of overall strength of studies and conclusions; national expertise; developed/revised within past 5 years.
B
Good – Officially sponsored by professional, public, private, or governmental agency; reasonably thorough and appropriate search strategy; reasonable consistency; sufficient number of well-designed studies; evaluations of strengths and limitations with fairly definitive conclusions; national expertise; developed/revised within past 5 years.
C
Low – Not sponsored by official agencies or organizations; poorly defined search strategies; no evaluation of strengths or weaknesses; insufficient evidence; conclusions cannot be drawn; older than 5 years.
Level V
Experiential and non-research evidence; includes literature integrative reviews, quality improvement, case reports, and opinion of nationally recognized experts based on experiential evidence.
Quality Rating
For Level V Evidence – Organizational Experience (QI, program, or financial evaluation)
A
High – Clear aims and objectives; consistent results across multiple settings; formal QI, financial, or program evaluation methods used; definitive conclusions; consistent recommendations with thorough reference to scientific evidence.
B
Good – Clear aims and objectives; consistent results in a single setting; formal QI, financial, or program evaluation methods; reasonably consistent recommendations with some reference to scientific evidence.
C
Low – Unclear or missing aims and objectives; inconsistent results; poorly defined QI, financial, or program evaluation methods; recommendations cannot be made.
Quality Rating
For Level V Evidence – Integrative review, literature review, expert opinion, case report, community standard, clinician experience, consumer preference)
A
High – Expertise is clearly evident; draws definitive conclusions; provides scientific rationale; thought leaders in field.
B
Good – Expertise appears to be credible; draws fairly definitive conclusions; provides logical argument for opinions.
C
Low – Expertise dubious; conclusions cannot be drawn.
Report from the Connecticut Nursing Collaborative-Action Coalition
To provide gap analysis of new graduates with suggested curricular improvements.
Several gaps were identified, such as that leadership, communication, systems-based practice, academia, and practice did not always speak the same language.
Book presenting a longitudinal mixed-methods study of prelicensure (all levels) RN education programs
To describe the changes in nursing education since the Lysaught study was released 40 years ago. Sponsored by the The Carnegie Foundation for the Advancement of Teaching.
A major finding was that nurses are undereducated for the current demands of practice. Other key findings include: (1) U.S. nursing programs are very effective in forming professional identity and ethical beliefs. (2) Clinical practice assignments provide powerful learning experiences, especially in programs where educators integrate clinical and classroom teaching. (3) U.S. nursing programs are not effective in teaching nursing science, natural sciences, social sciences, technology, and the humanities.
To describe the results of a national survey to a cross section of frontline nurse leaders on new graduate nurse proficiency. - To assess practice readiness.
Identified 36 graduate nurse competencies. - Only 10% of nurse leaders but 90% of faculty thought new graduates were prepared to practice. - Nurse leaders prioritized new graduate improvement needs in a remarkably similar manner. - There can be a relatively consistent approach for addressing new graduate nurses’ greatest improvement needs.
Draft report of entry-level competencies of RNs in Canada
To describe the competencies for entry-level RN practice developed by 11 jurisdictions in Canada.
Competencies were developed under the theme of clinician, communicator, collaborator, advocate, educator, leader, professional, scholar, and coordinator.
To describe a statewide study of 352 new graduates on their educational preparation for practice.
Gaps found included insufficient pharmacology content, lack of management and leadership preparation, and lack of preparation in electronic data measurement. Graduates felt educators did not prepare them for practice but instead to pass the NCLEX. A majority indicated they needed more clinical hours in the program.
To survey 183 new graduates from an accelerated prelicensure master’s program in Australia.
New graduates were weak in leadership, team management, responding to clinical emergencies, and recognizing abnormal laboratory findings; however, 94% felt equally or more prepared than other graduates in their work environments.
To provide overview of nursing education program approval in Ontario.
Standards were developed in the areas of nursing program governance, client and student safety, qualified faculty, entry to practice competencies, clinical learning opportunities, communication with preceptors, examination first-time pass rates, graduates’ rating of their preparation, and preceptors’ rating of students’ readiness to practice.
Commission on Collegiate Nursing Education accreditation manual
To provide standards to nursing education programs obtaining accreditation.
Standards: (1) program quality: mission and governance, (2) institutional commitment and resources, (3) program quality: curriculum and teaching-learning practices, (4) program effectiveness: assessment and achievement of program outcomes.
The IPEDS calculation excludes students who begin college part time, who enroll mid-year, and who transfer from one institution to another. Put another way, IPEDS counts only those students who enroll in an institution as full-time degree-seekers and finish a degree at the same institution within a prescribed period of time.
To describe the QSEN initiative, which includes adapting the Institute of Medicine competencies for nursing.
Definitions of essential features of a competent nurse are provided within each of the QSEN competencies; knowledge, skills, and attitudes for each QSEN competency are identified; the QSEN competencies include patient-centered care, teamwork and collaboration, evidence-based practice, quality improvement, safety, and informatics.
Report of the University of California, Los Angeles Higher Education Research Institute
To provide data-driven information on how to assess institutional graduation rates, emphasizing the importance of taking into account the characteristics of students whom institutions enroll.
Private universities have the highest raw degree completion rates and public 4-year colleges have the lowest. By comparing expected and actual graduation rates, much of the success of private universities in degree completion comes from the strength of the students they enroll. Instead of comparing raw degree completion rates, institutions can be evaluated by how they perform in moving students toward degree completion based on the characteristics and experiences of their students; in this manner, public institutions have lower overall graduation rates but more success in moving the students they enroll toward graduation.
To describe a cross-sectional descriptive study of 84 faculty on failing and grading of students.
The majority of faculty feel confident in their grading practices; however, the findings also suggest that faculty fail to fail students in both didactic and clinical courses.
Using grounded theory, to examine practice readiness from the perspective of nurse unit managers from the acute care practice sector and nursing program coordinators from the education sector.
The authors strongly advocate for nursing programs to have collaborative education-practice partnerships.
To provide data on the 2015 job market and provide insight into education and healthcare trends.
Evidence supports that the changing job market has an impact on employment rates by type of program and type of region. It also presents data on education plans following graduation and student loan load.
Descriptive study of 24 Italian engineering institutions to analyze factors related to academic productivity of universities.
To evaluate the quality of the educational process, account should be taken of the human capital entering the system. Caution should be taken when considering employment rates because they depend on employment conditions, the graduate X years from graduation, duration of the job search, pay at X years from graduation, type of the contract, relevance of the degree, and graduates’ degree of job satisfaction.
To compute a 95% confidence interval for 1,792 nursing program pass rates from 2010-2014 to determine whether programs that met or failed to meet pass rate standards may have done so by accident.
Application of confidence intervals to nursing program pass rates suggests that use of pass rate standards to evaluate nursing program quality may not be appropriate.
To make a case against using only NCLEX pass rates as outcomes standards.
- Multiple choice favors individuals with strengths in low-context applications. - Programs are ensuring NCLEX success by recruiting commercial third parties. - Need multiple indicators—graduation rates along with NCLEX pass rates.
To develop a concept-based learning method for clinical reasoning.
The author developed strategies to integrate clinical reasoning into teaching, such as focusing on documentation, diagnosis, communication, interventions, prioritization, putting it all together, and reflection.
To identify the relationship between NCLEX-RN success and the following: (1) prenursing GPA and final GPA, (2) age and gender, (3) ATI predictor scores.
The study findings include no relationship between NCLEX success and prenursing GPA, final GPA, and gender, but there was a relationship between NCLEX success and age and ATI predictor scores.
The NCSBN national simulation study: A longitudinal, randomized, controlled study replacing clinical hours with simulation in prelicensure nursing education.
Randomized controlled trial to investigate replacing clinical hours with simulation in prelicensure nursing education.
Provides evidence that substituting high-quality simulation experiences for up to half of traditional clinical hours produces comparable educational outcomes and new graduates who are ready for clinical practice.
To describe an integrative review of 50 articles related to the existence, extent, and significance of a preparation-practice gap.
The following three main themes were identified: a preparation-practice gap exists; this gap is costly; and closing the gap will likely rely on changes in undergraduate education and on-the-job remediation, such as nurse residencies and preceptorship programs.
To summarize findings from a review of 88 nursing education self-study reports across a 3-year period (2013–2015) and survey the programs regarding which interventions were most effective.
Three common interventions found to be extremely effective were (1) identifying at-risk students earlier, (2) providing timely remediation for at-risk students, and (3) enforcing program policies.
To conduct psychometric testing on a competency inventory to measure learning outcomes of baccalaureate students using a convenience sample of 599 nursing students.
The Competency Inventory of Nursing Students has satisfactory psychometric properties and could be a useful instrument for measuring learning outcomes of a nursing student. Ethics and accountability were the most important factors contributing to nursing students’ competencies.
Scoping review of the literature to compare the number of clinical practice hours across 4 countries mandated for students in nursing education programs that lead to RN licensure.
There were substantial differences in the requirements from 2,300 hours (U.K.) to no required hours (U.S.). The authors call for more research on clinical education and conclude that it is likely that it is the quality rather than quantity that matters.
Posthire and prestart performance-based development system assessments were administered to more than 5,000 new graduate nurses to assess entry-level competency and practice readiness.
Aggregate baseline data indicate that only 23% of new graduate nurses demonstrate entry-level competencies and practice readiness.
30075-2&id=gr1.jpg){kind=link}
30075-2&id=gr2.jpg){kind=link}
30075-2&id=gr3.jpg){kind=link}
30075-2&id=gr4.jpg){kind=link}