File preview
Lisa Carnahan Computer Scientist Standards Services Group lisa.carnahan@nist.gov
SCC Meeting 7
5/02/2011
1
NIST ISO standards for conformity assessment Example: three domains
◦ HHS EHR Certification Program ◦ Cryptographic Module Validation Program
Product testing handled in a operational context Moving forward
SCC Meeting 7
5/02/2011
2
US DOC/NIST Mission
To promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology … … in ways that enhance economic security and improve our quality of life.
DOC: US Dept of Commerce
Directs US Federal agencies with respect to their use of private sector standards and conformity assessment practices. The objective is for US Federal agencies to adopt private sector standards, wherever possible, in lieu of creating proprietary, non-consensus standards. Directs US NIST “to coordinate Federal, State, and local technical standards activities and conformity assessment activities, with private sector technical standards activities and conformity assessment activities, with the goal of eliminating unnecessary duplication and complexity in the development and promulgation of conformity assessment requirements and measures”. March 1996
Assist U.S. Federal Government Agencies in developing conformity policies and administrative infrastructure Design and assist in the implementation of related conformity assessment programs Develop test methods and tools for industry use in standards development and implementation Do not operationally test products Do not certify products
Certification
3rd party conformity assessment
Supplier’s Declaration
1st party conformity assessment
Independence and Rigor of Conformity Assessment
“demonstration that specified requirements relating to a product, process, system, person or body are fulfilled” ISO/IEC 17000
Used when the risks associated with non-conformity are moderate to high Includes evaluation, compliance decision, attestation of conformity and some form of surveillance or follow up Always conducted by a third party ISO/IEC Guide 65
Used when the critical characteristics can be evaluated via measurement under specified conditions Type test is a test carried out on samples that represent production for the purpose of determining conformity May be an element of a supplier‟s declaration or certification system ISO/IEC 17025
Used to assess and ensure/enhance ongoing conformity assessment body and program for competence, management and technical requirements Used to attain needed confidence in laboratory testing operation and results Used to attain needed confidence in certification system ISO/IEC 17011
• American Reinvestment & Recovery Act (ARRA) • §3001(c)(5) of the PHSA requires the [US HHS] National Coordinator in consultation with the Director of NIST to “keep or recognize a certification program or programs for the voluntary certification of HIT… such program shall include, as appropriate, testing… in accordance with §13201(b) of the HITECH Act” • §13201(b) “…the Director of NIST shall support the establishment of a conformance testing infrastructure… may include a program to accredit independent, non-Federal laboratories to perform testing.”
How Does All This Work?
“Meaningful User
Meaningful Use Objectives & Measures Regulations
of
Certified EHR Technology”
Certified Complete EHR
HIT Certification Programs Regulations ONC-ATCB / ONC-ACB
=
Combination of Certified EHR Modules
Correlated
Certification Criteria
HIT Standards & Standards Regulations
Complete EHR EHRs Modules Certification Criteria
12
http://healthit.hhs.gov/media/MU/n508/MU_SCC_CombinedGrid.pdf
From Recommendation to Certified Products
MU Recommendations from ARRA HIT Policy and Standards Committees
Based on the requirements in the ONC Final Rule, NIST published 42 test CMS Final Rule – Meaningful Use procedures which are in use by the authorized testing & Objectives and Measures certification bodies to test and certify EHR products for ONC Final Rule – Certificationthe Meaningful Use Program
Accredited Testing and Certification Bodies (ATCBs)
Criteria and Standards
Approved Test Procedures ATCB Test Scripts
Certification Commission for Health Information Technology Complete EHR and EHR Modules. Drummond Group, Inc. Complete EHR and EHR Modules. InfoGard Laboratories, Inc. –Complete EHR and EHR Modules. ICSA Labs - Complete EHR and EHR Modules. SLI Global Solutions Complete EHR and EHR Modules. Surescripts LLC EHR Modules: E-Prescribing, Privacy and Security. http://onc-chpl.force.com/ehrcert
ATCB Testing of EHRs ONC Certified Products List
Approved Test Procedures Version 1.1
§170.302.a Drug-drug, drug allergy, formulary checks §170.302.b Drug formulary checks §170.302.c Maintain up-to-date problem list * §170.302.d Maintain Active Medication List * §170.302.e Maintain Active Medication Allergy List §170.302.f.1 Vital Signs * §170.302.f.2 Body Mass Index §170.302.f.3 Growth Charts §170.302.g Smoking Status §170.302.h Incorporate Lab Test Results * §170.302.i Generate Patient Lists §170.302.j Medication Reconciliation §170.302.k Submission to Immunization Registries * §170.302.l Public Health Surveillance (v1.2) §170.302.u General Encryption §170.302.w Accounting of Disclosures
302: All environments 304: Ambulatory 306: In-patient
§170.302.v Encryption when exchanging electronic health information * §170.304.a Computerized Provider Order Entry * §170.304.b Electronic Prescribing * §170.304.c Record Demographics §170.304.d Generate Patient Reminder List §170.304.e Clinical Decision Support §170.304.f Electronic Copy of Health Information * §170.304.g Timely Access §170.304.h Clinical Summaries * §170.304.i Exchange Clinical Information and Patient Summary Record * §170.304.j Calculate & Submit Quality Measures §170.306.a Computerized Provider Order Entry
§170.302.m Education Resources *
§170.302.n Automate Measure Calculations §170.302.o Access Control §170.302.p Emergency Access §170.302.q Automatic Log-off §170.302.r Audit Log §170.302.s Integrity §170.302.t Authentication *Errata also posted
§170.306.b Record Demographics
§170.306.c Clinical Decision Support §170.306.d.1 Electronic Copy of Health Information * §170.306.d.2 Electronic Copy of Health Information §170.306.e Electronic Copy of Discharge Information §170.306.f Exchange Clinical Information and Patient Summary Record * §170.306.g Reportable Lab Results * §170.306.h Advance Directives §170.306.i Calculate & Submit Quality Measures
http://healthcare.nist.gov/use_testing/effective_requirements.html
NVLAP HIT Workshop, April 26, 2011 15
Publishes NIST Testing: Tools/Data/ Procedures Technical Requirements National Coordinator 5
Certified HIT Products List
ONC-Authorized Testing & Certification Bodies
Authorizes 4
Reports Certified Product Info
Complete EHR/ EHR Module Submitted
2 1
Test Report
Vendors/ Self-Developers
3
Certification Application and Agreement
45 CFR Part170 Subpart C is Normative ◦ Informative section of FR is not normative Constraining the test procedures to the functional and interoperable requirements as stated in the FR Describing what needs to be tested and how to perform the testing; not specifying how the Vendor‟s system should perform a particular function Every level of test method is traceable to the criteria Do not create new implicit (or explicit) requirements No explicit requirements for safety or s/w correctness – focus on functionality
NVLAP HIT Workshop, April 26, 2011 17
Exchange Clinical Information & Summary: The test procedure is organized into two sections:
Receive and Display. evaluates the capability to receive and display (render) a patient summary record in the EHR when received in HL7 CCD format and when received in ASTM CCR format.
◦ The patient summary record includes diagnostic test results, problem list, medication list, medication allergy list, immunizations, and procedures. Included in the test procedure is an evaluation of the capability of the EHR to display (render) structured data and vocabulary coded values in human-readable form
Transmit –evaluates the capability to transmit a patient summary record from the EHR in either HL7 CCD or ASTM CCR format as selected by the Vendor.
◦ The patient summary record includes diagnostic test results, problem list, medication list, medication allergy list, immunizations, and procedures. Included in the test procedure is an evaluation of the capability to communicate vocabulary coded values as defined by the referenced standards
NVLAP HIT Workshop, April 26, 2011
18
Informative section describing
◦ Certification criteria as published in 45 CFR Part 170 Subpart C of the Final Rule in the Federal Register on July 13, 2010 ◦ Informative description of how the test procedure is organized and conducted ◦ Standards referenced in the certification criteria
Normative Test Procedure
◦ Describes the required vendor information and test requirements for validating conformance to the criteria and standards
Example Test Data
◦ Provides examples of the test data to be used during the test procedure. The test data sets will be expanded as the test method matures.
Conformance Test Tools
◦ Provides a description and links to the associated conformance test tools, if applicable, to evaluate conformance to the referenced standards.
NVLAP HIT Workshop, April 26, 2011
19
Each subsection provides additional information on: ◦ Required vendor information ◦ Required test procedures ◦ Inspection test guide Traceability is provided in each subsection through a unique numbering sequence
NVLAP HIT Workshop, April 26, 2011
20
Program validates cryptographic modules to FIPS 140-2 (update to FIPS 140-1) Security Requirements
for Cryptographic Modules
Joint effort – NIST and Communications Security Establishment Canada (CSEC) h/w & s/w modules; four levels of increasing requirements (breadth and depth) Derived Test Requirements contain tests Implementaion Guidance – incorporate lessonslearned; new technologies Mature – established in July 1995 1500+ Validations issued!
SCC Meeting 7 5/02/2011 21
Cryptographic Module Validation Program – Model
Test and Validation
NVLAP Accredited FIPS 140-2 CST Lab
Vendor selects a lab; Submits module for testing
Cryptographic Module Vendor
Report Review
Testing and Implementation Guidance
CMVP performs Continuous Quality and Competency Review
Conformance Testing and Report Generation
Module Test Report to CMVP for review
CMVP
Outreach
Federal Agencies Industry Training
Module Validation
Module validation entries at www.nist.gov/cmvp
List of Validated
FIPS 140-2 Modules
Givens:
◦ Requirements in the form of functionality ◦ Requirements often don‟t include software quality, safety, correctness ◦ Regulatory timeframes present challenges ◦ Audience is not testing/certification aware
The resources for re-test/re-cert grow to become the larger expense.
◦ Planning for this in operations is critical After initial market saturation; re-test/re-cert is focus ◦ CMVP – experiences this (1500 validations issued) ◦ HHS EHR Cert Program – still in „initial test‟ in the market
SCC Meeting 7
5/02/2011
23
To the SCC participants: #1 How can the two approaches come together a priori (regulatory timeframe) for markets w/large numbers of vendors & products to certify? (e.g., HIT Certs = ~580 in 7 months) #2 Can s/w cert techniques be used to solve the re-test problem now? #3 Long-term strategy: Challenge to educate decision-makers that s/w process certification will give them success (functionality) for the cost
SCC Meeting 7
5/02/2011
24
SCC Meeting 7
5/02/2011
1
NIST ISO standards for conformity assessment Example: three domains
◦ HHS EHR Certification Program ◦ Cryptographic Module Validation Program
Product testing handled in a operational context Moving forward
SCC Meeting 7
5/02/2011
2
US DOC/NIST Mission
To promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology … … in ways that enhance economic security and improve our quality of life.
DOC: US Dept of Commerce
Directs US Federal agencies with respect to their use of private sector standards and conformity assessment practices. The objective is for US Federal agencies to adopt private sector standards, wherever possible, in lieu of creating proprietary, non-consensus standards. Directs US NIST “to coordinate Federal, State, and local technical standards activities and conformity assessment activities, with private sector technical standards activities and conformity assessment activities, with the goal of eliminating unnecessary duplication and complexity in the development and promulgation of conformity assessment requirements and measures”. March 1996
Assist U.S. Federal Government Agencies in developing conformity policies and administrative infrastructure Design and assist in the implementation of related conformity assessment programs Develop test methods and tools for industry use in standards development and implementation Do not operationally test products Do not certify products
Certification
3rd party conformity assessment
Supplier’s Declaration
1st party conformity assessment
Independence and Rigor of Conformity Assessment
“demonstration that specified requirements relating to a product, process, system, person or body are fulfilled” ISO/IEC 17000
Used when the risks associated with non-conformity are moderate to high Includes evaluation, compliance decision, attestation of conformity and some form of surveillance or follow up Always conducted by a third party ISO/IEC Guide 65
Used when the critical characteristics can be evaluated via measurement under specified conditions Type test is a test carried out on samples that represent production for the purpose of determining conformity May be an element of a supplier‟s declaration or certification system ISO/IEC 17025
Used to assess and ensure/enhance ongoing conformity assessment body and program for competence, management and technical requirements Used to attain needed confidence in laboratory testing operation and results Used to attain needed confidence in certification system ISO/IEC 17011
• American Reinvestment & Recovery Act (ARRA) • §3001(c)(5) of the PHSA requires the [US HHS] National Coordinator in consultation with the Director of NIST to “keep or recognize a certification program or programs for the voluntary certification of HIT… such program shall include, as appropriate, testing… in accordance with §13201(b) of the HITECH Act” • §13201(b) “…the Director of NIST shall support the establishment of a conformance testing infrastructure… may include a program to accredit independent, non-Federal laboratories to perform testing.”
How Does All This Work?
“Meaningful User
Meaningful Use Objectives & Measures Regulations
of
Certified EHR Technology”
Certified Complete EHR
HIT Certification Programs Regulations ONC-ATCB / ONC-ACB
=
Combination of Certified EHR Modules
Correlated
Certification Criteria
HIT Standards & Standards Regulations
Complete EHR EHRs Modules Certification Criteria
12
http://healthit.hhs.gov/media/MU/n508/MU_SCC_CombinedGrid.pdf
From Recommendation to Certified Products
MU Recommendations from ARRA HIT Policy and Standards Committees
Based on the requirements in the ONC Final Rule, NIST published 42 test CMS Final Rule – Meaningful Use procedures which are in use by the authorized testing & Objectives and Measures certification bodies to test and certify EHR products for ONC Final Rule – Certificationthe Meaningful Use Program
Accredited Testing and Certification Bodies (ATCBs)
Criteria and Standards
Approved Test Procedures ATCB Test Scripts
Certification Commission for Health Information Technology Complete EHR and EHR Modules. Drummond Group, Inc. Complete EHR and EHR Modules. InfoGard Laboratories, Inc. –Complete EHR and EHR Modules. ICSA Labs - Complete EHR and EHR Modules. SLI Global Solutions Complete EHR and EHR Modules. Surescripts LLC EHR Modules: E-Prescribing, Privacy and Security. http://onc-chpl.force.com/ehrcert
ATCB Testing of EHRs ONC Certified Products List
Approved Test Procedures Version 1.1
§170.302.a Drug-drug, drug allergy, formulary checks §170.302.b Drug formulary checks §170.302.c Maintain up-to-date problem list * §170.302.d Maintain Active Medication List * §170.302.e Maintain Active Medication Allergy List §170.302.f.1 Vital Signs * §170.302.f.2 Body Mass Index §170.302.f.3 Growth Charts §170.302.g Smoking Status §170.302.h Incorporate Lab Test Results * §170.302.i Generate Patient Lists §170.302.j Medication Reconciliation §170.302.k Submission to Immunization Registries * §170.302.l Public Health Surveillance (v1.2) §170.302.u General Encryption §170.302.w Accounting of Disclosures
302: All environments 304: Ambulatory 306: In-patient
§170.302.v Encryption when exchanging electronic health information * §170.304.a Computerized Provider Order Entry * §170.304.b Electronic Prescribing * §170.304.c Record Demographics §170.304.d Generate Patient Reminder List §170.304.e Clinical Decision Support §170.304.f Electronic Copy of Health Information * §170.304.g Timely Access §170.304.h Clinical Summaries * §170.304.i Exchange Clinical Information and Patient Summary Record * §170.304.j Calculate & Submit Quality Measures §170.306.a Computerized Provider Order Entry
§170.302.m Education Resources *
§170.302.n Automate Measure Calculations §170.302.o Access Control §170.302.p Emergency Access §170.302.q Automatic Log-off §170.302.r Audit Log §170.302.s Integrity §170.302.t Authentication *Errata also posted
§170.306.b Record Demographics
§170.306.c Clinical Decision Support §170.306.d.1 Electronic Copy of Health Information * §170.306.d.2 Electronic Copy of Health Information §170.306.e Electronic Copy of Discharge Information §170.306.f Exchange Clinical Information and Patient Summary Record * §170.306.g Reportable Lab Results * §170.306.h Advance Directives §170.306.i Calculate & Submit Quality Measures
http://healthcare.nist.gov/use_testing/effective_requirements.html
NVLAP HIT Workshop, April 26, 2011 15
Publishes NIST Testing: Tools/Data/ Procedures Technical Requirements National Coordinator 5
Certified HIT Products List
ONC-Authorized Testing & Certification Bodies
Authorizes 4
Reports Certified Product Info
Complete EHR/ EHR Module Submitted
2 1
Test Report
Vendors/ Self-Developers
3
Certification Application and Agreement
45 CFR Part170 Subpart C is Normative ◦ Informative section of FR is not normative Constraining the test procedures to the functional and interoperable requirements as stated in the FR Describing what needs to be tested and how to perform the testing; not specifying how the Vendor‟s system should perform a particular function Every level of test method is traceable to the criteria Do not create new implicit (or explicit) requirements No explicit requirements for safety or s/w correctness – focus on functionality
NVLAP HIT Workshop, April 26, 2011 17
Exchange Clinical Information & Summary: The test procedure is organized into two sections:
Receive and Display. evaluates the capability to receive and display (render) a patient summary record in the EHR when received in HL7 CCD format and when received in ASTM CCR format.
◦ The patient summary record includes diagnostic test results, problem list, medication list, medication allergy list, immunizations, and procedures. Included in the test procedure is an evaluation of the capability of the EHR to display (render) structured data and vocabulary coded values in human-readable form
Transmit –evaluates the capability to transmit a patient summary record from the EHR in either HL7 CCD or ASTM CCR format as selected by the Vendor.
◦ The patient summary record includes diagnostic test results, problem list, medication list, medication allergy list, immunizations, and procedures. Included in the test procedure is an evaluation of the capability to communicate vocabulary coded values as defined by the referenced standards
NVLAP HIT Workshop, April 26, 2011
18
Informative section describing
◦ Certification criteria as published in 45 CFR Part 170 Subpart C of the Final Rule in the Federal Register on July 13, 2010 ◦ Informative description of how the test procedure is organized and conducted ◦ Standards referenced in the certification criteria
Normative Test Procedure
◦ Describes the required vendor information and test requirements for validating conformance to the criteria and standards
Example Test Data
◦ Provides examples of the test data to be used during the test procedure. The test data sets will be expanded as the test method matures.
Conformance Test Tools
◦ Provides a description and links to the associated conformance test tools, if applicable, to evaluate conformance to the referenced standards.
NVLAP HIT Workshop, April 26, 2011
19
Each subsection provides additional information on: ◦ Required vendor information ◦ Required test procedures ◦ Inspection test guide Traceability is provided in each subsection through a unique numbering sequence
NVLAP HIT Workshop, April 26, 2011
20
Program validates cryptographic modules to FIPS 140-2 (update to FIPS 140-1) Security Requirements
for Cryptographic Modules
Joint effort – NIST and Communications Security Establishment Canada (CSEC) h/w & s/w modules; four levels of increasing requirements (breadth and depth) Derived Test Requirements contain tests Implementaion Guidance – incorporate lessonslearned; new technologies Mature – established in July 1995 1500+ Validations issued!
SCC Meeting 7 5/02/2011 21
Cryptographic Module Validation Program – Model
Test and Validation
NVLAP Accredited FIPS 140-2 CST Lab
Vendor selects a lab; Submits module for testing
Cryptographic Module Vendor
Report Review
Testing and Implementation Guidance
CMVP performs Continuous Quality and Competency Review
Conformance Testing and Report Generation
Module Test Report to CMVP for review
CMVP
Outreach
Federal Agencies Industry Training
Module Validation
Module validation entries at www.nist.gov/cmvp
List of Validated
FIPS 140-2 Modules
Givens:
◦ Requirements in the form of functionality ◦ Requirements often don‟t include software quality, safety, correctness ◦ Regulatory timeframes present challenges ◦ Audience is not testing/certification aware
The resources for re-test/re-cert grow to become the larger expense.
◦ Planning for this in operations is critical After initial market saturation; re-test/re-cert is focus ◦ CMVP – experiences this (1500 validations issued) ◦ HHS EHR Cert Program – still in „initial test‟ in the market
SCC Meeting 7
5/02/2011
23
To the SCC participants: #1 How can the two approaches come together a priori (regulatory timeframe) for markets w/large numbers of vendors & products to certify? (e.g., HIT Certs = ~580 in 7 months) #2 Can s/w cert techniques be used to solve the re-test problem now? #3 Long-term strategy: Challenge to educate decision-makers that s/w process certification will give them success (functionality) for the cost
SCC Meeting 7
5/02/2011
24