Health Canada publishes draft guidance for machine learning-enabled medical devices

On August 30, 2023, Health Canada published draft pre-market guidance for machine learning-enabled medical devices (“Guidance”). The Guidance is open for consultation until October 29, 2023.

This Guidance helps manufacturers that submit an application for a machine ‎learning-enabled medical device (“MLMD”), outlines Health Canada’s expectations for demonstrating the safety and ‎effectiveness of an MLMD throughout its lifecycle, and introduces a mechanism to pre-authorize planned changes to an MLMD to address risks through a pre-determined change ‎control plan (“PCCP”).

Application for a medical device licence

Within Health Canada’s medical device classification system, an MLMD could be considered a standalone software, a medical device that includes software, an in vitro diagnostic device, or a non-in vitro diagnostic device. The Guidance states that manufacturers should clearly state in their applications that the device uses machine learning and, if applicable, that the device has a PCCP. Further, manufacturers should include a justification for the proposed medical ‎device classification applied to the MLMD.

Applications for MLMDs must meet the requirements of the Medical Devices Regulations, including the availability of objective ‎evidence to support, among other things, the ‎safety and effectiveness of the device and any associated claims.‎

Demonstration of safety and effectiveness of an MLMD

Data referred to or used by manufacturers should adequately ‎represent the Canadian population and clinical practice. Further, data ‎used to develop the MLMD or demonstrate a device's safety and ‎effectiveness should reflect the population for whom the device is ‎intended.

Health Canada provides the following guidance with respect to demonstrating the safety and effectiveness of an MLMD:

• Good machine learning practice: This is important when designing, developing, evaluating, deploying, and maintaining an MLMD. An application for an MLMD should include a description of how the manufacturer has adopted good machine learning practice and implemented it throughout the product lifecycle. Health Canada has published Guiding Principles that provide the details of good machine learning practice, including important risk factors. These risk factors include, for example:
  • ensuring reference and training data sets are based upon the best available methods and have standards to address bias inherent in such data sets, and
  • ensuring that the interface of human and AI properly accounts for risks introduced by either human or AI, and ensures effective human interpretability of the model outputs, rather than just the output of the model in isolation.
• Pre-determined change control plan:
  • Concept: This is the documentation that is intended to characterize a device and its bounds, the intended changes to the machine learning system, the protocol for change management, and the change impacts. Modifications listed here must ensure that the device continues to operate within its intended use, and changes should not include changes to the medical conditions, purposes, or uses of an MLMD, as such changes requiringe a medical device licence amendment application prior to implementation.
  • Content: This should be a standalone section in the submission and include references to any application information related to the ‎PCCP that is outside of the PCCP section, such as in the ‎labelling or evidence used to demonstrate safety and ‎effectiveness. The PCCP consists of three components:‎ change description (i.e., documentation that characterizes the device and the proposed changes); change protocol (i.e., the set of policies and procedures that ‎control how changes, as outlined in the change description, will be ‎implemented and managed); and impact assessment (i.e., the potential influence and implications of the changes listed in the PCCP).
• Sex and gender-based analysis plus: This is an ‎analytical process used to assess how a product or initiative may ‎affect diverse groups of people and can be incorporated into ‎the risk management approach used across the lifecycle of the ‎device.‎
• Indications for use, intended use, and contraindications: The intended use or medical purpose ‎should be made clear in the application and all relevant ‎information should be provided.
• Device description: A detailed description of the MLMD, including any machine learning ‎systems used to achieve an intended medical purpose, should be provided.
• Risk management: Descriptions of the risks identified for the MLMD and the associated risk controls in place to eliminate or reduce those risks, the technique used to perform the initial and ongoing risk assessment and the system used for risk level categorization and acceptability, and the results of the risk assessment may be included in the application.
• Data selection and management: Manufacturers should consider providing: descriptions of the training, tuning and test datasets used to ‎develop and evaluate the machine learning system; data inclusion and exclusion criteria and a justification for removing any data; descriptions of techniques used to address data imbalances and a justification; a description of how data integrity was maintained during curation and how data quality and accuracy were ensured; and an explanation of how bias in the dataset was controlled during development.
• Development, training and tuning: Manufacturers should consider providing descriptions of the machine learning development, training and ‎tuning approaches.
• Testing and evaluation: Manufacturers should consider information on the machine learning system performance testing as part of the performance/bench testing or software verification and validation.
• Clinical validation: For applications for Class III or IV MLMDs, manufacturers should provide the appropriate clinical evidence, including clinical validation studies, to support the safe and effective clinical use of their device. For a Class II MLMD, this information should be available upon request.
• Transparency: Transparency requirements should consider the various stakeholders involved in a patient's healthcare across the lifecycle of the device. Manufacturers should provide copies of the instructions for use for the device, including those pertaining to the machine learning system, to Health Canada. Manufacturers should also consider including a description of the processes in place to ensure performance and inter-compatibility of the machine learning system.

Next steps

Comments on the Guidance will be accepted until October 29, 2023. Medical device stakeholders, including manufacturers of Class II to IV medical devices, regulatory representatives, and machine learning experts, are encouraged to make submissions to Health Canada during this consultation period. Health Canada will use the feedback received to finalize the guidance document.

For more information related to Health Canada’s consultation, please do not hesitate to contact the authors or this article with inquiries.