Add "audit" subcommand about rdm

[orwonthe]

Would this be the logical equivalent of doing a grep against the *.md files with each of the section identifiers and then reporting which grep commands failed?

[johndgiese]

Exactly!

[yujanshrestha]

In addition to the "official" checklist provided in the 62304 standard. There are plenty of unofficial ones that are just as useful, if not more. Check this one out:
FDA-ISO_QMS_audit_checklist_greenlight_guru.xlsx

Here are some requirements for this ticket:

• Checklists shall consist of a list of checklist items
• Checklist items shall contain a human readable description describing what the checklist item is auditing
• Checklist item shall contain a list of rules
• Each rule shall contain a list of references to standards or regulations.
• All references to standards or regulations shall be met for a rule to be accepted
• Only one rule needs to be accepted for the checklist item to be accepted
• All checklist items need to be accepted for the audit sub-command to complete without complaining
• Multiple checklists may exist. RDM audit sub-command shall allow users to select which checklists they choose to conform with. This should be a configuration file.
• Checklists shall be separate yaml files.

Here are some edge cases that I think the above requirements should handle

• Different device classifications. A medical image viewer (Class II) will have fewer checklist items than pacemaker firmware (Class III.) Each project can declare conformity to a different selection of checklists.
• Different countries. A medical device startup in the US first starts off with complying to looser US regulations before trying to sell product in the EU. For example, US only products need only comply with 21 CFR 820.20 whereas EU products need to comply with the more stringent ISO 13485. Similar situation exists for Japan and Canada.
• Different levels of risk tolerance. All things equal, larger medical device companies are expected to spend more resources being proactive about their quality management process. Regulators are more lenient to smaller startups and will usually give them a chance to address gaps. We can have different levels of checklists ranging from a "minimal" version to a "bulletproof" version.
• Different versions of the standard. The US FDA has only recognized the 2012 version of 14971 but the EU has recognized 2016. They are indeed significantly different in a few areas. We can have a checklist for each version.
• I am expecting these checklists to be customized by the user. I am envisioning a startup opting for the "minimal" checklist initially. The FDA auditor will likely find a few gaps during their first audit. The startup should add checklist items so they can ensure they do not expose the same gaps in the future.

[orwonthe]

Scenario: User targets standard X1. They customize generic checklist X1.yaml for their own use, naming it X1_ours.yaml . The standards body updates X1 to X2, adding a few requirements, modifying or clarifying some others, and dropping one or two requirements. What is the process by which user comes up with X2_ours.yaml ?

I am thinking some kind of #include X1.yaml inside of X1_ours.yaml solves the problem by making the latter a kind of delta file that adds and drops items. This implies that checklist items and their rules are named so they can be replaced or removed in such a delta file.

[orwonthe]

Scenario: Someone discovers that X1.yaml has some mistakes, typos or misreads of the X1 standard. So this rdm repository gets a PR to take care of the problem. What just broke?

I am thinking unit tests should have passing and failing examples. So the PR should pass and/or modify those tests.

[yujanshrestha]

I am thinking we should keep it simple and just have them copy the yaml file. They will probably not want the checklist to change from version to version unless they explicitly agree to and merge in the updates.

[orwonthe]

So the command will look like:
rdm gap -s 62304_2005.yaml release/*.md
Or, if using the configuration file, the config.yaml file would contain:

checklists:
  - '62304_2005.yaml'

And the command would look like:
rdm gap release/*.md

[yujanshrestha]

That sounds great!

[johndgiese]

@orwonthe @yujanshrestha here are a few common situations that can occur when performing gap analysis:

You have a standard. It has three sections 1, 2, and 3. Each of these sections have sub sections. E.g., 1.1, 1.2, 1.3. The subsections may in turn have subsubsections.

Within a subsubsection you may have a list of requirements. E.g., 62304 subsubsection 5.1.1 has a list with items a, b, c, d, and e. Each of these needs to show up in the document somewhere, and they are referred to by 5.1.1.a, 5.1.1.b, etc.

Other times, you may not have a list, but just a paragraph. In this paragraph you may have a couple of requirements. While rarer than an explicit list, I have sometimes split the paragraph up by labelling the requirements myself using a, b, c, d, ... It is debatable whether this is desired. In fact, I think it is too confusing and granular for a first stab at the problem.

Another situation that is common, and that I do think we should handle, is that we may only need to fulfill certain parts of a standard. E.g., Innolitics doesn't care about several entire sections of 13485, since we don't manufacture equipment. Thus, in our documentation, we will simply say:

[[13485:17,13485:18,13485:19 We don't manufacture products, so these sections don't apply]]

Thus, the reference to "17" will say all of the subsections within section 17 are met. If we didn't support this, someone would need to list out many subsections just to meet the gap analysis.

[orwonthe]

I was thinking of a way to make our checklists flexible in the sense of handling the variety of medical products. One idea was to have a file that does not show up in the output but gets read by the audit. This file could have keywords like RDM_SOFTWARE_ONLY. A checklist for a hardware only section could then have a usual rule plus a second rule containing just the reference RDM_SOFTWARE_ONLY.

[johndgiese]

FYI @yujanshrestha see the discussion in Slack.

[johndgiese]

Different levels of risk tolerance. All things equal, larger medical device companies are expected to spend more resources being proactive about their quality management process. Regulators are more lenient to smaller startups and will usually give them a chance to address gaps. We can have different levels of checklists ranging from a "minimal" version to a "bulletproof" version.

At least with the ISO standards, I think the leniency will be in how you meet a requirement, and not whether or not you have something in your docs that meets it. I say this, because the ISO auditors basically just go down the list and check that you address everything. As far as I can tell, very little judgement is involved in the process.

For this reason, I doubt the checklist would change based on how risk averse an institution is.

[johndgiese]

I am expecting these checklists to be customized by the user. I am envisioning a startup opting for the "minimal" checklist initially.

I'm a bit surprised by this. Can you provide an example of how a user would want to change the checklist?

[johndgiese]

Nice work!