innolitics / rdm Goto Github PK

When executing rdm pull without access token the user is prompted for github username and password. This interaction works fine except that the resulting history.yml file starts with:
GitHub username: change_requests:
instead of the expected:
change_requests:
This causes later commands to fail with errors like:
yaml.scanner.ScannerError: mapping values are not allowed here in "<unicode string>", line 1, column 33: GitHub username: change_requests:
The work around, until this is fixed, is to edit out the leading GitHub username:

It would be nice to have a custom jinja tag for keeping up with definitions and abbreviations.

In particular, we typically use the same definitions in across different documents, but only want to include the relevant definitions in the documents where they are used.

To avoid duplication across different templates, it would be nice if we:

1. Had a central store of definitions (data/definitions.yml) and abbreviations (data/abbreviations.yml).
2. Add a custom tag for dumping out all of the definitions used in the document

The current setup only supports generating a single release document.

• Determine whether we should keep multiple release documents within the repository simultaneously
• If so, figure out how to implement this
• Regardless, figure out how to only display change requests and completed problem reports that occurred within the current release (at the moment, we assume we are grabbing all of them!); the approach we use should work with various project managers

We want to make the variables in a document's YAML front matter available in templates.

It should work the same way Jekyll works:

https://jekyllrb.com/docs/front-matter/#custom-variables

Only, use "document" instead of "page".

Also, be sure to consider what would/should occur if there were a file named data/document.yml. I propose that we simply disallow the use of the file data/document.yml OR we merge it with the YAML front matter, having the front matter take precedence.

Once this change is made, we should update the init_files/documents/release_record.md to set the release_id with a document-level variable.

Use the GitHub API to construct the requirements.yml file and problem-reports.yml files.

Both of these files should have the same format:

1: "description of the requirement or problem report in GitHub flavored markdown"
3: "description of the requirement or problem report in GitHub flavored markdown"
4: "description of the requirement or problem report in GitHub flavored markdown"

We should generate both yaml files using a new Makefile rule and a new rdm subcommand.

This will require:

• Developing a thin interface to the GitHub API (authentication, depagination, issues)
• Processing API responses and producing the .yml file

We should use an existing GitHub API library, e.g. http://pygithub.readthedocs.io/en/latest/introduction.html

In the same way we provide a means for producing PDF documents, we should also provide a mechanism for producing word files.

When running rdm translate on xunit-style results, the flattened_gtest_results doesn't pick up req_ids (I'm not sure how it would?) and therefore when I go to make my documents, I get the following error:

File "./documents/test_record.md", line 36, in block "test_results" | {{ test_name }} | {{ unit_test_record[test_name].result }} | {{ unit_test_record[test_name].req_ids }} | {% if unit_test_record[test_name].note is defined %}{{ unit_test_record[test_name].note }}{% endif %} | jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'req_ids'

How would you plan to include req_ids from translate output? I use pytest for my python code and that provides the record_property fixture which can be used to inject into the XML report, which can then be picked up by translate.

Auditors and regulators require that we keep a revision and approval history for our documents.

§820.40 Document controls.
Each manufacturer shall establish and maintain procedures to control all documents that are required by this part. The procedures shall provide for the following:
(a) Document approval and distribution. Each manufacturer shall designate an individual(s) to review for adequacy and approve prior to issuance all documents established to meet the requirements of this part. The approval, including the date and signature of the individual(s) approving the document, shall be documented. Documents established to meet the requirements of this part shall be available at all locations for which they are designated, used, or otherwise necessary, and all obsolete documents shall be promptly removed from all points of use or otherwise prevented from unintended use.
(b) Document changes. Changes to documents shall be reviewed and approved by an individual(s) in the same function or organization that performed the original review and approval, unless specifically designated otherwise. Approved changes shall be communicated to the appropriate personnel in a timely manner. Each manufacturer shall maintain records of changes to documents. Change records shall include a description of the change, identification of the affected documents, the signature of the approving individual(s), the approval date, and when the change becomes effective.

When using Microsoft word and excel, this process is usually handled by manually updating the revision history and revision number.

For documents stored in git, it probably makes sense to rely on git commits to track this.

Git commit messages, however, will not be accessible to non-developers. To circumvent this:

• Inside the markdown version of the files, simply link to the github history
• Inside the PDF version of the files either (1) include an appendix with a dumped version of the history or (2) generate a second document that contains the full history.

The history ideally will show line-by-line changes, who made the change, when, who approved the change, when, and when the change went into effect. To start, it probably makes sense to say that the change goes into affect once the document is approved... but it would be worth thinking about how this could be handled more flexibly in the future.

After this change, we should no longer need to manually edit the "revision number" in the YAML front-mater. That said, some people may still want to manually update this number and have it displayed in the revision history. Perhaps there is some way we can incorporate this with git tags?? It is a tricky issue.

Before implementation on this is completed, the architecture of the change should be mapped out and discussed with @johndgiese and @yujanshrestha .

Per the 510(k) Format Tips document, item 3:

Provide a title and/or number for any diagrams, drawings, figures, illustrations, photos, charts, or tables that are used. Make sure citations in the text refer to them correctly.

We should figure out a way to support this sort of numbering.

It would be helpful to have categories in the requirements specification. The the requirements could be grouped by category with a header as the category name. For it is helpful to have a category for each software module and then have requirements for that module. Or to separate function and performance.

Pros of removing base templates:

• Don’t need to worry about templates changing if/when you upgrade RDM
• It is easier to customize documents for your needs, vs tracking back through blocks

Cons:

• You won’t be able to take advantage of process upgrades from RDM, but I suspect only us at Innolitics would want this anyway! I think the real thing that people will want to be able to upgrade is the audit system yaml files; this will let people find issues with their documents, and address them as appropriate.

The checkbox lists:

- [ ] Item 1
- [ ] Item 2

are very convenient for verification records. It would be nice if the latex document was able to format these a bit better.

• Update the default templates to use a data/abbreviations.yml and data/definitions.yml
• Add a small set of relevant example definitions

This is a big issue and will require some work to setup in a sufficiently flexible manner. It may require its own "backend" system.

We should also discuss whether it is possible to automatically vs manually tie tests to their requirements.

Finally, it would be great if the document(s) generated from the traceability can include links back to the github code!

One of our global template variables is a boolean called auditor_notes. When this flag is true, we include notes with references back to the 62304 standard which will streamline internal and external auditing.

As a result, this syntax is very common

Some sentence{% if system.auditor_notes %} [62304:6.2.4]{% endif %}.

It would be nice if we could condense this down to:

Some sentence [[62304:6.2.4]].

Note the potentially tricky white-space handling.

It would also be nice if we could color these notes green in the final latex output.

Currently missing numerous references from all 6 62304 checklists

I do not know if this is a feature, bug or neither, but when I run the make command the generated files are generated as ANSI format. This causes an error when trying to convert the generated document (e.g. release/510.md) to a word document using pandoc

Running MS Windows 10 Enterprise, Python 3.8

Currently, the github-flavored-markdown tables, when converted to latex with pandoc, do not line wrap! This is problematic for definitions.

It is useful to include section numbers in markdown files. It also would be nice to have a way to pin the section numbers, so that they don't change over time, and/or you can skip over certain numbers if appropriate to simplify updating references to the document.

A key part of RDM is connecting the commits to issues. The way we do this with Git and GitHub is to place Issue #201 in each commit message. Similar strategies would work with other version control systems and other issue trackers.

It can get annoying having to write Issue #201 into every commit you write. A common strategy to avoid this annoyance is to install githooks that place the issue number into your git commit message for you.

The way we have always done this in the past is to grab the issue number(s) off of the branch name.

If you are working on a branch for issue 201, you would name your branch: feature/201-brief-description. Note that there may be multiple branches for one issue.

The githook would then use the branch name to add Issue #201 to the bottom of your commit messages.

We probably want support for multiple issues as well. E.g. if you are working on 201 and 207, your branch name would be feature/201-207-brief-name. You would then want your commit messages to have Issues #201, #207 (or even Issue #201\nIssue #207.

There are a number of side cases to consider.

1. It would be nice if git commit -m "My message" would create a commit with the message:

My message

Issue #201

2. If you type git commit your editor should open up with the Issue #201 inserted in the commit message draft for you (you may need to change it; very occasionally you will have a commit on a branch that is actually not for Issue 201).

3. There are probably other cases I am not thinking about ... cherry picking---should be fine ... git commit -p ... should probably be fine. Anyway, try to think of other use cases.

We want to add an RDM subcommand that will install these githooks on a project. This sub command should work as follows:

rdm hooks

Should install the git hooks in the current git repository.

We may also want an option to install the hooks in a specified folder. Maybe something like:

rdm hooks specified/git/hooks/folder

The git hooks must be written with just bash, and should not use any other processes (except possibly git).

Some projects will have git hooks for other reasons! This complicates the process of installing them.

Ideally, we would have a way to install our scripts on top of existing scripts, but this may be complicated to setup.

It would be nice if we had some integration tests for this subcommand. Perhaps the integration test would:

1. create a new git repository in a temporary folder
2. install the hooks
3. do a few different git command line uses
4. test that the commit messages are formatted properly

Our GitHub-based project management backends require quite a few GitHub API hits. As a result, git pull can be slow and it is possible to run into the GitHub API limit.

• Research various methods to cache the results
• Implement the best solution

Ideally the solution would only pull down issues or pull requests which have changed since they were last pulled.

We want to add a rdm subcommand that lets us audit a set of documents against a particular version of the standard.

This audit tool will take as its input a yaml file that includes the various required sections in a standard, as well as a set of markdown files to search. It will then look through the auditor notes to cross reference that all of the sections of the standard are addressed.

It should then output to standard output (the option to generate a Microsoft word file would also be good) that can be given to auditors as a starting place for the standard.

This will also facilitate performing a gap analysis whe up upgrading from one version of the standard to another.

Perhaps the subcommand should be called using this rdm gap -s 62304:2005 release/*.md

See checklist for IEC 62304 below:

IEC_62304_Checklist.pdf

Hello,

Despite multiple efforts, I cannot generate the PDF files.
Make seems to stop after .md files generation.
Individual rdm tex and latexmk command works from command line.

My configuration is Windows 10 64 bits ; I tried Cygwin make and Gnuwin32 make with same result.

Any help is appreciated

Setup a public test repository with some data that exercises various corner cases.

Before implementing this, brainstorm with David + Yujan what corner cases we should cover.

It would be nice if we could preserve the key order in YAML files that we dump.

This will probably involve writing a custom "Dumper" class for https://github.com/innolitics/rdm/blob/master/rdm/util.py#L16

Also, we will need to use ordered dicts in various places.

Update the RDM hooks so that, if there is no issue number, instead of discarding the commit message, we preserver it somehow. E.g., in the least we could print it out to the console so you can copy/paste it back in!

At the moment, it is frustrating if you forget an issue number and your commit message is lost as a result.

Currently, rdm gap only prints out the missing checklist items. It would be nice to provide an option that prints out the matching checklist items too. A few requirements of this option include:

• showing the filename and line number of all matches
• showing some context around the matches
• showing the checklist item description from the checklist files

This issue requires two reviews. First, a review of the design specification (including the example output format). Second, a review of the code construction and tests.

I have run through the steps of the "Quick start", and added my own repo to the regulatory folder, by replacing repository: user/repository (here) with my-sample-project in the config.yml file
The regulatory folder in itself is not a git repo, so why does it need access to my source code?
Also when I make changes to the document templates, am I supposed to fork this project and push my changes to the forked repo, or should I create my own, private repo containing the regulatory folder ?

When we include snippets inside documents, the header indent level may be different than what it is in the snippet source. It would be nice to add a filter so we could adjust this:

# My Document

{{ snippets.other_doc | mardown_indentation -1 }}

where the snippet is grabbed from here:

RDOC other_doc
### Header

Header

Stuff here
ENDRDOC

See http://jinja.pocoo.org/docs/2.10/api/#custom-filters

• Write tests for various side cases (e.g., headers in blockquotes)
• Implement
• Add documentation for this (and also our other filters if possible, but you can skip this if you want; you'll probably want to add this somewhere around here https://github.com/innolitics/rdm#first-pass-output )

When I just run the Quick Start instructions from the README, I get the following error:

rdm pull config.yml > data/history.yml
Traceback (most recent call last):
  File "/home/dheater/src/rdm-test1/venv/lib/python3.8/site-packages/rdm/main.py", line 21, in main
    exit_code = cli(sys.argv[1:])
  File "/home/dheater/src/rdm-test1/venv/lib/python3.8/site-packages/rdm/main.py", line 43, in cli
    pull_from_project_manager(args.config)
  File "/home/dheater/src/rdm-test1/venv/lib/python3.8/site-packages/rdm/pull.py", line 14, in pull_from_project_manager
    BackendClass = load_class(config['project_management_backend'])
  File "/home/dheater/src/rdm-test1/venv/lib/python3.8/site-packages/rdm/util.py", line 123, in load_class
    module = import_module(module_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/dheater/src/rdm-test1/venv/lib/python3.8/site-packages/rdm/project_management/__init__.py", line 2, in <module>
    from .github import GitHubIssueBackend, GitHubPullRequestBackend
  File "/home/dheater/src/rdm-test1/venv/lib/python3.8/site-packages/rdm/project_management/github.py", line 6, in <module>
    from github import Github
ModuleNotFoundError: No module named 'github'

Either the Quick Start instructions should use pip install rdm[svg,github] to pull in Github support or maybe the default config.yml shouldn't assume the GithubIssueBackend (if that's possible?)

In particular, it would be nice if images posted in github issues would get pulled into the latex PDFs.

Images linked via URLs work in the markdown files already.

One possible (and probably necessary) solution will be to pull down images in URLs as part of the rdm tex stage, and to place these images inside regulatory/tmp.

I am trying out your software on a dummy project

The documentation and the code standard of your project is awesome, however I have a couple of problems with the history.yml file generation. I am running Windows 10 Enterprise (x64),, with make installed with Chocolatey
Python3.8

I do the following steps (in Anaconda console)

1. rdm init
2. cd regulatory
3. git clone https://github.com/hmogensen/my-sample-project
4. Change line 9 in config.yml to: my-sample-project
5. make

Output:

(base) C:\Users\226077\Documents\sandbox\regulatory>make
rdm pull config.yml > data/history.yml
No access token is stored in the GH_API_TOKEN environment variable.
See https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-linefor details.
Defaulting to username / password for login.
hmogensen
GitHub password (will not echo to console):
Traceback (most recent call last):
File "c:\users\226077\anaconda3\lib\site-packages\rdm\main.py", line 21, in main
exit_code = cli(sys.argv[1:])
File "c:\users\226077\anaconda3\lib\site-packages\rdm\main.py", line 43, in cli
pull_from_project_manager(args.config)
File "c:\users\226077\anaconda3\lib\site-packages\rdm\pull.py", line 15, in pull_from_project_manager
pm_backend = BackendClass(config)
File "c:\users\226077\anaconda3\lib\site-packages\rdm\project_management\github.py", line 21, in init
self.github_repository = self.github_browser.get_repo(self.config['repository'])
File "c:\users\226077\anaconda3\lib\site-packages\github\MainClass.py", line 345, in get_repo
headers, data = self.__requester.requestJsonAndCheck(
File "c:\users\226077\anaconda3\lib\site-packages\github\Requester.py", line 315, in requestJsonAndCheck
return self.__check(
File "c:\users\226077\anaconda3\lib\site-packages\github\Requester.py", line 340, in __check
raise self.__createException(status, responseHeaders, output)
github.GithubException.UnknownObjectException: 404 {"message": "Not Found", "documentation_url": "https://docs.github.com/rest"}

make: *** [Makefile:31: data/history.yml] Error 1
make: *** Deleting file 'data/history.yml'

If I rerun make, only one file from the documents folder will be processed unless I comment out the commands @mkdir -p $(@D) in the Makefile, since the script tries creating a new folder each time. After having performed this modification, running make yields the following output:

Traceback (most recent call last):
File "c:\users\226077\anaconda3\lib\site-packages\rdm\main.py", line 21, in main
exit_code = cli(sys.argv[1:])
File "c:\users\226077\anaconda3\lib\site-packages\rdm\main.py", line 36, in cli
render_template_to_file(config, args.template, context, sys.stdout)
File "c:\users\226077\anaconda3\lib\site-packages\rdm\render.py", line 38, in render_template_to_file
generator = generate_template_output(config, template_filename, context, loaders=loaders)
File "c:\users\226077\anaconda3\lib\site-packages\rdm\render.py", line 50, in generate_template_output
output_line_list = generate_template_output_lines(environment, template_filename, context)
File "c:\users\226077\anaconda3\lib\site-packages\rdm\render.py", line 62, in generate_template_output_lines
source_line_list = _generate_source_line_list(template, context)
File "c:\users\226077\anaconda3\lib\site-packages\rdm\render.py", line 91, in _generate_source_line_list
source = ''.join(generator)
File "c:\users\226077\anaconda3\lib\site-packages\jinja2\environment.py", line 1125, in generate
yield self.environment.handle_exception()
File "c:\users\226077\anaconda3\lib\site-packages\jinja2\environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "c:\users\226077\anaconda3\lib\site-packages\jinja2_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File ".\documents\release_history.md", line 31, in top-level template code
{% for cr in history.change_requests|rejectattr('is_problem_report') %}
File "c:\users\226077\anaconda3\lib\site-packages\jinja2\environment.py", line 471, in getattr
return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'history' is undefined

Our command rdm collect grabs snippets from various files. Sometimes, e.g. in markdown files, it would be nice if the delimeters had the form

<!-- RDOC snippet_id -->
Snippet stuff goes here
<!-- ENDRDOC -->

This lets us delimit snippets without adding garbage to our markdown files.

Perhaps we can replace start_token with start_regex, where the regex has a capturing group for the snippet id. Thus, the HTML comment feature could use:

start_regex = '<!-- RDOC (.*) -->'
stop_regex = '<!-- ENDRDOC -->'

(or something similar).

It would also be nice to be able to grab snippets from python docstrings. E.g.:

class MyClass:
    '''RDOC my_class
    '''
    # stuff goes here

Or, even more ideal, it would be nice if we could grab the python class snippets without needing RDOC at all!

Add windows-latest to the github actions tests workflow matrix and fix any bugs.

As of this writing, there is just one buy related to the tmp_editor pytestt fixture.

A very basic question here: in some of the files in the document folder, there are some sort of tags, like this for instance:
{{ system.project_name }}
Is there a way to specify what this tag will be replaced with in the generated release version of the document?
Or do I manually replace the above string with the name of my project, everywhere it occurs?

Questions:

• How much code/features can we remove if we use RST?
• Can we remove the rdm tex command completely?
• Can we remove our auto numbering support?
• Is the pandoc rst to docx conversion good enough that we don't need any docx customization?
• What are the downsides of using RST?

We will slowly add checklists for a few different FDA Guidance documents. Requirements in guidance documents will likely be less clearly defined than in ISO standards.

It may also be useful to extend the checklist format a bit so we can keep the documents simple. In particular, it may be nice to include lines like:

FDASG:4.2==62304:5.1.3 Optional comment goes here

to indicate that meeting this part of 62304 also meets the related guidance requirement.

Hi, do you have plans to add templates for 62366-1? If not, would you accept contributions towards such templates?

I have not investigated in detail if 62366-1 is suited for your template system, but I am interested in starting that discussion at least.

This issue is just a placeholder, so discuss with David + Yujan before implementing.

We need to produce a nice looking risk management matrix etc. from this document:

https://github.com/innolitics/rdm/blob/master/rdm/init_files/data/risk.yml

The templates have several TODO items. Many are necessary ones aimed at rdm users. They need to expand those to meet their specific situation. But some are really aimed rdm developers marking incomplete areas. The templates should be expanded to implement the latter. It would be useful, as a first step, to clearly mark which TODO is for whom.

The regulatory/data/system.yml file contains two types of information:

• Configuration values (e.g., repository and software_requirements_location.
• Global variables that are used in templates

These two types of information should be split into two separate files.

1. regulatory/_config.yml, which should contain configuration (this is analagous to the similarly named file in Jekyll)
2. regulatory/data/global.yml, which should contain global variables used in the default templates; these values should not have special semantics

I get the following output:
See https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-linefor details.

There should be a space between command-line and for.

This project is a great effort! Thanks a lot for starting it.

I'm giving it a try using the introductory guidelines.
However when I run rdm release, I get the following error:

usage: rdm [-h] <command> ...
rdm: error: argument <command>: invalid choice: 'release' (choose from 'init', 'render', 'tex', 'pull', 'hooks')

I've tried this from within both the root of my repository and the regulatory folder.
I get this error with any invalid command, e.g. rdm this-command-is-not-supported, however release is mentioned in the introductory guidelines, so it should be a valid command.

It would be nice that rdm handles gitlab in the same way it does with github.