outburn-il / certificator

FHIR certificator for MoH

certificator

FHIR certificator for MoH

certificator's Issues

Performance tests - concurrent requests response time (Test80)

Background

Article 3.5 in "Technical Guidelines and Requirements ‎Document for FHIR Infrastructure ‎Implementation"
https://www.fhir-il-community.org/%D7%A1%D7%A8%D7%98%D7%99%D7%A4%D7%99%D7%A7%D7%A6%D7%99%D7%94

‎3.5. Performance and availability requirements
Organizational FHIR platform SHALL maintain a minimum annual uptime of 99.95%.‎
Organizational FHIR platform SHALL meet the following performance targets for 100 ‎concurrent FHIR REST API requests:‎
‎-‎ ‎95th percentile of FHIR REST API responses SHALL be returned within 200 milliseconds.‎
‎-‎ ‎99th percentile of FHIR REST API responses SHALL be returned within 2000 ‎milliseconds.‎
Note that these requirements only define the criteria for the FHIR platform's uptime, response ‎times, and concurrency. Data latency requirements (i.e. - time between data updates in the ‎source system and their availability via FHIR REST API) are defined in the [B.3] Information ‎Requirements document.‎

Requirment

• Integrate performance testing capabilities into FUME - use loadtest
• Test loadtest client's ability to output the desired amount of requests - Expose an API in FUME to mimic a server that returns 200 OK on each response

תמיכה בכתיבה לקבצים

כסרטיפיקטור, אני רוצה אפשרות לכתוב את תוצאות המפה כקובץ לתיקיה בדיסק כדי שאוכל

1. לדגום רשימה של ריסורסים באופן אקראי ולשמור את הרפרנסים בדיסק
2. לשלוף את הרפרנסים שהוגרלו ולשמור אותם לדיסק כדי לחסוך שליפות חוזרות ונשנות בהמשך

בנוסף, אני רוצה אפשרות לבחור את סוג הקובץ - JSON או CSV.
עבור שני סוגי הקבצים נדרשת אפשרות להוסיף רשומה חדשה לקובץ כך שלא תדרוס את הרשומות הקיימות.

Support concurrent calls

Allow for sending a requested amount of concurrent requests to support performance testing

Test Patient Compartment - resource aggregation (TestID12)

Test12If a patient compartment is declared (CapabilityStatement.rest.compartment) - check it aggregates resources correctly

Override root route

When performing GET to the root (e.g. localhost:8400) the default FUME health-check json is returned. This should be replaced with a certificator one

Improve resource validation performance

In Laniado validation of a single resource took up to 12 sec.

Talking to Alex he mentioned on his laptop validation took 200 milisec.

Do root cause analysis as to the certification validator performance.

Map not working as expected

The below map and input wont work as expected.
Also, try, try.dev and rambam servers react diffrently

1. fume map
2. fume input
3. fume error (from rambam => baseURL: 'http://ec2-3-124-79-139.eu-central-1.compute.amazonaws.com:52773/csp/healthshare/rambam/fhir/r4',)
4. console error

fume map

`(
/* global parameters */
$pageSize := 10;
$maxRetriesForNewResource := 10;
/*****************/

/* date generator function */
$newRandomDate := function($min, $max){(
$minMs := $toMillis($min);
$maxMs := $toMillis($max);
$diffMs := $maxMs - $minMs;
$randomMs := $random() * $diffMs;
$fromMillis(($minMs + $randomMs),'[Y0001]-[M01]-[D01]');
)};

/* function that calculates the absolute diff between dates in milliseconds */
$dateDiffMillis := function($date1, $date2){($abs($toMillis($date1)-$toMillis($date2)))};

/* function that fetches resources above and below a requested date and returns them sorted by proximity /
$getClosestResources := function($resourceType, $date, $searchParam, $elementName) {(
/ get pages from above and below the date /
$geBundle := $search($resourceType, {$searchParam: 'ge' & $date, '_sort': $searchParam, '_count': $pageSize, '_elements': $elementName});
$leBundle := $search($resourceType, {$searchParam: 'le' & $date, '_sort': '-' & $searchParam, '_count': $pageSize, '_elements': $elementName});
/ sort the results by absolute diff from the selected date */
[$geBundle.entry.resource, $leBundle.entry.resource]^($dateDiffMillis($date, $lookup($, $elementName)));
)};

/* recursive function that adds a single new resource to an accumulating list /
/ educational note: /
/ note that the self-call is at the final step of the function. /
/ this is called "tail recursion" and it helps prevent stack overflow. /
/ see: https://docs.jsonata.org/programming#tail-call-optimization-tail-recursion /
$addResource := function($resourceType, $searchParam, $elementName, $accumulating, $iterationCounter){(
/ reset counter to 1 if not stated otherwise /
$iterationCounter := $exists($iterationCounter) ? $iterationCounter : 1;
/ initialize an empty list if accumulator not passed /
$accumulating := $exists($accumulating) ? $accumulating : [];
/ generate a new random date /
$randomDate := $newRandomDate($minDate, $maxDate);
/ get list of patients closest to the random date by birthday, sorted by proximity /
$searchResults := $getClosestResources($resourceType, $randomDate, $searchParam, $elementName);
/ get the closest one not collected yet /
$selection := $searchResults[$not(id in $accumulating)][0];
/ check the results /
$exists($selection) ? (
/ found a new patient - add its id to the list and return the updated list /
[$accumulating, $selection.id]
) : (
/ no new patient found for this date /
/ if we have reached max iterations - stop looking /
$iterationCounter >= $maxRetriesForNewResource ? (
/ return accumulated list as-is /
$accumulating
) : (
/ not reached max iterations - try again with a new random date /
$addResource($resourceType, $searchParam, $elementName, $accumulating, $iterationCounter + 1)
/ ^ this is the "tail call" */
)
)
)};

/* wrapper function that does the collection of the list of resources /
$collectResources := function($resourceType, $amountToCollect, $minDate, $maxDate, $searchParam, $elementName){(
/ iteration function that adds resources recursively, as long as: /
/ 1. the previous step succeeded in finding a new resource /
/ 2. the requested amount hasn't been collected yet /
$iterator := function($currentList, $previousList){(
$exists($currentList) and ($count($currentList) = $count($previousList) or $count($currentList) >= $amountToCollect) ? (
/ the stop condition is met - return list and stop iterating /
$currentList
) : (
/ try to add one patient and continue /
$updatedList := $addResource($resourceType, $searchParam, $elementName, $currentList);
/ tail call /
$iterator($updatedList, $currentList)
)
)};
/ call the iterator with empty initial lists */
$ids := $iterator();

/* transform the list of resource id's to a list of references (ready for $resolve) */
$ids.($join([$resourceType, $], '/'))

)};

/call the wrapper function with relevant keys, values will come from input/
/$collectResources(resourceType, amountToCollect, "2000-01-01", "2020-01-01", searchParam, elementName)/

/* testing the function: */
$minDate := '1950-01-01';
$maxDate := $now();
$collectResources('Patient', 7, $minDate, $maxDate, 'birthdate', 'birthDate');
)`

fume input
{ "resourceType": "Patient", "amountToCollect": 7, "minDate": "2000-01-01", "maxDate": "2020-01-01", "searchParam": "birthdate", "elementName": "birthDate" }

fume error
{ "__isFumeError": true, "message": "Request failed with status code 500", "code": "ERR_BAD_RESPONSE", "name": "AxiosError", "token": "search", "cause": "", "position": 4489 }

console error
_header: 'GET /csp/healthshare/rambam/fhir/r4/Patient?_count=10&birthdate=ge1953-10-01&_sort=birthdate&_elements=birthDate HTTP/1.1\r\n' + 'Accept: application/fhir+json;fhirVersion=4.0\r\n' + 'Content-Type: application/fhir+json;fhirVersion=4.0\r\n' + 'User-Agent: axios/1.6.7\r\n' + 'Accept-Encoding: gzip, compress, deflate, br\r\n' + 'Host: ec2-3-124-79-139.eu-central-1.compute.amazonaws.com:52773\r\n' + 'Connection: keep-alive\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'GET', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/csp/healthshare/rambam/fhir/r4/Patient?_count=10&birthdate=ge1953-10-01&_sort=birthdate&_elements=birthDate',

Allow for saving maps and calling between maps

As a user, I would like to be able to treat a map as a function + save maps to a backed-up location (such as one drive folder)

Test RI (referential integrity) - check for resolvability (Test34)

Requirment
For each reference pointing to the tested server in the sample (i.e - excluding references to other servers), check for resolvability, that is IF a GET of the referenced resource yields a 200 HTTP code AND the expected resourceType THAN the test is successful.

Implementation

• Create a list of reference paths and initially populate from base only
• Extract all listed paths and loop on non blank results with GET
• Create target file
• resourceID
• resourceType
• path
• reference.value
• result
• For each GET, document the result in the results file

Test for PHI exposure in resource ID - All resources, explicit (Test30)

Requirement
Resource IDs may be exposed on GET over HTTP and hence should not include protected health information (PHI) found in the Patient.identifier (Israeli ID number, passport number etc)

We assume:

1. If identifiers are used in resource IDs then this will occur in almost 100% of instances hence a small sample should suffice for detection.
2. Identifiers may be combined with other strings (prefixed by resourceType, combined with primary key etc) hence ids should be checked if they CONTAIN the identifiers (anywhere in the id).
3. For some identifiers we have prior knowledge about how they may show slightly transformed in Resource.id - for example, on Israeli ID number (Patient.identifier:il-id)

• Leading zeros may be trimmed
• A separating character such (a dash or other) can be added between the control digit and the id number
  Hence, for some identifiers,

Implementation

• Randomly sample N (small 10-20) patients (can be out of the patients sampled for certification).
• For sampled patients, extract all resources from Compartment Patient
• Check if Patient.identifier.value itself or any variation as mentioned in article 3 of the assumption above is contained in any of the resource ids in the compartment (including Patient itself).

Analyse dateTime/Period/date/time elements

Requirment
Allow for temporal analysis by extracting relevant paths into a file and then analyzing it in the dashboard.

Implementation
All relevant dateTime/Period/date/time elements (from base or profiles) will be listed in an auxiliary file.
Currently, only elements defined in the base will be included.

• Create paths file with the following attributes
• path (resourceType.path)
• Create a target file with the following attributes
• DateTime
• resource ID
• resourceType

https://hl7.org/fhir/R4/datatypes.html#dateTime:~:text=3%5B0%2D1%5D))%3F)%3F-,dateTime,-A%20date%2C%20date

Test Patient Compartment - Compare search forms (TestID13)

Test13
If a patient compartment is declared (CapabilityStatement.rest.compartment) - check the equivalence of the 2 search forms below (for existing and non existing patients)

GET [base]/Patient/333/Condition
GET [base]/Condition?patient=333

Basic SMART test

Open issues

• Consider a manual approach for the MVP phase based on Postman

Requirements
Tests 161 + 162

1. connect to an authorization server using mTLS with ORG-provided certificate, get an access token (in scopes request the same resources you are GETing anyway), use access token to connect to FHIR server (also with mTLS) and GET some resources. And then do the same with any other certificate unknown to ORG. This is pretty much basic SMART on FHIR backend services flow

2. Ask ORG to provide you with valid one. Generate your own (i.e. - openssl X509 -req ... etc.)

3. If mTLS fails you'll get an error in https.request - something about certificate usually. For access token request and also if FHIR server doesn't like your access token you'll usually get HTTP 4XX errors

Implementation
Client => PM (https://learning.postman.com/docs/sending-requests/authorization/certificates/)
Certificate =>
SMART2 FHIR Server => https://secure.server.fire.ly
Authentication =>
Authorization => Firely Auth (https://docs.fire.ly/projects/Firely-Server/en/latest/security/firely-auth/firely-auth.html#firely-auth-index)
Access Control =>

Test for PHI exposure in resource ID - All resources, explicit (Test30)

Requirement
Resource IDs may be exposed on GET over HTTP and hence should not include protected health information (PHI)

Implementation

• Extract Patient.identifier.value from all sampled patients and write it into "a PHI" file

• Patient.id

• Patient.identifier.value

• Patient.identifier.system

• Loop over all resources (including Patient), for each resource, compare (Resource.id) to all identifier values (Patient.identifier.value) in the "PHI" file

Analyse time elements

Requirment
Allow for time-based analysis by extracting relevant paths into a file and then analyzing it in the dashboard.

Implementation
All relevant timeelements (from base or profiles) will be listed in an auxiliary file.
Currently, only time elements defined in the base will be included.

• Create paths file with the following attributes
• path (resourceType.path)
• Create a target file with the following attributes
• time
• resource ID
• resourceType

https://hl7.org/fhir/R4/datatypes.html#dateTime:~:text=3%5B0%2D1%5D))%3F)%3F-,dateTime,-A%20date%2C%20date

תמיכה בקריאת HTTP status codes

בתור סרטיפיקטור אני רוצה לקרוא את קודי הסטאטוס שהשרת מחזיר כדי לוודא שהם על פי התקן.
קודים אלו כוללים שגיאות.

Analyse date elements

Requirment
Allow for date-based analysis by extracting relevant paths into a file and then analyzing it in the dashboard.

Implementation
All relevant date elements (from base or profiles) will be listed in an auxiliary file.
Currently, only dateTime elements defined in the base will be included.

• Create paths file with the following attributes
• path (resourceType.path)
• Create a target file with the following attributes
• date
• resource ID
• resourceType

https://hl7.org/fhir/R4/datatypes.html#dateTime:~:text=3%5B0%2D1%5D))%3F)%3F-,dateTime,-A%20date%2C%20date

Test Patient Compartment - inclusion & exclusion of resources (TestID1)

Test 1
If a patient compartment is declared (CapabilityStatement.rest.compartment) - check inclusion and exclusion of resources according to spec

תמיכה בקריאה מקבצים

כסרטיפיקטור, אני רוצה אפשרות לקרוא מקובץ CSV או JSON שנמצא בדיסק כדי שאוכל לאסוף רשימה של רפרנסים שנדגמו אקראית בשלב קודם ולשלוף אותם מהשרת.

לאפשר שימוש בPOST

בתור סרטיפיקטור אני רוצה לבצע פעולת POST לטובת בדיקת התנהגות הכתיבה, החיפוש, והעדכון של השרת.

בפעולת POST אני רוצה להגדיר את הכותרות וגוף ההודעה כדי לבדוק שהשרת דוחה\מבצע את הפעולה בהתאם להגדרות התקן לגבי תוכן הכותרות (למשל content types נתמכים) והגוף (האם מדובר בJSON, האם זה ריסורס וכו)

Test for PHI exposure in resource ID - Patient, explicit (Test30)

Requirement
Resource IDs may be exposed on GET over HTTP and hence should not include protected health information (PHI).

Implementation
Important => #25 provides the same functionality but also covers all other resources at slightly higher effort!

On Patient - compare ID (Patient.id) to all identifier values (Patient.identifier.value)