hl7 / bulk-data Goto Github PK
View Code? Open in Web Editor NEWBulk Data Implementation Guide
License: Other
Bulk Data Implementation Guide
License: Other
Resolution from https://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=21041&start=0 has not been applied
Below http://build.fhir.org/ig/HL7/bulk-data/#resources should be a bullet list of the 5 pages repeated from the titlebar, since some users did not find the titlebar
Where does this come from - is it a config somewhere? Will it fix itself when deployed to hl7.org?
http://build.fhir.org/ig/HL7/bulk-data/OperationDefinition-export.html
The official URL for this operation definition is:
http://hl7.org/fhir/us/bulkdata/OperationDefinition/export
Resolution from https://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=21935 has not been implemented.
For POST requests, the url returned as the "request" key in the completion manifest does not contain enough information to retry the request. Need to decide if this is a problem in practice and if so, how to remediate.
Exception in thread "main" java.lang.Error: The auto-build infrastructure does not publish IGs that contain HTML pages without the publish-box present. For further information, see note at http://wiki.hl7.org/index.php?title=FHIR_Implementation_Guide_Publishing_Requirements#HL7_HTML_Standards_considerations
In two spots we have a typo where EC384 should say ES384.
Clients can use the _since parameter to retrieve only resources that have been updated after a point in time. However, there is currently no way for a client to determine if a previously retrieved resource needs to be deleted without re-pulling all available data and performing a comparison (this could be across many GB of data). Examples: delete requests that need to be propagated to downstream systems or data ascribed to the wrong patient that needs to be removed.
Suggest we leave the backend services page as is, but add the following to section 1 on overview page too:
"Public health surveillance systems that do not require real-time exchange of data."
Add https://tools.ietf.org/html/rfc7240 to the underlying standards section on export.html
http://hl7.org/fhir/uv/bulkdata/STU1/export/index.html#request-flow shows that the $export operation is invocable via HTTP GET, but I don't think it says anything about POST.
Meanwhile, the base FHIR specification indicates that operations can be invoked via HTTP POST: https://www.hl7.org/fhir/operations.html#executing
Additionally, its says that
Operations may be invoked using a GET, with parameters as HTTP URL parameters, if:
- there are only simple input parameters - i.e. no complex datatypes like 'Identifier' or 'Reference', and
- and the operation does not affect the state of the server
Based on that, I think its OK to continue supporting $export
via GET, but I'd like to see the bulkdata specification indicate that POST must be supported as well (and maybe even should be preferred given that GETs should be idempotent/repeatable).
There was some discussion on the bulk data argonaut call on this front and, although no one was a big fan of the FHIR Parameters object, this is the wrapper that FHIR specifies for passing parameters while invoking extended operations like $export
via POST.
Think both shall be SHOULD?
For non-system-level requests, the Patient Compartment SHOULD be used as a point of reference for recommended resources to be returned.
FHIR-24434
http://hl7.org/fhir/uv/bulkdata/operations/index.html: "To declare conformance with this IG, a server should include the following URL in its own CapabilityStatement.instantiates: http://www.hl7.org/fhir/bulk-data/CapabilityStatement-bulk-data.html" should say "http://hl7.org/fhir/uv/bulkdata/CapabilityStatement/bulk-data (see http://hl7.org/fhir/uv/bulkdata/CapabilityStatement-bulk-data.json.html for details).
Also for technical correction: https://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=23864
Would be good also to add a parenthetical on DELETE like "after an export is complete, a server MAY use DELETE as a signal that a client is done retrieving files and that it is safe for the sever to remove those from storage."
Also for technical correct: https://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=24912
Update each operation to state as part of the description: "The FHIR server MUST support invocation of this operation using the FHIR Asynchronous Request Pattern.
Update each operation parameters list. Continue to include the _outputFormat parameter inline to provide a holistic view but condense the "Documentation " adding a reference to FHIR async.html. "The format for the requested bulk data files to be generated as per FHIR Asynchronous Request Pattern. Defaults to application/fhir+ndjson.
I don't know the HL7 mechanics here - should we change this to ballot #2?
Open Questions:
Auth: Do we want to only target servers that have pre-configured write permissions to a bucket, or do we need a way to pass in auth credentials to the server? If so, what would this look like?
Path: Does the we need additional information in addition to the bucket name, like file prefix (eg. to support a "folder" within the bucket that incorporates a timestamp), or service provider (to support use cases where the server is writing to a bucket provided by a different cloud vendor)?
Completion: Should we require that the output manifest file be written to the bucket last so it could be used as an event to trigger followup actions (eg. a de-id or db load) or would we expect clients to use job polling to determine all files have been written?
What are the places where a server developer might make a choice that clients need to be aware of? Can we list them in one spot? Examples would include:
...
Let's build out this list and create a section to track it.
The _since parameter currently limits returned resources to those whose state has changed after the supplied timestamp. In the case of a query like GET [fhir base]/Group/[id]/$export?_since=[timestamp]
, results will not include resources for patients added to the group after timestamp if their resource modification date is earlier than timestamp, necessitating a pull of all available data for a group on each data request (this could be many GB of data for a large group).
Suggest we delete the last sentence of this paragraph:
To begin the exchange, the client SHALL use the Transport Layer Security (TLS) Protocol Version 1.2 (RFC5246) or a more recent version of TLS to authenticate the identity of the FHIR authorization server and to establish an encrypted, integrity-protected link for securing all exchanges between the client and the authorization server’s token endpoint. All exchanges described herein between the client and the FHIR server SHALL be secured using TLS V1.2 or a more recent version of TLS .
http://build.fhir.org/ig/HL7/bulk-data/export/index.html#bulk-data-status-request has a table which now repeats the information below it with some variations.
Issues
The 5xx error has an example in the table and an example below in http://build.fhir.org/ig/HL7/bulk-data/export/index.html#response---error-status-1 but they're different. Is there a reason to include both, and is there a reason they're different? The example below perhaps is supposed to be an example associated with the File Request (http://build.fhir.org/ig/HL7/bulk-data/export/index.html#file-request) section instead, since its error string mentions a deleted file
Intro note says "Servers SHOULD supply a Retry-After header with a http date or a delay time in seconds" but this doesn't appear in our examples table or in http://build.fhir.org/ig/HL7/bulk-data/export/index.html#response---in-progress-status (it should probably appear in both places)
"The choice of when to determine that an export job has failed in its entirety (error status) vs returning a partial success (complete status) is left up to the implementer." belongs in the "Note: even if some..." paragraph
Also #21 applies here to http://build.fhir.org/ig/HL7/bulk-data/export/index.html#response---complete-status
Handling of binary resources is not directly addressed in the IG leading to a lack of clarity on how servers and clients should handle NDJSON files with resources that contain attachments (eg. DocumentReferences)
If resources in an output file contain elements of the type Attachment
, servers SHALL populate the Attachment.contentType
code as well as either the data
element or the url
element. The url
element SHALL be either an absolute url that can be de-referenced to the attachment's content or a relative url that identifies a Binary FHIR Resource included in the output of the export [*see Q1 below].
When the url
element is populated with an absolute URL and the requiresAccessToken
field in the Complete Status body is set to true
, the url location must be accessible by a client with a valid access token, and SHALL NOT require the use of additional authentication credentials. When the url
element is populated and the requiresAccessToken
field in the Complete Status body is set to false
, the url location must be accessible by a client without an access token.
Note that if a server copies files to the bulk data output endpoint or proxies requests to facilitate access from this endpoint, it may need to modify the Attachment.url
element when generating the FHIR bulk data output files.
change warning comment to
# update and merged PR publish stu2 bulk data #315 10/21/2021
I'll keep adding broken or problematic links to this issue as I read through. FYI @gotdan
At the very least include information from https://github.com/smart-on-fhir/fhir-bulk-data-docs/blob/e2ab3d9e8eb5a5e0e0a96c2d026d32860007c2a4/README.md
Based on Grahame's feedback during FHIR Infrastructure call, I think this should be http://hl7.org/fhir/uv/bulkdata
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.