Bulk Data Import

Bulk Data Import is an operation designed to upload large FHIR datasets to the FHIR server. Bulk Data Import works much like Bulk Data Export. Though this operation is not defined in the FHIR specification, they share similar principles.

Import Request

To import FHIR resources into the FHIR Server:

Upload the file, containing newline-delimited JSON, where each line is a single resource.
Make a post request to the server with a file path specified in the body.

Example - Import Request

curl --location --request POST 'https://example-develop.edenlab.dev/fhir/$import' \
--header 'Content-Type: application/json' \
--data-raw '{
    "filepath": "/import/file1.ndjson",
    "range": {
    "start": 1032,
    "end": 5000
    }
}'

Headers

Prefer (string, required): Specifies whether the response is immediate or asynchronous. The header is set to respond-async (based on ).

Example of input file

{"id":"5c41cecf-cf81-434f-9da7-e24e5a99dbc2","name":[{"given":["Brenda"],"family":["Jackson"]}],"gender":"female","birthDate":"1956-10-14T00:00:00.000Z","resourceType":"Patient"}
{"id":"3fabcb98-0995-447d-a03f-314d202b32f4","name":[{"given":["Bram"],"family":["Sandeep"]}],"gender":"male","birthDate":"1994-11-01T00:00:00.000Z","resourceType":"Patient"}
{"id":"945e5c7f-504b-43bd-9562-a2ef82c244b2","name":[{"given":["Sandy"],"family":["Hamlin"]}],"gender":"female","birthDate":"1988-01-24T00:00:00.000Z","resourceType":"Patient"}

ifNoneExist

The ifNoneExist parameter is designed to check whether the resource with a given identifier already exists in the database or not using the FHIR search. If it does, the procedure omits the resource. This way, it is possible to ensure that no duplicates of existing resources are created during the import procedure.

Examples of imported resources with “ifNoneExist“ parameter

{"ifNoneExist":"identifier=http://www.ohiogastro.com/procedure|2399234","resource":{"code":{"coding":[{"code":"99231","display":"SUBSEQUENT HOSPITAL CARE","system":"http://www.ama-assn.org/go/cpt"}]},"identifier":[{"system":"http://www.ohiogastro.com/procedure","value":"2399234"}],"location":{"reference":"Location?identifier=http://www.ohiogastro.com/location|14"},"meta":{"profile":["http://hl7.org/fhir/us/core/StructureDefinition/us-core-procedure"],"source":"http://www.ohiogastro.com"},"performedDateTime":"2022-03-29","performer":[{"actor":{"reference":"Practitioner?identifier=http://www.ohiogastro.com/practitioner|1104392281"}}],"resourceType":"Procedure","status":"completed","subject":{"reference":"Patient?identifier=http://www.ohiogastro.com/patient|982947"}}}
{"ifNoneExist":"identifier=http://www.ohiogastro.com/procedure|2399235","resource":{"code":{"coding":[{"code":"XRSFO","system":"http://www.ama-assn.org/go/cpt"}]},"identifier":[{"system":"http://www.ohiogastro.com/procedure","value":"2399235"}],"location":{"reference":"Location?identifier=http://www.ohiogastro.com/location|20"},"meta":{"profile":["http://hl7.org/fhir/us/core/StructureDefinition/us-core-procedure"],"source":"http://www.ohiogastro.com"},"performedDateTime":"2022-04-11","performer":[{"actor":{"reference":"Practitioner?identifier=http://www.ohiogastro.com/practitioner|1720160914"}}],"resourceType":"Procedure","status":"completed","subject":{"reference":"Patient?identifier=http://www.ohiogastro.com/patient|982948"}}}

Range

A user is able to start import from a specific line and end import to a specific line. In order to do this, the user needs to add parameter “range“ and put the number of rows to start - range.start and end - range.end

Example -Range

"range": {
    "start": 1032,
    "end": 5000
    }

Conditional References

When processing import, the Kodjin FHIR Server performs the next steps:

check all references for search URIs
For search URIs, use the search to locate matching resources
- if there are no matches or multiple matches, the transaction fails, and an error is returned to log import
- if there is a single match, the procedure replaces the search URI with a reference to the matching resource

Notes

The import process supports referential integrity. Therefore, if the input data contains invalid references or if some resources fail to be imported, the state of the FHIR store will not violate referential integrity.
If some resources fail to be imported, such as due to parsing errors, successfully imported resources aren't rolled back. For example, if 5 of 100 resources fail to import, the remaining 95 resources are imported into the store.

Bulk Data Delete Request

After a bulk data request has been started, a client may send a DELETE request to the URL provided in the Content-Location header to cancel the request.

Endpoint

DELETE [polling content location]

Bulk Data Status Request

After a bulk data request has been started, the client may poll the status URL provided in the Content-Location header.

Endpoint

GET [polling content location]

Responses

Response Type

Description

Example Response Headers + Body

In-Progress

Returned by the server while it is processing the $import request.
Count of processed row, count of errors and time of start import and updatedAt.

Status: 202 Accepted 
 {
    "id": "a91451b9-6678-43c8-beb1-1aa7577301cb", 
    "status": "in-progress", 
    "fileUrl": "https://example.com/fhir/$import", 
    "processed": { 
        "total": 350, 
        "imported": {
            "count": 350
        },
        "errors": {
            "count": 0
        }
    }, 
    "createdAt": "2022-09-23T08:42:21.532662101Z", 
    "updatedAt":"2022-09-23T08:42:28.247404672Z"
 }

Error

Returned by the server if the export operation fails.

Status: 500 Internal Server Error
Content-Type: application/json
{
    "resourceType": "OperationOutcome",
    "id": "1",
    "issue": [
        {
            "severity": "error",
            "code": "processing",
            "details": {
                "text": "An internal timeout has occurred"
            }
        }
    ]
}

Complete

Returned by the server when the import operation has completed.
Count of processed row, count of errors and time of start and finish import

Status: 200 OK
Expires: Mon, 22 Jul 2019 23:59:59 GMT
Content-Type: application/json
{
    "id": "a6d1255e-d976-4dd4-a944-1472156e77af",
    "status": "completed",
    "fileUrl": "https://example.com/fhir/$import",
    "processed": {
        "total": 195059,
        "imported": {
            "fileUrl": "https://example.com/fhir/$import",
            "count": 195059
        },
        "errors": {
        "count": 0
        }
    },
    "createdAt": "2022-09-22T11:43:52.666574356Z",
    "updatedAt": "2022-09-22T12:12:18.730391797Z"
}

Security and multi-tenancy in the import

All data imported during an import operation aligns with the scopes granted and provided in the access token for write interactions.

For example, if the client's scope is limited to patient/Observation.write and the source (ndjson) contains both Observations and Conditions, only Observations will be imported.

If the multi-tenancy option is enabled in the Kodjin FHIR Server, Kodjin will import data that belongs to the tenant provided in the token claim.