ctsit / export_large_projects

A REDCap module for exporting very large projects

Export Large Project

This is a REDCap external module that provides functionality to export large projects.

Easy Installation

• Obtain this module from the Consortium [REDCap Repo] (https://redcap.vanderbilt.edu/consortium/modules/index.php) from the Control Center.

Manual Installation

• Clone this repo into <redcap-root>/modules/export_large_projects_v0.0.0.
• Go to Control Center > Manage External Modules and enable Export Large Projects.
• Go to your project home page, click on Manage External Modules link, and then enable Export Large Projects.

Configuration

Access External Modules from the Control Panel to set the number of fields exported in each batch and the maximum execution time for the entire export.

How to use?

Once this module is enabled for a project, go to "Data Exports, Reports, and Stats" section. In here, the Export Large Projects button is presented as shown in the below image. Click that button to start the export. A progress indicator will show the stepwise export of the data. After all the chunks have been exported and assembled into a single CSV, a download link is presented to download the entire file.

export_large_projects's Issues

Data de-identification and subsetting is not available

ELP provides no accommodation for subsetting or de-identifying the data. i.e. These features are not supported:

Known Identifiers:
Remove all tagged Identifier fields (tagged in Data Dictionary)
Hash the Record ID field (converts record name to an unrecognizable value)
Free-form text:
Remove unvalidated Text fields (i.e. Text fields other than dates, numbers, etc.)
Remove Notes/Essay box fields
Date and datetime fields:
Remove all date and datetime fields
— OR —
Shift all dates by value between 0 and 364 days

Maybe it should?

Add Andy Martin to authors

Add Andy Martin to authors. This module is a port of his plugin code to a module and we need to acknowledge that. His info should read:

{ "name": "Andy Martin", "email": "[email protected]", "institution": "Stanford University" },

Non-superusers can not export data

When non-superusers attempt to export data they get a message that REDCap has crashed. This happens regardless of the choice of how User Rights are checked (Design/Setup Rights vs. module-specific) rights.

Note that if the non-superuser doesn't have the appropriate rights, they get the expected 'Access Denied' message.

Fix institution names

UF staff should have an institution of "University of Florida - CTSI"

Module throws a fatal error when the execution time is exceeded

When processing each batch export.php will throw a fatal timeout error if processing the batch takes too long.

The offending code is lines 62-80:

foreach ($batches as $count => $batch) {
    set_time_limit($_GET['max_execution_time_per_batch']);

    try {
        $records = REDCap::getData('csv', $batch);

        // Trim the header on all but the first row of the first batch.
        if ($count) {
            $first_lb = strpos($records, "\n");
            if ($first_lb !== false) {
                $records = substr($records, $first_lb + 1);
            }
        }

        fwrite($fh, $records);
    }
    catch (Exception $e) {
        $module->callJsCallback('displayErrorMsg');
    }

The set_time_limit method throws the fatal error. This type of error cannot by caught by try, catch blocks.

To recreate this issue insert a while(true) statement inside the foreach loop and set the max execution time (which is configured in the module's settings) to something small like 2 seconds.

No default global configuration of batch size and timeout

The module does not provide for a default or global configuration of the fields per batch or the timeout. Yet these fields are required to use the module. This requirement to do per-project configuration would make it hard to enable this module globally.

Please consider adding either a default value for these settings or a global setting so that per-project configuration of these settings is not required. I think per-project config should be allowed, but not required.

File downloads in excess of 500MB

I've noticed that file downloads in excess of 500MB do not complete and return 0 after a few seconds. The readfile_chunk function defined in init_functions is 100% correct but cannot seem to figure out why the download terminates after a few seconds. 10MB and 20MB downloads work fine. Any suggestions ?

RedCap Version: 8.4.2
PHP Version: 7
OS: SUSE 12

Error in plugins/export.php, easy solution

redcap_v8.5.2
On line 54 of plugins/export.php, the RCView::span functions gives an error, expecting two parameters. Adding an empty string as second paramter solves that problem.

Adjust log output written into the REDCap log

Reduce the number of messages written into the REDCap log. At the same time, add details to the logs that are there. e.g. an elapsed time would be good. see pbchase@b3c6035

@tlstoffs, what did you want reflected in the REDCap log? You had an opinion on what needed to be in the log for audit purposes.

strange character in config.json

line 43 of config.json looks like this on my linux server:
43 "key": "max_execution_time_per_batch",^[[>1;4602;0c

The part after the ',' is not selectable. I deleted the ',' which also deleted the extra garbage, and re-entered the ','.

Export Via API?

This module is great! We'd just like to be able to call it on a schedule. Any chance it could work with REDCap's API?