gh-davidr / nightscoutloader

A Java application that can synchronize meter/pump data with Nightscout, offers a quick way to modify notes field in treatment data and includes deep trend analysis of this data.

NightscoutLoader

Why the name 'Nightscout' Loader?

• Because Nightscout is the CGM in the Cloud environment that this tool supports.

What is Nightscout Loader?

A desktop application with the following purposes in mind:

• Synchronize data from meter/pump as treatments in Nightscout Care Portal.
• Optonally also now synchronize Diasend data from CGM as entries in Nightscout.
• Quick way of locating and allowing edits to the Notes field in treatment data.
• Analysis of BG Trends
• Analysis of CGM Data
• Easier access to Autotune

Synchronize data from meter/pump as treatments in Nightscout Care Portal.

Meter & Pump Data can be regularly loaded from a range of systems:

• Libre view (csv)
• Diasend (xls)
• Medtronic (csv)
• Tandem t:slim (csv)
• Cellnovo (csv)

A serial stream of BG, Carb and Insulin is intelligently grouped together into a Treatment. The tool will ignore previously loaded data. It will also identify contention between manually entered Care Portal entries and data from meter/pump.

Quick way of locating and allowing edits to the Notes field in treatment data.

Searches can be performed over combinations of different fields, date ranges, treatment types and also if the loaded entry is a duplicate with a manually entered Care Portal entry.

Analysis of BG Trends

Analysis can be run over selected date ranges and outputs to Excel for convenient archiving of analytical snapshots. Analysis also runs in background on start up and provides a condensed summary of top 3 trends

Analysis of CGM Data

Any CGM data available within selected analysis date range is also analyzed. A heat map shows areas of high frequency profiles.

Inference of CGM Date Ranges

For those without continual sensor use, identifies date ranges when sensor was worn - useful for reviewing past events & data.

Easier access to Autotune

Nightscout Loader can now help with running and maintaining profiles for Autotune

As well as integrating Autotune output in the Excel analysis results file, an "Autotune Management" window allows easy access to run Autotune once configured.

A Profile editor is provided that can edit the remote profile, open profile files on the local machine and also read profile data from a Diasend export.

Each time the Profile Editor updates the profile on the remote Autotune server, it takes a backup into a separate directory. The files in the backup directory can be listed (as above) and also downloaded onto the local machine for greater safe keeping.

Analytic approach used by NightscoutLoader

Nightscout Loader helps get real Insights from reams of Data.

Is it for me?

• Do you use Nightscout and want to upload Treatment (and or Diasend CGM) data to Nightscout from a supported Meter and or Pump?
  • Then set up for Nightscout use
• Do you want an extra tool to help analyze BGs, Carbs & Insulin? (Nightscout use is optional)
  • Then set up either for Nightscout use or non-Nightscout use
• Do you use Nightscout and want an extra tool to visualize repeated patterns of CGM data?
• Do you have Autotune configured on a separate Linux machine and want assistance with using it - particularly if you also use Diasend?
  • Features were initially incorporated to help with managing a virtual Linux server like AWS to run AutoTune. Since then, Autotune Web is available and NightscoutLoader does not yet integrate

If any of the above are true, then it could help you.

Installation Guide

System Requirements

The application is built for desktop or laptop computers with Java. It requires at least Java version 8 and the installation guide includes a link to check and install if necessary.

It loads the Nightscout Treatment and CGM into memory for comparison with new data being uploaded or for analysis. The amount of memory your system needs will depend on how much data you pull in. As a guide, 8 years' worth of treatment data and 20,000 CGM entries takes up 300Mb of RAM.

Installation Guide for Nightscout use on Windows

1. Check your version of Java at this link
2. Download the application from the Github Releases page.
   1. Occassionally, an early beta version is made available with fixes on the stable version or access to upcoming features.
3. Copy the NightScoutLoader.jar and NightScoutLoader.bat files to your desktop (or any other location), ensuring that both files are in the same folder.
   1. (Note that the loadLibreview.sh and loadDiasend.sh files are template Linux files for running NightscoutLoader as a batch and so don't need to be copied)
4. Launch the application by double-clicking the NightScoutLoader.bat file
   1. The NightScoutLoader.bat script checks the jar file is correctly located. If it is not correctly located, then it will alert and exit.
   2. It will also attempt to create C:\Temp folder for the main application log file. If this attempt fails then it will alert and exit.
5. Click Tools => Settings to launch the Settings panel
6. Complete the sections in Yellow
7. For example:
   1. if your user is "my_mongo_user"
   2. password is "my_mongo_pass"
   3. database is "my_mongodb"
   4. then Nightscout Server would be "mongodb+srv://my_mongo_user:[email protected]/my_mongo_db?retryWrites=true&w=majority" (Note that ds123.mongodb.net will be different for your own set up)
   5. Nightscout DB would be "my_mongo_db"
   6. Nightscout Collection is always �treatments�
8. Once the data is loaded, an analysis automatically runs but a more detailed report can be generated by selecting Action menu and then Analyze Results
9. Select the file type to load (it may show Cell Novo initially)
10. Navigate to the file to load by clicking the Select button
11. Start the synchronisation by selecting Action menu and then Synchronise
12. Once the data is loaded, an analysis automatically runs but a more detailed report can be generated by selecting Action menu and then Analyze Results To convert a previous installation for non-Nightscout to one for Nightscout use, simply complete steps 5 and onwards above, save the settings and re-launch the application.

Installation Guide for non-Nightscout use on Windows

1. Check your version of Java at this link
2. Download the application from the Github Releases page.
   1. Occassionally, an early beta version is made available with fixes on the stable version or access to upcoming features.
3. Copy the NightScoutLoader.jar and NightScoutLoader.bat files to your desktop (or any other location), ensuring that both files are in the same folder.
   1. (Note that the loadLibreview.sh and loadDiasend.sh files are template Linux files for running NightscoutLoader as a batch)
4. Launch the application by double-clicking the NightScoutLoader.bat file
   1. The NightScoutLoader.bat script checks the jar file is correctly located. If it is not correctly located, then it will alert and exit.
   2. It will also attempt to create C:\Temp folder for the main application log file. If this attempt fails then it will alert and exit.
5. Launch the application by double-clicking the NightScoutLoader.bat file
6. Select the file type to load (it may show Cell Novo initially)
7. Navigate to the file to load by clicking the Select button
8. Load the file by selecting Action menu and then Load Meter/Pump Only
9. Once the data is loaded, an analysis automatically runs but a more detailed report can be generated by selecting Action menu and then Analyze Results

Installation Guide for Mac

1. Sorry - I don't have regular access to a Mac that would enable the provision of instructions for use.
2. The application will definitely run on MacOS (as reported by existing MacOS users) and it should be sufficient to launch it from the NightScoutLoader.jar file without needing to have a run script.
3. Perhaps someone can advise.

Obtaining the MongoDB URI from Heroku

Setting MongoDB for Nightscout Server:

1. Login to Heroku (https://id.heroku.com/login)
2. Click on your Nightscout application
3. Click on "Settings" tab
4. In "Config Vars", click on "Reveal Config Vars"
5. Scroll down to "MONGODB_URI"
6. Copy value to right - which starts with "mongodb+srv://........"
7. Paste this URI text into "Nightscout Server" of NightscoutLoader's Settings screen

Running NightscoutLoader as a batch

NightscoutLoader can be run with arguments from the command line instructing it to synchronise data from a file (such as Diasend for example). In this way, it can be incorporated into a more streamlined set up to speed the process of getting data into Nightscout from a downloaded file.

A couple of redacted Bash shell scripts are included in the bin folder which the author uses on Linux for this purpose. They can be used to write equivalent Windows bat scripts if desired.

---

That's all there is to it!

To convert a previous installation for Nightscout to one for non-Nightscout use, simply clear the Nightscout Server field in the Settings window, save the settings and re-launch the application.

👫 Nightscout Loader Website 👫 Nightscout Loader Facebook Page

nightscoutloader's Issues

Allow NightscoutLoader to provide a feed to AutotuneWeb such that a user can run the online autotune from within Loader

As per the title, integrate NightScoutLoader Autotune tools with AutotuneWeb to allow non-Nightscout users to capture pump and diasend data and run Autotune effectively without the need to involve a nightscout set-up.

In addition, this would potentially allow clinics to use the tools with their pump users.

I've created the counter issue for AutotuneWeb: MarkMpn/AutotuneWeb#15

configuration with medtronic 640g cgm

HI! I am stuck in the config part where entering the mlab info and the nightscout info... please advise, i have a medtronic CGM not an accucheck...

Diasend Date Format

First, thank you so much for making this app available. It is especially useful for Omnipod users to get their data imported into Nightscout!

I've come across one issue, but perhaps I am using the app incorrectly... The available date formats in Settings does not include "MM/dd/yyyy HH:mm" which is the format used by Diasend in the USA for exported xls data files. This is discussed at openaps/docs#1316 and at autotuned gitter.

I made a fix in my fork of your dev branch to include this date fomat:
WinSettings.java, line 88
private String[] m_InputDateFormatList = {"Default", "dd/MM/yy hh:mm", "MM/dd/yy hh:mm", "dd/MM/yy", "MM/dd/yy", "MM/dd/yyyy HH:mm"};

I'm using IntelliJ rather than Eclipse though so am not sure I should create a PR on this.

ignores most of tslim x2 csv file

In stand alone mode, with tslim x2 "t:connect therapy timeline data export" app only grabs section labeled "Bolus" does not import "Basal" or BG (EGV)
loader-issue.zip

Diasend no longer available

Diasend has transferred over to Glooko. Glooko exports a ZIP file containing .csv files. The Diasend selection doesn't work for new Glooko export, as its looking for XLS files, and when I do select the ZIP or CSV from inside the ZIP I get "However this file is in an unrecognised format. Please export results form Diasend software in correct format."

The Glooko export is similar to the old XLS Diasend export. But instead of being a XLS file with multiple sheets, it instead is a .zip with multiple .csvs. I tired just editing the structure and making an XLS that looked similar to the old one, but the format for some of the CSVs have changed alot compared to the old XLS sheets.

no file select on macOS

Good morning,
yesterday i downloaded .jar and so on.
I start .JAR. Program opens, can make my settings.
If i want to import xls via search i dont see any data (all files) on my desktop.
MacOS is up to date (newest).
Regards
Sven

Problem loading diasend export

Using latest version, v2.6

Choose a diasend-exported file.
Doing a Load Meter/Pump only
Getting this msg

Any idea what to do?

May the diasend export be different because I'm in sweden?

Details screen is using a reverse index on the Treatment data loaded

Non-issue. Find / Details screen is correctly navigating the grid. Was reading the form incorrectly.

Make file

Do you have a project file for this repo so we can build it in IntelliJ or Eclipse? The jar file on Google Drive is too large to do a virus scan so I would like to build your code myself. Thank you.

NighscoutLoader won't load Medtronic carelink.eu CSV exports

Error returned is:
Option selected to load Medtronic file:
xxx.csv
However, this file is in an unrecognised format

I'm suspecting it's a date parsing issue?
I can provide example CSV files if required.

upload tidepool .json

Hello,

Right now tidepool is the only service I can use to extract basal data from my pump (t:slim with control:iq) and get it into a data file. If I export a .csv from tconnect I don't get any basal data from them. If I could use that .json to upload to nightscout that would be incredibly useful for me to get my automatically set basals into nightscout.

Connectivity test fails because of new(?) mongodb uri

When setting up Nightscout Loader for the first time, I'm getting this error:

**************************************************
* Testing connectivity to MongoDB for Nightscout *
**************************************************

Test Nightscout Failed.
Unable to connect to MongoDB

Nightscout Server(mongodb+srv://username:[email protected]/db_name?retryWrites=true&w=majority) DB Instance(dbname) Received the following error message:

uri needs to start with mongodb://-uri needs to start with mongodb://

My guess is that on these (relatively) new instructions, the uri for a mongodb has changed since the jar of Nightscout Loader was compiled.

Is it possible to let Nightscout Loader use mongodb+srv:// as the prefix?

Insulin tab is null

Thank you for providing this high-effort tool.
Though after building NightscoutLoader from source ( 😥 ), I get the following error when loading a diasend xls:

20-11-2019 21:59:13 : SEVR : <davidRichardson.DataLoadDiasend> Exception loading Insulin tab null

Indeed, I don't have any Insulin data:

I can try to anonymise a .xls if you would like to reproduce

Nightscout rest api

Howdy, phenomenal project.
FWIW, Nightscout does not support mongodb as an API for Nightscout, but rather hosts its own REST API.
I'd be delighted to help you develop a feature to use the Nightscout REST API rather than the mongodb interface.