uchic / iutahdata Goto Github PK

View Code? Open in Web Editor NEW

1.0 14.0 1.0 1.54 GB

The iUTAH Modeling and Data Federation website - data.iutahepscor.org.

License: BSD 3-Clause "New" or "Revised" License

Python 32.21% CSS 6.68% JavaScript 7.84% HTML 53.08% Jupyter Notebook 0.19%

iutahdata's Introduction

iUTAHData

This project contains the code for the iUTAH Modeling and Data Federation website - data.iutahepscor.org.

Installation

iUTAHData uses Django and Python 2.7.

Once you have cloned or downloaded the repository, install python dependencies with pip install -r requirements.txt

When installing packages in requirements.txt you may get an error dealing with pyodbc. Installing iODBC on your system should fix this error.

Linux/Debian

sudo apt-get install libiodbc2 iodbc msodbcsql

macOS (using HomeBrew)

brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
brew update
brew install --no-sandbox [email protected] [email protected] unixodbc

Database Connections

This section is only applicable to systems running Unix or Linux.

Proceeding is not for the faint of heart.

It might be necessary to connect to the production database for development, testing, or deployment. The production database runs on a Microsoft SQL Server (MSSQL). For developers working on a Unix system, you will need to configure FreeTDS on your system to connect to an MSSQL server.

FreeTDS is a set of libraries for Unix and Linux that allows your programs to natively talk to Microsoft SQL Server and Sybase databases.

FreeTDS uses iODBC to communicate with the MSSQL production database from a Unix system.

Installing FreeTDS

Linux/Debian: sudo apt-get install freetds-bin tdsodbc

MacOS: brew install freetds

Configuring FreeTDS

Configure a Data Source Name (DSN) file, which is usually located at /etc/odbc.ini on linux. A DSN is a configuration file that tells iODBC how to connect to a database.

# /etc/odbc.ini
[iUTAH_Logan_OD]
Driver          = ODBC Driver 13 for SQL Server
Description     = iUTAHDBS
Trace           = No
Server          = iutahdbs.uwrl.usu.edu
Database        = iUTAH_Logan_OD

Important: The value for Driver will change depending on which version of msodbcsql you have installed. A quick way to check the version is:

~$: sudo apt-get list msodbcsql
Listing... Done
msodbcsql/xenial,now XX.A.B.C-D amd64 [installed]

Replace the 13 in Driver = ODBC Driver 13 for SQL Server with XX.

More information on this may be found in Microsoft's documentation for installing odbc driver for sql-server.

When FreeTDS and iODBC are configured properly, DataParser.py will automatically configure a connection string to connect to the database.

Copying and License

It is open and licensed under the New Berkeley Software Distribution (BSD) License. Full text of the license follows.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Utah State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

iutahdata's People

Contributors

Stargazers

Watchers

Forkers

marioscience

iutahdata's Issues

Links to TSA on Watershed pages

The links on the right side of the watershed pages both link to the old map viewer pages. We only need one of these links, and it should link to the TSA. I think the caption should be 'Visualize Provo River data'. (or Red Butte Creek or Logan River).

Clicking photos should open them larger in a new window

This might be accomplished with a new photo gallery tool, but when the photo (not the thumbnail) is clicked on, the gallery should open larger in a new window.

Plots re-align when browser zoom level changes

I like to zoom out in the browser so that I can view more of the page and the plots, but this causes the plots to change from 3 across to 2 across. Is there any way to fix the plots to be 3 across?
This isn't super urgent.

Update Data Policy document

The link on the iUTAH Data Policy and Publication page to the Data Policy needs to be updated to version 1.6.

Nav Bar doesn't work on mobile.

Just need to change hover to touch using javascript/jQuery

Date Last Updated Issue

'Date Last Updated' for all of the sites on the production database is in November. Needs to be current and dynamic. Appears to be working on the sandbox.

Make entire line clickable on Sites table

I think the entire line (instead of just the site code) of a site in the table on the watershed page should be clickable and linked to the individual site page.

Update variables displayed on site pages

We don't need all variables displayed as sparkline plots- we should just use a list of selected variables. I've developed a list that is up in the Google drive data.iutahepscor.org folder: https://docs.google.com/a/aggiemail.usu.edu/document/d/1cEpW7cOES3kIXY4uH9nvGkZtCSjp_hUmM5AdCFVGtek/edit

Let me know if you need additional info (variableIDs, etc.)

Add GAMUT Fact Sheet

There is a document in the Google iUTAH CI/GAMUT folder called 'GAMUT Fact Sheet' to which we need to link on the website. I'm thinking on the GAMUT Network Overview page, have a link to it near the title. The linked text can say 'GAMUT Fact Sheet'.

Logan River menu link wrong

The link in the menu to Logan River is incorrect (links to Provo River).

Site Photos

I finally have received enough photos from the technicians that I think we should get them loaded to the website. There should be at least one photo for almost all of the sites (I think there is one site: PR_LM_AA, for which I have no photos).

These have been posted to the iUTAH CloudShare. The link is https://cloudshare.iutahepscor.org/public.php?service=files&t=3f551f9195c1a56ee6ec3a374f1857c5. I think this should give you the ability to access all of them- they are grouped by watershed and site. Let me know if it is a pain to download them from CloudShare. I could post them to the Cube or to the Google Drive.

We're going to have to address captions later.

Add a Link to repository under the Data menu

On the 'Data' menu, we need an item that is 'Dataset Repository' that links directly to http://repository.iutahepscor.org. This item should be #2 on the list after 'Data Policy and Publication'.

@jcaraballo17 or @mauriel133 , can you guys get this done?

Updated List of Sites

I've created an updated list of sites on the Google Drive in the data.iutahepscor.org folder. Can you make sure that the list of sites on the website pages are consistent?

Photo Viewer stretches

We may need a new tool to view photos- this one stretches photos that have a different aspect ratio. The photo selected below is in the middle of the and is vertical:

Issues with maps in Firefox

The maps are not loading correctly in Firefox:

url's with spaces

Can we eliminate spaces from url's? It is problematic when trying to give links to others:

http://django.uwrl.usu.edu/mdf/Data/Data%20Policy%20and%20Publication/

Crop picture

There is some graffiti that needs to be phtoshopped or cropped out of a picture for LR_MS_BA. Can you crop the photo from this:

to this:

Add GAMUT Data Workflow

We need to add the GAMUT Data Workflow image to the site. I'm thinking it should go under 'Development'. I'm wondering if we need text with it (could link the iEMS proceedings paper @horsburgh? ). If we wanted to be fancy, we could make it a clickable image map.

The image is in the Google drive. It also needs to be linked on the GAMUT Overview Page.

Need to Add Link

On 'The GAMUT Network' page, there are links for Red Butte Creek and the Logan River. A link needs to be added to the Provo River, too, linking to http://data.iutahepscor.org/mdf/provo-river

Show Scale on Plots

A technician suggests that we show some sort of scale on the plots:

I really think that we need some indication of scale on the plots. I recognize that this site is designed to just give an indication the overall trends at a given site, but consider precip. I am looking at the precip signal for ARBR_C right now and it goes up and then back down over the last 24 hours. Was this a gauge malfunction or an increase and decrease of .03cm? Without a scale it is unclear.

I'm unconvinced on this, but we should consider it.

Update Software Development Page

The text for the software development page has been updated in the Google Drive folder and needs to be pushed to the website. The images on this page can remain the same for now... I will work on them later.

Change Link

Let's change the link on the bottom of the homepage from 'GAMUT Map Viewer' to 'GAMUT Data Viewer' and link to the TSA.

Create New Page: 'Data Policy and Publication'

I've created a new document, 'Data policy and publication' that is up in the Google drive folder. It has summary text from the data policy and should eventually link to the final copy of the policy as well as example data collection plan, etc. I think we should implement this text as a new page under 'Development'. @horsburgh , you should take a look and make edits as needed. I may have included too much content.

Add LR_FB_BA site

We need to get a site page up for a new GAMUT site: the site code is LR_FB_BA. There are pictures up for it on the CloudShare.

Link needs modifying

The link at the end of the 'Sensor Data Quality Control' section on the Software Development page isn't working. I think the period at the end just needs to be excluded.

Server Error for all sites

I'm getting 500- Internal Server Error when clicking on any of the sites on the sandbox server. Not getting the error on the production server.

Add links to Development page

In the last paragraph on the 'Development' page, we need to add links to the words 'software' and 'hardware' which link to the 'Software Development' and 'Hardware Development' pages.

Change 'Click a box...' symbology

Remove the border from the 'Click a box to plot a variable' text as it looks like it should be clickable.

Add Site: PR_CH_C

We need to get a site page up for a new GAMUT site: the site code is PR_CH_BA. There are pictures up for it on the CloudShare.

Add links to iUTAH web services

We need to add links to the web services that we have deployed for each watershed. At this point, I think we should link them on the page for the watershed (for example: http://data.iutahepscor.org/mdf/Data/Logan_River/). I'm thinking it could be another thumbnail with a link on the right below the 'Visualize Logan River Data'.

At some point in the future, we should add some text about calling the web serviced.

http://data.iutahepscor.org/LoganRiverWOF/WaterML_1_1.aspx
http://data.iutahepscor.org/RedButteCreekWOF/WaterML_1_1.aspx
http://data.iutahepscor.org/ProvoRiverWOF/WaterML_1_1.aspx

Add link to homepage

Under 'Announcements' on the homepage, we need to add a link to the data publication system (http://repository.iutahepscor.org). Add the line 'Click here to submit a dataset.' with the link.

Table Colors

Are these the colors that we want for the tables? I am thinking a light gray background with a darker gray text.

Update Personnel Page

I've updated the 'iUTAH Staff Development' document in the data.iutahepscor.org Google Drive folder. This should take the place of the current Personnel page.

@horsburgh , you should have a look at the document and update as needed.

Stack the soil moisture variables to reflect physical depth

Could we change the arrangement the soil moisture, soil temperature, and permittivity plots so that they are stacked on top of each other to reflect the physical depth? This is how they should appear (with pipe delimiter):

Addional links on each site page

We need to add links/buttons to each site page. These links should go below the table/gallery and above the 'Most Recent instantaneous Measurements' text. They could be side by side and take most of the width of the page.

'Download the raw data for this site.' links to the associated record for the site on http://repository.iutahepscor.org/dataset. For example, LR_TG_C would link to: http://repository.iutahepscor.org/dataset/iutah-gamut-network-raw-data-at-tony-grove-climate-site-lr-tg-c
'Visualize time series from this site.' links to the TSA with that site pre-selected in the facets. No data series should be selected for plotting.

@jcaraballo17 and @marioscience , hopefully the two of you can work on this?

Variable names overlap values in some cases

Because each plot now includes variable code and variable name, some of the names are long enough to overlap with the values. Ideas to address:

Make the font even smaller for these variables
Truncate the variable names
Move the value lower in the box for these variables.

Add Windrose Plot

The technicians are requesting a windrose plot. I think the dynamic site pages are a good place to house this. I don't know what libraries we're using- a python example is here: http://youarealegend.blogspot.com/2008/09/windrose.html

Add USGS, CUWCD discharge data streams

We need to add the discharge data from the USGS and CUWCD (RB_ARBR_USGS, PR_BJ_CUWCD, PR_CH_CUWCD, PR_UM_CUWCD). @horsburgh , how do you suggest we handle this? Options:

We have these as separate sites in the database, so they could each have their own dynamic site page. We would not have any pictures for these.
It might be more straightforward to present them with their associated GAMUT site. We could add those data streams to the associated GAMUT site page.
I could change the sites in the databases and on the streaming dataloader so that the data is more directly associated with the GAMUT sites.

Indicate 24 hours of data

For the most recent instantaneous values, we need an indication of the timeframe that is being displayed. I would suggest a line that says 'Past 24 hours shown' after the 'Data update time'.

Updates to Variable List

Based on feedback from the iUTAH technicians, I've made some adjustments to the list of variables for display. Some of these variables are specific to particular sites, as indicated in the list. Please take a look and let me know if you have any questions.

Direct links to TSA down work when they select multiple series

When I click on a link to one of the time series from the spark line plots on and individual site page where that variable has both QualityControlLevel = 0 and QualityControlLevel = 1 data, the TSA isn't plotting anything - presumably because it doesn't know which series you want it to plot. For example:

http://data.iutahepscor.org/tsa/?sitecode=LR_Mendon_AA&variablecode=WaterTemp_EXO&view=visualization&plot=true

The links need to isolate a single series for plotting.

Server Error for Logan River and Provo River sites

I'm getting a server error when trying to click on any of the Logan River or Provo River sites. Red Butte sites seem to be fine. This is on the sandbox server.

Text of data disclaimer

There are some issues with the text in the disclaimer. It says "neither iUTAH..." but another party is not referred to. @horsburgh , did you have something in mind?

Display of Soil Moisture Variables

There are four soil moisture variables at each depth. We need a way to 1. separate them from the other climate variables and 2. distinguish data at each depth.

Update Documentation page

I've updated the list of presentations and posters that have been given related to iUTAH. This is in a Google doc in the Google Drive called 'Documentation'. The files named in this document correspond to files in the Presentations folder in the Google Drive.

We'll need to keep updating this list with the various presentations that we are giving this summer. I don't know if @Castronova has anything to add to the list? @horsburgh , should we be adding publications that are less official- i.e., presentations given at the various iUTAH meetings? Could be redundant.

DateTime needs to update

The date and time of the most recent values is not updating:

SSL Warning for CloudShare Link

There is an SSL warning when trying to access CloudShare via the link on the data.iutahepscor.org page. Can this be remedied?

Idea: side bar with click-able sites

When you are on one site page, you might want to click directly to another site in that watershed. It might be nice to have a menu or some sort of side bar with links to the other sites in the watershed.

GAMUT Overview Page

I've updated the content for the GAMUT Landing Page. I created a Google drawing to give an idea of arrangement of images, etc. It's on the Google Drive. Maybe wait a couple days to put it online as @horsburgh should review it. I will add the images to a folder. Let me know if you have any questions!

Update Training Page

I've updated the list of training materials related to iUTAH CI. This is in a Google doc in the Google Drive called 'Training'. The files named in this document correspond to files in the Presentations folder in the Google Drive.