Coder Social home page Coder Social logo

smithsonian / forestgeo Goto Github PK

View Code? Open in Web Editor NEW
12.0 7.0 4.0 59.29 MB

🌱🌳Web app helping a global network of scientists and forest research sites manage and validate data. Collaboration between the Smithsonian Institution's Forest Global Earth Observatory (ForestGEO), Microsoft, and CodeDay.

HTML 0.29% CSS 0.44% TypeScript 98.58% JavaScript 0.43% Shell 0.01% TSQL 0.24%
hacktoberfest sustainability forest forestry research science

forestgeo's People

Contributors

aklaran avatar alexkarcher-msft avatar cezapata avatar esscott avatar gabriellaarcilla avatar illume avatar jaro05 avatar justinpac avatar omarshehata avatar siddheshraze avatar vivihung avatar zmagar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

forestgeo's Issues

screenDuplQuadratInfo validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenDuplQuadratInfo Checks for duplicated quadrats Local Makes sure no quadrats (20 x 20 meter subsection of plot) are repeated
    public function screenDuplQuadratInfo ()
    {
      $q1 = "SELECT QuadratName,PersonnelID,RoleID,DateWorked,Form,COUNT(QuadratName) AS cnt FROM TempPersonnelInfo WHERE Errors<>'NONE' GROUP BY QuadratName,PersonnelID,RoleID,DateWorked,Form HAVING cnt > 1";
      $query1 = $this->screeningdb->query($q1);
      if ($query1->num_rows() > 0)
      {
        foreach($query1->result() as $row)
        {
          $q2 = 'UPDATE TempPersonnelInfo SET Errors = CONCAT(TRIM(Errors),";Duplicate Record") WHERE QuadratName = "'.$row->QuadratName.'" AND PersonnelID='.$row->PersonnelID.' AND RoleID='.$row->RoleID.' AND DateWorked="'.$row->DateWorked.'" and Form="'.$row->Form.'" and Errors <> "" AND Errors IS NOT NULL';
          $q3 = 'UPDATE TempPersonnelInfo SET Errors = "Duplicate Record" WHERE QuadratName = "'.$row->QuadratName.'" AND PersonnelID='.$row->PersonnelID.' AND RoleID='.$row->RoleID.' AND DateWorked="'.$row->DateWorked.'" and Form="'.$row->Form.'" AND (Errors = "" OR ISNULL(Errors))';
          $runQ2 = $this->screeningdb->query($q2);
          $runQ3 = $this->screeningdb->query($q3);
        }
      }
    }
DROP TABLE IF EXISTS `TempPersonnelInfo`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempPersonnelInfo` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `QuadratName` varchar(12) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `PersonnelID` smallint(5) unsigned DEFAULT NULL,
  `Form` varchar(32) DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `Role` varchar(128) DEFAULT NULL,
  `RoleID` smallint(5) unsigned DEFAULT NULL,
  `DateWorked` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexQuadratName` (`QuadratName`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

Add post-census summary statistics page

As a principal investigator, I want to view a report of post-census summary statistics (similar to post-screening in CTFSWeb) which crosschecks my new, complete census against historical data from past censuses, so that I can identify and address any outliers and abnormalities in the data before final upload to the Smithsonian Digital Repository.

checkDuplTagout validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

checkDuplTagout Check for duplicate Tag+StemTag (unique combination) in another database.  Uses ViewFullTable Cloud Checks against past censuses to make sure new trees don’t repeat a tag. Requires record of all tags that have ever been found at a site.

Tables:

  • filein: TempLocations, TempNewPlants, TempMultiStems
  • fileout: ViewFullTable (this data would be in the "cloud", it should contain all of the stems) Ask Suzanne about this: "Is it for one census, previous census, all data everywhere?"
    public function checkDuplTagout ($filein, $fileout, $plotid)
    {
        //Check for duplicate Tree tag in another database
        if ($fileout == 'ViewFullTable')
        {
          $q1 = "SELECT TempID,Tag FROM ".$filein." WHERE Tag IN (SELECT DISTINCT TAG FROM ".$fileout." WHERE PlotID=".$plotid.") AND Errors<>'NONE'";
        }
        else
        {
          $q1 = "SELECT TempID,Tag FROM ".$filein." WHERE Tag IN (SELECT DISTINCT TAG FROM ".$fileout." WHERE PlotID=".$plotid." UNION SELECT DISTINCT TAG FROM ".$fileout."Error WHERE PlotID=".$plotid.") AND Errors<>'NONE'";
        }

        $runQuery1 = $this->screeningdb->query($q1);

        if ($runQuery1->num_rows() > 0)
        {
           foreach($runQuery1->result() as $row)
           {
              $q2 = 'UPDATE '.$filein.' SET Errors = CONCAT(TRIM(Errors),";Tag exists in '.$fileout.' table") WHERE TempID = '.$row->TempID.' AND Errors <> "" AND Errors IS NOT NULL';
              $q3 = 'UPDATE '.$filein.' SET Errors = "Tag exists in '.$fileout.' table" WHERE TempID ='.$row->TempID.' AND (Errors = "" OR ISNULL(Errors))';
              $runQuery2 = $this->screeningdb->query($q2);
              $runQuery3 = $this->screeningdb->query($q3);
           }
        }
    }

Add page for administrators to update validation scripts

As a stakeholder, I want database managers to be able to add/edit/delete validation scripts (written in SQL and translated to other languages in the backend) as need be, so that the ForestGEO App is adaptable and maintainable long term.

Search "configure driven development" for more information.

Investigate authentication with Azure static web apps

We want to figure out how to deploy the app/what backend to use. One option is to publish the frontend as a static app that connects to a backend on Azure.

One requirement in whatever backend design we choose is that we need to be able to handle user authentication.

This tutorial says it can show you how to hook up your React app to an Azure backend + handle authentication:

https://docs.microsoft.com/en-us/learn/modules/publish-static-web-app-authentication/

It would be good to explore this to see if it will work for us.

Sample code from tutorial; https://github.com/MicrosoftDocs/mslearn-staticwebapp-authentication/tree/main

Design Backend Architecture diagram

As a developer, I want to see and visualize how the proposed solution looks like, what backend services we are going to use and how they are connected between each other.

screenMainStems validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenMainStems Assumes there are main stem codes. Check if there is more than one main stem, if there is none, if main stem died, etc. for multiple-stemmed trees Get all the codes Get all the multiple-stemmed trees Screen for more than one main stem Screen if no main stem Screen if main stem died and there are other live stems Screen if main stem does not have largest dbh Local Look for possible errors related to main stems. Is there more than one main stem (answer should be no, if yes, thrown an
    public function screenMainStems ($fileName)
    {
      //Assumes there are main stem codes
      //Check if there is more than one main stem, if there is none, if main stem died, etc. for multiple-stemmed trees


      // Get all the codes
      $codetmp = $this->_getCodes('main');
      $maincode = $codetmp[0];   //Should be one main stem code only

      $deadarr = $this->_getCodes('dead');
      $resprarr = $this->_getCodes('stem lost');

      // Get all the multiple-stemmed trees
      $q1 = "SELECT Tag, COUNT(Tag) as stems FROM ".$fileName." WHERE Errors<>'NONE' GROUP BY Tag HAVING stems>1";
      $runQ1 = $this->screeningdb->query($q1);

      foreach ($runQ1->result() as $q1Row)
      {

         $nummain = 0;
         $largestdbh = 0;
         $largestdbhid = 0;
         $numalive = 0;
         $maintempid = 0;
         $deadtreeid = 0;
         $mainst = 'n';
         $deadst = 'n';
         $resprst = 'n';
         $maindead = 'n';


         $q2 = "SELECT TempID, Codes, DBH, Errors FROM ".$fileName." WHERE Tag = '".$q1Row->Tag."' AND Errors<>'NONE'";
         $runQ2 = $this->screeningdb->query($q2);

         foreach($runQ2->result() as $q2Row)
         {
           if ($q2Row->DBH > $largestdbh)
           {
             $largestdbh = $q2Row->DBH;
             $largestdbhid = $q2Row->TempID;
           }

           $checkcodes = trim($q2Row->Codes);
           // Check codes

           if (!is_null($checkcodes) AND $checkcodes<>'*' AND $checkcodes<>'')
           {
             $nocodes = SUBSTR_COUNT($checkcodes,';');
             $arrcodes = explode(";", $checkcodes);
             for ($i = 0; $i <= $nocodes; $i++)
             {
                if (in_array($arrcodes[$i],$deadarr))
                {
                  $deadst = 'y';
                  $deadtreeid = $q2Row->TempID;
                }

                if (in_array($arrcodes[$i],$resprarr))
                {
                  $resprst = 'y';
                }

                if ($arrcodes[$i] === $maincode)
                {
                   $mainst = 'y';
                   $nummain++ ;
                   $maintempid = $q2Row->TempID;
                   // Change to this tempID even if dbh is the same, since this is main stem
                   if ($q2Row->DBH == $largestdbh)
                   $largestdbhid = $q2Row->TempID;
                 }  // end if main code
              }  //end for
            }  //end if there are codes

            if ($mainst === 'y' AND $deadst === 'y' and $deadtreeid == $maintempid)
            {
              $maindead = 'y';
            }
            if ($q2Row->DBH >= 10 || $resprst === 'y')
            {
              $numalive++;
            }
      }  // End for looking at stems of one tag

       // Screen for more than one main stem
       if ($nummain > 1 )
       {
           $q3 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";More than 1 main stem") WHERE Tag = "'.$q1Row->Tag.'" and codes like "%'.$maincode.'%"';
           $runQ3 = $this->screeningdb->query($q3);
           if ($largestdbhid > 0)
           {
             $q4 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid.' and Errors not like "%Main stem?%"';
             $runQ4 = $this->screeningdb->query($q4);
           }
        }

        // Screen if no main stem
        if ($nummain == 0 )
        {
           $q3 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";No main stem code") WHERE Tag = "'.$q1Row->Tag.'"';
           $runQ3 = $this->screeningdb->query($q3);
           if ($largestdbhid > 0)
           {
              $q4 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid.' and Errors not like "%Main stem?%"';
              $runQ4 = $this->screeningdb->query($q4);
           }
        }

        // Screen if main stem died and there are other live stems
        if ($maindead === 'y' && $numalive > 0 )
        {
           $q3 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Main stem is dead") WHERE TempID = '.$deadtreeid;
           $runQ3 = $this->screeningdb->query($q3);
           if ($largestdbhid > 0)
           {
             $q4 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid .' and Errors not like "%Main stem?%"';
             $runQ4 = $this->screeningdb->query($q4);
           }
         }

         // Screen if main stem does not have largest dbh
         if ($maintempid <> $largestdbhid AND $maintempid>0 AND $largestdbhid>0)
         {
            $q3 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Another stem has larger dbh?") WHERE TempID = '.$maintempid;
            $runQ3 = $this->screeningdb->query($q3);

            $q4 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid.' and Errors not like "%Main stem?%"';
            $runQ4 = $this->screeningdb->query($q4);
          }

      }  // End foreach

      $q3 = 'UPDATE '.$fileName.' SET Errors = substring(Errors,2) WHERE substring(Errors,1,1)=";"';
      $runQ3 = $this->screeningdb->query($q3);
    }



    public function _getCodes($description)
    {
      $this->screeningdb->select('TSMCode');
      $this->screeningdb->like('Description',$description);
      $query = $this->screeningdb->get('TSMAttributes');

      $codesarr = array();

      if($query->num_rows() > 0)
      {
        $j = 0;
        foreach($query->result() as $row)
        $codesarr[$j++] = $row->TSMCode;
      }
      else
      {
        $codesarr[0] = 'none';
      }

      return $codesarr;
    }




    public function screenMainStems2 ($file1, $file2)
    {
      //Check main stems for multiple-stemmed trees in 2 files: Old Trees and Multiple Stems

      // Get all the codes
      $codetmp = $this->_getCodes('main');
      $maincode = $codetmp[0];   //Should be one main stem code only

      $deadarr = $this->_getCodes('dead');
      $resprarr = $this->_getCodes('stem lost');

      $tagarray1 = array();
      // Get all the multiple-stemmed trees in first file
      $q1 = "SELECT Tag, COUNT(Tag) AS stems FROM ".$file1." WHERE Errors<>'NONE' GROUP BY Tag HAVING stems>1";
      $queryQ1 = $this->screeningdb->query($q1);

      if ($queryQ1->num_rows() > 0)
      {
        $i = 0;
        foreach($queryQ1->result() as $rowQ1)
        {
          $tagarray1[$i++] = $rowQ1->Tag;
        }
      }

      // Get all the tree tags from multiple-stemmed file
      $q2 = "SELECT DISTINCT Tag FROM ".$file2." WHERE Errors<>'NONE'";
      $queryQ2 = $this->screeningdb->query($q2);
      if ($queryQ2->num_rows() > 0)
      {
        $i = 0;
        foreach($queryQ2->result() as $rowQ2)
        {
          $tagarray2[$i++] = $rowQ2->Tag;
        }
        $multistems = 'y';
        $tagarray = array_unique(array_merge($tagarray1,$tagarray2));
      }
      else
      {
        $multistems = 'n';
        $tagarray = $tagarray1;
      }

      foreach ($tagarray as $key => $val)
      {
        $nummain = 0;
        $largestdbh = 0;
        $largestdbhid = 0;
        $numalive = 0;
        $maintempid = 0;
        $deadtreeid = 0;
        $mainst = 'n';
        $deadst = 'n';
        $resprst = 'n';
        $maindead = 'n';

        unset($treearr);

        $q3 = "SELECT TempID, Codes, DBH, Errors FROM ".$file1." WHERE Tag = '".$val."' AND Errors<>'NONE'";
        $queryQ3 = $this->screeningdb->query($q3);
        $i = 0;
        foreach($queryQ3->result() as $rowQ3)
        {
          $treearr['File'][$i] = $file1;
          $treearr['tempID'][$i] = $rowQ3->TempID;
          $treearr['codes'][$i] = $rowQ3->Codes;
          $treearr['DBH'][$i] = $rowQ3->DBH;
          $treearr['Errors'][$i] = $rowQ3->Errors;
          $i++;
        }

        if ($multistems == 'y')
        {
          $q4 = "SELECT TempID, Codes, DBH, Errors FROM ".$file2." WHERE Tag = '".$val."' AND Errors<>'NONE'";
          $queryQ4 = $this->screeningdb->query($q4);
          foreach($queryQ4->result() as $rowQ4)
          {
            $treearr['File'][$i] = $file2;
            $treearr['tempID'][$i] = $rowQ4->TempID;
            $treearr['codes'][$i] = $rowQ4->Codes;
            $treearr['DBH'][$i] = $rowQ4->DBH;
            $treearr['Errors'][$i] = $rowQ4->Errors;
            $i++;
          }
        }

        $numstems = sizeof($treearr['tempID']);

        for ($i = 0; $i < $numstems; $i++)
        {
          if ($treearr["DBH"][$i] > $largestdbh)
          {
            $largestdbh = $treearr["DBH"][$i];
            $largestdbhid = $treearr["tempID"][$i];
            $largestdbhfile = $treearr["File"][$i];
          }

          $checkcodes = trim($treearr["codes"][$i]);

          // Check codes
          if (!is_null($checkcodes) AND $checkcodes<>'*' AND $checkcodes<>'')
          {
            $nocodes = SUBSTR_COUNT($checkcodes,';');
            $arrcodes = explode(";", $checkcodes);
            for ($j = 0; $j <= $nocodes; $j++)
            {
              if (in_array($arrcodes[$j],$deadarr))
              {
                $deadst = 'y';
                $deadtreeid = $treearr["tempID"][$i];
                $deadtreefile = $treearr["File"][$i];
              }

              if (in_array($arrcodes[$j],$resprarr))
              {
                $resprst = 'y';
              }
              if ($arrcodes[$j] === $maincode)
              {
                $mainst = 'y';
                $nummain++ ;
                $maintempid = $treearr["tempID"][$i];
                $mainfile = $treearr["File"][$i];
                // Change to this tempID even if dbh is the same, since this is main stem
                if ($treearr["DBH"][$i] == $largestdbh)
                {
                  $largestdbhid = $treearr["tempID"][$i];
                  $largestdbhfile = $treearr["File"][$i];
                }
              }  // end if main code

            }  //end for

          }  //end if there are codes


          if ($mainst === 'y' AND $deadst === 'y' and $deadtreeid == $maintempid)
          {
            $maindead = 'y';
          }
          if ($treearr["DBH"][$i] >= 10 || $resprst === 'y')
          {
            $numalive++;
          }
        }  // End while looking at stems of one tag


        // Screen for more than one main stem
        if ($nummain > 1 )
        {
          $q2 = 'UPDATE '.$file1.' SET Errors = CONCAT(TRIM(Errors),";More than 1 main stem code") WHERE Tag = "'.$val.'" and Codes like "%'.$maincode.'%"';
          $runQ2 = $this->screeningdb->query($q2);

          $q3 = 'UPDATE '.$file2.' SET Errors = CONCAT(TRIM(Errors),";More than 1 main stem code") WHERE Tag = "'.$val.'" and Codes like "%'.$maincode.'%"';
          $runQ3 = $this->screeningdb->query($q3);

          if ($largestdbhid > 0)
          {
            $q4 = 'UPDATE '.$largestdbhfile.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid.' and Errors not like "%Main stem?%"';
            $runQ4 = $this->screeningdb->query($q4);
          }
        }

        // Screen if no main stem
        if ($nummain == 0 )
        {
          $q2 = 'UPDATE '.$file1.' SET Errors = CONCAT(TRIM(Errors),";No main stem code") WHERE Tag = "'.$val.'"';
          $runQ2 = $this->screeningdb->query($q2);

          $q3 = 'UPDATE '.$file2.' SET Errors = CONCAT(TRIM(Errors),";No main stem code") WHERE Tag = "'.$val.'"';
          $runQ3 = $this->screeningdb->query($q3);

          if ($largestdbhid > 0)
          {
            $q4 = 'UPDATE '.$largestdbhfile.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid.' and Errors not like "%Main stem?%"';
            $runQ4 = $this->screeningdb->query($q4);
          }
        }

        // Screen if main stem died and there are other live stems
        if ($maindead === 'y' && $numalive > 0 )
        {
          $q2 = 'UPDATE '.$deadtreefile.' SET Errors = CONCAT(TRIM(Errors),";Main stem is dead") WHERE TempID = '.$deadtreeid;
          $runQ2 = $this->screeningdb->query($q2);
          if ($largestdbhid > 0)
          {
            $q4 = 'UPDATE '.$largestdbhfile.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid .' and Errors not like "%Main stem?%"';
            $runQ4 = $this->screeningdb->query($q4);
          }
        }


        // Screen if main stem does not have largest dbh
        if ($maintempid <> $largestdbhid AND $maintempid>0 AND $largestdbhid>0)
        {
          $q2 = 'UPDATE '.$mainfile.' SET Errors = CONCAT(TRIM(Errors),";Another stem has larger dbh?") WHERE TempID = '.$maintempid;

          $q4 = 'UPDATE '.$largestdbhfile.' SET Errors = CONCAT(TRIM(Errors),";Main stem?") WHERE TempID = '.$largestdbhid.' and Errors not like "%Main stem?%"';
        }
      }  // End foreach ($tagarray as $key => $val)

      $q2 = 'UPDATE '.$file1.' SET Errors = substring(Errors,2) WHERE substring(Errors,1,1)=";"';
      $runQ2 = $this->screeningdb->query($q2);

      $q3 = 'UPDATE '.$file2.' SET Errors = substring(Errors,2) WHERE substring(Errors,1,1)=";"';
      $runQ3 = $this->screeningdb->query($q3);
    }
DROP TABLE IF EXISTS `TSMAttributes`;
CREATE TABLE `TSMAttributes` (
  `TSMID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TSMCode` char(10) NOT NULL,
  `Description` varchar(128) NOT NULL,
  `Status` enum('alive','alive-not measured','dead','missing','broken below','stem dead') DEFAULT NULL,
  PRIMARY KEY (`TSMID`)
) ENGINE=InnoDB AUTO_INCREMENT=66 DEFAULT CHARSET=latin1;

screenDiam validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenDiam Check for diameter range (d <0 and d> maxpossible raises an error). "SELECT TempID FROM ".$fileName." WHERE DBH<>0 AND (DBH>".$maxDBH." OR DBH<".$minDBH.") AND Errors<>'NONE' Minimum DBH for a stem is often 1.0, but this depends on whether site uses mm or cm. Local Checks to see if a tree’s diameter at breastheight (DBH) is within a predefined range.  Note that some sites record data in millimeters and others in centimeters, so minimum DBH is often 1.0 but not always.

Some was done on this already: see #107 (comment)

Separate pages from components

Right now we have both pages and components in FrontEnd/src/components.

We should separate these to make the codebase easier to navigate. For example, Browse.tsx, Report.tsx, and Validate.tsx should be under src/pages/, the rest are UI components and can stay where they are.

checkGrowth validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

checkGrowth Check for extreme growth - for Old Trees form Flags stems where absolute annual growth > $annualGrowth mm Check for shrinkage in recensuses - only for Old Trees form Flags records with annual relative growth <= -0.05 in stems with previous dbh>=50mm and no current stem lost code Local To calculate growth rates, need DBH and date of previous census.
   $tempTableName = 'TempOldTrees';

   //Check for extreme growth - for Old Trees form
   //Tags stems where absolute annual growth > 65 mm
   $annualGrowth = '65';

   //Check for shrinkage in recensuses - only for Old Trees form
   //Tags records with annual relative growth <= -0.05 in stems with previous dbh>=50mm and no current stem lost code
   $dbhCheck2 = '50';
   $annualShrink = '-0.05';

    ////Helper to above////
    public function _getCodes($description)
    {
      $this->screeningdb->select('TSMCode');
      $this->screeningdb->like('Description',$description);
      $query = $this->screeningdb->get('TSMAttributes');

      $codesarr = array();

      if($query->num_rows() > 0)
      {
        $j = 0;
        foreach($query->result() as $row)
        $codesarr[$j++] = $row->TSMCode;
      }
      else
      {
        $codesarr[0] = 'none';
      }

      return $codesarr;
    }



    //////Old Trees screening/////////
    public function checkGrowth ($filen,$minDBH,$annualGrowth,$dbhCheck2,$annualShrink)
    {
      //Check for extreme growth - for Old Trees form
      //Tags stems where absolute annual growth > $annualGrowth mm

      $q1 = 'UPDATE '.$filen.' AS a,TempQuadratDates AS b SET Errors = CONCAT(TRIM(Errors),";DBH too large?") WHERE a.QuadratName=b.QuadratName AND HOM=OldHOM AND
             Errors <> "" AND Errors IS NOT NULL AND Errors<>"NONE" AND DBH>='.$minDBH.' AND OldDBH>='.$minDBH.' AND (DBH-OldDBH)/((to_days(ExactDate)-to_days(b.PrevDate))/365.25) > '.$annualGrowth;
      $q2 = 'UPDATE '.$filen.' AS a,TempQuadratDates AS b SET Errors = "DBH too large?" WHERE a.QuadratName=b.QuadratName AND HOM=OldHOM AND
             (Errors = "" OR ISNULL(Errors)) AND Errors<>"NONE"AND DBH>='.$minDBH.' AND OldDBH>='.$minDBH.' AND (DBH-OldDBH)/((to_days(ExactDate)-to_days(b.PrevDate))/365.25) > '.$annualGrowth;
      $runQ1 = $this->screeningdb->query($q1);
      $runQ2 = $this->screeningdb->query($q2);


      //Check for shrinkage in recensuses - only for Old Trees form
      //Tags records with annual relative growth <= -0.05 in stems with previous dbh>=50mm and no current stem lost code

      $resprcodes = $this->_getCodes('stem lost'); //The helper public function above

      //$q1 = "SELECT TSMCode FROM TSMAttributes WHERE LOWER(DESCRIPTION) like '%resprout%' or LOWER(DESCRIPTION) like '%broken%'";
      //$q1 = "SELECT TSMCode FROM TSMAttributes WHERE LOWER(DESCRIPTION)='stem lost'";
      //$r1 = mysql_query($q1, $connect) or die('Selecting stem lost codes failed: ' . mysql_error());
      //$resprcodes=array();
      //if (mysql_num_rows($r1)> 0) {
      //   $j=0;
      //   while ($row = mysql_fetch_assoc($r1)) {
      //            $resprcodes[$j++] = $row["TSMCode"]; }
      //  }

      $q2 = "Select TempID FROM ".$filen;
      $queryQ2 = $this->screeningdb->query($q2);
      // Go through entire file
      foreach($queryQ2->result() as $rowQ2)
      {
        //Check whether tree has resprout or broken code, in which case do not run shrinkage test
        $tempID = $rowQ2->TempID;
        $q3 = "SELECT Codes FROM ".$filen." WHERE TempID = ".$tempID;
        $queryQ3 = $this->screeningdb->query($q3);
        $rowQ3 = $queryQ3->row();
        $codes = $rowQ3->Codes;
        $continue = 0;
        if (!is_null($codes) AND $codes<>'*' AND $codes<>'')
        {
          $nocodes = SUBSTR_COUNT($codes,';');
          $arrcodes = explode(";", $codes);
          for ($i = 0; $i <= $nocodes; $i++)
          {
            if (in_array($arrcodes[$i], $resprcodes))
            {
              $continue=1;
            }
          }
        }

         if($continue==0)
         {

            $q4 = 'UPDATE '.$filen.' AS a,TempQuadratDates AS b SET Errors = CONCAT(TRIM(Errors),";DBH shrank too much") WHERE a.QuadratName=b.QuadratName AND HOM=OldHom
                   AND Errors <> "" AND Errors IS NOT NULL AND Errors<>"NONE" AND DBH>='.$minDBH.' AND olddbh>='.$dbhCheck2.' AND ((DBH-OldDBH)/OldDBH)/((to_days(ExactDate)-to_days(b.PrevDate))/365.25) <='.$annualShrink.'
                   AND TempID = '.$tempID;
            $q5 = 'UPDATE '.$filen.' AS a, TempQuadratDates AS b SET Errors = "DBH shrank too much?" WHERE a.QuadratName=b.QuadratName AND HOM=OldHOM
                   AND (Errors = "" OR ISNULL(Errors)) AND Errors<>"NONE" AND DBH>='.$minDBH.' AND olddbh>='.$dbhCheck2.' AND ((DBH-OldDBH)/OldDBH)/((to_days(ExactDate)-to_days(b.PrevDate))/365.25) <='.$annualShrink.'
                   AND tempID = '.$tempID;
            $runQ4 = $this->screeningdb->query($q4);
            $runQ5 = $this->screeningdb->query($q5);

          }
      }
    }
DROP TABLE IF EXISTS `TSMAttributes`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TSMAttributes` (
  `TSMID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TSMCode` char(10) NOT NULL,
  `Description` varchar(128) NOT NULL,
  `Status` enum('alive','alive-not measured','dead','missing','broken below','stem dead') DEFAULT NULL,
  PRIMARY KEY (`TSMID`)
) ENGINE=InnoDB AUTO_INCREMENT=66 DEFAULT CHARSET=latin1;

(1,'F','Fallen or missing (1982)','alive'),(2,'L','Leaning','alive')




DROP TABLE IF EXISTS `TempOldTrees`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempOldTrees` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=6635 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `TempOldTrees`
--

LOCK TABLES `TempOldTrees` WRITE;
/*!40000 ALTER TABLE `TempOldTrees` DISABLE KEYS */;
INSERT INTO `TempOldTrees` VALUES (6117,492702,'27545',1026544,'',NULL,'0000',13,NULL,'protte',NULL,114,349,11,NULL,NULL,NULL,156,151,1.3,1.3,'','','','2019-08-27','',NULL,'n',NULL,NULL,NULL,'\r',NULL,NULL),(6118,492703,'27546',1026545,'',NULL,'0000',14,NULL,'soroaf',NULL,114,349,11,NULL,NULL,NULL,101,103,1.3,1.3,'','','','2019-08-27','',NULL,'n',NULL,NULL,NULL,'\r',NULL,NULL)

checkDuplTagin validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

checkDuplTagin Check for duplicate Tag +StemTag (unique combination) within temporary database. Local Checks against current census to make sure trees don’t repeat a tag.

Note: is this needed? Seems to be commented out in CTFS

    public function checkDuplTagin ($file)
    {
      //Check for duplicate Tree tags within temporary database.
      $q1 = "SELECT Tag, COUNT(Tag) AS cnt FROM ".$file." WHERE Errors<>'NONE' GROUP BY Tag HAVING cnt > 1";
      $runQuery1 = $this->screeningdb->query($q1);
      if ($runQuery1->num_rows() > 0)
      {
         foreach($runQuery1->result() as $row)
         {
            $q2 = 'UPDATE '.$file.' SET Errors = CONCAT(TRIM(Errors),";Duplicate Tag") WHERE Tag = "'.$row->Tag.'" AND Errors <> "" AND Errors IS NOT NULL';
            $q3 = 'UPDATE '.$file.' SET Errors = "Duplicate Tag" WHERE Tag = "'.$row->Tag.'" AND (Errors = "" OR ISNULL(Errors))';
            $runQuery2 = $this->screeningdb->query($q2);
            $runQuery3 = $this->screeningdb->query($q3);
         }
      }
    }
DROP TABLE IF EXISTS `TempLocations`;
CREATE TABLE `TempLocations` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTag` (`Tag`),
  KEY `indexSTemTag` (`Tag`,`StemTag`),
  KEY `indexTreeID` (`TreeID`),
  KEY `indexQuadratName` (`QuadratName`)
) ENGINE=InnoDB AUTO_INCREMENT=299 DEFAULT CHARSET=latin1;

(291,'0004','28504',NULL,15.4522,7.15468,NULL,NULL,11,349,114,'',NULL)

screenQuadratInfo validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenQuadratInfo Check for date ranges SELECT TempID FROM TempPersonnelInfo WHERE DateWorked NOT BETWEEN '".$start."' AND '".$end."' AND Errors<>'NONE' Local Makes sure all dates recorded in data collection, by quadrat, are within range of census. Appears to be related to startCensusDate.
    public function censusStartDate($plotID,$censusNumber)
    {
      $this->screeningdb->select('StartDate');
      $this->screeningdb->where('PlotID',$plotID);
      $this->screeningdb->where('PlotCensusNumber',$censusNumber);
      $query = $this->screeningdb->get('Census');
      if($query->num_rows() > 0)
      {
        $row = $query->row();
        $startDate = $row->StartDate;
      }
      else
      {
        echo "Error in screeningmodel/censusStartDate";
        die();
      }
      return $startDate;
    }


          $startDate = $this->screeningmodel->censusStartDate($plotID,$censusNumber);
          $enddate = date("Y-m-d");

    public function screenQuadratInfo ($start,$end)
    {
      //Check for date ranges
      $q1 = "SELECT TempID FROM TempPersonnelInfo WHERE DateWorked NOT BETWEEN '".$start."' AND '".$end."' AND Errors<>'NONE'";
      $query1 = $this->screeningdb->query($q1);
      if ($query1->num_rows() > 0)
      {
        foreach($query1->result() as $row)
        {
          $q2 = 'UPDATE TempPersonnelInfo SET Errors = CONCAT(TRIM(Errors),";Check Date") WHERE TempID = '.$row->TempID.' AND Errors <> "" AND Errors IS NOT NULL';
          $q3 = 'UPDATE TempPersonnelInfo SET Errors = "Check Date" WHERE tempID = '.$row->TempID.' AND (Errors = "" OR ISNULL(Errors))';
        }
      }
    }
DROP TABLE IF EXISTS `TempPersonnelInfo`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempPersonnelInfo` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `QuadratName` varchar(12) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `PersonnelID` smallint(5) unsigned DEFAULT NULL,
  `Form` varchar(32) DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `Role` varchar(128) DEFAULT NULL,
  `RoleID` smallint(5) unsigned DEFAULT NULL,
  `DateWorked` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexQuadratName` (`QuadratName`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Add file list page

As a data entry person, I want to browse a list of previously uploaded CSV files, so that I can keep track of the work my team has done on a particular census and manage duplicate data.

  • File list is organized by form type, quadrat number, date entered, who it was entered by, validation status, and any other relevant, distinguishing meta data
  • File list is mutable with edit and delete actions available

screenCodes validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenCodes Check for invalid codes and for Trees with both a D code and diameter measurement Get codes from TSMAttributes table and put in 1-D array Check if there are duplicate codes Flag if code does not appear in TSMAttributes table Flag if tree is dead and there is a dbh Local Multi-faceted validation.  Screens for trees that have a recorded DBH measurement but are also marked as “D” for dead.   Checks to see if there are any duplicate codes.   Throws an error if code doesn’t appear in TSMAttributes table (master table of codes available, vary by site).

For these tables:

  • TempOldTrees
  • TempMultiStems
  • TempNewPlants
    public function screenCodes ($fileName)
    {
      //Check for invalid codes AND
      //for Trees with both a D code and diameter measurement

      //Get codes from TSMAttributes table and put in 1-D array
      $queryCodes = "SELECT TSMCode FROM TSMAttributes";
      $runQueryCodes = $this->screeningdb->query($queryCodes);

      $j = 0;
      foreach($runQueryCodes->result() as $row)
      {
        $tsmcodesarray[$j++] = $row->TSMCode;
      }

      //Get codes that mean dead
      $queryCodes = "SELECT TSMCode FROM TSMAttributes WHERE LOWER(Description) LIKE '%dead%'";
      $runQueryCodes = $this->screeningdb->query($queryCodes);
      $j = 0;
      foreach($runQueryCodes->result() as $row)
      {
        $deadcodesarray[$j++] = $row->TSMCode;
      }

      $query = "SELECT TempID,Codes FROM ".$fileName." WHERE !ISNULL(Codes) AND Codes<>'*' AND Codes<>''";
      $runQuery = $this->screeningdb->query($query);
      foreach($runQuery->result() as $queryRow)
      {
          $nocodes = SUBSTR_COUNT($queryRow->Codes,';');
          $arrcodes = explode(";", $queryRow->Codes);

          //Check if there are duplicate codes
          $uniquecodes = array_unique($arrcodes);
          $cntcodes1 = count($arrcodes);
          $cntcodes2 = count($uniquecodes);
          if ($cntcodes1 <> $cntcodes2)
          {
             $q2 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Duplicate code") WHERE TempID = '.$queryRow->TempID.' AND Errors<>"NONE" AND Errors <>"" AND Errors IS NOT NULL';
             $q3 = 'UPDATE '.$fileName.' SET Errors = "Duplicate code" WHERE TempID = '.$queryRow->TempID.' AND Errors<>"NONE" AND (Errors = "" OR ISNULL(Errors))';
             $runQ2 = $this->screeningdb->query($q2);
             $runQ3 = $this->screeningdb->query($q3);
          }

          for ($i = 0; $i <= $nocodes; $i++)
          {
            //Tag if code does not appear in TSMAttributes table
            if (!in_array($arrcodes[$i], $tsmcodesarray))
            {
              $q2 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Check code:","'.$arrcodes[$i].'") WHERE TempID = '.$queryRow->TempID.' AND Errors<>"NONE" AND Errors <>"" AND Errors IS NOT NULL';
              $q3 = 'UPDATE '.$fileName.' SET Errors = "Check code:'.$arrcodes[$i].'" WHERE TempID = '.$queryRow->TempID.' AND Errors<>"NONE" AND (Errors = "" OR ISNULL(Errors))';
              $runQ2 = $this->screeningdb->query($q2);
              $runQ3 = $this->screeningdb->query($q3);
            }

            //Tag if tree is dead and there is a dbh
            if (in_array($arrcodes[$i], $deadcodesarray))
            {
              $q2 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Tree dead with dbh") WHERE tempID = '.$queryRow->TempID.' AND Errors<>"NONE" AND Errors <>"" AND Errors IS NOT NULL AND DBH>=10';
              $q3 = 'UPDATE '.$fileName.' SET Errors = "Tree dead with dbh" WHERE TempID = '.$queryRow->TempID.' AND Errors<>"NONE" AND (Errors = "" OR ISNULL(Errors)) AND DBH>=10';
              $runQ2 = $this->screeningdb->query($q2);
              $runQ3 = $this->screeningdb->query($q3);
            }

          }
       }
    }
DROP TABLE IF EXISTS `TSMAttributes`;
CREATE TABLE `TSMAttributes` (
  `TSMID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TSMCode` char(10) NOT NULL,
  `Description` varchar(128) NOT NULL,
  `Status` enum('alive','alive-not measured','dead','missing','broken below','stem dead') DEFAULT NULL,
  PRIMARY KEY (`TSMID`)
) ENGINE=InnoDB AUTO_INCREMENT=66 DEFAULT CHARSET=latin1;

(1,'F','Fallen or missing (1982)','alive'),(2,'L','Leaning','alive')


DROP TABLE IF EXISTS `TempOldTrees`;
CREATE TABLE `TempOldTrees` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=6635 DEFAULT CHARSET=latin1;

 (6117,492702,'27545',1026544,'',NULL,'0000',13,NULL,'protte',NULL,114,349,11,NULL,NULL,NULL,156,151,1.3,1.3,'','','','2019-08-27','',NULL,'n',NULL,NULL,NULL,'\r',NULL,NULL),(6118,492703,'27546',1026545,'',NULL,'0000',14,NULL,'soroaf',NULL,114,349,11,NULL,NULL,NULL,101,103,1.3,1.3,'','','','2019-08-27','',NULL,'n',NULL,NULL,NULL,'\r',NULL,NULL)

Removal of Prototypes folder from the main branch

The Prototypes folder contains several APIs that were used previously and are getting in the way of our Azure Functions API which is located in /api folder.
The Readme file says:

This is an archive of prototypes we built for the ForestGEO App. Since these are prototypes we built for the purpose of learning and proof of concept, we don't intend to build off the code in this folder. However, we archive these prototypes just in case we want to reference them later.

I'm going to backup the whole project, upload it to a cloud and remove the folder from GitHub (It is not intended for archival purposes, anyway)

screenDuplSpec validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenDuplSpec This public function checks to see if all the stems of a tree have the same species code Local All stems of a tree should have the same species code.
          $tempTableName = 'TempOldTrees';
          $tempTableName = 'TempNewPlants';

    public function  screenDuplSpec ($fileName)
    {
      //This public function checks to see if all the stems of a tree have the same species code
      //$q1 = "SELECT Tag,count(Tag) AS cnt FROM (SELECT DISTINCT Tag, Mnemonic, Errors FROM ".$fileName." GROUP BY Tag,Mnemonic,Errors UNION SELECT DISTINCT Tag, Mnemonic, Errors FROM ".$fileName."Error GROUP BY Tag,Mnemonic,Errors) AS t2 WHERE Errors<>'NONE' GROUP BY Tag HAVING cnt>1";
      $q1 = "SELECT Tag,count(Tag) AS cnt FROM (SELECT DISTINCT Tag, Mnemonic FROM ".$fileName." WHERE Errors<>'NONE' UNION SELECT DISTINCT Tag, Mnemonic FROM ".$fileName."Error WHERE Errors<>'NONE') AS t2 GROUP BY Tag HAVING cnt>1";
      $runQ1 = $this->screeningdb->query($q1);

      if ($runQ1->num_rows() > 0)
      {
         foreach($runQ1->result() as $row)
         {
            $q2 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";>1 Species") WHERE Tag = "'.$row->Tag.'" AND Errors <> "" AND Errors IS NOT NULL';
            $q3 = 'UPDATE '.$fileName.' SET Errors = ">1 Species" WHERE Tag ="'.$row->Tag.'" AND (Errors = "" OR ISNULL(Errors))';
            $runQ2 = $this->screeningdb->query($q2);
            $runQ3 = $this->screeningdb->query($q3);
         }
      }
    }
DROP TABLE IF EXISTS `TempOldTrees`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempOldTrees` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=6635 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;



DROP TABLE IF EXISTS `TempNewPlants`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempNewPlants` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=155 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

checkQuadrat validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

checkQuadrat Check for quadrats not matching another file, i.e. tempNewPlants. Is there an easier way??? Remember that there may be more than one record with the same tag in tempNewPlants "SELECT DISTINCT Tag, QuadratName FROM ".$fileout." WHERE PlotID=".$plotID Cloud Looks for quadrats that don’t match those available at a given site
    $filein     $tempTableName = 'TempLocations';
    $fileout    $tempNewTable = 'TempNewPlants';

    public function checkQuadrat ($filein, $fileout, $plotID)
    {
      //Check for quadrats not matching another file, i.e. tempNewPlants
      //Is there an easier way??? Remember that there may be more than one record with the same Tag in tempNewPlants

      $q1 = "SELECT Tag, QuadratName FROM ".$fileout." WHERE PlotID=".$plotID." GROUP BY Tag";
      $runQ1 = $this->screeningdb->query($q1);
      if ($runQ1->num_rows() > 0)
      {
        foreach($runQ1->result() as $row)
        {
           $q2 = 'UPDATE '.$filein.' SET Errors = CONCAT(TRIM(Errors),";Tree is in quadrat='.$row->QuadratName.' in '.$fileout.'") WHERE Tag = "'.$row->Tag.'" and QuadratName <> "'.$row->QuadratName.'"  AND Errors <> "NONE" AND Errors <> "" AND Errors IS NOT NULL';
           $q3 = 'UPDATE '.$filein.' SET Errors = "Tree is in quadrat='.$row->QuadratName.' in '.$fileout.'" WHERE Tag ="'.$row->Tag.'" AND QuadratName<>"'.$row->QuadratName.'" AND Errors <> "NONE" AND (Errors = "" OR ISNULL(Errors))';
           $runQ2 = $this->screeningdb->query($q2);
           $runQ3 = $this->screeningdb->query($q3);
        }
      }
    }
DROP TABLE IF EXISTS `TempLocations`;
CREATE TABLE `TempLocations` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTag` (`Tag`),
  KEY `indexSTemTag` (`Tag`,`StemTag`),
  KEY `indexTreeID` (`TreeID`),
  KEY `indexQuadratName` (`QuadratName`)
) ENGINE=InnoDB AUTO_INCREMENT=299 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
DROP TABLE IF EXISTS `TempNewPlants`;
CREATE TABLE `TempNewPlants` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=155 DEFAULT CHARSET=latin1;

Add version control capabilities

As a data entry person, after I make corrections to my data according to the error report and re-upload the CSV, I want this revised version to replace the old version, so that the data remains organized and duplicate free.

  • User is required to click to confirm replacement action
  • Implement version control methods to "soft" delete (mark as deleted but retain data until a certain point)

Fix logout link

As a data entry person, I want to be able to logout of my Microsoft account on ForestGEO App so that my data is protected and other users of my site's computer can login.

screenXY validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenXY Checks if the x and y-coordinates are within plot dimensions "SELECT TempID FROM ".$file." as a,Site as b WHERE a.PlotID=b.PlotID and a.X NOT BETWEEN 0 AND b.QDimX AND Errors<>'NONE' Local Plots have predefined dimensions, so this checks to make sure that stems’ X and Y coordinates are within range
TempLocations

    public function screenXY ($file, $plotID)
    {
      $q1 = "SELECT TempID FROM ".$file." as a,Site as b WHERE a.PlotID=b.PlotID and a.X NOT BETWEEN 0 AND b.QDimX AND Errors<>'NONE'";
      $runQuery1 = $this->screeningdb->query($q1);
      if ($runQuery1->num_rows() > 0)
      {
         foreach($runQuery1->result() as $row)
         {
            $q2 = 'UPDATE '.$file.' SET Errors = CONCAT(TRIM(Errors),";Check x-coordinate") WHERE TempID = '.$row->TempID.' AND Errors <> "" AND Errors IS NOT NULL';
            $q3 = 'UPDATE '.$file.' SET Errors = "Check x-coordinate" WHERE TempID = '.$row->TempID.' AND (Errors = "" OR ISNULL(Errors))';
            $runQuery2 = $this->screeningdb->query($q2);
            $runQuery3 = $this->screeningdb->query($q3);
         }
      }


      $q1 = "SELECT TempID FROM ".$file." as a,Site as b WHERE a.PlotID=b.PlotID and a.Y NOT BETWEEN 0 AND b.QDimY AND Errors<>'NONE'";
      $runQuery1 = $this->screeningdb->query($q1);
      if ($runQuery1->num_rows() > 0)
      {
         foreach($runQuery1->result() as $row)
         {
            $q2 = 'UPDATE '.$file.' SET Errors = CONCAT(TRIM(Errors),";Check y-coordinate") WHERE TempID = '.$row->TempID.' AND Errors <> "" AND Errors IS NOT NULL';
            $q3 = 'UPDATE '.$file.' SET Errors = "Check y-coordinate" WHERE TempID = '.$row->TempID.' AND (Errors = "" OR ISNULL(Errors))';
            $runQuery2 = $this->screeningdb->query($q2);
            $runQuery3 = $this->screeningdb->query($q3);
        }
      }
    }
```sql
DROP TABLE IF EXISTS `TempLocations`;
CREATE TABLE `TempLocations` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTag` (`Tag`),
  KEY `indexSTemTag` (`Tag`,`StemTag`),
  KEY `indexTreeID` (`TreeID`),
  KEY `indexQuadratName` (`QuadratName`)
) ENGINE=InnoDB AUTO_INCREMENT=299 DEFAULT CHARSET=latin1;

(291,'0004','28504',NULL,15.4522,7.15468,NULL,NULL,11,349,114,'',NULL)

Sample CSV files?

@esscott is there somewhere we can get access to/store some sample CSV that show what the data that users of the app would be uploading?

This is somewhat related to #38 but this is more to see examples of valid data that we can test with during development.

Would be nice to have a /data folder in the repo with a couple of examples

switch azure static web apps development accounts soon?

Do we need to switch the azure static web apps accounts soon? I think @OmarShehata mentioned so.

We can use some free azure credits I have allocated to me we can use for development.

Later on when it switches into production, we'll need to organize a ForestGEO/Smithsonian account.

Probably we have to change the tokens inside github -> settings -> secrets, and maybe some config in .github/workflows/azure-static-web-apps-agreeable-wave-08a957210.yml (I think that's it?)

screenDuplQuad validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

screenDuplQuad This public function checks to see if all the stems of a tree have the same quadrat Cloud All stems of a tree should be located within the same quadrat.
          $tempTableName = 'TempNewPlants';
          $tempTableName = 'TempOldTrees';
          $tempTableName = 'TempMultiStems';


    public function  screenDuplQuad ($fileName)
    {
      //This public function checks to see if all the stems of a tree have the same quadrat
      $q1 = "SELECT Tag,COUNT(Tag) AS cnt FROM (SELECT DISTINCT Tag, QuadratName, Errors FROM ".$fileName." GROUP BY Tag,QuadratName UNION SELECT DISTINCT Tag, QuadratName, Errors FROM ".$fileName."Error GROUP BY Tag, QuadratName) AS t2 WHERE Errors<>'NONE' GROUP BY Tag HAVING cnt>1";
      $runQ1 = $this->screeningdb->query($q1);

      if ($runQ1->num_rows() > 0)
      {
         foreach($runQ1->result() as $row)
         {
            $q2 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";>1 Quadrat") WHERE Tag = "'.$row->Tag.'" AND Errors <> "" AND Errors IS NOT NULL';
            $q3 = 'UPDATE '.$fileName.' SET Errors = ">1 Quadrat" WHERE Tag ="'.$row->Tag.'" AND (Errors = "" OR ISNULL(Errors))';
            $runQ3 = $this->screeningdb->query($q2);
            $runQ4 = $this->screeningdb->query($q3);
         }
      }
    }
DROP TABLE IF EXISTS `TempNewPlants`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempNewPlants` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=155 DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS `TempOldTrees`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempOldTrees` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=6635 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;


DROP TABLE IF EXISTS `TempMultiStems`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `TempMultiStems` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemID` int(10) unsigned DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `QuadratID` int(10) unsigned DEFAULT NULL,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Subquad` int(10) unsigned DEFAULT NULL,
  `SpeciesID` int(10) unsigned DEFAULT NULL,
  `Mnemonic` varchar(10) DEFAULT NULL,
  `SubSpeciesID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `PrimaryStem` varchar(20) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `DBH` float DEFAULT NULL,
  `OldDBH` float DEFAULT NULL,
  `HOM` float DEFAULT NULL,
  `OldHOM` float DEFAULT NULL,
  `Codes` varchar(50) DEFAULT NULL,
  `OldCodes` varchar(50) DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `ExactDate` date DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  `BigTrees` varchar(10) DEFAULT NULL,
  `Extra1` varchar(80) DEFAULT NULL,
  `Extra2` varchar(80) DEFAULT NULL,
  `Extra3` varchar(80) DEFAULT NULL,
  `Extra4` varchar(80) DEFAULT NULL,
  `Extra5` varchar(80) DEFAULT NULL,
  `DBHID` int(11) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTagStemTag` (`Tag`,`StemTag`),
  KEY `indexTreeIDStemID` (`TreeID`,`StemID`),
  KEY `indexQuadratNamePlotID` (`QuadratName`,`PlotID`),
  KEY `indexMnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=5115 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

checkSpecies validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

checkSpecies Check that species codes in filein are found in fileout "SELECT TempID,Mnemonic FROM ".$filein." WHERE Mnemonic IS NULL or Mnemonic NOT IN (SELECT DISTINCT Mnemonic FROM ".$fileout.") AND Errors<>'NONE' Cloud Makes sure species codes in dataset are in pre-defined list of acceptable species codes for a given site.
  • $filein
  • $fileout 'Species'
    public function  checkSpecies ($filein, $fileout)
    {
      //Check that species codes in filein are found in fileout

      $q1 = "SELECT TempID,Mnemonic FROM ".$filein." WHERE Mnemonic IS NULL or Mnemonic NOT IN (SELECT DISTINCT Mnemonic FROM ".$fileout.") AND Errors<>'NONE'";
      $runQ1 = $this->screeningdb->query($q1);

      if ($runQ1->num_rows() > 0)
      {
         foreach($runQ1->result() as $row)
         {
            $q2 = 'UPDATE '.$filein.' SET Errors = CONCAT(TRIM(Errors),";Check species") WHERE TempID = '.$row->TempID.' AND Errors <> "" AND Errors IS NOT NULL';
            $q3 = 'UPDATE '.$filein.' SET Errors = "Check species" WHERE TempID ='.$row->TempID.' AND (Errors = "" OR ISNULL(Errors))';
            $runQ2 = $this->screeningdb->query($q2);
            $runQ3 = $this->screeningdb->query($q3);
         }
      }
    }
DROP TABLE IF EXISTS `Species`;
CREATE TABLE `Species` (
  `SpeciesID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `CurrentTaxonFlag` smallint(6) DEFAULT NULL,
  `ObsoleteTaxonFlag` smallint(6) DEFAULT NULL,
  `GenusID` int(10) unsigned NOT NULL,
  `ReferenceID` smallint(5) unsigned DEFAULT NULL,
  `SpeciesName` char(64) DEFAULT NULL,
  `Mnemonic` char(10) DEFAULT NULL,
  `Authority` varchar(128) DEFAULT NULL,
  `IDLEVEL` enum('subspecies','species','superspecies','genus','family','multiple','none','variety') DEFAULT NULL,
  `FieldFamily` char(32) DEFAULT NULL,
  `Description` varchar(128) DEFAULT NULL,
  `Lifeform` enum('Emergent Tree','Tree','Midcanopy Tree','Understory Tree','Shrub','Herb','Liana') DEFAULT NULL,
  `LocalName` varchar(128) DEFAULT NULL,
  PRIMARY KEY (`SpeciesID`),
  KEY `Ref26208` (`GenusID`),
  KEY `Ref84209` (`ReferenceID`),
  KEY `Mnemonic` (`Mnemonic`)
) ENGINE=InnoDB AUTO_INCREMENT=1624 DEFAULT CHARSET=latin1;

Prototype CSV upload using Azure backend

Assuming we're using Azure Function Apps as our backend (see #53), we should prototype a simple flow for uploading & retrieving CSV files, so we can understand how this works, since this is the main data requirement of the app.

I think we will want to use Azure blob storage for this:

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction

So the backend/function app API should have (1) a POST route that can take a file & some metadata (the username of who uploaded it etc) (2) a route to download the file given some identifier

Wiring this up to the frontend can be a follow up task.

checkDuplStemin validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

checkDuplStemin Check for duplicate Tree and Stem tags within temporary database. Does not include dead stems that couldn't be tagged (StemTag=NULL) "SELECT Tag, StemTag, COUNT(Tag) as cnt FROM ".$fileName." GROUP BY Tag, StemTag HAVING cnt > 1" Local Appears to be the same as checkDuplTagin

Question: What are dead tags denoted by in the db?

For these tables:

  • TempOldTrees
  • TempMultiStems
  • TempNewPlants
  • TempLocations
    public function checkDuplStemin ($fileName)
    {
      //Check for duplicate Tree and Stem tags within temporary database.
      //Do not include dead stems that couldn't be tagged (StemTag=NULL)???
      //$q1 = "SELECT Tag, StemTag, COUNT(Tag) as cnt FROM (SELECT Tag, StemTag, Errors FROM ".$fileName." UNION SELECT Tag, StemTag, Errors FROM ".$fileName."Error) as a GROUP BY Tag, StemTag HAVING cnt > 1";
      $q1 = "SELECT Tag, StemTag, COUNT(Tag) as cnt FROM ".$fileName." GROUP BY Tag, StemTag HAVING cnt > 1";
      $runQuery1 = $this->screeningdb->query($q1);
      if ($runQuery1->num_rows() > 0)
      {
         foreach($runQuery1->result() as $row)
         {
            $q2 = 'UPDATE '.$fileName.' SET Errors = CONCAT(TRIM(Errors),";Duplicate Tag/StemTag") WHERE Tag = "'.$row->Tag.'" AND StemTag="'.$row->StemTag.'" AND Errors <> "" AND Errors IS NOT NULL';
            $q3 = 'UPDATE '.$fileName.' SET Errors = "Duplicate Tag/StemTag" WHERE Tag = "'.$row->Tag.'" AND StemTag="'.$row->StemTag.'" AND (Errors = "" OR ISNULL(Errors))';
            $runQuery2 = $this->screeningdb->query($q2);
            $runQuery3 = $this->screeningdb->query($q3);
         }
      }
    }
DROP TABLE IF EXISTS `TempLocations`;
CREATE TABLE `TempLocations` (
  `TempID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `QuadratName` varchar(12) DEFAULT NULL,
  `Tag` varchar(10) DEFAULT NULL,
  `StemTag` varchar(32) DEFAULT NULL,
  `X` float DEFAULT NULL,
  `Y` float DEFAULT NULL,
  `Comments` varchar(256) DEFAULT NULL,
  `TreeID` int(10) unsigned DEFAULT NULL,
  `PlotCensusNumber` int(10) unsigned DEFAULT NULL,
  `CensusID` int(10) unsigned DEFAULT NULL,
  `PlotID` int(10) unsigned DEFAULT NULL,
  `Errors` varchar(256) DEFAULT NULL,
  `OldTrees` varchar(10) DEFAULT NULL,
  PRIMARY KEY (`TempID`),
  KEY `indexTag` (`Tag`),
  KEY `indexSTemTag` (`Tag`,`StemTag`),
  KEY `indexTreeID` (`TreeID`),
  KEY `indexQuadratName` (`QuadratName`)
) ENGINE=InnoDB AUTO_INCREMENT=299 DEFAULT CHARSET=latin1;

(291,'0004','28504',NULL,15.4522,7.15468,NULL,NULL,11,349,114,'',NULL)

Alex's Developer Log

8/15

Starting to log my work for the project. Discussed next steps with Omar:

  • For security and other reasons change the upload function by creating an API route that will lead to Azure function on the server, which in turn will do the validation and the upload
  • some old Azure Functions APIs from Prototypes folder were causing issues when I was creating Azure API for our Static web app , I removed them in my ab-functions branch but Omar asked to create a separate issue for that, need to restore the files in my current branch.
  • need to make a drop-down menu to choose a plot where to upload file(s)
  • after that work on getting data from a single plot, as a first step show contents of the first file, then if the plot has several files, combine them, if they have the same rows, take data from the one uploaded later.

8/16

Went through several tutorials on Azure Functions to understand the structure and the logic to apply it when redesigning the upload function. Had issues with connecting local Azure Storage emulator (Azurite) for testing purposes but finally it's working.

8/17

The upload function works via API, need to create API route and add the code with Blob SDK to the function to process files. Restored Prototypes folder, it seems like after the Functions API folder was defined (/API) issues disappeared, so I'll keep the folder for now.

8/22

Okay, I finally figured it out, at least to the point that it works now. The problem was to use the parsed files from multipart component (they have the next fields: name, filename, FileBuffer, encoding, mimeType) to send it to blobUpload, The hard part for me was: to understand how to work with buffer and files in Nodejs; how to process the parsed files, save them in array(not necessary) and preserve the original file name and csv extension. I tried various options, from using NodeJS file system (storing a file and reading its contents), to uploading a file buffer to Azure storage (Blob upload function supports uploading buffers and strings), but finally I found a simpler solution, sending the array with parsed objects to our blobUpload function and rewrote it to process them.
Also I moved sensitive variables like Azure Connection String to .env file, will share it in Slack to test Azure storage.
I created blob containers with names of all plots.
Need to figure out how to use Response from SDK blobClient upload function because now when I test the staged version online, it doesn't have a connection string but it sends 500 Internal Server Error instead of more specific error.
PR is ready, it is not ideal but I'm working on the next steps already (choose plot)

8/29 Update for the whole week

I worked on the component to implement a plot selection for upload. I created a hard-coded array of plots , including the number of a plot that I found on https://forestgeo.si.edu (it can be convenient to use numbers instead of names because Azure storage limits characters in containers' names. For now, I compare chosen plot name with containers names by removing all non-letter characters. I added metadata to Cocoli container in Azure but for some reason when I pull the metadata is shows as undefined.

makeErrorBlank validation

See "instructions for writing a validation function". Below are notes from the spec, the original validation function (in the PHP language), and the SQL table structure.

makeErrorBlank Sets Errors as empty cell when errors are null or none. Local Not a validation. When the errors are none, the error column is empty.
    public function makeErrorBlank($fileName)
    {
      $query = "UPDATE ".$fileName." SET Errors='' WHERE Errors IS NULL || Errors<>'NONE'";
      $runQuery = $this->screeningdb->query($query);
      if($runQuery == TRUE)
      {
        $status = "Success";
      }
      else
      {
        $status = "Fail";
      }
      return $status;
    }

TODO:

  • table structure
  • structure of validation stored procedure
  • place in the code where it should go
  • way to test it (where/how to write test case)

Add ValidationTable to Validate page

We should add the ValidationTable component to the Validate page where we load the data. So that the user can see their data after they upload their CSV files. From there, it will show errors, if any.

startCensusDate validation

startCensusDate | Screens for dates entered, should not be earlier than the user defined start date. Sets the class variables  PlotID, PlotCensusNumber and returns StartDate | Local | Checks to make sure dates entered are not earlier than the start date of the census, which is defined by the user when setting up a new census.

Add error report page

As a data entry person, after using the Dropzone to upload a CSV, I want to be automatically directed to an error report for my data with the option to print or download, so that I can easily identify corrections that I need to make (either at my desk or tomorrow in the field) before re-uploading a revised file.

Improve repo documentation

  • Update "About" paragraph
  • Add README.md in the root folder
  • Add public spec to Wiki
    • Convert tables to Markdown
    • Add supporting images
  • Move ElectronApp to Prototypes folder

TypeScript configuration is set for es5 JS standard

Is there any particular reason to have target parameter set to es5? It doesn't support arrow functions so TS compiler will translate them into traditional ones, etc. ES6 revision is supported by all modern browsers for the last 5-6 years. I understand that some remote sites can have very old PCs with IE11 or something, is this a concern, @esscott?

Update the app's title

This is a good first issue to get familiar with the project.

Right now the app's title is "React App". We should change this to "Forest Geo" or something similar.

It looks like you can't just add a <title> tag in React, and we may need to use something like this: https://github.com/nfl/react-helmet

Add user login component

As a principal investigator for a research site, I want each member of my team to be required to login to the ForestGEO App with a unique user identity, so that multiple people can use app at the same time, and their user data is automatically attached to each submission of new census data.

Add data report page

As a scientist, I want to view and/or download data reports from past censuses and my partially-complete, current census, so that I can run analyses or distribute data for research.

  • Data reports can be filtered for only certain taxa (e.g. "Faramea" species), only certain areas of the plot (e.g. quadrat 0000-0010), and/or only certain trees (e.g. all trees with DBH ≥ 10 cm) to match current capabilities of CTFSWeb

Add in-line error component

As a data entry person, I want to see any errors associated with a specific data point displayed directly in-line with an error code and description of what's wrong, so that I can easily identify what needs correcting and whether it's a typo that I can fix at my desk or something that needs to be checked in the field.

Use Storybook #36 to create and test in-line error component without proper backend.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.