kleok / floodpy Goto Github PK

Include only the existing land cover categories in legend

• Modify Lancover func to return only the existing land cover categories
• Modify the create legend func to get as inputs the Lancover func outputs

Hello again! I'm running Floodpy for a new AOI however this error is showing:

I have increased the number of before_flood_days to 100 and still i get this error. I didn't get this in an earlier Floodpy version

I'm running FLOODPYapp.py FLOODPYapp_template.cfg all at once because doing with --dostep i would get an error when pre processing Sentinel 1 data.

Any way to solve this? Thanks!

Based on issue (#36) we have to support installation and running on windows machines.

The pre-flood water extent will be extracted by Sentinel-1 imagery and Sentinel-2 images before the flood.

• For Sentinel-1 acquisitions (@kleok )

• We select only recent (last month) Sentinel-1 acquisitions that are not related to "big" precipitations events.
• We calculate the median backscatter coefficient over time.
• We download a third-party dataset for water extent (e.g. JRC)
• We create a buffer zone of the water extent
• We calculate threshold (e.g. Otsu) and apply at median backscatter coefficient

• For Sentinel-2 acquisitions (@alekfal)

• We select only recent (last month) Sentinel-2 acquisitions with low cloud coverage (< 10%)
• We calculated MNDWI and apply cloud masks
• We calculate the median MNDWI values over time.
• We calculate a suitable MNDWI threshold at the buffer zone. We can have a look here.
• We apply thresholding at median MNDWI values.

• Combination of Sentinel-1 and Sentinel-2 water masks (@kleok, @alekfal)

• The final map is calculated as the union of Sentinel-1 and Sentinel-2 water masks

FLOMPY runs well untill it gets to STEP 4 (i.e. 'Statistical_analysis'). I get the following error, which seems to do with converting NaN values to integers from line 109 (i.e. utm_zone = int(np.floor((representative_longitude + 180) / 6) + 1)) of the file Generate_aux.py. Please see the error below, and the attached image of the config file.

******************** step - Statistical_analysis ********************
/home/lesetl/anaconda3/envs/flompy/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/home/lesetl/anaconda3/envs/flompy/lib/python3.10/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
File "/home/lesetl/testbed/FLOMPY/flompy/FLOMPYapp.py", line 386, in <module>
main()
File "/home/lesetl/testbed/FLOMPY/flompy/FLOMPYapp.py", line 377, in main
app.run(steps=inps.runSteps, plot=inps.plot)
File "/home/lesetl/testbed/FLOMPY/flompy/FLOMPYapp.py", line 348, in run
self.run_multitemporal_statistics(sname)
File "/home/lesetl/testbed/FLOMPY/flompy/FLOMPYapp.py", line 311, in run_multitemporal_statistics
get_S1_aux (self.Preprocessing_dir)
File "/home/lesetl/testbed/FLOMPY/flompy/Generate_aux.py", line 146, in get_S1_aux
UTM_CRS_EPSG = WGS84_to_UTM(lon_list, lat_list)
File "/home/lesetl/testbed/FLOMPY/flompy/Generate_aux.py", line 109, in WGS84_to_UTM
utm_zone = int(np.floor((representative_longitude + 180) / 6) + 1)
ValueError: cannot convert float NaN to integer
(flompy) -bash-4.2$ python FLOMPYapp.py FLOMPYapp_template.cfg
FLOod Mapping PYthon toolbox (FLOMPY) v.1.0
Copyright (c) 2021-2022 Kleanthis Karamvasis, [email protected]
Remote Sensing Laboratory of National Technical University of Athens

I've modified the configuration file to select my desired date and lat-long:

# B1. The datetime of flood event (Format is YYYYMMDDTHHMMSS)
Flood_datetime = 20170910T043000

# C2. AOI BBOX (WGS84)
LONMIN=81.29
LATMIN=24.61
LONMAX=81.46
LATMAX=24.74

After running the command FLOODPYapp.py FLOODPYapp_Pine.cfg --end Floodwater_classification, I get the following error. The previous steps seem to work fine. Any advice would be greatly appreciated! Note: I am using WSL (Ubuntu 22.04.1 LTS) and the default cfg file works without any problems.

******************** step - Floodwater_classification ********************
Calculating Bi/multi modality mask...
Calculating global flood mask using global threshold...
Traceback (most recent call last):
  File "/mnt/c/Users/acardaras/Documents/FLOODPY-main/floodpy/FLOODPYapp.py", line 441, in <module>
    main()
  File "/mnt/c/Users/acardaras/Documents/FLOODPY-main/floodpy/FLOODPYapp.py", line 432, in main
    app.run(steps=inps.runSteps, plot=inps.plot)
  File "/mnt/c/Users/acardaras/Documents/FLOODPY-main/floodpy/FLOODPYapp.py", line 400, in run
    self.run_get_flood_map(sname)
  File "/mnt/c/Users/acardaras/Documents/FLOODPY-main/floodpy/FLOODPYapp.py", line 351, in run_get_flood_map
    self.Flood_local_map_RG_morph,] = Calc_flood_map(Preprocessing_dir = self.Preprocessing_dir,
  File "/home/acardaras/.local/lib/python3.10/site-packages/floodpy/Floodwater_classification/Classification.py", line 97, in Calc_flood_map
    glob_thresh = threshold_Otsu(t_scores_flatten)
  File "/home/acardaras/.local/lib/python3.10/site-packages/floodpy/Floodwater_classification/Thresholding_methods.py", line 51, in threshold_Otsu
    return threshold_otsu(data.flatten())
  File "/home/acardaras/.local/lib/python3.10/site-packages/skimage/filters/thresholding.py", line 364, in threshold_otsu
    counts, bin_centers = _validate_image_histogram(image, hist, nbins)
  File "/home/acardaras/.local/lib/python3.10/site-packages/skimage/filters/thresholding.py", line 306, in _validate_image_histogram
    counts, bin_centers = histogram(
  File "/home/acardaras/.local/lib/python3.10/site-packages/skimage/_shared/utils.py", line 394, in fixed_func
    return func(*args, **kwargs)
  File "/home/acardaras/.local/lib/python3.10/site-packages/skimage/exposure/exposure.py", line 266, in histogram
    hist, bin_centers = _histogram(image, nbins, source_range, normalize)
  File "/home/acardaras/.local/lib/python3.10/site-packages/skimage/exposure/exposure.py", line 300, in _histogram
    hist, bin_edges = np.histogram(image, bins=bins, range=hist_range)
  File "<__array_function__ internals>", line 5, in histogram
  File "/usr/lib/python3/dist-packages/numpy/lib/histograms.py", line 793, in histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File "/usr/lib/python3/dist-packages/numpy/lib/histograms.py", line 426, in _get_bin_edges
    first_edge, last_edge = _get_outer_edges(a, range)
  File "/usr/lib/python3/dist-packages/numpy/lib/histograms.py", line 323, in _get_outer_edges
    raise ValueError(
ValueError: autodetected range of [-inf, inf] is not finite

Hello! I download the recent version of the repo and I started doing a new project to test a flood event that occured in Portugal.

However, when running the Run_preprocessing_S1_data it gives me a "NoneType" error as following:

TypeError Traceback (most recent call last)
Input In [13], in <cell line: 1>()
----> 1 app.run_preprocessing_S1_data('Preprocessing_S1_data')

File ~/FLOODPY/floodpy/FLOODPYapp.py:317, in FloodwaterEstimation.run_preprocessing_S1_data(self, step_name)
315 def run_preprocessing_S1_data(self, step_name):
--> 317 Get_images_for_baseline_stack(projectfolder = self.projectfolder,
318 S1_dir = self.S1_dir,
319 Precipitation_data = self.precipitation_df,
320 flood_datetime = self.flood_datetime,
321 days_back = self.days_back,
322 rain_thres=self.rain_thres)
324 Run_Preprocessing(projectfolder = self.projectfolder,
325 gpt_exe = self.gptcommand,
326 graph_dir = self.graph_dir,
327 S1_dir = self.S1_dir,
328 geojson_S1 = self.geojson_S1,
329 Preprocessing_dir = self.Preprocessing_dir)
331 return 0

File ~/FLOODPY/floodpy/Preprocessing_S1_data/Classify_S1_images.py:22, in Get_images_for_baseline_stack(projectfolder, S1_dir, Precipitation_data, flood_datetime, days_back, rain_thres)
10 def Get_images_for_baseline_stack(projectfolder,
11 S1_dir,
12 Precipitation_data,
13 flood_datetime,
14 days_back=5,
15 rain_thres=20):
16 '''
17 Creates a pandas DataFrame of Sentinel-1 acquisitions. Creates a column with
18 boolean values with name 'baseline'. If True the particular acquisition
19 can be used for calculation of baseline stack
20
21 '''
---> 22 Precipitation_data.index = Precipitation_data['Datetime']
23 Precipitation_data.drop(columns = ['Datetime'], inplace=True)
25 # calculate cumulative rain over the past days (given) for each date

TypeError: 'NoneType' object is not subscriptable

image:

previous step:

Hello, I am a third year bachelor's student in Geographic Information System and I am using FLOODPY to do a flood mapping study in my area. I have encountered the following error several times. Please help me on how I can resolve this. Also I don't understand the days_back part. Would you be so kind as to give me a short explanation because in the documentation its not so clear. Below is the error message. Thank you.

I have made the following modifications to my .cfg file and have noticed that the Sentinel_1_imagery directory
only contains data prior(Aug 18 - Sept 15) to my specified Flood_datetime(Sept 16). Is this the expected behavior and is there any way to ensure imagery is downloaded on or after the specified Flood_datetime?

# B1. The datetime of flood event (Format is YYYYMMDDTHHMMSS)
Flood_datetime = 20200916T050000

# B2. Days before flood event for baseline stack construction
before_flood_days = 40

# B3. Days after flood event
after_flood_days = 3

# C2. AOI BBOX (WGS84)
LONMIN=-87.33
LATMIN=30.34
LONMAX=-87.26
LATMAX=30.40

Hi Team,
I am running the FLOODPY for event happened in Solingen, Germany but getting the below error in Preprocessing_S1_data.py.

File "C:\Users\Pmathur\Anaconda3\envs\floodpy\lib\site-packages\floodpy\Preprocessing_S1_data\Preprocessing_S1_data.py", line 528, in Run_Preprocessing
Subset_AOI,

UnboundLocalError: local variable 'Subset_AOI' referenced before assignment

It might be happening because two adjacent GRD tile are getting used in pre-event tile but flood event has only one GRD tile as 2 and 3 tile in table below:
| ERA5_tp_mm | baseline | S1_GRD | Datetime
1 | 23.0889534 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1A_IW_GRDH_1SDV_20210609T055025_20210609T055050_038259_0483CC_2C8D.zip | 6/9/2021 5:50
2 | 23.73645536 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1B_IW_GRDH_1SDV_20210615T054935_20210615T055000_027363_0344A0_DAC1.zip | 6/15/2021 5:49
3 | 23.73645536 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1B_IW_GRDH_1SDV_20210615T055000_20210615T055025_027363_0344A0_13F8.zip | 6/15/2021 5:50
4 | 22.57493026 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1A_IW_GRDH_1SDV_20210621T055025_20210621T055050_038434_048908_F67D.zip | 6/21/2021 5:50
5 | 37.8658546 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1B_IW_GRDH_1SDV_20210627T054935_20210627T055000_027538_03498A_B833.zip | 6/27/2021 5:49
6 | 37.8658546 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1B_IW_GRDH_1SDV_20210627T055000_20210627T055025_027538_03498A_86D9.zip | 6/27/2021 5:50
7 | 47.67774148 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1A_IW_GRDH_1SDV_20210703T055026_20210703T055051_038609_048E45_DD91.zip | 7/3/2021 5:50
8 | 71.55205072 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1B_IW_GRDH_1SDV_20210709T054936_20210709T055001_027713_034EA7_4B41.zip | 7/9/2021 5:49
9 | 71.60350636 | TRUE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1B_IW_GRDH_1SDV_20210709T055001_20210709T055026_027713_034EA7_EA87.zip | 7/9/2021 5:50
10 | 164.908054 | FALSE | E:\Vishal_2022\FloodEventResponse\SOLINGEN\Sentinel_1_imagery\S1A_IW_GRDH_1SDV_20210715T055027_20210715T055052_038784_049389_5004.zip | 7/15/2021 5:50

PARAMETERS USED:
#A1. The name of your project withough special characters.
Projectname = 'Solingen'

#A2. The location that everything is going to be saved. Make sure
projectfolder = r'E:\Vishal_2022\FloodEventResponse\SOLINGEN'

#A3. The location of floodpy code
src_dir = r'C:\Users\Pmathur\FLOODPY-main\floodpy'

#A4. SNAP ORBIT DIRECTORY
snap_dir = r'C:\Users\Pmathur.snap\auxdata\Orbits\Sentinel-1'

#A5. SNAP GPT full path
GPTBIN_PATH = r"C:\Program Files\snap\bin\gpt.exe"
Flood_datetime = '20210714T000000'#20200921T030000'

before_flood_days = 40

after_flood_days = 3

AOI_File = "None”
LONMIN=6.5
LATMIN=51.1
LONMAX=7.2
LATMAX=51.3

days_back = 12

accumulated_precipitation_threshold = 120

#E1. The number of Sentinel-1 relative orbit. The default
S1_type = 'GRD'
relOrbit = 'Auto'

#E3. The minimum mapping unit area in square meters
minimum_mapping_unit_area_m2=4000

#E4. Computing resources to employ
CPU=8
RAM='20G'

Please have a look and let me know how we can run it for the above parameters.
Thanks
Vishal

• add Layer control in folium
• add more basemaps
• add measure control
• improve colormaps
• Add Sentinel-2 (RGB 432 band combination) as background image
• Add landcover map with legend

Good day,

I tried changing the relOrbit value to 50 and 116 as you suggested to my colleague in issue #1. Both resulted in a ValueError for the date, albeit different times. For relOrbit = 50 this error was "20211213T033455" and for relOrbit = 116 it was "20220128T164719".

Thanks in advance!
Natasha

Multiple flood map functionality

• should enable the user to select a particular time range
• Identify the time and dates that Floodpy can provide flood maps
• Download all data that should be processed in order to generate the flood maps
• Generate the different flood maps

TypeError: Download_S1_data() got an unexpected keyword argument 'S1_GRD_dir'

Hi,

we are getting the following error (AttibuteError: 'str' object has no attribute 'reset_index') in the 'Preporcessing_S1_data step'. It seems that 'S1_df' is empty.

Can you assist?

Thank you

The error occurs during the pre-processing step, the project template file and error image are attached

FLOODPYapp_template.txt

• Releasing a legacy LPS2022 version. ✔️
• Remove all the LPS2022 functionalities including:
  1. Crop_delineation sub-package and methods. ✔️
  2. Crop-delineation_unet sub-package and methods. ✔️
• Refactoring Sentinel 2 preprocessing procedures for future flood mapping applications using Sentinel 2. ✔️ (#44)

Download_Sentinel-2 data (@alekfal)
Preprocessing Sentinel-2 data (@alekfal )
Crop delineation (@Olyna )

I have been trying to figure out the error I keep getting on the final step of Floodwater_classification which has to do with non-empty take from an empty axes. I've looked at the numpy function relating to this error, and still can't figure out where it's coming from within the Flompy dataset.

Please find attached the error I keep getting. Any suggestions on how to go around this will be helpful.

I occasionally encounter an error related to the gpt subprocess on different AOI's. I've attached an image of the error message and provided an example .cfg file modification that causes this error on my machine.

# B1. The datetime of flood event (Format is YYYYMMDDTHHMMSS)
Flood_datetime = 20170910T043000

# B2. Days before flood event for baseline stack construction
before_flood_days = 60

# B3. Days after flood event
after_flood_days = 10

# C2. AOI BBOX (WGS84)
LONMIN=-81.46
LATMIN=24.61
LONMAX=-81.29
LATMAX=24.74

GFS data can be used in case of non-availability of ERA5 data. This is relevant for the analysis of ongoing flood events.

Hello, please advise how to solve this error

Good day Kleanthis,

You have been of invaluable assistance to myself and my colleague Nicolene Botha. After your last update we were able to obtain the flood map image for 5 February 2022. We are still going to look at the relOrbit variable as you suggested.

However, with the changes you made I can no longer get the flood map image for 9 December 2019, which I was able to get prior to the latest update. This is also the case for several other dates. It gives an assertion error and it appears that the file indicated in the flood_S1_filename.csv as the main image is not downloaded, hence the error.

I attach the error message, map.geojson and configuration file. In the configuration file you will see all the other dates of interest that are all giving the same assertion error.
FLOMPYapp.zip

Thanks again for your prompt assistance with every issue we have encountered. It is great when a developer is still actively maintaining their code.

Kind regards
Natasha

Hello!

I found this toolkit on linkedin and I find it super useful,

I'm trying to follow the instructions to do all the installation steps however I'm a windows user. As such, I'm using WSL to bypass that issue. However, when installing snap.sh, the installation never stops as shown in the following image, do you have any idea of how long it takes to install it? it is been like 2 hours cycling through processes

If i have snap on windows do I have to install it again through Linux?

Thank you!

EDIT1: I'm trying to do all the steps and when I reach here I can't download the precipitation data for my AOI because it says command is not found without the python3, and module not found with python3. Any help?