luc-girod / mmaster-workflows Goto Github PK

when I run mmaster_bias_correction.py -h in the terminal, some bug occurs: ModuleNotFoundError: No module named '_gdal', my operation system is mac os. and I can solve it successfully.

python mmaster_bias_correction.py 1.tif AST* -a exc_mask -b inc_mask
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:35: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:597: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:836: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will
not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:862: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, positive=False):
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1097: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/sit
e-packages/sklearn/linear_model/least_angle.py:1344: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the
numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1480: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, positive=False):
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:152: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
precompute=False, eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:320: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, random_state=None,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:580: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=4 * np.finfo(np.float).eps, n_jobs=None,
Number of image directories given: 1
Running bias correction on AST_L1A_00301232009055321
Writing DEMs cropped to common extent.
No date information read from filename.
Running iteration #1
2.7561726618515245 16.37609628579061 44.701101705782136
Percent-improvement threshold: 91.57105159759521
Magnitude threshold: 47.68607250564992
Running iteration #2
2.564140020613836 15.694076422192945 44.832883098010704
Percent-improvement threshold: 10.008838710838045
Magnitude threshold: 0.7206899229073495
FinalCHECK
Fin.
Original Sample size : 47566
Final Sample size : 47114
Remaining NaNs : 0
Initial Parameters: [-1.00657315 0.02204092]
Polynomial degree - 1 --> Status: True - 2
ftol termination condition is satisfied.
Lowest cost: 22544.329573323856
Parameters: [0.9119739 0.0021708]
Initial Parameters: [-0.03195836 -0.10664196 0.00229296]
Polynomial degree - 2 --> Status: True - 2
ftol termination condition is satisfied.
Lowest cost: 22489.289848837107
Parameters: [ 2.26389303 -0.16972806 0.00301988]
Initial Parameters: [-1.57215566e+00 1.93049367e-01 -8.53039720e-03 1.08045059e-04]
Polynomial degree - 3 --> Status: True - 2
ftol termination condition is satisfied.
Lowest cost: 22488.474920409553
Parameters: [ 1.85139708e+00 -9.44581716e-02 3.47744646e-04 2.64044930e-05]
Initial Parameters: [ 1.58564291e+00 -7.45298674e-01 6.68953345e-02 -1.90718142e-03
1.69911566e-05]
Polynomial degree - 4 --> Status: True - 2
ftol termination condition is satisfied.
Lowest cost: 22397.614422893068
Parameters: [ 5.50680182e+00 -1.12603031e+00 8.30456469e-02 -2.18179634e-03
1.85875041e-05]
Initial Parameters: [ 3.41884298e+00 -1.55952562e+00 1.66980230e-01 -6.22830084e-03
9.39475798e-05 -4.86480541e-07]
Polynomial degree - 5 --> Status: True - 2
ftol termination condition is satisfied.
Lowest cost: 22396.82362940726
Parameters: [ 4.87993974e+00 -8.40161367e-01 4.71737561e-02 -6.26950287e-04
-9.06681488e-06 1.74261333e-07]
Initial Parameters: [ 6.70042317e+00 -3.28732991e+00 4.36500123e-01 -2.33058372e-02
6.02771096e-04 -7.63408150e-06 3.82387064e-08]
Polynomial degree - 6 --> Status: True - 2
ftol termination condition is satisfied.
Lowest cost: 22386.839562209367
Parameters: [ 7.08452931e+00 -2.07157592e+00 2.45616429e-01 -1.35261841e-02
3.82078632e-04 -5.38437570e-06 2.99597854e-08]
Polynomial Order Selected: 6
Cross track robust Polynomial RMSE (all data): 9.236576240992907
Traceback (most recent call last):
File "mmaster_bias_correction.py", line 7, in
exec(compile(f.read(), file, 'exec'))
File "/home/rs/MMASTER-workflows/bin/mmaster_bias_correction.py", line 110, in
main()
File "/home/rs/MMASTER-workflows/bin/mmaster_bias_correction.py", line 96, in main
write_log=args.log)
File "/home/rs/test/MMASTER-workflows/pymmaster/mmaster_tools.py", line 1464, in mmaster_bias_removal
pp, pts=pts, robust=robust)
File "/home/rs/test/MMASTER-workflows/pymmaster/mmaster_tools.py", line 1037, in correct_along_track_bias
pts)
File "/home/rs/test/MMASTER-workflows/pymmaster/mmaster_tools.py", line 1004, in get_filtered_along_track
grp_dH = np.delete(grp_dH, [psize < pthresh])
File "<array_function internals>", line 6, in delete
File "/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/numpy/lib/function_base.py", line 4546, in delete
'length of {}'.format(N))
ValueError: boolean array argument obj to delete must be one dimensional and match the axis length of 40

In WorkFlowASTER_onescene.sh file at line no 92.
Please put space .
Change the code as :
eg:

if [ ! -d "RawData"]; then

to

if [ ! -d "RawData" ]; then

when i run the WorkFlowASTER.sh, and the code reach line: mm3d Satelib ASTERStrip2MM AST_L1A.* $name, the error ocurs as:
KIND OF ERR: User's error
Sorry, the following FATAL ERROR happended
can't open files:
[../data/aster_l1a_wkl/sorted/AST_L1A_00303142020052844_3N.tif](in mode wb)
Bye (press enter)

I encountered this error when running WorkFlowASTER.sh
I've checked that I'm on the IncludeALGLIB branch

For command = ApplyParralaxCor
------------------------------------------------------------
|   Sorry, the following FATAL ERROR happened               
|                                                           
|    Unknown command in mm3d
|                                                           
------------------------------------------------------------
-------------------------------------------------------------
|       (Elise's)  LOCATION :                                
|                                                            
| Error was detected
|          at line : 1697
|          of file : /home/wqh/Documents/micmac/src/CBinaires/mm3d.cpp
-------------------------------------------------------------
Bye  (press enter)

Any idea? Thanks in advance!

Hi so I'm having a problem with creating the conda environment supplied.
I create the environment from the .yml file and then type:
pip install -e MMASTER-workflows
I then get a conflict in the environment. I think this is because the .yml file specifies pybob=0.24 and the requirements for the pymmaster module is pybob>0.25. The .yml file is very constrained meaning that manual installation of pybob=0.26 then requires updating all the other dependencies.
I tried creating a seperate environment with just python=3.7 in it and then using the pip line above but that encountered issues with GDAL. Can you help?

Thanks,

Felix

I tried to run WorkFlowASTER_onescene.sh and WorkFlowASTER_onestrip.sh .
Both are throwing following error :

Error :
KIND OF ERR : User's error | Sorry, the following FATAL ERROR happened | | LArgMain , Don't understand :[ExportFitASTER=1]

And after this :
------------------------------------------------------------ | Sorry, the following FATAL ERROR happened | | GetSec::Sec | ------------------------------------------------------------
------------------------------------------------------------- | (Elise's) LOCATION : | | Error was detected | at line : 93 | of file : /home/kshakya/micmacdir/micmac/include/../src/uti_phgrm/MICMAC/uti_MICMAC.h -------------------------------------------------------------
And,

------------------------------------------------------------ | Sorry, the following FATAL ERROR happened | | FileLogMM3d: cannot open file for writing in [Ortho-MEC-Malt/../] | ------------------------------------------------------------
------------------------------------------------------------- | (Elise's) LOCATION : | | Error was detected | at line : 71 | of file : /home/kshakya/micmacdir/micmac/src/CBinaires/mm3d.cpp -------------------------------------------------------------
and then,

------------------------------------------------------------ | Sorry, the following FATAL ERROR happened | | Unknown command in mm3d | ------------------------------------------------------------
------------------------------------------------------------- | (Elise's) LOCATION : | | Error was detected | at line : 1411 | of file : /home/kshakya/micmacdir/micmac/src/CBinaires/mm3d.cpp -------------------------------------------------------------

and
cp: cannot stat 'MEC-MaltZ_Num9_DeZoom1_STD-MALT.tfw': No such file or directory mv: cannot stat 'TrackAngleMap.tif': No such file or directory ERROR 4: TrackAngleMap_nonGT.tif: No such file or directory rm: cannot remove 'TrackAngleMap_nonGT*': No such file or directory /home/kshakya/micmacdir/mmaster/mmasterworkflow/WorkFlowASTER_onestrip.sh: line 209: cd: Ortho-MEC-Malt: No such file or directory cp: cannot stat 'Orthophotomosaic.tif': No such file or directory mv: cannot stat 'Orthophotomosaic.tfw': No such file or directory ERROR 4: Orthophotomosaic_FullRes.tif: No such file or directory