luc-girod / mmaster-workflows Goto Github PK
View Code? Open in Web Editor NEWWorkflows for MMASTER
License: GNU General Public License v3.0
Workflows for MMASTER
License: GNU General Public License v3.0
when I run mmaster_bias_correction.py -h in the terminal, some bug occurs: ModuleNotFoundError: No module named '_gdal', my operation system is mac os. and I can solve it successfully.
python mmaster_bias_correction.py 1.tif AST* -a exc_mask -b inc_mask
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:35: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:597: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:836: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will
not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:862: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, positive=False):
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1097: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/sit
e-packages/sklearn/linear_model/least_angle.py:1344: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the
numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1480: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, positive=False):
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:152: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
precompute=False, eps=np.finfo(np.float).eps,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:320: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, random_state=None,
/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:580: DeprecationWarning: np.float
is a deprecated alias for the builtin float
. To silence this warning, use float
by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64
here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=4 * np.finfo(np.float).eps, n_jobs=None,
Number of image directories given: 1
Running bias correction on AST_L1A_00301232009055321
Writing DEMs cropped to common extent.
No date information read from filename.
Running iteration #1
2.7561726618515245 16.37609628579061 44.701101705782136
Percent-improvement threshold: 91.57105159759521
Magnitude threshold: 47.68607250564992
Running iteration #2
2.564140020613836 15.694076422192945 44.832883098010704
Percent-improvement threshold: 10.008838710838045
Magnitude threshold: 0.7206899229073495
FinalCHECK
Fin.
Original Sample size : 47566
Final Sample size : 47114
Remaining NaNs : 0
Initial Parameters: [-1.00657315 0.02204092]
Polynomial degree - 1 --> Status: True - 2
ftol
termination condition is satisfied.
Lowest cost: 22544.329573323856
Parameters: [0.9119739 0.0021708]
Initial Parameters: [-0.03195836 -0.10664196 0.00229296]
Polynomial degree - 2 --> Status: True - 2
ftol
termination condition is satisfied.
Lowest cost: 22489.289848837107
Parameters: [ 2.26389303 -0.16972806 0.00301988]
Initial Parameters: [-1.57215566e+00 1.93049367e-01 -8.53039720e-03 1.08045059e-04]
Polynomial degree - 3 --> Status: True - 2
ftol
termination condition is satisfied.
Lowest cost: 22488.474920409553
Parameters: [ 1.85139708e+00 -9.44581716e-02 3.47744646e-04 2.64044930e-05]
Initial Parameters: [ 1.58564291e+00 -7.45298674e-01 6.68953345e-02 -1.90718142e-03
1.69911566e-05]
Polynomial degree - 4 --> Status: True - 2
ftol
termination condition is satisfied.
Lowest cost: 22397.614422893068
Parameters: [ 5.50680182e+00 -1.12603031e+00 8.30456469e-02 -2.18179634e-03
1.85875041e-05]
Initial Parameters: [ 3.41884298e+00 -1.55952562e+00 1.66980230e-01 -6.22830084e-03
9.39475798e-05 -4.86480541e-07]
Polynomial degree - 5 --> Status: True - 2
ftol
termination condition is satisfied.
Lowest cost: 22396.82362940726
Parameters: [ 4.87993974e+00 -8.40161367e-01 4.71737561e-02 -6.26950287e-04
-9.06681488e-06 1.74261333e-07]
Initial Parameters: [ 6.70042317e+00 -3.28732991e+00 4.36500123e-01 -2.33058372e-02
6.02771096e-04 -7.63408150e-06 3.82387064e-08]
Polynomial degree - 6 --> Status: True - 2
ftol
termination condition is satisfied.
Lowest cost: 22386.839562209367
Parameters: [ 7.08452931e+00 -2.07157592e+00 2.45616429e-01 -1.35261841e-02
3.82078632e-04 -5.38437570e-06 2.99597854e-08]
Polynomial Order Selected: 6
Cross track robust Polynomial RMSE (all data): 9.236576240992907
Traceback (most recent call last):
File "mmaster_bias_correction.py", line 7, in
exec(compile(f.read(), file, 'exec'))
File "/home/rs/MMASTER-workflows/bin/mmaster_bias_correction.py", line 110, in
main()
File "/home/rs/MMASTER-workflows/bin/mmaster_bias_correction.py", line 96, in main
write_log=args.log)
File "/home/rs/test/MMASTER-workflows/pymmaster/mmaster_tools.py", line 1464, in mmaster_bias_removal
pp, pts=pts, robust=robust)
File "/home/rs/test/MMASTER-workflows/pymmaster/mmaster_tools.py", line 1037, in correct_along_track_bias
pts)
File "/home/rs/test/MMASTER-workflows/pymmaster/mmaster_tools.py", line 1004, in get_filtered_along_track
grp_dH = np.delete(grp_dH, [psize < pthresh])
File "<array_function internals>", line 6, in delete
File "/home/rs/anaconda3/envs/mmaster_environment/lib/python3.7/site-packages/numpy/lib/function_base.py", line 4546, in delete
'length of {}'.format(N))
ValueError: boolean array argument obj to delete must be one dimensional and match the axis length of 40
In WorkFlowASTER_onescene.sh file at line no 92.
Please put space .
Change the code as :
eg:
if [ ! -d "RawData"]; then
to
if [ ! -d "RawData" ]; then
when i run the WorkFlowASTER.sh, and the code reach line: mm3d Satelib ASTERStrip2MM AST_L1A.* $name, the error ocurs as:
KIND OF ERR: User's error
Sorry, the following FATAL ERROR happended
can't open files:
[../data/aster_l1a_wkl/sorted/AST_L1A_00303142020052844_3N.tif](in mode wb)
Bye (press enter)
I encountered this error when running WorkFlowASTER.sh
I've checked that I'm on the IncludeALGLIB branch
For command = ApplyParralaxCor
------------------------------------------------------------
| Sorry, the following FATAL ERROR happened
|
| Unknown command in mm3d
|
------------------------------------------------------------
-------------------------------------------------------------
| (Elise's) LOCATION :
|
| Error was detected
| at line : 1697
| of file : /home/wqh/Documents/micmac/src/CBinaires/mm3d.cpp
-------------------------------------------------------------
Bye (press enter)
Any idea? Thanks in advance!
MMASTER-workflows/WorkFlowASTER.sh
Line 73 in f968860
Hi so I'm having a problem with creating the conda environment supplied.
I create the environment from the .yml file and then type:
pip install -e MMASTER-workflows
I then get a conflict in the environment. I think this is because the .yml file specifies pybob=0.24 and the requirements for the pymmaster module is pybob>0.25. The .yml file is very constrained meaning that manual installation of pybob=0.26 then requires updating all the other dependencies.
I tried creating a seperate environment with just python=3.7 in it and then using the pip line above but that encountered issues with GDAL. Can you help?
Thanks,
Felix
I tried to run WorkFlowASTER_onescene.sh and WorkFlowASTER_onestrip.sh .
Both are throwing following error :
Error :
KIND OF ERR : User's error | Sorry, the following FATAL ERROR happened | | LArgMain , Don't understand :[ExportFitASTER=1]
And after this :
------------------------------------------------------------ | Sorry, the following FATAL ERROR happened | | GetSec::Sec | ------------------------------------------------------------
------------------------------------------------------------- | (Elise's) LOCATION : | | Error was detected | at line : 93 | of file : /home/kshakya/micmacdir/micmac/include/../src/uti_phgrm/MICMAC/uti_MICMAC.h -------------------------------------------------------------
And,
------------------------------------------------------------ | Sorry, the following FATAL ERROR happened | | FileLogMM3d: cannot open file for writing in [Ortho-MEC-Malt/../] | ------------------------------------------------------------
------------------------------------------------------------- | (Elise's) LOCATION : | | Error was detected | at line : 71 | of file : /home/kshakya/micmacdir/micmac/src/CBinaires/mm3d.cpp -------------------------------------------------------------
and then,
------------------------------------------------------------ | Sorry, the following FATAL ERROR happened | | Unknown command in mm3d | ------------------------------------------------------------
------------------------------------------------------------- | (Elise's) LOCATION : | | Error was detected | at line : 1411 | of file : /home/kshakya/micmacdir/micmac/src/CBinaires/mm3d.cpp -------------------------------------------------------------
and
cp: cannot stat 'MEC-MaltZ_Num9_DeZoom1_STD-MALT.tfw': No such file or directory mv: cannot stat 'TrackAngleMap.tif': No such file or directory ERROR 4: TrackAngleMap_nonGT.tif: No such file or directory rm: cannot remove 'TrackAngleMap_nonGT*': No such file or directory /home/kshakya/micmacdir/mmaster/mmasterworkflow/WorkFlowASTER_onestrip.sh: line 209: cd: Ortho-MEC-Malt: No such file or directory cp: cannot stat 'Orthophotomosaic.tif': No such file or directory mv: cannot stat 'Orthophotomosaic.tfw': No such file or directory ERROR 4: Orthophotomosaic_FullRes.tif: No such file or directory
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.