Silent failure of cloud class? Laz support? about pyfor

[brycefrank]

Try this. I have not added lastools to the environment.yml. Install and then re-run.

conda install -c conda-forge lastools -n pyfor_env

Even if this does fix it, .laz is not supported in a sensical way. Unfortunately, laspy uses a very slow procedure to extract the .laz information. It is something I am looking into.

[bw4sz]

Thanks for quick response.

Not quite there.

(pyfor_env) MacBook-Pro:pyfor ben$ conda install -c conda-forge lastools
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.4
  latest version: 4.5.11

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /Users/ben/miniconda3/envs/pyfor_env

  added / updated specs:
    - lastools


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blas-1.0                   |              mkl           5 KB
    numpy-1.15.4               |   py36h6a91979_0          35 KB
    mkl_fft-1.0.6              |           py36_0         159 KB  conda-forge
    scikit-learn-0.20.0        |   py36h4f467ca_1         5.4 MB
    lastools-20171231          |       hfc679d8_0         668 KB  conda-forge
    mkl-2019.0                 |              118       154.4 MB
    intel-openmp-2019.0        |              118         1.0 MB
    mkl_random-1.0.2           |           py36_0         377 KB  conda-forge
    scipy-1.1.0                |   py36h28f7352_1        15.4 MB
    numpy-base-1.15.4          |   py36h8a80b8c_0         4.1 MB
    ------------------------------------------------------------
                                           Total:       181.6 MB

The following NEW packages will be INSTALLED:

    intel-openmp: 2019.0-118
    lastools:     20171231-hfc679d8_0                   conda-forge
    mkl:          2019.0-118
    mkl_fft:      1.0.6-py36_0                          conda-forge
    mkl_random:   1.0.2-py36_0                          conda-forge
    numpy-base:   1.15.4-py36h8a80b8c_0

The following packages will be UPDATED:

    libgfortran:  3.0.0-1                               conda-forge --> 3.0.1-h93005f0_2
    numpy:        1.15.4-py36_blas_openblashb06ca3d_0   conda-forge [blas_openblas] --> 1.15.4-py36h6a91979_0

The following packages will be DOWNGRADED:

    blas:         1.1-openblas                          conda-forge --> 1.0-mkl
    scikit-learn: 0.20.0-py36_blas_openblash00c3548_201 conda-forge [blas_openblas] --> 0.20.0-py36h4f467ca_1
    scipy:        1.1.0-py36_blas_openblashb06ca3d_202  conda-forge [blas_openblas] --> 1.1.0-py36h28f7352_1

Proceed ([y]/n)? y


Downloading and Extracting Packages
blas-1.0             |    5 KB | ################################################################################################################################ | 100%
numpy-1.15.4         |   35 KB | ################################################################################################################################ | 100%
mkl_fft-1.0.6        |  159 KB | ################################################################################################################################ | 100%
scikit-learn-0.20.0  |  5.4 MB | ################################################################################################################################ | 100%
lastools-20171231    |  668 KB | ################################################################################################################################ | 100%
mkl-2019.0           | 154.4 MB | ############################################################################################################################### | 100%
intel-openmp-2019.0  |  1.0 MB | ################################################################################################################################ | 100%
mkl_random-1.0.2     |  377 KB | ################################################################################################################################ | 100%
scipy-1.1.0          | 15.4 MB | ################################################################################################################################ | 100%
numpy-base-1.15.4    |  4.1 MB | ################################################################################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(pyfor_env) MacBook-Pro:pyfor ben$ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfor
>>> pc=pyfor.cloud.Cloud("/Users/ben/Documents/DeepLidar/data/SJER/SJER_002.laz")
>>> print(pc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/Documents/pyfor/pyfor/cloud.py", line 122, in __str__
    min =  [float('{0:.2f}'.format(elem)) for elem in self.data.min]
AttributeError: 'Cloud' object has no attribute 'data'
>>>

[brycefrank]

Alright then! I will dig into this in a few hours. I have a suspicion what needs fixing. I will push the hotfix to master sometime today.

[brycefrank]

Ben,

The issue is sort-of fixed over on the laz_fix branch. This fix allows a file extension to be '.laz' and still go through the importing process. I was able to get the newly added test.laz in the testing suite to load and display points. I have not searched for other situations quite yet that may also break. But I think for the most part you should be good if you do the following:

(in your pyfor directory)

git fetch
git checkout laz_fix
pip install . --upgrade

This is assuming you already had a working installation of laszip.

I am having some trouble getting some new tests to pass on travis, there seems to be some weird issue with the path and conda and laspy. It may all be entertwined. Give the above a shot and let me know if it is still broken.

[brycefrank]

Another update, I switched from using laszip to lastools on conda, this seemed to do the trick for travis. I am going to leave that fix there for a day or two and ensure nothing else breaks.

[bw4sz]

Okay, working through this now. I've got a laszip version on the comp, but its probably not in path. If it s a dependency, maybe add that as a note? I'll check that in a sec.

(pyfor_env) MacBook-Pro:pyfor ben$ git status
On branch laz_fix
Your branch is up-to-date with 'origin/laz_fix'.
nothing to commit, working tree clean
(pyfor_env) MacBook-Pro:pyfor ben$ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfor
/Users/ben/miniconda3/envs/pyfor_env/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning:

the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

>>> pc=pyfor.cloud.Cloud("/Users/ben/Documents/DeepLidar/data/SJER/SJER_002.laz")
>>> print(pc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/Documents/pyfor/pyfor/cloud.py", line 126, in __str__
    return(out)
UnboundLocalError: local variable 'out' referenced before assignment

[bw4sz]

I take it back, laszip is in path. I wasn't sure how conda was going to play with brew install. Looks fine to me.

(pyfor_env) MacBook-Pro:pyfor ben$ which laszip
/Users/ben/miniconda3/envs/pyfor_env/bin/laszip

[bw4sz]

Just to confirm its not that file.

(pyfor_env) MacBook-Pro:pyfor ben$ ls
CHANGELOG.md		MANIFEST		docs			pyfor			readthedocs.yml		requirements.txt	setup.cfg
License.txt		README.md		environment.yml		pyfortest		requirements.rtd.txt	samples			setup.py
(pyfor_env) MacBook-Pro:pyfor ben$ cd pyfortest/
(pyfor_env) MacBook-Pro:pyfortest ben$ ls
__init__.py	data		performance.py	test_pyfor.py
(pyfor_env) MacBook-Pro:pyfortest ben$ cd data/
(pyfor_env) MacBook-Pro:data ben$ ls
chm.tif		clip.dbf	clip.prj	clip.qpj	clip.shp	clip.shx	test.las	test.laz
(pyfor_env) MacBook-Pro:data ben$ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfor
/Users/ben/miniconda3/envs/pyfor_env/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning:

the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

>>> pc=pyfor.cloud.Cloud("test.laz")
>>> print(pc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/miniconda3/envs/pyfor_env/lib/python3.6/site-packages/pyfor/cloud.py", line 126, in __str__
    return(out)
UnboundLocalError: local variable 'out' referenced before assignment

[brycefrank]

This is turning out to be a tricky one, Ben. I see you have had some trouble with laspy / laszip as well. I wonder if it may be related.

There are a couple of details worth noting, just for your information and for me to see later:

1. When I run unit tests in PyCharm I run into your original error. The test fails on

   pyfor/pyfortest/test_pyfor.py

   Line 73 in 0d45e18

   self.test_laz = cloud.Cloud(test_laz)

2. However, when I run unit tests from the console the error does not occur. I am able to load the .laz file with no error and all tests pass.

3. The latest build has passed with no errors on Travis. So I expect something is wonky with my local installation of the environment. I am also not testing the print statement that you are using in your above examples. I will add that to the unit tests and fix the tests.

4. I need to restructure the unit tests so that they cleanly test all of the three supported file types .ply, .las and .laz for all methods/attributes of Cloud. I'll make this objective secondary. Once that is completed and working I will merge laz_fix to master. If all goes well this should be done today.

1 vs. 2 are a total mystery to me. I will try uninstalling lastools/laszip completely from my environment and otherwise and see if that might help.

I have a chunk of the day I was going to commit to pyfor already, so I will tackle this beast first in a couple of hours or so.

[bw4sz]

Okay, i'll dig into it too. But your sense is that this an environment issue. I was just following the README, i'll start trying to isolate what's happening. I was running from shell, so too much shouldn't be declared.

[brycefrank]

I can tell you already the problem is here (you can also see I thought this would become an issue earlier on line 117!)

pyfor/pyfor/cloud.py

Lines 118 to 120 in 0d45e18

	if self.extension == '.las':
	las_version = self.data.header.version
	out = """ File Path: {}\nFile Size: {}\nNumber of Points: {}\nMinimum (x y z): {}\nMaximum (x y z): {}\nLas Version: {}

Something you can do now to alleviate your problem (tell me if anything else breaks) is to just override what the extension is. This may effect some things down the line when you want to write back to .laz which is not something I am totally sure laspy even supports. I will have to check.

pc = ("your_laz.laz")
pc.extension = '.las'

I really need to spend time building out handling multiple file types. I think I will just make the ad-hox fixes for you today and then spend a week or two building it properly on 0.3.2. I plan on adding more file type support so it is best to do this in a robust way.

[bw4sz]

Confirmed that the workaround is fine. I'll let you know if I run into anything else.

[brycefrank]

Ben,

Also to note:

laspy/laspy#73 (comment)

I am learning a lot of the internals of laspy right now to implement spatial indexing. There is a lot of work to be done with laspy I think with regard to .laz. It is a severe limitation and I may end up trying to get that sorted out over there in the next year. It will take time.

[brycefrank]

Last thing Ben.

It was a messy fight. I forgot to pull updates to my local repository yesterday! Aye-yai-yai....

I have added the new tests to check for .laz loading. Turns out my other issues were problems with pycharm. I installed a global version of lastools as a workaround for now. It is concerning. Something to do with how pycharm searches for the path that is different from the Linux terminal. A problem for another day.

The tests are rudimentary at best, but they passed on my local machine. I will ensure they do so on travis before merging to master.

[brycefrank]

Closing for now. If anything the remaining PyCharm issue is a problem in laspy

[Sajjadmanal]

@bw4sz @brycefrank How can I read .laz file usinf laspy? I am getting below error:
LaspyException: Error mapping file: Laszip was not found on the system