developmentseed / landsat-util

A utility to search, download and process Landsat 8 satellite imagery

Landsat-util

Landsat-util is a command line utility that makes it easy to search, download, and process Landsat imagery.

Docs

For full documentation visit: https://pythonhosted.org/landsat-util/

To run the documentation locally:

$ pip install -r requirements/dev.txt
$ cd docs
$ make html

Recently Added Features

• Improved pansharpening
• Use BQA bands for cloud/snow coverage and use in color correction
• Add support for different NDVI color maps (three included)
• Add support for image clipping using the new --clip flag

Change Log

See CHANGES.txt.

landsat-util's Issues

LC80351142015001LGN00 fails on image processing

Error:

File "/Users/user/lib/repos/git/landsat-util/landsat/image.py", line 174, in run
    band = self._color_correction(band, self.bands[i], 0, cloud_cover)
  File "/Users/user/lib/repos/git/landsat-util/landsat/image.py", line 207, in _color_correction
    p_low, cloud_cut_low = self._percent_cut(band, low, 100 - (cloud_cover * 3 / 4))
  File "/Users/user/lib/repos/git/landsat-util/landsat/image.py", line 266, in _percent_cut
    return numpy.percentile(color[numpy.logical_and(color > 0, color < 65535)], (low, high))
  File "/Users/user/.virtualenvs/landsat/lib/python2.7/site-packages/numpy/lib/function_base.py", line 3054, in percentile
    interpolation=interpolation)
  File "/Users/user/.virtualenvs/landsat/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2803, in _ureduce
    r = func(a, **kwargs)
  File "/Users/user/.virtualenvs/landsat/lib/python2.7/site-packages/numpy/lib/function_base.py", line 3143, in _percentile
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/Users/user/.virtualenvs/landsat/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 121, in take
    return take(indices, axis, out, mode)
IndexError: cannot do a non-empty take from an empty axes.

improve stdout and stderr messages across the application

add flag for continues download and image processing

About command without list

python download_landsat_scene.py -o scene -b LC8 -d 20130127 -s 181025 -u usgs.txt --output /landsat/image

I ran the following command that downloaded 28GB zip files. Now how can I find that this data is latest.
And is this the full data?

Images kind of grainy

I'm not sure if this is a valid "issue", but is there any way to get higher resolution imagery than this? Google has imagery that has at least 3 times as much resolution as this.

What is this, by the way, 2 meter resolution? What is the resolution of this imagery?

Error using command sudo pip install -U git+git://github.com/developmentseed/landsat-util.git

I am using the following command on ubuntu on digital ocean:
sudo pip install -U git+git://github.com/developmentseed/landsat-util.git

But this is giving the following error:::::
ERROR>>>
pip can't proceed with requirement 'GDAL==1.11.0 (from landsat==0.2.0)' due to a pre-existing build directory.
location: /tmp/pip_build_root/GDAL
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.

Storing debug log for failure in /root/.pip/pip.log

brew install fails

Seems like this might be fixed by changing ImageMagick-6.8.9-5 to ImageMagick-6.8.9-7?

==> Downloading https://downloads.sf.net/project/machomebrew/mirror/ImageMagick-

curl: (28) Operation timed out after 6371 milliseconds with 0 out of 0 bytes received
Trying a mirror...
==> Downloading http://www.imagemagick.org/download/ImageMagick-6.8.9-5.tar.xz

curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "imagemagick"
Download failed: http://www.imagemagick.org/download/ImageMagick-6.8.9-5.tar.xz

Image data labelled as no data

After processing, certain data pixels in a scene can be labelled as no data. This is probably due to the color correction algorithm labelling very dark pixels as no data.

To reproduce:

landsat process ./LC81990332014251LGN00.tar.bz -b 432 --pansharpen

QGIS inspection tool can be used to find that the pixel values at the white value points are actually no data.

cf. this tweet

cc @drewbo @scisco

Feature Request: Landsat image of a bounding box

At one point, I tried to use landsat-util to automatically generate landsat images of given bounding boxes, and I got totally stranded trying to deal with coordinate transforms.

Would be awesome if I could just say landsat-util process <path> --crop <bbox>. Or is that super easy to do with some magic gdal command, or some such?

tar unzipping issues

Add a fallback for rare cases where python tarfile module fails to open landsat tar.bz files

process fails if image is downloaded from google instead of AWS

Landast-util's default behavior is to try to download 2015 scenes from AWS. If the scene is missing from AWS, the process falls back to Google Storage.

However, if the process flag -p is used with the download command, landsat-util assumes the files were downloaded from aws, hence, the unzip step is skipped and the process fails as no images is found.

This is a bug and has to be fixed.

No matches found/GEOS Error

Hi, thanks for this great tool, thats what I've been looking for for a while now. Downloading with scene ID works fine, but searching gives me some errors:
For example the search with "landsat search --cloud 4 --start "january 1 2014" --end "december 10 2014" pr 192 023" delivers "No matches found"
The same search with the parameter country germany results in "ERROR 6: GEOS support not enabled" as does the search with shape files.

I appreciate any help!

Fails on systems where /usr/bin/python is Python 3

On Archlinux /usr/bin/python is actually Python 3 but landsat-util seems to only work in Python 2.
I have no idea if it is possible to specify the actual (major) version of Python a script should use, but if you can, it would be great.

Installation error on Ubuntu 12.04LTS

I have tried to install landsat-util on Ubuntu 12.04lts, and I get the following on the pip-log:
/usr/bin/pip run on Wed Feb 11 15:50:49 2015
Downloading/unpacking python-pip
Getting page http://pypi.python.org/simple/python-pip
Could not fetch URL http://pypi.python.org/simple/python-pip: HTTP Error 404: Not Found
Will skip URL http://pypi.python.org/simple/python-pip when looking for download links for python-pip
Getting page http://pypi.python.org/simple/
URLs to search for versions for python-pip:

• http://pypi.python.org/simple/python-pip/
  Getting page http://pypi.python.org/simple/python-pip/
  Could not fetch URL http://pypi.python.org/simple/python-pip/: HTTP Error 404: Not Found
  Will skip URL http://pypi.python.org/simple/python-pip/ when looking for download links for python-pip
  Could not find any downloads that satisfy the requirement python-pip
  No distributions at all found for python-pip
  Exception information:
  Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main
  self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 223, in run
  requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 948, in prepare_files
  url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 152, in find_requirement
  raise DistributionNotFound('No distributions at all found for %s' % req)
  DistributionNotFound: No distributions at all found for python-pip

Ubuntu 14.04

Lemme see how much I can post and have this make sense.
I use ubuntugis-unstable

1. landsat download LC80200362014197LGN00
2. landsat process ./zip/LC80200362014197LGN00.tar.bz

Output:
/tmp/tmpZkN3ao/LC80200362014197LGN00 folder created
/tmp/tmpZkN3ao/LC80200362014197LGN00/warp folder created
/tmp/tmpZkN3ao/LC80200362014197LGN00/scaled folder created
/tmp/tmpZkN3ao/LC80200362014197LGN00/final folder created
Unzipping LC80200362014197LGN00 - It might take some time
Creating output file that is 7720P x 7893L.
Processing input file /tmp/tmpZkN3ao/LC80200362014197LGN00/LC80200362014197LGN00_B4.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7720P x 7893L.
Processing input file /tmp/tmpZkN3ao/LC80200362014197LGN00/LC80200362014197LGN00_B3.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7720P x 7893L.
Processing input file /tmp/tmpZkN3ao/LC80200362014197LGN00/LC80200362014197LGN00_B2.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 15439P x 15786L.
Processing input file /tmp/tmpZkN3ao/LC80200362014197LGN00/LC80200362014197LGN00_B8.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Starting the Min/Max process with designated -percent cut- for band 4 of LC80200362014197LGN00
...
Finished processing band 4 of LC80200362014197LGN00
Starting the Min/Max process with designated -percent cut- for band 3 of LC80200362014197LGN00
...
Finished processing band 3 of LC80200362014197LGN00
Starting the Min/Max process with designated -percent cut- for band 2 of LC80200362014197LGN00
...
Finished processing band 2 of LC80200362014197LGN00
scaling pan to min max with 2% cut for band 4
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 3
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 2
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
/tmp/tmpZkN3ao/LC80200362014197LGN00/warp/LC80200362014197LGN00_B4.TIF[0] TIFF 7720x7893 7720x7893+0+0 16-bit Grayscale DirectClass 121.9MB 1.890u 0:01.919
/tmp/tmpZkN3ao/LC80200362014197LGN00/warp/LC80200362014197LGN00_B3.TIF[0] TIFF 7720x7893 7720x7893+0+0 16-bit Grayscale DirectClass 121.9MB 1.300u 0:01.300
/tmp/tmpZkN3ao/LC80200362014197LGN00/warp/LC80200362014197LGN00_B2.TIF[0] TIFF 7720x7893 7720x7893+0+0 16-bit Grayscale DirectClass 121.9MB 0.720u 0:00.719
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
/tmp/tmpZkN3ao/LC80200362014197LGN00/scaled/LC80200362014197LGN00_B4.TIF[0] TIFF 7720x7893 7720x7893+0+0 8-bit Grayscale DirectClass 61MB 1.860u 0:01.889
/tmp/tmpZkN3ao/LC80200362014197LGN00/scaled/LC80200362014197LGN00_B3.TIF[0] TIFF 7720x7893 7720x7893+0+0 8-bit Grayscale DirectClass 61MB 1.240u 0:01.259
/tmp/tmpZkN3ao/LC80200362014197LGN00/scaled/LC80200362014197LGN00_B2.TIF[0] TIFF 7720x7893 7720x7893+0+0 8-bit Grayscale DirectClass 61MB 0.650u 0:00.649
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
Starting the image processing
global name 'cv2' is not defined
Skipping Image Correction using OpenCV
Convertin image tweaks
Convert: averaging
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
convert.im6: Unknown field with tag 42113 (0xa481) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
OSError: [Errno 2] No such file or directory

Update documentation

@ianschuler @scisco @KAPPS- @YKCzoli From a satellite imagery newb's perspective, I would like to see a more detailed overview of what this tool can do, such as the following.

Overview: What can landsat-util do?

Landsat-util has three main functions:

• Search for landsat tiles based on several search parameters.
• Download landsat images.
• Image processing and pan sharpening on landsat images.

These three functions can be performed separately or all at once.
Help: Type landsat -h for detailed usage parameters.

Step 1: Search

Search for landsat tiles in a given geographical region, using any of the following:

• Landsat paths and rows: If you know the paths and rows you want to search for.
• Country name: If you know what country you want imagery for.
• Custom shapefile: Use a tool such as http://geojson.io/ to generate custom shapefiles bounding your geographical region of interest. Landsat-util will download tiles within this shapefile.

Additionally filter your search using the following parameters:

• Start and end dates for when imagery was taken
• Maximum percent cloud cover (default is 20%)

Search returns information about all landsat tiles that match your search criteria. This includes a link to an unprocessed preview of the tile. The most important thing is the tile's sceneID, which you will need to download the tile (see step 2 below).

Examples of search:
Search by path and row:
$: landsat search --cloud 6 --start "july 01 2014" --end "august 1 2014" pr 165 100

Search by country:
$: landsat search --cloud 6 --start "july 01 2014" --end "august 1 2014" country 'Singapore'

Search by custom shapefile:
$: landsat search --cloud 6 --start "july 01 2014" --end "august 1 2014" shapefile path/to/shapefile.shp

Step 2: Download

You can download tiles using their unique sceneID, which you get from landsat search.

Examples of download:
Download images by their custom sceneID, which you get from landsat search:
$: landsat download LC80030032014142LGN00 LC80030032014158LGN00

Search and download tiles all at once with the --download flag:
$: landsat search --download --cloud 6 --start "july 01 2014" --end "august 1 2014" pr 165 100

Step 3: Image processing

You can process your downloaded tiles with our custom image processing algorithms. In addition, you can choose to pansharpen your images.

Examples of image processing:
Process images that are already downloaded. Remember, the program only accepts zip files:
$: landsat process path/to/LC80030032014158LGN00.tar.bz

Process and pansharpen a downloaded image:
$: landsat process --pansharpen path/to/LC80030032014158LGN00.tar.bz

Search, download, and process images all at once using the --imageprocess flag:
$: landsat search --imageprocess --cloud 6 --start "july 01 2014" --end "august 1 2014" shapefile path/to/shapefile.shp

Processed "final.TIF" images are not georeferenced (as warned by error messages when processing)

My final.TIF images are not openable with Google Earth Pro or any other program which loads georeferenced TIF (raster) images into a coordinate reference system. I'm seeing in the output when I run the process command that there are certain attributes that are not being transposed to the final output. I assume this is it. I'm running this process from Ubuntu on a VirtualBox, everything has been installed through apt-get.

So, I just ran this command:

david:~$ landsat process LC80160362015028LGN00.tar.bz

and got this output from the command line:

/tmp/tmpJenmEl/LC80160362015028LGN00 folder created
/tmp/tmpJenmEl/LC80160362015028LGN00/warp folder created
/tmp/tmpJenmEl/LC80160362015028LGN00/scaled folder created
/tmp/tmpJenmEl/LC80160362015028LGN00/final folder created
/home/david/landsat/processed/LC80160362015028LGN00 folder created
Unzipping LC80160362015028LGN00 - It might take some time
Creating output file that is 7715P x 7895L.
Processing input file /tmp/tmpJenmEl/LC80160362015028LGN00/LC80160362015028LGN00_B4.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7715P x 7895L.
Processing input file /tmp/tmpJenmEl/LC80160362015028LGN00/LC80160362015028LGN00_B3.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7715P x 7895L.
Processing input file /tmp/tmpJenmEl/LC80160362015028LGN00/LC80160362015028LGN00_B2.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 15428P x 15789L.
Processing input file /tmp/tmpJenmEl/LC80160362015028LGN00/LC80160362015028LGN00_B8.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Starting the Min/Max process with designated -percent cut- for band 4 of LC80160362015028LGN00
...
Finished processing band 4 of LC80160362015028LGN00
Starting the Min/Max process with designated -percent cut- for band 3 of LC80160362015028LGN00
...
Finished processing band 3 of LC80160362015028LGN00
Starting the Min/Max process with designated -percent cut- for band 2 of LC80160362015028LGN00
...
Finished processing band 2 of LC80160362015028LGN00
scaling pan to min max with 2% cut for band 4
Input file size is 7715, 7895
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 3
Input file size is 7715, 7895
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 2
Input file size is 7715, 7895
0...10...20...30...40...50...60...70...80...90...100 - done.
/tmp/tmpJenmEl/LC80160362015028LGN00/warp/LC80160362015028LGN00_B4.TIF[0] TIFF 7715x7895 7715x7895+0+0 16-bit Grayscale DirectClass 121.9MB 1.410u 0:02.759
/tmp/tmpJenmEl/LC80160362015028LGN00/warp/LC80160362015028LGN00_B3.TIF[0] TIFF 7715x7895 7715x7895+0+0 16-bit Grayscale DirectClass 121.9MB 1.140u 0:02.489
/tmp/tmpJenmEl/LC80160362015028LGN00/warp/LC80160362015028LGN00_B2.TIF[0] TIFF 7715x7895 7715x7895+0+0 16-bit Grayscale DirectClass 121.9MB 0.670u 0:01.799
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
/tmp/tmpJenmEl/LC80160362015028LGN00/scaled/LC80160362015028LGN00_B4.TIF[0] TIFF 7715x7895 7715x7895+0+0 8-bit Grayscale DirectClass 60.97MB 1.430u 0:01.700
/tmp/tmpJenmEl/LC80160362015028LGN00/scaled/LC80160362015028LGN00_B3.TIF[0] TIFF 7715x7895 7715x7895+0+0 8-bit Grayscale DirectClass 60.97MB 1.090u 0:01.260
/tmp/tmpJenmEl/LC80160362015028LGN00/scaled/LC80160362015028LGN00_B2.TIF[0] TIFF 7715x7895 7715x7895+0+0 8-bit Grayscale DirectClass 60.97MB 0.720u 0:00.850
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
Starting the image processing
Starting the image processing
Starting the image processing
Convertin image tweaks
Convert: averaging
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
Input file size is 7715, 7895
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7715, 7895
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7715, 7895
0...10...20...30...40...50...60...70...80...90...100 - done.
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
convert.im6: Unknown field with tag 42113 (0xa481) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
World file written to '/tmp/tmpJenmEl/LC80160362015028LGN00/warp/LC80160362015028LGN00_B4.tfw'.
The output is stored at /home/david/landsat/processed.

In the middle there you can see the errors that are likely causing this issue. Would appreciate any help you guys can give. Thanks

Inconsistency detected by ld.so under Ubuntu 14.04

New one - lets see how this works in the copying and pasting:

landsat process --pansharpen ./zip/LC80200362014197LGN00.tar.bz
/tmp/tmptn_r7_/LC80200362014197LGN00 folder created
/tmp/tmptn_r7_/LC80200362014197LGN00/warp folder created
/tmp/tmptn_r7_/LC80200362014197LGN00/scaled folder created
/tmp/tmptn_r7_/LC80200362014197LGN00/final folder created
Unzipping LC80200362014197LGN00 - It might take some time
Creating output file that is 7720P x 7893L.
Processing input file /tmp/tmptn_r7_/LC80200362014197LGN00/LC80200362014197LGN00_B4.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7720P x 7893L.
Processing input file /tmp/tmptn_r7_/LC80200362014197LGN00/LC80200362014197LGN00_B3.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7720P x 7893L.
Processing input file /tmp/tmptn_r7_/LC80200362014197LGN00/LC80200362014197LGN00_B2.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 15439P x 15786L.
Processing input file /tmp/tmptn_r7_/LC80200362014197LGN00/LC80200362014197LGN00_B8.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Starting the Min/Max process with designated -percent cut- for band 4 of LC80200362014197LGN00
...
Finished processing band 4 of LC80200362014197LGN00
Starting the Min/Max process with designated -percent cut- for band 3 of LC80200362014197LGN00
...
Finished processing band 3 of LC80200362014197LGN00
Starting the Min/Max process with designated -percent cut- for band 2 of LC80200362014197LGN00
...
Finished processing band 2 of LC80200362014197LGN00
scaling pan to min max with 2% cut for band 4
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 3
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 2
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
/tmp/tmptn_r7_/LC80200362014197LGN00/warp/LC80200362014197LGN00_B4.TIF[0] TIFF 7720x7893 7720x7893+0+0 16-bit Grayscale DirectClass 121.9MB 0.500u 0:00.570
/tmp/tmptn_r7_/LC80200362014197LGN00/warp/LC80200362014197LGN00_B3.TIF[0] TIFF 7720x7893 7720x7893+0+0 16-bit Grayscale DirectClass 121.9MB 0.330u 0:00.359
/tmp/tmptn_r7_/LC80200362014197LGN00/warp/LC80200362014197LGN00_B2.TIF[0] TIFF 7720x7893 7720x7893+0+0 16-bit Grayscale DirectClass 121.9MB 0.170u 0:00.190
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
/tmp/tmptn_r7_/LC80200362014197LGN00/scaled/LC80200362014197LGN00_B4.TIF[0] TIFF 7720x7893 7720x7893+0+0 8-bit Grayscale DirectClass 61MB 0.500u 0:00.530
/tmp/tmptn_r7_/LC80200362014197LGN00/scaled/LC80200362014197LGN00_B3.TIF[0] TIFF 7720x7893 7720x7893+0+0 8-bit Grayscale DirectClass 61MB 0.330u 0:00.339
/tmp/tmptn_r7_/LC80200362014197LGN00/scaled/LC80200362014197LGN00_B2.TIF[0] TIFF 7720x7893 7720x7893+0+0 8-bit Grayscale DirectClass 61MB 0.170u 0:00.170
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 33922 (0x8482) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768. convert.im6: Unknown field with tag 34737 (0x87b1) encountered.TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
Starting the image processing
global name 'cv2' is not defined
Skipping Image Correction using OpenCV
Convertin image tweaks
Convert: averaging
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7720, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
convert.im6: Unknown field with tag 42113 (0xa481) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
World file written to '/tmp/tmptn_r7_/LC80200362014197LGN00/warp/LC80200362014197LGN00_B4.tfw'.

2014 Sep 02 18:22:42 : Application.logger (INFO) Elevation management: setting default height above ellipsoid to 0 meters
Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 0% [ Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 2% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 4% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 6% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 8% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 10% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 12% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 14% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 16% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 18% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 20% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 22% [******** Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 24% [Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 26% [Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 28% [Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 30% [Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 32% [Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 34% [* Writing /tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF...: 100% [******](1e+02 seconds)
Inconsistency detected by ld.so: dl-close.c: 762: dl_close: Assertion `map->l_init_called' failed!
subprocess.CalledProcessError: Command '['otbcli_BundleToPerfectSensor', '-inp', '/tmp/tmptn_r7/LC80200362014197LGN00/warp/LC80200362014197LGN00_B8.TIF', '-inxs', '/tmp/tmptn_r7_/LC80200362014197LGN00/final/comp.TIF', '-out', '/tmp/tmptn_r7_/LC80200362014197LGN00/final/pan.TIF', 'uint16']' returned non-zero exit status 127

Ask before creating folder

If the ~/landsat folder doesn't exist, we should ask the user if he wants to create it. Also, if it's the first time using landsat-util, we should ask for config preferences.

Update tests to better use mocks

Use mock to shorten and improve tests

selective band download does not always work and returns a tar of all bands

This one works fine: landsat download LC80420362015082LGN00 --bands 57
it downloads to a subfolder B5,B7,MTL as expected

This one does not: landsat download LC80420362013108LGN01 --bands 57
it downloads a tar.bz with all bands

Search function is missing a lot of results that are in fact stored by Google

Whats up homies!

I noticed that the search function does not capture a lot of results. I am running into this issue more often than not.

Basically if I run a search with 'landsat search' I will get zero results, but then find that imagery does indeed exist on EarthExplorer. And it is not just that this imagery is not yet on the Google repo. I know this because I will enter in the scene ID number that I get from EarthExplorer and successfully use 'landsat download' to retrieve that scene for processing.

@scisco any reason why this might be the case?

Best,
Alex

magick error

After updating Xcode to 4.6.3 and running brew doctor, still i get this error after running
the install for landsat-util--has anyone seen this before?

1 error generated.
make[1]: *** [magick/magick_libMagickCore_6_Q16_la-annotate.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [install] Error 2

download band 8 in continuous download/process/pansharpen command

This command landsat download LC81410412015088LGN00 --bands 432 -p --pansharpen fails because band 8 is not downloaded from aws.

Have to make sure that band 8 is downloaded if pansharpen flag is used for 2015 and newer imagery.

GeoJSON output for landsat-search

Being able to landsat search <params> --geojson | geojsonio and get a map showing footprints could be handy at times. It would require either including the WRS databases as module data or setting up an API to serve path/row footprints. I'm happy to work on this if you like the idea.

doesn't work

Its probably a simple fix but I'm not an IT guy. I keep getting the following error

landsat search --cloud 4 --start "january 1 2014" --end "january 10 2014" -p 009,045
Traceback (most recent call last):
File "/Users/schull/anaconda/bin/landsat", line 5, in
import landsat.landsat
File "/Users/schull/anaconda/lib/python2.7/site-packages/landsat/landsat.py", line 21, in
from image import Process, FileDoesNotExist
File "/Users/schull/anaconda/lib/python2.7/site-packages/landsat/image.py", line 13, in
import rasterio
File "/Users/schull/anaconda/lib/python2.7/site-packages/rasterio/init.py", line 8, in
from rasterio._base import eval_window, window_shape, window_index
ImportError: dlopen(/Users/schull/anaconda/lib/python2.7/site-packages/rasterio/_base.so, 2): Library not loaded: libgdal.1.dylib
Referenced from: /Users/schull/anaconda/lib/python2.7/site-packages/rasterio/_base.so
Reason: image not found

Any ideas how to fix it?

add pg-pages site

@kamicut

Unnecessary gdal_warp

In https://github.com/developmentseed/landsat-util/blob/master/landsat/image_helper.py#L423-L425 we're warping band 8 even though we don't use it if the pansharpen flag isn't set.

Colour & contrast correction

Finding that some scenes returning results that are really dark after processing and pansharpening.
Two examples attached.
SceneIDs:
LC81950292014015LGN00
LC81760362014250LGN00

installation on windows error + use in QGIS

Hi

I'm a complete beginner in command line and programming, and tried to install landsat-util. When you say that landsat-util is a command line utility, does that means that i can run it in windows powershell?

Had this error during installation:
Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Tiago Monge\Documents\GitHub> pip install -U git+git://github.com/devel
opmentseed/landsat-util.git
Downloading/unpacking git+git://github.com/developmentseed/landsat-util.git
Cloning git://github.com/developmentseed/landsat-util.git to c:\users\tiago mo
nge\appdata\local\temp\pip-66ylie23-build
Running setup.py (path:C:\Users\Tiago Monge\AppData\Local\Temp\pip-66ylie23-bu
ild\setup.py) egg_info for package from git+git://github.com/developmentseed/lan
dsat-util.git
Traceback (most recent call last):
File "", line 17, in
File "C:\Users\Tiago Monge\AppData\Local\Temp\pip-66ylie23-build\setup.py"
, line 31
print error
^
SyntaxError: invalid syntax
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 17, in

File "C:\Users\Tiago Monge\AppData\Local\Temp\pip-66ylie23-build\setup.py", li
ne 31

print error

          ^

SyntaxError: invalid syntax

---

Cleaning up...
Command python setup.py egg_info failed with error code 1 in C:\Users\Tiago Mong
e\AppData\Local\Temp\pip-66ylie23-build
Storing debug log for failure in C:\Users\Tiago Monge\pip\pip.log
C:\Users\Tiago Monge\Documents\GitHub>

What should i do?

Instead of using comand line, i would like to know if it is possible to run the Landsat-util application on the software QGIS 2.4? (I downloaded the ZIP that is on the right pannel of your GITHub page, but didn't figure out what py file should i run as a script in QGIS to use the whole landsat-util tool). How to use it in QGIS 2.4?

Thanks in advance, and sorry for the total beginner questions and confusion. I would appreciate that you explain in a simple way.

Tiago Monge

returned non-zero exit status 4

Hi. Congratulations on this very usefull tool.
I'm having trouble processing the images after dowloading them.
Here's the console text:

"C:\Programs\landsat-util-master>python C:\Programs\landsat-util-master\landsat\l
andsat.py search --download --imageprocess --cloud 20 --start "january 1 2014" -
-end "april 10 2014" pr 204 032
1 items were found
{
"limit": 100,
"results": [
{
"cloud": 4.16,
"date": "2014-03-19",
"path": "204",
"row": "032",
"sat_type": "L8",
"sceneID": "LC82040322014078LGN00",
"thumbnail": "http://earthexplorer.usgs.gov/browse/landsat_8/2014/20
4/032/LC82040322014078LGN00.jpg"
}
],
"status": "SUCCESS",
"total": 1,
"total_returned": 1
}
Starting the download:
Skipping existing item: file://C:\Users\Administrator/landsat/zip/LC820403220140
78LGN00.tar.bz
1 images were downloaded
c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC82040322014078LGN00 folder cr
eated
c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC82040322014078LGN00/warp fold
er created
c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC82040322014078LGN00/scaled fo
lder created
c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC82040322014078LGN00/final fol
der created
Unzipping LC82040322014078LGN00 - It might take some time
Creating output file that is 7828P x 7932L.
Processing input file c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC8204032
2014078LGN00/LC82040322014078LGN00_B4.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7828P x 7932L.
Processing input file c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC8204032
2014078LGN00/LC82040322014078LGN00_B3.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7828P x 7932L.
Processing input file c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC8204032
2014078LGN00/LC82040322014078LGN00_B2.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 15655P x 15862L.
Processing input file c:\users\admini1\appdata\local\temp\1\tmpybepa2/LC8204032
2014078LGN00/LC82040322014078LGN00_B8.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Starting the Min/Max process with designated -percent cut- for band 4 of LC82040
322014078LGN00
...
Finished processing band 4 of LC82040322014078LGN00
Starting the Min/Max process with designated -percent cut- for band 3 of LC82040
322014078LGN00
...
Finished processing band 3 of LC82040322014078LGN00
Starting the Min/Max process with designated -percent cut- for band 2 of LC82040
322014078LGN00
...
Finished processing band 2 of LC82040322014078LGN00
scaling pan to min max with 2% cut for band 4
Input file size is 7828, 7932
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 3
Input file size is 7828, 7932
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 2
Input file size is 7828, 7932
0...10...20...30...40...50...60...70...80...90...100 - done.
Invalid Parameter - -combine
Traceback (most recent call last):
File "C:\Programs\landsat-util-master\landsat\landsat.py", line 285, in
main()
File "C:\Programs\landsat-util-master\landsat\landsat.py", line 282, in __main
__
main(args)
File "C:\Programs\landsat-util-master\landsat\landsat.py", line 236, in main
p.full()
File "C:\Programs\landsat-util-master\landsat\image_helper.py", line 108, in f
ull
self._combine()
File "C:\Programs\landsat-util-master\landsat\image_helper.py", line 340, in _
combine
subprocess.check_call(argv)
File "C:\OSGeo4W64\apps\Python27\lib\subprocess.py", line 542, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['convert', '-identify', '-combine', u'c
:\users\admini1\appdata\local\temp\1\tmpybepa2/LC82040322014078LGN00/war
p/LC82040322014078LGN00_B4.TIF', u'c:\users\admini1\appdata\local\temp\1
\tmpybepa2/LC82040322014078LGN00/warp/LC82040322014078LGN00_B3.TIF', u'c:\users
\admini1\appdata\local\temp\1\tmpybepa2/LC82040322014078LGN00/warp/LC8204
0322014078LGN00_B2.TIF', u'c:\users\admini1\appdata\local\temp\1\tmpybep
a2/LC82040322014078LGN00/final/rgb-null.TIF']' returned non-zero exit status 4
"

Projection occasionally fails for Polar Regions

The way the shape/pixel size is being set is based on the scene "pointing up" which is not always the case for regions which are projected as polar stereographic. Relevant code is here and should be adjusted to be conditional on crs (or generalized)

Create a Github Pages branch

I took the liberty of forking this and creating a gh-pages branch + associated html/css/js.

• Website
• Repository

My intention was to create a pull-request, but as far as I can tell you only only do that across existing branches, and in this case, you don't yet have a gh-pages branch.

Landsat-util is so much nicer to use than the Earth Explorer, I thought it could use a public facing website in addition to the README. Hope this is useful to you folks!

add test coverage

landsat process fails with "OSError: [Errno 2] No such file or directory"

Running on Ubuntu 14.04, here is the output from the attempt to process a file, with the error message at the end.

root@ip-10-147-29-195:~/landsat# landsat process /root/landsat/zip/LC80060512014211LGN00.tar.bz
/tmp/tmpLwHYf6/LC80060512014211LGN00 folder created
/tmp/tmpLwHYf6/LC80060512014211LGN00/warp folder created
/tmp/tmpLwHYf6/LC80060512014211LGN00/scaled folder created
/tmp/tmpLwHYf6/LC80060512014211LGN00/final folder created
Unzipping LC80060512014211LGN00 - It might take some time
Creating output file that is 7609P x 7822L.
Processing input file /tmp/tmpLwHYf6/LC80060512014211LGN00/LC80060512014211LGN00_B4.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7609P x 7822L.
Processing input file /tmp/tmpLwHYf6/LC80060512014211LGN00/LC80060512014211LGN00_B3.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7609P x 7822L.
Processing input file /tmp/tmpLwHYf6/LC80060512014211LGN00/LC80060512014211LGN00_B2.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 15217P x 15642L.
Processing input file /tmp/tmpLwHYf6/LC80060512014211LGN00/LC80060512014211LGN00_B8.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Starting the Min/Max process with designated -percent cut- for band 4 of LC80060512014211LGN00
...
Finished processing band 4 of LC80060512014211LGN00
Starting the Min/Max process with designated -percent cut- for band 3 of LC80060512014211LGN00
...
Finished processing band 3 of LC80060512014211LGN00
Starting the Min/Max process with designated -percent cut- for band 2 of LC80060512014211LGN00
...
Finished processing band 2 of LC80060512014211LGN00
scaling pan to min max with 2% cut for band 4
Input file size is 7609, 7822
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 3
Input file size is 7609, 7822
0...10...20...30...40...50...60...70...80...90...100 - done.
scaling pan to min max with 2% cut for band 2
Input file size is 7609, 7822
0...10...20...30...40...50...60...70...80...90...100 - done.
OSError: [Errno 2] No such file or directory

Process is eating up all the space on my SSD

Yesterday I tried to get some satellite imagery from Landsat-8 using your utility, but the process stopped because I ran out of free space on my SSD. I started with about 20 GB of free space (I'm on a Mac, 10.10, 6 GB of ram) but after a while it stopped. I had the final image in the ~/landsat folder (about 200 MB), I opened it, it seems fine nothing interesting. But the process in the terminal was still running, although it seemed pretty much dead, so eventually I quit it.
So the image seems to be okay, but most of my free space has gone.
I checked /private/var/folders/_3/67r5_rl53y99y3sxth7bk95m0000gn/T/ where I found two tmp***** folders full of TIF satellite images, taking up approx. 16 GB of space. After several shutdown/restart they are till there.
I assume they are supposed to be deleted after completion, right? Maybe because the process never finished normally, they ended up undeleted.
What went wrong? Is it safe to remove them by hand?

Search function not working 100%

Hi and thanks for the nice tool you are developing.

I started to play with it and noticed the search function is not providing 100% of the scene available (compared to the same search in Glovis).
As an example I searched for a LS8 image path/row 190/036 between 1 october 2014 and 11 december 2012 (even with 20% cloud cover) and landsat-util returned none, when glovis found 3 (max 6% cloud cover).
On the other side, once the sceneid found in Glovis, landsat-util is able to download it.

Best regards,
Bruno

convert exit functions to return for main function

Currently the main function writes messages to stdout and exit at the end of each successful task.

Change this behavior to return the message to the invoking entry point and exit there. This will enable better integrations with tools that are using landsat-util programattically.

memory conservation on image processing

To further conserve memory on non-pansharpen images, process each band separately.

Make PyPi package

Working on making a PyPi package to ease the installation process.

seg fault on run after brew install

When running landsat -h it errors out with this message: Segmentation fault: 11
Running python 2.7 and OSX 10.9.4.

GDAL issue

Hi,
I installed all dependencies but I'm running into this error (regarding GDAL)

//
i-a$ landsat -h
ImportError: dlopen(//anaconda/lib/python2.7/site-packages/osgeo/_gdal.so, 2): Symbol not found: __ZNSt8ios_base4InitD1Ev
Referenced from: //anaconda/lib/python2.7/site-packages/osgeo/_gdal.so
Expected in: dynamic lookup
//

I installed GDAL through Macport but I also have anaconda installed on my system.

Thanks

unexpected behavior downloading multiple scenes

Running the following:

landsat download -p LC80110272014262LGN00 LC80100272014255LGN00

Yields:

===> Downloading: LC80110272014262LGN00.tar.bz
     LC80110272014262LGN00.tar.bz already exists on your system
===> Downloading: LC80100272014255LGN00.tar.bz
     LC80100272014255LGN00.tar.bz already exists on your system
===> Unzipping LC80110272014262LGN00 - It might take some time
* Image processing started for bands 4-3-2
===> Getting boundaries
===> Projecting
     band 4
     band 3
     band 2
===> Final Steps
     Color correcting band 4
     Color correcting band 3
     Color correcting band 2
     Writing to file
===> The output is stored at /Users/j/landsat/processed/LC80110272014262LGN00/LC80110272014262LGN00_bands_432.TIF
===> Done!
Time spent : 224.74 seconds

I'd have expected to get both scenes processed, but it's only generating a composite for the first scene.

add flag for custom download folder

Explore ways to remove scikit image dependencies

Explore whether it makes sense to reimplement the two scikit images functions used in image processing.

can not find gdal-config

C:\Python27\landsat-util-master>python setup.py install
Traceback (most recent call last):
File "setup.py", line 20, in
if subprocess.call(['which', 'gdal-config']):
File "C:\Python27\lib\subprocess.py", line 522, in call
return Popen(_popenargs, *_kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 710, in init
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

OSError: [Errno 12] Cannot allocate memory

when i try to render the image of the manual, the following error occurs:

marcello@ubuntu:SEM → landsat process --pansharpen LC80090452014008LGN00.tar.bz
/tmp/tmpEJwZXS/LC80090452014008LGN00 folder created
/tmp/tmpEJwZXS/LC80090452014008LGN00/warp folder created
/tmp/tmpEJwZXS/LC80090452014008LGN00/scaled folder created
/tmp/tmpEJwZXS/LC80090452014008LGN00/final folder created
/home/marcello/landsat/processed/LC80090452014008LGN00 folder created
Unzipping LC80090452014008LGN00 - It might take some time
Creating output file that is 7657P x 7857L.
Processing input file /tmp/tmpEJwZXS/LC80090452014008LGN00/LC80090452014008LGN00_B4.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7657P x 7857L.
Processing input file /tmp/tmpEJwZXS/LC80090452014008LGN00/LC80090452014008LGN00_B3.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 7657P x 7857L.
Processing input file /tmp/tmpEJwZXS/LC80090452014008LGN00/LC80090452014008LGN00_B2.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Creating output file that is 15312P x 15712L.
Processing input file /tmp/tmpEJwZXS/LC80090452014008LGN00/LC80090452014008LGN00_B8.TIF.
0...10...20...30...40...50...60...70...80...90...100 - done.
Starting the Min/Max process with designated -percent cut- for band 4 of LC80090452014008LGN00
...
Finished processing band 4 of LC80090452014008LGN00
Starting the Min/Max process with designated -percent cut- for band 3 of LC80090452014008LGN00
...
Finished processing band 3 of LC80090452014008LGN00
Starting the Min/Max process with designated -percent cut- for band 2 of LC80090452014008LGN00
...
Finished processing band 2 of LC80090452014008LGN00
scaling pan to min max with 2% cut for band 4
OSError: [Errno 12] Cannot allocate memory

My machine has 8 gb of RAM and I believe this can be a problem in Python

Inconsistency detected by ld.so - Ubuntu 14.04

I think everything ran to completion - but it did toss an error at the end. I had a Final.tif in the Processed folder though. I only included the last part of the output (since that seemed most important).

Convert: averaging
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
Input file size is 7789, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7789, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 7789, 7893
0...10...20...30...40...50...60...70...80...90...100 - done.
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
/usr/bin/gdal_calc.py:266: FutureWarning: comparison to None will result in an elementwise object comparison in the future.
myNDVs=1_numpy.logical_or(myNDVs==1, myval==myNDV[i])
0 .. 10 .. 20 .. 30 .. 40 .. 50 .. 60 .. 70 .. 80 .. 90 .. 100 - Done
convert.im6: Unknown field with tag 42113 (0xa481) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
World file written to '/tmp/tmpE2llmD/LC80210352014012LGN00/warp/LC80210352014012LGN00_B4.tfw'.

2014 Sep 21 17:13:49 : Application.logger (INFO) Elevation management: setting default height above ellipsoid to 0 meters
Writing /tmp/tmpE2llmD/LC80210352014012LGN00/final/pan.TIF...: 100% [**************************************************](1e+02 seconds)
Inconsistency detected by ld.so: dl-close.c: 762: _dl_close: Assertion `map->l_init_called' failed!
subprocess.CalledProcessError: Command '['otbcli_BundleToPerfectSensor', '-inp', '/tmp/tmpE2llmD/LC80210352014012LGN00/warp/LC80210352014012LGN00_B8.TIF', '-inxs', '/tmp/tmpE2llmD/LC80210352014012LGN00/final/comp.TIF', '-out', '/tmp/tmpE2llmD/LC80210352014012LGN00/final/pan.TIF', 'uint16']' returned non-zero exit status 127
rjhale@gis2:~/landsat$ ls

add feature for uploading to s3

Installing on Yosemite

Trying to get this running on a Yosemite machine. Running into some dependency issues:

$: brew install https://raw.githubusercontent.com/developmentseed/landsat-util/master/Formula/landsat-util.rb
######################################################################## 100.0%
######################################################################## 100.0%
######################################################################## 100.0%
==> Downloading https://github.com/developmentseed/landsat-util/archive/v0.2.0.tar.gz
Already downloaded: /Library/Caches/Homebrew/landsat-util-0.2.0.tar.gz
==> Patching
==> python setup.py install --prefix=/usr/local/Cellar/landsat-util/0.2.0/libexec --install-scri
simplejson.tests.__init__: module references __file__
Adding simplejson 3.6.5 to easy-install.pth file

Installed /usr/local/Cellar/landsat-util/0.2.0/libexec/lib/python2.7/site-packages/simplejson-3.6.5-py2.7-macosx-10.10-intel.egg
error: Installed distribution six 1.4.1 conflicts with requirement six>=1.6.1
couldn't understand kern.osversion `14.0.0'

READ THIS: http://git.io/brew-troubleshooting

i also ran:

$: pip show six

---
Name: six
Version: 1.9.0
Location: /usr/local/lib/python2.7/site-packages
Requires: 

Any suggestions?