yeesian / gdalutils.jl Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 1.0 50.24 MB

Convenience utilities for GDAL in Julia

License: Other

Julia 100.00%

gdalutils.jl's People

Contributors

Watchers

Forkers

juliatagbot

gdalutils.jl's Issues

Have a consistent scheme for choosing names

A rough organization into a few themes:

get_count() = n_()

Examples:

    - getgcpcount -> ngcp
    - datasetgetlayercount -> nlayer
    - getrastercount -> nraster
    - getdrivercount -> ndriver
    - getpointcount -> npoint 
    - getgeometrycount -> ngeometry
    - getfieldcount -> nfield 
    - getgeomfieldcount -> ngeomfield
    - getfeaturecount -> nfeature
    - getfieldcount -> nfield
    - getoverviewcount -> noverview

Remove typenames

(rely on multi-dispatch instead) Examples:

    - datasetcopywholeraster -> copywholeraster
    - deletedataset -> delete
    - renamedataset -> rename
    - copydatasetfiles -> copyfiles
    - getrasterunittype -> getunittype
    - setrasterunittype -> setunittype
    - getrasteroffset -> getoffset
    - setrasteroffset -> setoffset 
    - getrasterscale -> getscale
    - setrasterscale -> setscale
    - getrasterdatatype -> getdatatype
    - getrasteraccess -> getaccess
    - getrasterband -> fetchband (new syntax)
    - getbandnumber -> getindex (new syntax)
    - getbanddataset -> getdataset
    - getrasternodatavalue -> getnodatavalue
    - setrasternodatavalue -> setnodatavalue
    - getrastersampleoverview -> getsampleoverview
    - getrastercolorinterpretation -> getcolorinterp (name shortening)
    - setrastercolorinterpretation -> setcolorinterp (name shortening)
    - getrastercolortable -> getcolortable
    - setrastercolortable -> setcolortable/clearcolortable (new syntax)
    - rasterbandcopywholeraster -> copywholeraster
    - getdatasetdriver -> getdriver
    - destroydriver -> destroy
    - registerdriver -> register
    - deregisterdriver -> deregister 
    - getdrivercreationoptionlist -> options (name shortening)
    - getdrivershortname -> getshortname
    - getdriverlongname -> getlongname
    - datasetrasterio -> rasterio
    - destroygeometry -> destroy
    - datasetdeletelayer -> deletelayer
    - datasetcreatelayer -> createlayer
    - datasetcopylayer -> copylayer
    - datasetexecutesql -> executesql
    - datasetreleaseresultset -> releaseresultset
    - creategeomfield -> createfield
    - addgeomfielddefn -> add
    - deletegeomfielddefn -> delete

Remove "ref"

(no pointers); Examples

    - getprojectionref -> getproj (name shortening)
    - getgeomfielddefnref -> fetchgeomfielddefn (new syntax)
    - getgeomfieldref -> fetchgeomfield (new syntax)
    - getgeometryref(geom, i) -> fetchgeom (new syntax)
    - getgeometryref(feature) -> getgeom (name shortening)
    - getfielddefnref(feature, i) -> getfielddefn
    - getdefnref -> getfeaturedefn
    - getnameref -> getname

Format

(Export/Import, From/To); Examples:

    - createfromwkb -> fromWKB
    - createfromwkt -> fromWKT
    - createfromgml -> fromGML
    - creategeometryfromjson -> fromJSON
    - importfromwkb -> fromWKB
    - exporttowkb -> toWKB
    - exporttoisowkb -> toISOWKB
    - exporttogml -> toGML
    - exporttokml -> toKML
    - exporttojson -> toJSON

New Syntax

    - [new] fetch/update
    - getrasterband -> fetchband
    - datasetgetlayer/datasetgetlayerbyname -> fetchlayer
    - openshared/openex -> read/update
    - close(createcopy(...)) -> write(...)
    - value -> pointalongline
    - setspatialfilterex -> setspatialfilter/clearspatialfilter
    - getbandnumber -> getindex
    - setrastercolortable -> setcolortable/clearcolortable
    - getfieldasinteger -> asint, etc
    - setfieldinteger -> setfield, etc
    - getnextfeature -> fetchnext
    - getrasterxsize, getrasterysize -> width, height
    - getrasterbandxsize, getrasterbandysize -> width, height
    - assignspatialreference -> setspatialref
    - removegeometry -> removegeom /removeallgeoms

Name dropping/shortening

Examples:

    - coordinate -> coord
    - dimension -> dim
    - spatialreference -> spatialref
    - geometry -> geom
    - interpretation -> interp
    - buildpolygon -> polygon
    - nativemedia -> media
    - fillunsetwithdefault -> fillunset
    - drivercreationoptionlist -> options
    - projection -> proj

instances:

    - getcoordinatedimension -> getcoorddim
    - setcoordinatedimension -> setcoorddim
    - getgeometrytype -> getgeomtype
    - getgeometryname -> getgeomname
    - assignspatialreference -> setspatialref (new syntax)
    - getspatialreference -> getspatialref 
    - addgeometry -> addgeom
    - addgeometrydirectly -> addgeomdirectly
    - removegeometry -> removegeom /removeallgeoms (new syntax)
    - hascurvegeometry -> hascurvegeom
    - getlineargeometry -> getlineargeom
    - getgeometrycolumn -> getgeomcolumn
    - getcolorinterpretationbyname -> getcolorinterp
    - validatecreationoptions -> validate
    - buildpolygonfromedges -> polygonfromedges
    - getnativemediatype -> getmediatype
    - setnativemediatype -> setmediatype
    - fillunsetwithdefault -> fillunset 
    - getdrivercreationoptionlist -> options
    - getprojectionref -> getproj
    - setprojection -> setproj

Provide easy way to list the "head" of a FeatureLayer

will need to dig into the way DataFrames.jl prints her REPL output

reference: http://gis-lab.info/docs/gdal/gdal_ogr_user_docs.html#ogrinfo

list creation options in display of drivers at the REPL

julia> r = LightXML.parse_string(GU.options(GU.driver("GTiff")))

julia> collect(child_elements(root(r)))
35-element Array{Any,1}:
 <Option name="COMPRESS" type="string-select">       <Value>NONE</Value>
<Value>LZW</Value>       <Value>PACKBITS</Value>       <Value>JPEG</Value>
<Value>CCITTRLE</Value>       <Value>CCITTFAX3</Value>       <Value>CCITTFAX4</Value>
<Value>DEFLATE</Value>   </Option>
 <Option name="PREDICTOR" type="int" description="Predictor Type (1=default, 2=horizontal
differencing, 3=floating point prediction)"/>
[...]

julia> collect(child_elements(root(r)))[3]
<Option name="DISCARD_LSB" type="string" description="Number of least-significant bits to 
set to clear as a single value or comma-separated list of values for per-band values"/>

julia> options = [attribute(child, "name") for child in collect(child_elements(root(r)))]
35-element Array{Union{ASCIIString,UTF8String,Void},1}:
 "COMPRESS"
 "PREDICTOR"
 "DISCARD_LSB"
 "JPEG_QUALITY"
 "JPEGTABLESMODE"
 "ZLEVEL"
 "NUM_THREADS"
 "NBITS"
 "INTERLEAVE"
 "TILED"
 ⋮
 "SOURCE_PRIMARIES_GREEN"
 "SOURCE_PRIMARIES_BLUE"
 "SOURCE_WHITEPOINT"
 "TIFFTAG_TRANSFERFUNCTION_RED"
 "TIFFTAG_TRANSFERFUNCTION_GREEN"
 "TIFFTAG_TRANSFERFUNCTION_BLUE"
 "TIFFTAG_TRANSFERRANGE_BLACK"
 "TIFFTAG_TRANSFERRANGE_WHITE"
 "STREAMABLE_OUTPUT"
 "GEOTIFF_KEYS_FLAVOR"

julia> print(options)
Union{ASCIIString,UTF8String,Void}["COMPRESS","PREDICTOR","DISCARD_LSB","JPEG_QUALITY",
"JPEGTABLESMODE","ZLEVEL","NUM_THREADS","NBITS","INTERLEAVE","TILED","TFW","RPB",
"RPCTXT","BLOCKXSIZE","BLOCKYSIZE","PHOTOMETRIC","SPARSE_OK","ALPHA","PROFILE",
"PIXELTYPE","BIGTIFF","ENDIANNESS","COPY_SRC_OVERVIEWS","SOURCE_ICC_PROFILE",
"SOURCE_PRIMARIES_RED","SOURCE_PRIMARIES_GREEN","SOURCE_PRIMARIES_BLUE",
"SOURCE_WHITEPOINT","TIFFTAG_TRANSFERFUNCTION_RED","TIFFTAG_TRANSFERFUNCTION_GREEN",
"TIFFTAG_TRANSFERFUNCTION_BLUE","TIFFTAG_TRANSFERRANGE_BLACK",
"TIFFTAG_TRANSFERRANGE_WHITE","STREAMABLE_OUTPUT","GEOTIFF_KEYS_FLAVOR"]

provide fetch and update methods for RasterBands

Support windowed and masked reads and writes

relevant blogpost: https://sgillies.net//2013/12/21/rasterio-windows-and-masks.html
source code for block iterators: https://github.com/mapbox/rasterio/blob/master/rasterio/_base.pyx#L346-L384

Document Motivations for this Package

(Thinking of renaming this package to ArchGDAL.jl)

Like Fiona/etc, this package has a raison d'etre, and this issue tries to elicit what it is, to

guide users towards other packages that might suit their needs better, and
to facilitate decision-making for developers in PRs/contributions.

Principles (The Arch Way)

(adapted from: https://wiki.archlinux.org/index.php/Arch_Linux#Principles)

simplicity: without unnecessary additions or modifications.
(i) Preserves GDAL Data Model, and makes available GDAL/OGR methods without trying to mask them from the user.
(ii) does not import other packages
modernity: ArchGDAL strives to maintain the latest stable release versions of GDAL as long as systemic package breakage can be reasonably avoided. You can stay with older versions by pinning them
pragmatism: The principles here are only useful guidelines. Ultimately, design decisions are made on a case-by-case basis through developer consensus. Evidence-based technical analysis and debate are what matter, not politics or popular opinion.
user-centrality: whereas other libraries attempt to be more user-friendly, ArchGDAL shall be user-centric. It is intended to fill the needs of those contributing to it, rather than trying to appeal to as many users as possible.
versatility: ArchGDAL will strive to remain small in its assumptions about the range of user-needs, and to make it easy for users to build their own extensions/conveniences.

File Organization

# GDAL.jl
Thin GDAL wrapper for Julia; provides
(i) build scripts,
(ii) GDAL's C API as well as,
(iii) a cleaned-up low-level interface to GDAL
## File Organization
src/C/*:                raw GDAL/OGR C API
src/*:                  cleaned-up low-level API (that doesn't unnecessarily drop function arguments)

#ArchGDAL.jl
Lightweight philosophy on GDAL; provides
(i) julian types (look Ma/Pa, no pointers!)
(ii) error checking for you
(iii) context managers using do-blocks
## File Organization
src/types.jl            julian types ("stateless" wrappers; only contains pointer to respective GDAL objects)
src/metadata.jl         
src/driver.jl           
src/dataset.jl          
src/spatialref.jl
src/gdal/*
    - rasterband.jl
    - rasterio.jl
    - ...
src/ogr/*
    - featuredefn.jl
    - featurelayer.jl
    - fielddefn.jl
    - geometry.jl
    - ...
src/display.jl          how objects/types gets rendered at the REPL

Usage

Two modes of use envisaged:

Heavyweight/backend-processing: lightweight wrappers around pointers makes it easy for you to ingest just the information you need
- context managers using do-blocks that closes-datasets/finalizes-resources for you.
Interactive REPL exploration: gdalinfo/fio/rio-like environment that provides you with context/information for (otherwise opaque) GDAL pointers
- also makes available gdalinfo/etc via the shell mode
- docstring/help available
- facilitate easier visualizations/plots in a jupyter notebook, or via TerminalExtensions

Strikes a balance between

stateful class/objects (which are great for convenience; giving you information without having to ask),
context managers (which are great for safety; closing/finalizing resources), and
lightweight handles (great for pipelines that removes the human-in-the-middle, and when ingesting heavy amounts of data that doesn't fit well with REPL usage)

cc @visr @ljwolf @meggart
(Keeping it as an issue for now to reduce the noise in changes)

Implement remaining getters/setters for OGRFields

Just an internal TODO reminder

relevant file(s): src/ogr/featurelayer.jl

systematically identify and implement iterators

Iterating over

Provide methods for GCPs

Just an internal TODO reminder

relevant files: src/dataset.jl and src/gcp.jl

systematically identify and organize methods with equivalent do-block syntax

Reminder: document risks of floating pointers when closing/destroying objects

 | | |_| | | | (_| |  |  Version 0.4.2 (2015-12-06 21:47 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin13.4.0

julia> import GDALUtils; const GU = GDALUtils
GDALUtils

julia> raster = GU.read("pyrasterio/RGB.byte.tif")
GDAL Dataset (Driver: GTiff/GeoTIFF)

File(s): pyrasterio/RGB.byte.tif
Dataset (width x height): 791 x 718 (pixels)
  [ReadOnly] Band 1 (Red): 791 x 718 (UInt8)
  [ReadOnly] Band 2 (Green): 791 x 718 (UInt8)
  [ReadOnly] Band 3 (Blue): 791 x 718 (UInt8)
Number of bands: 3

julia> band = GU.fetchband(raster, 1)
[ReadOnly] Band 1 (Red): 791 x 718 (UInt8)
    blocksize: 791x3, nodata: 0.0, units: 1.0px + 0.0
    overviews:

julia> band.ptr
Ptr{GDAL.GDALRasterBandH} @0x00007fa3be49c810

julia> GU.close(raster)
Ptr{GDAL.GDALDatasetH} @0x0000000000000000

julia> raster
Closed Dataset

julia> band
Error showing value of type GDALUtils.RasterBand:
ERROR: KeyError: 164004320 not found
 in summarize at /Users/yeesian/.julia/v0.4/GDALUtils/src/display.jl:35
 in show at /Users/yeesian/.julia/v0.4/GDALUtils/src/display.jl:46
 in anonymous at show.jl:1294
 in with_output_limit at /Applications/Julia-0.4.2.app/Contents/Resources/julia/lib/julia/sys.dylib
 in showlimited at show.jl:1293
 in writemime at replutil.jl:4
 in display at REPL.jl:114
 in display at REPL.jl:117
 [inlined code] from multimedia.jl:151
 in display at multimedia.jl:162
 in print_response at REPL.jl:134
 in print_response at REPL.jl:121
 in anonymous at REPL.jl:624
 in run_interface at /Applications/Julia-0.4.2.app/Contents/Resources/julia/lib/julia/sys.dylib
 in run_frontend at /Applications/Julia-0.4.2.app/Contents/Resources/julia/lib/julia/sys.dylib
 in run_repl at /Applications/Julia-0.4.2.app/Contents/Resources/julia/lib/julia/sys.dylib
 in _start at /Applications/Julia-0.4.2.app/Contents/Resources/julia/lib/julia/sys.dylib

julia> band.ptr
Ptr{GDAL.GDALRasterBandH} @0x00007fa3be49c810

Provide support for virtual datasets

Relevant header file: http://www.gdal.org/gdal__vrt_8h_source.html

The VRT driver is a format driver for GDAL that allows a virtual GDAL dataset to be composed from other GDAL datasets with repositioning, and algorithms potentially applied as well as various kinds of metadata altered or added. VRT descriptions of datasets can be saved in an XML format normally given the extension .vrt.

(taken from the tutorial: http://www.gdal.org/gdal_vrttut.html)

OGR Virtual Format is a driver that transforms features read from other drivers based on criteria specified in an XML control file. It is primarily used to derive spatial layers from flat tables with spatial information in attribute columns. It can also be used to associate coordinate system information with a datasource, merge layers from different datasources into a single data source, or even just to provide an anchor file for access to non-file oriented datasources.
The virtual files are currently normally prepared by hand.

(taken from the manual: http://www.gdal.org/drv_vrt.html)