lambgeo / geo-layer Goto Github PK

also add basic gdal layers

ref: lambgeo/docker-lambda#24

after #6

Jinja2 is present in lambci image but not on AWS Lambda

quick fix:

pip install jinja2 --user --ignore-installed

I've just pushed a lot of fix in docker-lambda so we need to publish new layer here too!

https://github.com/RemotePixel/cogeo-layer

But only include

numpy
pyproj==2.4.1
rasterio>=1.1.3
requests
shapely
supermercado
pygeos

It's too hard to maintain this repo because and it's not really a great idea to fix the versions for all the module.

I'm going to Archive this repo and focus on https://github.com/lambgeo/docker-lambda

after #6

ref #5
Because I originally had -geo layer in my aws account, the version count was different throught the region and layers.

To keep everything uniq, I'm going to release the layers and docker images with a new geolayer suffix

The new ARN will be
arn:aws:lambda:{REGION}:524387336408:layer:gdal{GDAL_VERSION_NODOT}-py{PYTHON_VERSION_NODOT}-geolayer:{LAYER_VERSION}

and the docker name
lambgeo/lambda:gdal{GDAL_VERSION_SHORT}-py{PYTHON_VERSION_SHORT}-geolayer

Hi

I am trying to use the geo-layer with a NodeJS 10.x AWS Lambda function and calling ogr2ogr using child_process.exec however I get the following error

ogr2ogr: error while loading shared libraries: libexpat.so.1: cannot open shared object file: No such file or directory

Looking at the docker image it does seem to be built with expat so maybe I am not correctly setting the environment variables or maybe I need to set the LD_LIBRARY_PATH?

I have used the same method from a Python 3.7 AWS Lambda function using subprocess and it worked fine.

Any ideas?

Many thanks

🤦 I though I cleanup all old -geo layer in my account but it might not be true

    {
        "region": "us-east-1",
        "layers": [
            {
                "name": "gdal24-py37-geo",
                "arn": "arn:aws:lambda:us-east-1:524387336408:layer:gdal24-py37-geo:9",
                "version": 9
            },
            {
                "name": "gdal30-py37-geo",
                "arn": "arn:aws:lambda:us-east-1:524387336408:layer:gdal30-py37-geo:8",
                "version": 8
            },
            {
                "name": "gdal31-py37-geo",
                "arn": "arn:aws:lambda:us-east-1:524387336408:layer:gdal31-py37-geo:1",
                "version": 1
            },
            {
                "name": "gdal24-py38-geo",
                "arn": "arn:aws:lambda:us-east-1:524387336408:layer:gdal24-py38-geo:1",
                "version": 1
            },
            {
                "name": "gdal30-py38-geo",
                "arn": "arn:aws:lambda:us-east-1:524387336408:layer:gdal30-py38-geo:1",
                "version": 1
            },
            {
                "name": "gdal31-py38-geo",
                "arn": "arn:aws:lambda:us-east-1:524387336408:layer:gdal31-py38-geo:1",
                "version": 1
            }
        ]
    },

cc @kylebarron

Hi.
This error has appearead again in this new layer. And I can reproduce it, finally.

I've cloned this repo and created a makefile with the commands described in the README. However, when trying to run a simple python function, it gives the error from the title.

Makefile:

.PHONY: layer test

layer:
	docker build --tag package:latest . --build-arg GDAL_VERSION=3.0 --build-arg PYTHON_VERSION=3.8
	docker run --name lambda -w /var/task -itd package:latest bash
	docker cp scripts/create-lambda-layer.sh lambda:/create-lambda-layer.sh
	docker exec -it lambda bash /create-lambda-layer.sh
	docker cp lambda:/tmp/package.zip package.zip
	docker stop lambda
	docker rm lambda

test:
	unzip package.zip -d layer
	docker run \
		--rm \
		-v $(shell pwd):/var/task:ro,delegated \
		-v $(shell pwd)/layer:/opt:ro,delegated \
    	-e GDAL_DATA="/opt/share/gdal" \
    	-e PROJ_LIB="/opt/share/proj" \
		lambci/lambda:python3.8 handler.handler '{"some": "data"}'

Python function:

import numpy
import rasterio

def handler(event, context):
    print(numpy.__version__)
    print(rasterio.__version__)

I've already checked the linking using ldd and it seems to be linked correctly. But when executing the function, it does not work.

$ docker run --rm -it lambgeo/lambda:gdal3.0-py3.8 bash -c "ldd /opt/lib/libgdal.so | grep expat"
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb514a28000)

Am I missing something?

Again, thanks for your work with these layers. They help A LOT. And sorry for pestering you with my issues 😝

I do not know if it is your plan to make these layers publicly available. I hope it is. They are very good! But, I was trying to use the new ones today and I am getting access denied (see below).

$ aws lambda get-layer-version-by-arn --arn "arn:aws:lambda:eu-central-1:524387336408:layer:gdal31-py38-geolayer:1" --region "eu-central-1"

An error occurred (AccessDeniedException) when calling the GetLayerVersionByArn operation: User: arn:aws:iam::1234567890:user/user is not authorized to perform: lambda:GetLayerVersion on resource: arn:aws:lambda:eu-central-1:524387336408:layer:gdal31-py38-geolayer:1

I added a new layer with following ARN: arn:aws:lambda:ap-northeast-1:524387336408:layer:gdal24-python37-geo:2
Even if the python code finds libraries like shapely, it does not find osgeo so gdal. Could you help me please?