aws-iot-builder-tools / aws-greengrass-provisioner Goto Github PK

Need to add NodeJS and npm to the container

The provisioner creates the group and installation package, fails to obtain SSH keys to setup the ec2 instance.

[INFO] BasicDeploymentHelper: Deployment successful
[ERROR] AwsGreengrassProvisioner: Is a directory
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Is a directory
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.handleProvisionerFailure(AwsGreengrassProvisioner.java:45)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$main$0(AwsGreengrassProvisioner.java:31)
at io.vavr.control.Try.onFailure(Try.java:633)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.main(AwsGreengrassProvisioner.java:31)
Caused by: java.io.IOException: Is a directory
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.nio.file.Files.read(Files.java:3105)
at java.nio.file.Files.readAllBytes(Files.java:3158)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.IoHelper.lambda$readFile$c72e3cae$1(IoHelper.java:64)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.IoHelper.readFile(IoHelper.java:64)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.IoHelper.readFileAsString(IoHelper.java:58)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.lambda$getPrivateKeyFilesForSsh$0(BasicIoHelper.java:45)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.getPrivateKeyFilesForSsh(BasicIoHelper.java:47)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.lambda$getJschWithPrivateKeysLoaded$8d6a4564$1(BasicIoHelper.java:113)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.getJschWithPrivateKeysLoaded(BasicIoHelper.java:113)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.getSshSessionTask(BasicIoHelper.java:143)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:742)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:55)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.executeOrDisplayHelp(Operation.java:34)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.lambda$execute$1(Operation.java:22)
at io.vavr.control.Try.onSuccess(Try.java:648)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.execute(Operation.java:22)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$null$2(AwsGreengrassProvisioner.java:80)
at java.util.Optional.map(Optional.java:215)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$run$da6d192b$1(AwsGreengrassProvisioner.java:80)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.run(AwsGreengrassProvisioner.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.runProvisioner(AwsGreengrassProvisioner.java:53)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$main$249538b4$1(AwsGreengrassProvisioner.java:30)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.main(AwsGreengrassProvisioner.java:30)

https://assets.ubuntu.com/v1/15d52dd3-greengrass-snap-release-notes.pdf

For testing purposes it would be helpful to not have to manually launch EC2 instances. An option like —ec2-launch could build the configuration, launch an instance, scp the files to it, and run the bootstrap script.

Greengrass support in CloudFormation was recently launched

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-reference-greengrass.html

The provisioner should be able to output CloudFormation templates that use these features.

After creating a container with java -jar ../aws-greengrass-provisioner/build/libs/AwsGreengrassProvisioner.jar -g xx -d deployments/cdd-skeleton.conf -c -a X86_64 and then trying to start it with docker run -p 8883:8883 greengrass:xx it fails with:

Setting up greengrass daemon
Validating hardlink/softlink protection
Found cgroup subsystem:  name=systemd
Found cgroup subsystem:  pids
Found cgroup subsystem:  hugetlb
Found cgroup subsystem:  net_prio
Found cgroup subsystem:  perf_event
Found cgroup subsystem:  net_cls
Found cgroup subsystem:  freezer
Found cgroup subsystem:  devices
Found cgroup subsystem:  memory
Found cgroup subsystem:  blkio
Found cgroup subsystem:  cpuacct
Found cgroup subsystem:  cpu
Found cgroup subsystem:  cpuset
failed to create overlay fs for container nosysRootfs operation not permitted
The Greengrass daemon process with [pid = 62] died

Maybe I'm just missing something from the startup command?

There are multiple scenarios that need to be tested with integration tests. They are a combination of the two ways GGP itself it can be run and the kinds of builds it can perform.

Ways to run GGP:

• Native builds (running GGP from the JAR)
• Docker builds (running GGP in Docker)

Kinds of builds it can perform:

• Java functions
• NodeJS functions without additional dependencies
• NodeJS functions with additional dependencies
• Python functions without additional dependencies
• Python functions with additional dependencies
• Combined builds (building Java + NodeJS + Python)

For the first pass that gives us 12 combinations of builds we need to test. The first step is to do this with a simple script and then move it to something more formal if necessary.

When launching java functions with --docker-launch the deployment fails because of seemingly missing java8 executable.

java -jar ../aws-greengrass-provisioner/build/libs/AwsGreengrassProvisioner.jar -g xx -d deployments/cdd-skeleton.conf --docker-launch

docker exec -it xx cat /greengrass/ggc/var/log/system/runtime.log

[2019-01-14T14:27:29.675Z][INFO]-Starting worker arn:aws:lambda:eu-west-1:xx:function:xx-CDDSkeletonJava:19
[2019-01-14T14:27:29.675Z][ERROR]-Deployment xx of type NewDeployment for group xx failed error: unable to create worker process for arn:aws:lambda:eu-west-1:xx:function:xx-CDDSkeletonJava:19. cannot find executable java8 under any of the provided paths [/usr/bin /usr/local/bin]
[2019-01-14T14:27:29.675Z][INFO]-Updating status to Failure of deployment xx for group xx

However java 8 is actually present in the container, just not in the path where ggc is looking for it. A quick fix for example is to create a symlink ln -s /etc/alternative/java /usr/local/bin/java8 and then redeploying the function.

Some example functions have an environment variable called PORT that is used to tell a pinned Greengrass Lambda function what port to listen on. When running in EC2 we should update the security groups so this function is accessible externally. When running in Docker we should map this port to the host.

Error looks like this with --ec2-launch

install: unrecognized option '--upgrade'
Try 'install --help' for more information.
install: missing destination file operand after 'AWSIoTPythonSDK'
Try 'install --help' for more information.
install: missing destination file operand after 'mxnet'
Try 'install --help' for more information.
install: missing destination file operand after 'opencv-python'
Try 'install --help' for more information.

1.8.0 is out, we need to support it

The root CA in the docs is root.ca.pem. We used root-ca.pem. We cannot stand for an egregious discrepancy like this.

1.8.0 has been superseded by 1.8.1

Detection of running Docker containers reports that a container is running if it exists, rather than if it is actually running. This needs to be fixed.

See https://docs.aws.amazon.com/greengrass/latest/apireference/createfunctiondefinitionversion-post.html

Partial snippet of function configuration:

{
	"Environment": {
		"Execution": {
			"IsolationMode": "GreengrassContainer|NoContainer",
			"RunAs": {
				"Uid": "integer",
				"Gid": "integer"
			}
		}
	}
}

Adding a device to a Greengrass group no longer works because the list returned from the latest SDK is unmodifiable. It throws an unsupported operation exception.

Released 2019-05-01

Add support for running Device Tester

Add support for connectors, there appears to be a generic structure to the connector configuration. This needs to be a generic conf with a connector ARN and connector params so that you don't have to build something for each connector. All connectors have an ARN, I suspect many more will be coming.

Python users prefer requirements.txt to specifying the dependencies in the function.conf. This is a standard way of declaring dependencies and GGP needs to switch to this.

Need to add pip to the container

No dependencies get installed, throws a bunch of exceptions and even Python doesn't get installed so the core fails to start.

Being OCD I would like my account cleaned up of any resources that were created during the lab(s) that are not cleaned up when deleting the cloud formation stack(s).

While there exists support for creating a Greengrass Docker container it is not utilizing the official AWS Greengrass Docker containers. Those containers can be found in the Running AWS IoT Greengrass in a Docker Container documentation.

We should migrate to these Docker containers so we can support running Greengrass on Linux, Mac OS, and Windows.

Greengrass SDKs are still at 1.2 even though we've switched to Greengrass Core version 1.7.0. These should be updated soon.

If the SSH connection to an EC2 instance fails due to the wrong SSH key being used an infinite loop occurs where the provisioner just reports "Auth fail" forever. This needs to bail out in the normal timeout period or it needs to fail immediately.

Before the project gets much (any?) larger we should switch to the most recent Java SDKs

The library that we are using (docker-java) is showing signs of dying. We should switch to a library that is more actively maintained like Spotify's Docker library.

If no SSH keys are found it doesn't make sense for GGP to launch an EC2 instance. It should detect that the keys are missing and then refuse to launch the instance.

EC2 launch only launches X86 instances, it should support ARM at least so we can do more testing

CI will be nice once we get some pull requests in

It can take a long time to pull down Gradle depending on network conditions. This could be sped up in Docker by:

• Installing SDK man - curl -s "https://get.sdkman.io" | bash
• Installing the latest Gradle version in the container - sdk install gradle 5.4.1

--ec2-launch saves a lot of time when working with EC2 instances. This needs to support local systems too so that bootstrapping and rebuilding systems is faster.

The provisioner's bootstrap script checks to see if node is present by executing which node. If that executable is present it doesn't check to see if the nodejs6.10 symlink is present.

Instead of which node it should really do which nodejs6.10.

For example, if Gradle fails to download due to CDN issues an exception is thrown but the provisioner hangs forever.

Dagger just isn't worth it anymore

Add support for secrets, also defining the secrets via secrets manager.

On systems that need to use the no isolation the default isolation mode must be set. Otherwise the core will fail to start.

Currently it is not immediately clear how the provisioner should be used with maven based functions. Looking at BasicMavenBuilder, it seems to be very opinionated but gives little information in the way of javadocs or other documentation.

I'm sure we can live with the hard coded values for now, but it should be clear what files are expected where. Maybe just a simple sample directory structure with explanations?

.
├── deployments
│   ├── deployment.defaults.conf
│   ├── function.defaults.conf
│   └── my-function.conf
├── foundation
│   ├── 1-I-wonder
│   ├── 2-what-should
│   └── 3-be-here
└── functions
    └── my-function
        └── function.conf

See exception below, looks like there may need to be retries here.

software.amazon.awssdk.services.ec2.model.Ec2Exception: The instance ID 'i-05071b42256ca7e1d' does not exist (Service: Ec2, Status Code: 400, Request ID: e49fc004-c7c1-4bf8-b810-b8e4e69b229e)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleErrorResponse(HandleResponseStage.java:115)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleResponse(HandleResponseStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:58)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:64)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.ec2.DefaultEc2Client.describeInstances(DefaultEc2Client.java:9429)
        at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:718)
        at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:58)
        at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.executeOrDisplayHelp(Operation.java:34)
        at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.lambda$execute$1(Operation.java:22)
        at io.vavr.control.Try.onSuccess(Try.java:648)
        at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.execute(Operation.java:22)
        at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$null$2(AwsGreengrassProvisioner.java:83)
        at java.util.Optional.map(Optional.java:215)
        at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$run$da6d192b$1(AwsGreengrassProvisioner.java:83)
        at io.vavr.control.Try.of(Try.java:75)
        at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.run(AwsGreengrassProvisioner.java:78)
        at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.runProvisioner(AwsGreengrassProvisioner.java:56)
        at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$main$249538b4$1(AwsGreengrassProvisioner.java:30)
        at io.vavr.control.Try.of(Try.java:75)
        at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.main(AwsGreengrassProvisioner.java:30)

This exception is preventing Java functions from starting in Greengrass 1.7.1 today:

Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/services/lambda/runtime/LambdaLogger

Add support for Greengrass HSI and that support should work with Device Tester integration as well

Device Tester for GG doesn't support 1.8.1 as of today so this is on hold.

See: https://docs.aws.amazon.com/greengrass/latest/developerguide/device-tester-for-greengrass-ug.html#dev-test-versions

Some operating systems obviously don't support systemd. There must be an option to turn it off.

Device Tester support doesn't work in Docker. Errors indicate missing credentials.

The bootstrap script should detect when cgroups are not enabled and enable them.

It appears that a check to see if a reboot was required was put in the wrong place which broke this test.

While testing on Cloud9 I noticed that pip dependency installations fail. It appears that the system has the --user option hard coded so it needs to be disabled.