The agent always starts in Java debug mode, with a debug port open.

Once you managed to fire up a local agent, and reboot the system. The container is gone and not restarted automatically.

Potential Null Dereference Path

In file: CommandShellExecutor.java there is the following code segment

public static <V, E> CommandShellResultSet<V, E> executeCommand(String command, Function<CommandShellResultSet<List<String>, List<String>>, CommandShellResultSet<V, E>> mapper) {
        return executeCommand(command).map(mapper);
}

which invokes another method executeCommand

public static CommandShellResultSet<List<String>, List<String>> executeCommand(String command) {
        String[] fullCommand = computeCommand(command);
        return execute(fullCommand);
}

which invokes execute

private static CommandShellResultSet<List<String>, List<String>> execute(String[] fullCommand) {
        CommandShellResultSet<List<String>, List<String>> resultSet = null;
        try {
            Process process = Runtime.getRuntime().exec(fullCommand);
            List<String> value = readOutput(process, Process::getInputStream);
            List<String> errors = readOutput(process, Process::getErrorStream);
            resultSet = new CommandShellResultSet<>(value, errors);
        } catch (IOException e) {
            LoggingService.logError(MODULE_NAME, e.getMessage(), e);
        }
        return resultSet;
}

this method initialize the resultSet variable with null value and only assign any value to it inside ta try-catch block. But before assigning any value to resultSet if IOException occurs due to readOutput or Runtime.getRuntime().exec() then resultSet will be returned as null which can later lead to NullPointerException in the very first method due to null.map(mapper)

Possible Workaround

One possible workaround is always check the returned value of execute() like

public static <V, E> CommandShellResultSet<V, E> executeCommand(String command, Function<CommandShellResultSet<List<String>, List<String>>, CommandShellResultSet<V, E>> mapper) {
        CommandShellResultSet<List<String>, List<String>> resultSet = executeCommand(command);
        if(resultSet != null){
            return resultSet.map(mapper);
        }else{
            // return something appropriate
        }
}

another workaround can be use of try-catch which is generally not recommended as a best practice

public static <V, E> CommandShellResultSet<V, E> executeCommand(String command, Function<CommandShellResultSet<List<String>, List<String>>, CommandShellResultSet<V, E>> mapper) {
        try{
            return executeCommand(command).map(mapper);
        }catch(NullPointerException e){
            // do something appropriate
        }finally{
            // if needed
        }
}

as the executeCommand method was used in the several places in the codebase so, if any of the places this scenario arises which can lead to an unexpected behavior or crash the program.

Fix from triage team!!!

We have mentioned to return something appropriate because of lack of knowledge of the codebase, it is not possible to know which value should be the appropriate value to return in case of any null vlaue or which fix type is more appropriate for this codebase.

Sponsorship and Support:

This work is done by the security researchers from OpenRefactory and is supported by the Open Source Security Foundation (OpenSSF): Project Alpha-Omega. Alpha-Omega is a project partnering with open source software project maintainers to systematically find new, as-yet-undiscovered vulnerabilities in open source code - and get them fixed - to improve global software supply chain security.

The bug is found by running the iCR tool by OpenRefactory, Inc. and then manually triaging the results.

I when installing the agent I noticed that is still uses an init.d script. Event on a Raspberry Pi, that seems rather outdated.

The Application template contains:

          commands:
            - '/usr/bin/contextBroker'
            - '-fg'
            - '-multiservice'
            - '-ngsiv1Autocast'
            - '-dbhost'
            - 'localhost:2017'
            - '-corsOrigin'
            - '__ALL'

After instanciation and deployement

 iofogctl describe microservice

shows:

    commands:
    - /usr/bin/contextBroker
    - -fg
    - -multiservice
    - -ngsiv1Autocast
    - -dbhost
    - localhost:2017
    - -corsOrigin
    - __ALL

But running container get:

/usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -corsOrigin -dbhost -fg -mult
iservice -ngsiv1Autocast /usr/bin/contextBroker __ALL localhost:2017

The agent requires Java to run. Installing the deb package does not install a JRE. So that running the Agent will fail:

pi@raspberrypi:~/deploy $ sudo /etc/init.d/iofog-agent start
Starting iofog-agent service...
Using Java version  found at ''
/etc/init.d/iofog-agent: 30: /etc/init.d/iofog-agent: java: not found

I attempted to connect ioFog agent and ioFog controller.
However, the ioFog controller does not recognize the ioFog agent.

I checked the log of ioFog agent(/var/log/iofog-agent/iofog-agent.0.log).
In the log, "controller certificate verification failed" was described as follows.

[12/04/2018 11:16:19.022] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:19.022] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:19.024] [INFO] [Message Bus] : STARTING MESSAGE BUS SERVER
[12/04/2018 11:16:19.024] [INFO] [Message Bus Server] : starting...
[12/04/2018 11:16:19.553] [INFO] [Message Bus] : MESSAGE BUS SERVER STARTED
[12/04/2018 11:16:19.586] [INFO] [LOCAL API ] : Local Api Instantiated
[12/04/2018 11:16:19.592] [INFO] [Local API] : Container configuration retrieved
[12/04/2018 11:16:19.601] [INFO] [Supervisor] : started
[12/04/2018 11:16:19.950] [INFO] [Local API] : Local api server started at port: 54321

[12/04/2018 11:16:23.097] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:24.587] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:29.014] [INFO] [Process Manager] : monitoring containers
[12/04/2018 11:16:29.101] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:29.591] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:29.951] [INFO] [Local API] : Initiating control signals for unacknowledged signals
[12/04/2018 11:16:29.953] [INFO] [Local API] : Initiating message sending for the unacknowledged messages
[12/04/2018 11:16:34.592] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:35.104] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:39.128] [INFO] [Process Manager] : monitoring containers
[12/04/2018 11:16:39.593] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:39.951] [INFO] [Local API] : Initiating control signals for unacknowledged signals
[12/04/2018 11:16:39.953] [INFO] [Local API] : Initiating message sending for the unacknowledged messages
[12/04/2018 11:16:41.107] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:44.594] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:47.110] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:49.139] [INFO] [Process Manager] : monitoring containers
[12/04/2018 11:16:49.595] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:49.952] [INFO] [Local API] : Initiating control signals for unacknowledged signals
[12/04/2018 11:16:49.954] [INFO] [Local API] : Initiating message sending for the unacknowledged messages
[12/04/2018 11:16:52.139] [INFO] [Field Agent] : post IOFog status
[12/04/2018 11:16:52.140] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:52.141] [WARN] [Field Agent] : controller certificate verification failed
[12/04/2018 11:16:52.141] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:52.141] [INFO] [Field Agent] : start posting IOFog status

My ioFog agent setting is below.

$ sudo iofog-agent info
[sudo] password for test:
Iofog UUID : not provisioned
IP Address : 192.168.11.19
Network Interface : ens33
Developer's Mode : on
ioFog Controller : http://192.168.11.10:51121/api/v3/
ioFog Certificate : /etc/iofog-agent/cert.crt
Docker URL : unix:///var/run/docker.sock
Disk Usage Limit : 50.00 GiB
Message Storage Directory : /var/lib/iofog-agent/
Memory RAM Limit : 4096.00 MiB
CPU Usage Limit : 80.00%
Log Disk Limit : 10.00 GiB
Log File Directory : /var/log/iofog-agent/
Log Rolling File Count : 10
Status Update Frequency : 30
Get Changes Frequency : 60
Scan Devices Frequency : 60
Post Diagnostics Frequency : 10
Isolated Docker Containers Mode : off
GPS mode : auto
GPS coordinates(lat,lon) : 35.6906,139.77
Fog type : intel_amd

My ioFog controller setting is below.

$iofog-controller config list
Port: 51121
Home url: https://google.com
Email activation: false
Log files directory: /var/log/iofog-controller
Log files size: 1048576
Dev mode: true

※I am deploying iofog agent and iofog controller in separate virtual machines.

Why is ioFog controller unable to recognize ioFog agent?

I have a ioFog agent running with coredns/coredns:latest(Image id: ead0a4a53df8) containers on it. I deployed a microservice on that agent and then I wanted to delete the microservice and failed. The iofog-agent log gave

[08/12/2023 09:52:04.997] [SEVERE] [PMCM] [Process Manager] : com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.dockerjava.api.model.Capability` from String "all": value not one of declared Enum instance names: [ALL, SYS_BOOT, DAC_OVERRIDE, NET_RAW, BLOCK_SUSPEND, FOWNER, IPC_LOCK, IPC_OWNER, SYS_PACCT, NET_BIND_SERVICE, WAKE_ALARM, FSETID, DAC_READ_SEARCH, SYS_CHROOT, SYS_RAWIO, SYS_ADMIN, KILL, MAC_ADMIN, SYS_RESOURCE, CHOWN, SETPCAP, SYS_PTRACE, NET_ADMIN, SETFCAP, SYS_NICE, LINUX_IMMUTABLE, AUDIT_CONTROL, LEASE, AUDIT_WRITE, SYS_MODULE, MKNOD, SYSLOG, MAC_OVERRIDE, SYS_TIME, SETGID, SETUID, SYS_TTY_CONFIG, NET_BROADCAST]
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 2054] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["HostConfig"]->com.github.dockerjava.api.model.HostConfig["CapDrop"]->java.lang.Object[][0]) - Exception: Error monitoring container - Stack trace: org.eclipse.iofog.exception.AgentSystemException: Error monitoring container
        at org.eclipse.iofog.process_manager.ProcessManager.lambda$new$1(ProcessManager.java:117)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.client.ResponseProcessingException: com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.dockerjava.api.model.Capability` from String "all": value not one of declared Enum instance names: [ALL, SYS_BOOT, DAC_OVERRIDE, NET_RAW, BLOCK_SUSPEND, FOWNER, IPC_LOCK, IPC_OWNER, SYS_PACCT, NET_BIND_SERVICE, WAKE_ALARM, FSETID, DAC_READ_SEARCH, SYS_CHROOT, SYS_RAWIO, SYS_ADMIN, KILL, MAC_ADMIN, SYS_RESOURCE, CHOWN, SETPCAP, SYS_PTRACE, NET_ADMIN, SETFCAP, SYS_NICE, LINUX_IMMUTABLE, AUDIT_CONTROL, LEASE, AUDIT_WRITE, SYS_MODULE, MKNOD, SYSLOG, MAC_OVERRIDE, SYS_TIME, SETGID, SETUID, SYS_TTY_CONFIG, NET_BROADCAST]
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 2054] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["HostConfig"]->com.github.dockerjava.api.model.HostConfig["CapDrop"]->java.lang.Object[][0])
        at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:873)
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
        at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:428)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:324)
        at com.github.dockerjava.jaxrs.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:30)
        at com.github.dockerjava.jaxrs.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:13)
        at com.github.dockerjava.jaxrs.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:23)
        at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
        at com.github.dockerjava.core.command.InspectContainerCmdImpl.exec(InspectContainerCmdImpl.java:52)
        at org.eclipse.iofog.process_manager.DockerUtil.lambda$getRunningContainers$2(DockerUtil.java:363)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.eclipse.iofog.process_manager.DockerUtil.getRunningContainers(DockerUtil.java:367)
        at org.eclipse.iofog.process_manager.ProcessManager.deleteRemainingMicroservices(ProcessManager.java:199)
        at org.eclipse.iofog.process_manager.ProcessManager.lambda$new$1(ProcessManager.java:114)
        ... 1 more

docker inspect with the coredns container gave this

    "HostConfig": {
        "CapAdd": [
            "NET_BIND_SERVICE"
        ],
        "CapDrop": [
            "all"
        ],

and my docker version on that agent is

Client:
 Version:           20.10.25
 API version:       1.40
 Go version:        go1.18.1
 Git commit:        20.10.25-0ubuntu1~20.04.1
 Built:             Fri Jul 14 22:00:45 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.25-0ubuntu1~20.04.1
  Built:            Thu Jun 29 21:55:06 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~20.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

I do not know how to fix it. My work depends on ioFog deploying and deleting microservices properly and hope it can be fixed.

The awk command used for determining the agent's CPU utilization assumes the output to use a . as the decimal limiter. However, on a system using European (here: German) locale, the value output uses a , as decimal limiter which leads to a failure to parse the value using Java's Float.parseFloat() method.
I can see two options to fix this:

1. Prepend the awk command with a LC_NUMERIC=en_US.UTF-8 && ` in order to use a decimal point instead of a comma when formatting floats, or
2. Prevent the problem by not outputting a float in the first place. I doubt the value of knowing that the CPU utilization is 56.345 instead of 56 percent. Thus, the awk command could be changed to format the output like this:

   grep 'cpu' /proc/stat | awk '{usage=($2+$3+$4)*100/($2+$3+$4+$5+$6+$7+$8+$9)} END {printf (%d, usage)}'

   And then parse the value using Integer.parseInt() ...