aws-robotics / cloudwatchlogs-ros1 Goto Github PK

View Code? Open in Web Editor NEW

16.0 16.0 11.0 225 KB

ROS packages for facilitating the use of AWS cloud services.

License: Apache License 2.0

CMake 6.94% C++ 93.06%

cloudwatchlogs-ros1's People

Contributors

Stargazers

Watchers

Forkers

hyandell juanrh ross-desmond xabxx aalon ryanewel jerwallace fran6co ojasjoshi jikawa-az

cloudwatchlogs-ros1's Issues

Possible to send logs by Node

Hi,

Quick question - is it possible to send logs by node and/or instead of topic?

Thanks

Mark

Integration with CloudWatch Insights

Hi,

I've had the CloudWatch logger working well now for a few weeks.

Just looking at what I can do to integrate with Insights.

At the moment my Robots' nodes publish messages using ROS_INFO("blah blah blah");

One of the messages it publishes is:

"Redocking Attempts: 0"

Which is a simple String with an integer, basically indicating if the robot has had problems docking.

What I'd like to use Insights for is to add a filter to my selection such that if the redocking attempts are > 0 I get and alert.

Current Filter Looks Like:

fields @timestamp, @message
| filter @message like /Redocking Attempts/
| sort @timestamp desc
| limit 20

Any ideas re: how I could achieve this? Will I have to create a JSON message instead (Is there a JSON message API readily available?)

Thanks & Regards

Mark

Issue Building Cloudwatch_Logger

Hi,

Just upgrading and rebuilding by Robots OSs. Bumped into an issue re: log output below building the CloudwatchLogger. Not yet found a solution. Do you have any ideas?

This is ROS 1 on Ubuntu 20.04 ROS version Noetic

Many Thanks

Mark

`==> Processing catkin package: 'cloudwatch_logger'
==> Building with env: '/home/mark/ros_ws/install_isolated/env.sh'
Makefile exists, skipping explicit cmake invocation...
==> make cmake_check_build_system in '/home/mark/ros_ws/build_isolated/cloudwatch_logger'
==> make -j1 in '/home/mark/ros_ws/build_isolated/cloudwatch_logger'
[ 60%] Built target cloudwatch_logger_lib
[ 80%] Building CXX object CMakeFiles/cloudwatch_logger.dir/src/main.cpp.o
In file included from /usr/include/x86_64-linux-gnu/c++/9/bits/c++allocator.h:33,
from /usr/include/c++/9/bits/allocator.h:46,
from /usr/include/c++/9/memory:63,
from /home/mark/ros_ws/install_isolated/include/aws/core/utils/memory/AWSMemory.h:12,
from /home/mark/ros_ws/install_isolated/include/aws/core/utils/memory/stl/AWSAllocator.h:11,
from /home/mark/ros_ws/install_isolated/include/aws/core/utils/memory/stl/AWSString.h:10,
from /home/mark/ros_ws/install_isolated/include/aws/core/utils/logging/LogLevel.h:10,
from /home/mark/ros_ws/install_isolated/include/aws/core/Aws.h:7,
from /home/mark/ros_ws/src/cloudwatchlogs-ros1/cloudwatch_logger/src/main.cpp:16:
/usr/include/c++/9/ext/new_allocator.h: In instantiation of ‘void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = Aws::Utils::Logging::AWSROSLogger; _Args = {}; _Tp = Aws::Utils::Logging::AWSROSLogger]’:
/usr/include/c++/9/bits/alloc_traits.h:226:6: required by substitution of ‘template<class _Alloc2, class> static std::true_type std::allocator_traits<Aws::AllocatorAws::Utils::Logging::AWSROSLogger >::__construct_helperAws::Utils::Logging::AWSROSLogger::__test<_Alloc2, >(int) [with _Alloc2 = Aws::AllocatorAws::Utils::Logging::AWSROSLogger; = ]’
/usr/include/c++/9/bits/alloc_traits.h:233:40: required from ‘struct std::allocator_traits<Aws::AllocatorAws::Utils::Logging::AWSROSLogger >::__construct_helperAws::Utils::Logging::AWSROSLogger’
/usr/include/c++/9/bits/alloc_traits.h:250:2: required by substitution of ‘template<class _Tp, class ... _Args> static std::_Require<std::_and<std::_not<typename std::allocator_traits<Aws::AllocatorAws::Utils::Logging::AWSROSLogger >::__construct_helper<_Tp, _Args>::type>, std::is_constructible<_Tp, _Args ...> > > std::allocator_traits<Aws::AllocatorAws::Utils::Logging::AWSROSLogger >::_S_construct<_Tp, _Args ...>(Aws::AllocatorAws::Utils::Logging::AWSROSLogger&, _Tp*, _Args&& ...) [with _Tp = Aws::Utils::Logging::AWSROSLogger; _Args = {}]’
/usr/include/c++/9/bits/alloc_traits.h:350:26: required by substitution of ‘template<class _Tp, class ... _Args> static decltype (std::allocator_traits<Aws::AllocatorAws::Utils::Logging::AWSROSLogger >::_S_construct(__a, __p, (forward<_Args>)(std::allocator_traits::construct::__args)...)) std::allocator_traits<Aws::AllocatorAws::Utils::Logging::AWSROSLogger >::construct<_Tp, _Args ...>(Aws::AllocatorAws::Utils::Logging::AWSROSLogger&, _Tp*, _Args&& ...) [with _Tp = Aws::Utils::Logging::AWSROSLogger; _Args = {}]’
/usr/include/c++/9/bits/shared_ptr_base.h:548:39: required from ‘std::_Sp_counted_ptr_inplace<_Tp, _Alloc, _Lp>::_Sp_counted_ptr_inplace(_Alloc, _Args&& ...) [with _Args = {}; _Tp = Aws::Utils::Logging::AWSROSLogger; _Alloc = Aws::AllocatorAws::Utils::Logging::AWSROSLogger; __gnu_cxx::_Lock_policy _Lp = __gnu_cxx::_S_atomic]’
/usr/include/c++/9/bits/shared_ptr_base.h:679:16: required from ‘std::__shared_count<_Lp>::__shared_count(_Tp*&, std::_Sp_alloc_shared_tag<_Alloc>, _Args&& ...) [with _Tp = Aws::Utils::Logging::AWSROSLogger; _Alloc = Aws::AllocatorAws::Utils::Logging::AWSROSLogger; _Args = {}; __gnu_cxx::_Lock_policy _Lp = __gnu_cxx::_S_atomic]’
/usr/include/c++/9/bits/shared_ptr_base.h:1344:71: required from ‘std::__shared_ptr<_Tp, _Lp>::__shared_ptr(std::_Sp_alloc_shared_tag<_Tp>, _Args&& ...) [with _Alloc = Aws::AllocatorAws::Utils::Logging::AWSROSLogger; _Args = {}; _Tp = Aws::Utils::Logging::AWSROSLogger; __gnu_cxx::_Lock_policy _Lp = __gnu_cxx::_S_atomic]’
/usr/include/c++/9/bits/shared_ptr.h:359:59: required from ‘std::shared_ptr<_Tp>::shared_ptr(std::_Sp_alloc_shared_tag<_Tp>, _Args&& ...) [with _Alloc = Aws::AllocatorAws::Utils::Logging::AWSROSLogger; _Args = {}; _Tp = Aws::Utils::Logging::AWSROSLogger]’
/usr/include/c++/9/bits/shared_ptr.h:701:14: required from ‘std::shared_ptr<_Tp> std::allocate_shared(const _Alloc&, _Args&& ...) [with _Tp = Aws::Utils::Logging::AWSROSLogger; _Alloc = Aws::AllocatorAws::Utils::Logging::AWSROSLogger; _Args = {}]’
/home/mark/ros_ws/install_isolated/include/aws/core/utils/memory/stl/AWSAllocator.h:96:58: required from ‘std::shared_ptr<_Tp> Aws::MakeShared(const char*, ArgTypes&& ...) [with T = Aws::Utils::Logging::AWSROSLogger; ArgTypes = {}]’
/home/mark/ros_ws/src/cloudwatchlogs-ros1/cloudwatch_logger/src/main.cpp:36:65: required from here
/usr/include/c++/9/ext/new_allocator.h:145:20: error: invalid new-expression of abstract class type ‘Aws::Utils::Logging::AWSROSLogger’
145 | noexcept(noexcept(::new((void *)__p)
| ^~~~~~~~~~~~~~~~~~
146 | _Up(std::forward<_Args>(__args)...)))
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/mark/ros_ws/src/cloudwatchlogs-ros1/cloudwatch_logger/include/cloudwatch_logger/log_node.h:19,
from /home/mark/ros_ws/src/cloudwatchlogs-ros1/cloudwatch_logger/src/main.cpp:19:
/home/mark/ros_ws/install_isolated/include/aws_ros1_common/sdk_utils/logging/aws_ros_logger.h:27:7: note: because the following virtual functions are pure within ‘Aws::Utils::Logging::AWSROSLogger’:
27 | class AWSROSLogger : public AWSLogSystem
| ^~~~~~~~~~~~
In file included from /home/mark/ros_ws/install_isolated/include/aws/core/Aws.h:8,
from /home/mark/ros_ws/src/cloudwatchlogs-ros1/cloudwatch_logger/src/main.cpp:16:
/home/mark/ros_ws/install_isolated/include/aws/core/utils/logging/LogSystemInterface.h:46:30: note: ‘virtual void Aws::Utils::Logging::LogSystemInterface::Flush()’
46 | virtual void Flush() = 0;
| ^~~~~
make[2]: *** [CMakeFiles/cloudwatch_logger.dir/build.make:63: CMakeFiles/cloudwatch_logger.dir/src/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1155: CMakeFiles/cloudwatch_logger.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
<== Failed to process package 'cloudwatch_logger':
Command '['/home/mark/ros_ws/install_isolated/env.sh', 'make', '-j1']' returned non-zero exit status 2.

Reproduce this error by running:
==> cd /home/mark/ros_ws/build_isolated/cloudwatch_logger && /home/mark/ros_ws/install_isolated/env.sh make -j1

Command failed, exiting.`

CloudWatch Logger node generating a lot of logging info

Reviewing logs that are coming in and finding the cloudwatch_logger node itself generating a ton of INFO messages that seems to be consistent with the publish_frequency. The original 5s interval hit a API limit compared to the little logs our application sends.

Here is a logs with a 60s frequency with min_log_verbosity set to INFO. We are concious about uncessary logs since these are shipped over cellular.

Is it possible to suppress this logging from the node vs bumping our logging levels within our entire application? Or would it be better to set sub_to_rosout to False and start pushing logs to a dedicated topic?

Any guidance appreciated!

Ensure Log Batch Doesn't Span More Than 24 Hours

I've noticed that occasionally the cloudwatch node crashes when it tries to batch old logs spanning 24 hour time spans. We should handle this more gracefully and output an error to the user about which logs got dropped from the batch.

[ERROR] [1576699115.494884036]: [AWSClient] HTTP response code: 400
Exception name: InvalidParameterException
Error message: The batch of log events in a single PutLogEvents request cannot span more than 24 hours.
5 response headers:
connection : close
content-length : 139
content-type : application/x-amz-json-1.1
date : Wed, 18 Dec 2019 19:58:35 GMT
x-amzn-requestid : {omitted}

[ERROR] [1576699115.496820436]: [SendLogsRequest] Send log request failed due to: The batch of log events in a single PutLogEvents request cannot span more than 24 hours., with error code: 131

[ERROR] [1576699115.496979688]: [SendLogsToCloudWatch] Failed to send to CloudWatch in Log Group: robot_app Log Stream: {omitted} with error code: 1
[cloudwatch_logger-18] process has died [pid 22428, exit code -11, cmd /catkin_ws/devel/lib/cloudwatch_logger/cloudwatch_logger __name:=cloudwatch_logger __log:=/root/.ros/log/b7895c34-21d0-11ea-a9e1-d0c5d3afb37d/cloudwatch_logger-18.log].
log file: /root/.ros/log/b7895c34-21d0-11ea-a9e1-d0c5d3afb37d/cloudwatch_logger-18*.log

Build Issue

Hi,

I'm had this build before - those about a year back. Just refreshing and rebuilding now I'm running into this error:

/opt/ros/kinetic/include/aws_ros1_common/sdk_utils/ros1_node_parameter_reader.h:34:12: error: ‘Aws::AwsError Aws::Client::Ros1NodeParameterReader::ReadMap(const char*, std::map<std::__cxx11::basic_string, std::__cxx11::basic_string >&) const’ marked ‘override’, but does not override
AwsError ReadMap(const char * name, std::map<std::string, std::string> & out)

Any idea what this cause is?

Thanks

Mark

Can't start CW Log node - authentication issues

Hi,

I am having issue just getting the CW Log node to push logs. It seems to be related to an authentication issue and not clear where the issue lies.

Here is what I get when running the sample application:

... logging to /home/ubuntu/.ros/log/db0c15d6-531b-11e9-9e7d-4061864b8ae7/roslaunch-ros-11343.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ros:41131/

SUMMARY
========

PARAMETERS
 * /cloudwatch_logger/aws_client_configuration/connect_timeout_ms: 9000
 * /cloudwatch_logger/aws_client_configuration/max_retries: 10
 * /cloudwatch_logger/aws_client_configuration/region: us-west-2
 * /cloudwatch_logger/aws_client_configuration/request_timeout_ms: 9000
 * /cloudwatch_logger/log_group_name: robot_application...
 * /cloudwatch_logger/log_stream_name: device_name
 * /cloudwatch_logger/min_log_verbosity: DEBUG
 * /cloudwatch_logger/publish_frequency: 5.0
 * /cloudwatch_logger/sub_to_rosout: True
 * /cloudwatch_logger/topics: []
 * /rosdistro: kinetic
 * /rosversion: 1.12.14
NODES
  /
    cloudwatch_logger (cloudwatch_logger/cloudwatch_logger)

auto-starting new master
process[master]: started with pid [11353]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to db0c15d6-531b-11e9-9e7d-4061864b8ae7
process[rosout-1]: started with pid [11366]
started core service [/rosout]
process[cloudwatch_logger-2]: started with pid [11379]
[ INFO] [1553971472.289850374]: [main] Starting cloudwatch_logger...
[ INFO] [1553971472.298896958]: [ReadPublishFrequency] Publish frequency is set to: 5
[ INFO] [1553971472.300726748]: [ReadLogGroup] Log group name is set to: robot_application_name
[ INFO] [1553971472.302639950]: [ReadLogStream] Log stream name is set to: device_name
[ INFO] [1553971472.304349319]: [ReadSubscribeToRosout] Whether to subscribe to rosout_agg topic is set to: 1
[ INFO] [1553971472.305739801]: [ReadMinLogVerbosity] Log verbosity is set to DEBUG.
[ INFO] [1553971472.305873877]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/ubuntu//.aws/config and using profilePrefix = 1
[ INFO] [1553971472.305916400]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/ubuntu//.aws/credentials and using profilePrefix = 0
[ INFO] [1553971472.305970677]: [ProfileConfigFileAWSCredentialsProvider] Setting provider to read credentials from /home/ubuntu//.aws/credentials for credentials file and /home/ubuntu//.aws/config for the config file , for use with profile
 default
[ INFO] [1553971472.306014492]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/ubuntu//.aws/config and using profilePrefix = 1
[ INFO] [1553971472.306174145]: [Aws::Config::AWSProfileConfigLoader] Successfully reloaded configuration.
[ INFO] [1553971472.306257575]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/ubuntu//.aws/credentials and using profilePrefix = 0
[ INFO] [1553971472.306334625]: [Aws::Config::AWSProfileConfigLoader] Successfully reloaded configuration.
[ INFO] [1553971472.340198505]: [CurlHttpClient] Initializing Curl library
[ INFO] [1553971472.343481810]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/ubuntu//.aws/config and using profilePrefix = 1
[ INFO] [1553971472.343534508]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/ubuntu//.aws/credentials and using profilePrefix = 0
[ INFO] [1553971472.343584441]: [ProfileConfigFileAWSCredentialsProvider] Setting provider to read credentials from /home/ubuntu//.aws/credentials for credentials file and /home/ubuntu//.aws/config for the config file , for use with profile
 default
[ INFO] [1553971472.343634195]: [EC2MetadataClient] Creating HttpClient with max connections2 and scheme http
[ INFO] [1553971472.343683503]: [CurlHandleContainer] Initializing CurlHandleContainer with size 2
[ INFO] [1553971472.343728052]: [InstanceProfileCredentialsProvider] Creating Instance with default EC2MetadataClient and refresh rate 300000
[ INFO] [1553971472.343760221]: [DefaultAWSCredentialsProviderChain] Added EC2 metadata service credentials provider to the provider chain.
[ INFO] [1553971472.343878271]: [Aws::Config::AWSProfileConfigLoader] Successfully reloaded configuration.
[ INFO] [1553971472.343917604]: [InstanceProfileCredentialsProvider] Credentials have expired attempting to repull from EC2 Metadata Service.
[ INFO] [1553971472.344109841]: [CurlHandleContainer] Pool grown by 2
[ INFO] [1553971472.344138121]: [CurlHandleContainer] Connection has been released. Continuing.
[ERROR] [1553971473.344781421]: [CurlHttpClient] Curl returned error code 28
[ERROR] [1553971473.344997725]: [EC2MetadataClient] Http request to retrieve credentials failed.
[ INFO] [1553971473.345083534]: [Aws::Config::AWSProfileConfigLoader] Failed to reload configuration.
[ INFO] [1553971473.345322229]: [CurlHandleContainer] Initializing CurlHandleContainer with size 25
[ INFO] [1553971473.345643092]: [StartPublisherThread] Started publisher thread
[ INFO] [1553971473.346427372]: [CurlHandleContainer] Pool grown by 2
[ INFO] [1553971473.346538664]: [CurlHandleContainer] Connection has been released. Continuing.
[ INFO] [1553971473.355820614]: [ReadSubscriberList] Subscribing to rosout_agg
[ INFO] [1553971473.355869571]: [main] Initialized cloudwatch_logger.
[ WARN] [1553971473.742467271]: [AWSErrorMarshaller] Encountered AWSError 'MissingAuthenticationTokenException': Missing Authentication Token
[ERROR] [1553971473.742719541]: [AWSClient] HTTP response code: 400
Exception name: MissingAuthenticationTokenException
Error message: Missing Authentication Token
5 response headers:
connection : close
content-length : 89
content-type : application/x-amz-json-1.1
date : Sat, 30 Mar 2019 18:44:32 GMT
x-amzn-requestid : dc478dc5-531b-11e9-8d7d-df38900c1a74
[ WARN] [1553971473.742868071]: [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[ WARN] [1553971473.743092549]: [CheckLogGroupExists] Request to check if log group named robot_application_name exists failed. Error message: Missing Authentication Token, with error code: 8
[ INFO] [1553971473.743350965]: [CurlHandleContainer] Connection has been released. Continuing.
[ WARN] [1553971474.074042260]: [AWSErrorMarshaller] Encountered AWSError 'MissingAuthenticationTokenException': Missing Authentication Token
[ERROR] [1553971474.074265678]: [AWSClient] HTTP response code: 400
Exception name: MissingAuthenticationTokenException
Error message: Missing Authentication Token
5 response headers:
connection : close
content-length : 89
content-type : application/x-amz-json-1.1
date : Sat, 30 Mar 2019 18:44:33 GMT
x-amzn-requestid : dc7a5e57-531b-11e9-8e3b-df68653f3e8a
[ WARN] [1553971474.074420090]: [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[ERROR] [1553971474.074620496]: [CreateLogGroup] Failed to create Log Group :robot_application_name due to: Missing Authentication Token, with error code: 8
[ERROR] [1553971474.074699993]: [Run] Failed to create log group, retrying ...
[ INFO] [1553971474.074884171]: [CurlHandleContainer] Connection has been released. Continuing.
[ WARN] [1553971474.409767874]: [AWSErrorMarshaller] Encountered AWSError 'MissingAuthenticationTokenException': Missing Authentication Token
[ERROR] [1553971474.409960100]: [AWSClient] HTTP response code: 400
Exception name: MissingAuthenticationTokenException
Error message: Missing Authentication Token
5 response headers:
connection : close
content-length : 89
content-type : application/x-amz-json-1.1
date : Sat, 30 Mar 2019 18:44:33 GMT
x-amzn-requestid : dcad545e-531b-11e9-9df6-01f20a4e62a5

Thinking this was related to a misconfigured IAM policy I am able to create a log group with no issues.

$ aws logs create-log-group --log-group-name testdeleteme

AWS Credential Files

#robomaker
[default]
region = us-east-1

$ cat ~/.aws/credentials 
#aws robomaker creds
[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX

System time is also synchronized

$ timedatectl status                                            
      Local time: Sat 2019-03-30 13:48:19 CDT
  Universal time: Sat 2019-03-30 18:48:19 UTC
        RTC time: Sat 2019-03-30 18:48:20
       Time zone: America/Chicago (CDT, -0500)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

Version

$ aptitude show ros-kinetic-cloudwatch-logger
Package: ros-kinetic-cloudwatch-logger   
State: installed
Automatically installed: no
Version: 1.0.0-0xenial-20190325-063824-0800
Priority: extra
Section: misc
Maintainer: AWS RoboMaker <[email protected]>
Architecture: amd64
Uncompressed Size: 132 k
Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.0), libstdc++6 (>= 5.2), ros-kinetic-aws-common,
         ros-kinetic-aws-ros1-common, ros-kinetic-cloudwatch-logs-common, ros-kinetic-roscpp
Conflicts: ros-kinetic-cloudwatch-logger:i386
Description: CloudWatch Logger node for publishing logs to AWS CloudWatch Logs
 
Homepage: http://wiki.ros.org/cloudwatch_logger

Logger fails to authenticate using config file, but works with envars

I am having a weird behavior where the setup works using AWS envars, but doesn't with a ~/.aws/credentials file.

While using the credentials file I am able to create a log group and push a message via awscli. When launching the ros project getting the following output:

SUMMARY                                                                                                                                                                                                                               [50/303]
========

PARAMETERS
 * /cloudwatch_logger/aws_client_configuration/connect_timeout_ms: 10000
 * /cloudwatch_logger/aws_client_configuration/max_retries: 100
 * /cloudwatch_logger/aws_client_configuration/region: us-east-1
 * /cloudwatch_logger/aws_client_configuration/request_timeout_ms: 60000
 * /cloudwatch_logger/log_group_name: mbbssim
 * /cloudwatch_logger/log_stream_name: 81e63cbe-d7d2-4ba...
 * /cloudwatch_logger/min_log_verbosity: INFO
 * /cloudwatch_logger/publish_frequency: 120
 * /cloudwatch_logger/sub_to_rosout: True
 * /cloudwatch_logger/topics: []
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES
  /
    base (basestation/base)
    cloudwatch_logger (cloudwatch_logger/cloudwatch_logger)

auto-starting new master
process[master]: started with pid [10158]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to fac359f2-a65e-11e9-83c0-cc3d822d9092
process[rosout-1]: started with pid [10171]
started core service [/rosout]
process[base-2]: started with pid [10178]
process[cloudwatch_logger-3]: started with pid [10189]
[ INFO] [1563126248.197236689]: [main] Starting cloudwatch_logger...
[ INFO] [1563126248.204495265]: [ReadPublishFrequency] Publish frequency is set to: 120
[ INFO] [1563126248.205949694]: [ReadLogGroup] Log group name is set to: mbbssim
[ INFO] [1563126248.207294833]: [ReadLogStream] Log stream name is set to: 81e63cbe-d7d2-4bab-4073-aed39f7359bf
[ INFO] [1563126248.209071818]: [ReadSubscribeToRosout] Whether to subscribe to rosout_agg topic is set to: 1
[ INFO] [1563126248.210649525]: [ReadMinLogVerbosity] Log verbosity is set to INFO.
[ INFO] [1563126248.211587951]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/mr337//.aws/config and using profilePrefix = 1
[ INFO] [1563126248.211638603]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/mr337//.aws/credentials and using profilePrefix = 0
[ INFO] [1563126248.211679273]: [ProfileConfigFileAWSCredentialsProvider] Setting provider to read credentials from /home/mr337//.aws/credentials for credentials file and /home/mr337//.aws/config for the config file , for use with profile
 default
[ INFO] [1563126248.211713663]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/mr337//.aws/config and using profilePrefix = 1
[ INFO] [1563126248.211762464]: [Aws::Config::AWSConfigFileProfileConfigLoader] Unable to open config file /home/mr337//.aws/config for reading.
[ INFO] [1563126248.211789842]: [Aws::Config::AWSProfileConfigLoader] Failed to reload configuration.
[ INFO] [1563126248.211818241]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/mr337//.aws/credentials and using profilePrefix = 0
[ INFO] [1563126248.211961798]: [Aws::Config::AWSProfileConfigLoader] Successfully reloaded configuration.
[ INFO] [1563126248.258529704]: [CurlHttpClient] Initializing Curl library
[ INFO] [1563126248.262861790]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/mr337//.aws/config and using profilePrefix = 1
[ INFO] [1563126248.262960796]: [Aws::Config::AWSConfigFileProfileConfigLoader] Unable to open config file /home/mr337//.aws/config for reading.
[ INFO] [1563126248.262997453]: [Aws::Config::AWSProfileConfigLoader] Failed to reload configuration.
[ INFO] [1563126248.263073625]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/mr337//.aws/config and using profilePrefix = 1
[ INFO] [1563126248.263115513]: [Aws::Config::AWSConfigFileProfileConfigLoader] Initializing config loader against fileName /home/mr337//.aws/credentials and using profilePrefix = 0
[ INFO] [1563126248.263165278]: [ProfileConfigFileAWSCredentialsProvider] Setting provider to read credentials from /home/mr337//.aws/credentials for credentials file and /home/mr337//.aws/config for the config file , for use with profile
 default
[ INFO] [1563126248.263226385]: [EC2MetadataClient] Creating AWSHttpResourceClient with max connections2 and scheme http
[ INFO] [1563126248.263260030]: [CurlHandleContainer] Initializing CurlHandleContainer with size 2
[ INFO] [1563126248.263296261]: [InstanceProfileCredentialsProvider] Creating Instance with default EC2MetadataClient and refresh rate 300000
[ INFO] [1563126248.263327261]: [DefaultAWSCredentialsProviderChain] Added EC2 metadata service credentials provider to the provider chain.
[ INFO] [1563126248.263477389]: [Aws::Config::AWSProfileConfigLoader] Successfully reloaded configuration.
[ INFO] [1563126248.263518831]: [InstanceProfileCredentialsProvider] Credentials have expired attempting to repull from EC2 Metadata Service.
[ INFO] [1563126248.263706538]: [CurlHandleContainer] Pool grown by 2
[ INFO] [1563126248.263819304]: [CurlHandleContainer] Connection has been released. Continuing.
.
.
.
.
[ERROR] [1563126249.264218528]: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
[ERROR] [1563126249.264374689]: [EC2MetadataClient] Http request to retrieve credentials failed
[ WARN] [1563126249.264417702]: [EC2MetadataClient] Request failed, now waiting 0 ms before attempting again.
2019-07-14 17:44:09,393 - app:129 - INFO - Starting telem loop...
[ERROR] [1563126250.264838784]: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
[ERROR] [1563126250.264909642]: [EC2MetadataClient] Http request to retrieve credentials failed
[ WARN] [1563126250.264959605]: [EC2MetadataClient] Request failed, now waiting 2000 ms before attempting again.
[ INFO] [1563126252.265243072]: [CurlHandleContainer] Connection has been released. Continuing.
[ERROR] [1563126253.265100463]: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
[ERROR] [1563126253.265193679]: [EC2MetadataClient] Http request to retrieve credentials failed
[ WARN] [1563126253.265244195]: [EC2MetadataClient] Request failed, now waiting 4000 ms before attempting again.
[ INFO] [1563126257.265451658]: [CurlHandleContainer] Connection has been released. Continuing.
[ERROR] [1563126258.265866716]: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
[ERROR] [1563126258.265939342]: [EC2MetadataClient] Http request to retrieve credentials failed
[ WARN] [1563126258.265967638]: [EC2MetadataClient] Request failed, now waiting 8000 ms before attempting again.
[ INFO] [1563126266.266241045]: [CurlHandleContainer] Connection has been released. Continuing.
[ERROR] [1563126267.266728018]: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
[ERROR] [1563126267.266791009]: [EC2MetadataClient] Http request to retrieve credentials failed
[ERROR] [1563126267.266830843]: [EC2MetadataClient] Can not retrive resource /latest/meta-data/iam/security-credentials
[ INFO] [1563126267.266901035]: [Aws::Config::AWSProfileConfigLoader] Failed to reload configuration.
[ INFO] [1563126267.267074366]: [CurlHandleContainer] Initializing CurlHandleContainer with size 25
[ INFO] [1563126267.267318236]: [StartPublisherThread] Started publisher thread
[ INFO] [1563126267.267908661]: [CurlHandleContainer] Pool grown by 2
[ INFO] [1563126267.267987055]: [CurlHandleContainer] Connection has been released. Continuing.
[ INFO] [1563126267.278248063]: [ReadSubscriberList] Subscribing to rosout_agg
[ INFO] [1563126267.278314980]: [main] Initialized cloudwatch_logger.
[ WARN] [1563126267.586875648]: [AWSErrorMarshaller] Encountered AWSError 'MissingAuthenticationTokenException': Missing Authentication Token
[ERROR] [1563126267.586963778]: [AWSClient] HTTP response code: 400
Exception name: MissingAuthenticationTokenException
Error message: Missing Authentication Token
5 response headers:
connection : close
content-length : 89
content-type : application/x-amz-json-1.1
date : Sun, 14 Jul 2019 17:44:27 GMT
x-amzn-requestid : 06a855e0-a65f-11e9-8dca-8506383ed1a4
[ WARN] [1563126267.587026461]: [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[ WARN] [1563126267.587122962]: [CheckLogGroupExists] Request to check if log group named mbbssim exists failed. Error message: Missing Authentication Token, with error code: 8
[ INFO] [1563126267.587242780]: [CurlHandleContainer] Connection has been released. Continuing.
[ WARN] [1563126267.776558040]: [AWSErrorMarshaller] Encountered AWSError 'MissingAuthenticationTokenException': Missing Authentication Token
.
.
.

Export AWS key and secret via envars and everything works fine. I don't mind injecting envars but don't know how to securely do that in the robomaker environment. Setting AWS envars on deployment doesn't sound that attractive.

Test Environment

AWS credential file (chmod 777):

$ cat ~/.aws/credentials 
[default]
aws_access_key_id =  XXXX
aws_secret_access_key = XXX
region=us-east-1

No AWS configuration file to interfere.

Roslaunch params:

 * /cloudwatch_logger/aws_client_configuration/connect_timeout_ms: 10000
 * /cloudwatch_logger/aws_client_configuration/max_retries: 100
 * /cloudwatch_logger/aws_client_configuration/region: us-east-1
 * /cloudwatch_logger/aws_client_configuration/request_timeout_ms: 60000
 * /cloudwatch_logger/log_group_name: mbbssim
 * /cloudwatch_logger/log_stream_name: 81e63cbe-d7d2-4ba...
 * /cloudwatch_logger/min_log_verbosity: INFO
 * /cloudwatch_logger/publish_frequency: 120
 * /cloudwatch_logger/sub_to_rosout: True
 * /cloudwatch_logger/topics: []
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

Installed packages:

$ aptitude show ros-kinetic-cloudwatch-logger ros-kinetic-cloudwatch-logs-common
Package: ros-kinetic-cloudwatch-logger   
New: yes
State: installed
Automatically installed: no
Version: 2.1.0-2xenial-20190624-162117-0800
Priority: extra
Section: misc
Maintainer: AWS RoboMaker <[email protected]>
Architecture: amd64
Uncompressed Size: 151 k
Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.0), libstdc++6 (>= 5.2), ros-kinetic-aws-common, ros-kinetic-aws-ros1-common, ros-kinetic-cloudwatch-logs-common, ros-kinetic-roscpp
Conflicts: ros-kinetic-cloudwatch-logger:i386
Description: CloudWatch Logger node for publishing logs to AWS CloudWatch Logs
 
Homepage: http://wiki.ros.org/cloudwatch_logger


Package: ros-kinetic-cloudwatch-logs-common
New: yes
State: installed
Automatically installed: yes
Version: 1.0.2-1xenial-20190624-161838-0800
Priority: extra
Section: misc
Maintainer: AWS RoboMaker <[email protected]>
Architecture: amd64
Uncompressed Size: 151 k
Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.0), libstdc++6 (>= 5.2), ros-kinetic-aws-common
Conflicts: ros-kinetic-cloudwatch-logs-common:i386
Description: AWS CloudWatch management library used by ROS1/2 node to publish logs to CloudWatch service

Feature Flag to turn off topic listing for a node

For some nodes that are subscribed to a lot of topic can cause a log message to be filled with a lot of unnecessary information.

158.352000000 INFO [node name: /cloud] [topics: /waypoint/parameter_updates, /rosout, /waypoint/parameter_descriptions, /cloud/parameter_updates, /steering/pid/parameter_descriptions, /autopilot/parameter_descriptions, /lidar/parameter_descriptions, /vel, /steering/pid/parameter_updates, /safety_halt, /waypoint, /autosteer/parameter_descriptions, /cloud/parameter_descriptions, /hal/parameter_descriptions, /autosteer/parameter_updates, /clock, /autopilot/parameter_updates, /lidar/parameter_updates, /hal/parameter_updates] Valid Config

Would it be possible to have a flag to turn the [topics: ...] information off? Would help to clean up the logs and reduce cellular bandwidth.

AWS RoboMaker Notice

Dear AWS RoboMaker Customer,

Thank you for using AWS RoboMaker. AWS RoboMaker extends the most widely used open-source robotics software framework, Robot Operating System (ROS), developed and maintained by the Open Source Robotics Foundation and Open Robotics (https://www.openrobotics.org/).

On March 18th 2020, Open Robotics released an updated version of ROS Melodic and later discovered a bug via ros/ros_comm#1913. This issue caused the /rosout and /rosout_agg topics to not output any messages and impacts all ROS applications using the updated ROS Melodic version, including those running in AWS RoboMaker simulation. In addition, the AWS RoboMaker CloudWatch extension emits output messages from /rosout and /rosout_agg to Amazon CloudWatch Logs so one can easily review and analyze these messages. If your application uses the impacted version of ROS Melodic and you use the RoboMaker CloudWatch extension, you will not see any /rosout or /rosout_agg messages in Amazon CloudWatch Logs because of the reported issue. Open Robotics has identified a fix for this issue and expect to release a fix to ROS Melodic in the next few days. We will closely monitor the release of the fix and keep you updated.

If you have any questions, please reach out to us by opening an AWS support case.

Thank you

cloudwatch_logger node not working

repro steps
on a mac or linux machine,

docker pull ros:kinetic-ros-core
docker run -it \
    --network host \
    -v "/tmp/.X11-unix:/tmp/.X11-unix" \
    -v "${HOME}:/opt/${USER}" \
    -v "${HOME}/.gitconfig:/root/.gitconfig" \
    --name=integrationtest_cloudwatchlogs \
    --privileged \
    ros:kinetic-ros-core	    
    bash

at this point you should be inside the docker container
now, inside the docker container, run

#install pip3
apt install -y python3-pip python-rosinstall
#install aws cli
pip3 install --upgrade --user awscli
# make sure awscli is in your path
aws --version
# (optional) if cannot find awscli in current path, try to look for it, my path is
# export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# configure aws creds, make sure to set the region
aws configure

source /opt/ros/kinetic/setup.bash
apt-get update
apt-get install -y ros-$ROS_DISTRO-cloudwatch-logger
roslaunch cloudwatch_logger cloudwatch_logger.launch

I seeing error messages:

root@linuxkit-025000000001:~/test# roslaunch cloudwatch_logger cloudwatch_logger.launch ... logging to /root/.ros/log/126497be-8c70-11e9-9aa8-025000000001/roslaunch-linuxkit-025000000001-4288.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://localhost:33841/

SUMMARY
========

PARAMETERS
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES
  /
    cloudwatch_logger (cloudwatch_logger/cloudwatch_logger)

ROS_MASTER_URI=http://localhost:11311

process[cloudwatch_logger-1]: started with pid [4305]
[ERROR] [1560274910.439882900]: [CurlHttpClient] Curl returned error code 6 - Couldn't resolve host name
[ERROR] [1560274910.440286100]: [AWSClient] HTTP response code: 31306048
Exception name: 
Error message: Unable to connect to endpoint
0 response headers:

Throttling Exception

Hi,

I've changed the way my robots work. Basically CloudWatch is now running all the time. I suspect it's not the amount of data getting uploaded (given the error) but the frequency.

I've changed publish_frequency from the default 5 sec to 120 but still after a while this error gets thrown and I can't upload more data.

Any suggestions?

Thanks

Mark

[ERROR] [1594043186.700089703]: [AWSClient] HTTP response code: 400
Exception name: ThrottlingException
Error message: Rate exceeded
5 response headers:
connection : close
content-length : 58
content-type : application/x-amz-json-1.1
date : Mon, 06 Jul 2020 13:46:26 GMT
[ WARN] [1594043186.700169487]: [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[ERROR] [1594043186.700320532]: [SendLogsRequest] Send log request failed due to: Rate exceeded, with error code: 13
[ERROR] [1594043186.700381692]: [SendLogsToCloudWatch] Failed to send to CloudWatch in Log Group: mxnet_robotics Log Stream: mxnet_kobuki_robot_log_stream with error code: 1
[ WARN] [1594043186.700663183]: [SendLogs] Unable to send logs to CloudWatch, retrying ...

Request cannot span more than 24 hours Error

I am starting to see odd errors in our logs from CloudWatch Log node when running the application live.

[ WARN] [1593087736.376730661, 30.901000000]: [AWSErrorMarshaller] Encountered AWSError 'InvalidParameterException': T
he batch of log events in a single PutLogEvents request cannot span more than 24 hours.
[ERROR] [1593087736.376854123, 30.901000000]: [AWSClient] HTTP response code: 400
Exception name: InvalidParameterException
Error message: The batch of log events in a single PutLogEvents request cannot span more than 24 hours.
5 response headers:
connection : close
content-length : 139
content-type : application/x-amz-json-1.1
date : Thu, 25 Jun 2020 12:22:15 GMT
x-amzn-requestid : 6e2c66d4-baf8-48b8-af82-bb56bad020f0
[ WARN] [1593087736.376925890, 30.901000000]: [AWSClient] If the signature check failed. This could be because of a ti
me skew. Attempting to adjust the signer.
[ERROR] [1593087736.377026834, 30.901000000]: [SendLogsRequest] Send log request failed due to: The batch of log event
s in a single PutLogEvents request cannot span more than 24 hours., with error code: 131
[ERROR] [1593087736.377057525, 30.901000000]: [SendLogsToCloudWatch] Failed to send to CloudWatch in Log Group: ab200s
im Log Stream: 4a54bac8-efb7-47a4-9ee5-493d928ae446 with error code: 1

What is really odd is after this error we still get application shipped to CloudWatch and have been using them for weeks.

Things I have checked:

The timestamp is consistent with the epoch time, no skewed system time
Latest updates for the package

Using Version: 2.3.1-1xenial-20200330-153719+0000

add diagnostics support

diagnostics is a ROS diagnostics package.
We publish CPU usage,actuator temperature etc...
So, it is important to support this message type.

CloudWatcher Failing To Start

Hi,

I've been testing cloudwatcher. I updated the sample_configuration.yaml to a custom log_stream_name

However I get the error messages below:

[ERROR] [1543682968.675952432]: [AWSClient] HTTP response code: 400
Exception name: ResourceAlreadyExistsException
Error message: The specified log group already exists
5 response headers:
connection : close
content-length : 94
content-type : application/x-amz-json-1.1
date : Sat, 01 Dec 2018 16:49:28 GMT
x-amzn-requestid : 116c39c3-f589-11e8-be76-d3e7ff879f8f
[ERROR] [1543682968.676457696]: [CreateLogGroup] Failed to create Log Group :robot_application_name due to: The specified log group already exists, with error code: 135
[ERROR] [1543682968.790863450]: [AWSClient] HTTP response code: 400
Exception name: ResourceAlreadyExistsException
Error message: The specified log stream already exists
5 response headers:
connection : close
content-length : 95
content-type : application/x-amz-json-1.1
date : Sat, 01 Dec 2018 16:49:27 GMT
x-amzn-requestid : 117c8dd5-f589-11e8-bd16-65b3e1494de1
[ERROR] [1543682968.791192148]: [CreateLogStream] Failed to create Log Stream :mxnet_kobuki_robot_log_stream in Log Group :robot_application_name due to: The specified log stream already exists, with error code: 135

According to the config file - it will try to create the log stream if it doesn't exist. Although the error messages seem to indicate otherwise.

How can I resolve this? And where should I expect the log stream to appear - I'm assuming eu-west-2 (I updated this config to reflect this.

Thanks and Regards

Mark

Test

test

Support std_msgs/String as logging message to CloudWatch

It'd be nice to use cloudwatch_logger for logging raw strings into CloudWatch without using rosgraph_msgs/Log. A use case is pushing JSON strings for easier use with CloudWatch Insights. It also is more general and consistent with the CloudWatch Logs API.

AWS CW API Throttle Issue

Hi,

I have been running into an issue with hitting the API limits and throttled back. I thought the issue was related to #20 generating a lot of log entries, but that has been fixed and confirmed.

I am still getting the dreaded ThrottlingException. I have tried various publish_frequency

publish_frequency = 10s

[ERROR] [1554434092.587344572, 32.603000000]: [AWSClient] HTTP response code: 400
Exception name: ThrottlingException
Error message: Rate exceeded
5 response headers:
connection : close
content-length : 58
content-type : application/x-amz-json-1.1
date : Fri, 05 Apr 2019 03:14:51 GMT
x-amzn-requestid : faa1d28b-5750-11e9-9b5c-49f40dc0bbd4

publish_frequency = 60s

[ERROR] [1554433884.447696369, 406.472000000]: [AWSClient] HTTP response code: 400
Exception name: ThrottlingException
Error message: Rate exceeded
5 response headers:
connection : close
content-length : 58
content-type : application/x-amz-json-1.1
date : Fri, 05 Apr 2019 03:11:23 GMT
x-amzn-requestid : 7e92c988-5750-11e9-a09f-0db3a37537c2

When this is encountered from what I can tell the batch of logs sent go missing. Majority of the time logging will resume and show in CW and other times it just continuously error out.

aws-robotics / cloudwatchlogs-ros1 Goto Github PK

cloudwatchlogs-ros1's People

Contributors

Stargazers

Watchers

Forkers

cloudwatchlogs-ros1's Issues

Test Environment

Recommend Projects

Recommend Topics

Recommend Org