heyoutline / cloud_watch Goto Github PK
View Code? Open in Web Editor NEWLogger backend for Elixir applications
License: MIT License
Logger backend for Elixir applications
License: MIT License
๐
I'm getting
CaseClauseError: no case clause matching: {:error, {"InvalidSequenceTokenException", "The given sequenceToken is invalid. The next expected sequenceToken is: 49619543027323424921201786257049013329532568378470856290", "49619543027323424921201786257049013329532568378470856290"}}
File "lib/cloud_watch/aws_proxy.ex", line 124, in CloudWatch.AwsProxy.request/2
File "lib/cloud_watch.ex", line 186, in CloudWatch.do_flush/4
File "lib/cloud_watch.ex", line 47, in CloudWatch.handle_info/2
File "gen_event.erl", line 627, in :gen_event.server_update/4
File "gen_event.erl", line 609, in :gen_event.server_notify/4
File "gen_event.erl", line 398, in :gen_event.handle_msg/6
File "proc_lib.erl", line 226, in :proc_lib.init_p_do_apply/3
since the match in CloudWatch.AwsProxy.request/2
expects {:error, {type, info}}
and {:error, {type, into, sequenceToken}}
is returned from ex_aws.
I'm using ex_aws 2.2.4
that was recently released.
Fix: #25
My production application uses this package to push our server logs to CloudWatch and everything has been great until this morning. Our server ended up crashing. We got a lot of error messages related to this package, but it's still unclear what lead the application to crash completely. From my understanding, the processes created by this package should be spawned by Logger and so a cascading failure here shouldn't affect the main app, but I could be wrong there.
We ended up getting about 200 of the following messages:
** (exit) {:EXIT, {{:case_clause, {:error, {:closed, ""}}}, [{CloudWatch.AwsProxy, :request, 2, [file: 'lib/cloud_watch/aws_proxy.ex', line: 61]}, {CloudWatch, :flush, 2, [file: 'lib/cloud_watch.ex', line: 93]}, {CloudWatch, :handle_info, 2, [file: 'lib/cloud_watch.ex', line: 46]}, {:gen_event, :server_update, 4, [file: 'gen_event.erl', line: 577]}, {:gen_event, :server_notify, 4, [file: 'gen_event.erl', line: 559]}, {:gen_event, :handle_msg, 6, [file: 'gen_event.erl', line: 347]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}
It seems that the request to flush the buffer failed and there was no clause in the case
statement opened on line 93 to handle it. Looking over the AwsProxy, it seems that, when using the AWS
package, it should return {:error, HTTPoison.Error.t}
and that's what the case statement expects. However, we're using ExAWS
and that seems to only return {:error, {type, message}}
so it can't match.
We're using ex_aws @ 2.1.0
and cloud_watch @ 0.3.0
Both Timber and CloudWatch installed in project so I was able to view the logs on Timber even though CloudWatch was crashing due to the log formatter.
Getting this error in logs of Timber:
CloudWatch installed in Logger terminating ** (Protocol.UndefinedError) protocol Jason.Encoder not implemented for {{2020, 11, 13}, {14, 32, 36, 913}} of type Tuple, Jason.Encoder protocol must always be explicitly implemented. This protocol is implemented for the following type(s): CloudWatch.InputLogEvent, Ecto.Association.NotLoaded, Ecto.Schema.Metadata, DateTime, Integer, List, Float, Map, Time, Atom, Decimal, NaiveDateTime, Date, Any, BitString, Jason.Fragment (jason 1.2.2) lib/jason.ex:150: Jason.encode!/2 (ex_aws 2.1.6) lib/ex_aws/request.ex:19: ExAws.Request.request/6 (ex_aws 2.1.6) lib/ex_aws/operation/json.ex:49: ExAws.Operation.ExAws.Operation.JSON.perform/2 (cloud_watch 0.3.2) lib/cloud_watch/aws_proxy.ex:61: CloudWatch.AwsProxy.request/2 (cloud_watch 0.3.2) lib/cloud_watch.ex:121: CloudWatch.do_flush/4 (cloud_watch 0.3.2) lib/cloud_watch.ex:41: CloudWatch.handle_info/2 (stdlib 3.11) gen_event.erl:577: :gen_event.server_update/4 (stdlib 3.11) gen_event.erl:559: :gen_event.server_notify/4 (stdlib 3.11) gen_event.erl:347: :gen_event.handle_msg/6 (stdlib 3.11) proc_lib.erl:249: :proc_lib.init_p_do_apply/3 Last message: :flush State: %{buffer: [%CloudWatch.InputLogEvent{message: "[debug] QUERY OK source=\"event\" db=1.7ms\nSELECT max(e0.\"createTimeStamp\") FROM \"event\" AS e0 WHERE (e0.\"gameID\" = $1) [<<196, 234, 188, 198, 236, 184, 69, 45, 155, 81, 180, 126, 71, 252, 32, 236>>] \n", timestamp: {{2020, 11, 13}, {14, 32, 36, 913}}}], buffer_length: 1, buffer_size: 225, client: %{}, flushed_at: nil, format: {Statstrack.CloudWatchFormatter, :format}, level: :debug, log_group_name: "statstrack-web", log_stream_name: "statstrack-web-statstrack-prod-prod", max_buffer_size: 10485, max_timeout: 60000, sequence_token: nil}
The previous hurdle I managed to clear was CloudWatch crashing on CloudWatch.InputLogEvent
in a similar fashion to how it is crashing here with the Erlang date type. I added a custom log formatter so it would catch this type so Jason wouldn't crash.
defmodule MyApp.CloudWatchFormatter do
require Protocol
Protocol.derive(Jason.Encoder, CloudWatch.InputLogEvent, only: [:message, :timestamp])
# If I could put in another derive here to catch the tuple that would be great but there's no way for derive to do that afaik
@pattern Logger.Formatter.compile("[$level] $message $metadata\n")
def format(level, message, timestamp, metadata) do
Logger.Formatter.format(@pattern, level, message, timestamp, [])
rescue
error -> "could not format: #{inspect(error)} #{inspect({level, message, metadata})}\n"
end
end
And then plugged that in to the cloud_watch config:
config :logger, CloudWatch,
format: {MyApp.CloudWatchFormatter, :format},
level: :debug,
log_group_name: "myapp-web",
log_stream_name: "myapp-logs",
max_buffer_size: 10_485,
max_timeout: 60_000,
metadata: []
This can't address the Erlang timestamp which looks like: {{2020, 11, 13}, {14, 32, 36, 913}}
and I can't accommodate the tuple in the formatter the way I can with a module like CloudWatch.InputLogEvent
.
I tried forking this lib and doing the formatter myself making it safe for Jason but I couldn't get it working so in the end I just switched ex_aws json parser back to Poison and now it's working... or possibly just on to the next iteration of problem.
It would be great if someone smarter than I could make this change so Jason can handle it.
I'd like to propose following updates:
release_0.3
for compatibility with older applicationsrelease_0.3
if 1.5 is neededhttpoison
. We should not restrict users in their choicesThe logs are showing up in the console with local time, not UTC.
Hello, I had this problem here, and I do not know what I can do to solve it.
It looks like I've exceeded the limit, but I do not have a boundary value set.
Process #PID<0.20400.23> terminating: {:EXIT, {{:case_clause, {:error, {"ThrottlingException", "Rate exceeded"}}}, [{CloudWatch, :flush, 2, [file: 'lib/cloud_watch.ex', line: 93]}, {:gen_event, :server_update, 4, [file: 'gen_event.erl', line: 577]}, {:gen_event, :server_notify, 4, [file: 'gen_event.erl', line: 559]}, {:gen_event, :handle_msg, 6, [file: 'gen_event.erl', line: 300]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}
Apparently, my application went offline because of this error. Can this happen or does the dependency work in isolation and an error returned from Amazon would not knock out an application?
Hi, I'm ready to submit a pull request with metadata filtering and matching feature but I just saw @pmenhart fork and it has some interesting features that I would like to use in my code.
@pmenhart what do you think about pushing your code in this repo (or @lboekhorst the other way around)? I think it would be more useful for the community to have a single repo with all developed features.
I'm trying to run mix phx.digest in my app and it throws the following error.
== Compilation error in file lib/cloud_watch/input_log_event.ex == ** (ArgumentError) Poison.Encoder is not available (elixir) lib/protocol.ex:69: Protocol.assert_protocol!/2 lib/cloud_watch/input_log_event.ex:4: (module) (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6 could not compile dependency :cloud_watch, "mix compile" failed. You can recompile this dependency with "mix deps.compile cloud_watch", update it with "mix deps.update cloud_watch" or clean it with "mix deps.clean cloud_watch"
It seems that there is not a dependency for poison in the cloudwatch project, I'm currently using Jason, there is an option for selecting another JSON encoder?
It looks like it will only flush() when the buffer is full? If there is a log entry sitting there that hasn't filled the buffer, it will sit forever? Is there any time-based flush as well?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.