wistia / elixir_nsq Goto Github PK
View Code? Open in Web Editor NEWAn NSQ client for Elixir and Erlang, written in Elixir.
License: MIT License
An NSQ client for Elixir and Erlang, written in Elixir.
License: MIT License
Even in low volumes, both my consumer and producer timeout from time to time as per the logs below. Any config
params that could be tweaked to avoid this?
Publisher:
13:49:15.706 [error] GenServer NsqPublisher.Publisher terminating
** (stop) exited in: GenServer.call(#PID<0.10321.0>, {:pub, "message"}, 5000)
** (EXIT) time out
(elixir) lib/gen_server.ex:604: GenServer.call/3
(elixir_nsq) lib/nsq/producer.ex:205: NSQ.Producer.pub/2
(nsq_publisher) lib/publisher.ex:23: NsqPublisher.Publisher.handle_cast/2
(stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:681: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
13:49:15.707 [error] GenServer #PID<0.10321.0> terminating
** (Protocol.UndefinedError) protocol String.Chars not implemented for {:pub, "messages", "message"}
(elixir) lib/string/chars.ex:3: String.Chars.impl_for!/1
(elixir) lib/string/chars.ex:17: String.Chars.to_string/1
(elixir_nsq) lib/nsq/connection.ex:217: NSQ.Connection.cmd/3
(elixir_nsq) lib/nsq/producer.ex:256: NSQ.Producer.do_pub/3
(stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:661: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Consumer:
13:49:25.968 [error] GenServer #PID<0.6130.4> terminating
** (FunctionClauseError) no function clause matching in NSQ.ConnInfo.conn_id/2
(elixir_nsq) lib/nsq/conn_info.ex:9: NSQ.ConnInfo.conn_id(#PID<0.6129.4>, "parent:#PID<0.6129.4>:conn:192.168.0.63:4150")
(elixir_nsq) lib/nsq/consumer/connections.ex:184: NSQ.Consumer.Connections.stop_connection/3
(elixir_nsq) lib/nsq/consumer/connections.ex:166: anonymous fn/3 in NSQ.Consumer.Connections.stop_connections/3
(elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir_nsq) lib/nsq/consumer/connections.ex:165: NSQ.Consumer.Connections.stop_connections/3
(elixir_nsq) lib/nsq/consumer/connections.ex:95: NSQ.Consumer.Connections.update/3
(elixir_nsq) lib/nsq/consumer/connections.ex:82: NSQ.Consumer.Connections.discover_nsqds_and_connect/2
(elixir_nsq) lib/nsq/consumer/connections.ex:51: NSQ.Consumer.Connections.refresh/1
13:49:25.968 [error] Task #PID<0.10873.5> started from #PID<0.10871.5> terminating
** (stop) exited in: GenServer.call(#PID<0.6135.4>, {:send, "FIN 06fbac5b95a01000\n"}, 5000)
** (EXIT) no process
(elixir) lib/gen_server.ex:604: GenServer.call/3
(elixir_nsq) lib/nsq/connection/buffer.ex:101: NSQ.Connection.Buffer.send!/2
(elixir_nsq) lib/nsq/message.ex:99: NSQ.Message.fin/1
(elixir_nsq) lib/nsq/message.ex:208: NSQ.Message.respond_to_nsq/2
(elixir_nsq) lib/nsq/message.ex:158: NSQ.Message.process_without_timeout/1
(elixir) lib/task/supervised.ex:94: Task.Supervised.do_apply/2
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: #Function<1.124640675/0 in NSQ.Message.process/1>
Args: []
Hey,
thank you for writing this package! I encountered some problems when downloading the package in the way suggested in the readme ({:elixir_nsq, "~> 1.0.3"}
). If I choose to pull directly from git ({:elixir_nsq, github: "wistia/elixir_nsq"}
), everything works fine. It seems to be related to ibrowse not being pulled correctly from git when the release version was fetched from hex.
Steps to reproduce:
mix new test
test/mix.exs
defmodule Test.Mixfile do
use Mix.Project
def project do
[app: :test,
version: "0.0.1",
elixir: "~> 1.2",
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
deps: deps]
end
# Configuration for the OTP application
#
# Type "mix help compile.app" for more information
def application do
[applications: [:logger, :elixir_nsq]]
end
# Dependencies can be Hex packages:
#
# {:mydep, "~> 0.3.0"}
#
# Or git/path repositories:
#
# {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}
#
# Type "mix help deps" for more examples and options
defp deps do
[{:elixir_nsq, "~> 1.0.3"}]
end
end
mix deps.get; mix deps.compile --all
mix run --no-halt
=INFO REPORT==== 17-Jun-2016::16:09:40 ===
application: logger
exited: stopped
type: temporary
** (Mix) Could not start application ibrowse: could not find application file: ibrowse.app
Currently, httpotion is now deprecated. The possible option changes it to use another client such as, :tesla
which can switch to any client base on the user, or :finch
, :req
, etc.
I can help port the client after your consensus about the HTTP client. ๐
As soon as this is called:
NSQ.Producer.Supervisor.start_link(
"tracks",
%NSQ.Config{
nsqds: ["127.0.0.1:4150", "127.0.0.1:4151"]
}
)
The following error is thrown:
** (FunctionClauseError) no function clause matching in anonymous fn/1 in Socket.arguments/1
(socket) lib/socket.ex:170: anonymous fn({:active, false}) in Socket.arguments/1
(elixir) lib/enum.ex:1003: Enum.flat_map_list/2
(elixir) lib/enum.ex:1004: Enum.flat_map_list/2
(socket) lib/socket/tcp.ex:285: Socket.TCP.arguments/1
(socket) lib/socket/tcp.ex:108: Socket.TCP.connect/3
(elixir_nsq) lib/nsq/connection/initializer.ex:24: NSQ.Connection.Initializer.connect/1
(elixir_nsq) lib/nsq/connection.ex:99: NSQ.Connection.init/1
(stdlib) gen_server.erl:365: :gen_server.init_it/2
(stdlib) gen_server.erl:333: :gen_server.init_it/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.1202.0>, {:function_clause, [{Socket, :"-arguments/1-fun-2-", [active: false], [file: 'lib/socket.ex', line: 170]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1003]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1004]}, {Socket.TCP, :arguments, 1, [file: 'lib/socket/tcp.ex', line: 285]}, {Socket.TCP, :connect, 3, [file: 'lib/socket/tcp.ex', line: 108]}, {NSQ.Connection.Initializer, :connect, 1, [file: 'lib/nsq/connection/initializer.ex', line: 24]}, {NSQ.Connection, :init, 1, [file: 'lib/nsq/connection.ex', line: 99]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}
State: {:state, {#PID<0.1205.0>, NSQ.Message.Supervisor}, :one_for_one, [], :undefined, 3, 5, [], 0, NSQ.Message.Supervisor, :ok}
[error] GenServer #PID<0.1199.0> terminating
** (MatchError) no match of right hand side value: {:error, {{:EXIT, {{:badmatch, {:error, {:function_clause, [{Socket, :"-arguments/1-fun-2-", [active: false], [file: 'lib/socket.ex', line: 170]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1003]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1004]}, {Socket.TCP, :arguments, 1, [file: 'lib/socket/tcp.ex', line: 285]}, {Socket.TCP, :connect, 3, [file: 'lib/socket/tcp.ex', line: 108]}, {NSQ.Connection.Initializer, :connect, 1, [file: 'lib/nsq/connection/initializer.ex', line: 24]}, {NSQ.Connection, :init, 1, [file: 'lib/nsq/connection.ex', line: 99]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}}, [{NSQ.Connection, :start_link, 8, [file: 'lib/nsq/connection.ex', line: 82]}, {:supervisor, :do_start_child, 2, [file: 'supervisor.erl', line: 365]}, {:supervisor, :handle_start_child, 2, [file: 'supervisor.erl', line: 724]}, {:supervisor, :handle_call, 3, [file: 'supervisor.erl', line: 422]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 636]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 665]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}, {:child, :undefined, "parent:#PID<0.1198.0>:conn:127.0.0.1:4150", {NSQ.Connection, :start_link, [#PID<0.1198.0>, {"127.0.0.1", 4150}, %NSQ.Config{client_id: nil, max_backoff_duration: 120000, max_in_flight: 2500, tls_key: nil, user_agent: nil, write_timeout: 1000, backoff_multiplier: 1000, nsqlookupds: [], hostname: nil, tls_min_version: nil, max_reconnect_attempts: 30, msg_timeout: 60000, snappy: false, tls_cert: nil, output_buffer_size: 16384, read_timeout: 60000, heartbeat_interval: 30000, lookupd_poll_interval: 60000, auth_secret: "", deflate_level: 6, tls_insecure_skip_verify: false, dial_timeout: 1000, rdy_redistribute_interval: 5000, rdy_retry_delay: 5000, low_rdy_idle_timeout: 10000, nsqds: [{"127.0.0.1", 4150}, {"127.0.0.1", 4151}], max_attempts: 5, deflate: false, lookupd_poll_jitter: 0.3, output_buffer_timeout: 250, max_requeue_delay: 900000, backoff_strategy: :exponential, sample_rate: 0, event_manager: nil, tls_v1: false, message_handler: nil}, "tracks", nil, #PID<0.1200.0>, #PID<0.1201.0>]}, :temporary, 5000, :worker, [NSQ.Connection]}}}
(elixir_nsq) lib/nsq/producer.ex:183: anonymous fn/3 in NSQ.Producer.connect_to_nsqds/3
(elixir) lib/enum.ex:1270: Enum."-map/2-lists^map/1-0-"/2
(elixir_nsq) lib/nsq/producer.ex:182: NSQ.Producer.connect_to_nsqds/3
(elixir_nsq) lib/nsq/producer.ex:105: NSQ.Producer.init/1
(stdlib) gen_server.erl:365: :gen_server.init_it/2
(stdlib) gen_server.erl:333: :gen_server.init_it/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.1198.0>, {{:badmatch, {:error, {{:EXIT, {{:badmatch, {:error, {:function_clause, [{Socket, :"-arguments/1-fun-2-", [active: false], [file: 'lib/socket.ex', line: 170]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1003]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1004]}, {Socket.TCP, :arguments, 1, [file: 'lib/socket/tcp.ex', line: 285]}, {Socket.TCP, :connect, 3, [file: 'lib/socket/tcp.ex', line: 108]}, {NSQ.Connection.Initializer, :connect, 1, [file: 'lib/nsq/connection/initializer.ex', line: 24]}, {NSQ.Connection, :init, 1, [file: 'lib/nsq/connection.ex', line: 99]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}}, [{NSQ.Connection, :start_link, 8, [file: 'lib/nsq/connection.ex', line: 82]}, {:supervisor, :do_start_child, 2, [file: 'supervisor.erl', line: 365]}, {:supervisor, :handle_start_child, 2, [file: 'supervisor.erl', line: 724]}, {:supervisor, :handle_call, 3, [file: 'supervisor.erl', line: 422]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 636]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 665]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}, {:child, :undefined, "parent:#PID<0.1198.0>:conn:127.0.0.1:4150", {NSQ.Connection, :start_link, [#PID<0.1198.0>, {"127.0.0.1", 4150}, %NSQ.Config{client_id: nil, max_backoff_duration: 120000, max_in_flight: 2500, tls_key: nil, user_agent: nil, write_timeout: 1000, backoff_multiplier: 1000, nsqlookupds: [], hostname: nil, tls_min_version: nil, max_reconnect_attempts: 30, msg_timeout: 60000, snappy: false, tls_cert: nil, output_buffer_size: 16384, read_timeout: 60000, heartbeat_interval: 30000, lookupd_poll_interval: 60000, auth_secret: "", deflate_level: 6, tls_insecure_skip_verify: false, dial_timeout: 1000, rdy_redistribute_interval: 5000, rdy_retry_delay: 5000, low_rdy_idle_timeout: 10000, nsqds: [{"127.0.0.1", 4150}, {"127.0.0.1", 4151}], max_attempts: 5, deflate: false, lookupd_poll_jitter: 0.3, output_buffer_timeout: 250, max_requeue_delay: 900000, backoff_strategy: :exponential, sample_rate: 0, event_manager: nil, tls_v1: false, ...}, "tracks", nil, #PID<0.1200.0>, #PID<0.1201.0>]}, :temporary, 5000, :worker, [NSQ.Connection]}}}}, [{NSQ.Producer, :"-connect_to_nsqds/3-fun-0-", 3, [file: 'lib/nsq/producer.ex', line: 183]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1270]}, {NSQ.Producer, :connect_to_nsqds, 3, [file: 'lib/nsq/producer.ex', line: 182]}, {NSQ.Producer, :init, 1, [file: 'lib/nsq/producer.ex', line: 105]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}
State: {:state, {#PID<0.1199.0>, NSQ.Connection.Supervisor}, :one_for_one, [], :undefined, 3, 5, [], 0, NSQ.Connection.Supervisor, :ok}
Running Elixir 1.5.2, Erlang/OTP 20.
{:elixir_nsq, "~> 1.0.3"},
NSQ.Producer.close
is referenced in the README however it does not exist
https://github.com/wistia/elixir_nsq/blob/master/README.md#publish-messages
Using socket 0.3.8 (maybe anything after 0.3.5; this is not known to me) results in an error like this when trying to connect:
I updated the mix.exs to restrict the package version from 0.3.1 to 0.3.5, but that kind of fix isn't a good long term strategy. We should update our code to be compatible from 0.3.1 up.
This error occurred after our application failed to write some db records. While the initial issue is app-based, we think it uncovered a bug in this area.
Elixir.ArithmeticError bad argument in arithmetic expression
lib/nsq/consumer/backoff.ex:175 NSQ.Consumer.Backoff.exponential_backoff/1
lib/nsq/consumer/backoff.ex:83 NSQ.Consumer.Backoff.backoff/3
lib/nsq/consumer/backoff.ex:29 NSQ.Consumer.Backoff.start_stop_continue/3
lib/nsq/consumer/backoff.ex:36 NSQ.Consumer.Backoff.start_stop_continue!/3
lib/nsq/consumer.ex:221 NSQ.Consumer.handle_call/3
gen_server.erl:661 :gen_server.try_handle_call/4
gen_server.erl:690 :gen_server.handle_msg/6
proc_lib.erl:249 :proc_lib.init_p_do_apply/3
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.