wistia / elixir_nsq Goto Github PK

View Code? Open in Web Editor NEW

90.0 90.0 26.0 372 KB

An NSQ client for Elixir and Erlang, written in Elixir.

License: MIT License

Elixir 98.94% Procfile 1.06%

elixir_nsq's People

Contributors

Stargazers

Watchers

elixir_nsq's Issues

[upwork post] I saw the job-post and would love to collaborate on here.

Timeouts on both Consumers and Producers

Even in low volumes, both my consumer and producer timeout from time to time as per the logs below. Any config params that could be tweaked to avoid this?

Publisher:

13:49:15.706 [error] GenServer NsqPublisher.Publisher terminating
** (stop) exited in: GenServer.call(#PID<0.10321.0>, {:pub, "message"}, 5000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:604: GenServer.call/3
    (elixir_nsq) lib/nsq/producer.ex:205: NSQ.Producer.pub/2
    (nsq_publisher) lib/publisher.ex:23: NsqPublisher.Publisher.handle_cast/2
    (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

13:49:15.707 [error] GenServer #PID<0.10321.0> terminating
** (Protocol.UndefinedError) protocol String.Chars not implemented for {:pub, "messages", "message"}
    (elixir) lib/string/chars.ex:3: String.Chars.impl_for!/1

    (elixir) lib/string/chars.ex:17: String.Chars.to_string/1
    (elixir_nsq) lib/nsq/connection.ex:217: NSQ.Connection.cmd/3
    (elixir_nsq) lib/nsq/producer.ex:256: NSQ.Producer.do_pub/3
    (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:661: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

Consumer:

13:49:25.968 [error] GenServer #PID<0.6130.4> terminating
** (FunctionClauseError) no function clause matching in NSQ.ConnInfo.conn_id/2
    (elixir_nsq) lib/nsq/conn_info.ex:9: NSQ.ConnInfo.conn_id(#PID<0.6129.4>, "parent:#PID<0.6129.4>:conn:192.168.0.63:4150")
    (elixir_nsq) lib/nsq/consumer/connections.ex:184: NSQ.Consumer.Connections.stop_connection/3
    (elixir_nsq) lib/nsq/consumer/connections.ex:166: anonymous fn/3 in NSQ.Consumer.Connections.stop_connections/3
    (elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir_nsq) lib/nsq/consumer/connections.ex:165: NSQ.Consumer.Connections.stop_connections/3
    (elixir_nsq) lib/nsq/consumer/connections.ex:95: NSQ.Consumer.Connections.update/3
    (elixir_nsq) lib/nsq/consumer/connections.ex:82: NSQ.Consumer.Connections.discover_nsqds_and_connect/2

    (elixir_nsq) lib/nsq/consumer/connections.ex:51: NSQ.Consumer.Connections.refresh/1

13:49:25.968 [error] Task #PID<0.10873.5> started from #PID<0.10871.5> terminating
** (stop) exited in: GenServer.call(#PID<0.6135.4>, {:send, "FIN 06fbac5b95a01000\n"}, 5000)
    ** (EXIT) no process
    (elixir) lib/gen_server.ex:604: GenServer.call/3
    (elixir_nsq) lib/nsq/connection/buffer.ex:101: NSQ.Connection.Buffer.send!/2
    (elixir_nsq) lib/nsq/message.ex:99: NSQ.Message.fin/1
    (elixir_nsq) lib/nsq/message.ex:208: NSQ.Message.respond_to_nsq/2
    (elixir_nsq) lib/nsq/message.ex:158: NSQ.Message.process_without_timeout/1
    (elixir) lib/task/supervised.ex:94: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: #Function<1.124640675/0 in NSQ.Message.process/1>
    Args: []

Could not start application ibrowse: could not find application file: ibrowse.app

Hey,
thank you for writing this package! I encountered some problems when downloading the package in the way suggested in the readme ({:elixir_nsq, "~> 1.0.3"}). If I choose to pull directly from git ({:elixir_nsq, github: "wistia/elixir_nsq"}), everything works fine. It seems to be related to ibrowse not being pulled correctly from git when the release version was fetched from hex.

Steps to reproduce:

mix new test

test/mix.exs

defmodule Test.Mixfile do
use Mix.Project

def project do
  [app: :test,
   version: "0.0.1",
   elixir: "~> 1.2",
   build_embedded: Mix.env == :prod,
   start_permanent: Mix.env == :prod,
   deps: deps]
end

# Configuration for the OTP application
#
# Type "mix help compile.app" for more information
def application do
  [applications: [:logger, :elixir_nsq]]
end

# Dependencies can be Hex packages:
#
#   {:mydep, "~> 0.3.0"}
#
# Or git/path repositories:
#
#   {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}
#
# Type "mix help deps" for more examples and options
defp deps do
  [{:elixir_nsq, "~> 1.0.3"}]
end
end

mix deps.get; mix deps.compile --all
mix run --no-halt

=INFO REPORT==== 17-Jun-2016::16:09:40 ===
application: logger
exited: stopped
type: temporary
** (Mix) Could not start application ibrowse: could not find application file: ibrowse.app

Port HTTP client to another client

Currently, httpotion is now deprecated. The possible option changes it to use another client such as, :tesla which can switch to any client base on the user, or :finch, :req, etc.

I can help port the client after your consensus about the HTTP client. 🙇

(FunctionClauseError) no function clause matching in anonymous fn/1 in Socket.arguments/1 when creating Producer

As soon as this is called:

NSQ.Producer.Supervisor.start_link(
      "tracks",
      %NSQ.Config{
        nsqds: ["127.0.0.1:4150", "127.0.0.1:4151"]
      }
    )

The following error is thrown:

** (FunctionClauseError) no function clause matching in anonymous fn/1 in Socket.arguments/1
    (socket) lib/socket.ex:170: anonymous fn({:active, false}) in Socket.arguments/1
    (elixir) lib/enum.ex:1003: Enum.flat_map_list/2
    (elixir) lib/enum.ex:1004: Enum.flat_map_list/2
    (socket) lib/socket/tcp.ex:285: Socket.TCP.arguments/1
    (socket) lib/socket/tcp.ex:108: Socket.TCP.connect/3
    (elixir_nsq) lib/nsq/connection/initializer.ex:24: NSQ.Connection.Initializer.connect/1
    (elixir_nsq) lib/nsq/connection.ex:99: NSQ.Connection.init/1
    (stdlib) gen_server.erl:365: :gen_server.init_it/2
    (stdlib) gen_server.erl:333: :gen_server.init_it/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.1202.0>, {:function_clause, [{Socket, :"-arguments/1-fun-2-", [active: false], [file: 'lib/socket.ex', line: 170]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1003]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1004]}, {Socket.TCP, :arguments, 1, [file: 'lib/socket/tcp.ex', line: 285]}, {Socket.TCP, :connect, 3, [file: 'lib/socket/tcp.ex', line: 108]}, {NSQ.Connection.Initializer, :connect, 1, [file: 'lib/nsq/connection/initializer.ex', line: 24]}, {NSQ.Connection, :init, 1, [file: 'lib/nsq/connection.ex', line: 99]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}
State: {:state, {#PID<0.1205.0>, NSQ.Message.Supervisor}, :one_for_one, [], :undefined, 3, 5, [], 0, NSQ.Message.Supervisor, :ok}
[error] GenServer #PID<0.1199.0> terminating
** (MatchError) no match of right hand side value: {:error, {{:EXIT, {{:badmatch, {:error, {:function_clause, [{Socket, :"-arguments/1-fun-2-", [active: false], [file: 'lib/socket.ex', line: 170]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1003]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1004]}, {Socket.TCP, :arguments, 1, [file: 'lib/socket/tcp.ex', line: 285]}, {Socket.TCP, :connect, 3, [file: 'lib/socket/tcp.ex', line: 108]}, {NSQ.Connection.Initializer, :connect, 1, [file: 'lib/nsq/connection/initializer.ex', line: 24]}, {NSQ.Connection, :init, 1, [file: 'lib/nsq/connection.ex', line: 99]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}}, [{NSQ.Connection, :start_link, 8, [file: 'lib/nsq/connection.ex', line: 82]}, {:supervisor, :do_start_child, 2, [file: 'supervisor.erl', line: 365]}, {:supervisor, :handle_start_child, 2, [file: 'supervisor.erl', line: 724]}, {:supervisor, :handle_call, 3, [file: 'supervisor.erl', line: 422]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 636]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 665]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}, {:child, :undefined, "parent:#PID<0.1198.0>:conn:127.0.0.1:4150", {NSQ.Connection, :start_link, [#PID<0.1198.0>, {"127.0.0.1", 4150}, %NSQ.Config{client_id: nil, max_backoff_duration: 120000, max_in_flight: 2500, tls_key: nil, user_agent: nil, write_timeout: 1000, backoff_multiplier: 1000, nsqlookupds: [], hostname: nil, tls_min_version: nil, max_reconnect_attempts: 30, msg_timeout: 60000, snappy: false, tls_cert: nil, output_buffer_size: 16384, read_timeout: 60000, heartbeat_interval: 30000, lookupd_poll_interval: 60000, auth_secret: "", deflate_level: 6, tls_insecure_skip_verify: false, dial_timeout: 1000, rdy_redistribute_interval: 5000, rdy_retry_delay: 5000, low_rdy_idle_timeout: 10000, nsqds: [{"127.0.0.1", 4150}, {"127.0.0.1", 4151}], max_attempts: 5, deflate: false, lookupd_poll_jitter: 0.3, output_buffer_timeout: 250, max_requeue_delay: 900000, backoff_strategy: :exponential, sample_rate: 0, event_manager: nil, tls_v1: false, message_handler: nil}, "tracks", nil, #PID<0.1200.0>, #PID<0.1201.0>]}, :temporary, 5000, :worker, [NSQ.Connection]}}}
    (elixir_nsq) lib/nsq/producer.ex:183: anonymous fn/3 in NSQ.Producer.connect_to_nsqds/3
    (elixir) lib/enum.ex:1270: Enum."-map/2-lists^map/1-0-"/2
    (elixir_nsq) lib/nsq/producer.ex:182: NSQ.Producer.connect_to_nsqds/3
    (elixir_nsq) lib/nsq/producer.ex:105: NSQ.Producer.init/1
    (stdlib) gen_server.erl:365: :gen_server.init_it/2
    (stdlib) gen_server.erl:333: :gen_server.init_it/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.1198.0>, {{:badmatch, {:error, {{:EXIT, {{:badmatch, {:error, {:function_clause, [{Socket, :"-arguments/1-fun-2-", [active: false], [file: 'lib/socket.ex', line: 170]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1003]}, {Enum, :flat_map_list, 2, [file: 'lib/enum.ex', line: 1004]}, {Socket.TCP, :arguments, 1, [file: 'lib/socket/tcp.ex', line: 285]}, {Socket.TCP, :connect, 3, [file: 'lib/socket/tcp.ex', line: 108]}, {NSQ.Connection.Initializer, :connect, 1, [file: 'lib/nsq/connection/initializer.ex', line: 24]}, {NSQ.Connection, :init, 1, [file: 'lib/nsq/connection.ex', line: 99]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}}, [{NSQ.Connection, :start_link, 8, [file: 'lib/nsq/connection.ex', line: 82]}, {:supervisor, :do_start_child, 2, [file: 'supervisor.erl', line: 365]}, {:supervisor, :handle_start_child, 2, [file: 'supervisor.erl', line: 724]}, {:supervisor, :handle_call, 3, [file: 'supervisor.erl', line: 422]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 636]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 665]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}, {:child, :undefined, "parent:#PID<0.1198.0>:conn:127.0.0.1:4150", {NSQ.Connection, :start_link, [#PID<0.1198.0>, {"127.0.0.1", 4150}, %NSQ.Config{client_id: nil, max_backoff_duration: 120000, max_in_flight: 2500, tls_key: nil, user_agent: nil, write_timeout: 1000, backoff_multiplier: 1000, nsqlookupds: [], hostname: nil, tls_min_version: nil, max_reconnect_attempts: 30, msg_timeout: 60000, snappy: false, tls_cert: nil, output_buffer_size: 16384, read_timeout: 60000, heartbeat_interval: 30000, lookupd_poll_interval: 60000, auth_secret: "", deflate_level: 6, tls_insecure_skip_verify: false, dial_timeout: 1000, rdy_redistribute_interval: 5000, rdy_retry_delay: 5000, low_rdy_idle_timeout: 10000, nsqds: [{"127.0.0.1", 4150}, {"127.0.0.1", 4151}], max_attempts: 5, deflate: false, lookupd_poll_jitter: 0.3, output_buffer_timeout: 250, max_requeue_delay: 900000, backoff_strategy: :exponential, sample_rate: 0, event_manager: nil, tls_v1: false, ...}, "tracks", nil, #PID<0.1200.0>, #PID<0.1201.0>]}, :temporary, 5000, :worker, [NSQ.Connection]}}}}, [{NSQ.Producer, :"-connect_to_nsqds/3-fun-0-", 3, [file: 'lib/nsq/producer.ex', line: 183]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1270]}, {NSQ.Producer, :connect_to_nsqds, 3, [file: 'lib/nsq/producer.ex', line: 182]}, {NSQ.Producer, :init, 1, [file: 'lib/nsq/producer.ex', line: 105]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 365]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 333]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}
State: {:state, {#PID<0.1199.0>, NSQ.Connection.Supervisor}, :one_for_one, [], :undefined, 3, 5, [], 0, NSQ.Connection.Supervisor, :ok}

Running Elixir 1.5.2, Erlang/OTP 20.

{:elixir_nsq, "~> 1.0.3"},

Stale NSQ.Producer.close in README

NSQ.Producer.close is referenced in the README however it does not exist

https://github.com/wistia/elixir_nsq/blob/master/README.md#publish-messages

TODO: Update to to work with newer versions of socket package

Using socket 0.3.8 (maybe anything after 0.3.5; this is not known to me) results in an error like this when trying to connect:

I updated the mix.exs to restrict the package version from 0.3.1 to 0.3.5, but that kind of fix isn't a good long term strategy. We should update our code to be compatible from 0.3.1 up.

Elixir.ArithmeticError

This error occurred after our application failed to write some db records. While the initial issue is app-based, we think it uncovered a bug in this area.

Elixir.ArithmeticError bad argument in arithmetic expression
lib/nsq/consumer/backoff.ex:175 NSQ.Consumer.Backoff.exponential_backoff/1
lib/nsq/consumer/backoff.ex:83 NSQ.Consumer.Backoff.backoff/3
lib/nsq/consumer/backoff.ex:29 NSQ.Consumer.Backoff.start_stop_continue/3
lib/nsq/consumer/backoff.ex:36 NSQ.Consumer.Backoff.start_stop_continue!/3
lib/nsq/consumer.ex:221 NSQ.Consumer.handle_call/3
gen_server.erl:661 :gen_server.try_handle_call/4
gen_server.erl:690 :gen_server.handle_msg/6
proc_lib.erl:249 :proc_lib.init_p_do_apply/3

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.