ostinelli / apnotic Goto Github PK

Hi!

I had notifiaction.apns_collapse_id = order.id and app was crashing with HTTP2::Error::ProtocolError (HTTP2::Error::ProtocolError) and no backtrace. Sometimes exception was rescued with message undefined method 'bytesize' for 2:Fixnum. But sometimes it wasn't even rescued, app just crashed.

Changing to order.id.to_s fixed this.

APNOTIC_POOL = Apnotic::ConnectionPool.new({
  cert_path: Rails.root.join("config", "certificates", "AuthKey_########.p8")
}, size: 5) do |connection|
  connection.on(:error) { |exception| puts "Exception has been raised: #{exception}" }
end

Yields the error OpenSSL::PKey::RSAError: Neither PUB key nor PRIV key: nested asn1 error

Is there different setup to use when using a .p8 file with ConnectionPool?

Hello,

This probably a known issue/behavior #14 but I have this kind of code:

# this function is called from a Sidekiq worker

    notification = Apnotic::MdmNotification.new(...)
    push = connection.prepare_push(notification)
    push.on(:response) do |response|
      if response.ok?
        XWorker.perform_async(device.id) # 🔒
      end
    end
    connection.join
    connection.close

But the worker (XWorker) "lock" the action and we loop forever in https://github.com/ostinelli/net-http2/blob/master/lib/net-http2/client.rb#L59.

So instead I have to run the code immediately and avoid using the worker. Is it normal? I tried to understand without finding the solution.

Hello.
I use Apnotic push_async in Sidekiq.
The fix from this issue #68 resolve problem when Sidekiq is crushed because of exception in main thread. But from time to time one of my 10 Sidekiq workers is stuck forever with job where we need to send a push async.
I've explored this problem and found what if SocketError happened, for example, here in socket_loop https://github.com/ostinelli/net-http2/blob/master/lib/net-http2/client.rb#L142 (to reproduce just raise it here) next try of push_async will stuck here https://github.com/ostinelli/apnotic/blob/master/lib/apnotic/connection.rb#L83
streams_available? will always be false because SocketError resetting @client.remote_settings to default value from http-2 gem https://github.com/igrigorik/http-2/blob/master/lib/http/2/connection.rb#L11-L19
and remote_max_concurrent_streams always return zero because of that condition https://github.com/ostinelli/apnotic/blob/master/lib/apnotic/connection.rb#L94-L98.

To find this bug I've used monkey patch:

module Apnotic
  class Connection
    private
    def delayed_push_async(push)
      i = 1
      until streams_available? do
        if i < 5000
          sleep 0.001
          i+=1
        else
          raise StandardError.new("Timeout 5s on Apnotic::Connection#delayed_push_async: #{ @client.remote_settings[:settings_max_concurrent_streams] }/#{ @client.stream_count }")
        end
      end
      @client.call_async(push.http2_request)
    end
  end
end

And, as workaround - this monkey patch:

module Apnotic
  class Connection
    private
    def remote_max_concurrent_streams
      # 0x7fffffff is the default value from http-2 gem (2^31)
      if @client.remote_settings[:settings_max_concurrent_streams] == 0x7fffffff
        1
      else
        @client.remote_settings[:settings_max_concurrent_streams]
      end
    end
  end
end

So, I just changed 0 to 1. But actually I didn't understand why we use 0 if we have default value from http-2 gem. If it means some connection troubles and because of that we say that we can't use any concurrent connection, so maybe better raise some kind of exception.

Also, important to know, what another Sidekiq workers continue to successfully use push_async until next SocketError happened in another worker and it also will stuck. If I apply monkey patch, which break loop after 5 seconds, the next push_async in same worker will also trapped in this loop. So, only that worker, which raise SocketError goes into degraded state.

useful comment: #64 (comment)

First of all, thanks for a great library!

If the :error callback is not set, the underlying socket thread may raise an error in the main thread at unexpected execution times.

What would happen if I did this?

connection.on(:error) { |exception| raise exception }

Would this be raised in the underlying socket thread, or in the main thread?

Hi, I have looking at your gem and am considering using it in a production environment for a rails app. My app sends out large numbers of APNs to client devices to manage background updates to client data. I was originally considering using sidekiq and rolling my own APNs manager. Based on my very cursory review, it looks like I can use Apnotic with async push support instead. I was wondering if you feel this gem is ready for production use for the scenario I'm considering.

I assume I am somehow getting in my own way here, but would appreciate any advice that you can provide.

I am currently using the recommended connection.on(:error) do |exception| callback to catch socket errors and re-attempt sending push notifications using .push_async. These exceptions are fairly common in my setup, usually due to a DNS resolution error or closed connection. I am not using a background worker like DelayedJob or similar.

The problem I am experiencing is that .push_async blocks when I attempt to resend a notification. The apnotic documentation states that the underlying connection will be automatically repaired, but this does not seem to be the case. Perhaps this is only in the context of a background worker engine that causes the connection object to be completely reinitialized?

One additional note: I am not using a ConnectionPool currently, though I do not believe this affects the situation one way or the other.

I'm considering using ConnectionPool in order to send a considerable amount of messages.
I'm afraid that keeping connections alive for long will cause memory leaking issues.
Has someone encountered this issue?

Thanks

Push notification api requires topic to be passed with the request. Can we pass default topic for notifications?

From time to time

response = connection.push(notification)

is returning a nil response resulting in an

NoMethodError(undefined method `ok?' for nil:NilClass)

error. I've no clue so far why this is happening. It looks to me that NetHTTP2 might be swallowing an error here that could be helpful for debugging.

I yet have to see if there is any logging/additional information inside apnotic or NetHTTP2 that could help me debug this, cause I'd love to know the underlying cause.

Like many people probably, I'm considering moving to this gem from grocer or houston. I'm wondering if anyone has used it in production successfully and reliably, and what performance was like, and how it matches up to grocer. (Basically, I'm hoping for a thread similar to alloy/lowdown#11). Thanks!

The same issue described here: #68 started biting us in production since this morning. It seems Apple changed something in production since this particular platform has been working flawlessly in production for months.

In our case this had the effect of all our Sidekiq jobs crashing and our queues being filled up.

The error we were getting was:

SocketError: Socket was remotely closed
	from /home/deploy/rails_app/shared/bundle/ruby/2.3.0/gems/net-http2-0.15.0/lib/net-http2/client.rb:122:in `callback_or_raise'
	from /home/deploy/rails_app/shared/bundle/ruby/2.3.0/gems/net-http2-0.15.0/lib/net-http2/client.rb:107:in `rescue in block (2 levels) in ensure_open'
	from /home/deploy/rails_app/shared/bundle/ruby/2.3.0/gems/net-http2-0.15.0/lib/net-http2/client.rb:101:in `block (2 levels) in ensure_open'

As described in #68, the fix has already been merged. Upgrading to latest version of this gem from master upgraded the gem and its underlying http libraries and fixed the issue immediately:

bundle update --source apnotic

...
Using http-2 0.9.0 (was 0.8.2)
Fetching net-http2 0.18.0 (was 0.15.0)
Installing net-http2 0.18.0 (was 0.15.0)
Using apnotic 1.4.0 (was 1.1.0) from git://github.com/ostinelli/apnotic.git (at master@fc9eb88)
...

Opening this issue so it will be easy to find by anyone who is currently being bitten by this.

This is an enhancement request.

net-http2 gem supports proxy parameters (proxy host, port, user, password).
So, it would be great if we can pass these parameters in Apnotic::Connection.new() which in turn creates net-http2 client with these proxy params.
It will be useful for Apnotic gem users, working behind a corporate proxy.

Thanks!

From net-http2 documentation:

It is RECOMMENDED to set the :error callback: if none is defined, the underlying socket thread may raise an error in the main thread at unexpected execution times.

But the problem is that Apnotic::ConnectionPool does not allow calling on on actual Apnotic::Connection to pass in the error handler.

Here's example scenario that leads to lost jobs - we've been seeing this in production for quite some time now but couldn't point a finger on it, but thanks to #68 it's got easily reproducible:

• There's a sidekiq worker set up to initialize pool as suggested in documentation with no error handler
• Fresh sidekiq process is booted and it starts processing jobs, some of them call that worker
• APNS connection fails with SocketError or similar
• Sidekiq process crashes completely since that SocketError is raised on a main thread
• Unprocessed jobs that were already picked up by that process are lost completely

Here is an example report of this exact behavior: sidekiq/sidekiq#3886

I think apnotic should definitely do a better job here to improve reliability and also stop suggesting unsafe usage in documentation. Here's what I would suggest:

• Allow to pass through connection error handler into Apnotic::ConnectionPool
• Make this handler required or at least change the documentation to have it explicitly provided

Currently we work this around by creating a pool manually:

class Worker
  POOL = ConnectionPool.new(size: 5) do
    connection = Apnotic::Connection.new(...)

    connection.on(:error) do |err|
      Bugsnag.notify(ConnectionError.new(err.inspect))
    end

    connection
  end
end

Please let me know if there are any thoughts. Thanks!

I got an error in my app where it says that the response given back from connection.push was nil. Is this expected behavior, and if so what does it mean?

The code where this occurs is as follows:

CONNECTION_POOL.with do |connection|
   notification.topic = Rails.application.config.apns_topic
   response = connection.push(notification)
   if response.status == '410' ||
       (response.status == '400' && response.body['reason'] == 'BadDeviceToken')

The specific error was undefined method 'status' for nil:NilClass

When I looked at connection here, it seems the nil is likely coming upstream from the HTTP2 library.

Any help is appreciated!

Connection reset by peer
/home/alex/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/openssl/buffering.rb:182:in sysread_nonblock' /home/alex/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/openssl/buffering.rb:182:in read_nonblock'
/home/alex/apps/dskb-web/shared/bundle/ruby/2.2.0/gems/net-http2-0.15.0/lib/net-http2/client.rb:133:in block in socket_loop' /home/alex/apps/dskb-web/shared/bundle/ruby/2.2.0/gems/net-http2-0.15.0/lib/net-http2/client.rb:130:in loop'
/home/alex/apps/dskb-web/shared/bundle/ruby/2.2.0/gems/net-http2-0.15.0/lib/net-http2/client.rb:130:in socket_loop' /home/alex/apps/dskb-web/shared/bundle/ruby/2.2.0/gems/net-http2-0.15.0/lib/net-http2/client.rb:102:in block (2 levels) in ensure_open'

is this like socket loop thread abort_on_exception (was: Thread Safety) #4

How to resolve it?

APNOTIC_POOL = Apnotic::ConnectionPool.new(
    {
      cert_path: Rails.root.join('config', 'apns-production.pem'),
      cert_pass: ''
    }, size: 5
  )

def perform(post_id, device_id, message)
    APNOTIC_POOL.with do |connection|
      notifications = Redis::HashKey.new(
        "notifications:#{post_id}", marshal: true
      )
      object = JSON.parse(message)
      noti = Apnotic::Notification.new(object['token'])
      noti.alert = object['alert']
      noti.badge = 0
      noti.sound = 'default'
      noti.topic = 'xxx'
      noti.custom_payload = object['custom_data']
      response = connection.push(noti)

      raise 'Timeout sending a push notification' unless response

      if response.status == '410' ||
         (response.status == '400' &&
          response.body['reason'] == 'BadDeviceToken')
        notifications.delete(device_id.to_s)
        # TODO: SHOULD REMOVE THE DEVICE FROM NotificationDevice
      end
    end
  end

Is there a reason that JRuby isn't supported at https://github.com/ostinelli/apnotic/blob/master/lib/apnotic.rb#L9, or is it that it just hasn't been tested? Looking through the dependencies it looks like both connection_pool and http-2 both support jruby and test against it...

Been seeing this in production. I assume the root cause is the connection was remotely closed, but this looks like a bug in the error handling.

vendor/bundle/ruby/2.3.0/gems/http-2-0.8.2/lib/http/2/connection.rb:659:in `connection_error': undefined method `message' for nil:NilClass (HTTP2::Error::ProtocolError)
    from vendor/bundle/ruby/2.3.0/gems/http-2-0.8.2/lib/http/2/connection.rb:430:in `connection_management'
    from vendor/bundle/ruby/2.3.0/gems/http-2-0.8.2/lib/http/2/connection.rb:225:in `receive'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:109:in `block in socket_loop'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:105:in `loop'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:105:in `socket_loop'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:88:in `block (2 levels) in ensure_open'

Caused: vendor/bundle/ruby/2.3.0/gems/http-2-0.8.2/lib/http/2/connection.rb:659:in `connection_error': undefined method `message' for nil:NilClass (NoMethodError)
    from vendor/bundle/ruby/2.3.0/gems/http-2-0.8.2/lib/http/2/connection.rb:430:in `connection_management'
    from vendor/bundle/ruby/2.3.0/gems/http-2-0.8.2/lib/http/2/connection.rb:225:in `receive'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:109:in `block in socket_loop'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:105:in `loop'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:105:in `socket_loop'
    from vendor/bundle/ruby/2.3.0/gems/net-http2-0.12.1/lib/net-http2/client.rb:88:in `block (2 levels) in ensure_open'

Using Apnotic 0.10.0

Hi. According to the Apple Docs: https://developer.apple.com/library/content/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/PayloadKeyReference.html#//apple_ref/doc/uid/TP40008194-CH17-SW5

I can use an action-loc-key field to customize the texts shown in action buttons in push notifications.
I've been trying this:

notification       = Apnotic::Notification.new(token)
        notification.alert = {
          body: alert,
          action-loc-key: 'Accept'
          }

but action-loc-key is an invalid key in Ruby.
Any ideas?

Will like to write integration tests it will be easier if we had the DummyServer implementation available also.

The README states:

...it is recommended to use a queue engine that will retry unsuccessful pushes.

However, the example worker raises errors only in the case of a timeout, and swallows any unsuccessful response codes/reasons other than those indicating an invalid device.

Is it recommended to raise on any of the many other possible errors? While some clearly indicate a problem with the notification and won't recover on retries, others indicate server problems or other temporary conditions that may resolve, allowing automatic retries (such as sidekiq's default error-handling) to eventually succeed.

I'm happy to PR a README modification, but wondering if there's a clear best practice here.

Hi!

WDYT about supporting input hashes in #push and similar methods? I've found that there is new field thread-id in notification body.

If gem would support something like this, it'll be possible to use it without modification after every new field added:

connection.push(
  headers: {
    'apns-id' => apns_id,
    ...
  },
  body: {
    alert: { ... },
    'thread-id' => 1,
  }
)

And Notification can still be supported. It may have #to_h, returning {headers: , body:}, and there will be single check notification = notification.to_h if notification.is_a?(Notification).

If you like the idea, I can prepare PR for it.

UPD. forgot about :token field.

when i'm trying to call : response = connection.push(notification) i got EOFError: end of file reached. Is this some kind of bug or did i make a mistake in my code / connection?

NB: i use the same code like in documentation, sry for my bad english

Taking a look at the connection pool example, after we check out a connection, we have to wait for the full request-response cycle before any other thread can make a request on that connection, which isn't necessary with HTTP/2. I'd suggest Apnotic::Connection#push having an option to return a Future immediately so we can put the connection back into the pool as soon as possible, and then we can await the Future after that. That's the way the pushy library for Java does it it seems

Hello, I'm running into an issue where my Sidekiq process is crashing after some period of time when attempting to use a connection from a pool. I'm not entirely sure if it's an issue with Apnotic, with how I'm managing the connection pool, or with my own code being thread safe. I would appreciate any help, or maybe even pointing in the right direction for how to determine exactly what the issue is.

The stack trace is

2016-06-24T20:19:17.083Z 11526 TID-ovumo5ako Deliveries::IosDeliveryWorker JID-f2ac3377caf5893f8a8941cb BID-hZm_HvyLaQMLZw INFO: start
2016-06-24T20:19:17.084Z 11526 TID-ovuna5ck0 Deliveries::IosDeliveryWorker JID-a45d7efd9590715caccd576f BID-hZm_HvyLaQMLZw INFO: start
Connection reset by peer
/Users/eebs/.rubies/ruby-2.3.1/lib/ruby/2.3.0/openssl/buffering.rb:178:in `sysread_nonblock'
/Users/eebs/.rubies/ruby-2.3.1/lib/ruby/2.3.0/openssl/buffering.rb:178:in `read_nonblock'
/Users/eebs/.gem/ruby/2.3.1/gems/net-http2-0.11.1/lib/net-http2/client.rb:90:in `block in socket_loop'
/Users/eebs/.gem/ruby/2.3.1/gems/net-http2-0.11.1/lib/net-http2/client.rb:87:in `loop'
/Users/eebs/.gem/ruby/2.3.1/gems/net-http2-0.11.1/lib/net-http2/client.rb:87:in `socket_loop'
/Users/eebs/.gem/ruby/2.3.1/gems/net-http2-0.11.1/lib/net-http2/client.rb:71:in `block (2 levels) in ensure_open'

This is happening after I:

• Start Sidekiq
• Enqueue a few notifications
• Wait 30-60 minutes
• Try and send a few more notifications

This is crashing the entire sidekiq process, not just a single job.

Here is a paired down version of the code I'm running with hopefully all the important bits left in:

ios_delivery_worker.rb

Each IosDeliveryWorker is responsible for sending one push notification to the target token belonging to the given native app identified by app_id, and recording the response. It retrieves a native app specific Apnotic::ConnectionPool from the PoolManager::Ios class stored in the POOLS constant. It uses the connection pool to send the notification as suggested in the Readme.

class IosDeliveryWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'push', :backtrace => true

  attr_reader :token, :app_id, :payload

  def perform(token, app_id, paylod)
    @token   = token
    @app_id  = app_id
    @payload = payload

    response = publish_message
    update_delivery(response)
  end

  POOLS = PoolManager::Ios.new

  private

  def publish_message
    POOLS.for(app_id) do |pool|
      pool.with do |connection|
        notification = Apnotic::Notification.new(token)
        # ... assign payload, etc
        connection.push(notification)
      end
    end
  end

  def update_delivery(response)
    # ... update response, etc
  end
end

pool_manager/ios.rb

The PoolManager::Ios class is used to retrieve an Apnotic::ConnectionPool for a given native application. It uses a hash to store the connection pools, keyed by the native application's ID. Access to the hash is wrapped in a Mutex to protect against multiple Sidekiq threads from accessing the hash at the same time, and potentially corrupting it. This seems to work in practice, as I can see that only one connection pool is created (by logging when creating the connection pool), but concurrent processing is an area in which I am less familiar. Do you think something I'm doing while managing the pools is causing a problem?

module PoolManager
  class Ios

    attr_reader :pool_size

    def initialize(pool_size=5)
      @store = {}
      @mutex = Mutex.new
      @pool_size = pool_size
    end

    def for(id)
      @mutex.synchronize do
        pool = @store[id]

        unless pool
          app = IosApp.find(id)
          push_url = app.production? ? Apnotic::APPLE_PRODUCTION_SERVER_URL : Apnotic::APPLE_DEVELOPMENT_SERVER_URL

          pool = Apnotic::ConnectionPool.new({
            url: push_url,
            cert_path: Paperclip.io_adapters.for(app.certificate),
            cert_pass: app.certificate_password
          }, size: pool_size)
          @store[id] = pool
        end

        yield pool
      end
    end
  end
end

Will the connections in the pool become unusable after some period of time? It seems they are closing or being reset, and then crashing the Sidekiq process when they are used again. I see that there is a close method on Apnotic::Connection, but if I close each connection after pushing the notification, I see a tremendous decrease in throughput (from ~200ms per job to ~1600ms per job).

I'm not entirely sure what the issue is, but I believe that the connections in the pool are getting closed or disconnected, and the error happens when they are attempted to be used again. If this is the case, it would be great if the connection pool would be able to recover from closed connections rather than crashing.

Sorry for the overload of information, but I hope I have given you enough context to help me find the issue. If there is anything else I can provide, please let me know. I would also happily submit a PR if I knew what the issue was and I was able to fix it.

Thanks for the great library!

n = Apnotic::Notification.new(token)
n.priority = 10.to_s

It would be nicer if you could just write: n.priority = 10

I am a bit at a loss because we've been running into what I consider somewhat of a thread-safety/isolation issue. We have an AMQP queue based architecture that we use to send out pushes (to APN and other push systems) where we are seeing a lot of these errors:

Errno::ETIMEDOUT(Connection timed out):
  /opt/ruby-2.3.1/lib/ruby/2.3.0/openssl/buffering.rb:178:in `sysread_nonblock'
  /opt/ruby-2.3.1/lib/ruby/2.3.0/openssl/buffering.rb:178:in `read_nonblock'
  net-http2 (0.14.0) lib/net-http2/client.rb:116:in `block in socket_loop'
  net-http2 (0.14.0) lib/net-http2/client.rb:113:in `loop'
  net-http2 (0.14.0) lib/net-http2/client.rb:113:in `socket_loop'
  net-http2 (0.14.0) lib/net-http2/client.rb:93:in `block (2 levels) in ensure_open'

(this is the full trace)

The problem is this is happening in unrelated processors and making them fail hard, like the GCMProcessor as an example. To give you a very rough idea we are effectively using one connection per processor:

class Http2PushConnection
  @connections = {}

  def self.send_push(notification)
    response = nil
    Retry.retry_on_exception(max_retries: 2, wait_s: 0.2) do
      get_connection(notification.notification_env) do |connection|
        response = connection.push(notification, timeout: 2 * CONNECT_TIMEOUT)
      end
    end
    response
  end

  def self.get_connection(env)
    @connections[env] = establish_connection(env) unless @connections.key?(env)

    conn = @connections[env]
    yield(conn)
    if close_connection?(env)
      conn.close
      @connections.delete(env)
    end
  end
end

class ApnsHttp2Processor < AmqpProcessor
  def process
    notification = build_notification(...)
    Http2Connection.send_push(notification)
  end
end

class GCMProcessor < AmqpProcessor
  def process
    // Send push to Google's GCM service, for example
  end
end

I don't know how to handle this one in the application layer.

• Catching it: not possible as far as I can see
• Retrying: not possible because some of our processors are not (and cannot be fully) idempotent

Any ideas? Any input is welcome.

n.alert = "hi"
puts n.body
{"aps":{"alert":"hi"}}

n.custom_payload = payload_apns
puts n.body
{"aps":{"alert":"hi"},"aps":{"alert":"Hello","sound":"default"}}

"aps" is now twice in the JSON

MDM push notifications don't contain the standard body, the only include:

  {"mdm":"A47EA72E-0A82-4B05-8ADE-5EEB3F103EB1"}

where "A47EA72E-0A82-4B05-8ADE-5EEB3F103EB1" is the "push magic" token of the device.

Doing some light testing with this gem and one thing I'm seeing is that after a certain length of time, connections to APNS time out, regardless of what the timeout parameter is set to. (I think the default timeout is 30s, while it takes ~15 minutes to time out here. Here is what the backtrace fails with: [2].

Are there any best practices for keeping the connection alive over a sporadically used connection? And if the connection does fail eventually, will the HTTP2 client take care of reconnects in pooled mode, as the ConnectionPool library says it should [1]? Testing this scenario right now but figured I'd ask here as well. Using dev servers, for what it's worth.

[1] https://github.com/mperham/connection_pool
[2] https://gist.github.com/j0sh/a215eebb8f1f0cc4d6f6270bc02df352

• Add notice to clarify that Apnotic is expected to raise an error when these are encountered, so that a job manager can restart a job that errored out.
• Clarify that there's no timeout option in async pushes.

E.g., if I by accident use a dev. certificate while connecting to production, I receive some weird
HTTP2::Error::ProtocolError: undefined method 'message' for nil:NilClass

Apple is now allowing developers to use a JWT token instead of a cert:
https://developer.apple.com/library/content/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/APNsProviderAPI.html

Go implementation here:
RobotsAndPencils/buford#63

Interesting things about this new approach:

1. you need the topic.
2. you can push to all apps in the org. (hence why you need the topic).
3. you can only open 1 stream until you send a push. at that point apple will advertise a new max # of streams to you.

I've just started using Apnotic behind my APNS Proxy app (basically an HTTP endpoint that can receive notifications and send them over to Apple).

Under normal usage our worker works like this:

1. We make a connection to APNS.
2. We keep the connection object around and use it each time a notification needs to be sent.
3. If there's an error sending a notification on the connection, we catch the error, disconnect and make a new connection.

Everything is mostly working very well except every few hours we get socket errors triggered outside of the normal flow.

rake aborted!
SocketError: Socket was remotely closed
/app/vendor/bundle/ruby/2.3.0/gems/net-http2-0.14.1/lib/net-http2/client.rb:104:in `rescue in block (2 levels) in ensure_open'
/app/vendor/bundle/ruby/2.3.0/gems/net-http2-0.14.1/lib/net-http2/client.rb:98:in `block (2 levels) in ensure_open'
EOFError: end of file reached
/app/vendor/bundle/ruby/2.3.0/gems/net-http2-0.14.1/lib/net-http2/client.rb:122:in `block in socket_loop'
/app/vendor/bundle/ruby/2.3.0/gems/net-http2-0.14.1/lib/net-http2/client.rb:119:in `loop'
/app/vendor/bundle/ruby/2.3.0/gems/net-http2-0.14.1/lib/net-http2/client.rb:119:in `socket_loop'
/app/vendor/bundle/ruby/2.3.0/gems/net-http2-0.14.1/lib/net-http2/client.rb:99:in `block (2 levels) in ensure_open'
Tasks: TOP => apns_proxy:worker

From looking at the source for HTTP2 & Apnotic, these seem to be raised within a thread that has abort_on_exception set to true. Whenever these errors occur, the whole process dies with no option for us to catch the error and handle it gracefully.

For APNS Proxy, we have a single worker which maintains multiple connections to Apple - one for each application that uses it so the whole worker dying because a single connection has failed isn't desirable.

The actual worker code can be found here if you're interested: https://github.com/adamcooke/apns-proxy/blob/master/lib/apns_proxy/worker.rb

Any help would be appreciated.

I'm trying to use socket error callback and NoMethodError thrown during the execution of it during a Sidekiq job.

connection.on(:error) do |exception|
  logger.error("exception has been raised: #{exception}")
end

Also, tried from documentation:

connection.on(:error) { |exception| puts "Exception has been raised: #{exception}" }

Gem versions:

    apnotic (1.3.1)
      connection_pool (~> 2.0)
      net-http2 (>= 0.17, < 2)

Edit:
I'm using the token based APNS method. I can see :on when I log connection.methods

Are you receptive to a modification of the Apnotic::Connection initializer so that a certificate string can be provided in lieu of a file? This provides additional flexibility as a certificate can then be pulled from sources other than local storage and a workaround such as initiating a StringIO object is not necessary.

If you are, please provide attribute naming guidance and I will submit a pull request.

In the last 2 days, our production server has been raising "SocketError: Socket was remotely closed" whenever it tries to push to APN server.
I didn't update production source code for weeks. It suddenly happens on all environments: production, staging, development.
I also checked all Apple certificates. They are valid (not expired).

I am not alone. There are 2 people reporting about this issue on Apple forums:
https://forums.developer.apple.com/thread/106078

I really appreciate if you know any thing about this issue.

I was having an issue where my sidekiq workers were leaking memory when sending notifications using apnotic.

The source of the leak is indicated in the code below. It's related to the use of a connection pool when sending notifications.

  def perform(token)
    # build the notification outside of the connection block.
    # if you build the notification inside the block memory associated with it is never released.
    notification       = Apnotic::Notification.new(token)
    notification.alert = "Hello from Apnotic!"

    APNOTIC_POOL.with do |connection|
      # making a notification inside the block will leak the object.
      #  notification       = Apnotic::Notification.new(token)
      #  notification.alert = "Hello from Apnotic!"

      response = connection.push(notification)
      raise "Timeout sending a push notification" unless response

      if response.status == '410' ||
        (response.status == '400' && response.body['reason'] == 'BadDeviceToken')
        Device.find_by(token: token).destroy
      end
    end
  end

We send such a large volume of notifications that we'd end up using over 1GB of memory every 12 hours which would eventually run us out of memory.

The above example is the synchronous code, but the same is true with async. You need to build the notification outside of the pool or you leak.

I've just updated to 1.4.0 version (I did that accidentally, I wanted to use 1.3.1), and I started getting this error:
LocalJumpError: no block given (yield)

for this code:

apnotic_pool.with do |connection|
      logger.info "Sending push notification to APNS"
end

I would like to include handling for timeout situations when using .push_async and would appreciate some guidance on how best to accomplish this.

To provide some context, we first discovered timeout-related issues in our own project upon using Apnotic's connection.join. The join was never returning and there were also notifications that had been queued via .push_async that had not run their on(:response) yet. Our knee jerk solution is to wrap .join in a timeout block and run .close shortly after, whether a timeout occurs or not.

This got me to thinking... if two notifications are sent with the async method, are they always delivered in sequence? If there is some sort of delay with sending the first one, does the second one stay stuck? My guess is that it depends upon how the http2 streams multiplex the notifications. Is this accurate?

Ultimately, I'm hoping to find a robust way to time out these problems and retry but would greatly benefit from some insights on the best way to approach it with this library.

Hello

I'm trying to use apnotic in development/sandbox environment with the Push notification sandbox server like so:

connection = Apnotic::Connection.new(cert_path: "certificate.pem", url: "https://gateway.sandbox.push.apple.com:2195")

& then setting up a notification as shown in the README.

However, when I try to send the notification via
response = connection.push(notification), I get a nil response, in other words response is a nil object. According to the documentation this is what happens if the notification request times out, so I tried setting the timeout to something like 30 seconds, and it still returned nil & quickly too (not waiting 30 secs to return nil).

Any idea what this means? I'm using a sandbox/development cert, which appears to be in order given that Apnotic returns a failure if you don't have the right type of cert.

Not sure what to look for next. I might try switching to a production cert but that kind of defeats the purpose of using the sandbox.

Are there any other circumstances where sending a notification would return nil instead of details on the response? Any suggestions welcome.

We were using the async method to send notifications and while sending it to around 40K devices on a single connection we got the HTTP2::Error::StreamLimitExceeded exception.

Can anyone explain what this error means? Is there a limit to the amount of data I can send on a HTTP/2 connection? A part of the notifications did go out, and the remaining failed.

We were on version 0.10.0.

I'd have an instance which is kept around for the lifetime of my application, and I'd like to have a single connection be open. I've been playing with the code, and I'm not sure where the @connection.join line should go, or if it's even required - my app seems to function fine without it. It actually didn't perform as well when following each push notification, but that just may be push notification performance (it's always hard to say). Also, are there any problems in the way my code is setup?

I appreciate your support.

class PushNotificationManager
  def initialize()
    # create a persistent connection
    if development
      @connection = Apnotic::Connection.development(cert_path: "PushCertDev.pem")
    else
      @connection = Apnotic::Connection.new(cert_path: "PushCertDist.pem")
    end
    # wait for all requests to be completed
    # @connection.join
  end

  def clean_up
    # close the connection
    @connection.close
  end

  def send_message_to_device(message, device_token)
    # Sends the message asynchronously
    # If this causes problems then it can be done synchronously instead

    notification = Apnotic::Notification.new(device_token)
    notification.alert = {
      title: message.from_user_name,
      body: message.text,
      action: 'View'
    }
    notification.content_available = true
    notification.category = 'new'
    notification.custom_payload = {
      text: message.text,
      message_id: message.message_id,
      from_user_id: message.from_user_id,
      to_user_id: message.to_user_id
    }
    notification.topic = 'ProjectDent.TwIMTopic'

    # prepare push
    push = @connection.prepare_push(notification)
    push.on(:response) do |response|
      reason = nil

      if response.body.class == Hash
        reason = response.body['reason']
      end

      if response.ok?
        yield true, reason
        # Reason is logged - may be nil, but just in case there is ever a reason while still saying it's ok.
        TwIMLog.log('push notification', "sent notification with message id: #{message.message_id}, to device with token: #{device_token}, reason: #{reason}")
      else
        yield false, reason
        TwIMLog.log('push notification', "failed to send notification with message id: #{message.message_id}, to device with token: #{device_token}, reason: #{reason}")
      end
    end

    # send
    @connection.push_async(push)
  end
end

Hi,

The sample code in readme for push_async is not working resulting the following error:

NoMethodError: undefined methodprepare_push' for #<Apnotic::Connection:`

Should I use different commit for push_async?

Currently the ConnectionPool returns only the PRODUCTION objects of Apnotic::Connection. There's no option to return development objects in case someone wants to use the connection pool with development objects.

After the Errno::ECONNRESET - Connection reset by peer error
the remote_max_concurrent_streams changed to 0 (the @client.remote_settings[:settings_max_concurrent_streams] == 0x7fffffff)
and the request enters into an infinite loop.

def delayed_push_async(push)
until streams_available? do
sleep 0.001
end
@client.call_async(push.http2_request)
end

def streams_available?
remote_max_concurrent_streams - @client.stream_count > 0
end

Within the last 24 hours we've been unable to send push notifications in development/sandbox environment.

It looks like one issue may be that the development url used by apnotic doesn't match apple's docs.

https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns#overview has:

* Development server: api.sandbox.push.apple.com:443
* Production server: api.push.apple.com:443

While the current master branch of apnotic sets APPLE_DEVELOPMENT_SERVER_URL to "https://api.development.push.apple.com:443"

However, setting the url to https://api.sandbox.push.apple.com:443 does not fix the issue.

We observe the issue with a stack trace like this:

Exception: SocketError: Socket was remotely closed
--
0: /Users/jason/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/net-http2-0.16.0/lib/net-http2/client.rb:130:in `callback_or_raise'
1: /Users/jason/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/net-http2-0.16.0/lib/net-http2/client.rb:115:in `rescue in block (2 levels) in ensure_open'

We just implemented Apnotic and are receiving SocketError: Socket was remotely closed when we try to send a notification. How do we debug?

I keep getting this issue when trying to send notifications, anyone stumpled upon this before?

EOFError: end of file reached
    from /Users/alexander/.rbenv/versions/2.2.3/lib/ruby/2.2.0/openssl/buffering.rb:182:in `sysread_nonblock'
    from /Users/alexander/.rbenv/versions/2.2.3/lib/ruby/2.2.0/openssl/buffering.rb:182:in `read_nonblock'
    from /Users/alexander/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/net-http2-0.13.3/lib/net-http2/client.rb:111:in `block in socket_loop'
    from /Users/alexander/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/net-http2-0.13.3/lib/net-http2/client.rb:108:in `loop'
    from /Users/alexander/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/net-http2-0.13.3/lib/net-http2/client.rb:108:in `socket_loop'
    from /Users/alexander/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/net-http2-0.13.3/lib/net-http2/client.rb:93:in `block (2 levels) in ensure_open'

I created an apns token for my app, and then uninstalled and reinstalled my app to get a 2nd token. I am getting 200 return codes for both tokens, when I was expecting a 400 for one of them.

I'm sure its something I am doing wrong, any help appreciated. The notification is being sent by a resque scheduler task call to a model method.

module ApplePush

  @queue = :apple_push

  def self.perform()
    ApnsNotification.where(:sent_time => nil).each do |notification|
      notification.send_to_apple
    end
  end

end


require 'apnotic'

class ApnsNotification < ApplicationRecord

  APNOTIC_POOL = Apnotic::ConnectionPool.new({
    cert_path: Rails.root.join("config", "certs", "MyCert.p12"),
    cert_pass: "mypassword"
  }, size: 5)

   def send_to_apple
       Resque.logger.info "   sending message: '" + self.message + "' to " + self.token

       stripped_token = self.token.delete(' <>')

       Resque.logger.info "   stripped token: " + stripped_token

       APNOTIC_POOL.with do |connection|

         notification          = Apnotic::Notification.new(stripped_token)
         notification.alert    = self.message
         notification.topic    = "mybundle"

         begin
            response = connection.push(notification)
         rescue Exception => e
            Resque.logger.info "        exception: " + e.message
         end

         Resque.logger.info response.to_json

         if response.status == '410' ||
            (response.status == '400' && (response.body['reason'] == 'BadDeviceToken'))

            Apns.where(:token=>self.token).each do |apns|
               Resque.logger.info "   destroying old apns: " + apns.device_id + " " + apns.token
               apns.destroy
            end
         end

         self.sent_time = Time.now.to_i
         self.save

       end
   end

end


I, [2018-05-17T01:48:50.109327 #6351]  INFO -- :    sending message: 'Eric desk unit: Lost communication' to <a37a55f4 11c7ec74 8aa35d53 08f1ff39 781b1ae1 fbda8db5 11dd9933 7ca16a46>
I, [2018-05-17T01:48:50.109515 #6351]  INFO -- :    stripped token: a37a55f411c7ec748aa35d5308f1ff39781b1ae1fbda8db511dd99337ca16a46
I, [2018-05-17T01:48:50.533902 #6351]  INFO -- : {"headers":{":status":"200","apns-id":"14965060-43c3-4585-b443-21342ad93ca9"},"body":""}
I, [2018-05-17T01:48:50.539204 #6351]  INFO -- :    sending message: 'Eric desk unit: Lost communication' to <1d867019 f5d67a87 ec876092 b52af730 49771704 7a93ff57 48cb1619 56a4dc18>
I, [2018-05-17T01:48:50.539322 #6351]  INFO -- :    stripped token: 1d867019f5d67a87ec876092b52af730497717047a93ff5748cb161956a4dc18
I, [2018-05-17T01:48:50.609411 #6351]  INFO -- : {"headers":{":status":"200","apns-id":"99457b80-0876-4bcb-a546-946fdbaefff2"},"body":""}