Comments (19)
psofiterol
commented on September 27, 2024
1
Hey @phumpal, do we have any updates on this?
Only asking cause potentially having some log lines not reach us as notifications is kind of concerning/stressing for us ;S
from pybrake.
phumpal
commented on September 27, 2024
1
Now, is it safe to assume that this is the cause of these errors (as the aforementioned routes/route_metrics/queries are part of that)?
AFAICT a rather small # of requests to APM endpoints seem to be timing out. I've made a few minor changes on our backend and will observe impact over the next few days.
@smurf-U is there anything we can do in pybrake to more robustly handle timeout w/retries or optionally enabling more verbose logging (of status codes) while we work on the backend?
How was this not spotted?
Good question! [email protected] is routed through HubSpot where our CSM (support) team manages customer interactions. I suspect communicating through two different mediums (GitHub and HubSpot) between two teams introduced some oversight. My apologies.
from pybrake.
phumpal
commented on September 27, 2024
@psofiterol I can probably guess which / project(s) but I may be wrong. Would you mind sending those to [email protected] and referencing this issue?
from pybrake.
psofiterol
commented on September 27, 2024
Hey @phumpal, we've opened a thread with [email protected] in the first place, we were encouranged to post here. Replied in that thread with the info requested 👍
from pybrake.
phumpal
commented on September 27, 2024
@psofiterol Our CSM team typically handles all support correspondence; sorry was out of the loop here. I got the info I need and will take a look, thanks!
from pybrake.
phumpal
commented on September 27, 2024
@psofiterol whoops - I got shuffled into other things. Apologies for timing out :)
Can you send an example of a notice (error) failure?
from pybrake.
psofiterol
commented on September 27, 2024
@phumpal We don't have evidence that any errors failed to produce notifications, but that's why we're worried. We don't know if the pybrake ERRORS (see original issue description) are thrown while trying to forward a notification to airbrake or are independently occurring.
from pybrake.
phumpal
commented on September 27, 2024
The pybrake errors are confusing and I'm wondering if it's not an issue with how pybrake handles responses from the backend when APM data for the accout becomes capped.
I'm curious if there is more recent output from pybrake in your logs? Perhaps 5/22, 5/23 and 5/24
from pybrake.
psofiterol
commented on September 27, 2024
Hey @phumpal, the errors were not contained to the original dates, we keep getting them. Here are the most recent ones:
Note: These are just from one environment of one project, but the situation is the same across other env/projects as well.
2022-05-29 10:08:43,441 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 10:08:43,442 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 10:25:04,162 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 10:25:14,165 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 10:31:04,440 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 10:32:14,501 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 10:32:14,799 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 10:32:24,509 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 11:09:06,244 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 11:09:16,267 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 12:11:09,815 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 12:26:00,999 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 12:53:22,125 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 12:53:32,139 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 13:00:52,587 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 13:01:02,789 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 13:01:02,789 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 13:01:02,934 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 13:04:33,177 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 13:56:06,101 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 13:56:16,112 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 14:13:06,144 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 14:13:06,145 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 15:28:01,074 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:11:53,597 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:13:32,881 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:13:33,675 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 16:13:33,676 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:13:52,899 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 16:13:52,911 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:35:03,865 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:36:24,764 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:39:04,021 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 16:39:04,029 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:39:05,066 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:39:14,940 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:39:44,061 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:49:05,598 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:49:15,515 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:51:14,621 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 16:51:15,631 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 17:19:27,231 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 17:47:47,762 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 18:10:39,105 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 18:10:49,106 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 18:11:01,109 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 18:11:10,340 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 18:36:20,566 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 19:01:13,279 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 20:24:36,729 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 21:08:20,415 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 21:08:20,417 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 21:38:51,681 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 21:42:32,359 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 21:42:41,156 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 21:42:41,916 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 22:35:34,748 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 22:59:05,322 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 23:00:35,401 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 23:09:25,867 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 23:13:16,879 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 23:35:47,123 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 23:35:58,115 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 23:35:58,117 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 23:36:07,133 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 23:36:08,135 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-29 23:36:17,180 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-29 23:45:17,621 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 00:16:09,396 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 00:38:20,715 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 00:38:21,598 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 00:56:51,797 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 01:00:22,001 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:19:27,005 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:19:27,456 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:23:57,252 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:41:28,177 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:49:59,227 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:50:08,345 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:51:18,384 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:51:29,292 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:51:29,292 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:51:38,764 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:51:48,406 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:51:48,407 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:51:49,318 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:52:08,804 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:52:19,345 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:52:38,453 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:52:48,460 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:53:28,500 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:53:29,472 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:54:09,454 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:54:09,454 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:54:18,539 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:54:18,540 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:54:19,461 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:54:28,935 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:54:48,951 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:54:58,579 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:55:09,520 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:55:19,023 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:55:28,640 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:55:38,633 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:55:58,625 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:55:59,540 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:56:08,625 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:56:08,626 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:56:29,056 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:58:19,158 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:58:19,158 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:59:08,788 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:59:08,789 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:59:09,200 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 02:59:09,201 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 02:59:09,673 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:04:19,072 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:06:20,053 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:08:39,317 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:08:39,325 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:08:39,748 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:11:39,926 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:12:00,393 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:12:00,394 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:12:09,956 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:18:20,310 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:18:20,762 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:18:20,762 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:18:40,330 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:18:40,349 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:18:50,791 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:18:50,792 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:22:20,980 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:22:30,124 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 03:22:30,141 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 03:22:40,581 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 06:01:39,348 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 06:10:10,089 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 06:10:10,090 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 06:33:11,021 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 06:33:11,166 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 06:33:21,016 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 07:00:12,468 routes.py:119 pybrake ERROR - The read operation timed out
2022-05-30 07:00:12,468 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 07:00:12,552 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-30 07:39:54,825 route_metric.py:112 pybrake ERROR - The read operation timed out
from pybrake.
andylamp
commented on September 27, 2024
@phumpal, since its been 3 days since your last update - any luck with these logs or you require more info from us? We also upgraded to 1.4 to see if things improve (but so far they have not).
from pybrake.
psofiterol
commented on September 27, 2024
Also attaching some fresh logs with pybrake 1.4.0 installed
2022-06-02 12:48:03,249 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 12:49:34,296 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 12:49:34,299 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 12:49:38,322 queries.py:98 pybrake ERROR - The read operation timed out
2022-06-02 12:49:39,557 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 12:50:33,390 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 12:50:44,445 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 12:54:24,571 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 12:54:33,631 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 12:54:43,610 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 12:56:33,715 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 13:01:54,989 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 13:02:04,301 queries.py:98 pybrake ERROR - The read operation timed out
2022-06-02 13:02:04,311 queries.py:98 pybrake ERROR - The read operation timed out
2022-06-02 13:02:04,409 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 13:02:14,016 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 13:02:55,117 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 13:03:34,098 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 13:03:35,072 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 13:04:15,123 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 13:05:25,275 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 13:05:35,186 routes.py:119 pybrake ERROR - The read operation timed out
2022-06-02 13:07:44,344 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 13:07:45,291 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-06-02 13:08:05,423 routes.py:119 pybrake ERROR - The read operation timed out
from pybrake.
phumpal
commented on September 27, 2024
We don't know if the pybrake ERRORS (see original issue description) are thrown while trying to forward a notification to airbrake or are independently occurring.
Based on the output provided
2022-05-16 13:00:11,383 route_metric.py:112 pybrake ERROR - The read operation timed out
2022-05-16 13:03:30,420 routes.py:119 pybrake ERROR - The read operation timed out
seems like pybrake is having issues w/the APM endpoints. Looking at our logs your project does not seem to be having issues sending errors.
- To confirm logs are UTC?
- Does your account or project have any caps set?
- Has the account reached limits?
- Do you have logs prior to 5/16?
We're working with our team to reproduce.
from pybrake.
psofiterol
commented on September 27, 2024
seems like pybrake is having issues w/the APM endpoints. Looking at our logs your project does not seem to be having issues sending errors.
Right, that's good to know.
- To confirm logs are UTC?
Affirmative, all the logs posted here are UTC.
- Does your account or project have any caps set?
We have a cap in place for the account.
- Has the account reached limits?
AFAIK this has only happened once, last Autumn. Due to the cap we stopped getting airbrake logging completely so we almost immediately realized this. This hasn't happened recently though. And we've been getting the errors for 2 billing cycles now so even if this had to do with the cap it would probably have stopped 10 days ago when the cycle refreshed - alas that's not the case.
- Do you have logs prior to 5/16?
Unfortunately no, I believe we don't retain logs going that far back.
from pybrake.
smurf-U
commented on September 27, 2024
@phumpal I tried reproducing the issue with pybrake (1.2.0, 1.4.0), python 3.9, and Django 4.0.4, but I was unable to do so.
- Does this problem affect only route metrics, or does it affect error/query/queue notification as well?
- Could you please try making a curl request from the same environment?
- Could you please run the sample example and test route on that project once?
from pybrake.
psofiterol
commented on September 27, 2024
Good news. Following the questions from @smurf-U (and seeing that we're having this problem for routes, route_metrics and queries), I went back into the account for a more careful look.
When answering @phumpal questions earlier today I was only checking the Error monitoring service, not Performance monitoring. This has a cap in place as well and we are hitting it.
Now, is it safe to assume that this is the cause of these errors (as the aforementioned routes/route_metrics/queries are part of that)?
Also I have to ask you this. The first action [email protected] took was ask and receive our permission to log into our account to assess the situation. How was this not spotted?
from pybrake.
smurf-U
commented on September 27, 2024
@phumpal If necessary, we can raise the timeout on the Pybrake side. https://github.com/airbrake/pybrake/blob/master/src/pybrake/routes.py#L115
If you notice that it doesn't work after the backend modification, please let me know and I'll release a new release with a timeout of 5 to 10 minutes.
from pybrake.
smurf-U
commented on September 27, 2024
@psofiterol Do you still have this problem? If not, we can close this issue.
from pybrake.
psofiterol
commented on September 27, 2024
@psofiterol Do you still have this problem? If not, we can close this issue.
As mentioned previously, the cause was us going over our quotas. Safe to close.
from pybrake.
smurf-U
commented on September 27, 2024
Thanks @psofiterol for update. Please feel free to open issue 🙂
from pybrake.
Related Issues (20)
- Pybrake query notifier throws error when query is a psycopg2 Composed object HOT 1
- [PROPOSAL] Disable stats calls by default HOT 5
- Setting "level" in log config does not change what is sent to airbrake HOT 1
- m
- TypeError: filter() missing 1 required positional argument: 'record' caused by socket timeout HOT 9
- Turn Notifier class into a singleton? HOT 2
- Setup Airbrake for your Python application
- Pypi Source not working HOT 1
- 400 Bad Request: Did not attempt to load JSON data because the request Content-Type was not 'application/json'.
- Errors filtered out when middleware involve in backtrace HOT 1
- Usage outside of Django fails due to Django references HOT 2
- App fails to launch requiring pybrake middleware
- Fastapi/Sanic middleware does not support lack of SqlAlchemy package
- Django project not alerting airbrake on logger.error HOT 4
- How to notify from behind a proxy
- Logging integration not working in lambdas
- get_full_name optional on AbstractBaseUser HOT 1
- Airbrake Starlette middleware throws error when running behind a reverse proxy
- Pybrake raises depecration warnings with the FastAPI middleware HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pybrake.