Program still crashes about sneakpeek

[fterh]

This is happening very frequently now. I suspect it's to do with the read/write operations to the database file... perhaps now that the database file is growing larger in file size, read/write ops take longer and result in simultaneous accesses...?

Complete hunch. I'll look at this next week when I'm freer.

[fterh]

@yleong @xlanor I've been looking at main.py, and I'm wondering...

I'm catching the exception at line 24 except Exception as e:, as can be seen in the "log file": Exception occurred while scanning. Since the exception has been properly caught and handled, why does the Python process crash?

[yleong]

I'm not sure either, the exception handler at main.py:24 should have caught any exception in scan. I have just noticed that DatabaseManager.disconnect and the praw instantiation are not covered by the exception handler. Could the crashes be due to those lines? Can you share more crash logs if available? Maybe we can consider integrating with sentry.io rollbar so that we all can be notified of crashes and examine the exceptions. Or the graylog cluster would work too.

[xlanor]

Definitely, I'm currently tied down by my schedule but I can assist. my graylog cluster is still up, should we move the discussion to slack/gitter?

[yleong]

Sure. Do we have one setup?

[fterh]

@yleong No worries, I read the documentation for the schedule library, and turns out it's that. When an exception bubbles up in a thread task, it breaks the scheduler, so even though it was caught and handled in main.py, the scheduling thing breaks. The fix is documented here: https://schedule.readthedocs.io/en/stable/faq.html#what-if-my-task-throws-an-exception

@xlanor Yup, let's do that. You have my Telegram handle. No rush on the graylog cluster, I can push out a fix for the exception handling thing by the end of the week as perhaps v0.6.0-beta, and when graylog is up we can move to v1.0.

[changhuapeng]

@fterh If my understanding is correct, the Schedule library is used in this project to only scan for new submissions on the subreddit. Maybe you will want to replace the Schedule library with PRAW's own method of monitoring new submissions if the scheduling part is really causing all the crashes.

[fterh]

@fterh If my understanding is correct, the Schedule library is used in this project to only scan for new submissions on the subreddit. Maybe you will want to replace the Schedule library with PRAW's own method of monitoring new submissions if the scheduling part is really causing all the crashes.

That's super cool, never knew about that! Will give it a read, thanks :)

[xlanor]

@changhuapeng @yleong , we are getting a slack channel up and going for anyone to join and discuss about sneakpeak.
slack

Feel free to join!

[fterh]

@yleong I've pushed out 0.6.0-beta that uses Praw's mechanism for monitoring a subreddit. So far so good, will monitor for a bit and close this issue if it seems to work.

@xlanor for 1.0, since this issue is (hopefully) fixed already, I'm thinking we do logging and refactoring.

[xlanor]

@fterh

Sounds good. I'm wondering if you'd also like to automate the deployment process via travis, which is free for open source repositories. Part of my job scope at work involves setting up a deployment pipeline and infrastructure.

Maybe you can open a travis integration to this repository first as preparation, once my holidays roll around next month I'll definetly be more involved

[fterh]

Sure. I'm familiar with Travis, but I'd thought that it was unnecessary because I originally envisioned this to be a minimal maintenance/development project in the sense that once the handlers are up and running, it shouldn't require any form of maintenance or development.

Looks like that plan got slightly thwarted :P But I agree that CI/CD is a good v1.0 milestone. Do you want to create a new issue for that? I'm closing this.