fterh / sneakpeek Goto Github PK

View Code? Open in Web Editor NEW

102.0 11.0 17.0 175 KB

Reddit bot to preview and post hyperlinks as comments

License: MIT License

Python 99.50% Dockerfile 0.50%

reddit reddit-bot article-extractor news-articles preview

sneakpeek's People

Contributors

Stargazers

Watchers

Forkers

yleong magicalbeacon rickithadi darrenwee iedr wesleyzxc endian675 soemyatmyat gbacalso chungnicholas redditporean midicronica mcx bayar21 sirciel dontstopnowbaby u1i

sneakpeek's Issues

Remove unused files and dependencies

cnalifestyle.channelnewsasia.com has title only (no body)

https://www.reddit.com/r/singapore/comments/al7bbh/the_story_of_homegrown_bak_kwa_specialist_lim/efbetpl

Program stops running after a while without error/exception

Unlike the previous crash issues, this one has no error/exception in the "logs". I'm still monitoring the situation, and compiling all the "logs" (nohup.out) to see if there's a common pattern.

WIll update when I have more info.

Today article has title only (no body)

See https://www.reddit.com/r/singapore/comments/abp0zv/why_we_are_pushing_for_divestment_from_fossil/ed1zbz8/

Refactor config options to environment variables

Run-time settings should be configurable without having to modify source code.

Warn when env var `SUBREDDIT` is not set

I have tried to simply some of the code, have a look.

see:
https://github.com/NoahSee/reddit_commenter
https://www.reddit.com/r/test/comments/ammfbr/testing/

Detect and handle premium Straitstimes articles

Program still crashes

Here is the output copied off the end of nohup.out:

Sleeping
Starting main
Starting scan
Exception occurred while scanning
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/bin/invoke", line 11, in <module>
    sys.exit(program.run())
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/program.py", line 363, in run
    self.execute()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/program.py", line 532, in execute
    executor.execute(*self.tasks)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/executor.py", line 129, in execute
    result = call.task(*args, **call.kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/tasks.py", line 128, in __call__
    result = self.body(*args, **kwargs)
  File "/root/sneakpeek/tasks.py", line 7, in start
    c.run("ENV=prod python main.py")
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/context.py", line 94, in run
    return self._run(runner, command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/context.py", line 101, in _run
    return runner.run(command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 271, in run
    return self._run_body(command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 365, in _run_body
    raise ThreadException(thread_exceptions)
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (OSError):


Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>},
 'target': <bound method Runner.handle_stdin of <invoke.runners.Local object at 0x7fb67916bba8>>}

Traceback (most recent call last):

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/util.py", line 233, in run
    super(ExceptionHandlingThread, self).run()

  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 641, in handle_stdin
    data = self.read_our_stdin(input_)

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 601, in read_our_stdin
    bytes_ = input_.read(bytes_to_read(input_))

OSError: [Errno 9] Bad file descriptor

Ricemedia handler unable to handle links with tracking query

While handling https://www.reddit.com/r/singapore/comments/asyfwk/, ricemedia handler threw an uncaught exception and caused the program to crash

Operating on submission ID: asyfwk
Submission qualifies
Traceback (most recent call last):
  File "main.py", line 33, in <module>
    schedule.run_pending()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 493, in run_pending
    default_scheduler.run_pending()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 78, in run_pending
    self._run_job(job)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 131, in _run_job
    ret = job.run()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 411, in run
    ret = self.job_func()
  File "main.py", line 22, in start
    scan(reddit.subreddit(config.SUBREDDIT))
  File "/root/sneakpeek/scan.py", line 21, in scan
    comment_raw = handler.handle(submission.url)
  File "/root/sneakpeek/handlers/RicemediaHandler/__init__.py", line 28, in handle
    unwrapped_body = re.search(f"{start_marker}(.+?){end_marker}", soup.text).group(1)
AttributeError: 'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/bin/invoke", line 11, in <module>
    sys.exit(program.run())
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/program.py", line 363, in run
    self.execute()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/program.py", line 532, in execute
    executor.execute(*self.tasks)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/executor.py", line 129, in execute
    result = call.task(*args, **call.kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/tasks.py", line 128, in __call__
    result = self.body(*args, **kwargs)
  File "/root/sneakpeek/tasks.py", line 7, in start
    c.run("ENV=prod python main.py")
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/context.py", line 94, in run
    return self._run(runner, command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/context.py", line 101, in _run
    return runner.run(command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 271, in run
    return self._run_body(command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 365, in _run_body
    raise ThreadException(thread_exceptions)
invoke.exceptions.ThreadException: 
Saw 1 exceptions within threads (OSError):


Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>},
 'target': <bound method Runner.handle_stdin of <invoke.runners.Local object at 0x7fe62970bb70>>}

Traceback (most recent call last):

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/util.py", line 233, in run
    super(ExceptionHandlingThread, self).run()

  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 641, in handle_stdin
    data = self.read_our_stdin(input_)

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 601, in read_our_stdin
    bytes_ = input_.read(bytes_to_read(input_))

OSError: [Errno 9] Bad file descriptor

Program does not resume running if scan() method throws

Lines 32-37 of main.py.

This happens when e.g. praw receives 504 HTTP response.

Special characters not escaped

This lead to accidental formatting issues. Example: This article used single * to censor vulgarities which lead to markdown making text in-between as italics

Edit: URL markdown.

Incorrect article.text parsing (ricemedia.co)

E.g. http://ricemedia.co/culture-events-sir-stamford-raffles-statue-disappeared/

Probably because the first chunk of text (before the image in the article):

Has 5 sentences < MIN_SENT_COUNT (default 7)
Has 113 words < MIN_WORD_COUNT (default 300)

Implement proper logging

Right now the program only prints bare lines to stdout/stderr. We ought to use a proper logging library that adds timestamps, log levels and log rotations.

Program crashes after running for a few days

The bot is running on an Ubuntu 18.10 x64 Digitalocean droplet using the command nohup python3 -m pipenv run invoke start &.

After a few days of running, it crashes. This is the output log:

Starting main
Starting scan
Traceback (most recent call last):
  File "main.py", line 33, in <module>
    schedule.run_pending()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 493, in run_pending
    default_scheduler.run_pending()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 78, in run_pending
    self._run_job(job)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 131, in _run_job
    ret = job.run()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/schedule/__init__.py", line 411, in run
    ret = self.job_func()
  File "main.py", line 22, in start
    scan(reddit.subreddit(config.SUBREDDIT))
  File "/root/sneakpeek/scan.py", line 12, in scan
    for submission in subreddit.new(limit=config.LIMIT):
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/praw/models/listing/generator.py", line 52, in __next__
    self._next_batch()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/praw/models/listing/generator.py", line 62, in _next_batch
    self._listing = self._reddit.get(self.url, params=self.params)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/praw/reddit.py", line 391, in get
    data = self.request('GET', path, params=params)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/praw/reddit.py", line 506, in request
    params=params)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/sessions.py", line 185, in request
    params=params, url=url)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/sessions.py", line 116, in _request_with_retries
    data, files, json, method, params, retries, url)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/sessions.py", line 101, in _make_request
    params=params)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/rate_limit.py", line 35, in call
    kwargs['headers'] = set_header_callback()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/sessions.py", line 145, in _set_header_callback
    self._authorizer.refresh()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/auth.py", line 328, in refresh
    password=self._password)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/auth.py", line 138, in _request_token
    response = self._authenticator._post(url, **data)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/prawcore/auth.py", line 31, in _post
    raise ResponseException(response)
prawcore.exceptions.ResponseException: received 503 HTTP response
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/bin/invoke", line 11, in <module>
    sys.exit(program.run())
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/program.py", line 363, in run
    self.execute()
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/program.py", line 532, in execute
    executor.execute(*self.tasks)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/executor.py", line 129, in execute
    result = call.task(*args, **call.kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/tasks.py", line 128, in __call__
    result = self.body(*args, **kwargs)
  File "/root/sneakpeek/tasks.py", line 7, in start
    c.run("ENV=prod python main.py")
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/context.py", line 94, in run
    return self._run(runner, command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/context.py", line 101, in _run
    return runner.run(command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 271, in run
    return self._run_body(command, **kwargs)
  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 365, in _run_body
    raise ThreadException(thread_exceptions)
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (OSError):


Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>},
 'target': <bound method Runner.handle_stdin of <invoke.runners.Local object at 0x7fbbf201aba8>>}

Traceback (most recent call last):

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/util.py", line 233, in run
    super(ExceptionHandlingThread, self).run()

  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 641, in handle_stdin
    data = self.read_our_stdin(input_)

  File "/root/.local/share/virtualenvs/sneakpeek-osnio7yv/lib/python3.7/site-packages/invoke/runners.py", line 601, in read_our_stdin
    bytes_ = input_.read(bytes_to_read(input_))

OSError: [Errno 9] Bad file descriptor

Handle articles with comment.body longer than 10000 characters

Set up automated code review in deployment pipeline

E.g. codebeat/codacy or any other similar product