olegeech-me / mvcr-application-checker

A set of services to check MVCR application status

MVCR Residential Application Status Notifier

This project offers automation for the residential application status from the Ministry of Interior of the Czech Republic. It diligently tracks changes in status and immediately notifies users of any updates. Users can conveniently subscribe to these notifications via the Telegram bot MVCR Status Řízení Bot. After providing their application details, the integrated monitor service will fetch and compare the application status at periodic intervals to keep users informed.

Modules and Architecture

The project is architecturally divided into two primary modules:

1. Telegram Bot Module

Bot Features

• Telegram Bot: This consists of the main Telegram bot code equipped with various command handlers.
• Database: The bot is integrated with a PostgreSQL database to store user data and application details.
• RabbitMQ Integration: It's integrated with a RabbitMQ server, which aids in the generation and consumption of fetch/refresh requests and responses for the fetchers.
• Application Monitor: This feature periodically scans the database, generating refresh requests whenever an application's last timestamp exceeds the configured time period.

2. Fetcher Module

Fetcher Features

• Messaging: This class listens to fetch/refresh queues on RabbitMQ.
• Browser Class: Leverages Selenium and Firefox to emulate user actions, thereby fetching or refreshing each application status.
• Application Processor: Employed for each new request to ensure streamlined and efficient data processing.

The Fetcher module is engineered with scalability and resilience in mind. It's designed for multi-instance deployment, ensuring that even if one instance encounters issues, others can seamlessly continue the task. This design choice significantly minimizes service interruptions.

For enhanced security and data integrity, the RabbitMQ server (within the Bot module) and its client (within the Fetcher module) employ SSL certificates, ensuring encrypted traffic and safeguarded data.

Getting Started

To make use of this service, simply visit the Telegram Bot link and follow the instructions to subscribe.

Acknowledgments

This project stands as a testament to open-source collaboration and the ongoing efforts to simplify and automate bureaucratic processes for foreigners in the Czech Republic.

Credits

Big thanks to Inessa Vasilevskaya for her major contributions to this project. Thanks to Fedir "Theo" L. (https://theodorthegreathe.mojeid.cz/) for providing Ukraine translations.

Development

Please see the Development Guide for more information.

mvcr-application-checker's Issues

Leading 000 zeros in the application might need be filtered out

Update/refresh requests should be requeued after certain timeout

Otherwise "Message 30873ed8795ec3a84aeaac207ea0e37e has already been published. Skipping." will be always seen.

Add handler for messages, when out of subscribe scenario

Users should be offered with a little helper message

/remind feature

Introduce /settimer or /remind

User can set time for which they will get daily notification of their current application status
/unsettimer or /forget removes reminder

Initial welcome message is not translated when switching language

When use joins and /start command is executed, switching language doesn't render start message in the selected lang, only subscribe helper is translated

Users can put non-valid year

When parsing years from the args or a single full input, year should be validate against "allowed years"

Ugly network error traceback

Periodically, when the telegram's httpx service cannot reach API, this mega-ugly dump shows up in the logs:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/telegram/ext/_updater.py", line 628, in _network_loop_retry
    if not await action_cb():
  File "/usr/local/lib/python3.10/site-packages/telegram/ext/_updater.py", line 333, in polling_action_cb
    raise exc
  File "/usr/local/lib/python3.10/site-packages/telegram/ext/_updater.py", line 322, in polling_action_cb
    updates = await self.bot.get_updates(
  File "/usr/local/lib/python3.10/site-packages/telegram/ext/_extbot.py", line 553, in get_updates
    updates = await super().get_updates(
  File "/usr/local/lib/python3.10/site-packages/telegram/_bot.py", line 394, in decorator
    result = await func(self, *args, **kwargs)  # skipcq: PYL-E1102
  File "/usr/local/lib/python3.10/site-packages/telegram/_bot.py", line 3542, in get_updates
    await self._post(
  File "/usr/local/lib/python3.10/site-packages/telegram/_bot.py", line 482, in _post
    return await self._do_post(
  File "/usr/local/lib/python3.10/site-packages/telegram/ext/_extbot.py", line 335, in _do_post
    return await super()._do_post(
  File "/usr/local/lib/python3.10/site-packages/telegram/_bot.py", line 510, in _do_post
    return await request.post(
  File "/usr/local/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 168, in post
    result = await self._request_wrapper(
  File "/usr/local/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 288, in _request_wrapper
    raise exc
  File "/usr/local/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 278, in _request_wrapper
    code, payload = await self.do_request(
  File "/usr/local/lib/python3.10/site-packages/telegram/request/_httpxrequest.py", line 243, in do_request
    raise NetworkError(f"httpx.{err.__class__.__name__}: {err}") from err
telegram.error.NetworkError: httpx.ReadError:

Suffix should be restricted to 2 digits only

Otherwise this can happen:
2023-10-12 12:33:57,607 - bot.handlers - INFO - [SUBSCRIBE] Adding application OAM-24164-2023/TP-2023 for chat_id: 5211378, first_name: XXX

/admin_broadcast function to send news to subscribers

This function will be useful for sending any new bot features or scheduled bot maintenance periods.

DB race conditions

Example:

2023-09-26 02:56:19,094 - bot.rabbitmq - INFO - Received status update message: {'chat_id': 129963852, 'number': '12345', 'suffix': '4', 'type': 'TP', 'year': 2023, 'last_updated': '2023-09-25T20:45:04.145405', 'status': 'Řízení s číslem jednacím OAM-12345/TP-2023 je stále ve stavu zpracovává se. Více o tom, co můžete dělat, se dozvíte na stránce věnované průběhu správního řízení.'}
2023-09-26 02:56:19,094 - bot.rabbitmq - INFO - Reply received for message ID 3f73007367967edd42fae1577afe10e1
2023-09-26 02:56:19,094 - bot.database - ERROR - Error while fetching application status for chat ID: 129963852. Error: cannot perform operation: another operation is in progress
2023-09-26 02:56:19,094 - bot.rabbitmq - ERROR - Failed to get current status from db for user 129963852
2023-09-26 02:56:19,097 - bot.rabbitmq - INFO - Status didn't change for user 1374884023 application
2023-09-26 02:56:19,097 - bot.database - INFO - Updating last_updated timestamp for chatID 1374884023 in DB

Write unit tests

:/

Unit tests coverage sucks :-/

we need to cover more core functions with unit tests ...

Limit how many applications fetcher takes

The maximum number of applications to take for processing at one should be limited per fetcher.

add /force_refresh command

This will request immediate status refresh, but should be limited only to 1 use a day

latency check error trace stack

2023-10-14 03:04:38,679 - asyncio - ERROR - _GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=ClientConnectorError(ConnectionKey(host='frs.gov.cz', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None), ConnectionResetError(104, 'Connection reset by peer'))>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "uvloop/loop.pyx", line 2084, in create_connection
  File "uvloop/loop.pyx", line 2079, in uvloop.loop.Loop.create_connection
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/src/fetcher/metrics_collector.py", line 94, in send_metrics
    await self.get_website_latency()
  File "/code/src/fetcher/metrics_collector.py", line 28, in get_website_latency
    async with session.get(self.url) as response:
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 1167, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 562, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host frs.gov.cz:443 ssl:default [Connection reset by peer]

user experience: any command should allow to break out of subscribe scenario

For now it seems only /start allows to do that

Fetchers should be monitored / provide statistics

• Fetcher should report when they connect, that should be visible to admins via bot admin commands
• Fetcher should provide statistics on how many successful / failed applications it had recently

Implement language switch mechanism

DB foreing key issue

2023-10-13 22:52:03,647 - bot.database - ERROR - Error while removing application 645, type MK, year 2023 for user 381159628. Error: update or delete on table "applications" violates foreign key constraint "reminders_application_id_fkey" on table "reminders"
2023-10-13 22:52:07,084 - bot.database - ERROR - Error while removing application 645, type MK, year 2023 for user 381159628. Error: update or delete on table "applications" violates foreign key constraint "reminders_application_id_fkey" on table "reminders"
2023-10-13 22:52:18,990 - bot.database - ERROR - Error while removing application 645, type MK, year 2023 for user 381159628. Error: update or delete on table "applications" violates foreign key constraint "reminders_application_id_fkey" on table "reminders"
2023-10-13 22:52:58,456 - bot.database - ERROR - Error while removing application 645, type MK, year 2023 for user 381159628. Error: update or delete on table "applications" violates foreign key constraint "reminders_application_id_fkey" on table "reminders"
2023-10-13 22:53:04,946 - bot.database - ERROR - Error while removing application 645, type MK, year 2023 for user 381159628. Error: update or delete on table "applications" violates foreign key constraint "reminders_application_id_fkey" on table "reminders"

Persistent log for Bot module

Previous logs vanish after docker compose rebuild

Force refresh requests are limited same way as regular refresh requests

Translation for menu buttons

Can be implemented using https://core.telegram.org/bots/api#setmycommands
Also, admin commands can be added via this method specifically for admins only

2024 missing

В боте проверки статуса заявления в ЧР нету выбора года 2024. Будете править ?

Allow tracking more than 1 application

Currently the /subcribe command allows you to input just 1 application, all subsequent invocations results in You are already subscribed message.
Unfortunately in real life it is often necessary to track applications of several family members (5 sounds like a reasonable number imho),

Subscribe conv handler should exit on any command

Fetcher dedup issues

2023-10-03 12:58:51,644 - fetcher.browser - INFO - Looks like connection error, will retry https://frs.gov.cz/informace-o-stavu-rizeni/ again later in 14
2023-10-03 12:59:05,651 - fetcher.browser - INFO - User-Agent for this request will be Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0
2023-10-03 12:59:05,733 - fetcher.browser - INFO - Initialized virtual display
2023-10-03 12:59:13,805 - fetcher.browser - ERROR - An error has occurred during page loading Message: Unable to locate element: .//div[text()='2033']

2023-10-03 12:59:16,497 - fetcher.application_processor - ERROR - Failed to fetch status for application number 25436
2023-10-03 12:59:16,497 - fetcher.application_processor - ERROR - Message exceeded max retries: {'chat_id': 1000043073, 'number': '25436', 'suffix': '0', 'type': 'DP', 'year': '2033', 'force_refresh': False, 'last_updated': '0'}
2023-10-03 12:59:16,530 - fetcher.messaging - INFO - Successfully published message to StatusUpdateQueue
2023-10-03 12:59:16,530 - asyncio - ERROR - Exception in callback <function _task_done at 0x7f93880a5dc0>
handle:
Traceback (most recent call last):
File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
File "/usr/local/lib/python3.8/dist-packages/aio_pika/tools.py", line 47, in _task_done
raise exc
File "/code/src/fetcher/application_processor.py", line 41, in fetch_callback
await self.process_fetch_request(message)
File "/code/src/fetcher/application_processor.py", line 105, in process_fetch_request
await self.end_processing(app_details["number"])
File "/code/src/fetcher/application_processor.py", line 34, in end_processing
self.processing_apps.remove(app_number)
KeyError: '25436'
2023-10-03 12:59:16,563 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-35' coro=<consumer() done, defined at /usr/local/lib/python3.8/dist-packages/aio_pika/queue.py:31> exception=KeyError('25436')>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/aio_pika/queue.py", line 37, in consumer
return await create_task(callback, message)
File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
File "/usr/local/lib/python3.8/dist-packages/aio_pika/tools.py", line 47, in _task_done
raise exc
File "/code/src/fetcher/application_processor.py", line 41, in fetch_callback
await self.process_fetch_request(message)
File "/code/src/fetcher/application_processor.py", line 105, in process_fetch_request
await self.end_processing(app_details["number"])
File "/code/src/fetcher/application_processor.py", line 34, in end_processing
self.processing_apps.remove(app_number)
KeyError: '25436'

Applications that are not present on MVCR site should be treated in a special way

Currently, if the status of application is "Řízení s číslem jednacím OAM-xxxxx-x/tt-yyyy nebylo nalezeno. Číslo jednací zadávejte bez úvodních nul (příklad: OAM-000123-XX/DP-2000)", application is resolved and periodic refresh is turned off. Sometimes this kind of applications are just "not there" yet, so it makes sense to let users know that and, possibly, lazy-poll status for the application within a month before backing off completely.

Improve subscribe user experience

Users should be able to subscribe via interactive dialog. It's hard to type in all the details after /subscribe

Deduplications of requests

Fetcher should group together similar requests

Fetcher reqeue issues

Quite often requests don't get requeued cuz of:
2023-10-15 17:25:25,740 - fetcher.browser - ERROR - [36992] An error has occurred during page loading: Couldn't fetch application status 2023-10-15 17:25:27,271 - fetcher.browser - WARNING - [36992] Fetch failed, retrying https://frs.gov.cz/informace-o-stavu-rizeni/ later in 11 seconds 2023-10-15 17:25:27,273 - fetcher.browser - ERROR - [36992] An error has occurred during page loading: Couldn't fetch application status 2023-10-15 17:25:27,274 - urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcbaf683580>: Failed to establish a new connection: [Errno 111] Connection refused')': /session/2482e46a-5e6f-48ea-a50b-5492387d088b/source 2023-10-15 17:25:27,275 - urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcbaf630490>: Failed to establish a new connection: [Errno 111] Connection refused')': /session/2482e46a-5e6f-48ea-a50b-5492387d088b/source 2023-10-15 17:25:27,276 - urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcbaf630a30>: Failed to establish a new connection: [Errno 111] Connection refused')': /session/2482e46a-5e6f-48ea-a50b-5492387d088b/source 2023-10-15 17:25:27,281 - fetcher.application_processor - ERROR - [36992/DP-2023][REFRESH] Error processing request: HTTPConnectionPool(host='127.0.0.1', port=38725): Max retries exceeded with url: /session/2482e46a-5e6f-48ea-a50b-5492387d088b/source (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcbaf65c490>: Failed to establish a new connection: [Errno 111] Connection refused')) 2023-10-15 17:25:27,282 - fetcher.application_processor - INFO - [36992/DP-2023][REFRESH] Unlocking, processing finished