adfinis / timed-backend

Django API for the Timed application

Timed Backend

Timed timetracking software REST API built with Django

Archived

This Repository has been merged into a monorepo and is therefore archived.

Installation

Requirements

• docker
• docker-compose

After installing and configuring those requirements, you should be able to run the following commands to complete the installation:

Add the timed.local entries to your hosts file:

echo "127.0.0.1 timed.local" | sudo tee -a /etc/hosts

Then just start the docker-compose setup:

make start

This brings up complete local installation, including our Timed Frontend project.

You can visit it at http://timed.local.

The API can be accessed at http://timed.local/api/v1 and the admin interface at http://timed.local/admin/.

The Keycloak admin interface can be accessed at http://timed.local/auth/admin with the account admin and password admin

Development

To get the application working locally for development, make sure to create a file .env with the following content:

ENV=dev
DJANGO_OIDC_CREATE_USER=True

If you have existing users from the previous LDAP authentication, you want to add this line as well:

DJANGO_OIDC_USERNAME_CLAIM=preferred_username

Import some development fixtures to get started:

make loaddata

The test data includes 3 users: admin, fritzm and axels with which you can log into http://timed.local. Their passwords are identical to the username.

To access the Django admin interface you will have to change the admin password in Django directly:

$ make bash
root@0a036a10f3c4:/app# poetry run python manage.py changepassword admin
Changing password for user 'admin'
Password: 
Password (again): 
Password changed successfully for user 'admin'

Then you'll be able to login in the Django admin interface http://timed.local/admin/.

Adding a user

If you want to add other users with different roles, add them in the Keycloak interface (as they would be coming from your LDAP directory). You will also have to correct their employment in the Django admin interface as it is not correctly set for the moment. Head to http://timed.local/admin/ after having perform a first login with the user. You should see that new user in the Employment -> Users. Click on the user and scroll down to the Employments section to set a Location. Save the user and you should now see the Timed interface correctly under that account.

Configuration

Following options can be set as environment variables to configure Timed backend in documented format according to type.

Parameter	Description	Default
DJANGO_ENV_FILE	Path to setup environment vars in a file	.env
DJANGO_DEBUG	Boolean that turns on/off debug mode	False
DJANGO_SECRET_KEY	Secret key for cryptographic signing	not set (required)
DJANGO_ALLOWED_HOSTS	List of hosts representing the host/domain names	not set (required)
DJANGO_HOST_PROTOCOL	Protocol host is running on (http or https)	http
DJANGO_HOST_DOMAIN	Main host name server is reachable on	not set (required)
DJANGO_DATABASE_NAME	Database name	timed
DJANGO_DATABASE_USER	Database username	timed
DJANGO_DATABASE_HOST	Database hostname	localhost
DJANGO_DATABASE_PORT	Database port	5432
DJANGO_OIDC_DEFAULT_BASE_URL	Base URL of the OIDC provider	http://timed.local/auth/realms/timed/protocol/openid-connect
DJANGO_OIDC_OP_AUTHORIZATION_ENDPOINT	OIDC /auth endpoint	{DJANGO_OIDC_DEFAULT_BASE_URL}/auth
DJANGO_OIDC_OP_TOKEN_ENDPOINT	OIDC /token endpoint	{DJANGO_OIDC_DEFAULT_BASE_URL}/token
DJANGO_OIDC_OP_USER_ENDPOINT	OIDC /userinfo endpoint	{DJANGO_OIDC_DEFAULT_BASE_URL}/userinfo
DJANGO_OIDC_OP_JWKS_ENDPOINT	OIDC /certs endpoint	{DJANGO_OIDC_DEFAULT_BASE_URL}/certs
DJANGO_OIDC_RP_CLIENT_ID	Client ID by your OIDC provider	timed-public
DJANGO_OIDC_RP_CLIENT_SECRET	Client secret by your OIDC provider, should be None (flow start is handled by frontend)	not set
DJANGO_OIDC_RP_SIGN_ALGO	Algorithm the OIDC provider uses to sign ID tokens	RS256
DJANGO_OIDC_VERIFY_SSL	Verify SSL on OIDC request	dev: False, prod: True
DJANGO_OIDC_CREATE_USER	Create new user if it doesn't exist in the database	False
DJANGO_OIDC_USERNAME_CLAIM	Username token claim for user lookup / creation	sub
DJANGO_OIDC_EMAIL_CLAIM	Email token claim for creating new users (if DJANGO_OIDC_CREATE_USER is enabled)	email
DJANGO_OIDC_FIRSTNAME_CLAIM	First name token claim for creating new users (if DJANGO_OIDC_CREATE_USER is enabled)	given_name
DJANGO_OIDC_LASTNAME_CLAIM	Last name token claim for creating new users (if DJANGO_OIDC_CREATE_USER is enabled)	family_name
DJANGO_OIDC_BEARER_TOKEN_REVALIDATION_TIME	Time (in seconds) to cache a bearer token before revalidation is needed	60
DJANGO_OIDC_CHECK_INTROSPECT	Use token introspection for confidential clients	True
DJANGO_OIDC_OP_INTROSPECT_ENDPOINT	OIDC token introspection endpoint (if DJANGO_OIDC_CHECK_INTROSPECT is enabled)	{DJANGO_OIDC_DEFAULT_BASE_URL}/token/introspect
DJANGO_OIDC_RP_INTROSPECT_CLIENT_ID	OIDC client id (if DJANGO_OIDC_CHECK_INTROSPECT is enabled) of confidential client	timed-confidential
DJANGO_OIDC_RP_INTROSPECT_CLIENT_SECRET	OIDC client secret (if DJANGO_OIDC_CHECK_INTROSPECT is enabled) of confidential client	not set
DJANGO_OIDC_ADMIN_LOGIN_REDIRECT_URL	URL of the django-admin, to which the user is redirected after successful admin login	dev: http://timed.local/admin/, prod: not set
DJANGO_ALLOW_LOCAL_LOGIN	Enable / Disable login with local user/password (in admin)	True
EMAIL_URL	Uri of email server	smtp://localhost:25
DJANGO_DEFAULT_FROM_EMAIL	Default email address to use for various responses	webmaster@localhost
DJANGO_SERVER_EMAIL	Email address error messages are sent from	root@localhost
DJANGO_ADMINS	List of people who get error notifications	not set
DJANGO_WORK_REPORT_PATH	Path of custom work report template	not set
DJANGO_SENTRY_DSN	Sentry DSN for error reporting	not set, set to enable Sentry integration
DJANGO_SENTRY_TRACES_SAMPLE_RATE	Sentry trace sample rate, Set 1.0 to capture 100% of transactions	1.0
DJANGO_SENTRY_SEND_DEFAULT_PII	Associate users to errors in Sentry	True
HURRICANE_REQ_QUEUE_LEN	Django Hurricane's request queue length. When full, the readiness probe toggles	250
STATIC_ROOT	Path to the static files. In prod, you may want to mount a docker volume here, so it can be served by nginx	/app/static
STATIC_URL	URL path to the static files on the web server. Configure nginx to point this to $STATIC_ROOT	/static

Contributing

Look at our contributing guidelines to start with your first contribution.

License

Code released under the GNU Affero General Public License v3.0.

timed-backend's Issues

Static files don't load on test aks

Static files for django admin don't load on the test aks.
Maybe it's an issue with the STATIC_ROOT not being correct.
Gunicorn PR

Drop CustomerPassword

S324:

timed/subscription/admin.py:27:38: S324 Probable use of insecure hash functions in hashlib: md5 | 25 | password = self.cleaned_data.get("password") 26 | if password is not None: 27 | self.instance.password = hashlib.md5(password.encode()).hexdigest() | ^^^^^^^^^^^ S324 28 | return super().save(commit=commit) |

class CustomerPassword(models.Model):
    """Password per customer used for login into SySupport portal.

    Password are only hashed with md5. This model will be obsolete
    once customer center will go live.
    """

Can this be dropped, respectively ignored and dropped in a separate PR?
Since https://github.com/adfinis/customer-center does exist

I'm not 100% if this is still in use or not. It looks like it was part of the old SySupport portal and might have been replaced with a proper solution when the portal was migrated to being an Ember based frontend.

If this is in fact the case then we should create an issue to track the codes removal. Maybe @winged or @trowik know if it's still needed?

I'm also not 100% sure, but I'd say it's safe to remove it in another PR.

Originally posted by @trowik in #1049 (comment)

Add "Auto-Submitted" mail header to mail notifications

Timed generates notification e-mails such as "Verification of reports". Those mails may lead to bounces in case one has an e-mail auto-responder active (such as out-of-office notifications), replying to a non-existing sender address (such as [email protected]).

To prevent auto responders from replying to notifications, the Auto-Submitted: mail header should be included in such e-mails.

Quoting RFC 3834:

The purpose of the Auto-Submitted header field is to indicate that the message was originated by an automatic process, or an automatic responder, rather than by a human; and to facilitate automatic filtering of messages from signal paths for which automatically generated messages and automatic responses are not desirable.
[...]
Automatic responses SHOULD NOT be issued in response to any message which contains an Auto-Submitted header field (see below), where that field has any value other than "no".

I hope that I opened this feature request issue on the correct repository, otherwise feel free to move it.

Automate pushing container images to ghcr.io

Currently there is no automation for publishing container images.

Replace uses of str.format with fstrings

Currently we still have str.format, this can be replaced with the more readable f-strings

str.format

class Attendance(models.Model):
    """Attendance model.

    An attendance is a timespan in which a user was present at work.
    Timespan should not be time zone aware hence splitting into date and
    from resp. to time fields.
    """

    date = models.DateField()
    from_time = models.TimeField()
    to_time = models.TimeField()
    user = models.ForeignKey(
        settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name="attendances"
    )

    def __str__(self) -> str:
        """Represent the model as a string."""
        return "{}: {} {} - {}".format(
            self.user,
            self.date.strftime("%Y-%m-%d"),
            self.from_time.strftime("%H:%M"),
            self.to_time.strftime("%H:%M"),
        )

f-string

class Attendance(models.Model):
    """Attendance model.

    An attendance is a timespan in which a user was present at work.
    Timespan should not be time zone aware hence splitting into date and
    from resp. to time fields.
    """

    date = models.DateField()
    from_time = models.TimeField()
    to_time = models.TimeField()
    user = models.ForeignKey(
        settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name="attendances"
    )

    def __str__(self) -> str:
        """Represent the model as a string."""
        return f"{self.user}: {self.date.strftime('%Y-%m-%d')} {self.from_time.strftime('%H:%M')} - {self.to_time.strftime('%H:%M')}"

f-string and without strftime

class Attendance(models.Model):
    """Attendance model.

    An attendance is a timespan in which a user was present at work.
    Timespan should not be time zone aware hence splitting into date and
    from resp. to time fields.
    """

    date = models.DateField()
    from_time = models.TimeField()
    to_time = models.TimeField()
    user = models.ForeignKey(
        settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name="attendances"
    )

    def __str__(self) -> str:
        """Represent the model as a string."""
        return f"{self.user}: {self.date:%Y-%m-%d} {self.from_time:%H:%M} - {self.to_time:%H:%M}"

Replace rest_condition with native DRF permission operators

Dep rest_condition is not needed anymore and can be replaced with new DRF permission operators which got released in DRF 3.9.

As Timed is also affected from issue we will have to wait for 3.9.1 to implement this.

Export Metrics

We should expose application metrics to help us pinpoint performance or other issues.

The way to go seems to be django-prometheus. The Prometheus metrics will integrate nicely with the Kube Prometheus Stack that is available for this on production.

Apart from exposing metrics in the backend we will also want to configure a PodMonitor to scrape them in the corresponding Helm chart.

• django-prometheus
• PodMonitor

don't pass today as arg

Passing today seems like something we'd want to take a closer look at and refactor in a diff PR.

Originally posted by @hairmare in #1071 (comment)

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

bug: remaining effort on initial report creation is buggy

Scenario

Create a report and add an estimated remaining effort to it.

Expected behavior

It updates the remaining effort of the certain task in the statistics view.

Actual behavior

After submitting the report the remaining effort does not update in stats view. But after editing the rem. effort and saving the report again in the timesheet, it will update the values in the stats view.

The attributes are fine in both requests to the backend. I suspect that a the POST and PATCH requests have different handling of the remaining effort attribute. Maybe the POST does not properly trigger the update?

bug(auth): requests to the api with an invalid token receive a response status 500 instead of 401

rename default branch from master to main

i think we should do this all over the place, most modern projects have already switched

Update from psycopg2 to psycopg >= 3

It's called psycopg again and theire is no psycopg3 package.

Afaict Django 4.2 made s ready for this update and it's time.

Compatibility with newer django rest framework >= 3.8.0

See the change log in #244:

Deprecated list_route & detail_route in favor of action decorator with detail boolean.

bug: reschedule reports leads to 500 error

Rescheduling reports from one day to another leads to a 500 error on the /api/v1/reports/40 endpoint.

Latest version (rc10)

The logs from the backend container display the following:

backend_1   | Internal Server Error: /api/v1/reports/40
backend_1   | Traceback (most recent call last):
backend_1   |   File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
backend_1   |     response = get_response(request)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
backend_1   |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
backend_1   |     return view_func(*args, **kwargs)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
backend_1   |     return self.dispatch(request, *args, **kwargs)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
backend_1   |     response = self.handle_exception(exc)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
backend_1   |     self.raise_uncaught_exception(exc)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
backend_1   |     raise exc
backend_1   |   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
backend_1   |     response = handler(request, *args, **kwargs)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/rest_framework/mixins.py", line 82, in partial_update
backend_1   |     return self.update(request, *args, **kwargs)
backend_1   |   File "/app/timed/tracking/views.py", line 177, in update
backend_1   |     tasks.notify_user_rejected_report(instance, request.user)
backend_1   |   File "/app/timed/tracking/tasks.py", line 96, in notify_user_rejected_report
backend_1   |     _send_notification_emails([user_changes], reviewer, True)
backend_1   |   File "/app/timed/tracking/tasks.py", line 42, in _send_notification_emails
backend_1   |     connection.send_messages(messages)
backend_1   |   File "/usr/local/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages
backend_1   |     new_conn_created = self.open()
backend_1   |   File "/usr/local/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 67, in open
backend_1   |     self.connection.starttls(keyfile=self.ssl_keyfile, certfile=self.ssl_certfile)
backend_1   |   File "/usr/local/lib/python3.9/smtplib.py", line 771, in starttls
backend_1   |     raise SMTPNotSupportedError(
backend_1   | smtplib.SMTPNotSupportedError: STARTTLS extension not supported by server.
backend_1   | 500 PATCH /api/v1/reports/40 (172.24.0.1) 346.06ms

@trowik This seems to be connected to the rejected report feature.

Upgrade PostgreSQL to a non EOL version

As per 923f8e2 we seem to be using a PostgreSQL version that was EOLed February 13, 2020.

We should upgrade to a version that is not EOL, basically all versions >9.4 are still in support.

See here for the available versions and their support policy: https://www.postgresql.org/support/versioning/

Once updated the Helm chart for timed also needs updating. In fact this issue surfaced during updating/implementing the chart: adfinis/timed-chart#9 (comment)

Format code with black

Format code with black:
https://github.com/ambv/black

This way #327 can be fixed as well.

User assignment on various objects

We should have users assigned on various objects, to allow some more elaborate permissions and planning functions.

• Customers: Assigned users should then also be able to create / manage projects for this customer
• Projects: Assigned users should be able to manage tasks
• Tasks: Assigned users can be used for "light-weight" resource planning.

Ideally, I think each assignment should contain a "role" or "type" field to describe the relation (such as "manager", or "resource"). This way, one could be assigned to a project as a resource for planning purposes, or as a manager for permission purposes.

In a first step, only the assignment should be done - the permissions would probably best be done in a separate PR / feature.

Introduce type hinting

The code should provide type hints where they make sense without overloading the code with hard to maintain typing information.

Ruff has ignore-untyped rules that we actived, in several cases satisfying the rules with some simple hints will improve the code base and further aid modern IDEs.

PEPs

• PEP 563 – Postponed Evaluation of Annotations

Rename default branch to main

also needs to be updated in references like ci config

Tracking of remaining effort

We propose to add the possibility to track remaining effort estimates on Timed Tasks while right next to the field where spent time is tracked (to save horizontal space, "Not billable" and "Needs review" would be replaced by icon buttons).

Using this information, we can add the total remaining effort to project tooltips and task tooltips.

Provided that everyone working on a project updates the remaining effort estimates on the tasks they are working on, the project tooltip yields an always up-to-date remaining effort estimate. The "statistics" page of timed can then be extended to show the budget (red dashed line), spent time (blue bar) and remaining effort estimate (shallow red or green bar).

Since the remaining effort estimates are only useful for some projects (flat fee or cost ceiling, medium to large size), we'd add a setting to the "Projects" screen of timed where PMs can activate / deactivate the function on a per-project basis (default setting: deactivated). For projects without activated remaining effort tracking the extra field in the tracking screen would not be displayed.

What goals are to be achieved

• PMs can react earlier when budget overshoot becomes apparent -> easier to take appropriate measures (e.g. talk to customer)
• More transparency: Everyone in the company can see if specific projects are "on track" without having to talk to the PM

remove secret from settings.py

Replace raw Content-Disposition header creation with util

In Django 4.2 there is django.utils.http.content_disposition_header() which can help us ensure that the headers are RFC 6266 compliant.

Previous Code

response["Content-Disposition"] = f"attachment; filename={name}"

New Code

from django.utils.http import content_disposition_header

response["Content-Disposition"] = content_disposition_header(as_attachment=True, filename=name)

I'm not sure if this is the canonical way to call it, but it does add unicode support to the filename and hides some implementation specifics.

I'm not sure if its worth it, so feel free to close this if you don't think it is.

Originally posted by @hairmare in #1072 (comment)

Improve configuration

I took a look at your instructions for installation, settings.py and #8
I believe this can be improved.

Further lecture: https://12factor.net/config

Environment variables

You use a single .env file.
I would define .env-files for each environment: .env.development, .env.test, .env.build etc.
Each of them contains the configuration settings for each environment.
To avoid repeating the configuration definitions, a .env.base may be defined, the more specific .env.* files then define a more specific set of settings.
I would use .env just to override the configuration settings of the other files (and for production - i.e. there should be no .env.production).
Also add a .env.template which contains the required configuration variables to deploy the application, to simplify operations' tasks (when creating the .env.production).
I would also use dotenv to load these environment variables.

Modes

It would be great if the settings for each environment were kept in classes using django classy settings.
Then you can define the settings required in production only in the production class (for instance a sentry backend).
This also allows to set defaults per environment (like the secret key, the database host, etc.) in the class - yet I do not recommend to do so and rather keep the settings in the environment specific .env-file (read further lecture).

editing rejected report results in weird mail

Issue

At first I had a Report actually rejected and got this mail:

Some of your reports have been rejected by a reviewer. Please get in contact
with them to clarify the reports. Most likely, you will just need to move
the reports to the correct project / task.

Reviewer: <Actualy Reviewer>


Date: mm/dd/yyyy
Duration: 7:00 (h:mm)
Task: x > y > z
Comment: <comment>
---

Then after i edited the Report i got this mail:

Some of your reports have been rejected by a reviewer. Please get in contact
with them to clarify the reports. Most likely, you will just need to move
the reports to the correct project / task.

Reviewer: <Myself>


Date: mm/dd/yyyy
Duration: 7:00 (h:mm)
Task: x > y > z
Comment: <edited comment>
---

After every edit i got another mail, where I am marked as the reviewer.