horvathmarton / stock-buddy-api

REST API for the Stock Buddy portfolio monitoring platform.

Stock Buddy API

Development Notes

Getting Started

To start working with the Stock Buddy API, follow these steps:

1. Start PostgreSQL either as a service or container. If using Docker, run the following command:

docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=mysecretpassword postgres

2. Install a recent Python version (3.9 or higher) and pip.

3. Install PostgreSQL development dependencies for binary support.

4. Install the project's Python dependencies using pip:

python -m venv .venv
source ./.venv/bin/activate
pip install -r requirements.txt

5. Prepare your environment variables file based on the provided example.env file and load it.

6. (Destructive Action) Recreate the database schema:

python manage.py recreate_db

7. (Destructive Action) Perform the initial database migration:

python manage.py migrate

8. Start the Stock Buddy API by running the following command:

python manage.py runserver

Alternatively, you can start the Stock Buddy API using Docker Compose. Build and run it with the following command:

docker-compose build && docker-compose up

Contributing

We welcome contributions to the Stock Buddy API project. If you'd like to contribute, please read our Contribution Guidelines to get started.

License

This project is licensed under the MIT License - see the LICENSE file for details.

stock-buddy-api's Issues

Add extra dividend info fields to stock position snapshot

Admin dashboard is not exposed on deployment

Expected Behavior

/api/admin loads on both deployed instances. Any user with proper privileges could log in and use the admin dashboard.

Actual Behavior

/api/admin serves the fronted index.html.

Steps to Reproduce the Problem

1. Navigate to https://sb.noelsr.dev/api/admin.

Specifications

• Version: 1.0
• Platform: MacOS
• Browser: Chrome

Move the sync resource to a context manager

Description

Technical details

Add a full Postman documentation for the API

Description

We need detailed and up-to-date documentation about API usage for easier development.

The implementer of this task is going to create a full Postman docs about each endpoint of the API with easy-to-understand naming.

Technical details

• This issue should be started after #11 is done.
• Add as much automation for ease of use as possible (e.g.: the login endpoint saves the token for other requests etc.)
• Add some examples if applicable.
• Use self-explanatory naming for the requests and folder structure.
• Use environment vars for secrets and parameterize variable elements (e.g.: the base URL).

Automate release

Description

Technical details

Add a helper script for both the API and dashboard project.

Implement a feature flagging system

Description

Technical details

The dividend distribution section in portfolio summary contains empty values

Expected behavior

The server calculates the dividend distribution of the portfolio and attaches it to the response under the dividend_distribution key as a map. The key is the ticker and the value is the percentage value between 0 and 1. (The zero values are filtered from the map.

Actual behavior

The map is present in the response, but the values are all zero even for stocks that pay dividends.

Steps to Reproduce the Problem

1. Log in as a user with an existing portfolio.
2. Request a summary for the current day.
3. The response's dividend distribution section only contains 0 values.

Add the cash API

Description

We need an API to track the cash amounts in the user's accounts. It is a read-only API that uses a similar replay strategy on the transaction to calculate the cash size at a given data. It has to be split into different portfolios and currencies, but also provide a sum field.

Technical details

Stock transaction edit and delete

Description

Technical details

Apply new migrations automatically during deployment

Description

If the current deployment has new migrations on an instance the deployment should apply them to the database automatically.

The implementer should do some research what is the best way to achieve this task.

Sync SEC filings and company reports

Description

Technical details

Add the dashboard API

Description

We need to create an API for the dashboard section of the app. This is a high-level summary view of the current status of the user's entire portfolio.

Strategy

One sub-section of the dashboard API is strategy. In this section, we need two new models, a Strategy where we store asset distribution types (so a name and a percentage value and all the values in a strategy adds up to 100%) and a StrategyItem which is in many-to-one connection with the Strategy. The Strategy model has to have a visibility property that defines if other users can see it or not (defaults to private).

After this, we need a full CRUD for both resources. On the dashboard, we are going to have an editor where the user could create new strategies and edit/delete existing strategies. And we are going to have a recommendation view where the user could see its current strategy (this is calculated from the current asset allocation) and can select a targeted strategy (with a recommendation helper). The target and real strategy will be displayed side by side for the user to reference.

Portfolio indicators

The other part of the dashboard API is the portfolio indicators sub-section. This contains high-level info about the user's portfolio. Namely:

• largest position exposure (%)
• largest sector exposure (%)
• total AUM ($)
• gross capital deployed (% of AUM)
• total invested capital ($)
• total floating PnL ($)
• ROIC per year and since inception (%)
• annualized ROIC (%)

Add error reporting

Description

To avoid unhandled errors in the API we need to set up error reporting.

Technical details

This issue is going to need some research because Python might have a different error reporting methodology from Node. My recommendation is to use Sentry, because it is easy to use, has a lib for Python and the UI is user-friendly, but I'm not totally opinionated in this case.

The core points for the implementation:

• No error could "escape", we have to wrap the whole application with a try-catch (don't have to be exact if we could implement it with some kind of hook), to catch every unhandled exception in the app.
• The unknown exceptions should be handled graciously in the response, so the frontend receives a valid answer (even if error response) and no internal info is leaked to the client. A 500 response with a general message (e.g.: an unhandled exception occurred) will do. The frontend is responsible to handle it and display it properly for the user (the app should not break in this case either).
• Sentry has an option to disable reporting, this should be configured based on the environment variables. So e.g. we should not report errors while developing (if NODE_ENV === 'development' we don't report, just using the python equivalent).
• We should add an option to filter exceptions that are not in our interest. E.g. 401 and 404 exceptions are not really an error from the API monitoring point of view so we should not report it. In the final error handler, we should check the type and if some condition is met we shouldn't report it.
• The code should be scanned briefly for current error handling occurrences and the reporting should wire in where applicable. In this case, attach as many contexts as possible (e.g.: relevant variable values, class state, etc.). To have a concrete example the CSV parsing uses its own error handling, but if we encounter some special parsing error, we should report it to the error handling dashboard too.

Some reading resources for the implementer:

• https://docs.djangoproject.com/en/4.0/howto/error-reporting/
• https://docs.sentry.io/platforms/python/

Add version and commit hash info endpoint

Description

We need to be able to check the current version and deployed the commit hash for the API.

Technical details

• Research how to dynamically determine the commit hash of the latest commit in a controller.
• The version information is contained in version.py, we should import it from there.
• Expose an endpoint that contains the version information and current commit hash of the application.

Unify the API format (snake case or camel case)

Description

At the moment we use a mixed version of the snake case and camel case in the API. We need to do a short research and negotiate a unified API format based on some pros and cons.

The implementer should do research about industry standards, create a short doc about our convention and introduce the standardization on the existing codebase. (API, dashboard, and sycner).

Technical details

• The output of this task is documentation about the chosen method and some why-s.
• The existing code should be also modified to follow the new conventions.
• The affected parts:
  1. Route sections (these are possibly going to be kebab case)
  2. Query params
  3. Properties in payloads
• Pro for the camel case is that it is general in JS
• Pro for the snake case is that it is general in Python and databases

Add bonds API

Description

We need to expose an API for the user to handle bonds in Stock Buddy. This API is going to be really similar to its stock counterpart.

Technical details

• Create a model for bond, bond_transaction and bond_portfolio in the database.
• Expose API-s for both resources.
• We would only want to track interest payment and price info for these instruments so we need to extend the raw_data schema with these tables.
• We need to add endpoints for the syncer bot to insert stock price and payment data.
• We need to expose a bonds service where we could calculate bond portfolio AUM for the strategy service.
• Extend the bonds service with a snapshot calculation capability.

Create the CD flow

Use bandit in the linting process

Description

Bandit is a code analysis tool for python that could find logical errors in the codebase. We want to wire it up for the lining process.

Technical details

Bandit is already installed to the project and the check code is implemented, just commented out (it is a custom Django command: core/management/commands/lint.py). The task is to uncomment this check, make it run without an error and after that fix, every issue in the code, pointed out by bandit.

Add FX to raw data

Description

We should handle currencies as a dynamic entity instead of just hard coding USD, EUR, and HUF. We should have a separate table where we could add extra currencies dynamically. We also have to update their price information on a weekly basis instead of providing it manually. This way we could easily represent cryptocurrencies as well.

Technical details

• Have an extra column in each security table to provide which currency it is nominated in.
• We should have a table to list the supported currencies.
• We should have a price sync and price table to sync FX prices (e.g. relative to USD).
• We need to update our calculations to consider currency exchanges.

Add environment and dependency management

Description

E.g.: pipenv or poetry

Technical details

Stock portfolio API is read-only now

Expected behavior

An investor could create, update and delete their own stock portfolios. The administrator could manage everyone's portfolios.

Current behavior

The stock portfolio API is read-only. The user cannot create or modify a stock portfolio.

Technical details

The API is implemented in the StockPortfolioViewSet class.

We had to disable the editing of the portfolio when introducing the summary and detail endpoints because the serializer couldn't work with both payloads. The simplest solution to this issue is to select the serializer conditionally on the viewset based on the action. Also, don't forget to check if the serializer is active both in and out, so the input is validated and the output is serialized properly.

Add logging

Description

We need to add logging for the API for better visibility and debugging.

The implementer should:

1. Research the issue of logging and decide on a proper logging practice for the app.
2. After that, based on the guidelines they should apply the new guidelines to the existing codebase.
3. If all that is done a short doc should be added to describe the logging practice.

Technical details

• Don't spam info level and above, these should contain only the most relevant information that is useful to be collected during runtime.
• Everything should be logged, but use the TRACE and DEBUG levels excessively.
• The implementer should find a way how to enforce the active use of the new logging practice. One solution is to add it to the PR template's reviewer checklist, but if a better solution is found during the research, it is welcomed.

References before implementation:

• https://docs.djangoproject.com/en/4.0/topics/logging/
• https://docs.djangoproject.com/en/4.0/howto/logging/#logging-how-to

Can't delete portfolio with existing transactions

Expected Behavior

I can delete a portfolio with transactions. In this case, both the portfolio and the belonging transactions are removed.

Actual Behavior

The API throws an error when I try to delete a portfolio with transactions.

Steps to Reproduce the Problem

1. Log in as a general user
2. Create a new stock portfolio
3. Add some transactions to the portfolio
4. Try to delete the portfolio via the API

Specifications

• Version: 1.0
• Platform: MacOS
• Browser: Chrome

Annualized ROIC is not calculated correctly

Expected Behavior

The ROIC calculation reflects the return on the initially invested capital.

Actual Behavior

The ROIC calculation uses the current capital base which could contain realized PnL from closed positions.

Steps to Reproduce the Problem

1. Create a portfolio with a large, profitable, closed position.
2. Open new positions for the portfolio that are slightly at a loss.
3. The annualized ROIC will be negative, although it should be positive.

Technical details

We should deduct the invested capital from cash inflows rather than the size of investments when calculating the portfolio.

Sync raw data automatically via bots

Add support for extra auth types (SSO and 2FA)

Unify the route handling

Description

This is a pure refactoring task. No new feature or bugfix should be introduced here.

We need to unify the route handling in the API. At the moment we use the view handler types inconsistently, so short research should be done on which one is the most appropriate for our needs and replace each handler with that one.

Technical details

• Comment on this issue the whys of the choice.
• https://www.django-rest-framework.org/api-guide/views/
• https://www.django-rest-framework.org/api-guide/generic-views/
• https://www.django-rest-framework.org/api-guide/viewsets/

Add consistency check to raw data import

• Data continuity
• Big jumps in price or large dividend increase
• Schedule checks about the health of the data

Create the CI/CD flow

Description

We need to automate the run of the code quality checks and the deployment of the app to different environments (staging and production). We need to implement this automation on every repository (API, dashboard, syncer).

We need to run the unit test suite, code formatting, and lining process on push. Check the API tests merging into staging or production and release the app with proper versioning.

For access to the deployment target contact @horvathmarton.

Technical details

• I recommend using GitHub Actions as we have some quota with the premium subscription and the integration is simple here. But if the implementer has a strong other preference, we should discuss it.
• Check each step in the CI flow, so we could save some time in the actions. pylint and the whole test suite together is outstandingly slow now. If it could be optimized it should be done as part of this task.
• We shouldn't run the whole test suite on each push. We should separate the API tests from the traditional unit tests somehow and only run the unit tests on push and the API tests only before merging.
• We need another long-running branch next to master so we could separate the production and staging environment (it should be develop or staging or following some other convention). Contact @horvathmarton to protect the branch after creation. We shouldn't push to it nor to master directly.
• I have no preferences on version tracking, git tags seem a bit redundant in this flow, but nice and transparent, while adding version info to the merge commit is a simpler, but maybe less robust solution.

Introduce a time weighted return calculation methodology

Description

We need to have a refined and unified methodology to calculate portfolio performance. This service is also going to serve as a foundation for our soon-to-be performance view.

Technical details

• Study the linked docs to determine an algorithm for calculating portfolio performance OSV-portfolio-calculation-explanation.pdf and fit it to our current data modeling.
• Create a separate service to determine ROIC on a specific portfolio.
• Design the service so that on a later day we could adjust costs and calculate a portfolio value in a time series.

API testing

Description

API testing is the last part of the backend test suite that should be added before any refactoring could take place. We need to thoroughly test the current functionality using in-memory DB and the mock API client of Django.

We should add a test case for each endpoint's happy path (possibly multiple) and every edge case that seems to be relevant.

Technical details

• A couple of API tests have already been added as part of a fix (#28) it could be used as a template for further testing.
• Make sure that the test suite still runs fast after the API tests are added. We want it to be run frequently during development and don't want to waste CI time in the future either.
• If the data mocking could be consolidated somehow as part of this task, that would be great. E.g.: Create a helper function that populates the database with mock data.

Add stats about the last update date and latest price info per ticker

Description

Technical details

Automate issue and PR flow

Description

Technical details

Dockerize the project

Description

Technical details

Support transactions and broker costs

Crypto API

Description

Technical details

Calculate monetary values in its change

Description

Currently, we calculate Euros and US dollars with fractional units that could lead to errors due to floating-point math. We should replace this functionality to use the change of these currencies.

Technical details

1. modify the cash service to use Euro cents and US dollar cents (use integers instead of floats)
2. update the DB models using price values to allow integers (migrate existing values with a 100x ❗ )
3. update the serializer to incorporate the change
4. modify the frontend to accept cent value and divide it down

Configure Sentry

Description

Technical details

• Differentiate between environments.
• Filter common errors (400, 401 etc.)

A general user can see other user's portfolios

Expected Behavior

I as a general user can't fetch the portfolio of another user. Only admins can.

Actual Behavior

I as a general user can fetch the portfolio of another user.

Steps to Reproduce the Problem

1. Log in as a general user
2. Try to access another user's portfolio via the API (select an existing ID)

Specifications

• Version: 1.0
• Platform: MacOS
• Browser: Chrome

Fix every linting error

Description

Currently, the linting command produces a ton of output. We should strive to keep the code without linting errors because that is a good base to fix errors in fresh code more effectively and have any use for a CI/CD system. After the first cleanup, the maintenance of code quality will be less of a pain.

Technical details

We have two solutions to fix a linting error:

1. Fix the code so the error disappears.
2. Suppress the linting rule.

Prefer the first solution where it is sensible. We don't want to disable too many linting rules, because in that case the linting itself becomes meaningless, but don't hack the code itself, just to remove a linting error. Also, prefer suppressing an error locally instead of turning it off globally.

Run the test suites after fixing to check if the functionality is still correct.

Result is not representable

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

• Version:
• Platform:
• Browser:

Portfolio summary endpoint breaks if a portfolio requested before its first transaciton

Expected behavior

A not found error is sent from the server containing an error message that the portfolio has no transaction data before the selected period.

The dashboard displays the message as an empty state on the view.

Actual behavior

The server responds with internal server error because a 0 division happens.

The dashboard displays the unhandled error message.

Steps to Reproduce the Problem

1. Log in as a user with an existing portfolio.
2. Go to the portfolio summary view.
3. Select a date that is before the first transaction in the portfolio.

Reorder API folder structure

Description

We need to restructure the folders in the API project so the python module structure is less fragile and the config files and other folders won't pollute the project root.

Sync market data automatically via a syncer

Description

Create an application that could synchronize stock price, dividend and sync data to the stock buddy API.

We are going to use the Yahoo Finance API to fetch information about the tracked stocks.

Technical details

• Possibly the /raw-data API is going to be modified as part of this task. When this is done, add API test coverage for that module.
• We are going to need to sync other data types in the future so the architecture of this component has to be highly scalable.
• We should support both an initial fetch that synchronizes data from the stocks IPO and synchronization that is only the "new" data for the stock, that has happened since the last sync.

Position in summary has no price information

Expected Behavior

When one of my stock positions has no price information the stock is calculated as a no-growth position. (The stock price stays flat.)

Actual Behavior

If one of my positions has no price information in the database the snapshot creation fails.

Steps to Reproduce the Problem

1. Log in as a general user.
2. Create a new position with a stock that has no price information in the database.
3. Try to fetch the portfolio summary. It'll response 500.

Specifications

• Version: 1.1
• Platform: MacOS
• Browser: Chrome

Add performance API (portfolio, market, position)

Description

We want to create an API to support calculating different performance metrics for a single position, a portfolio, or an entire index.

Technical details

• The underlying service has been already implemented in #55. The implementer should use that to expose it via endpoints.
• We need two separate endpoints, for now, one for a single position (expecting a ticker and a time range) and one for a portfolio (expecting a portfolio ID or summary and a time range).
• We don't support indices yet so we don't need to implement that performance calculation.
• The performance service already contains everything that we should return. What we need in a report is a time series of the following values for the selected range:
  • annualized performance
  • accumulated total performance
  • timeslot performance
  • cash flow for the timeslot
  • the current size of the capital
  • accumulated total cash flow
  • cash yield for the current timeslot
  • total cash yield
• As part of this issue we should wire up the new calculation on the dashboard as well.

Enhance database backup command

Description

Extend the database backup command so it complies with our current backup naming convention. This way we could backup the databases by issuing a single command.

Technical details

• It should detect in which environment we run it.
• It should generate a unique name for the dump containing the current timestamp, version, and the name of the environment.

Portfolio summary endpoint breaks

Expected Behavior

The user can request a specific portfolio and the portfolio details are being sent as a response.

Actual Behavior

The servers run on an error during date comparison.

Steps to Reproduce the Problem

[24/Mar/2022 06:52:31] "GET /stocks/portfolios/summary/?asOf=2022-03-24 HTTP/1.0" 200 1964
[2022-03-24 06:52:33,830+0000] INFO - The user mhorvath has requested 2 portfolio as of 2022-03-24 00:00:00.
[2022-03-24 06:52:33,848+0000] ERROR - Internal Server Error: /stocks/portfolios/2/
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/app/apps/stocks/views.py", line 72, in retrieve
    snapshot = self.stocks_service.get_portfolio_snapshot(
  File "/home/app/lib/services/stocks.py", line 26, in get_portfolio_snapshot
    return cls.get_portfolio_snapshot_series(portfolios, [snapshot_date])[
  File "/home/app/lib/services/stocks.py", line 58, in get_portfolio_snapshot_series
    while action.date > snapshot_dates[0]:
TypeError: can't compare datetime.datetime to datetime.date

Specifications

• Version: 1.0
• Platform: MacOS
• Browser: Chrome