openwisp / docker-openwisp Goto Github PK

Provide a redis instance in a dedicated container and ensure the django settings point to it

source: http://openwisp.org/gsoc/ideas-2019.html

Provide mounted volumes that allow to store persistently files that are uploaded by users (eg: floor plan images).

source: http://openwisp.org/gsoc/ideas-2019.html

I think I mistakenly declared celery two times:

• https://github.com/openwisp/docker-openwisp/blob/master/docker-compose.yml#L130
• https://github.com/openwisp/docker-openwisp/blob/master/docker-compose.yml#L143

Remove psql statements from containers for a more pythonic approach that give proper error codes (like using psycopg2).
The psql statements are used in the shell script: common/utils.sh.

Blocked: Read comments on the issue

openwisp-orchestration container's certbot script needs to be tested in a container with public IP address.
(Read conversation below)

Provide a celery worker, initially used only by OpenWISP Controller (connections branch), but which later will executes background tasks of other modules as well, in a dedicated container.

source: http://openwisp.org/gsoc/ideas-2019.html

Research work:

• https://nick.sarbicki.com/blog/deploy-django-celery-on-k8s/

Blocked by: #40

The build time and more importantly the image size should be reduced as much as possible.

• Reduce the number of layers.
• Remove apk and python cache.
• Find and remove all the build dependencies.
• Leverage multistage build and reduce size further

Research:

• https://bit.ly/2GBtVtI
• psycopg/psycopg2#684
• Python multistage benefits (See here)

Currently, the configurations in the kubernetes/ need to be repeated for every file that requires a specific value,
Example, the domain and IP addresses need to changed in 4 files, Config.yml, ExtServices.yml, Services.yml & Ingress.yml, this is not ideal and may cause different settings in different files.

Error:

(1) rest: EXPAND {"username": "%{User-Name}", "password": "%{User-Password}"}
(1) rest: --> {"username": "admin", "password": "admin"}
(1) rest: Processing response header
(1) rest: ERROR: Malformed HTTP header: Status line too short
(1) rest: ERROR: Received 13 bytes of response data: HTTP/2 200
(1) rest: ERROR: Request failed: 23 - Failed writing received data to disk/application
(1) rest: ERROR: Server returned no data

The following were first thought of as the cause of the issue but testing proved these theories wrong:

1. HTTP/2 related bug in upstream:
   Tested using HTTP/1.1 instead of HTTP/2

2. Certificate issue, tested with:

tls = {
   tls_check_cert = no
   tls_check_cert_cn = no
}

Implement automated testing with basic checks for each service, eg: for the admin interface, send and HTTP request expecting to see the login page (at least one similar checks should be implemented for each service); the tests shall be executed in a travis-ci build

source: http://openwisp.org/gsoc/ideas-2019.html

Note:

• Add Test for checking postfix working on a fresh install by checking the creation of a new user.

While adding a new user, even if an SMTP error is received, user is still added to the database. Expected response should be Error code: 5XX.

Example errors:

• smtplib.SMTPRecipientsRefused: {'[email protected]': (550, b'5.1.1 [email protected]: Recipient address rejected: User unknown in local recipient table')}
• smtplib.SMTPRecipientsRefused: {'[email protected]': (454, b'4.7.1 [email protected]: Relay access denied')}
• socket.gaierror: [Errno -3] Try again

To reproduce:

• Intensionally misconfigure postfix so that it either can't relay(incase relay host is used).
  or
• Send response to non-existing users in "mydestinations".
  or
• Turn off the postfix service (postfix stop)

Blocked: Mentor review needed

The openwisp-orchestration container takes a while to create certificates and nginx server starts in the meanwhile and failes due to absence of SSL certificates.

• Information regarding reaching the dashboard needs to be improved. (Making the hosts part clear)
• Currently it also says /etc/hosts/ when it should be /etc/hosts (Its a file not a directory

The openwisp base image installs python packages hard-coded into the Dockerfile.
We want to make this process a little more configurable so that the user can supply their own python package links to create the image.

The builds for travis take a lot of time due to docker-compose build command.
It'll be great if we can come up with something to cache the intermediate layers or some other way to reduce build time.

Create a small version of documentation of the description of the variables in .env so that we can test them in deployment and discuss about them should that be required.

Datetime issue while migrating:

For this issue, look at the logs of dashboard container logs while migrating (first time the volume is created), you should see:

/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py:1421: RuntimeWarning: DateTimeField Ca.validity_start received a naive datetime (2019-10-04 00:00:00) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py:1421: RuntimeWarning: DateTimeField Cert.validity_start received a naive datetime (2019-10-04 00:00:00) while time zone support is active.
  RuntimeWarning)

Provide the OpenWISP Controller (connections branch) views and APIs in a dedicated container

source: http://openwisp.org/gsoc/ideas-2019.html

Currently the dashboard container has APIs that belong in controller, radius and topology containers only.

• Update disable-services documentation.

Provide a default management VPN based on OpenVPN in a dedicated container. The VPN shall be also pre-configured in OpenWISP, both as a VPN Server and a related VPN client template (ask questions to your mentors to know more about this), the OpenVPN configuration used in the container shall be kept in sync with the definition available in the VPN Server of OpenWISP. The Certificate Revocation List of the VPN shall be downloaded periodically from openwisp on the filesystem (a script run in a crontab) of the VPN server and the configuration of the VPN server

source: http://openwisp.org/gsoc/ideas-2019.html

The SSLMODE needs to be set as required and tested.

Provide a deafult working postfix instance which is able to send emails but can also be turned off (users may want to use an external SMTP service)

source: http://openwisp.org/gsoc/ideas-2019.html

Provide the OpenWISP Admin interface and the views managing account information (password reset, email confirmation) in a dedicated container.

source: http://openwisp.org/gsoc/ideas-2019.html

Allow users to easily configure some django settings:

• CORS settings
• Sentry logging
• DEFAULT_FROM_EMAIL
• SMTP settings
• language code
• timezone
• Leaflet settings
• default cert validity for django-x509
• default CA validity for django-x509

source: http://openwisp.org/gsoc/ideas-2019.html

Blocked What CORS settings need to be configurable?

The make publish command helps me to put the docker images on docker hub, but that's not enough, we want use Makefile to be able:

1. Tag the new release with a version number, example openwisp/openwisp-base:0.1.0 (Images with latest tag should also be uploaded.)
2. User to easily upload images in their own docker hub account, example atb00ker/openwisp-base:0.1.0
3. Release edge image directly from the master branch as openwisp/openwisp-base:edge

Edit:

• 3rd point taken care by docker hub autobuilds! 😄

We should use selenium for more robust testing of the functionality.

Task list:

Provide a dedicated container for celery-beat (used to automatically execute background tasks periodically)

Done:

1. Fixed: #43 : Fixed in #44

Updates on issues:

1. #32 : #to-do#
2. #2 : #to-do#

Review:

1. #44
2. #30
3. #28
4. Pull requests related to #26:
   4.1. openwisp/django-netjsonconfig#127
   4.2. openwisp/django-x509#62
   4.3. openwisp/openwisp-controller#117 & openwisp/openwisp-controller#116
5. Need review : #58

Blocked:

1. Blocked by #44 :
   1.1. #51 & #50 (#11)
   1.2. #48 (#8)
2. Pulls related to #26 : #52
3. Need review : #28
4. Need review : #30
5. #2

Post-GSoC:

1. #53 : Improvement for future.
2. #47 : Testing to-be done after all services are ready.
3. #54 : Finding the reason for this issue.

Set correct headers for openwisp configurations by replicating the configurations of the ansible-openwisp2.

Ensure following are enabled by default but can be turned off if needed:

• Gzip
• HTTP/2
• IPv6

Blocks: #7 , #2

Provide the OpenWISP Radius views and APIs in a dedicated container

source: http://openwisp.org/gsoc/ideas-2019.html

Some tasks need to run perodically for openwisp-radius which are currently not in a cronjob.
A celery-beat instance or a kubernetes's cronjob object needs to be implemented for this!

See comment: #23 (comment)

@hispanico how can this be achieved? Do you know?

Provide the websocket server of OpenWISP in a dedicated container

source: http://openwisp.org/gsoc/ideas-2019.html

1. Find all the urls that are required by admin.

• django_netjsonconfig.urls (django-netjsonconfig schemas)
• openwisp_controller.pki (CRL view)
• openwisp_controller.config (get_default_templates)

2. Rewrite them to be used as admin urls.

Provide the OpenWISP Network Topology views and APIs in a dedicated container

source: http://openwisp.org/gsoc/ideas-2019.html

last_ip field of the Device model should point to the IP of the AP.

Include installation, upgrade and usage information as well as an explaination of configuration settings in the README.

source: http://openwisp.org/gsoc/ideas-2019.html

• Documentation for image requirements:

• Geo-django: gdal geos proj4

• Pillow: zlib-dev jpeg-dev

• service_identity: openssl (openssl-dev: for installation) libffi-dev

• gettext: envsubst

• Document Makefile

Received:

daemon.err openwisp: Invalid url: missing X-Openwisp-Controller header

Note: adding correct nginx headers should fix it.

Blocked by: #1

Connection issue: When trying to setup the openwisp_controller.connection, I am facing an issue that when I go to change any of the devices configuration and apply, I get a timeout because request takes too long.

Note 1:

• I put the openwisp-controller in a container and tested it. (Working)
• I put the common/ files in a seperate folder beside the openwisp-controller in the container and tested. (Not Working)
• I copied the settings.py and urls.py to the common/ folder from the openwisp-controller/tests/ and tested. (Not Working)

Now,
Since the first bullet worked, I think that dependencies are not causing it.
Since, the third bullet fails, I think that misconfiguration in settings.py or urls.py is not causing it.

Note 2:
The file /usr/local/lib/python3.7/site-packages/channels/worker has line consumer_finished.send(sender=self.__class__) which "hangs" the execution.

Note 3:
It's not that channels / django / connection branch or dependencies have any issue with alpine packages or postgresql backend or dockerization of worker and runserver into different containers. It's all working in an image here: openwisp/openwisp-controller#111

Note 4:
Since the openwisp/openwisp-controller#111 was working, I decided to use it as a base image and build the images from there and that lead the testing in Note 1. Just incase that might be useful for debugging here is the commit for that: atb00ker@ec427d4

Add letsencrypt dns01 option to get the certificates.

Blocked by: Waiting for review on comment. (read below)

We need to keep settings in different variables. Read discussion here for context.

The function to create production certificates have errors and is not working properly.

Make 3 stages:

1. Build: should build the docker images and upload on dockerhub as mentioned here with format: openwisp/openwisp-<image>:travis-<build-number>.

2. Test: should test the images. Multiple test builds should test images with different environment variables.

3. Destroy: should delete the openwisp/openwisp-<image>:travis-<build-number>.

Test following:

• With HTTPS
• Mounting files (raddb/, nginx.conf )

Provide a PostgreSQL instance in a dedicated container, ensure all the other services point to this postgres instance, ensure the data is stored on persistent storage, allow using a PostgreSQL hosted elsewhere as an alternative (some users may want to use managed PostgreSQL services)

source: http://openwisp.org/gsoc/ideas-2019.html

Create automated tests for all the images, remember to create tests to:

• Test postfix by adding a new user
• Test redis by logging in.

Some of the containers don't take the timezone from the variable in the .env file.
For this task,

1. Change the timezone to some timezone other than UTC or IST in the .env
2. Run the containers
3. Check the timezone of each of the container.
4. If a container doesn't follow the timezone, fix it and submit the PR.

Provide a configurable freeradius instance in a dedicated container

source: http://openwisp.org/gsoc/ideas-2019.html

Some tasks need to run perodically for openwisp-network-topology which are currently not in a cronjob.
A celery-beat instance or a kubernetes's cronjob object needs to be implemented for this!

The mtu_disc option in the VPN server is not supported in some routers and hence causes OpenVPN not initalize properly without removing:

        option mtu_disc 'no'                                               
        option mtu_test '0'

Following routers have been tested to have this problem:

• GL.iNET GL-MT300N-V2 Mini Travel Router
• Virtual Machine running OpenWRT 18.06