pi-top / pi-topos-web-portal

React / Python application for providing a web-based interface to pi-topOS.

pi-topOS Web Portal

Build Status: Latest

About

A Python server that hosts a web application to interact directly with a Linux machine.

In addition to the core web server, desktop management of app pages is managed via [web-renderer](https://github.com/pi-top/web-renderer) is handled via a separate -desktop binary Debian package.

pt-os-web-portal is included out-of-the-box with pi-topOS.

Ensure that you keep your system up-to-date to enjoy the latest features and bug fixes.

This application is installed as a Python 3 script that is managed by a systemd service, configured to automatically run on startup and restart during software updates.

Installation

pt-os-web-portal is installed out of the box with pi-topOS, which is available from pi-top.com. To install on Raspberry Pi OS or other operating systems, check out the Using pi-top Hardware with Raspberry Pi OS page on the pi-top knowledge base.

More Information

Please refer to the additional documentation for more information about the application.

pi-topos-web-portal's Issues

Improve handling of closing `web-renderer`

Can we use wmctrl to get the PID or possibly even close the window?

Add quick access to recovery mode/shell during setup

Users who want to perform a full upgrade quickly will want to boot into recovery quickly and easily. Providing a simple way to do this would be beneficial.

Perhaps this could be triggered by pressing the build ID (similar to Android's Developer Options)

Move `expand-fs.sh` and `notify-on-port-busy.sh` from `pkg` to `src`

Add custom favicon

Currently the react favicon is displayed.

Fix failing test run CI

======================== 63 passed, 11 skipped in 1.63s ========================
Cleaning up...
yarn install v1.13.0
[1/4] Resolving packages...
[2/4] Fetching packages...
info There appears to be trouble with your network connection. Retrying...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
error @testing-library/[email protected]: The engine "node" is incompatible with this module. Expected version ">=12". Got "10.24.0"
error Found incompatible module
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Error: Process completed with exit code 1.

Seems as though test-runner Docker container needs node to be updated from 10.24.0 to > 12.

OS updater: show what is going to be/has been updated

Create new service that runs once on boot
    if current version of pt-os is newer than last accessed:
        show notification asking user to check out latest pt-os version notes
            create a page in the docs that just chronicles latest releases as a rolling doc (newest first)
                e.g. pi-top.github.io/docs/releases-and-updates.md#3.4 (use `pt-os` version)

                    using anchor points so that they only go to whatever version they have

            options:
                "Don't ask me about this release again"
                    store current `pt-os` version as last accessed
                "Don't ask me again"
                    disable service
                "Show me!"
                    store current `pt-os` version as last accessed
                    open docs to anchor point matching version in release

Though we might just want to handle all of this in the web portal. if there's an OS docs page with a list of the updates on it, the OS updater UI can show the updates that it's gonna update by parsing the markdown

Handle pi-top [4] miniscreen directly

OS build pipeline/on web portal install: disable pt-sys-oled
Web portal: handle the miniscreen itself if sys oled is not enabled
Web portal: enable pt-sys-oled during finalise stage of onboarding

Simplify finalise logic

Use OS interface from SDK: pi-top/pi-top-Python-SDK#431

Use config file for tracking state in backend

e.g. state=onboarding, state=main, etc.

Add check for major OS version updates

As well as checking for package updates, major version updates of pi-topOS should be tracked.
Perhaps we can do point-releases similar to Debian (e.g. 10.4) where new major version means new underlying OS. This would allow web portal to know if a new version is out, which we can use to notify users that they need to update.
This could be handled by a package in https://github.com/pi-top/pt-os-core/ somehow - possibly related to news updates regarding pi-topOS that web portal can show.

OS updater GUI: add KB link for failed update

When the web portal is responsible for OS updating...

If the OS update is not during onboarding, then we should include a link to https://knowledgebase.pi-top.com/knowledge/error-updating-sirius. Perhaps the best way to handle this would be via a redirect that we can change later on (e.g. https://pi-top.com/error-updating)

Decouple Debian packaging from core app

e.g. we check if onboarding is completed by looking for if /etc/xdg/autostart/pt-os-setup.desktop exists or not. This is only relevant because it is handled by the postinst Debian maintainer script. This means that the application is not going to work as intended if it is installed in any other way.

I am sure that there are other examples, so we should track these down and address them.

Review redirection handling

Ideally, we would have it so that all pages check the following:

• If OS setup not completed, redirect
• If tour not completed, redirect
• Else go to desired page, or landing page if none selected (i.e. / root via http://localhost/)

First boot: improve UX

First boot takes a little while because the swap file is blocking while it is being generated. By making this something that runs in the background, we no longer need to actually show any kind of window at the start saying "expanding fs..." - this isn't what is taking a long time, and - in fact - when it is run in the background, we don't need a UI at all.

We can simply do what Raspberry Pi OS does, which is to expand the filesystem using init= in /boot/cmdline.txt.
Possibly this script can also handle some of our finalise steps that are related to first boot configuration (e.g. fbcon).

This is probably most appropriate as a new package: pt-os-onboarding/pt-os-setup. Ideally, web portal has no knowledge of anything to do with configuring the actual operating system outside of the options provided via the UI.

Add easy file upload/download support

Web-based file management makes it the most intuitive way of moving files to/from another device.

Add support for whitelisting IPs

Any non pi-top [4] running pi-topOS will have a web dashboard that is likely not needed. Ideally, we should limit access in these cases to localhost only.

This is probably easiest to achieve via a mechanism for permitted client access.

Support onboarding through mobile devices

The onboarding app is not displayed correctly in mobile devices.

Improve how to determine if filesystem is expanded

We should replace the FS expand breadcrumb with something more robust. One possible option would be to check the filesystem size, though I'm sure there are other ways to detect.

Migrate `pt-shutdown-helper` to a web portal GUI app

Just need a simple interface to open for power management.

"Shutdown" - "/sbin/shutdown -h now"
"Reboot" - "/sbin/reboot"
"Reboot into pi-topOS Recovery" - "/usr/bin/rm /recovery/autoboot.txt && /sbin/reboot"

When rebooting into recovery, provide a simple warning notice:

⚠️ WARNING: Rebooting into pi-topOS Recovery may cause you to lose access to your device and delete your data. Read this KB article before continuing.

Take my back
I understand the risk

Node v16 causes CI test run to fail

There's a bug on node 16 that caused tests not to run, the job fails in the preparation steps. Downgrading to use an earlier version of node solves the issue.

yarn run v1.22.11
$ react-scripts test --coverage


#
# Fatal error in , line 0
# Check failed: !holder_map.has_named_interceptor().
#
#
#
#FailureMessage Object: 0x7fffc4ec0dc0
 1: 0xb691f1  [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 2: 0x1bf3094 V8_Fatal(char const*, ...) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 3: 0x10ac4a1 v8::internal::ConcurrentLookupIterator::TryGetPropertyCell(v8::internal::Isolate*, v8::internal::LocalIsolate*, v8::internal::Handle<v8::internal::JSGlobalObject>, v8::internal::Handle<v8::internal::Name>) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 4: 0x1c5aa55  [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 5: 0x1c5b9eb v8::internal::compiler::JSGlobalObjectRef::GetPropertyCell(v8::internal::compiler::NameRef const&, v8::internal::compiler::SerializationPolicy) const [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 6: 0x1e61ba3 v8::internal::compiler::JSNativeContextSpecialization::ReduceNamedAccess(v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::compiler::NamedAccessFeedback const&, v8::internal::compiler::AccessMode, v8::internal::compiler::Node*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 7: 0x1e623c4 v8::internal::compiler::JSNativeContextSpecialization::ReducePropertyAccess(v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::base::Optional<v8::internal::compiler::NameRef>, v8::internal::compiler::Node*, v8::internal::compiler::FeedbackSource const&, v8::internal::compiler::AccessMode) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 8: 0x1e6274e v8::internal::compiler::JSNativeContextSpecialization::ReduceJSLoadNamed(v8::internal::compiler::Node*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
 9: 0x1df5e0a v8::internal::compiler::Reducer::Reduce(v8::internal::compiler::Node*, v8::internal::compiler::ObserveNodeManager*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
10: 0x1c92604  [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
11: 0x1df60be v8::internal::compiler::GraphReducer::Reduce(v8::internal::compiler::Node*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
12: 0x1df7895 v8::internal::compiler::GraphReducer::ReduceTop() [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
13: 0x1df7c48 v8::internal::compiler::GraphReducer::ReduceNode(v8::internal::compiler::Node*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
14: 0x1ca03fd v8::internal::compiler::InliningPhase::Run(v8::internal::compiler::PipelineData*, v8::internal::Zone*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
15: 0x1ca1291 v8::internal::compiler::PipelineImpl::CreateGraph() [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
16: 0x1ca1668 v8::internal::compiler::PipelineCompilationJob::PrepareJobImpl(v8::internal::Isolate*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
17: 0xd94f5d v8::internal::OptimizedCompilationJob::PrepareJob(v8::internal::Isolate*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
18: 0xd96358  [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
19: 0xd97e70 v8::internal::Compiler::CompileOptimized(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::ConcurrencyMode, v8::internal::CodeKind) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
20: 0x11f4463 v8::internal::Runtime_CompileOptimized_Concurrent(int, unsigned long*, v8::internal::Isolate*) [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
21: 0x15e67f9  [/opt/hostedtoolcache/node/16.9.0/x64/bin/node]
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 1.

Handle swap file

Reference:

• https://arcanesciencelab.wordpress.com/2019/08/03/increasing-swapfile-space-on-the-raspberry-pi-4/

Here is a good example. However, instead of handling the service (sudo /etc/init.d/dphys-swapfile, it might be possible to handle directly:

sudo dphys-swapfile setup
sudo dphys-swapfile swapon

Add dpkg trigger support

Service is reluctant to restart (as it should be). However, if a dependency updates (particularly like the SDK), then it needs to be made aware that it should restart, and notify the user accordingly.

This is most easily done with a dpkg trigger.

Upgrade page error message overflows container

The error message should have overflow: hidden so it does not overlap with buttons and red warning message.

Rename onboarding to "quick setup" and allow user to re-run at any time

Increases re-use of useful parts of the application.

Create server health check notification service

Standalone service that monitors for issues with the web portal, diagnoses, and either fixes/alerts the user. It should be able to detect issues like unavailable port, failed service, etc. This would run tests like trying to acquire the port.

This would take on the responsibility of pt-os-web-portal-port-busy.

OS Upgrade doesn't work when running onboarding through a browser in a different host

OS Upgrade page uses websockets to send/receive messages to/from the backend through ws://127.0.0.1:80/os-upgrade. This means that if the application is opened on a different host through a web browser, it won't work, since the connection fails.

Prepare for open sourcing

Review the EULA, and ensure that we don't risk anything by open sourcing. Look into FS Me font licensing, etc.

Clean up openbox-session config after use

/home/pi/.config/openbox/rc.xml is installed as part of the build process. We should clean it up when we're done with it (finalise stage).

Add `mypy` static typing

More info: http://mypy-lang.org/

Config:

-   repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.910
    hooks:
    -   id: mypy

Current error log:

src/server/tests/utils.py:8: error: Incompatible types in assignment (expression has type "Callable[[Dict[_KT, _VT], _KT, _VT], None]", base class "object" defined the type as "Callable[[object, str, Any], None]")
src/server/tests/utils.py:9: error: Incompatible types in assignment (expression has type "Callable[[Dict[_KT, _VT], _KT], None]", base class "object" defined the type as "Callable[[object, str], None]")
src/server/backend/helpers/command_runner.py:83: error: Incompatible return value type (got "Optional[str]", expected "str")
src/server/backend/helpers/mocks/pywifi_mock.py:54: error: Need type annotation for "results" (hint: "results: List[<type>] = ...")
src/server/backend/helpers/finalise.py:24: error: Incompatible return value type (got "str", expected "int")
src/server/backend/helpers/finalise.py:29: error: Incompatible types in assignment (expression has type "None", variable has type "str")
src/server/backend/helpers/finalise.py:32: error: Incompatible types in assignment (expression has type "None", variable has type "str")
src/server/backend/helpers/finalise.py:35: error: Incompatible return value type (got "str", expected "int")
src/server/backend/helpers/language.py:14: error: Need type annotation for "utf8_locales" (hint: "utf8_locales: List[<type>] = ...")
src/server/backend/helpers/language.py:34: error: Incompatible return value type (got "None", expected "str")
src/server/backend/helpers/language.py:47: error: Need type annotation for "utf8_locales" (hint: "utf8_locales: List[<type>] = ...")
src/server/backend/helpers/about.py:50: error: Need type annotation for "most_unstable_source" (hint: "most_unstable_source: Dict[<type>, <type>] = ...")
src/server/backend/helpers/about.py:58: error: Incompatible return value type (got "Optional[Any]", expected "str")
src/server/backend/helpers/wifi_country.py:30: error: Incompatible types in assignment (expression has type "None", variable has type "str")
src/server/backend/helpers/timezone.py:41: error: Incompatible return value type (got "Optional[str]", expected "str")
src/server/backend/helpers/keyboard.py:29: error: Incompatible return value type (got "Tuple[Optional[str], Optional[str]]", expected "Tuple[str, str]")
src/server/backend/helpers/device_registration.py:9: error: Library stubs not installed for "requests" (or incompatible with Python 3.9)
src/server/backend/helpers/device_registration.py:9: note: Hint: "python3 -m pip install types-requests"
src/server/backend/helpers/device_registration.py:9: note: (or run "mypy --install-types" to install all missing stub packages)
src/server/backend/helpers/device_registration.py:9: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
src/server/backend/helpers/wifi_manager.py:40: error: Missing return statement
src/server/backend/helpers/wifi_manager.py:89: error: Incompatible types in assignment (expression has type "float", variable has type "int")
src/server/backend/helpers/os_updater.py:13: error: Name "apt.progress.base.AcquireProgress" is not defined
src/server/backend/helpers/os_updater.py:35: error: Name "apt.progress.base.InstallProgress" is not defined
src/server/backend/helpers/os_updater.py:66: error: "None" has no attribute "update"
src/server/backend/helpers/os_updater.py:67: error: "None" has no attribute "open"
src/server/backend/helpers/os_updater.py:82: error: "None" has no attribute "upgrade"
src/server/backend/helpers/os_updater.py:83: error: "None" has no attribute "upgrade"
src/server/backend/helpers/os_updater.py:85: error: "None" has no attribute "install_count"
src/server/backend/helpers/os_updater.py:87: error: "None" has no attribute "required_download"
src/server/backend/helpers/os_updater.py:90: error: "None" has no attribute "required_space"
src/server/backend/routes.py:250: error: Item "None" of "Optional[Any]" has no attribute "route"
Found 29 errors in 13 files (checked 51 source files)

Miniscreen during onboarding: improve UX

Currently not very performant, but mainly the issue is communication.

Could do a better job of keeping the process engaging! Most people will have this on their desk.

Sometimes it loses the AP connection for one frame and the animation repeats - should we wait for multiple repeat errors before resetting state?

I think that we should primarily focus on getting and keeping the user connected here, but should explore opportunities to managing messaging particularly when the system is down during the onboarding process.

Some initial thoughts -

Stage 1: get the user to connect to the pi-top

• add getting start URL with "connect" CTA
• add some basic guidance for navigation (e.g. navbar? animating arrow?)

Stage 2: get the user to find the onboarding app

• show URL to go to on connected device

Stage 3: indicate that the miniscreen is no longer the priority

• communicate that the user needs to "complete onboarding via web app"

The real question is this - how can we show the connectivity information when the user needs it, and the "find the onboarding app" information when the user needs that?

Build in Docker, deploy with `dh-virtualenv`

https://dockyard.readthedocs.io/en/latest/packaging-howto.html

Create allow-list of external commands that can be executed using `run_command`

Fix the uncontrolled command line in src/server/backend/helpers/command_runner.py by creating a look-up of all of the permitted commands (preferably with the full path to each command to ensure robustness).

Bullseye: Rework desktop files

• Create "pt-onboarding" Debian package for handling maintainer scripts stuff (e.g. update-alternatives), separate from pt-web-portal-desktop
  • Also a better package to handle Conflicts: piwiz
• Rename .desktop files to better suit, etc.

The whole pt-web-portal-desktop area should be reworked - a new package name may make sense as part of that.

Update during onboarding fails when trying to reach raspbian.raspberrypi.org

Steps to Reproduce:

• Download OS from pi-top.com
• Flash SD card with OS
• Go through onboarding via VNC
• Connect to WiFi during onboarding
• Update should fail.

Add support for new OS versioning system

pi-top/pi-topOS-Build#6

• Update "about" page to use new format (continue supporting previous format if found)
• Update miniscreen onboarding app to show the OS version and build ID separately

Review adding recovery OS management

pi-top/pi-topOS-Recovery#21

pi-topOS Recovery needs to have a separate interface for managing its recovery partition files.

The simplest solution would be something that can checkout the repo, and diff with /recovery for verification. Downloading from GitHub releases and moving the files into /recovery.

This can be added as a sort of notification widget, which checks periodically for updates.

Browsing to a wrong link is a blank page

e.g. pi-top.local/onbording
A 404 page or redirect would be better.

Miniscreen app: text isn't rendered correctly

Sometimes the text in the miniscreen is not rendered correctly.

Adding an extra space to the rendered string fixes the issue (e.g.: "pi-top" + " "). I think it might be related to the calculations performed by PIL when drawing the text.

Also, this doesn't happen with some fonts (e.g.: Roboto-Medium)

Replace `pt-os-updater` with web portal's OS updater

Currently the OS updater uses a systemd service that runs on each boot. It checks /etc/pi-top/pt-os-updater/last_checked_date to see if a day has passed, before deciding if the user should receive a notification that contains the action command to start the update in it.

This new system should be robust and easy to test with confidence. Therefore, I propose for a first version:

• Web portal notifies user if there's an update, says to start Updates from the Start Menu
• .desktop file runs web-renderer at update page
• Updater window can be closed and update will continue

Notably, we drop the config file for 'last checked date' until we build a better system for the web portal to handle how it checks for updates. This is likely to grow to encompass many different use cases:

• Schools don't want to update and grind their networks to a halt - they need a way to bypass regular checks
• Some users may want to auto-download in the background and upgrade on shutdown

Some thought should be given to the scope of the use-cases and an appropriate configuration pattern should be chosen.

Possibly we can consider some of the following:

• Add an action to open web-renderer at update page directly from notification
• Detect if user closes window, use notification to confirm
• Window opens without option to close except for from the UI itself
• IPC commands (e.g. web-portal-ctl update-window)?

Review if this should be the app that handles filesystem expansion etc. on first boot of the OS

Perhaps this should be another app? Its behaviour is quite separate.

Re-add python tests

Merging pitopcommon to pitop.common (#190) breaks tests.... we need to re-add them, hopefully without using the test-run docker image

Wi-Fi country: also add for `hostapd.conf`

Write

HOSTAPD_country_code=GB

in /etc/default/wifi-ap-sta - requires this

App will not run without Debian packaging

Currently the app requires the files to be in these locations:

pkg/pwdptwp.sh                              /usr/lib/pt-web-portal
src/expand-fs.sh                            /usr/lib/pt-web-portal
src/notify-on-port-busy.sh                  /usr/lib/pt-web-portal
src/server/Pipfile*                         /usr/lib/pt-web-portal
src/server/run.py                           /usr/lib/pt-web-portal
src/server/backend                          /usr/lib/pt-web-portal

The server should be able to be installed without Debian packaging, with full functionality. The src/*.sh scripts should be invoked by the backend with a relative path (or similar).

Ideally, the install file would look something like this:

pkg/pwdptwp.sh                        /usr/lib/pt-web-portal
src/server                            /usr/lib/pt-web-portal

Also, the app checks if onboarding is completed by looking for if /etc/xdg/autostart/pt-os-setup.desktop exists or not. This is only relevant because it is handled by the postinst Debian maintainer script. This means that the application is not going to work as intended if it is installed in any other way.

I am sure that there are other examples, so we should track these down and address them.

Onboarding: offer to set date and time manually if no internet

We can also use this to suggest to people who have no internet that their system is likely out of date

Add firmware upgrade to OS setup

When this is done, we should review replacing pt-firmware-updater with web portal's firmware updater.

Add JS coverage report to CodeCov

Add basic authentication for remote clients

If host device is NOT on a pi-top [4], then display "Sorry, accessing this pi-top's web portal from another device is not available for devices other than pi-top [4] right now."

Otherwise, we should provide instructions on how to get a code from a page in pt-sys-oled, to prove that the user is at the device. If there are repeated failed attempts, then it should lock until "unlocked" somehow from the local device. This should provide a basic form of authentication for security.

Remove unnecessary breadcrumbs files & associated functionality

Some of the files in /etc/pi-top/ & other breadcrumbs are no longer necessary. We need to make sure the associated functionality in the server is removed.

e.g:

• /etc/pi-top/.licenceAgreed
• /etc/pi-top/.silentBoot
• /etc/pi-top/.expandedFs (Related to #143)

Review `pi-language-support`

piwiz uses pi-language-support to get the names of all packages required to add translations and fonts to support a particular locale. Is this useful?

Add settings page to replace `pt-settings`

See same style as for the other applications that are packaged as part of the pt-web-portal-desktop package.