onecdonly / sherpa Goto Github PK

View Code? Open in Web Editor NEW

107.0 6.0 13.0 6.73 MB

A mini-package-manager for QNAP NAS

License: GNU General Public License v3.0

Shell 100.00%

bash script sherpa qnap entware nas qts package-manager qpkg usenet

sherpa's People

Contributors

Stargazers

Watchers

Forkers

rivy unixengineer erikdj r0b3rdv pccarer d9001089 superpawko dirtyoldbiker unrared omega62 mstrrlm

sherpa's Issues

permit packages to follow remote tags as well as branches

SickChill clones from the most recent remote tag found.

Most other packages clone from master branch.

SAB3 package needs to follow a specific hardcoded tag instead.

add check for QTS version before installing anything

package downloads are not being fully debug logged

Must also include the command executed.

Watcher3 not starting daemon

Python crash shown in init:

-------------------- start requested --------------------
= Sun Jun 21 08:49:41 EST 2020
= daemon is not active
* updating application 'OWatcher3': OK
= result: [0]
= PullGitRepo(): "Already up to date."
* launching: failed!
= result: [1]
= daemon startup messages: "Userdata directory exists, continuing.
Connected to database sqlite:////share/MD0_DATA/.qpkg/OWatcher3/config/watcher.sqlite
Config file found, merging any new options.
Clearing Mako cache.
Traceback (most recent call last):
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/lib/reprconf.py", line 213, in as_dict
    value = unrepr(value)
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/lib/reprconf.py", line 508, in unrepr
    return b.build(obj)
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/lib/reprconf.py", line 354, in build
    raise TypeError('unrepr does not recognize %s' %
TypeError: unrepr does not recognize 'Constant'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "watcher.py", line 139, in <module>
    root = cherrypy.tree.mount(App(), '{}/'.format(core.URL_BASE), 'core/conf_app.ini')
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/_cptree.py", line 226, in mount
    app.merge(config)
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/_cptree.py", line 100, in merge
    _cpconfig.merge(self.config, config)
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/_cpconfig.py", line 139, in merge
    for section, value_map in reprconf.as_dict(other).items():
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/lib/reprconf.py", line 46, in as_dict
    config = Parser().dict_from_file(config)
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/lib/reprconf.py", line 228, in dict_from_file
    return self.as_dict()
  File "/share/MD0_DATA/.qpkg/OWatcher3/OWatcher3/lib/cherrypy/lib/reprconf.py", line 219, in as_dict
    raise ValueError(msg, x.__class__.__name__, x.args)
ValueError: ("Config error in section: '/', option: 'tools.auth.on', value: 'True'. Config values must be valid Python.", 'TypeError', ("unrepr does not recognize 'Constant'",))"

switch Watcher3 repo

Use https://github.com/barbequesauce/Watcher3.git instead

SickRage: "'fatal: No names found, cannot describe anything."

This happens as SickRage version tags are set periodically, and the sherpa SickRageinit only performs a shallow git clone which means it's unlikely to capture a tag.

Described here: https://forum.qnap.com/viewtopic.php?p=675114#p675114

Current solution is to clone back 100 changes and hope this captures a tag. Got to find a better way.

invalid target app not identified early-on

after downgrading Py3, all installed Py3 apps should be restarted

service port number not shown on first install

Might be due to odd return code when running initial clone from repo

ensure crashed app daemon recovers correctly

Presently, the init script doesn't confirm that an existing PID file matches an actual in-memory process.

NZBGet kills every running instance on exec

This should only kill the default instance launched by Entware and leave all others running.

Entware: update IPKGs

Whenever a package is updated/reinstalled, ensure Entware IPKGs are also updated. This isn't presently happening via --update.

ensure failure to downgrade Py3 is recognised

new init script features!

history - show the current contents of /var/log/package.log
status - test if app is in memory and UI responds on designated port

perform installed RAM size check before running

Units running QTS with 1GB or less are known to result in unstable SABnzbd installations.

warn user when reinstalling Entware about defaults

Request: https://forum.qnap.com/viewtopic.php?p=754117#p754117

When re-installing Entware, warn user that all IPKGs will be reverted to defaults, and only those required for sherpa will be reinstalled.

Sherpa should prompt user with agree/cancel.

Sabnzbd install removed all configurations.

Attempted to install sabnzbd over clint hall's version and my configurations were lost.
Here is the relevant part of the debug log.

[ dbug ] (**) SCRIPT:          finished: Mon Nov 26 22:00:05 MST 2018
[ dbug ] (**) SCRIPT:      elapsed time: 00h:01m:14s
[ dbug ] (II) ======================================================================
[ dbug ] (<<) <DisplayResult> [0]
[ dbug ] (>>) <Init>
[ dbug ] (II) ======================================================================
[ dbug ] (**) SCRIPT:           started: Mon Nov 26 22:41:20 MST 2018
[ dbug ] (**) SCRIPT:           version: 181116
[ dbug ] (II) ----------------------------------------------------------------------
[ dbug ] (II) Markers: (**) detected, (II) information, (WW) warning, (LL) log file,
[ dbug ] (II)  (EE) error, (==) processing, (--) done, (>>) f entry, (<<) f exit,
[ dbug ] (II)  (vv) variable name & value, ($1) positional argument value.
[ dbug ] (II) ----------------------------------------------------------------------
[ dbug ] (**) NAS   :             model: TS-563
[ dbug ] (**) NAS   :  firmware version: 4.3.4
[ dbug ] (**) NAS   :    firmware build: 20180315
[ dbug ] (**) NAS   :            kernel: 4.2.8 x86_64
[ dbug ] (**) NAS   :         OS uptime: 3:11
[ dbug ] (**) NAS   :       system load: 1 min=0.12, 5 min=0.13, 15 min=0.21
[ dbug ] (**) NAS   :              EUID: 0
[ dbug ] (**) NAS   :    default volume: /share/CACHEDEV1_DATA
[ dbug ] (**) NAS   :             $PATH: /share/CACHEDEV1_DATA/.qpkg/container-stati
[ dbug ] (**) NAS   :              /opt: /share/CACHEDEV1_DATA/.qpkg/Entware
[ dbug ] (**) NAS   :   /share/Download: CACHEDEV1_DATA/Download
[ dbug ] (**) SCRIPT:    user arguments: Sabnzbd
[ dbug ] (**) SCRIPT:        target app: SABnzbdplus
[ dbug ] (II) ----------------------------------------------------------------------
[ dbug ] (vv) QPKG_ARCH [x64]
[ dbug ] (vv) STEPHANE_QPKG_ARCH [x64]
[ dbug ] (vv) PREF_ENTWARE [Entware]
[ dbug ] (**) QPKG  : Entware installer: std
[ proc ] downloading sherpa package list ...
[ done ] downloaded sherpa package list
[ dbug ] (<<) <Init> [0]
[ dbug ] (>>) <PauseDownloaders>
[ dbug ] (--) paused existing SABnzbd queue
[ dbug ] (<<) <PauseDownloaders> [0]
[ dbug ] (>>) <RemoveOther>
[ dbug ] (**) QPKG  :  'Par2cmdline-MT': not installed [250]
[ dbug ] (**) QPKG  :         'Optware': not installed [250]
[ dbug ] (<<) <RemoveOther> [0]
[ dbug ] (>>) <DownloadQPKGs>
[ dbug ] (>>) <DownloadQPKG>
[ proc ] downloading QPKG (Par2_0.7.4.0_x86_64.qpkg.zip) ...
[ done ] downloaded QPKG (Par2_0.7.4.0_x86_64.qpkg.zip)
[ dbug ] (<<) <DownloadQPKG> [0]
[ dbug ] (>>) <DownloadQPKG>
[ proc ] downloading QPKG (SABnzbdplus_180909.qpkg) ...
[ done ] downloaded QPKG (SABnzbdplus_180909.qpkg)
[ dbug ] (<<) <DownloadQPKG> [0]
[ dbug ] (<<) <DownloadQPKGs> [0]
[ dbug ] (>>) <InstallEntware>
[ dbug ] (--) adjusted $PATH
[ dbug ] (vv) PATH [/opt/bin:/opt/sbin:/share/CACHEDEV1_DATA/.qpkg/container-station/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin/X11:/usr/local/sbin:/usr/local/bin:/Apps/bin:/Apps/sbin]
[ dbug ] (>>) <PatchEntwareInit>
[ dbug ] (II) patch: do the "opt shuffle" - already done
[ dbug ] (<<) <PatchEntwareInit> [0]
[ dbug ] (<<) <InstallEntware> [0]
[ dbug ] (>>) <InstallExtras>
[ proc ] installing QPKG (Par2_0.7.4.0_x86_64.qpkg) - this can take a while ...
[ done ] installed QPKG (Par2_0.7.4.0_x86_64.qpkg)
[ dbug ] (>>) <InstallIPKGs>
[ dbug ] (>>) <UpdateEntware>
[ proc ] updating Entware package list ...
[ done ] updated Entware package list
[ dbug ] (<<) <UpdateEntware> [0]
[ dbug ] (>>) <InstallIPKGBatch>
[ proc ] calculating number and size of IPKGs required ...
[ dbug ] (II) requested IPKG names: python python-pip git git-http nano less python-pyopenssl python-dev gcc unrar p7zip coreutils-nice ionice ffprobe python-pyopenssl python-lxml pytho
n-lib2to3
[ dbug ] (==) finding all IPKG dependencies ...
[ dbug ] (--) complete
[ dbug ] (II) found all IPKG dependencies in 11 iterations
[ dbug ] (==) excluding packages already installed ...
[ dbug ] (--) complete
[ dbug ] (II) required IPKG names:
[ dbug ] (vv) IPKG_download_size [0]
[ done ] no IPKGs are required
[ dbug ] (<<) <InstallIPKGBatch> [0]
[ dbug ] (<<) <InstallIPKGs> [0]
[ dbug ] (>>) <InstallPIPs>
[ proc ] downloading & installing (PIP modules) ...
[ done ] downloaded & installed (PIP modules)
[ dbug ] (<<) <InstallPIPs> [0]
[ dbug ] (<<) <InstallExtras> [0]
[ dbug ] (>>) <InstallTargetApp>
[ dbug ] (>>) <BackupConfig>
[ proc ] stopping daemon (sabnzbd.sh) ...
[ done ] daemon stopped (sabnzbd.sh)
[ dbug ] (<<) <BackupConfig> [0]
[ proc ] uninstalling QPKG 'SABnzbdplus' ...
[ done ] uninstalled QPKG 'SABnzbdplus'
[ proc ] installing QPKG (SABnzbdplus_180909.qpkg) - this can take a while ...
[ done ] installed QPKG (SABnzbdplus_180909.qpkg)
[ proc ] waiting for 10 seconds ...
[ done ] wait complete
[ dbug ] (>>) <RestoreConfig>
[ dbug ] (<<) <RestoreConfig> [0]
[ proc ] starting daemon (sabnzbd.sh) - this can take a while ...
[ done ] daemon started (sabnzbd.sh)
[ dbug ] (<<) <InstallTargetApp> [0]
[ dbug ] (>>) <Cleanup>
[ dbug ] (--) resumed existing SABnzbd queue
[ dbug ] (<<) <Cleanup> [0]
[ dbug ] (>>) <DisplayResult>
[ done ] 'SABnzbdplus' has been successfully installed!
[ dbug ] (**) SCRIPT:          finished: Mon Nov 26 22:42:46 MST 2018
[ dbug ] (**) SCRIPT:      elapsed time: 00h:01m:26s
[ dbug ] (II) ======================================================================
[ dbug ] (<<) <DisplayResult> [0]

check compatibility with existing alternative Entware installations

ensure IPKGs for all installed QPKGs are installed

If Entware is removed and reinstalled as part of a sherpa run, IPKGs for other installed QPKGs are not automatically included.

InstallIPKGs()

... should check for all other QPKGs that require IPKGs.

auto-save Entware IPKG list to file when reinstalling Entware

Request: https://forum.qnap.com/viewtopic.php?p=754117#p754117

We won't restore these, but should append datetime and existing package list to a backup file.

No internet access

I've run the script successfully in the past but got an error with SABnzbd today and tried to run the script again to reinstall.

Am getting a fail message saying No Internet access, not sure where to go from here.

[ dbug ] (II) ========================================================================
[ dbug ] (**) SCRIPT:             started: Fri Jan 3 17:52:49 CST 2020
[ dbug ] (**) SCRIPT:             version: 191106
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (II) Markers: (**) detected, (II) information, (WW) warning, (LL) log file,
[ dbug ] (II)  (EE) error, (==) processing, (--) done, (>>) f entry, (<<) f exit,
[ dbug ] (II)  (vv) variable name & value, ($1) positional argument value.
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) NAS   :               model: TS-439
[ dbug ] (**) NAS   :    firmware version: 4.2.6
[ dbug ] (**) NAS   :      firmware build: 20191107
[ dbug ] (**) NAS   :              kernel: 3.4.6 i686
[ dbug ] (**) NAS   :           OS uptime: 3:01
[ dbug ] (**) NAS   :         system load: 1 min=0.29, 5 min=0.34, 15 min=0.32
[ dbug ] (**) NAS   :                EUID: 0
[ dbug ] (**) NAS   :      default volume: /share/MD0_DATA
[ dbug ] (**) NAS   :               $PATH: /opt/sbin:/opt/bin:/sbin:/bin:/usr/sbin:/us
[ dbug ] (**) NAS   :                /opt: /share/MD0_DATA/.qpkg/Optware
[ dbug ] (**) NAS   :     /share/Download: MD0_DATA/Qdownload
[ dbug ] (**) SCRIPT:      user arguments: SABnzbd
[ dbug ] (**) SCRIPT:       target app(s): SABnzbdplus
[ dbug ] (**) SCRIPT:        working path: /share/Public/sherpa.tmp
[ dbug ] (**) QPKG  :       download path: /share/Public/sherpa.tmp/qpkg-downloads
[ dbug ] (**) IPKG  :       download path: /share/Public/sherpa.tmp/ipkg-downloads
[ dbug ] (**) QPKG  :                arch: x86
[ proc ] testing Internet access ...
[ fail ] No Internet access
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (<<) <EnvironCheck> [13]
[ dbug ] (>>) <Cleanup>
[ dbug ] (<<) <Cleanup> [13]
[ dbug ] (>>) <DisplayResult>
[ fail ] 'SABnzbdplus' install failed! [13]
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) SCRIPT:            finished: Fri Jan  3 17:52:50 CST 2020
[ dbug ] (**) SCRIPT:        elapsed time: 00h:00m:01s
[ dbug ] (II) ========================================================================
[ dbug ] (<<) <DisplayResult> [13]

Watcher3 not shipped with initial config

display bug: unwanted empty line after "installed file"

as per https://forum.qnap.com/viewtopic.php?p=746363#p746363

add option to backup all installed package configs

User specifies --backup, all config are zipped and placed into [sherpa.sh] working directory. Failing that, maybe [/share/Public]?

update Entware to 1.01

Can also remove Entware-ng as installation candidate - there's now one package to rule them all.

SickRage Script

Hey,

Latest version of SickRage seems to have renamed the init script from SickBeard.py to SiCKRAGE.py and launching the package now throws the error:

/opt/bin/python2.7: can't open file 'SickBeard.py': [Errno 2] No such file or directory

Updating the local TARGET_SCRIPT within ./sickrage.sh doesn't seem to resolve the issue; SickRage fails to start.

add NZBGet as installable package

Upgrade Medusa Python version to >= 3.5

Hello! Medusa dev here ✋

We have noticed that your script is a popular way to install Medusa! 🎉
I'm here to inform you that we are fully Python 3 compatible since a few releases.

As you may already know Python 2.7 has reached its EOL on 01.01.2020.
For this reason we will end support for any Medusa installs using Python 2.7 in the near future.

We kindly ask you to update the Python version to >= 3.5 (preferably 3.6 or 3.7) as soon as possible. The update won't require any intervention from the end users.

Thank you for your continued support 👏

on-screen debugging is starting too late to show debug helper text

add support for new 'Entware' QPKG

The internal package name is 'Entware'.

all QPKGs should update qpkg.conf with web port address

Code for this is only used in the SABnzbd QPKG init.

Need to add it to every other sherpa QPKG init.

create a Transmission QPKG using IPKG

main: enable multiple package installs

Presently, sherpa requires multiple runs to install multiple packages. Let's get it processing multiple arguments.

restart all sherpa QPKGs after reinstalling Entware

update fail due to pip3 missing, QTS-4.3.6, sabnzbdplus

Using sherpa.sh I've been running sabnzbdplus on QTS 4.3.6.1070 for maybe a year now. Recently I ran into some par2 failures and decided to see if I could update anything using sherpa.sh. When I tried to update, I got a Fail message about pip3 not being found.

Details:

sabnzbd is running and has been working well, except for the occasional par2 failure.

I downloaded the latest sherpa.sh following the directions on p1 of the forum thread.

I ran --update-all, and this is the error I got.

[/share/Public] # ./sherpa.sh --update-all
sherpa.sh (200610)

[ done ] Internet is accessible            
[ done ] updated 'Entware' package list            
[ done ] 2 IPKGs (292kB) to be downloaded                              
[ done ] downloaded & installed 2 IPKGs                         
[ done ] downloaded & installed Python 2 modules             
[ fail ] A required NAS system file is missing (/opt/bin/pip3)

I haven't tried to fix it yet. I would need specific directions I think.
Here's sherpa.debug.log

Thanks for working on this!

use QTS BusyBox 'grep' if possible

The new DowngradePy3() function doesn't cherry-pick a grep, so it uses Entware if $PATH has been modified, and QTS if not.

In this function grep uses the -o option, which isn't available in some older QTS BusyBoxes.

Need to recode using QTS BusyBox grep and without the -o option.

Thanks to redmercury2006 for finding this. :)

ensure SickChill shows a version number in configuration screen

Need to clone from latest tag forward to most recent commit. It did this previously - just need to restore code.

main log should also contain contents of temp logs

When a package download (or install) operation fails, the contents of that specific operation log should be inserted into the main run log so we don't have to locate the specific operation log afterward.

check compatibility with existing Optware-NG installations

Probably don't want to install Entware if Optware-NG is already installed.

error when installing SABnzbdplus via sherpa

Hi
I'm getting the following error when attempting to install SABnzbdplus on a QNAP TVS-682T. Entware has installed ok via sherpa.

sherpa.sh (200108)

[ done ] Internet is accessible
[ done ] downloaded file (SABnzbdplus_190928.qpkg)
[ done ] updated Entware package list
[ done ] no IPKGs are required
[ fail ] Download & install PIP modules failed [1]

[ fail ] 'SABnzbdplus' install failed! [19]

As requested I've attached the sherpa.debug.log
sherpa.debug.log

Any assistance would be greatly appreciated.

speedup IPKG size calc

Current method is horribly inefficient and slow.

Should only check each dependency once.

permit Entware reinstallation without requiring another QPKG install too

find a new Python 2.7.16 source

Python 2 has been pulled from Entware/OpenWRT.

SABnzbd 2.3.9, SickChill and Headphones packages are temporarily unavailable.

SABnzbd cannot start after downgrading Python 3

Install SAB3, then install Watcher3.

Python 3.8.1 is downgraded to 3.7.4-2

Restart SAB3.

-------------------- start requested --------------------
= Sun May 31 08:58:31 EST 2020
= (SABnzbd) is not active
* updating (SABnzbd): OK
= result: 0
= PullGitRepo(): 'Already up to date.'
* updating (nzbToMedia): OK
= result: 0
= PullGitRepo(): 'Already up to date.'
* starting (SABnzbd): failed!
= result: 1
= startup messages: 'Not all required Python modules are available, please check requirements.txt
Missing module: cryptography
You can read more at: https://sabnzbd.org/python3
If you still experience problems, remove all .pyc files in this folder and subfolders'

SABnzbd UI for Python 3 crashes with a 500 error on first run

But, restarting it then works:

/etc/init.d/sabnzbd3.sh restart

add download speed to display

Not sure if this can be done, but should be investigated anyway. Even if the speed isn't 'live', let's record it in the debug log.

add support for replacing QSickRage with SickChill

package inits can fail but sherpa shows install as fine

Clearly, return codes from the init scripts are being lost on the way back to sherpa. 😕

These packages have been fixed:

LazyLibrarian is moving

Hi,

Just a heads-up, on Saturday (10 Nov) we will be moving the repo to

https://gitlab.com/LazyLibrarian/LazyLibrarian.git

The github/dobytang one will be mothballed as dobytang is no longer around and we cannot get full access rights to it. We would have liked the LazyLibrarian group at github as that's inactive but the owner hasn't responded to requests to release it, so we're moving to gitlab.

The new location also has a brand new much needed documentation section, and we hope to include automated builds of deb rpm and snap packages.

Many thanks for your support of lazylibrarian

Phil.

QBoost removes 'wait-for-Entware.sh'

@tazzi has reported that QBoost removes this script from the .qpkg root, so I'll move that code into the top of each QPKG init script.