https://github.com/aardappel/lobster

This ticket is about recurring small updates in installation scripts under folders user-install and sysadmin-install.

NOTE 1: Branch https://github.com/frgomes/bash-scripts/tree/minor_updates runs in parallel to master.

NOTE 2: This ticket is not expected to be closed for the lifetime this repository is maintained.

Hello,
I copied the script into my Linux system (chmod a+x was done) and for a test want to get the script itself be downloaded in /tmp

Would you please so nice to tell me how this can be achieved. I found an example how to call it here but did not work:
https://stackoverflow.com/questions/60190759/how-do-i-clone-fetch-or-sparse-checkout-a-single-directory-or-a-list-of-directo/60190760#60190760

Help would be appreciated.

It's also necessary to install Ant and make Ant import Ivy.
https://ant.apache.org/ivy/history/2.5.0-rc1/install.html

I'm not sure why... but existing scripts which were working fine stopped work properly when wget is called.
It may be a misconfiguration or just a bug somewhere else... but the point is that the toolbox sometimes uses wget sometimes use curl. We should use only one of those.

• Replace wget by curl in files under bin folder;
• Replace wget by curl in all other files

• kpvm
• cryptsetup

Related: #23 #35

Background

Application installation scripts under folder user-install aim to install applications without need of root access, at least in theory if everything goes according to plan.

Every application has it's own behaviour and requirements, meaning that there's no uniform way to tell what the end user has to do in order run a certain application of interest. Certain applications require the definition of environment variables, certain applications are provided as container images and so on.

In order to tackle this problem, when an application is installed we provide a shell script which initializes whatever needs to be initialized in order to make it work out when required. In particular, a very common thing involves updating the PATH environment variable so that the installation folder is included.

Terminology

With so many scripts all over the place eventually changing PATH, it becomes necessary to make it clear what we are talking about:

• postactivate is a shell script created by mkvirtualenv in a virtual environment of your choice. This script is initially empty. You can add logic to it according to your requirements.

• Installation scripts are shell scripts under folder user-scripts of this project. These scripts may download applications, install them under a certain folder and may create "application activation scripts" under a standard location.

• Application activation scripts should not be confused with postactivate scripts. Application activation scripts are created by installation scripts, managed by us. These application application scripts should not be touched by the user in general. These scripts must be executed in order to configure the user session so that the application can be launched when the user requests it.

What this ticket is about

One issue we had before was that it was necessary to add some logic manually to postactivate scripts (i.e.: in virtual environments) in order to guarantee that all application activation scripts are executed when the user switched from one virtual environment to another.

The installation scripts should be clever enough to do everything automatically. So, if application activation scripts need to be executed... and this seem to be inescapable... the installation scripts should add some logic to the postactivate script of the current virtual environment, if any.

Also, application activation scripts should also be executed in case the user is not using virtual environments at all.

But the most important aim of this ticket is providing the ability to install/activate multiple versions of applications. In a nutshell: you may be interested to have:

• Python2, Java8, Scala11 in virtual environment p2j8s11;
• Python3, Java12, Scala13 in virtual environment p3j12s13.

This will be possible easily, since each virtual environment has its own postactivate script and it's private set of application activation scripts.

Despite all the flexibility depicted above, nothing prevents you of simply ignoring virtual environments entirely, in case you don't need them and you simply don't care. In this case, the current implementation provided by #35 already provides a fall back behaviour which does just that.

Tasks

• Make sure that installation scripts save application activation scripts under current virtual environment, if any. Otherwise, fall back to ~/.bashrc-scripts/installed.
• Make sure that installation scripts have ability to instrument the postactivate script under the current virtual environment, if any. This way, activating a virtual environment triggers the execution of all application activation scripts. Nothing needs to be done in case there's no current virtual environment.
• Make sure that configuration at startup run application activation scripts from hinted virtual environment, if any. Otherwise, fall back to ~/.bashrc-scripts/installed.

Add comprehensive documentation about:

• environment variables and their use.
• virtual environments and application installation.
• initial setup process and how to pass a hint of a virtual environment.

Related: frgomes/.emacs.d#3

Executing user-install scripts should install binaries, etc under $HOME/.local.
A possible alternate path would be detecting that root is running user-install scripts and, in this case, allow installation under /usr/local.

Apparently, install-jupyter.sh does everything it should do and always did well.
However, command jupyter is not available in the environment.

After some investigation, it became clear what the problem was:

1. I've removed Debian packages virtualenv, virtualenv-wrapper and friends;
2. I'm now using the simple, barebones, replacement for commands workon and deactivate, i.e.: instead of using these commands provided by virtualenv-wrapper, I'm now using some shell functions which are small, very simple and avoid pulling a full blown of undesirable dependencies. In particular, in server and restricted environments, we are definitely not interested on lots of dependencies pulled by virtualenv and friends.
3. pip3 install --verbose jupyter revealed that jupyter is under $HOME/.local/bin

So, in a nutshell, we need to make sure that $HOME/.local/bin is in $PATH.

Stride was called HipChat before.
https://www.stride.com/downloads

sudo sh -c 'echo "deb https://packages.atlassian.com/debian/stride-apt-client $(lsb_release -c -s) main" > /etc/apt/sources.list.d/atlassian-stride.list'
wget -O - https://packages.atlassian.com/api/gpg/key/public | sudo apt-key add -
sudo apt-get update
sudo apt-get install stride

Help wanted: I need volunteers willing to test drive on CentOS, since I don't use it anymore.

At the moment, bash-scripts rely on apt-get for everything.

We should support :

• apt on Debian and derivative distributions;
• yum on RedHat based distributions, such as CentOS, Fedora and RHEL;
• zypper on OpenSUSE;
• something else somewhere else?

Related: #37

Functions named ff* and fg* should perform better, should be simpler and should avoid getting confused with file names containing white spaces.

In particular, we should discard .git and .hg directories early in the find command which runs as part of these functions, instead of obtaining a full dataset of file names and only after that getting rid of all files under .git and .hg folders.

https://developer.gnome.org/integration-guide/stable/desktop-files.html.en

Modern distributions offer Firefox and Thunderbird at system level.

The intent of our installation process is leveraging recent or even nighlty versions Firefox and Thunderbird for power users.

For this reason, our installation should be done at user's level and not at system's level.

https://github.com/eclipse/lemminx

This package should be generic and usable by everybody, not only me.

At the moment, I'm sharing my virtual environments, which may prevent other users of using this project.

• convert scripts under user_install to scripts under bin
• provide nice usage screen messages for all install_* scripts

Remove hardcoded dependency from $HOME/scripts

$ fgfile HOME | fgrep -v _HOME | fgrep -v /Downloads  | fgrep -v /websites | fgrep -v /bin | fgrep -v /.config | fgrep /scripts
./README.md:35:$ source $HOME/scripts/bashrc
./README.md:38:Then add a call to ``$HOME/scripts/bashrc`` into your ``$HOME/.bashrc``:
./README.md:41:$ echo 'source $HOME/scripts/bashrc' >> $HOME/.bashrc
./postinstall-user.sh:79:  [[ ! -e $HOME/scripts ]] && ln -s $HOME/workspace/bash-scripts $HOME/scripts
./bash_04virtualenv.sh:6:  if [ -d $HOME/scripts/bashrc-virtualenvs ] ;then
./bash_04virtualenv.sh:7:    pushd $HOME/scripts/bashrc-virtualenvs > /dev/null
./bash_04virtualenv.sh:46:    ln -s $HOME/scripts/bashrc-virtualenvs/${source} ${target}

When scripts run under sudo, references like
source ~/scripts/...
... become equivalent to
source ~root/scripts/...
... which is incorrect.

The current user must be kept.

Related: #23 #36

Background

Virtual environments is an interesting concept, heavily used by Python programmers. Since I usually use several versions of Java and Scala, I borrowed the concept from the Python ecosystem given that in reality, the benefits apply equally to several other ecosystems, not only Python.

Some time ago, my preferred virtual environments were part of the codebase and were initialized (if necessary) when a new terminal window was opened. The reasons for that are explained in Design Concept below.

Since this project started to become more popular, it became clear that I should never push my virtual environments to users of this project. For this reason, I've moved my virtual environments to somewhere else. However, there are still leftovers around here and, in particular, I'm still forcing a "default" (let's put this way) virtual environment called p3 to users of this project.

Design Concept

Starting from a brand new laptop with only the base operating system installed, I would like to be able to quickly have my environment setup. I would like to download a shell script from the Internet which sets up everything for me. Then I open a new terminal window, all remaining bits and pieces are automagically configured and I'm ready to go.

Requirements

• Support for virtual environments is enable only if there's support for Python in the underlying distribution. Even though the vast majority of distributions have some sort of Python version pre-installed, some small distributions don't. And we should accommodate for that.
• Given that Python is supported but pip is not available, we should download pip from the Internet and install it.
• Given that pip was successfully installed, we then install/upgrade virtualenv and virtualenvwrapper.
• Given that virtualenvwrapper was successfully installed, we should source virtualenvwrapper.sh in order to make commands workon and mkvirtualenv available, among other commands.

Additional tasks:

• We should NOT enforce or configure any virtual environment, i.e.: p3 should go!
• We should make sure that the Downloads folder is configurable.
• Review current documentation.
• Add temporary hack. See: #36

Related: #19 #23 #35 #37

Background

One problem detected along the way with daily usage of bash-scripts is that functions and aliases many times are not visible depending on the operation being employed. This is daunting and unfortunate.

Everything (or almost everything!) we define should always be available for end users, no matter if calling a functionality from the terminal, from shell scripts, from functions, shells or subshells or whatever it might be.

About this ticket

We should convert (potentially!) everything into shell scripts, instead of functions and aliases. This way, given that bash_scripts is on $PATH, all functionality should always be visible, since shell scripts sitting on the file system are first class citizens.

Performing such code review and reorganization is a big task and will take a lot of effort, but it is also an opportunity for addressing other issues still open, such as #36, #37 and #19.

In scope

• turn functions onto shell scripts;
• find a reasonable solution for aliases;
• #37: make sure that file names are always between quotes, so that whitespaces are supported;
• #19: make sure we employ tar -C, git -C, etc wherever possible, avoiding cd and pushd;
• install python3, python3-pip, if necessary;
• employ python3-venv;
• should work just fine with virtual environments;
• should work just fine without virtual environments;
• make sure that it does not pollute the user's environment variables in any way;
• make sure that the current directory the user is at is always preserved;

Write documentation about:

• Python packages to be removed
• How virtual environments are managed
• convenience command workon

Files and directories containing whitespaces is an inconvenient reality for people forced to use Windoze by their employers or customers.

This ticket aims to try to make life easier in this case, making this toolbox "whitespace friendly".

Some interesting ideas in this article:
https://www.linuxjournal.com/article/10455

ditto.

Related: #35 #36

Remove hardcoded dependency from $HOME/workspace

$ fgfile HOME | fgrep -v _HOME | fgrep -v /Downloads  | fgrep -v /websites | fgrep -v /bin | fgrep -v /.config | fgrep /workspace
./bash_50mobile.sh:48:###   g8 file:///${HOME}/workspace/mobile.g8 --name=$project --package=$package -b expo-support
./README.md:30:$ mkdir $HOME/workspace && cd $HOME/workspace
./README.md:34:$ ln -s $HOME/workspace/bash-scripts scripts
./README.md:76:  *)            export WORKSPACR=$HOME/workspace
./bash_20functions.sh:146:  local dir=${WORKSPACE:=$HOME/workspace}
./bash_20functions.sh:157:  local dir=${WORKSPACE:=$HOME/workspace}
./bashrc-tricks/bashrc.scripts.before:15:  *)                   export WORKSPACR=$HOME/workspace
./bash_50scala.sh:47:  pushd $HOME/workspace/scala-native
./bash_62containers.sh:8:WORK_WORKSPACE=${HOME}/workspace
./bash_25scm.sh:175:    if [ ! -d $HOME/workspace ] ;then
./bash_25scm.sh:176:        mkdir -p $HOME/workspace
./bash_25scm.sh:178:    pushd $HOME/workspace
./bash_25scm.sh:180:        pushd $HOME/workspace
./bash_25scm.sh:185:        pushd $HOME/workspace/${module}
./bash_25scm.sh:199:    if [ -d $HOME/workspace/${module} ] ;then
./bash_25scm.sh:200:        pushd $HOME/workspace/${module}
./postinstall-user.sh:70:  [[ ! -d $HOME/workspace ]] && mkdir -p $HOME/workspace
./postinstall-user.sh:71:  pushd $HOME/workspace
./postinstall-user.sh:79:  [[ ! -e $HOME/scripts ]] && ln -s $HOME/workspace/bash-scripts $HOME/scripts
./postinstall-user.sh:83:  [[ ! -d $HOME/workspace ]] && mkdir -p $HOME/workspace
./postinstall-user.sh:84:  pushd $HOME/workspace
./user-install/install-mxnet.sh:8:    cd $HOME/workspace
./bash_50jupyter.sh:5:  if [ ! -d $HOME/workspace/jupyter-notebooks ] ;then
./bash_50jupyter.sh:6:    [[ ! -d $HOME/workspace ]] && mkdir -p $HOME/workspace
./bash_50jupyter.sh:7:    pushd $HOME/workspace
./bash_50jupyter.sh:11:  pushd $HOME/workspace/jupyter-notebooks

https://github.com/apple/sourcekit-lsp
https://clangd.llvm.org/

We should refrain of cd/pushd all over the place.
In most cases we are doing that just before performing a tar, git or zip but all these applications allow passing option -C for convenience.

See PR #34

This is an alternative to ddclient.
Source here:
https://gist.github.com/frgomes/9346671a12b621ffbf8e57f8accdeece

This may help, maybe:

apt install kde-zeroconf avahi-ui-utils avahi-discover avahi-dnsconfd mdns-scan

http://www.linux-magazine.com/Online/Features/AirPrint
https://www.youtube.com/watch?v=O4sSNaTzTaQ

These are some options:

• https://github.com/sonota88/anbt-sql-formatter/tree/master/lib/anbt-sql-formatter, which is written in Ruby and apparently installs only in system space.
• https://github.com/nrempel/sleek which is written in Rust and installs in the userspace

ditto

At the moment it is being installed as system packages.

https://github.com/REditorSupport/languageserver

https://github.com/KDE/extra-cmake-modules

If I try to use a branch tag in the git_sparse_checkout script I get an error in the checkout step:
error: pathspec 'branch_tag_name' did not match any file(s) known to git

I found that if I use the branch instead of the branchtag and then add additional commands:
git -C /path/to/submodule fetch -t -depth=1
git -C /path/to/submodule checkout branchtag

It will do what I need it to do, however I think that the fetch command could be improved to get less.

Decide how scripts which require root privileges should be handled.

Chances are that they simply become scripts under folder sbin or maybe simply bin, but explicitly warn that root privileges are necessary.

• #61: Sort out applications dependent on .deb packages;
• convert scripts under sysadmin_install to scripts under sbin
• provide nice usage screen messages for all install_* scripts

hi,
i tested git_sparse_checkout main [email protected]:NicolasMICAUX/github_test.git -- README.md
and git_sparse_checkout [email protected]:NicolasMICAUX/github_test.git -- README.md but i don't understand the message i get :
fatal: impossible de trouver la référence distante master error: le spécificateur de chemin 'master' ne correspond à aucun fichier connu de git

i want to get only README.md from the test repo [email protected]:NicolasMICAUX/github_test.git

Thanks for your help

• Reorganize post-install scripts so that they can be run separately;
• Reorganize other scripts somewhere else which should run by root;
• Make sure that installation of WP-34s works;
• Provide a script which runs them all.

These are applicaitons which depend on .deb packages:

$ rg [.]deb

sysadmin-install/install-openfire.sh
7:  sudo dpkg -i "${DOWNLOADS}"/openfire_4.3.1_all.deb 

sysadmin-install/install-slack.sh
6:  local file=slack-desktop-${version}-${arch}.deb

sysadmin-install/install-mozilla-sops.sh
5:  local file=sops_${SOPS_VERSION}_amd64.deb

sysadmin-install/install-keybase.sh
4:  local file=keybase_amd64.deb

sysadmin-install/install-printer-dcp9020cdw.sh
19:  download http://download.brother.com/welcome/dlf100441/dcp9020cdwlpr-1.1.2-1.i386.deb
20:  download http://download.brother.com/welcome/dlf100443/dcp9020cdwcupswrapper-1.1.4-0.i386.deb
21:  download http://download.brother.com/welcome/dlf006645/brscan4-0.4.4-4.amd64.deb
22:  download http://download.brother.com/welcome/dlf006652/brscan-skey-0.2.4-1.amd64.deb
23:  download http://download.brother.com/welcome/dlf006654/brother-udev-rule-type1-1.0.2-0.all.deb
25:  if [ -f "${DOWNLOADS}"/dcp9020cdwlpr-1.1.2-1.i386.deb -a \
26:       -f "${DOWNLOADS}"/dcp9020cdwcupswrapper-1.1.4-0.i386.deb -a \
27:       -f "${DOWNLOADS}"/brscan4-0.4.4-4.amd64.deb -a \
28:       -f "${DOWNLOADS}"/brscan-skey-0.2.4-1.amd64.deb -a \
29:       -f "${DOWNLOADS}"/brother-udev-rule-type1-1.0.2-0.all.deb ] ;then \
33:          "${DOWNLOADS}"/dcp9020cdwlpr-1.1.2-1.i386.deb \
34:          "${DOWNLOADS}"/dcp9020cdwcupswrapper-1.1.4-0.i386.deb \
35:          "${DOWNLOADS}"/brscan4-0.4.4-4.amd64.deb \
36:          "${DOWNLOADS}"/brscan-skey-0.2.4-1.amd64.deb \
37:          "${DOWNLOADS}"/brother-udev-rule-type1-1.0.2-0.all.deb \

sysadmin-install/install-remarkable.sh
13:  download https://remarkableapp.github.io/files/remarkable_${REMARKABLE_VERSION}.deb
17:  sudo dpkg -i "${DOWNLOADS}"/remarkable_${REMARKABLE_VERSION}.deb

sysadmin-install/install-discord.sh
7:  local file=discord-${version}.deb

sysadmin-install/install-zoom.sh
4:  local file=zoom_amd64.deb

sysadmin-install/install-skype.sh
4:  local file=skypeforlinux-64.deb

ditto

https://github.com/dhall-lang/dhall-haskell/tree/master/dhall-openapi

Please let me know how you use bash-scripts, your opinion, suggestions and criticisms. From this discussion here we will be deriving more specific tickets.