https://verdaccio.org/docs/en/docker

To speed up installs

after renaming the project:exec command to project:shell. #35

we can introduce a new project command project:exec which executes a given command in the container

dde project:exec bin/console --help

dde project:exec-root rm -rf /var/www/var/cache

nginx-proxy allows having Multiple Hosts

if we define in the docker-compose.yml something like this:

        environment:
            - VIRTUAL_HOST=xxx.test,api.xxx.test

it will output:

Finished startup successfully
visit: https://xxx.test,api.xxx.test

which then does not open correctly.

fix should be here:

in newer Docker Version the compose.yml is used instead of docker-compose.yml

Using compose.yml throws the dde error docker-compose.yml not found

Problem

We should check if a new version requires dde to upgrade the database (for example after the mariadb image has been updated).
Otherwise it will crash like this:

[ERROR] InnoDB: Plugin initialization aborted with error Generic error
[Note] InnoDB: Starting shutdown...
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[Note] Plugin 'FEEDBACK' is disabled.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
[Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.4+maria~ubu2204 started.
[Note] [Entrypoint]: Switching to dedicated user 'mysql'
[Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.4+maria~ubu2204 started.
[Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
[Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
[Note] Starting MariaDB 10.11.4-MariaDB-1:10.11.4+maria~ubu2204 source revision 4e2b93dffef2414a11ca5edc8d215f57ee5010e5 as process 1
[Note] InnoDB: Compressed tables use zlib 1.2.11
[Note] InnoDB: Number of transaction pools: 1
[Note] InnoDB: Using ARMv8 crc32 + pmull instructions
[Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
[Note] InnoDB: Using liburing
[Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
[Note] InnoDB: Completed initialization of buffer pool
[Note] InnoDB: Buffered log writes (block size=512 bytes)
[ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.21. You must start up and shut down MariaDB 10.7 or earlier.
[ERROR] InnoDB: Plugin initialization aborted with error Generic error
[Note] InnoDB: Starting shutdown...
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[Note] Plugin 'FEEDBACK' is disabled.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting

Current workaraound/fix:

docker exec -it dde_mariadb mysql_upgrade -uroot -proot

if the project is not up, start it command exec or exec-root is callled

Description
When executing the dde system:update command, an error occurs if the dde network is not present. This results in an error message and the termination of the script. The exact error is: Error response from daemon: network dde not found.

Steps to Reproduce

1. Run the dde system:update command without the dde network being present.
2. Observe the error.

Expected Behavior
The script should handle the absence of the dde network gracefully and display an appropriate message instead of failing with an error.

Actual Behavior
The script crashes with the error message Error response from daemon: network dde not found.

Possible Solution
Add a check in line 11 of the destroy.sh script to verify if the dde network exists before attempting to remove it. This could be achieved with the following bash code:

if docker network ls | grep -q dde; then
  docker network rm dde
else
  echo "Network 'dde' not found, no removal necessary."
fi

Removing network if created
Remove network dde
dde
Finished destroying successfully
Removing data
find: ‘./data/mailhog’: No such file or directory
find: ‘./data/mariadb’: No such file or directory
find: ‘./data/reverseproxy’: No such file or directory

Currently nuke only works if the current directory is the dde directory.

If the current directory is the dde directory #12 occurs

export NODE_EXTRA_CA_CERTS=~/dde/data/reverseproxy/etc/nginx/certs/ca.pem

dde system-nuke delete's data/reverseproxy/etc/nginx/conf.d/dde.conf and recreates it as a directory on dde system-up.
After this the container is stuck in a reboot-loop until the file is replaced.

$ docker logs -f dde_reverseproxy

nginx.1    | 2020/01/24 12:40:58 [crit] 28#28: pread() "/etc/nginx/conf.d/dde.conf" failed (21: Is a directory)
forego     | starting nginx.1 on port 5200
forego     | sending SIGTERM to nginx.1
forego     | sending SIGTERM to dockergen.1
forego     | starting dockergen.1 on port 5000
forego     | starting nginx.1 on port 5100
nginx.1    | 2020/01/24 12:41:51 [crit] 28#28: pread() "/etc/nginx/conf.d/dde.conf" failed (21: Is a directory)
forego     | starting nginx.1 on port 5200
forego     | sending SIGTERM to nginx.1
forego     | sending SIGTERM to dockergen.1
forego     | starting dockergen.1 on port 5000
forego     | starting nginx.1 on port 5100
nginx.1    | 2020/01/24 12:42:52 [crit] 33#33: pread() "/etc/nginx/conf.d/dde.conf" failed (21: Is a directory)
forego     | starting nginx.1 on port 5200
forego     | sending SIGTERM to nginx.1
forego     | sending SIGTERM to dockergen.1
forego     | starting dockergen.1 on port 5000
forego     | starting nginx.1 on port 5100
nginx.1    | 2020/01/24 12:43:53 [crit] 27#27: pread() "/etc/nginx/conf.d/dde.conf" failed (21: Is a directory)
forego     | starting nginx.1 on port 5200
forego     | sending SIGTERM to nginx.1
forego     | sending SIGTERM to dockergen.1

f.ex

WARN[0000] network default: network.external.name is deprecated in favor of network.name
WARN[0000] volume ssh-agent_socket-dir: volume.external.name is deprecated in favor of volume.name

would be easy to migrate automatically on dde update with yq or similar

There's no note about how to install/use Mutagen.

It's only listed in the dependencies table, which isn't enough.

TBD

https://mutagen.io

SSH Key missing is missing after host restart respectively after ssh-agent container restart without using dde

the command project:exec is ambiguous.

project:exec make essentially a docker compose exec sh, so it opens a shell in the container.

to further develop dde i like to introduce a project-command #36 to execute a command in the container, for example docker compose exec bin/console --help. but how call this new project-command?

so to clarify the project-command i like to rename:

• project:exec to project:shell
• project:exec-root to project:root-shell

rm .docker-sync/daemon.pid

Happens if docker-sync container gets forcibly shut down (e.g. kernel panic of host system). docker-sync will try to kill previously running daemon on startup but fail since it doesn't exist anymore.

Description

I propose implementing a functionality in dde that automatically checks and sets the permissions in the DATA_DIR or DDE_DATA_DIR before the application starts, using chown based on the environment variables DDE_UID and DDE_GID. This aims to improve user experience and prevent issues with access rights.

Proposed Solution

Implementation of a Bash function to be executed as part of the startup process:

function set_correct_permissions() {
    if [ -d "$DDE_DATA_DIR" ]; then
        echo "Setting correct ownership for the directory $DDE_DATA_DIR to UID: $DDE_UID and GID: $DDE_GID"
        chown -R "$DDE_UID":"$DDE_GID" "$DDE_DATA_DIR"
    else
        echo "Directory $DDE_DATA_DIR does not exist."
    fi
}

Expected Benefit

This adjustment would simplify handling for the user and ensure that permission problems are avoided.

For multi-container projects (e.g. decoupled API/frontend) it should be possible to run dde exec [container] to get a shell session on that container.

If you use a passkey with your ssh key and you try to share this key with your dde, you have to run dde system:update pretty much every day because your ssh key is locked again.
Instead, it would be cool to have a command that just reactivates the ssh key and adds it to the docker containers again.
This would cut the waiting time because the containers would not have to be rebuilt (and downloaded again).

Docker Compose is now in the Docker CLI, try docker compose up

Run in separate worker to improve performance: https://de.slideshare.net/radekb/docker-for-mac-local-developer-environment-optimization

It has been observed that the scripts shell.sh/shell/root.sh and exec.sh/exec/root.sh contain similar code blocks, complicating maintenance and potentially leading to inconsistencies.

Improvement Suggestion:

Consolidating the duplicate code into shared internal functions could enhance code quality and simplify maintenance:

1. Identification of duplicate code.
2. Development of internal functions for shared logic.
3. Replacement of the duplicate code with calls to these functions, followed by testing to ensure functionality.

Implementing these changes would significantly improve efficiency and maintainability.

The current list of commands when running dde, has descriptions for the flags like --autocomplete but no description for the individual commands.

Not all commands are self explaining. Like system:nuke and system:cleanup for example, do not really tell me how much I get into trouble when running these, or whether I get in trouble at all.

It would be helpful if all commands had a description like the two flags have. Or at least, if all commands are described in the readme.

Arguments:
   --autocomplete                  add `eval $(~/dde/dde.sh --autocomplete)`  to  your `~/.zshrc` or `~/.bash_profile`
   --help                          show extended help for command, if exists

System Commands:
   system:cleanup                  
   system:config                   
   system:dde:install              
   system:dde:install:alias        
   system:dde:install:autocomplete 
   system:destroy                  
   system:env                      
   system:exec                     
   system:list                     
   system:log                      
   system:logs                     
   system:nuke                     
   system:services                 
   system:ssh-key-add              
   system:status                   
   system:stop                     
   system:up                       
   system:update                   

Project Commands:
   project:destroy                 
   project:docker-override         
   project:env                     
   project:exec                    
   project:fix-permissions         
   project:log                     
   project:open                    
   project:services                
   project:shell                   
   project:shell:root              
   project:status                  
   project:stop                    
   project:up                      
   project:update                  

Local Commands:

deprecated native docker volumes should be used

When dde System containers have ports in conflict with other programs, there is currently no way to change them with docker-compose.override.yml due to the nature how the YAML arrays will be merged in this case.

The only solution in case of port conflicts, is to either kill the software that is using the same port as the dde service (not possible in my case), or to deactivate the dde system service all together with following lines in the docker-compose.override.yml:

Example for Mailcrab:

services:
    mailcrab:
        deploy:
            replicas: 0 # disables mailcrab as port is in conflict with protonmail bridge

According to other people, the PR #38 might make that possible, but is stale since a while ago.

Since the official acceptance of both .yml and .yaml extensions by Docker Compose exists, I propose implementing support for .yaml extensions for Docker Compose files in the DDE project. This extension would enable compatibility with a wider range of Docker Compose configurations and allow users to use their preferred file extension without the need to adjust file names for compatibility with the DDE project. Incorporating this change could increase flexibility for developers and simplify integration into existing workflows.

• needed for clean packaging

https://mutagen.io/documentation/transports/tunnels

This would allow us to handle container restarts / recreation more gracefully. If we can somehow include the tunnel broker in the dde system compose stack, that would be optimal - until then we could make this optional, and allow users to specify their mutagen.io credentials.

Needed when running different containers in a project.
example: dev-environment using fancy zsh stuff during development & a dependency running in production mode

Introduced in: #69

The newest yq version now parses comments in yml files, so the following docker-compose.yml will not work with dde exec anymore:

services:
    web: # Example web service

We should absolutely specify which yq-version to use, but also fix our parsing logic to filter out comments of service definition lines.

Maybe even get rid of it completely, and remove the "default to first service" logic?

normally, we use this synax:

    storage:
        container_name: example-storage
        image: minio/minio:RELEASE.2022-01-08T03-11-54Z
        environment:
            - VIRTUAL_HOST=storage.example.test

if we use this array syntax, the certificate wont be generated becuase we use a grep for VIRTUAL_HOST=

https://github.com/whatwedo/dde/blob/master/commands/project/up.sh#L17

    storage:
        container_name: example-storage
        image: minio/minio:RELEASE.2022-01-08T03-11-54Z
        environment:
            VIRTUAL_HOST: 'storage.example.test'

We should use YQ to read the environment variables.

instead of analyzing the YAML files use "docker-compose config".

docker-compose config, includes the docker-compose.override.yml file

https://mutagen.io/documentation/orchestration/compose

Analog to project:shell:root & project:shell

In Docker Compose version 2.24.7, the output format of environment variables (environments) in the docker-compose config command has changed. It no longer defaults to a dictionary format. Instead, the formatting is preserved as specified in the Docker Compose file: a dictionary remains a dictionary, and a list remains a list.

This change means that if the environment variables are defined as a list, scripts located at https://github.com/whatwedo/dde/blob/master/commands/project/up.sh#L17 and https://github.com/whatwedo/dde/blob/master/commands/_internals/openUrl.sh#L2 may encounter parsing issues with these values. As a workaround, it is recommended to define all environment variables in Docker Compose as dictionaries. Additionally, the mentioned lines in the scripts should be adjusted to accommodate this format change.

I just had the issue that I had checked out a feature of the dde-system.

While trying to setup a project, things failed and I didn't know why. It ended up that I did not checkout the master branch.
I wrongly thought a dde system:update would reset the branch and update to the latest version in the master branch.

There are two options for me to resolve this problem for others:

1. Automatically checkout the master branch. It could cause a problem when developing a feature (maybe handle this with a flat)
2. Output a warning into the command line when not on the master branch