deref / exo Goto Github PK
View Code? Open in Web Editor NEWA process manager & log viewer for dev
Home Page: https://exo.deref.io
License: Apache License 2.0
A process manager & log viewer for dev
Home Page: https://exo.deref.io
License: Apache License 2.0
Implement the process from the installtest repo for delivering a binary release to users.
Describe the bug
Long log lines are truncated and/or split across lines.
To Reproduce
Steps to reproduce the behaviour:
services:
web:
image: bash
command: "printf '.%.0s' {1..9000}"
exo run docker-compose.yaml
Expected behaviour
One log line printed with 9000 dots on one line in both the UI and the CLI. Instead what happens is that some smaller number of dots is printed in the UI (this seems to be consistently 8080) and none are printed in the CLI.
If the 9000 is increased to a much larger number the dots appear to get split across multiple lines in the UI but still don't sum to the correct number.
Describe the bug
Exporting a procfile does not preserve procfile order, and includes redundant PORT variables.
diff --git a/Procfile b/Procfile
index bd49e77..2903ab6 100644
--- a/Procfile
+++ b/Procfile
@@ -1,3 +1,3 @@
-server: ./script/dev-server.sh
-gui: ./script/dev-gui.sh
-storybook: ./script/storybook.sh
+gui: PORT=5100 ./script/dev-gui.sh
+server: PORT=5000 ./script/dev-server.sh
+storybook: PORT=5200 ./script/storybook.sh
To Reproduce
On current main, just do exo apply Procfile
and you'll immediately see an "out of date" warning in the GUI. Press "export" then, get diff above.
Expected behavior
If the PORT
would be redundant from the implicit port order in the procfile, we should omit the PORT=whatever
from the exported service lines in the outputted procfile.
Somehow, we should preserve the order of entries in the procfile. Not sure how this should work, maybe some kind of metadata on the components. Like procfile:index=2
or whatever & then a best-effort stable sort used in export? Open to other ideas too.
Describe the bug
In my actual local time, it is 12:38. The logs from tick
correctly show this time, but the timestamps we create on the far left shows 4:38, an incorrect time.
Expected behavior
Our timestamps === actual user local time.
System Info:
Describe the bug
If a docker-compose file doesn't specify a command it won't work with exo run
.
To Reproduce
Steps to reproduce the behavior:
FROM alpine
CMD ["sleep", "infinity"]
services:
alpine:
build: .
exo run docker-compose.yml
, which will give something like:โฏ dexo run docker-compose.yml
GUI available at: http://localhost:4000/#/workspaces/sfbbetx0x4y6bwf8keb9xv0mem
Error: applying manifest: internal server error
applying manifest: internal server error
In the server logs one sees:
20:00:39 server 2021/08/06 19:00:39 http 20b8fkx46p0y8zpfw8w0480e40: error processing request: adding "alpine": creating container: Error response from daemon: No command specified
Expected behavior
The compose file to run.
Not sure if this is intentional or not, but it leads to some annoying behavior when developing.
When I run exo run tools/goreman/procfiles/Procfile.rexec
with this procfile in our repo: https://github.com/buildbuddy-io/buildbuddy
app: bazel run enterprise/server -- --config_file=enterprise/config/buildbuddy.local.yaml
exec: bazel run enterprise/server/cmd/executor:executor -- --monitoring_port=9091 --executor.docker_socket=
redis: redis-server
And hit stop in the UI on exec
, I get no logs about a shutdown. When I hit play again - the error message
07:05:47 exec 2021/08/02 14:05:47.344 FTL listen tcp 0.0.0.0:9091: bind: address already in use
This leads me to believe the server that bazel run
spun up was never killed.
When I run the same exec
process on it's own in a terminal and hit Ctrl+C, I get the following logs:
^CCaught interrupt signal; shutting down...
2021/08/02 14:13:04.832 INF Stopping queue processing, machine is shutting down. name=rN5IABEi
2021/08/02 14:13:04.861 INF Graceful stop of executor succeeded.
Server "prod-buildbuddy-executor" stopped.
And don't get the port collision error when I start it again.
My workaround for now is to hit Ctrl+C on exo run tools/goreman/procfiles/Procfile.rexec
, which solves the issue - but means I have to restart all 3 processes.
as discussed
When starting a new process imported from a Procfile, exo automatically injects a PORT
environment variable into the new process. This can have unintended consequences. For example, my application relies on a config file to specify the port, but it allows PORT to override this value. I expect that if I do not set this environment variable in a Procfile
, .envrc
, or similar that it will not be set magically.
from exod.log:
2021/07/28 14:55:37 error processing request: running update script: fork/exec /var/folders/cn/q2gg_ww95f5249g5xtfm4jp40000gn/T/example827031344: no such file or directory
2021/07/28 14:56:38 error processing request: running update script: fork/exec /var/folders/cn/q2gg_ww95f5249g5xtfm4jp40000gn/T/example214949991: no such file or directory
via @siggisim
should also add check-engines
Foreman now supports cd
and command chaining within a Procfile, e.g.:
gui: cd gui && npm i && npm run dev
Exo does not allow this behviour. The workaround is the following:
gui: sh -c 'cd gui && npm i && npm run dev'
I think two things have to happen:
exo exit
if exo
is on the pathensureDaemon
health check should validate the version matches correctlybetween https://github.com/deref/exo/pull/42/files and #44 we should just give up on command line flags and pass supervisor config as json on stdin
Is your feature request related to a problem? Please describe.
When I am developing exo, I constantly have to switch my client config in config.toml
to point between the default installed version and the development version.
Describe the solution you'd like
I would like the exo
cli to be aware of config profiles. exo profiles ls
should display the available profiles with an *
next to my current profile, and exo profiles use <profile-name>
should set my current profile to <profile-name>
. exo profiles create <profile-name>
should create a new profile and set it as my default profile.
Describe alternatives you've considered
Additional context
This feature may make more sense after we have to ability to update config from the cli, since creating a profile would probably just entail creating a new, empty TOML file that still must be edited by hand.
Is your feature request related to a problem? Please describe.
When I have a large volume of logs, and I am looking for a particular string, I cannot do this easily with exo.
Describe the solution you'd like
I would like the log viewer to include a filter button. Clicking on this button should bring up a text box that I can type in to filter the logs. As I type, the logs are filtered to only those that contain the string that I have entered (case insensitive).
Describe alternatives you've considered
Go back to logging to a file then searching with grep
. ๐ข
There is no way to distinguish between inherited environment variables and blank environment variables.
See https://docs.docker.com/compose/environment-variables/#set-environment-variables-in-containers vs https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers
- PRESENT=1
- BLANK=
- INHERITED
This applies both to the UI and the CLI.
This should already be possible as part of the Cobra feature set, but we should figure out how to automate this as part of the install process.
Describe the bug
host.internal.docker is a host that is added on both Windows and MacOS but doesn't exist in Linux. Exo uses this host for the syslog-address.
We need to do something like this: https://stackoverflow.com/a/62431165
If you view a process that has lots of logs it will load the logs forwards from the first line. This can be very slow if there are many thousands of lines of logs. Instead I'd propose that we load the most recent logs and then page backwards.
Describe the bug
When logs from multiple processes are multiplexed in the terminal a process can interfere with the logs of another process. For instance, if process A emits \u001b[1mhello\nbye\u001b[0m
, equivalent to:
hello
bye
and process B emits oh no\n
, the output in the terminal can look like this:
13:38:26 hello
13:38:26 oh no
13:38:26 bye
Here the output of process B is incorrectly bold, as is the timestamp prefix added by exo.
Describe the bug
When using exo run
the first few seconds of logs are missing.
To Reproduce
Steps to reproduce the behavior:
services:
test:
image: bash
command: "bash -c 'i=0; while true; do echo $((i++)); sleep 1; done'"
โฏ dexo run docker-compose.y*
GUI available at: http://localhost:4001/#/workspaces/hk6vv1fe87qsjqn7kbb8ywaanr
Job URL: http://localhost:4001/#/jobs/t9sayfxh6031e8tjrk9f0pxq88
applying success
โโ updating test success
16:08:15 test 3
16:08:16 test 4
Notice that the first log line printed is "3" when it should have been "0".
We probably should make this configurable. It looks like 4500 is commonly used for IPSec NAT traversal, so this might break some VPNs.
Originally posted by @kendru in #23 (comment)
When an exo run
command fails it is unclear that this has happened in the UI.
To Reproduce
Steps to reproduce the behavior:
services:
alpine:
image: "alpine"
command: "notacommand"
exo run docker-compose.yml
โฏ dexo run docker-compose.yml
GUI available at: http://localhost:4000/#/workspaces/hah3yrwaa4mz7zp1sj275y5wkm
Expected behavior
Terminate the exo run
with a non-zero status code or at least output an error message on stderr.
Is your feature request related to a problem? Please describe.
When filtering logs, I usually want to search case insensitive, but sometimes I want to search case-sensitive.
Describe the solution you'd like
See "smartcase" in vim:
https://vim.fandom.com/wiki/Searching
I always have this on by default. Basically, if you search for an all-lowercase string, then the search is case insensitive. If there are any uppercase characters, then the search is case sensitive.
Describe alternatives you've considered
Providing some UI or search/filter directives for toggling case sensitivity on and off. I think that should probably happen first: some more general work to improve filtering/querying expression.
Is this correct? If I'm reading this right, we are looking for a log with the same name as the key, but the key is `(log + <255>, id).
Originally posted by @kendru in #23 (comment)
I did a fmt.Printf("blah")
(note: no \n
) and then no other log messages happened. No log messages got produced. This seems reasonable, but maybe we should have a timeout on individual log messages, and split them, so that no one else winds up in this confused position?
cc @siggisim
Describe the bug
When an exo run
command fails it is unclear that this has happened in the UI.
To Reproduce
Steps to reproduce the behavior:
services:
alpine:
image: "alpine"
command: "notacommand"
That will give an error like the following in the server but no error in the UI.
19:47:31 server 2021/08/06 18:47:31 starting container "73d8bd1d97b21c1c00d94a9b9b83fbc985829e3045597e8af9c223fd15f2cbf3": Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "notacommand": executable file not found in $PATH: unknown
Expected behavior
Some error message in the UI
Describe the bug
The UI doesn't show any processes or inform you that anything is happening when a docker image is building.
To Reproduce
Steps to reproduce the behavior:
exo run docker-compose.yaml
for a docker compose file for which you don't have any cached docker imagesExpected behavior
A loading bar or spinner should be displayed
Two main formats to support, in order of priority:
.env
- simple key/value pairs (trivially safe).envrc
- scripts as defined by https://direnv.net/ โ naive handling would be risky, running arbitrary code in a directory. have to consider security threat model.An enviornment file should be loaded prior to any process start, etc. The workspace configuration screen should have some UI for inspecting/editing these files too.
If an environment file changes (or some other way to detect env changes?), that should invalidate the spec for anything that depends on the environment (when we have a dependency system, at least). That is, we might want to cause an update
or even a destroy/recreate for some component.
docker-compose supports an --env-file
cli command. we should consider when/how env impacts individual commands, profiles, etc.
Also consider secrets.
exo feedback
Describe the bug
When using the dexo exit
command, it should exit the dev process. It currently exits the installed instance rather than the dev one.
To Reproduce
Steps to reproduce the behavior:
cd
to the repo root directoryexo run
, then press Ctl+C to stop the logs process.dexo status
dexo exit
Expected behavior
After exiting, dexo status
should indicate that the dev server is no longer running.
Is your feature request related to a problem? Please describe.
When processes start, stop, exit, etc, it's not obvious in the logs.
Describe the solution you'd like
There should be a log stream for system events like this, which appear in the aggregate log stream.
compose files supports a depends_on
option, documented here: https://docs.docker.com/compose/compose-file/compose-file-v3/#depends_on and here: https://github.com/compose-spec/compose-spec/blob/master/spec.md#depends_on
As of 8c40b56 we load compose files and ignore this configuration. Ignoring this configuration is reasonable, since well-behaved services need to fail gracefully if their dependencies are temporarily unavailable. There is prior art for this as well, since depends_on is not supported in Docker's Swarm mode.
However, we may want dependencies for other reasons. For instance, interactive start
operations should also start their dependencies by default.
If we do support depends_on
, we should do so in a more general way, as this affects orchestration in exo beyond just docker.
PortMapping
and Dictionary
in that package.compose.Convert
process, we should produce dependencies in whatever way we model that in the exo manifest. Lots of design work needed here, but I have some ideas worth discussing.The advertised UI in the readme is not the same as the web app.
Describe the bug
Service named with an underscore fails as invalid component name.
This is the same problem as #72
Expected behavior
We should apply the same solution as we did for procfiles. See this PR and the highlighted line in particular:
Note that I'd argue that underscores should be rejected by docker compose itself, especially given what the spec says about aliases and hostnames for links.
Containers for the linked service MUST be reachable at a hostname identical to the alias, or the service name if no alias was specified.
Hello,
My Procfile has some processes named like sidekiq_invoice_outbound
and exo does not accept that, I don't know why exactly.
Is your feature request related to a problem? Please describe.
I love using exo for logs and process management, but I find that I constantly go to my terminal to check what branch I have checked out with Git.
Describe the solution you'd like
I would love to have my current git branch displayed in some status area (like in VSCode). When I click on this, I would like to see a list of branches that I have checked out locally, and if I click on one, I would like to change to that branch. If there are changes preventing me from switching branches, it would be fine to show an error message.
Describe alternatives you've considered
Going back and forth between exo and my terminal ๐
Additional context
N/A
When the daemon is not running, the UI should display an "exo is not running" message.
The current one has a lots of problems, starting a list:
go run
leaves behind binaries (h/t @BenElgar)Perhaps more importantly, builtin file watcher support would be a welcome feature! Once we have custom exo manifests, we could add watch configuration for each service/process/container. Exo would then automatically restart processes via the control interface when things change.
We're using fonts that are not bundled with the release, so if the user doesn't have them installed, they don't work!
Is your feature request related to a problem? Please describe.
When something goes wrong between docker & exo, it may be difficult to figure out what docker resources need cleanup.
Describe the solution you'd like
labels/tags/annotations are the standard tool in cloud-computing world for this, and docker is no different. Docker compose sets a com.docker.compose.project
on every resource it creates and we should do the same.
Every container, network, and volume we create should get at least these two labels set:
io.deref.exo.workspace
- id stringio.deref.exo.component
- id stringOther labels that docker compose sets are worth studying and considering which apply to exo:
LABEL_CONTAINER_NUMBER = 'com.docker.compose.container-number'
LABEL_ONE_OFF = 'com.docker.compose.oneoff'
LABEL_PROJECT = 'com.docker.compose.project'
LABEL_WORKING_DIR = 'com.docker.compose.project.working_dir'
LABEL_CONFIG_FILES = 'com.docker.compose.project.config_files'
LABEL_ENVIRONMENT_FILE = 'com.docker.compose.project.environment_file'
LABEL_SERVICE = 'com.docker.compose.service'
LABEL_NETWORK = 'com.docker.compose.network'
LABEL_VERSION = 'com.docker.compose.version'
LABEL_SLUG = 'com.docker.compose.slug'
LABEL_VOLUME = 'com.docker.compose.volume'
LABEL_CONFIG_HASH = 'com.docker.compose.config-hash'
Beyond labels, we should probably produce better friendly names. Compose uses some code like this:
def build_container_name(project, service, number, slug=None):
bits = [project.lstrip('-_'), service]
if slug:
bits.extend(['run', truncate_id(slug)])
else:
bits.append(str(number))
return '_'.join(bits)
Do you think that we should just let microseconds be the highest resolution we support?
Originally posted by @kendru in #23 (comment)
Describe the bug
If a command outputs a vast amount of logs exo run doesn't appear able to stop.
To Reproduce
Steps to reproduce the behavior:
services:
busy:
image: bash
command: "bash -c 'while true; do echo hello; done'"
exo run
processExpected behavior
The workspace should stop, terminating the container.
My assumption is that this happens because of the very large amount of output that the script generates.
When we stop a process, we send SIGKILL to the logio process, orphaning the supervised process (and not stopping it). Instead, we should send a SIGTERM
to logio, indicating that it should gracefully stop its child process, wait for termination, then stop itself. Configurable timeouts should be used in both exo and the supervisor, after which the child process should be killed. @BenElgar let me know if you have questions.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.