emnudge / logpipe Goto Github PK

Some users have mentioned that the automatic log entry is jarring, despite this being the current behavior of every terminal.

I fear adding an animation here will have implications for future perf work in virtual lists. However, it could be interesting to explore.

We've had (#64, #67) some issues with regex performance.

We'll need to look into this going forward. As a quick fix to unblock the main thread and terminate long-running parsing stages, we can shove all this into a worker thread.

We'll need to serialize this data, but it shouldn't be too difficult.

Add a version type for v<num>.<num>.<num> or just any 3 numbers in general that are dot-separated.

Don't parse time if it's inside of a word. It should start and end on a word boundary.

Might want to also look into ANSI parsing not removing the first few bits of ANSI from certain messages.

logpipe will swallow all logs. We should document the use of tee to place logs into a file before logpipe gets it.

Something like

my-program |& tee logs.txt | logpipe

A lot of issues can be solved by simply using a much larger terminal space.

I like keeping it small for simplicity's sake, but I understand how this can introduce a big burden for some people.

We also may want to look into word-break CSS.

This is a cool project.
Just so you're aware, in the ancient times, sysadmins would use CCZE.
=> https://github.com/cornet/ccze

I've written this tool with as much vanilla HTML as I can.
It's a bit unusual, but I do this:

• To make the dependencies and outgoing network traffic transparent (safety)
• Lower download size
• Improve debugging
• Allow use of native web features without being gated by the framework (certain optimizations)
• Make the code easier to reason about

Making It Simple

That last bit is primarily for people who aren't web devs, because deviating from a framework actually makes the code a whole lot more difficult since you remove the "style guides" these frameworks make.

Luckily, there is probably a way to bring in component behavior without using a new framework.

Component Idea

If we make our "components" based on HTML pages, we can keep the idea of a "component" model still.

Take our command palette for example. Right now, it has its HTML inside of our index.html file. The JS for it is sizable so we put it in a separate command-palette.js file. But the CSS is together with the rest of the style.css data.

What if, we instead imported the command palette in our HTML as if it were another HTML file that we inlined.
We could have a directory

command-palette/
  index.html
  style.css
  index.js

The HTML makes reference to local files and by inlining the HTML, we get the entire related component tree.

The JS file can still communicate with the rest of the codebase by importing utility files where we might store state. It's a single use component, but we get the benefits of keeping it in one place.

1h2m3s
and (12.31s) are common for timing systems.
Let's add a new time type for these.

Also should fix 2.332 (decimals)

Broken off from #16
We've added the context menu in #19. Now it's time to add the "jump to log" action.

This is similar to Discord where you can search for messages, but then view the surrounding context right after. There's often important information around the message that you won't see when performing a regular search.

Centering the log output makes placing an optional "settings" menu difficult.
Cramming everything into a command palette has been surprisingly effective for previous projects, as brutish as it sounds.

Possible features:

• Set title
  • changes the page h1 and head title. Similar to setting the initial --title flag
• Remove ANSI escape parsing
  • Some programs are bad at syntax highlighting. Allow logpipe to use its own syntax highlighting.
• Change theme
  • I like darkmode, but astigmatism is a thing
• Stop parsing input
  • This one I'm less sure about, but we can let the user continue the output in the terminal and close the browser
• Save settings to disk
  • Let Node.js persist your settings between ports.
• Advanced filter
  • Fancier regex options
• Highlight text
  • don't filter, but highlight specific patterns.

The last 2 are related to direct filtering. I don't know that the command palette is the best place to do it, but it can be a convenient area if no other ideas crop up.

We can trivially get around problems by moving away from the current use of .innerHTML to DOM mutations. This also let's us remove that glaring warning in the readme.

I tried viewing some large logs (>1k lines), but after a few seconds of loading the website mostly stayed empty, and logged this:

Uncaught (in promise) InternalError: too much recursion
    replacePath http://localhost:37647/highlight.js:155
    highlightText http://localhost:37647/highlight.js:205
    getLogEl http://localhost:37647/index.js:42
    onmessage http://localhost:37647/index.js:86
    onmessage http://localhost:37647/index.js:86

It also shows a line count, albeit incorrect, but no lines:

(Firefox 124.0.1, NixOS, Node v21.7.1)

Sometimes it's helpful to copy/paste a log's text content.
Sometimes, you want to document some strange behavior and it's not as important for people to examine the text contents.

Screenshotting the view manually can be a bit messy. We can add a command palette option to automatically screenshot the current view.

I don't yet know exactly what this will look like, but it's an interesting idea. We could also embed the log's content into the png via a TXT section. I've done this work in earlier projects, so it shouldn't be too difficult.

It is impossible to query for content which contains quotes at the moment. We should fix this.

We can use subshells to pipe both stderr and stdout to logpipe.

If you want to separate these 2 output streams, this can be very helpful.

my_program > >(logpipe -t "stdout") 2> >(logpipe -t "stderr")

Also 2>&1 | can be shortened to |& in ZSH or newer versions of bash. I should add that as well to the docs.

my_program > >(logpipe -t "stdout" -p 8080) 2> >(logpipe -t "stderr" -p 8080)

We need to parse files better.

Right now we only parse files if it ends in :\d+, but we should also match

• :\(\d+,\d+\)
• Anything with a path and an extension
• Maybe also parse folders or change the name to "location"

Make a new view in the command palette that shows a "help" view with useful information from the readme.

It should include

• How to use bash redirection if no logs show up
• Intro to custom query language
• context menu intro
• tips on specifying a port

One disadvantage of using the web browser is that there's no shell history to refer back to.

It may be helpful to allow saving the current log state. Syntax highlighting is automatic, so we only need to save timestamps and raw log chunks.

This feature forces us to start thinking about storage solutions. This can't be done in the browser since we change ports (and therefore origins), making it impossible to share data between sessions.

Our parsing logic does not properly remove all ansi escapes - particularly the reset when at the end of a message.

I don't have a windows machine to test, but currently we're only capturing Mac's Cmd+K.

Windows browsers remap ctrl+k to some browser search functionality, but enough websites (github, angular docs) still use this mapping that it should be fine. We just need to e.preventDefault().

Some commands use ansi escape parsing, but just color the entire input one color.

This is not very useful. It might help to differentiate the inputs form the rest in the terminal, but it's actively unhelpful with our automatic syntax highlighting.

We can do this in two ways. We should perhaps add both.

• Turn off globally parsing ansi escapes. Remove them and pass them to other commands.
• Individually turn off ansi escape parsing for a specific message.

We're currently doing some unconventional things Shoelace is getting in the way of our movement.

We need to take over control so that ArrowUp and ArrowDown are not intercepted.

Unrelated to the main project goals.

It was expressed that, being a web tool, we're in a good position to provide an easy view into the tool's behavior without asking people to install a package directly from git.

We can make a dedicated branch that we regularly rebase off of master. We can then either

1. Edit the index.js file to take input from some function
2. Keep the current behavior, but additionally register a server worker that sends fake events.

The 2nd option is more interesting since it lets us almost completely preserve the main code (which makes rebasing simpler), but there are some issues with service workers taking a second to register on the initial load, making for a poorer user experience.

We'll likely move to a non-networked version unless it becomes important to communicate that the demo uses SSE at some point in the future.

This should link to the github, docs, and display the current version.

It's usually complete fine, but when using multi redirection such as in #44, we get a confusing output where it may be hard to tell which command started which port.

❯ my_program > >(logpipe) 2> >(logpipe)

Logs are displayed on http://localhost:55716

Logs are displayed on http://localhost:55717

We can fix this if the user supplies a title. Currently this won't resolve the problem, even if the user supplies a title.

As I keep adding features, it's become useful to check what version we're running.

I've had times where something was confusing me, but it turned out my installation was pulling from an old version.

A few things are missing from the log filtering that I think would be a pretty big QoL feature

• copy log timestamp
  • currently viewable, but hard to copy if you want to compare something
• copy log ID
  • could be useful for future features
• scroll to log
  • see the context around a log item

Like Warp's feature, it could be nice to get a ping when the command finishes executing.

We have a pretty bad perf situation with massive log inputs.

Our biggest obstacle is rendering. Querying itself seems performant, even on massive datasets. But rendering is where we choke.

Possible Solution

• Virtual Lists
• Bring query data to top of div. Expose it as attributes so we can avoid query selector.

Silly me, just realized that all of our docs saying to use logpipe --port <number> were wrong.

Right now node swallows these args, so we need to do logpipe -- --port <number> which looks pretty annoying.

There should be a way to get around this. One way is to capture the arguments before passing it to node through bash.

Currently we do text-based filtering for tags.

This is done so we can re-use the text filter box while still supporting a helpful feature.
However, text-based filtering has some disadvantages. Here we see that Info is selected, but both [INFO] and Info are filtered:

To fix this, we need to do 2 things:

• Have tag filter buttons specifically match class names, such as "tag" instead of raw text
• Propagate this to the UI somehow. Perhaps, a custom type=tag, but it needs to be something people wouldn't normally type.

We currently do text-based filtering exclusively. This is limiting and we have the power to do something more interesting.
This stems from #25 where we described the need for a query language.

For something really basic, we can support

type=<classname>

in text boxes which run a queryselector on each log item for matching node types. This way we can re-use our syntax highlighter for the query system.

An initial issue I had with this approach is that it may limit us if someone has a log containing "type=" and wants to search for this content. We can initially support instead lp-type= which is unlikely to clash with anything for now.

We can also fix the issue with people copy/pasting logs directly into the search box by creating a "search for logs like this" option to the context menu. Also a really cool idea that would be based off of the work in this feature.

This is gonna be really cool!

The extended tag container is still having trouble with some massive tags.

Another approach is capping the length of tags in the display. Maybe doing [tag name...] if it's too long, but keeping the full tag in some data-key.

As I mentioned on the reddit thread about it make the video stand out more in the README. I didn't know to look for it and only your comment on reddit made me look.

Allow the user to change their current theme using the command palette.

This also makes for a good follow up on allowing the user to save data using Node.js (so it sticks between hosts). Baby steps.