brimdata / zui

Zui is a powerful desktop application for exploring and working with data. The official front-end to the Zed lake.

Zui

The Official Front-End to the Zed Lake

Zui is a desktop app for exploring and working with data.

Highlights:

• Drag-and-drop data ingest
• Automatic detection of common data formats
• Schema inference during ingest
• Beautiful result views for nested or tabular data
• Named queries with version history
• Query session history to keep track of your work
• Pinnable query fragments to keep your search box uncluttered
• Right-click menus for pivoting and filtering

Ready To Try It Out?

Download Zui for your operating system.

Refer to the installation guide and release notes for more information.

Powered By Zed

Zed offers an innovative approach to working with data known as "Super-Structured Data".

Behind Zui is a local Zed Lake instance where you can load your data into pools and use the powerful Zed language to search, analyze, and transform it. Use it to:

• Explore deeply nested JSON objects
• View Parquet and Arrow IPC stream files
• Clean up CSV files by adding type information
• Search heterogeneous NDJSON logs
• Transform data from a legacy database's CDC logs
• Investigate Zeek security logs

Zed provides a system to make working with data easier and more efficient. The storage layer, type system, query language, and zq command-line utility are just a few of the tools Zed offers to the data community.

Formerly Known as "Brim"

For many years, the app was known as Brim, named after the company Brim Data that created it. In 2023, it was renamed to Zui (a play on "Zed User Interface") to better reflect its connection to the Zed technology that powers it.

Zui retains the security-specific features that made Brim popular while expanding its reach to anyone who has data to work with. For example, you'll still find the customized views, histograms, and correlations relevant to the security domain appearing in Zui via its nascent plugin system. In the future, developers will be able to create custom plugins that make Zui even more effective for their specific needs.

Related Packages

This Zui code repository is actually a monorepo (managed with nx) that includes several packages on which the app depends. They may also be used as standalone tools. These include:

• zed-js: the JavaScript library for browsers
• zed-node: the JavaScript library for Node.js
• zed-wasm: the Zed command-line tools in the browser
• zui-player: the end-to-end testing framework for Zui

Need Help?

Please browse the support resources to review common problems and helpful tips before opening an issue.

Contributing

We welcome your contributions! Please refer to our contributing guide for information on how to get involved in development.

Join the Community

Join our public Slack workspace to stay up to date on announcements, ask questions, and exchange tips with other users.

zui's Issues

master starts to error screen with existing space

If I try to run Brim off of current master (16e606a) when I've previously created a space with the Brim 0.6.0 release, the application loads directly to an error screen, with this error:

TypeError: Invalid attempt to spread non-iterable instance
    at _nonIterableSpread (/Users/alfred/work/brim/dist/js/state/LogDetails/reducer.js:15:39)
    at _toConsumableArray (/Users/alfred/work/brim/dist/js/state/LogDetails/reducer.js:13:95)
    at toHistory (/Users/alfred/work/brim/dist/js/state/LogDetails/reducer.js:83:43)
    at /Users/alfred/work/brim/dist/js/state/LogDetails/selectors.js:32:33
    at /Users/alfred/work/brim/node_modules/reselect/lib/index.js:76:25
    at /Users/alfred/work/brim/node_modules/reselect/lib/index.js:36:25
    at /Users/alfred/work/brim/node_modules/reselect/lib/index.js:90:33
    at Object.getHistory (/Users/alfred/work/brim/node_modules/reselect/lib/index.js:36:25)
    at Function.stateToProps [as mapToProps] (/Users/alfred/work/brim/dist/js/components/RightPane.js:172:40)
    at mapToPropsProxy (/Users/alfred/work/brim/node_modules/react-redux/lib/connect/wrapMapToProps.js:53:92)

repro on windows & mac:

• run 0.6.0 release, ingest a pcap, exit brim
• run master (currently at 16e606a)

Ingest Zeek Logs into Brim

When given a array of files and their associated types, we need to validate that the array contains:

• One pcap
• One zeek log
• Multiple zeek logs

Any other combination will throw an error explaining what we expect.

If it passes validation, return the default dataDir and spaceName params.

If one file:
spaceName: same name as that file
dataDir: same directory as that file/spacename.brim

If more than one file:
spaceName:
if all logs in same dir, the name of that dir
else a randomly generated name
dataDir: $HOME/.brim/spaceName.brim

Profiling feature

Can you please evaluate if makes sense to add a "Profiling" feature? Wireshark has this feature - https://www.wireshark.org/docs/wsug_html_chunked/ChCustConfigProfilesSection.html.

This feature is really useful when you want to analyze different type of captures. For Wi-Fi(802.11), you have the Metageek profile - https://support.metageek.com/hc/en-us/articles/115013527388-Wireshark-Configuration-Profile

Failure when clicking "License" link on Windows

When verifying the fix in #521, I happened to try clicking all the links in the new "About Brim" box. When I click the one for "License", it failed to come up.

I have a good guess what the problem is. The "Acknowledgments" link points to a file called "acknowledgments.txt", and Windows defines Notepad as a default handler for the .txt extension. Meanwhile, as the screenshot shows, the "LICENSE" file has no extension. Whereas Windows is extension-obsessed, MacOS seems more forgiving and capable of popping up the extension-free text file in its viewer.

Since a filename called LICENSE is certainly standard in GitHub, perhaps we just have to copy it to a different .txt name when we build the artifact?

UX: Open button to read zeek logs

Provide button to read/ingest Zeek logs.

1. We will not ask the user to specify the type of logs when opening log files. We discussed we would use auto-detect logic from zq. Indicate the list of supported log formats.
2. Call out ZNG format as one of the supported ones (no need to use the red color as in the wireframe, but putting it as first in list helps highlighting the best format, ZNG ).
3. Ideally design a folder icon with zeek logo as a visual hint of the most common option.

Wireframe Open button: Showing Open button. The usage of Fin icon and Zeek folder icon will serve as a visual clue for the Wireshark persona and the Zeek persona.

Defaults: User can chose to accept our defaults for naming convention and .brim location. Our defaults today save the Brim folder in the same location of the selected file. If users un-select the defaults checkbox, they can choose:

1. name: the Name of the brim folder, which will still have .brim extension
2. location: the location for the saving of the brim folder.

Drag operations

1. PCAP: we will support dragging a Single PCAP file only. Supporting more is future functionality. Today users can use the mergecap command available from Wireshark installation.
2. LOGS: we will support dragging a single log file, dragging a Folder that contains log files or multiple file selections. In order to process logs we assume they will be in one of our supported formats, using our auto-detection logic.

Refactor viewer component

Our viewer can be optimized for faster mounting on the tab pages and this library seems like it might facilitate that. Let's refactor the viewer to use that lib.

Additionally, we can connect that new viewer with the up and down arrow shortcuts for selecting logs and ensure that the viewer is always focusing on an area with the selected log in sight when one of those keys has been pressed (i.e. scroll down/up along with the selector).

• Scroll position
• Scroll to item
• Scroll to index
• Render a message at the end (end of results)
• Render an empty message (no results)
• Sync the selection with the detail pane
• Show headers / hide headers
• Have a message if the headers are hidden

In a recent team discussion, @jameskerr noted that this refactor could help us address a set of issues with the Table view, e.g., slowness or problems rendering complex things. I'm linking those issues as being "Blocked" by this one so they're easy to find.

We also acknowledged that we're not sure how long this refactor might take. It could be weeks or it could be days. We'd ideally like to address the related bugs before we release Zui v1.0, so we've committed to spending at least some time in the short term to size up the effort, and if it's too big we'll consider rolling it over to Zui v1.1.

Unable to load pcapng on Windows

I just installed Brim on Windows 10 x64 and got the following error after loading a pcapng file.

Unable to generate full summary logs from PCAP 

Detail: zeek exited with status 1: warning in <command line>, line 7: problem initializing NB-DNS: no valid nameservers in resolver config fatal error in <command line>, line 7: problem with trace file - (bad dump file format) exit status 1

Opening Zeek logs in Brim app

In Brim v0, ingesting pcaps was the primary use case, which left non-pcap-having Zeek users out in the cold a bit. An advanced user could surely use zq to manually turn their Zeek logs into an all.bzng and put it in the right location with the other trimmings to access it as a Space in the app. However, to give these users the experience they expect, we'll provide the same drag-&-drop kind of UX experience for Zeek logs that we do today for pcaps.

Modal to display ingest errors

When the user clicks on the errors counter, we'll display a modal with a scrolling list of all the errors.

Form for dataDir & space

Add a checkbox next to the file input that says use defaults.

If that is not checked, when the user uploads files, after validation, present a modal form with inputs for the space name, and the dataDir (brim folder).

Pass this data on to the function that will send the post request.

Design UI for the errors counter

This will be displayed to the right of the progress bar.

Add Formula to Homebrew

This will allow for easier installs and management of the software on macOS.

Homebrew: https://github.com/Homebrew/brew/

Show "Whole Space" by default when sensible, or a reasonable maximum otherwise

Background: In Brim v0 where the user workflow always starts with a pcap, we learn the start/end times of the pcap as we're indexing it and use them to set the initial time span of the Space while the pcap is still ingesting. This way the histogram time range is consistently at the whole time range throughout ingest and fills in with more bars as the screen is refreshed.

In v1 when we'll let users bring in Zeek logs without pcaps, we won't have that guidance from the pcap anymore. Also, there's concern that always setting the time span to cover 100% of the ingested data might make queries frustratingly slow, so it might make sense to select a default time range that covers a substantial amount of data, but not necessarily everything.

@mccanne had a proposal:

The app could fetch timespan of space GET then bucket time into say 1000 buckets (i.e., delta = (maxTs - minTs) / 1000)), then run a * | every $delta count(). Now we know the density of events and we can choose the largest default time window (rounded to something sensible 1 min, 15 min, 1 hour, 1 day, etc) such that the default window is the largest window containing not more than say 1 million events. This would be especially useful UX once we have a time index on the bzng file things only get slow when you zoom out to the whole space as opposed to starting slow from the start.

Update design of open file area

Add the zeek Icon
Add the zeek to the accepted formats

Revive the histogram deep inspection test

@mikesbrown says:

Revive the histogram deep inspection test. The logic is probably fine to remain, but the expected values (e.g., number of distinct _paths) will be different due to data set differences.

Consolidate Zealot and Boom clients

@mason-fish says:

Once we have client tests written, let's consolidate the boom and zealot clients since they are talking to the same service.

Better "zed serve" process management

A few issues regarding zed serve process management:

1. Brim ignores unexpected zed serve shutdowns. Would be nice for zed serve to have some kind of automatic restart with growing retry intervals (similar to the behavior of kubernetes pods). If Brim is unable to restart zed serve, the client should be alerted with some kind of diagnostic information to provide support (maybe a link to generate a github issue).

2. If Brim experiences an ungraceful shutdown (e.g. kill -9 brim) zed serve does not also receive the signal and shutdown. If we take care of the above problem, this becomes an issue because on subsequent restarts of brim, zed serve would be unable to bind to :9867 because the port would be in use by the orphaned zed serve process.

3. We also need to consider the case of a host having another service listening on :9867. Perhaps this is another version of zed serve that a user is purposefully running- the application should notify the user of this case and confirm if they want to continue running against an unmanaged zed serve process. If it is an unknown service the application should prompt the user to terminate the service bound to this port.

repeated key navigation for log detail locks up UI

If I open up a log details pane, then hold down the 'down' or 'up' keys, the UI locks up, and eventually all of the records briefly to display themselves at once in the pane.
I saw this on macos and windows, using the latest master.

Populate request ID header

Now in zqd master, the http endpoint will receive a string value in the http header X-Request-ID and use this value for logs generated from this request. This functionality will be helpful for end-to-end debugging of errors.

Ideally the client should start saving its own logs, but this out of scope for this ticket. For now the client should use something like the node-uuid to populate X-Request-ID header value.

zqd will return the X-Request-ID in the response header.

In the meantime, as long as the Brim app is not populating this field, zqd is generating a unique one for each response. See https://github.com/brimsec/zq/blob/a1ffe45a27a1366cddfa75e2c0f1581346f0d1f0/zqd/middleware.go#L24-L26.

Cleaner right-click tests and dealing with native menus

Native right-click menus present a problem for Spectron-based testing because the native menus aren't in the DOM. In brief, an out-of-band event listener receives intermediate representations of records, and desired actions.

It won't be easy to duplicate the exact solution we'd used in the past. Whereas before the "unit-of-test data" was static Zeek TSV, now it's PCAP. In the latter case, each test run will produce different UIDs. This means it won't be possible to have hard-coded intermediate representations of records, which worked well enough before. The UID will change each time.

(Ref: PROD-1489)

Write jest client tests

@mason-fish says:

per @jameskerr's research, using https://jestjs.io/docs/en/puppeteer would give jest access to the fetch api's that the app uses, and would enable us to write client testing for both boom and zealot.

Ideally we would only have one client lib, so this is a good preliminary step to take before we refactor/consolidate the two clients.

Revive pcap tests

@mikesbrown says:

This was a side feature before that now is more important. There were 3 tests, and I think we can bring them back. The tests will take some time to work on to find proper flows with different test data. Note that this is to test integration points, not exhaustively test pcap handling.

UX: errors during log ingest

Provide a visual indication to the the user that the space he is looking at has ingest errors. ( the user may be looking at partial data).

1. Show warning in the space area indicating there were errors during ingest. User can "clean/erase" this message.
2. Count the number of errors and show during ingest.
3. If user clicks on errors warning message, he will get details about the errors.

We will use the auto-detect logic of zq to figure out the format of zeek logs. We will try to ingest all we can, count errors and warn user that the space they are looking at does not have all the data. Providing details of the errors to the user will help him correct the problem or try to troubleshoot.

Wireframe Error Warning: showing warning to the user to indicate that the space he is looking at does not include all the data and contains ingest errors.

If user hovers over the Warning icon, he can see details and also remove the warning message.

When ingesting, progress bar will show movement. If errors are found, show a warning with number of errors. If user clicks on this warning message, provide error Details.

This feature depends on backend improvements.

Reassess end-to-end test strategy

@mikesbrown set the stage for this project with the description below. Implementation issues will ultimately be linked to this Epic.

---

The problem addressed by #561 is an example of an annoying feature of webdriver-based testing. Different environments yield different test bugs (not software functionality bugs). After a year, now is the time to reassess end to end testing.

Requirements:

• MUST work on Windows, Mac.
• MUST be able to test something written for Electron
• MUST be able to test an installed Electron app
• MUST have a mechanism for file upload

Nice to haves:

• Abstracts away waiting for element interaction (Cypress yes; TestCafe yes but less so; Spectron/Webdriver no)

There are claims Spectron can do all of the above, but with Spectron you get problems like the one addressed by #561.

There are some other options out there that fix some of these. Cypress and TestCafe are examples of end-to-end test frameworks that abstract away some of the element waiting/retrying we have to do when using Spectron/Webdriver. Cypress in particular looks like a pleasure to work with. https://docs.cypress.io/guides/getting-started/writing-your-first-test.html

The problem is that Cypress Electron support is in alpha, and the development for Electron support, while surging last August/September, seems to have stalled. Choosing Cypress is a bit risky then. TestCafe doesn't look as compelling as Cypress, but site docs imply better Electron support. Nevertheless on the side, I'm going to see what it's like trying to get our app to run in these.

Note there isn't much else for Electron apps that I've seen based on searches at https://discuss.atom.io/c/electron and https://www.reddit.com/r/electronjs/ . I will look at a few Electron app repos to see what else they might be using.

on-close handler prevents Spectron's app.stop() from functioning on Windows

@mikesbrown explains:

---

PR #499 prevents Spectron's app.stop() from working in Windows. This is a regression in the sense that reliable methods of driving the app in automated testing are broken for the windows platform. Being able to use app.stop() is important because it clears out chromedriver and electron and removes that responsibility from a test maintainer to do that themselves.

You can reproduce the problem by doing the following:

1. Getting a windows machine and installing Node 12 and GIt
2. Cloning Brim
3. Running:

$ npm install
$ npm run build
$ npm run itest -- -t tab itest/test/smoke

You may have to run the last command (that includes itest) a subsequent time to produce the problem.

Before #499 at hash e0b01933e8bb564eca680f3d245870cdc28be3ac, you could observe the test pass and the app stop at the end of the test (including all windows disappearing). At the next hash 990eb2f0160d65748eeee1f6abcc8cc16e9edbc0 you can observe the app still up after the test passes.

Not only does Brim fail to exit using Spectron's stop(), but also Brim fails to close using the browser-window API close(). Tests using spectron can issue something like:

app.browserWindow.close()

Brim responds to this call on Mac; in Windows the window remains open. On both platforms, messages like the following are logged:

'[ec4b8f1316]:  Confirming Unload...',
      '[ec4b8f1316]:  Unloading',
      '[ec4b8f1316]:  Saving window state'

I can the problem go away by applying this patch:

diff --git a/src/js/electron/tron/window.js b/src/js/electron/tron/window.js
index 3dea96e7..732043cd 100644
--- a/src/js/electron/tron/window.js
+++ b/src/js/electron/tron/window.js
@@ -32,10 +32,6 @@ function searchWindow(params) {
       nodeIntegration: true,
       experimentalFeatures: true
     }
-  }).on("close", (e) => {
-    // Close handled by the search renderer
-    e.preventDefault()
-    e.sender.webContents.send("close")
   })

   if (size) {

The preventDefault is probably interfering in the Spectron-on-Windows case. Why this happens with Spectron-on-Windows only isn't clear.

Windows support: "GA" quality

Brim v0 claimed only beta-quality support for Microsoft Windows, due to several limitations we couldn't overcome in the allotted time. Issues linked to this Epic describe the additional work necessary to bring it up to a "GA" level of quality for which it would be appropriate to remove the "beta" label.

UX: progress bar to show processing logs

Update the message for the progress bar to reflect that we are ingesting zeek logs.

Reusing same progress bar as for packets but changing the message to "Processing Logs"

Path not being set properly for zqd to find zeek on Windows in dev/ci env

In a Windows development environment where Brim is checked out and built, Brim cannot ingest a pcap because zeek can't be found.

This is the reason for CI in Windows failing. Why zeek can't be found is unclear. There's nothing in zqd logs that indicate this one way or another. I confirmed when I run npm run electron that the running zqd is brim.git\zdeps\zqd.exe.

Brim derives the path of zqd here, and zeek just after.

From https://github.com/brimsec/brim/blob/3b912ea0f689e1a90cc9684cf7c5c858e0130c63/src/js/zqd/zqd.js#L13 :

const zqdPath = join(app.getAppPath(), "zdeps")
const zqdZeekPath = join(zqdPath, "zeek")

The rest of the code in zqd.js looks reasonable. We've already established zqdPath is working, so it's odd zqdZeekPath isn't.

To reproduce:

1. Get a windows machine
2. Download Git and Node 12
3. Clone brim.git; cd brim
4. npm install && npm run build && npm run electron
5. Attempt to ingest a pcap and note error

Revive query integration tests

Revive the old query integration tests. The queries had a result that returned logs; analytics (scalar result); analytics by ts (every); analytics by a key; analytics by ts, key (every group by). They aren't exhaustive zql language tests by any means.

New window for Log Details pane

Per @siskojr:

---

Proposal: Decouple the main Log (events) pane from the Log Details pane. The Log Details pane would become a new window.

When a user double-clicks an event for the first time, a new window with the Log Details will appear.

When a user clicks an event in the main Log pane, we update the Log Details window.

When a user moves to a new event in the main Log pane with arrow keys, we also update the Log Details window.

The user might end up using various simultaneous Log Details windows. On double-click, provide the option to open a new Log Details window (tied to current Logs window) or re-use the current active one. This is:

1. Mode with a single Log Details window open. All changes in the Logs window update the same Log Details Window.
2. Dual Log Details Window (A use case with two active Log Details windows using two monitors is the most likely one. In this case user has to decide to what Side to “project” the log.)
3. N- Log Details windows. This case may be less common but users will associate windows of logs (would contain multiple tabs) to the same window of Log Details.

Mode 1,2 are required. Mode 3 is desired.

Possibility to support real time ("live") captures

One of the best feature that wireshark has is the fact that can capture traces in real time.
Can you please add similar feature to brim?

Improve closing of Log Details pane

Proposal: Make it easy to click the control to close the Log Details pane easy.

A current problem is that when the user wants to use the mouse to hide the Log Details pane (as opposed to hotkey), the tooltips sometimes cover the button.

Here's an example of when the button is easy to find:

Here's an example where it's obscured:

Ideas discussed to resolve this have included ensuring the tooltips are displayed in a way so that they never cover the button or, per @jameskerr's idea, to reposition the close button.

Consistent skinny bars in histogram view

The width of the bars in the histogram can vary widely, somewhat randomly, depending on the duration of the search query.

Sometimes you get nice skinny bars:

Other times you get thick bars:

I would argue that the skinny bars provides for a better user experience. Skinny bars make it is easier to find anomalous events.

Revive the "helpers" integration tests

@mikesbrown says:

These were "View AST" etc. Some of the tests may need to be modified slightly to deal with different data set availability.

Clicking History entry for deleted Space causes loss of context for current Space

When reviewing post-merge state of the functionality in #547, @alfred-landrum noted:

The rationale for keeping the history entries was to allow the user store them & run them again in live space. I don’t think we should try launch the search on the non-existent space.

Indeed, here's an example of an undesirable side-effect of the current behavior that feels like a bug. When I click on a History entry associated with a deleted Space, in addition to the error messages (which seem fine) it resets the app's context away from the Space I was looking at, which I found quite jarring as a user.

Video: repro.zip

npm cache problems in ci

Ref: https://github.com/brimsec/brim/actions/runs/74573741

1. Windows cache isn't being found
2. MacOS cache is ~1154 MB. Can this be shrunk? See also actions/cache#187 . For comparison the Linux and Windows caches are about 78M.

Linux release

Please release a Linux version.

Flatpack packaging is probably ideal for you, to quickly reach all users regardless of the distro.

Store ingest errors in redux

Right now we store the ingest data in the space object. We could add an errors array there.

Or we can make a new reducer call the 'ingests' reducer. This could then store the progress, and any errors it encountered. We'd have to index it by cluster and space just like we're doing now with the 'spaces' reducer.

I might like the second idea more.

enable auto-update for windows

Now that we have code-signing support for our windows builds, we should be able to create the zip needed for the auto-update code.

EDIT: Our reading of the docs says that the exe file we produce should be enough for windows auto-update to work, but it doesn't appear to do so on the recent v0.7.0 release.

Wi-Fi - 802.11 captures support

Can it be possible to support Wi-Fi (802.11) captures.
I have tested Wi-Fi captures and it seems that is not supported.

Negative integration test for bad pcaps

Whereas #452 was about a positive integration test for pcaps, this is about writing a negative integration test. The goal should be to send some sort of bad file to Brim and make sure the backend error handling is made clear in the app.

Ensure Brim user can find ZQL docs

A user made the suggestion on Slack:

oh yeah, you guys should have a link to the query syntax somewhere in the brim app
i wasn't sure what i could type into the search box

I'd been thinking users would have come to the app after having sat through the demo video and heard about the dependency on zq/ZQL, the pointers to the repos with the language docs, etc. But the user's experience is validation that not all will be that thorough/patient (duh!)

Can we add some kind of prominent link near the search box to https://github.com/brimsec/zq/tree/master/zql/docs?

Adding also the link to query language as a new list item in the Help area is useful.

zqd binary not present

Hello,

sudo npm install:

(node:15643) UnhandledPromiseRejectionWarning: Error: zqd binary not present at /home/xxxx/install/brim/zdeps/zqd

[email protected] build:css /home/alfon/install/brim
node-sass-chokidar src/css --output dist/css "--watch" "--skip-initial"

(node:16754) UnhandledPromiseRejectionWarning: Error: zqd binary not present at /home/alfon/install/brim/zdeps/zqd
at zqdCommand (/home/alfon/install/brim/dist/js/zqd/zqd.js:86:11)
at ZQD.start (/home/alfon/install/brim/dist/js/zqd/zqd.js:123:44)
at zqdMainHandler (/home/alfon/install/brim/dist/js/electron/ipc/zqd/mainHandler.js:12:9)
at _callee$ (/home/alfon/install/brim/dist/js/electron/main.js:69:42)
at tryCatch (/home/alfon/install/brim/node_modules/regenerator-runtime/runtime.js:45:40)
at Generator.invoke [as _invoke] (/home/alfon/install/brim/node_modules/regenerator-runtime/runtime.js:271:22)
at Generator.prototype. [as next] (/home/alfon/install/brim/node_modules/regenerator-runtime/runtime.js:97:21)
at asyncGeneratorStep (/home/alfon/install/brim/dist/js/electron/main.js:33:103)
at _next (/home/alfon/install/brim/dist/js/electron/main.js:35:194)
at /home/alfon/install/brim/dist/js/electron/main.js:35:364
(node:16754) UnhandledPromiseRejectionWarning: Error: zqd binary not present at /home/alfon/install/brim/zdeps/zqd
at zqdCommand (/home/alfon/install/brim/dist/js/zqd/zqd.js:86:11)
at ZQD.start (/home/alfon/install/brim/dist/js/zqd/zqd.js:123:44)
at zqdMainHandler (/home/alfon/install/brim/dist/js/electron/ipc/zqd/mainHandler.js:12:9)
at _callee$ (/home/alfon/install/brim/dist/js/electron/main.js:69:42)
at tryCatch (/home/alfon/install/brim/node_modules/regenerator-runtime/runtime.js:45:40)
at Generator.invoke [as _invoke] (/home/alfon/install/brim/node_modules/regenerator-runtime/runtime.js:271:22)
at Generator.prototype. [as next] (/home/alfon/install/brim/node_modules/regenerator-runtime/runtime.js:97:21)
at asyncGeneratorStep (/home/alfon/install/brim/dist/js/electron/main.js:33:103)
at _next (/home/alfon/install/brim/dist/js/electron/main.js:35:194)
at /home/alfon/install/brim/dist/js/electron/main.js:35:364
(node:16754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:16754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:16754) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:16754) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Best regards,

Handle line breaks & super wide lines in error messages

Found in Brim commit 3b912ea talking to zqd tagged v0.7.0.

While verifying an issue, I saw that some of this helpful error message ran off the side of the screen.

Based on how it looks, there almost certainly were line breaks in the original error message. It'd be ideal if we could include those so the user can more easily cut & paste the whole thing, such as for sharing with us in a support situation.

@mikesbrown also discovered a similar class of issue: Super long lines that may not even have line breaks in them. Perhaps we'd want a horizontal scrollbar, or perhaps we'd want to wrap them. I know our CI automation takes screenshots during failures and would probably like to be able to see as much text as possible, so perhaps lean toward the latter?

Detect File Type

We need a function to detect if a file is a pcap or a zeek log. I'm not sure how we're going to do this. I was thinking we could check the encoding and see if its binary or utf-8.

Instantly jump to top/bottom of rows in main logs panel

Today in Brim, a user attempting to scroll to the bottom of the displayed events experiences in the "infinite scroll" behavior where additional events are added to the bottom and the scroll controller moves "up". This may be great if the user is looking to see a few more events, or might be frustrating if they want to quickly jump to the oldest event in the Space.

Further in the spirit of the arrow keys movement that was introduced in #550, we've discussed that it would make sense to offer some kind of control (e.g. a hotkey) that would allow the user to instantly jump to the top/bottom in this manner. This seems similar in nature to other familiar tools like Google Sheets where the user can jump to the first/last row in a spreadsheet via Cmd-Up/Cmd-Down on MacOS or Ctrl-Up/Ctrl-Down on Windows.

Agree on contract for "reset state" for tests

@mikesbrown says:

---

Tests need to be able to set themselves up. The "Reset State" mechanism supported by Brim is a pretty convenient way to get to a "good-enough" initial state for tests.

"Reset State" has shown to be buggy in the past (see PROD-515 and PROD-539 for examples).

For both these reasons, integration tests made extensive use of "Reset State": both as a setup step, and also in a specific test in which the app was "dirtied" with state and minimally verified to be reset.

When @alfred-landrum was working on #452 , he reported that the mechanism we use for resetting state in integration tests was causing his test to fail, and he commented it out. While I could not reproduce it locally, indeed, https://www.electronjs.org/docs/api/web-contents#contentssendchannel-args confirms this is asynchronous.

While reviewing how "reset state" works, I was also reminded that the integration test's "reset state" is not a "full reset state".

The integration tests send "resetState" to the renderer process through the webcontents that spectron makes available. The code for handling a resetState sent to the render process is here:

https://github.com/brimsec/brim/blob/d468eeeda55187a108fea51c0aab5a8e689e7952/src/js/initializers/initShortcuts.js#L36

ipcRenderer.on("resetState", () => {
    clearState()
    location.reload()
  })

But the menu-driven reset state, which we don't have access to in integration tests, sends "resetState" to the renderer AND removes a file. That code is here:

https://github.com/brimsec/brim/blob/d468eeeda55187a108fea51c0aab5a8e689e7952/src/js/electron/menu/searchAppMenu.js#L96

{
          label: "Reset State",
          click: () => {
            send("resetState")
            lib
              .file(config.windowStateFile())
              .remove()
              .catch(() => {})
          }
        }

I'm mentioning all this stuff because the asynchronous nature of webcontents.send("resetState") needs to be dealt with, but at the same time, we should come up with a contract for how integration tests will reset state.

1. Ideally the test could do something like webcontents.sendSync(), but I can't find its equivalent at https://www.electronjs.org/docs/api/web-contents . Absent that, I'd be happy for ideas here. A strawman absent of something better is to have something in the DOM that reports a timestamp in unix seconds of when state was last "set". The test can get that element, issue resetState, and wait for that element's value to increase.

2. Ideally the test would simulate the "full reset state" that includes the deletion of config.windowStateFile() through the app. How this is done sort of depends on 1. For example, if it's possible to move the deletion of the state file to the renderer process, then this problem goes away. If it can't, should we punt on that and make the test responsible for deleting the state file?

These almost certainly require a change to Brim. cc: @jameskerr and @mason-fish. And while this doesn't seem like a feature need for brim-v0, it's important in order to facilitate stable integration testing.

UPDATE:

Messing with #452 locally has caused me to go down a state persistence rabbit hole. Here's an update. I was testing it locally and noticed on subsequent runs, we were always being presented with the "Open PCAPs" screen. This is odd because the old behavior of integration tests with boomd would present you with "wherever you had been before". The state I expected to see upon running an integration test that ingests PCAPs is that to run it a second time, the same PCAP file was already ingested, unless you "reset state", which I had pointed out @alfred-landrum had already disabled. Instead, state appeared to be reset even without a "reset state" call by the test. This was bugging the hell out of me.

When I run the app using Spectron, Electron’s app.getPath(“userData”) is not the OS X ~/Library/Application Support/Brim. Instead, for each test, it’s a temp directory, such as /var/folders/x9/b_56b0s947b58yxjgdktg1th0000gn/T/.org.chromium.Chromium.A9chgr. This must have to do with Spectron and Chromedriver. I'm able to work around this passing a user-data-dir value to Spectron's chromeDriverArgs. I have not yet been able to find why this differs. The docs I've found so far claim this is supposed to be a path resembling ~/Library/Application Support/Brim.

This has important implications for tests. One is that tests by default won't be using the same userData path as the actual app. If we want to do so, I believe we'll have to do it ourselves, possibly reimplementing logic. This is a bit unfortunate: it's workable but the implications should be understood. It also makes the priority of a synchronous "reset state" lower. As long as we are OK paying the price to reingest PCAPs on any test that needs them, we can do nothing: state will be reset on every test anyway.

Brim should use PacketPostStatus snapshot counter

Brim should use the PacketPostStatus count implemented in zqd in brimdata/zed#454
to optimize when to issue new searches during ingest.

"Copy for zq" provides an invalid command line

Let's say a user has imported a pcap hello.pcapng into Brim (GA verison tagged v0.12.0) and then entered ZQL count() by uid. When they select the "Copy for zq" option, they're presented with the following for possible cut & paste:

zq -f table "count() by uid" hello.pcapng.brim/all.zng

This has three problems:

1. It assumes that zq is installed and in the user's $PATH, which may not be the case.
2. It references the Space via the name that appears in the app ("hello.pcapng.brim") as opposed to the directory name under which the all.zng exists. For example, here's what it looks like on my Mac where I've just reproduced the issue:

$ tree -s "$HOME/Library/Application Support/Brim/data/spaces"
/Users/phil/Library/Application\ Support/Brim/data/spaces
└── [        192]  sp_1e154F3difAvnV9J8lCpui6zgyR
    ├── [       3510]  all.zng
    ├── [        235]  config.json
    ├── [         92]  info.json
    └── [       1802]  packets.idx.json

1 directory, 4 files

3. The path is relative, so even if the app had used the correct directory name, a pasted command line into a freshly-opened command window would not work.

Since zq is now bundled with the app (in resources/app/zdeps/zq), it seems we should be able to address this by making sure to generate an absolute pathname that uses the proper Space directory name. Care should be taken to ensure we've done any necessary quoting/escaping is necessary to deal with typical gotchas: Directory names that have space chars in them, forward/backslash differences on MacOS vs. Windows, etc.

Close/reopen laptop lid - Brim shows Network error

This can be considered low priority, as it does not always happen. It's hard to reproduce and may be related to the size of ingested pcaps. Here's a screenshot of one time the problem reproduced:

More detail from the user who originally reported it:

When this happened, the ingest process stopped and I had to open file again and start over. I found the same problem when the screen saving/automatic lock of account happened.