Running the cache collection in a WebWorker/WorkerThread about autocomplete-paths

[aminya]

One of the hot parts of the code is this function call:

autocomplete-paths/lib/paths-cache.js

Line 154 in b6d3b09

const entries = await this._getDirectoryEntries(directory)

It uses some Atom API (directory.getEntries) to get the Files or Directories under a certain directory.

[lierdakil]

WebWorkers in Atom can't access Node API. Not entirely sure why it's that way, but perhaps for a good reason, e.g.

The only way to load a native module safely for now, is to make sure the app loads no native modules after the Web Workers get started.
(source)

(this condition is pretty much impossible to guarantee in Atom as far as I can tell)

So the only way to make caching non-blocking that I currently can think of is splitting it out into a separate worker process. But that creates a potential bottleneck in JSON deserialization if the number of files is large (on the order of tens of thousands), since processes in Node can't share memory. This can be worked around perhaps by offloading filtering onto the worker process as well, but that creates some latency concerns.

That is to say, this is a bit complicated.

[aminya]

WebWorkers in Atom can't access Node API. Not entirely sure why it's that way, but perhaps for a good reason, e.g.

Using Node API is enabled in this PR:
atom/atom#21139

The community version of Atom already supports Node API in the web workers. For the official Atom, it is just a switch that Atom maintainers need to enable. @sadick254 @darangi

Currently, the performance problem is that we are using the getEntries function from Atom API, which is async, but still blocking! If we could move that to another thread, then the problem is solved. We should look into this Atom API and see if there is room for improvement.

Another possibility is to use some node package instead of Atom API.

autocomplete-paths/lib/paths-cache.js

Line 209 in b6d3b09

directory.getEntries((err, entries) => {

The only way to load a native module safely for now, is to make sure the app loads no native modules after the Web Workers get started.

This just means that we should not use a new native module in the webworker. I don't think this is a big limitation. That means that we should use JavaScript on the worker. If we want to use a native module for this, there is no need for a separate worker because the native module can start its own threads.

[github-actions]

🎉 This issue has been resolved in version 2.13.0 🎉

The release is available on:

• GitHub release
• Atom package

Your semantic-release bot 📦🚀