Research Next.js hot reload performance about clutch

[tmota900]

When change is done (i.e add a property to a component) there is a doOperation request that is done to API to manipulate the metadata, the project then does a full reconcile of file system and finally the next server is restarted. In the frontend this trigger of operation is locked to a debounced mechanism that is triggered when there are no more changes in the property for 300ms.

This debounced mechanism is responsible to prevent multiple unwanted operations that might be redundant or part of a final operation (i.e editing a text content from a text primitive). This constitutes an overhead of at least 300ms before the server is rendered. To tackle this limitation an different approach should be followed:
Decouple doOperation actions from render actions: The render action should be done in a AOT render style for when a change occurs, the application tries immediately to render it. If a new change is detected this mechanism should cancel the previous render request and start a new one. This would constitute an "elastic debounced" approach where the time it takes to render the changes is the tick of change's listener.

Regarding the reconcile approach, today when a change to a component is done there is a call to the same mechanism used when you load project which tries to validate all the components from the metadata (used preload() hook). This also constitutes a smaller overhead since its coupled with how much components a project has, in tests with production grade projects I'm getting values between 50ms and 150ms. This could be improved if only the changed components were required to be checked by the FSGen.

Other issues:
While investigating this case I was able to identify an issue with the paths of the files in the project. If a project is opened in a windows machine, the Cluch folder is in a absolute location (different from unix systems that rely on the ~ [home] folder) that are being uploaded to the project with that absolute path, making a copy of it when you open the project in an unix system.

[bruno12mota]

The debounce on operations is not really to debounce the work since the work to be done is done in a queue fashion. The issue is with creating thousands of similar operations that then need to go into the server. Ideally these should merge if they have the same purpose, this would be done in the operation manager that already defers the communication of operations to the server.

As to hot reloading, that needs to be handled by next, if the fs gen is fast and all that is working efficiently, then it should be up to the hot reloading to handle it fast. a good one would be to try out next turbopack and see if we can already use for projects and see if it has big performance benefits.

As to the fs gen, checking if certain components fs should be reconciled should be fast unless that mechanism is broken somehow.

[tmota900]

If I understand, regarding the doOperation mechanism we would insert the wanted renders in said operations queue but pre-process them before sending the request to API, clearing/merging changes that are redundant, reducing this way the number of done operations but keeping them coupled with render process (possibly overcoming any synching issues that might arrive if we went for a AOT render process)?

Regarding the hotreload. I have 2 separated behaviours atm. If I change a property on a component this render is not triggered by a file system watch trigger but instead uses a memoized change on project metadata: https://github.com/clutch-creator/creator/blob/development/packages/core/src/features/creator-project/manager/hooks.ts#L105
If a change is done via code mode for instance, this will trigger a FS watch event.

The fsGen part was already optimized to work with nodejs runtime in parallel (as a worker process) instead of being done in the DOM itself. It has a cache validation that is fairly fast (also done in parallel agains the required file array). The quirk I found here was a BUG I believe that included more files than it should (referring to the windows issue with files in C:\.... The optimization was instead of sending an event to reconcile every file (even tho most of them are already cached), we could send only the required ones based on the changes made.