Middleware for curl? about curl

[unscriptable]

Hello @manast,

This is an interesting approach -- especially if some modules need more than one transformation. So far, I haven't seen any project hampered by a lack of multiple transformation handlers. The current plugin system seems to work fine.

You're not the first person to suggest using localStorage/sessionStorage, though. (Side note: I am dubious about it's effectiveness versus the browser's regular cache.) This may be the first substantial use case for multiple transformation handlers.

We devised a concept called "implicit plugins". These plugins are not explicitly mentioned in dependencies (unlike normal plugins: "text!template.html" or "link!app.css"). It is certainly possible to chain these.

Actually, it wouldn't be too hard to chain normal plugins, either, but I don't know if that really makes sense. It seems like there'd have to be lots of imperative rules in order to cooperate with current, non-chainable plugins.

I like the idea and will continue to keep it in mind while designing the implicit plugins. Maybe I won't call them "plugins" at all. Chainable "handlers" or "transformers" sound like better names now that I've read your note. :)

Thank you very much for the suggestion!

• John

[manast]

Hello, I think that implicit plugins would be effectively the same thing, the advantage with the middleware approach is that you will use middlewares in the implementation of the core itself which maybe keeps the code shorter.

About the localStorage as caching, I have a use case where it is very important: offline applications. ApplicactionCache is nice and everything, but it is a PITA to keep it updated in the server. When developing a rich web application I would not need to care to update the applicationCache all the time, instead If I can rely on the module loader subsystem, my life will be much simpler... I also advocate for not needing to have a static resource builder in the server, this should all be handled automatically by the module loader system, with niceties such as minification and caching for optimum performance.

I am happy that you liked my idea, if I can help with anything please tell me. I am really in need of an implicit plugin/middleware mechanism in my application, currently I made a hack on require.js to make it work, but this is far from optimal.

regards.

[unscriptable]

I think that implicit plugins would be effectively the same thing

I agree.

the advantage with the middleware approach is that you will use middlewares in the implementation of the core itself

Again, I agree. I am calling them plugins, but 90% of the code would be in the core.

About the localStorage as caching, I have a use case where it is very important: offline applications

Yet again, I agree. :) I was totally not thinking about HTML5 apps / offline apps.

I am happy that you liked my idea, if I can help with anything please tell me. I am really in need of an implicit plugin/middleware mechanism in my application, currently I made a hack on require.js to make it work, but this is far from optimal.

I could certainly use some feedback and platform testing. I have to beg my co-workers to test Android atm. :)

I want to have implicit loader/plugin thingies (see what I did there?) in a dev branch by the end of this month.

Have you seen this by @rbackhouse / @zazl? https://groups.google.com/group/amd-implement/browse_thread/thread/baeb3c8daaf0ec5e?fwc=1 He did something similar, but created an entire AMD loader.

Regards,

-- John

[manast]

lsjs is indeed really cool I will take a look at it.

[manast]

another crazy feature that comes to my mind is this: instead of requesting the whole javascript files when some files have changed, why not just do it right and have a simple version control system where you actually only get from the server the changes of all the modified files? (I have some experience with impl. of control revision systems and I know this is easier than it seems :).

On Sep 13, 2011, at 3:18 AM, John Hann wrote:

I think that implicit plugins would be effectively the same thing

I agree.

the advantage with the middleware approach is that you will use middlewares in the implementation of the core itself

Again, I agree. I am calling them plugins, but 90% of the code would be in the core.

About the localStorage as caching, I have a use case where it is very important: offline applications

Yet again, I agree. :) I was totally not thinking about HTML5 apps / offline apps.

I am happy that you liked my idea, if I can help with anything please tell me. I am really in need of an implicit plugin/middleware mechanism in my application, currently I made a hack on require.js to make it work, but this is far from optimal.

I could certainly use some feedback and platform testing. I have to beg my co-workers to test Android atm. :)

I want to have implicit loader/plugin thingies (see what I did there?) in a dev branch by the end of this month.

Have you seen this by @rbackhouse / @zazl? https://groups.google.com/group/amd-implement/browse_thread/thread/baeb3c8daaf0ec5e?fwc=1 He did something similar, but created an entire AMD loader.

Regards,

-- John

Reply to this email directly or view it on GitHub:
https://github.com/unscriptable/curl/issues/24#issuecomment-2077285

[manast]

and in fact, even easier since there is already a git implementation in js: https://github.com/danlucraft/git.js

:D

On Sep 13, 2011, at 3:18 AM, John Hann wrote:

I think that implicit plugins would be effectively the same thing

I agree.

the advantage with the middleware approach is that you will use middlewares in the implementation of the core itself

Again, I agree. I am calling them plugins, but 90% of the code would be in the core.

About the localStorage as caching, I have a use case where it is very important: offline applications

Yet again, I agree. :) I was totally not thinking about HTML5 apps / offline apps.

I am happy that you liked my idea, if I can help with anything please tell me. I am really in need of an implicit plugin/middleware mechanism in my application, currently I made a hack on require.js to make it work, but this is far from optimal.

I could certainly use some feedback and platform testing. I have to beg my co-workers to test Android atm. :)

I want to have implicit loader/plugin thingies (see what I did there?) in a dev branch by the end of this month.

Have you seen this by @rbackhouse / @zazl? https://groups.google.com/group/amd-implement/browse_thread/thread/baeb3c8daaf0ec5e?fwc=1 He did something similar, but created an entire AMD loader.

Regards,

-- John

Reply to this email directly or view it on GitHub:
https://github.com/unscriptable/curl/issues/24#issuecomment-2077285

[rbackhouse]

That's actually what the lsjs loader will do if the "timestampUrl" configuration property is specified. The loader will attempt to use it to obtain timestamp information on the modules it currently knows about. For each page load the "timestampUrl" is POSTed an array of objects in JSON. Each object contains a "url" property and a "timestamp" property containing a date string. The called server is expected to check each url's timestamp and if different return the url in an array of strings back to the lsjs loader. The lsjs loader will load these modules via xhr and update the stored value in local storage.

[manast]

yes I know that. But what I mean is that instead of sending the whole js files, you will only need to send the differences, which are often much smaller than the whole file...

On Sep 14, 2011, at 11:45 AM, Richard Backhouse wrote:

That's actually what the lsjs loader will do if the "timestampUrl" configuration property is specified. The loader will attempt to use it to obtain timestamp information on the modules it currently knows about. For each page load the "timestampUrl" is POSTed an array of objects in JSON. Each object contains a "url" property and a "timestamp" property containing a date string. The called server is expected to check each url's timestamp and if different return the url in an array of strings back to the lsjs loader. The lsjs loader will load these modules via xhr and update the stored value in local storage.

Reply to this email directly or view it on GitHub:
https://github.com/unscriptable/curl/issues/24#issuecomment-2091524

[unscriptable]

Once curl.js 0.6dev is stable, I'll be hoping you guys have some time to help me out with this feature. :) -- J

[rbackhouse]

TBH honest I don't see that much benefit of doing this over pulling down the whole file. You would now have to have code in the client that is doing the merge. That would add to the size of the loader etc. Also it could be quite time consuming depending on the complexity of the changes.

[manast]

I think its worth to test. The code for applying a diff patch cant be much larger than 50 lines. You are probably right that in most cases it will not make any difference to send all the modified files again, its just that the engineer in me wants to make the optimal solution, and wasting broadband sending redundant data is not optimal, is it? ;P

On Sep 14, 2011, at 4:56 PM, Richard Backhouse wrote:

TBH honest I don't see that much benefit of doing this over pulling down the whole file. You would now have to have code in the client that is doing the merge. That would add to the size of the loader etc. Also it could be quite time consuming depending on the complexity of the changes.

Reply to this email directly or view it on GitHub:
https://github.com/unscriptable/curl/issues/24#issuecomment-2093962

[unscriptable]

Hey @manast,

The implicit module loaders code is operational. It's still a work in progress since I am sure we'll find new and exciting ways to exploit the feature. The new curl/loader/cjsm11 module uses it to process unwrapped CommonJS Modules/1.1 modules. Take a look at that file and let me know how you feel about creating a localStorage loader/processor.

-- John

[unscriptable]

You need some more info in order to use the module loader. I'll have that ready by Monday. Just wanted to see what you thought of the "curl/loader/cjsm11" implementation. -- J

[manast]

Hello John,

I am taking a quick look at it now (I am still at holidays with limited computer time :).

I think it is quite elegant to re-use the text plugin for fetching the javascript source as you did.

For the cache plugin, it will be needed to get some timestamp or checksum information from the server for the modules that are available in the local storage. I was thinking that in a webpage bootstrap, all the checksums of the available assets are collected in an array and sent to the server. The server will just return which assets are old, and with that information we will know which modules to fetch from the server and which to read from the cache. So the question is if this initial server request would be part of the plugin, and in that case we need an initialization part that is only executed once, or if we have it as a separate operation and the metadata is just passed to the plugin somehow. I believe the former is the best way to do it, for several reasons, but one of them is that it is quite convenient to have everything that the plugin needs to work as part of the plugin itself.

Finally, just as a side note, wouldn't be nice if functions such as injectSource, injectScript, etc are part of curl's private API? oftentimes the most complicated part for a plugin writer are just implementing these kind of functions. I understand that this will make the plugins incompatible with other loaders, but still I think it would be a nice addition that will also reduce redundancy...

I apologize if my remarks are incorrect, since I am a newbie and I do not understand yet all the details of your solution.

regards.

On Jan 6, 2012, at 2:46 PM, John Hann wrote:

Hey @manast,

The implicit module loaders code is operational. It's still a work in progress since I am sure we'll find new and exciting ways to exploit the feature. The new curl/loader/cjsm11 module uses it to process unwrapped CommonJS Modules/1.1 modules. Take a look at that file and let me know how you feel about creating a localStorage loader/processor.

-- John

---

Reply to this email directly or view it on GitHub:
https://github.com/unscriptable/curl/issues/24#issuecomment-3384644

[unscriptable]

Hey @manast! Happy holidays! Glad some people are still celebrating. :)

For the cache plugin, it will be needed to get some timestamp or checksum information from the server for the modules that are available in the local storage. I was thinking that in a webpage bootstrap, all the checksums of the available assets are collected in an array and sent to the server. The server will just return which assets are old, and with that information we will know which modules to fetch from the server and which to read from the cache. So the question is if this initial server request would be part of the plugin, and in that case we need an initialization part that is only executed once, or if we have it as a separate operation and the metadata is just passed to the plugin somehow. I believe the former is the best way to do it, for several reasons, but one of them is that it is quite convenient to have everything that the plugin needs to work as part of the plugin itself.

I think the former is the best way, too. Both are possible, though. Here's a rough sketch, iiuc:

load: function (absId, require, loaded, config) {
    // figure out urlToDetermineOldFiles from config (to get url) and inspection of cache
    if (urlToDetermineOldFiles) {
    require([urlToDetermineOldFiles], function (oldFiles) {
        // remove old files from cache
        removeOldFiles();
        loadFile(absId, loaded);
    });
    }
    else {
        loadFile(absId, loaded);
    }
}

Finally, just as a side note, wouldn't be nice if functions such as injectSource, injectScript, etc are part of curl's private API? oftentimes the most complicated part for a plugin writer are just implementing these kind of functions. I understand that this will make the plugins incompatible with other loaders, but still I think it would be a nice addition that will also reduce redundancy...

injectSource isn't needed by curl. injectScript is implemented in curl as loadScript, but loadScript does a whole lot more than injectScript. I guess I could extract and reuse the part that might be useful to plugins and loaders. For now, I opted to try to write the module loader like a plugin which could help make it compatible with other loaders.

I could go either way.

-- J