Need hotfix override system about manageiq-ui-service

[himdel]

Well, that's the problem :) we don't have those files on the appliance at all. All we have is /var/www/miq/vmdb/public/self_service/js/app-*.js and lib-*.js, which are respectively the concatenation of all the files from client/**/*.js and the concatenation of all the library dependencies. That's it :).

That's why I was proposing having a copy of the ssui source on the appliance, and having all that's needed to regenerate these app* and lib* files from that source.

[chriskacerguis]

@himdel so, some thoughts to your comments:

RE: frequency of hotfixing
I can't comment about how rare hotfixing is, but it seems that it is needed enough for @dclarizio to request a system to manage it (whatever that system is).

RE: security
I agree with you there, we do open a possible vector, but it seems like a good point to start a discussion for something to do.

[AparnaKarve]

I'll go with option 2 also since it seems like the most feasible option at the moment.
Maybe we could deploy a tool like appliance_console that we currently have on our appliances to apply the hotfix and rebuild SSUI.

[dclarizio]

Adding @martinpovolny @AparnaKarve @himdel for thoughts

[chriskacerguis]

With all the new things we are doing, I wonder if the rails asset pipeline is going to continue to cause issues; it really doesn't help us (the SUI team) much.

[AllenBW]

@eclarizio and I went through this once... for sui because we cat everything into 4 or so files, ja just gotta rebuild the file in question (but really only two, because two of the files are dependency files)

[chriskacerguis]

@AllenBW wrong "clarizio"

[AllenBW]

Nope, right one :-P, roping him in to see if this ended up working in the end or what became of our fix...

[himdel]

Well, moving to a separate node app means it would be easy. But that's not something we can backport, however, we can do something similar for euwe & darga..

• ensure the appliance contains a copy of the self-service repo
• ensure the appliance contains all the dependencies for gulp build
• then, to hotfix we simply patch the repo copy (or untar the changes over it, same as in the ops ui)
• aand run gulp build (and possibly npm install for hotfixes containing dependency changes, if any)

That way, it works the same whether we just have that copy or a running node app, it works the same as in the ops ui, hotfixes can be stacked (with care - obviously that means patching, not replacing), and everything works with minimal changes to the existing process :).

EDIT: though note that to support hotfixes with dependency changes, we'd need to install a newer npm (ie. run npm install -g npm during the appliance build process or before the first patch).

[chriskacerguis]

@himdel You are right, this really isn't something we can backport. I'm not sure I fully understand your solution. We are trying to get away from from doing what you are describing (again, not sure I fully understand). That way we can provide the customer with the ability to drop in a file, and restart the server. There would be nothing more for them to do.

[himdel]

@chriskacerguis dropping a file and restarting the server works only if the file is a file containing all the javascript from SSUI, concatenated (and you don't actually have to restart the server in that case).

That means we would never be able to do 2 hotfixes on one machine (because you have no way to even find out if a hotfix had been applied or not), and certainly makes the process different from what is done in the ops ui - replacing only the changed source files.

My solution means you can do exactly that - replace the source files that need changing, it also means you can find out if a hotfix had been applied or not (diffing against the release source), and you can apply as many hotfixes as you want (that one only provided you patch the files instead of replacing, or that there's no overlap).

It also means that if we ever do move towards node serving stuff on the appliance, it will continue to work exactly the same, because you'd still be replacing the source files. (There will be a difference in that before node server, you run gulp build after applying the patch, after node server, you'd restart the node server after applying the patch instead.)

So.. depends on the priorities.. If the goal is indeed to just give the customer a file and nothing else, then what we already have works that way :). But the whole reason we're having this discussion is that we can't just expect to replace the built artifact and be done with it because it breaks the moment you need more than one hotfix, it breaks with any custom local changes, etc.., isn't it?

[chriskacerguis]

@dclarizio To give some context here, can you please chime in with the details of what is prompting this?

[chriskacerguis]

@himdel I'm not sure I understand you when you say "dropping a file and restarting the server works only if the file is a file containing all the javascript from SSUI". Wouldn't that file just need to contain the contents of the replaced file?

[chriskacerguis]

@himdel thank you for clarifying. That is what I thought was being done here #217 by the rails asset pipeline. So, as a part of that, we need to stop doing that concatenation.

[himdel]

@chriskacerguis No, we cannot stop concatenating files, we need to serve these files concatenated in production - otherwise, you'd be downloading ~200 files on each load, instead of 2. That's why, to be able to hotfix, we need to be able to do the concatenating on the appliance itself.

[chriskacerguis]

@himdel I wonder if we need to rethink that. I would want to dig in more regarding why we would need to load ~200 files. One of the big questions that I have is everything we are loading required, or are we loading things that wouldn't necessarily be required? I agree that the less http requests the faster the app, but if we start to break things up wouldn't that allow the browser to cache a bit more efficiently?

I tend to break things up in 3 way:

1. Things that are needed on every page
2. Things that are needed per module
3. Things that are needed per page

If we move this to a task, there is a lot of work for sure. Another thing we need to consider is do the pros outweigh the cons (multiple files make for easy hotfixing, but bring up all the hints that you have mentioned).

Finally, for the time being we can't run gulp build on the appliance as it doesn't have Node 6.x. I'm sure it will soon as it is now LTS.

[himdel]

@chriskacerguis Well, 122 files is our source, and we're using about 47 external dependencies, so at least that many files, probably more :).

Loading things that are not necessarily required is a moot point, sorry, angular 1 does not support lazy loading, so we're out of luck. Angular 2 does however, so we could revisit that then. (And I have heard about projects doing their own lazy-loading in angular 1, but it was always described as an ugly hack which doesn't really give me any confidence.)

As for gulp build, you're right, but for darga, we definitely don't need node 6, and as for euwe, I guess @AllenBW knows better than me, but imho the only thing requiring node 6 was eslint which we don't have to use during gulp build, and node-sass which doesn't have to go in euwe, right?

[AllenBW]

Actually most things can be made happy with node 4 (eslint is >=4) but if we're going to update, why not update (unless it breaks something unfixable). I say this not knowing the context/direction of the conversation, don't mind me.

[chriskacerguis]

@himdel yeah, again there's lots of work to be done...and not a "simple" thing. I know Angular 2 does lazy loading, and I think that is something we will want to look at eventually.

[chalettu]

So the suggestion that @himdel made earlier about possibly using gulp-build on the appliance would be the best route to make this possible. The practice we are using of taking all our source files and combining them for ease of use at runtime is a fairly common industry practice these days. We could actually go a similar direction and generate new JS files with the hotfix for the user , maybe as a zip file and then have the user upload the zip file via a web interface. If we started using cache busting gulp plugins, we could make sure the new concatenated JS files have a different filename automatically than the old ones and just repoint index.html to the new js files.

Just a thought

[himdel]

we could make sure the new concatenated JS files have a different filename

@chalettu the generated files already have a hash in the name, so.. yeah, we already do that :).

[chalettu]

Cool. Didn't look close enough to see that, well that makes that easier then.

[chriskacerguis]

@himdel so, I think @chalettu might be on to something. I think that the goal is to make hot fixing as easy as possible for the end-user. So, any suggestions on making that easy are welcome to get this requirement moving...and I like @chalettu thought on that (some sort of upload tool)

[himdel]

Aaah, sorry @chalettu, I completely missed the upload suggestion.

Well, yeah, from the user's perspective, this is a great idea! I'm not as sure if it's still great given the context though - 2 things concern me.. a) hotfixing is (AFAIK) relatively rare, so it feels like making a web UI for that could be an overkill, especially since who'd be applying the patches would be sysadmins anyway, right?

And b), security. By implementing something like that, we're opening a huge possible attack vector, since the app needs to be able to write in it's own directory without restriction, and also to the manageiq (or at least manageiq/public) directory, and to run scripts in the system. Which means the impact of any security oversight in SSUI would essentially change from a mild inconvenience to a possibility of complete control of the appliance.

So.. it's a trade off, and I'd say, if we expect people to do that daily, sure, go for it, but it seems a disproportionate amount of work for something used rarely. (Also, this is something we can't do for darga or euwe retrospectively.)

[chriskacerguis]

Just to make sure we are all on the same page, here are the requirements that I have been given:

1. This "hotfix" system would be for the "F release" or above
2. Need way for customer to easily apply hotfixes (and be able to apply hotfix on top of hotfix)
3. Build tools (e.g. Node 6.x) will not be available to customer.

@dclarizio did I miss any?

Current (high level) solutions from this thread (if I missed one, please let me know).

Proposed Solution 1
Break up the JS files into the individual files (do not concatenate), create a system in which the system will look for a "hotfix" file first, and if it doesn't fine one, then it will load the stock file.

• CON: that's a lot of files to load (would need to move to a lazy load like system)
• PRO: easiest way to hotfix the system

Proposed Solution 2
Keep a copy of the source files on the appliance, if a hotfix is needed, put the new file in the source, and "rebuild" the SSUI as needed

• CON: Build tools may not be available on the appliance
• PRO: Simplest solution from engineering, could be complicated for end-user

Proposed Solution 3
Add some form of web UI to manage hotfixes that would upload a new file, then reload the server.

• CON: Possibly opens a security hole
• PRO: Would allow for a robust hotfix / web-update solution

From where I sit, I like option 3 the best (the web-based updater, not the hotfix system). My 2nd choice would be option 1.

@dclarizio @AllenBW @chalettu @himdel @martinpovolny @AparnaKarve what's your thoughts on which solution to go with (or feel free to propose a new solution that meets the requirements)?

/cc @chessbyte @Fryguy @bascar

[himdel]

Aaah, IMO requirement n.3 (build tools not available) severely limits what we can do ... essentially down to only solution 1, so I think that one needs to be relaxed.

I'm for solution 2 (4.), mainly because I think solution 1 is unrealistic in production, and 3 (7.) feels like a possible overkill, and still needs to use either 1 or 2 under the hood.

[chriskacerguis]

@himdel I very much agree with you, but I can't do anything about the build tools. Node 6.x is not available, and so until Node is updated to 6.x in the "blessed" repos we can't add it to the appliance.

[himdel]

tbh I'm still hoping it will get RPM'd before the F release.. but if not, then yes, we do have a problem. (I'd say just install it manually when doing the first hotfix, but not at all sure how feasible that really is..)

[chalettu]

I vote for the 3rd option. There are a lot of ways you could minimize an
attack vector to make it safer. My second choice would be option 2. I
would have to learn a little more about how we package our codebase for use
on appliances before talking more about it.

On Wed, Oct 5, 2016 at 6:13 AM, Martin Hradil [email protected]
wrote:

tbh I'm still hoping it will get RPM'd before the F release.. but if
not, then yes, we do have a problem. (I'd say just install it manually when
doing the first hotfix, but not at all sure how feasible that really is..)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#219 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOElRhqd48HQvNdP3jWJGEqcJ7EVuk-ks5qw4ZogaJpZM4KNAeq
.

Chris Hale

Senior Software Engineer
[email protected]
512-585-7286
https://bluejeans.com/chhale

[bascar]

I like option 2 the best for many reasons but live troubleshooting/screensharing/fixing corner case issues on the fly and testing immediately is the top of the list.

Hotfix on hotfix is also a concern. Any way we could use git/hashing/other as a way to verify that what we are laying down as a fix is in fact loading into the "known" environment we expect the fix to lay into?

[chriskacerguis]

@bascar with option 2 we are limited until we can get Node 6.x (which hopefully will be QUICK) on the appliance, but that's not up to our team. That said, what option do you like?

[dclarizio]

Option 3 has an issue, load balancers/multiple appliances. The customer would have to upload that file to every UI appliance that has our source code, from the Web UI. Granted, this issue does exist even for server based hot fixes, so perhaps we can build that into the Web UI screens somehow.

I'm for option 2, which follows more closely how we do existing hot fixes, but simply adds a script step at the end, which could hopefully be automated.

[chriskacerguis]

@dclarizio I think with option 2 we could have a system setup that as we update the "branch" in Git, it would auto-update from that branch. So, if they were running E release then every time we updated the E branch it would pull down the latest (maybe even have an auto-update script that runs), so it would update all the UI's.

[bascar]

@chriskacerguis I think I still like option 2 :)

What is the dependency on Node 6? Can't we carry our own packages for it until they are pulled into the "blessed repos"?

[chriskacerguis]

@bascar As I understand things (which could be incorrect) for the appliance we can only pull from a "blessed" repo, at the present time only Node 4.x is available. We cannot add the NodeJS rpm repo. @simaishi could you please confirm / clarify things for me and @bascar?

[edit] forgot the word "rpm"

[simaishi]

@chriskacerguis @bascar For downstream, we can only install RPMs from the "blessed" repos. We can package our own and use that, but that means we have to maintain the RPM as well. We had some discussion about this in email and I think we decided not to build our own. cc @Fryguy @chessbyte

[chriskacerguis]

Thanks @simaishi !!!!!!

[chriskacerguis]

@Fryguy any concerns from your end about doing "Proposed Solution 3", and basically make a web updater?

[martinpovolny]

To start I'd clarify the requirements:

1. This "hotfix" system would be for the "F release" or above
2. Need way for customer to easily apply hotfixes (and be able to apply hotfix on top of hotfix)
3. Build tools (e.g. Node 6.x) will not be available to customer.

From the above only 1. is actually true.

Number 2.: hotfixing this way should be very rare (we have RPMs or DEB packages do to easy upgrades and the person doing the hotfixing has to be an experienced sysadmin not an accountant.

Number 3.: as was written earlier in the discussion we don't need the latest Node for the build.

On solutions:
Solution number 1 was rejected due to technical reasons. We don't want the files to be split nor does our toolchain and used libraries support that.

Solution number 3 is an overkill. We are not writing software for home modem where end user needs to upload fixes. We actually distribute appliances and in Linux distributions we distribute packages. Also we do not intend to support combinations of hotfixes. We actually do not want so support hot-fixed installs except for some rare exceptions.

That leaves us with Solution 2 that I personally find the only reasonable and feasible one.

[chessbyte]

@chriskacerguis Option 3 does seem like overkill to me as well. We would be investing in something that we would not need once Node 6 is on the appliance. Additionally, the Web UI of Option 3 would need to make 2 big assumptions: 1) only allow system admin to do this and 2) all CloudForms appliances may live inside the corporate firewall and have no "internet" access. So, I vote for Option 2, aka KISS

[chriskacerguis]

@martinpovolny because we have updated the dependancies, we need Node 6.x. As we saw, when Node was running an old version, the build failed. Also, those requirements are neither true nor false, they were communicated to me; they simply just are (their validity may be questionable).

@chessbyte I agree an update system for just hot fixes is overkill, however from an overall process I do think that an auto-update system could be beneficial for the project; but I think that is a larger discussion.

So, as it stands, Node 6.x is not available on the appliance; so until then we will just have to build files for a hot fix until Node 6.x is available. I will leave it in the very capable hands of @simaishi to get that figured out. Once it is available, we will go with Solution 2.

[chriskacerguis]

Re-opening this for additional discussion.

[Fryguy]

@chriskacerguis We will also need to find a way to deploy this differently for downstream (probably RPM, but we are still discussing), so whatever choice is made should be aware of that.

[Fryguy]

Reading back, option 2 is probably the most flexible and allows us to deploy hotfixes via RPM downstream (and with RPM allowing us to auto-build the SSUI afterwards).

One downside to the rebuilding is how we deploy the files downstream anyway though. We pre-build the SSUI files and make them part of the RPM. If we remove and replace them, then I think RPM will not be happy. We'll have to find a way to avoid that, perhaps by not pre-building in the first place. cc @simaishi

[chriskacerguis]

@Fryguy Was that Martin P's "option 2", b/c option 2 from my list was keeping a copy of the source on the machine and rebuilding as needed.

[Fryguy]

@chriskacerguis I was referring to proposed solution 2 from here #219 (comment)

[chriskacerguis]

cools. just wanted to make sure :)

[chriskacerguis]

@Fryguy If we go with option 2, then that's more of a system management, not really a "code" issue per se. So, going to close this issue out and we can stick with option 2 (which I'm not sure is ideal, but keeping with KISS principles and "I don't have a better idea"™ it seems like the way to go)