Comments (23)
Thats definitely an interesting idea. Worth trying out I think.
from quarkus-super-heroes.
Although maybe the folder showing up as a different color in the IDE may have given you a clue
from quarkus-super-heroes.
Thanks, @edeandrea. Is this an accurate picture, or is there stuff in deploy/docker-compose
that doesn't go into src/main/docker-compose
? For the left side, I've only shown one service, and I've skipped the native/java aspect.
from quarkus-super-heroes.
Not quite. The deploy/docker-compose
is entirely generated by a script as well.
app/src/main/docker-compose/infra.yml
contains any infrastructure "stuff" that the app needs, but on its own is not a valid docker compose file. It only has the snippets containing the service definitions. Kind of like whats in the files withinapp/src/main/kubernetes
app/src/main/docker-compose/java17.yml
(andnative.yml
) - this is the docker compose file for the app itself. Again, not a valid docker compose file, as its only the snippets containing the service definitions. In comparison with the kubernetes stuff, this is the stuff that the kubernetes extensions would generate, but since we have no docker compose extension, we have to hand-craft
The generate-docker-compose-resources.sh
script then goes through app/src/main/docker-compose
for each type (i.e. java17, native), combines whats in infra.yml
with the java17.yml
or native.yml
file and creates a valid docker compose file that lives in app/deploy/docker-compose
(as either java17.yml
or native.yml
.
The script also then appends the services to the root /deploy/docker-compose
directory into the appropriate file (java17.yml
, native.yml
).
Does that make sense?
I've also thought about building a docker compose extension that could re-use Dekorate to be able to generate the app specific docker compose files like it does for kubernetes, but I never invested in that (no time unfortunately) and wasn't sure if there are others that would benefit from it either.
from quarkus-super-heroes.
Not quite. The
deploy/docker-compose
is entirely generated by a script as well.
Ah, ok. So the yellow line I proposed is there, but going in the opposite direction?
Actually, I see it's the same direction, but I have the labels on the two files backwards. I was getting quite muddled between the two, trying to remember which path was what.
We should perhaps do the same thing I suggested elsewhere, and .gitignore the generated content, and have the script that generates it run in the CI. And also put a comment in the file explaining that it's generated. Hopefully between those three, it might stop people (like me :) ) investing in maintaining those files.
The ideal would be not even having them in source control, but that would need an extension, as you say.
The
generate-docker-compose-resources.sh
script then goes throughapp/src/main/docker-compose
for each type (i.e. java17, native), combines whats ininfra.yml
with thejava17.yml
ornative.yml
file and creates a valid docker compose file that lives inapp/deploy/docker-compose
(as eitherjava17.yml
ornative.yml
.The script also then appends the services to the root
/deploy/docker-compose
directory into the appropriate file (java17.yml
,native.yml
).Does that make sense?
It does. When I was working through the generate-docker-compose-resources.sh
script I spotted the bit that created the aggregated files, and the bit that created the downstream files, but not the bit that created the per-project files. (I'd already manually updated them, so there was no change.)
So I think then there's no more scripting needed, it's just a question of having the generated files be more clearly labelled as generated, so people don't spend time changing them. The comment has to be in the file itself, because we can't make people cross-reference files they might change against readmes to see if the file is listed in the readme as a generated file.
Even this would do the trick, I think, but CI + .gitignore would be more complete:
touch $output_file
echo '# this file is generated by generate-docker-compose-resources.sh. do not change it!' >> $output_file
echo 'version: "3"' >> $output_file
echo 'services:' >> $output_file
from quarkus-super-heroes.
... and ideally also a similar comment in the 'donor' yaml files explaining where that content ends up, and the script strips out that comment ... but that's slightly less trivial to implement.
from quarkus-super-heroes.
How would you envision the .gitignore
thing actually working? The CI process would have to make multiple commits...one which eliminates the .gitgnore
and commits the files, then another commit that reinstantes the .gitignore
? Or is there another way?
from quarkus-super-heroes.
I'm pretty sure the CI doesn't actually need to commit the change to the .gitignore
, it just has to have it locally. So I think something like
echo whatever >> interesting.file
rm .gitignore
git add interesting.file # being specific, here
git commit -m "this only changes the interesting file"
git push
would work. It leaves the CI with a dirty git workspace, but we don't care, because it's an ephemeral machine.
from quarkus-super-heroes.
It ends up being "one rule for the CI, one rule for the humans," which is slightly confusing, but in combination with a comment on the file is probably clearest overall.
from quarkus-super-heroes.
Although it still wouldn't have kept you from modifying the files in the first place. Just would have made you scratch your head as to why your changes didn't commit :)
from quarkus-super-heroes.
Yes, you'd need the comment on the files themselves to alert people to the fact that hand-editing them is (a) ineffectual and (b) a waste of time. :)
from quarkus-super-heroes.
I'll also have to check because we are not manually doing git add
/git commit
/git push
. We are using a GitHub action to do it...Which may be fine, but need to investigate.
quarkus-super-heroes/.github/workflows/create-deploy-resources.yml
Lines 59 to 66 in 1a847cb
from quarkus-super-heroes.
See also quarkusio/quarkus#34025 for the verbosity of the config. I'll update this WI title to keep the scope just at the handling of the generated files.
from quarkus-super-heroes.
Hey @holly-cummins I was circling back around to this and I have it mostly working in a test repo.
The files look ignored in the IDE, then CI regenerates them, deletes the .gitignore
, then commits the generated resources back into source control, leaving the dirty .gitignore
alone.
See this commit for an example.
Then in my IDE I see the updates and I can pull them in. Yay!
BUT......
What I'm seeing after I pull in the updated files is that the IDE doesn't seem to treat them as ignored anymore, even though the .gitignore
file is still the way it should be. I hand-edited one of the generated files and it shows up as changed in the git view (see screenshots below).
Have you come across anything like this before?
from quarkus-super-heroes.
Actually upon further investigation, it doesn't seem that its the IDE that is the problem. If I do a git status
I see that the file is modified too..
╰─ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: rest-villains/deploy/docker-compose/java17.yml
no changes added to commit (use "git add" and/or "git commit -a")
from quarkus-super-heroes.
It seems that .gitignore
is not used when selecting files to commit, it's used for adding only: files listed there are not suggested to be added. But, once they are added (i.e. via a commit once the local .gitignore
is removed), they appear in git status
.
from quarkus-super-heroes.
So I'm not sure what we are trying to achieve is something that is technically possible.
I can definitely add something like this to the top of all generated files:
#####################################################################
# THIS FILE IS AUTOMATICALLY GENERATED DURING CI/CD.
# ANY LOCAL CHANGES YOU MAKE SHOULD NOT BE COMMITTED TO SOURCE CONTROL.
#####################################################################
There are ways to tell git locally to ignore local changes to tracked files by doing something like
git update-index --assume-unchanged <filename>
but then any developer would have to do that on their local machine, which I think is a road we do not want to go down.
There are actually 2 options that could get around this a bit: https://www.baeldung.com/git-assume-unchanged-skip-worktree
But again, each user would have to do it for themselves because it is working at the local staging area. And then if you were to git pull
changes, it would revert.
from quarkus-super-heroes.
Hmm, annoying.
I agree that no matter what, we should add the comment to the generated files.
I guess the other option is that we don't have the CI commit the generated files. If we think people will need them when deploying locally, we could move the generation process to happen in a build step, rather than the CI.
from quarkus-super-heroes.
I guess the other option is that we don't have the CI commit the generated files. If we think people will need them when deploying locally, we could move the generation process to happen in a build step, rather than the CI.
I guess I don't understand what you mean here? The files are there so it is stupid simple to docker compose up
or kubectl apply -f
the entire system. We can't assume anyone who wants to deploy the system even has a java runtime installed.
In the case of kubectl apply -f
, the user doesn't even need to have the repo cloned locally. They can do something like kubectl apply -f https://raw.githubusercontent.com/quarkusio/quarkus-super-heroes/main/deploy/k8s/java17-kubernetes.yml
from quarkus-super-heroes.
I just merged #316 which adds the comment to the top of the generated headers.
from quarkus-super-heroes.
One thing we could do, which would involve the user actually doing something, is that we could create a script which "untracks" those files. We could include instructions for users on how to register it as a pre-commit hook in their local repo.
But we couldn't control whether or not they actually did that.
from quarkus-super-heroes.
I think asking users to run a script undoes the aim of the changes, which is to try and ensure users can 'do the right thing' in the project without having to read lots of documentation.
So I think we should leave it with just the comment in the files.
from quarkus-super-heroes.
I agree. I tend to shy away from anything that would make the user have to do anything else (for which they would need to read something that they already aren't :) )
from quarkus-super-heroes.
Related Issues (20)
- Drop Java 11 HOT 10
- Fights service when deployed to k8s doesn't get random fighters HOT 8
- Switching from jmeter to hyperfoil HOT 4
- Rewrite UI in a more modern framework HOT 1
- Clean up various kubernetes profiles config HOT 4
- Add a narration microservice that can narrate fights
- Fight microservice website information is outdated
- InvalidDefinitionException with Instant in tests with quarkus dev mode HOT 5
- Missing config key in hero register rest client annotation
- Support native compilation in narration service HOT 1
- Introduce Java 21 HOT 8
- Introduce some gRPC? HOT 1
- Update azure containerapps docs & script with narration service
- Introduce some gRPC?
- Stork env vars not working properly when deployed to k8s HOT 4
- Switch narration app to use quarkus-langchain4j HOT 12
- Change fighters/location UI around a bit to use flip cards
- Add microcks to UI service
- Add openshift-ai integration into narration service
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quarkus-super-heroes.