Comments (8)
If you ask about the per agent limit of DRONE_MAX_PROCS
then yes, it is respected.
Continuing on the above example, if you have lint.yml
, test.yml
and build.yml
, and you have a single agent with DRONE_MAX_PROCS=2, then one of the 3 pipelines will wait until there is an available agent. If you have two agents with DRONE_MAX_PROCS=2, then they can all start, and you have spare room for one additional pipeline.
The behavior is exactly the same how matrix builds work today.
from woodpecker.
Docs:
https://github.com/laszlocph/docs/blob/master/content/usage/concepts/multi-pipeline.md
How to try it out on a branch while keep existing builds running?
- Change the "Pipeline Path" in the repo settings to .drone/
- Click "Fallback to .drone.yml if path not exists"
- Add your yamls in the .drone/ folder
It will search for the pipeline files under .drone/
and it will find it on your branch. While for existing branches where there is the existing .drone.yml
it will fall back to that file.
from woodpecker.
The design that I'm implementing:
The pipeline is either in the .drone.yml
file or the .drone
folder.
- pipeline location may be configurable in a later iteration
- with the defualt pipeline location, the
.drone.yml
takes precedence if both the.drone.yml
file and the.drone
folder exists
For multiple pipelines you must use separate files
- the pipeline file format does not change, each file describes one pipeline with the well known control flows: parallelism, branch and pipeline triggers
- an example
.drone
folder
.drone
├── lint.yml
├── test.env
├── build.yml
└── deploy.yml
- the different pipelines run in parallel on separate agents
- with custom status lines feeding back to Github
- pipelines can have a
depends_on
triggers to block thedeploy.yml
until the prerequisites are finished
With this approach we tackle two problems:
- Drone files growing too big
- Feedback loop is slow on purely lint/test errors
from woodpecker.
the different pipelines run in parallel on separate agents
Even if a limitation is set to an agent / process?
I understand the thoughts behind and this is very useful to accelerate everything.
However, a specified limitation should not be ignored (maximum parallel instances).
Or do I understand something wrong?
But your idea is thought through.
I like it!
from woodpecker.
That's great! - Thanks for the answer.
from woodpecker.
I've still have a question or maybe a feature request.
It is possible to have "Named Agent" for different build tasks?
Example:
Fileserver : QuadCore + 8 GB RAM (Intel(R) Pentium(R) CPU G4560 @ 3.50GHz)
Workstation: QuadCore + 16 GB RAM (Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz)
drone server and drone agent are always running on Fileserver...
Default = Fileserver
Some bash logic for demonstrating:
if [ "Workstation" == "online" ] ; then
DRONE_USE_AGENT="Workstation"
else
DRONE_USE_AGENT="Fileserver"
fi
I hope that I have shown understandable.
Edit: If this not the right topic, i can create a new issue for this...
from woodpecker.
Distinguishing between agent capabilities is not in scope for this task.
I believe that it's best if agents have identical capabilities in general. Now some cases would require named agents, like when you attach some special hardware to your agent nodes. But even then, the mapping would be static, so the presented if-else logic would not be a feature I would implement.
from woodpecker.
The if-else logic I have just introduced for more clarity.
It is true what you say that actually all agents should have the same conditions to deliver a reproducible result. I was more concerned with the temporary and faster execution of a build, which could be reduced to a quarter of the time if you run the agent on an another computer.
Thanks again!
from woodpecker.
Related Issues (20)
- Document "when and what to contribute" HOT 1
- Document how the workflow is parsed
- Woodpecker daily cron builds get stuck
- Show linter errors as highlighted text in UI
- WebUI does not display Oauth2 login error
- Refactoring: use DAG for steps in backend. HOT 1
- Agent configuration WOODPECKER_FILTER_LABELS= should not fail
- Improve cli
- Setting WOODPECKER_DATABASE_DATASOURCE_FILE does not work as expected when using docker image HOT 12
- Gitea Webhook - https://git-repo.local/api/v1/version: context deadline exceeded HOT 1
- Restarting pipelines does not work
- Linter should not report warning for missing when in steps if global when is set HOT 2
- Add custom timeout and retrys to gitea/forgejo client HOT 4
- Volume Mount error: name is not a valid kubernetes dns name HOT 3
- Gitlab user email, name and avatar of
- Add new PR event that is triggered on any changes to a PR. HOT 1
- Server panic on manual run & push
- Pipeline stuck in pending status
- No logs when cannot connect to database
- matrix workflows set up environment in wrong way
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from woodpecker.