mserranom / bbrun Goto Github PK
View Code? Open in Web Editor NEWRun Bitbucket Pipelines locally
License: MIT License
Run Bitbucket Pipelines locally
License: MIT License
Hi, I tried to use Private images hosted by AWS ECR (EC2 Container Registry) as you can see in documentation, and I receive an error.
The context is, I have a custom pipelines image, hosted in private AWS repository, and I want to use it, to build code with controled environment.
executing step in "[object Object]" docker: invalid reference format. See 'docker run --help'.
If I use a public image, everything works ok.
Thanks
Currently deployment to npm.com is performed after a master push in Travis.
After this change:
If I run it interactively, it works.
My script called deploy.sh
bbrun $1 $2 $3 --pipeline branches:$BRANCH --env "AWS_SECRET_ACCESS_KEY=xxx, AWS_ACCESS_KEY_ID=yyy"
I made a script to basically copy everything to a /tmp area, and only them run bb in it. Else, it ends changing my project files, since I do npm install and I kind of change a few file contents depending on the deployment environment (dev / prod strings, stuff like that). But when running on Bitbucket, everything happens in a docker and gets disposed.
Anyway, the problem is that it fails directly. I also commented out my aws commands on yml file and it keeps saying "my aws command" is faulty. Interactively, it works like a charm.
At the moment I am at a feature branch, say "feature-1".
But I want to test the scripts for a different branch, say "dev".
I tried
bbrun --pipline branches:dev
and
bbrun --pipline dev
but only the default scripts will be executed.
The bitbucket-pipelines.yml
looks like:
image: node:6.10.3
clone:
depth: 5
pipelines:
branches:
dev:
-step:
scripts:
- echo dev
master:
-step:
scripts:
- echo master
default:
-step:
scripts:
- echo default
Can you explain how i can tell bbrun
to excute the dev scripts ?
Thank you for your help !
Many pipelines need a custom auth.json that will be passed as env to be secure. Ie:
name: Initial Setup
caches:
- composer
script:
- echo "$AUTH" > auth.json
- php -d memory_limit=-1 $(which composer) install --prefer-dist
artifacts:
- vendor/**
The parseVars function is very simple and need to be more complex for this scenarios.
using node:12-alpine as my base image
after running bbrun --pipeline branches:development
I get the following error
throw err;
^
Error: Cannot find module '/ws/bash'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:982:15)
at Function.Module._load (internal/modules/cjs/loader.js:864:27)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
at internal/main/run_main_module.js:18:47 {
code: 'MODULE_NOT_FOUND',
requireStack: []
}```
I think Bitbucket use the environment Variable "CI" to indicate that the script is running on a pipeline.
Should bbrun automatically set CI to "true" ?
returning following error steps.forEach is not a function
when running the following
command:
bbrun --template test-pipelines.yml
test-pipelines.yml
image: node:10.14.1
definitions:
caches:
node: node_modules
steps:
- step: &Test
name: Run Tests
caches:
- node
script:
- npm run test:pipeline-ob
- npm run test:pipeline-webauth
pipelines:
default:
"**":
- step: &Test
Often a Bitbucket pipeline has some sort of deployment step that'll execute some SSH commands on a remote server, rsync some files etc.
At the moment debugging this scenario is quite difficult if you have a local Docker container acting as the "remote server", but the containers running each step don't have access to this "remote server" because they run on different Docker networks.
One way of tackling this is by giving the ability to specify the Docker network to run the containers in, an example would be:
bbrun --network example-network --env "SSH_HOST=example, SSH_USER=bitbucket
Notice the --network example-network
, this would allow each step to have access to your container running outside of bbrun using the container name of the "remote server" (assuming that your container is "example" and connected to the "example-network").
It would be beneficial to add a flag for ssh keys, because a pipeline could be installing a private package.
BBrun should support persisting files between steps. As it stands, each call to src/docker.js#run runs a separate container, erasing created files.
Please document which node version needed to use bbrun
I found out, that node 12.0.0 is needed?
As in:
options:
size: 2x # all steps in this repo get 8GB memory
pipelines:
default:
- step:
size: 2x # or configure at step level
script:
- ...
More informatio in Pipelines stocking stuffers: test reporting, Docker run and large builds
I managed to create a null step in an otherwise valid bitbucket-pipelines.yml
, and got the uninformative error above. I'm proposing (1) guarding each of the config[attr] || inScopeAttr
in bbrun.js and (2) providing a traceback to the null/undefined part of a pipeline.
Problem: The engines dependency is too strict in package.json, requiring node 8.5.0 exactly.
Recommended Solution: Be more forgiving. "node": ">= 8.5.0"
Current Behaviour:
When the --interactive
flag is passed, the container is run with /bin/bash
. However in the course of a normal pipeline run, sh
is used though the .bbrun.sh
file has it's first line pointing to bash via #!/usr/bin/env bash
.
This causes bash specific features like the source
command that's used to run a script that doesn't have executable permission to fail (like source venv/bin/activate
that's common in python projects)
Expected Behaviour:
Since we already use /bin/bash
in the interactive shell, we could use /bin/bash
for non-interactive mode script execution as well.
I started a bbrun without parameters and the fist default step was successful.
But then also another default step wich has deployment: staging & trigger: manual set.
This should be prevented!
attached you will find a test bitbucket-pipline.yml:
pipelines:
default:
- step:
name: Build & Test
script:
- /bin/echo "Build & Test"
- step:
name: Deploy to STAGING
deployment: staging
trigger: manual
script:
- /bin/echo "Deploy to STAGING - this should not be run!"
- step:
name: Deploy to LIVE
deployment: production
trigger: manual
script:
- /bin/echo "Deploy to LIVE - this should not be run!"
For expediency, bitbucket pipelines support caching directories between steps. bbrun
should be able to emulate this.
Would be helpful to have examples available for documentation
How can I pass --net=host
to "docker run"?
Additionally, I think that if you send this all the time, there would be no harm at all even if you have a direct internet connection.
After #19 was fixed, this no longer breaks the builds, but builds are executed sequentially. They should be executed in parallel.
As described in Use services and databases in Bitbucket Pipelines
Both in README.md, and when running bbrun --help
the --pipelines
argument is not correct.
First off, thank you for your work creating this tools, it's made debugging BB pipelines locally so much easier.
I noticed that the working directory always seems to be /ws
even if a different one is specified in the Dockerfile
WORKDIR
. Is there a command line flag or other option that can be setting to specific this?
I appreciate any help you can give me.
Hello, this is not related to your add-on more a docker thing. but maybe you know the answer... is it possible to debug a pipeline script like in vagrant? In vagrant i can enter "vagrant ssh" and can go into the machine. Currently it sucks very when i wait 20 minutes for execution and then everything breaks up because of a error in the build script. I mean thanks to your module its much better then real testing but a command for going into the machine would be awesome.
Bitbucket can run steps in parallel (docs). Currently, bbrun expects each item in a pipeline to contain a script
member. This causes parallel: [...steps]
to fail.
Currently docker
commands are run with --rm
option, that destroys the container after execution.
A --keep-container
will not destroy the containers, which is useful for debugging purposes.
Hi @mserranom . I think I got all access to everything.
On NPM the organization, for me, is empty.
There is the release.sh script, which I would assume to run locally to get something released to GH.
You also have Travis config file.
Can I think if GitHub actions, from Master, calling all steps AND releasing the package to NPMJS?
One thing I ended using in a few Node projects was to set the package.json version ONLY when releasing (like it was 1.0.0 and at release time become 1.0.20200120) and never push this back to the repo. Instead, just tag the repo with v1.0.20200120. Everything gets automatic then, and releases can be fully automatic from GH itself. Travis would not be needed anymore (or use Travis and forget GH).
What do you think? I would like to have latest master released to NPMJS.
Best
Marco
Current Behavior:
Currently each time a pipeline is run, one docker container per step is created. The steps may install packages that utilize disk space. At the end of pipeline completion the containers aren't removed.
Expected Behavior:
After completion of pipeline run, the containers should be removed from docker, cleaning up any space utilized.
Used the hello-world bitbucket-pipelines.yml
and changed the docker image to docker:stable
See more here
So the file ends-up like this:
pipelines:
default:
- step:
image: docker:stable
script:
- echo "hello world!"
After running bbrun
I get the following error:
executing step in "docker:stable"
/usr/local/bin/docker-entrypoint.sh: exec: line 35: bash: not found
If you check the line 35 of docker-entrypoint.sh file in the docker image repo you see it's just an "exec" command.
Checking on docker.js file, I see it uses bash
as shell and Alpine Linux doesn't uses bash
it uses sh
instead.
I've tested changing the bash
references with sh
on docker.js and it works both for Ubuntu and Alpine images. Ideally we should point the shell to be used dynamically. Maybe using $SHELL
or $0
enviroment variables. I've tried quickly but couldn't make it work. So I'll submit a Pull Request with what I have shortly.
Currently the execution environment doesn't support running docker commands against a docker daemon.
Potentially we can use a local Docker daemon and add support as described in Run Docker commands in Bitbucket Pipelines
That particular container, which I use on bitbucket, works perfectly nice there.
But when running locally, I got an error.
Why? Because there is a mandatory entrypoint which runs and fails, perhaps because it is not inside Pipelines.
Solution: remove entrypoint on bbrun ... I am not sure if this has side effects, but by doing that, my problem was gone.
docker.js : 36
:
run --rm -P --entrypoint="" -v ${pwd()}:${workDir} -w ${workDir} ${image} bash ${BUILD_SCRIPT};
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.