Comments (7)
Oh neat! So I'm actually done writing it, just coming up with good tests. I should have a PR soon after that.
Like I said, totally okay if you don't want to use this.
from singularity.
I am looking forward to seeing what you've done.
from singularity.
yeah!! Here is a sneak preview:
$ sudo ./singularity build container.sif Dockerfile
INFO: Starting build...
Getting image source signatures
Copying blob 2408cc74d12b skipped: already exists
Copying config a366738a18 done
Writing manifest to image destination
Storing signatures
2022/06/22 13:00:12 info unpack layer: sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8
INFO: Running post scriptlet
+ apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
v3.16.0-196-g143603d2cf [https://dl-cdn.alpinelinux.org/alpine/v3.16/main]
v3.16.0-201-g5317e55db7 [https://dl-cdn.alpinelinux.org/alpine/v3.16/community]
OK: 17026 distinct packages available
INFO: Adding startscript
INFO: Creating SIF file...
INFO: Build complete: container.sif
Sorry I'm slow on the tests, as @dtrudg can attest it's my weak point!
from singularity.
We've shuffled this issue around on & off milestones now, because we have a few concerns about how complete and consistent this type of feature could be, if added to Singularity at this time.
A couple of thoughts:
-
Many people we've spoken to requesting better support for containers that come from Dockerfiles have been clear that they mainly work in OCI land, and then pull OCI images to SIF to use Singularity as a runtime.
podman
,buildah
, etc. are valid options for many sites at this point - to build from Dockerfiles, faithfully. Therefore for the Singularity runtime we've concentrated more on the idea of runtime compatibility. -
Singularity hasn't supported, and still doesn't support, (at runtime) everything that can be expressed in a container built from a
Dockerfile
. We've been working to improve OCI compatibility with each release, and we have a big push coming up to integrate a low-level OCI runtime that can provide a big jump in that regard (https://sylabs.staging.sycloud.io/2022/02/singularityce-4-0-and-beyond/). The general theme of the roadmap is for continuous incremental OCI compatibility improvements, so there is a bit of concern about bringing in new features that add a new angle to OCI compatibility considerations,, but with significant caveats in what is faithfully supported. Essentially we don't want to add to the list of 'this OCI thing works in Singularity, except a, b, c don't` that users have to consider. We only want to remove the limitations.
I think we'd like to see a clear path to fully faithful Dockerfile builds, if we are going to add functionality and promote Singularity as a Dockerfile
builder. So we need to work through:
- What things in a Dockerfile can / cannot be supported?
- (a) at build time
- (b) at run time (like USER)
- How will this fit in with the move toward using runc, and gradually away from the singularity native runtime (note we are not replacing it in 4.0 / removing functionality) - which is the big direction on the roadmap through 4.0 (we don't want to do a lot of work that is specific to the existing runtime)?
- How would this impact what works with remote builds vs local builds?
- Does any of the work have implications around support for older distros, such as RHEL7?
Note - none of this precludes exploring approaches. I just want to be up-front that fitting this feature into the Singularity roadmap is more complex than it might appear at first.
from singularity.
Going to add our notes from chat here for keeping! My plan is to give this a first pass so we can then have deeper discussion (and it's really no worries if we don't use what I ultimately make, this is going to be fun!)
- I would parse the Dockerfile sections into their corresponding Singularity sections, and return that "same" set of sections to build. I've done it in Python so I have a sense of how to go about it, so the only part of buildkit I'd want is the nice parser (to give me original pieces).
- at build time we'd start with FROM RUN COPY/ADD and CMD, and issue warnings for others for the time being. We cannot support USER in singularity until there is actual support (and I saw an issue about runc and that I think?)
- I don't know enough about what "singularity native runtime" means, but if I can parse the same content into the singularity stages arguably it could be compatible?
- No idea about RHEL7! I used to have a RHEL7 machine but it zonked with an update and I'm sending it back to the lab.
For this first pass it might just be for thinking / testing, and then either we can think through a plan that works (and for future singularity changes) or it could be something akin to a plugin. Either way I decided to work on it because (in my head at least) it seems straight forward and fun to do, even if it can't be added 🙂
from singularity.
Hi @vsoch.
In the interest of being open about other Sylabs plans around Dockerfile builds, we are also planning to add the ability to cloud.sylabs.io / Singularity Enterprise, for users to submit Dockerfiles. They would be built in the service using an OCI builder backend, not Singularity, and both OCI and SIF output would be produced.
This obviously doesn't block exploring building Dockerfiles in Singularity itself. We just want to be up-front about another angle where we'll need to consider consistency, user-experience, etc.
from singularity.
Noting here that the title of this issue has been edited so that it is clear it applies to building from a Dockerfile, using Singularity's native runtime, to a Singularity SIF file that is not an OCI-SIF image.
A separate issue #2218 now exists concerned with the addition of an OCI-mode build that uses OCI tooling to build from a Dockerfile into an OCI-SIF.
from singularity.
Related Issues (20)
- Pip install uses too much disk space - error with /tmp as tmpfs HOT 2
- Unknown image format/type in nextflow pipeline HOT 5
- Drop remaining direct containers/common usage
- CNI dhcp plugin does not work - netns bind issues
- oci-sif: inefficent copies of oci (layout) and tarball images
- delete sandbox
- allow user ns in singularity.conf
- allow ipc ns in singularity.conf
- e2e: OCI CustomHomePreservesRootShell failure
- Guidance on Choosing Between Apptainer and SingularityCE for Computational Materials Science Applications.
- Error when installing from Ubuntu package HOT 1
- Review / Fix dnf/yum/zypper ConveyorPacker unit tests
- `--data` convenience flag for --bind with image-src=/
- pkg/ocibundle/ocisif test dependent on squashfuse, but not gated HOT 1
- pkg/image TestSIFInitializer test failure
- OCI ConveyorPacker test failures
- Library ConveyorPacker Test Failure HOT 1
- SIFAssembler test failure
- tmpfs error using singularity with PID namespace on RHEL 8.8 HOT 5
- Does not build an .img file on Vagrant 2.4.1 on Mac(Sonoma 14.5 (23F79)) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from singularity.