Coder Social home page Coder Social logo

bestouff / genext2fs Goto Github PK

View Code? Open in Web Editor NEW
51.0 5.0 30.0 715 KB

genext2fs - ext2 filesystem generator for embedded systems

License: GNU General Public License v2.0

Makefile 0.23% Shell 6.97% C 83.34% M4 4.53% Roff 4.93%
ext2 embedded-linux mkfs

genext2fs's Introduction

GENEXT2FS

genext2fs - ext2 filesystem generator for embedded systems

SYNOPSIS

genext2fs [ options ] [ output-image ]

DESCRIPTION

genext2fs generates an ext2 filesystem as a normal (non-root) user. It does not require you to mount the image file to copy files on it, nor does it require that you become the superuser to make device nodes.

The filesystem image is created in the file output-image. If not specified, it is sent to stdout. The -d and -a options support reading from stdin if a single hyphen is given as an argument. Thus, genext2fs can be used as part of a pipeline without any temporary files.

By default, the maximum number of inodes in the filesystem is the minimum number required to accommodate the initial contents. In this way, a minimal filesystem (typically read-only) can be created with minimal free inodes. If required, free inodes can be added by passing the relevant options. The filesystem image size in blocks can be minimised by trial and error.

OPTIONS

-x, --starting-image image

Use this image as a starting point.

-d, --root directory[:path]

Add the given directory and contents at a particular path (by default the root).

-D, --devtable spec-file[:path]

Use spec-file to specify inodes to be added, at the given path (by default the root), including files, directories and special files like devices. If the specified files are already present in the image, their ownership and permission modes will be adjusted accordingly (this can only occur when the -D option appears after the options that create the specified files). Furthermore, you can use a single table entry to create many devices with a range of minor numbers (see examples below). All specified inodes receive the mtime of spec-file itself.

-a, --tarball file[:path]

Add the given archive (tarball) contents at a particular path (by default the root). If file is a hyphen, then the tarball will be read from standard input. Note: if not compiled with libarchive, genext2fs will use a builtin tarball parser with very primitive capabilities (e.g. no sparse file support, generally no support other than for modern GNU tar without fancy options).

-b, --size-in-blocks blocks

Size of the image in blocks.

-B, --block-size bytes

Size of a filesystem block in bytes.

-N, --number-of-inodes inodes

Minimum number of inodes. The required inode number will be computed automatically for all input that is not read from stdin. The number given by this option sets the minimum number of inodes. If you add anything from standard input, you should set this value because in that case the required number of inodes cannot be precomputed. The value set by this option will be overwritten by the value computed from the -i option, if the resulting number of inodes is larger.

-L, --volume-label name

Set the volume label for the filesystem.

-i, --bytes-per-inode ratio

Used to calculate the minimum number of inodes from the available blocks. Inodes are computed by multiplying the number of blocks (-b) by the blocksize (1024) and dividing that by the ratio given in this option. If the result is larger, then the number of required inodes counted from the input or the minimum number of inodes from the -N option, then the value computed by this option is used.

-m, --reserved-percentage N

Number of reserved blocks as a percentage of size. Reserving 0 blocks will prevent creation of the lost+found directory.

-o, --creator-os name

Value for creator OS field in superblock.

-g, --block-map path

Generate a block map file for this path.

-e, --fill-value value

Fill unallocated blocks with value.

-z, --allow-holes

Make files with holes.

-f, --faketime

Use a timestamp of 0 for inode and filesystem creation, instead of the present. Useful for testing.

-q, --squash

Squash permissions and owners (same as -P -U).

-U, --squash-uids

Squash ownership of inodes added using the -d option, making them all owned by root:root.

-P, --squash-perms

Squash permissions of inodes added using the -d option. Analogous to umask 077.

-v, --verbose

Print resulting filesystem structure.

-V, --version

Print genext2fs version.

-h, --help

Display help.

EXAMPLES

genext2fs -b 1440 -d src /dev/fd0

All files in the src directory will be written to /dev/fd0 as a new ext2 filesystem image. You can then mount the floppy as usual.

genext2fs -b 1024 -d src -D devicetable.txt flashdisk.img

This example builds a filesystem from all the files in src, then device nodes are created based on the contents of the file devicetable.txt. Entries in the device table take the form of:

          <name> <type> <mode> <uid> <gid> <major> <minor> <start> <inc> <count>

where name is the file name and type can be one of:

          f    A regular file
          d    Directory
          c    Character special device file
          b    Block special device file
          p    Fifo (named pipe)

uid is the user id for the target file, gid is the group id for the target file. The rest of the entries (major, minor, etc) apply only to device special files.

An example device file follows:

          # name    type mode uid gid major minor start inc count

          /dev      d    755  0    0    -    -    -    -    -
          /dev/mem  c    640  0    0    1    1    0    0    -
          /dev/tty  c    666  0    0    5    0    0    0    -
          /dev/tty  c    666  0    0    4    0    0    1    6
          /dev/loop b    640  0    0    7    0    0    1    2
          /dev/hda  b    640  0    0    3    0    0    0    -
          /dev/hda  b    640  0    0    3    1    1    1    16
          /dev/log  s    666  0    0    -    -    -    -    -

This device table creates the /dev directory, a character device node /dev/mem (major 1, minor 1), and also creates /dev/tty, /dev/tty[0-5], /dev/loop[0-1], /dev/hda, /dev/hda1 to /dev/hda15 and /dev/log socket.

genext2fs's People

Contributors

bestouff avatar josch avatar vapier avatar xoneca avatar yefuwang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

genext2fs's Issues

Could you please tag your release commits?

Hello!

Could you please push tags for the 1.4.1 and the recent 1.4.2 commits? That would allow distributions packaging your work to easily pick up when a new version is available.

Thanks!

cheers, josch

Support for symbolic links in the device table

They don't exactly fit in the format current table format. But I need it to set their uid / gid (the link can be created in the source directory even if user is unprivileged, but ids can not).

I couldn't find a workaround: tried creating the link in the source directory, and adding its path as a regular file / directory /... in the device table file, but genext2fs complains (legitimately) that node types don't match.

How to change owner of root inode

We use genext2fs in a Buildroot (i.e. genimage) environment. We need to change the owner of the root (/) node in order to control the owner of the mountpoint when the created ext2 partition is mounted in the target embedded system.
The ownership of the root directory tree passed via -d is ignored.
So I tried with -D, but a devtable containing something like

/ d 755 1000 1000 - - - - -

is unfortunately rejected with

genext2fs: bad name '/' (contains a slash)

As creation of the ext2 image is part of a (user) build process, we cannot temporarily mount the filesystem to do the required chmod at the mountpoint level, as this would require root permissions. All we can use is a fakeroot environment.

Can you please tell us how to change the root node owner, or could you maybe point us in another direction (maybe some idea for post-processing; unfortunately there is no e2chmod in e2tools - and e2mkdir does nothing for existing directories).

Thank you very much!

superblocks not shadow copied

According to the ext2 specification, superblocks and group descriptors are supposed to be copied per block group (or only in certain block groups if the sparse superblock feature is enabled). However, I don't see any copies being made of the superblock at all in this tool. Since it doesn't add in the sparse superblock feature flag it should be making copies for every block group.

The -d, -D and -a options do not allow paths containing a colon

Currently, the argument to -d, -D and -a is split by the first colon. This means that the path to the directory, specfile or tarball must not have a colon in it. This limitation should either be fixed or at least documented somewhere.

For fixing this I thought about replacing the call to strchr by strrchr but that will just shift the problem to the path inside the ext2 filesystem.

Another problem is, that strchr and strrchr operate on bytes and not on characters. This means, that all unicode characters containing 0x3A are also not supported in input paths -- that's a lot of characters.

One way to fix it would be to allow escaping the colon but then parsing becomes hard.

Another way would be to drop the colon syntax in favour of new command line arguments or by allowing the -d, -D and -a options to receive two arguments, so that the shell takes care of the splitting for us.

no support for hardlinks in tarballs

steps to reproduce:

$ mkdir test
$ echo foo > test/foo
$ ln test/foo test/bar
$ tar -C test -cf test.tar .
$ tar tvf test.tar
drwxr-xr-x josch/josch       0 2020-06-24 15:37 ./
-rw-r--r-- josch/josch       4 2020-06-24 15:37 ./bar
hrw-r--r-- josch/josch       0 2020-06-24 15:37 ./foo link to ./bar
$ ./genext2fs -B 1024 -N 17 -b 200 -a test.tar -f -o Linux -q test.img
$ sudo mount test.img /mnt
$ ls -lha /mnt
ls: cannot access '/mnt/foo': Structure needs cleaning
total 23K
drwxr-xr-x  4 root root 1.0K Jan  1  1970 .
drwxr-xr-x 25 root root 4.0K Jun 12 16:28 ..
-rw-------  1 root root    4 Jun 24 15:37 bar
??????????  ? ?    ?       ?            ? foo
drwx------  2 root root  16K Jan  1  1970 lost+found
$ head /mnt/foo                                                                                                                                                                /tmp/genext2fs
head: cannot open '/mnt/foo' for reading: Structure needs cleaning

If hardlinks from tarballs will not be supported or until they are supported, genext2fs should at least throw an error when the tarball contains them.

Support for converting tarballs to ext2

Hi,

would you accept a pull request allowing genext2fs to read a tarball (for example by using libarchive) and turn it into an ext2 image?

Even non-root users can easily create, store and manipulate tarballs containing files, directories and device nodes with arbitrary ownership, permissions, timestamps and other attributes. The interface could be as simple as:

$ genext2fs < rootfs.tar > rootfs.img

What do you think?

Make it clearer -d/-D args are applied as layers

Genext2fs applies its inputs (-d and -D arguments) as layers. As seen in #15 this may not be obvious. So, make it obvious:

  • state it clearly in the manpage, maybe in the helptext
  • display each stage as it happens

Issue with symlinks where target is exactly 60 characters

When a symlink target path is exactly 60 characters long, the final character is being truncated in the generated image.

For example, here is a symlink in my source directory:

/tmp/genimage/root/etc/ssl/certs/ca-cert-QuoVadis_Root_CA_3_G3.pem -> /usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_3_G3.crt

Looking at the resulting image (mounted at /mnt), the final "t" is missing:

/mnt/etc/ssl/certs/ca-cert-QuoVadis_Root_CA_1_G3.pem -> /usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_1_G3.cr

wrong filename if tarball entry is exactly 100 characters long

steps to reproduce:

$ mkdir test
$ touch test/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01
$ tar -C test -cf test.tar .
$ ./genext2fs -B 1024 -N 17 -b 200 -a - -f -o Linux test.img < test.tar
$ sudo mount test.img /mnt
$ ls -lha /mnt
drwxr-xr-x  3 root  root  1.0K Jan  1  1970 .
drwxr-xr-x 25 root  root  4.0K Jun 12 16:28 ..
-rw-r--r--  1 josch josch    0 Jun 25 13:57 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef010000644
drwx------  2 root  root   16K Jan  1  1970 lost+found

As you can see, the filename has "0000644" appended at the end.

Please support reading tarballs from stdin.

Thanks for accepting the libarchive pull request!

Next I'd like to be able to read tarballs from standard input. I wanted to use this issue to discuss how the interface for that should best look like. The major difference is, that when the data comes from stdin, it cannot be read twice to automatically determine how many inodes to allocate. In principle, the caller can do their own inode computations and then call genext2fs with the right -N argument. But maybe it makes sense for the caller to only supply the size -b and have the number of inodes be chosen by a heuristic if the input comes from stdin? After all, when creating an ext2 filesystem, I also only specify its size and I don't have to specify the number of inodes.

What do you think?

Fix autotools warnings/errors

Running autoreconf -i produces the following errors (and a few warnings):

aclocal: warning: autoconf input should be named 'configure.ac', not 'configure.in'
automake: warning: autoconf input should be named 'configure.ac', not 'configure.in'
configure.in:10: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated.  For more info, see:
configure.in:10: https://www.gnu.org/software/automake/manual/automake.html#Modernize-AM_005fINIT_005fAUTOMAKE-invocation
configure.in:13: installing './compile'
configure.in:10: installing './install-sh'
configure.in:10: installing './missing'
Makefile.am: error: required file './README' not found
Makefile.am: error: required file './ChangeLog' not found
Makefile.am: installing './depcomp'
automake: warning: autoconf input should be named 'configure.ac', not 'configure.in'
parallel-tests: installing './test-driver'
autoreconf: automake failed with exit status: 1

genext2fs is painfully slow for multi-GB input

Hi,

I'm now using genext2fs with multi-GB tarballs as input. While this works well it also takes several hours on my machine. So I profiled genext2fs:
gprof.txt

If I read the profiling output correctly, then most time is spent in the function allocate().

Do you have any ideas how to improve the speed by introducing better data structures?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.