vathpela / createrepo_c

C implementation of the createrepo.

createrepo_c

C implementation of createrepo

Run createrepo -h for usage syntax.

Devel tips

Building

Package build requires - Pkg name in Fedora/Ubuntu:

• bzip2 (http://bzip.org/) - bzip2-devel/libbz2-dev
• cmake (http://www.cmake.org/) - cmake/cmake
• expat (http://expat.sourceforge.net/) - expat-devel/libexpat1-dev
• file (http://www.darwinsys.com/file/) - file-devel/libmagic-dev
• glib2 (http://developer.gnome.org/glib/) - glib2-devel/libglib2.0-dev
• libcurl (http://curl.haxx.se/libcurl/) - libcurl-devel/libcurl4-openssl-dev
• libxml2 (http://xmlsoft.org/) - libxml2-devel/libxml2-dev
• python (http://python.org/) - python2-devel/libpython2.7-dev
• rpm (http://www.rpm.org/) - rpm-devel/librpm-dev
• openssl (http://www.openssl.org/) - openssl-devel/libssl-dev
• sqlite3 (https://sqlite.org/) - sqlite-devel/libsqlite3-dev
• xz (http://tukaani.org/xz/) - xz-devel/liblzma-dev
• zlib (http://www.zlib.net/) - zlib-devel/zlib1g-dev
• Documentation: doxygen (http://doxygen.org/) - doxygen/doxygen
• Documentation: sphinx (http://sphinx-doc.org/) - python-sphinx/
• Test requires: check (http://check.sourceforge.net/) - check-devel/check
• Test requires: python-nose (https://nose.readthedocs.org/) - python-nose/python-nose
• Experimental support: drpm (https://git.fedorahosted.org/git/drpm.git)

From your checkout dir:

mkdir build
cd build/
cmake ..
make

To build the documentation, from the build/ directory: make doc

Note: For build with debugging symbols you could use (from the build/ directory):

cmake -DCMAKE_BUILD_TYPE:STRING=DEBUG .. && make

Building from an rpm checkout

E.g. when you want to try weak and rich dependencies.

cmake -DRPM_PATH="/home/tmlcoch/git/rpm" .. && make

Note: The RPM must be buit in that directory

Commands I am using for building of RPM:

cd /home/tmlcoch/git/rpm
CPPFLAGS='-I/usr/include/nss3/ -I/usr/include/nspr4/' ./autogen.sh --rpmconfigure --with-vendor=redhat --with-external-db --with-lua --with-selinux --with-cap --with-acl --enable-python
make clean && make

Building with delta rpm support (drpm)

At first, you have to checkout drpm library from the https://git.fedorahosted.org/git/drpm.git and build it.

git clone ssh://git.fedorahosted.org/git/drpm.git
cd drpm/
make

Then run cmake for createrepo_c with param -DDRPM_PATH="/home/tmlcoch/git/drpm" where the path is path to your build of drpm library.

cmake -DDRPM_PATH="/home/tmlcoch/git/drpm" .. && make

Building for a different Python version

By default, cmake should set up things to build for Python 2, but you can do a build for Python 3 like this::

cmake -DPYTHON_DESIRED=3 .

Other build options

-DENABLE_LEGACY_WEAKDEPS=ON

Enable old SUSE weaktags (Default: OFF)

-DENABLE_THREADED_XZ_ENCODER=ON

Threaded XZ encoding (Default: OFF)

Note: This option is disabled by default, because Createrepo_c parallelize a lot of tasks (including compression) by default, this only adds extra threads on XZ library level which causes thread bloat and for most usecases doesn't bring any performance boost. On regular hardware (e.g. less-or-equal 4 cores) this option may even cause degradation of performance.

Build tarball

utils/make_tarball.sh [git revision]

Without git revision specified HEAD is used.

Build RPM package

Modify createrepo_c.spec and run:

utils/make_rpm.sh

Note: Current .spec for Fedora rawhide

Testing

All unit tests run from librepo checkout dir

Build C tests && run c and python tests

make tests && make test

Note: For a verbose output of testing use: make ARGS="-V" test

Run only C unittests (from your checkout dir):

build/tests/run_gtester.sh

Note: The C tests have to be builded by make tests)!

Run only Python unittests (from your checkout dir):

PYTHONPATH=`readlink -f ./build/src/python/` nosetests -s tests/python/tests/

Or, for an alternative python version, specify the appropriate nosetests executable:

PYTHONPATH=`readlink -f ./build/src/python/` nosetests-3.4 -s tests/python/tests/

Links

Bugzilla

Important notes

In original createrepo sha is a nickname for the sha1 checksum. Createrepo_c mimics this behaviour.

---

Differences in behavior between createrepo_c and createrepo

Checksums after update

Use case:

• Repodata in repo/ are has checksum xxx
• Params: --update --checksum=yyy repo/

createrepo_c result:

• All package checksums are recalculated into yyy

original createrepo result:

• Only new and changed packages has yyy checksums other packages has still xxx checksums

Skip symlinks param

Use case:

• Some packages in repo/ are symlinks
• Params: --skip-symlinks repo/

createrepo_c result:

• Symlinked packages are ignored

original createrepo result:

• Symlinked packages are processed (https://bugzilla.redhat.com/show_bug.cgi?id=828848)

Base path from update-md-path repo

Use case:

• A somebody else's repo is somewhere
• The repo items have set a base path to http://foo.com/
• We want to create metadata for our repo
• Some packages in our repo are same as packages in somebody else's repo
• We want to speed up creation of our repodata with combo --update and --update-md-path=somebody_else's_repo
• Params: --update --update-md-path=ftp://somebody.else/repo our_repo/

createrepo_c results:

• All our packages have no base path set (if we don't set --baseurl explicitly)

original createrepo result:

• Some packages in metadata (which was same in our repo and in somebody else's repo) have base path set to http://foo.com/
• (https://bugzilla.redhat.com/show_bug.cgi?id=875029)

Crippled paths in filelists.xml after update

Use case:

• A repo with old metadata exists
• We want to update metadata
• Params: --update repo/

createrepo_c results:

• All is fine

original createrepo result:

• Some paths in filelists.xml are crippled (https://bugzilla.redhat.com/show_bug.cgi?id=835565)

--update leaves behind some old repodata files

Use case:

• A repo with repodata created with --simple-md-filenames exists
• We want to update repodata to have checksums in filenames
• Params: --update repo/

createrepo_c results:

• All repodata contains checksum in the name

original createrepo result:

• All repodata contains checksum in the name
• There are old metadata without checksum in the name too
• (https://bugzilla.redhat.com/show_bug.cgi?id=836917)

Modifyrepo_c

Modifyrepo_c is compatible with classical Modifyrepo except some misbehaviour:

• TODO: Report bugs and add reference here

Batch file

When there is need to do several modification to repository (repomd.xml) a batch file could be used.

Batch file is Modifyrepo_c specific. It is not supported by the classical Modifyrepo - at least not yet.

Example

# Add:
#   [<path/to/file>]
#   <options>

# Metadata that use a bunch of config options
[some/path/comps.xml]
type=group
compress=true
compress-type=gz
unique-md-filenames=true
checksum=sha256
new-name=group.xml

# Metadata that use default settings
[some/path/bar.xml]

# Remove:
#   [<metadata name>]
#   remove=true

[updateinfo]
remove=true

Supported options

Option name	Description	Supported value(s)	Default
path	Path to the file. When specified it override the path specified in group name (name between [] parenthesis)	Any string	group name (string between '[' ']')
type	Type of the metadata	Any string	Based on filename
remove	Remove specified file/type from repodata	true or false	false
compress	Compress the new metadata before adding it to repo	true or false	true
compress-type	Compression format to use	gz, bz2, xz	gz
checksum	Checksum type to use	md5, sha, sha1, sha224, sha256, sha384, sha512	sha256
unique-md-filenames	Include the file's checksum in the filename	true or false	true
new-name	New name for the file. If compress is true, then compression suffix will be appended. If unique-md-filenames is true, then checksum will be prepended.	Any string	Original source filename

Notes

• Lines beginning with a '#' and blank lines are considered comments.
• If remove=true is used, no other config options should be used