jackyf / cupt Goto Github PK

View Code? Open in Web Editor NEW

36.0 4.0 4.0 10.97 MB

high-level package manager for Debian, inspired from APT

CMake 0.71% C++ 75.25% Shell 0.13% Perl 13.92% Perl 6 9.97% Terra 0.02%

cupt's People

Contributors

Stargazers

Watchers

Forkers

ryan52 santex cvalka4 jelmer

cupt's Issues

Incomplete support for APT config list variables

When using apt in a docker container I noticed several error messages.

root@e4a9be148f73:/# cupt update
E: syntax error: line 11, character 33: expected: option value (quoted string) or opening curly bracket ('{')
E: unable to parse the config file '//etc/apt/apt.conf.d/docker-gzip-indexes'
W: skipped the configuration file '//etc/apt/apt.conf.d/docker-gzip-indexes'

originating from an APT config line

Acquire::CompressionTypes::Order:: "gz";

I first suspected it was a faulty APT config file and filed an issue in moby. It turned out the file has legal syntax but cupt seems to lack support for appending elements to list variables by appending :: to the variable name.

From https://manpages.debian.org/buster/apt/apt.conf.5.en.html

To append a new element to a list, add a trailing :: to the name of the list. (As you might suspect, the scope syntax can't be used on the command line.)

Note that appending items to a list using :: only works for one item per line, and that you should not use it in combination with the scope syntax (which adds :: implicitly). Using both syntaxes together will trigger a bug which some users unfortunately depend on: an option with the unusual name "::" which acts like every other option with a name. This introduces many problems; for one thing, users who write multiple lines in this wrong syntax in the hope of appending to a list will achieve the opposite, as only the last assignment for this option "::" will be used. Future versions of APT will raise errors and stop working if they encounter this misuse, so please correct such statements now while APT doesn't explicitly complain about them.

несколько вопросов...

приветствую!

скажите, где располагается документация о том, как собрать/установить этот менеджер пакетов?
жестко ли код "прибит" к линукс? есть ли надежда портировать его на виндоус?
откуда этот менеджер пакетов получает "знания" о репозитарии?

спасибо.

зы
почитал немного код и заметил, что используется boost::lexical_cast<std::string>(integral type). код написан с использованием C++11, а в C++11 есть же std::to_string(integral type)

cupt throws errors when APT config. files have comments in them

Whenever I use cupt to install a package with it, it gives this error:

E: syntax error: line 1, character 1: expected: clear directive ('#clear') or option name (letters, numbers, slashes, points, dashes, double colons allowed)
E: unable to parse the config file '//etc/apt/apt.conf.d/50apt-file.conf'
W: skipped the configuration file '//etc/apt/apt.conf.d/50apt-file.conf'

After some troubleshooting, I found that unless I removed every comment from that configuration file, it would always display those messages. Is there a particular reason why cupt isn't merely ignoring these comments on those files?.

Use CPack to build the binary deb package instead of debhelper

Implement signed-by

cupt fails to update repositories

I'm trying out cupt in my Ubuntu system, but I've encountered a problem. When I execute the command sudo cupt update it fails with the following output:

W: failed to download (In)Release for 'http://archive.ubuntu.com/ubuntu bionic-updates/'
E: no download handlers available for the protocol 'http'
#11: starting http://archive.ubuntu.com/ubuntu bionic Release                                                                                                                                  
0% [#11 bionic Release 0B]                                                                                                                                                     | 0B/s | ETA: 0sW: failed to download (In)Release for 'http://archive.ubuntu.com/ubuntu bionic/'
E: no download handlers available for the protocol 'https'
#11: not available                                                                                                                                                                             
0%  
Fetched 0B in 0s.                                                                                                                                                                              
E: there were errors while downloading release and index data
E: error performing the command 'update'

I can't seem to find what's the issue.

Update: I was missing libcupt4-2-downloadmethod-wget or libcupt4-2-downloadmethod-curl

Download Manager can deadlock if a signal is received at the wrong time

During the package download part of a safe-upgrade action, cupt emitted the following error and then just hung:

22% [15 libclang1 394KiB/4605KiB 9%]                      | 1432KiB/s | ETA: 17s
E: download client: the download server socket timed out

Attaching GDB to the parent process of the now zombied downloader shows the following backtrace:

(gdb) bt
#0  0xb76e9430 in __kernel_vsyscall ()
#1  0xb72a6511 in __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:97
#2  0xb72258fb in _L_lock_10900 () at malloc.c:5104
#3  0xb72233c6 in __GI___libc_malloc (bytes=17) at malloc.c:2856
#4  0xb7407395 in operator new(unsigned int) () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#5  0xb7469565 in std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#6  0xb746b4d6 in char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) ()
#7  0xb746bb60 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) ()
#8  0xb76324fa in cupt::internal::sendPingMessage () at /home/mccoyj1/src/cupt/cpp/lib/src/download/manager.cpp:330
#9  <signal handler called>
#10 _int_malloc (av=av@entry=0xb7355440 <main_arena>, bytes=bytes@entry=16) at malloc.c:3424
#11 0xb72233d3 in __GI___libc_malloc (bytes=16) at malloc.c:2859
#12 0xb7407395 in operator new(unsigned int) () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#13 0xb763ba61 in allocate (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/4.6/ext/new_allocator.h:92
#14 _M_allocate (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/4.6/bits/stl_vector.h:150
#15 std::vector<pollfd, std::allocator<pollfd> >::_M_insert_aux<pollfd>(__gnu_cxx::__normal_iterator<pollfd*, std::vector<pollfd, std::allocator<pollfd> > >,, pollfd&&) (this=0xbf890d90, __position=...) at /usr/include/c++/4.6/bits/vector.tcc:327
#16 0xb7632bbc in emplace_back<pollfd> (this=<optimized out>) at /usr/include/c++/4.6/bits/vector.tcc:102
#17 push_back (__x=..., this=<optimized out>) at /usr/include/c++/4.6/bits/stl_vector.h:840
#18 push_back (__x=..., this=<optimized out>) at /home/mccoyj1/src/cupt/cpp/lib/src/download/manager.cpp:673
#19 cupt::internal::ManagerImpl::pollAllInput (this=0x9a86a80, workerQueue=..., persistentSockets=..., clientSockets=..., exitFlag=false, debugging=false) at /home/mccoyj1/src/cupt/cpp/lib/src/download/manager.cpp:694
#20 0xb76363de in cupt::internal::ManagerImpl::worker (this=0x9a86a80) at /home/mccoyj1/src/cupt/cpp/lib/src/download/manager.cpp:797
#21 0xb7637fc3 in cupt::internal::ManagerImpl::ManagerImpl (this=0x9a86a80, config_=..., progress_=...) at /home/mccoyj1/src/cupt/cpp/lib/src/download/manager.cpp:249
#22 0xb76380dc in cupt::download::Manager::Manager (this=0xbf891140, config=..., progress=...) at /home/mccoyj1/src/cupt/cpp/lib/src/download/manager.cpp:1216
#23 0xb75d4af8 in cupt::internal::PackagesWorker::__do_downloads (this=0x9793904, downloads=..., downloadProgress=...) at /home/mccoyj1/src/cupt/cpp/lib/src/internal/worker/packages.cpp:2000
#24 0xb75db588 in cupt::internal::PackagesWorker::changeSystem (this=0x9793904, downloadProgress=...) at /home/mccoyj1/src/cupt/cpp/lib/src/internal/worker/packages.cpp:2121
#25 0xb762fc09 in cupt::system::Worker::changeSystem (this=0x98dc6b0, progress=...) at /home/mccoyj1/src/cupt/cpp/lib/src/system/worker.cpp:89
#26 0x080f1ca9 in managePackages (context=..., mode=ManagePackages::Install) at /home/mccoyj1/src/cupt/cpp/console/handlers/managepackages.cpp:1321
#27 0x080a2eb3 in operator() (c=..., this=<optimized out>) at /home/mccoyj1/src/cupt/cpp/console/misc.cpp:254
#28 std::_Function_handler<int(Context&), getHandler(const string&)::<lambda(Context&)> >::_M_invoke(const std::_Any_data &, Context &) (__functor=..., __args#0=...) at /usr/include/c++/4.6/functional:1764
#29 0x0809ff55 in operator() (__args#0=..., this=<optimized out>) at /usr/include/c++/4.6/functional:2161
#30 mainEx (argc=3, argv=0xbf8916d4, context=..., command=...) at /home/mccoyj1/src/cupt/cpp/console/cupt.cpp:78
#31 0x0809f922 in main (argc=164371856, argv=0x9af6f10) at /home/mccoyj1/src/cupt/console/cupt.cpp:64

sources.list: non-space-surrounded braces for options not recognised

cupt gives an error message when a repository is specifying an architecture in its configuration file and results in cupt skipping the repository. e.g., this is the output from the Google Chrome repository.

E: no download handlers defined for the protocol '[arch=amd64]/dists/http'
E: no download handlers defined for the protocol '[arch=amd64]/dists/http'
W: failed to download (In)Release for '[arch=amd64] http://dl.google.com/linux/chrome/deb//'
W: failed to download (In)Release for '[arch=amd64] http://dl.google.com/linux/chrome/deb//'
E: there were errors while downloading release and index data
E: error performing the command 'update'

If I remove the line specifying the architecture cupt doesn't give any warning or error message.

Failing test `query/repo-signatures/validation-errors:"empty signature"` in a Debian stretch container

When trying to build the debian package in a debian stretch docker container two of the tests fail:

   #   Failed test 'validation failed'
    #   at tt/query/repo-signatures/validation-errors.t line 15.
    #                   'Package: p
    # Version: 1
    # Status: not installed
    # Source: p
    # Priority: extra
    # Uncompressed size: 0B
    # Architecture: all
    # URI: copy://./localrepo//
    # SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
    # 
    # '
    #           matches '(?^:Package)'
    
    #   Failed test 'error message is right'
    #   at tt/query/repo-signatures/validation-errors.t line 16.
    #                   'Package: p
    # Version: 1
    # Status: not installed
    # Source: p
    # Priority: extra
    # Uncompressed size: 0B
    # Architecture: all
    # URI: copy://./localrepo//
    # SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
    # 
    # '
    #     doesn't match '(?^m:empty signature)'
    # Looks like you failed 2 tests of 2.

#   Failed test 'empty signature'
#   at tt/query/repo-signatures/validation-errors.t line 18.
# Looks like you failed 1 test of 9.
[08:47:47] tt/query/repo-signatures/validation-errors.t ................................................... 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/9 subtests

This happens on master and the current release

Harden the binaries

Add the module below
import it in the highest level CMakeLists.txt
call harden against each target

harden.cmake

#This is free and unencumbered software released into the public domain.
#Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means.
#In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law.
#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#For more information, please refer to <https://unlicense.org/>

include(CheckCXXCompilerFlag)

function(determineSupportedHardeningFlags property)
	set(FLAGS_HARDENING "")
	foreach(flag ${ARGN})
		unset(var_name)
		string(REPLACE "=" "_eq_" var_name ${flag})
		string(REPLACE "," "_comma_" var_name ${var_name})
		set(var_name "SUPPORTS_HARDENING_${property}_${var_name}")
		check_cxx_compiler_flag(${flag} ${var_name})#since linker flags and other flags are in the form of compiler flags
		if(${${var_name}})
			list(APPEND FLAGS_HARDENING "${flag}")
		endif()
	endforeach(flag)
	list(JOIN FLAGS_HARDENING " " FLAGS_HARDENING)
	#message(STATUS "FLAGS_HARDENING ${FLAGS_HARDENING}")
	set(HARDENING_${property} "${FLAGS_HARDENING}" CACHE STRING "Hardening flags")
endfunction(determineSupportedHardeningFlags)

function(processFlagsList target property)
	get_target_property(FLAGS_UNHARDENED ${target} ${property})
	if(FLAGS_UNHARDENED MATCHES "FLAGS_UNHARDENED-NOTFOUND")
		set(FLAGS_UNHARDENED "")
	endif()
	#message(STATUS "processFlagsList ${target} ${property} ${FLAGS_UNHARDENED}")
	#message(STATUS "HARDENING_${property} ${HARDENING_${property}}")
	if(HARDENING_${property})
	else()
		determineSupportedHardeningFlags(${property} ${ARGN})
	endif()
	
	set(FLAGS_HARDENED ${FLAGS_UNHARDENED})
	list(APPEND FLAGS_HARDENED ${HARDENING_${property}})
	list(JOIN FLAGS_HARDENED " " FLAGS_HARDENED)
	#message(STATUS "${target} PROPERTIES ${property} ${FLAGS_HARDENED}")
	set_target_properties(${target} PROPERTIES ${property} "${FLAGS_HARDENED}")
endfunction(processFlagsList)

function(setupPIC target)
	set_property(TARGET ${target} PROPERTY POSITION_INDEPENDENT_CODE ON) # FUCK, doesn't work
	if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
		get_target_property(type ${target} TYPE)
		if(type STREQUAL "EXECUTABLE")
			list(APPEND HARDENING_COMPILER_FLAGS
				"-fPIE"
			)
		else()
			list(APPEND HARDENING_COMPILER_FLAGS
				"-fPIC"
			)
		endif()
		list(APPEND HARDENING_LINKER_FLAGS
			"-pie"
		)
	elseif(MSVC)
		list(APPEND HARDENING_COMPILER_FLAGS
			"/dynamicbase" "/HIGHENTROPYVA"
		)
	else()
		message(ERROR "The compiler is not supported")
	endif()
endfunction(setupPIC)

function(harden target)
	setupPIC("${target}")
	if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
		list(APPEND HARDENING_COMPILER_FLAGS
			"-Wall" "-Wextra" "-Wconversion" "-Wformat" "-Wformat-security" "-Werror=format-security"
			"-fno-strict-aliasing" "-fno-common"
			#"-fstack-check"
			#"-fcf-protection=full" # conflicts to "-mindirect-branch"
			"-fstack-clash-protection"
			"-mcet"
			"-fsanitize=cfi"
			"-fsanitize=cfi-cast-strict"
			"-fsanitize=cfi-derived-cast"
			"-fsanitize=cfi-unrelated-cast"
			"-fsanitize=cfi-nvcall"
			"-fsanitize=cfi-vcall"
			"-fsanitize=cfi-icall"
			"-fsanitize=cfi-mfcall"
			
			#"-fsanitize=safe-stack;compiler-rt" # https://clang.llvm.org/docs/SafeStack.html
			"-fsanitize=address" # https://clang.llvm.org/docs/AddressSanitizer.html
			
			# TODO implement compiler flag dependence on libs linking
			#"-fsanitize=undefined;ubsan" # https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html , gcc also has it
			
			#"-fsanitize=thread" # https://clang.llvm.org/docs/ThreadSanitizer.html , 15x slowdown and 10x memory overhead
			#"-fsanitize=memory" # https://clang.llvm.org/docs/MemorySanitizer.html 3x slowdown
			#"-fsanitize=dataflow" # https://clang.llvm.org/docs/DataFlowSanitizer.html, taint analysis, requires explicit annotation of code
			
			#"-fvtable-verify=std;vtv"
			
			# this conflicts with gcc which now has -fcf-protection=full hardcoded
			"-fcf-protection=none -mindirect-branch"
			"-fcf-protection=none -mindirect-branch=thunk-extern"
			"-fcf-protection=none -mindirect-branch=thunk-inline"
			"-fcf-protection=none -mindirect-return"
			"-fcf-protection=none -mindirect-branch-register"
			"-fcf-protection=none -mindirect-branch-loop"
			
			"-x86-speculative-load-hardening"
			"-mno-indirect-branch-register"
		)
		
		# some flags are bugged in GCC
		if(CMAKE_CXX_COMPILER_ID MATCHES "GNU")
		else()
			list(APPEND HARDENING_COMPILER_FLAGS
				"-ftrapv" # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=35412
			)
		endif()
		
		# GCC 9 has removed these flags
		if(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 9 AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 10))
			message(STATUS "GCC 9 removes some hardening flags but doesn't fail if they are present, instead shows deprecation message. In order to not to put garbage into warnings we don't insert them. See the code of Harden.cmake for the details.")
		else()
			list(APPEND HARDENING_COMPILER_FLAGS
				"-mmitigate-rop" 
				"-fcheck-pointer-bounds"
				"-fchkp-treat-zero-size-reloc-as-infinite"
				"-fchkp-first-field-has-own-bounds"
				"-fchkp-narrow-bounds"
				"-fchkp-narrow-to-innermost-array"
				"-fchkp-optimize"
				"-fchkp-use-fast-string-functions"
				"-fchkp-use-nochk-string-functions"
				"-fchkp-use-static-const-bounds"
			)
		endif()
		
		list(APPEND HARDENING_LINKER_FLAGS
			"-Wl,-O1"
			"-Wl,--sort-common"
			"-Wl,--as-needed"
			"-Wl,-flto"
		)
		if(CMAKE_SYSTEM_NAME MATCHES "Windows")
			list(APPEND HARDENING_LINKER_FLAGS
				"-Wl,--export-all-symbols"
				"-Wl,--nxcompat"
				"-Wl,--dynamicbase"
			)
			if(CMAKE_SIZEOF_VOID_P EQUAL 8)
			#	list(APPEND HARDENING_LINKER_FLAGS "-Wl,--image-base,0x140000000") # doesn't work for this project
			endif()
		elseif(CMAKE_SYSTEM_NAME MATCHES "Linux") # other using ELF too?
			list(APPEND HARDENING_COMPILER_FLAGS
				# on MinGW hello world works, but more complex things just exit without any output or crash in the middle of execution
				"-fstack-protector"
				"-fstack-protector-strong"
			)
			list(APPEND HARDENING_LINKER_FLAGS
				# not present in MinGW
				"-Wl,-z,relro"
				"-Wl,-z,now"
				"-Wl,-z,ibtplt"
				"-Wl,-z,ibt"
				"-Wl,-z,shstk"
			)
		endif()
		list(APPEND HARDENING_MACRODEFS
			"-D_FORTIFY_SOURCE=2"
			"-D_GLIBCXX_ASSERTIONS"
		)
	elseif(MSVC)
		set(HARDENING_COMPILER_FLAGS "/sdl" "/GS" "/SafeSEH" "/guard:cf" "/HIGHENTROPYVA")
		set(HARDENING_LINKER_FLAGS "/guard:cf")
	else()
		message(ERROR "The compiler is not supported")
	endif()

	processFlagsList(${target} COMPILE_FLAGS ${HARDENING_COMPILER_FLAGS})
	processFlagsList(${target} LINK_FLAGS ${HARDENING_LINKER_FLAGS})
	
	#list(JOIN HARDENING_MACRODEFS " " HARDENING_MACRODEFS) # unneeded, list is needed, not string
	set(HARDENING_MACRODEFS "${HARDENING_MACRODEFS}" CACHE STRING "Hardening flags CMake list (not string!)")
	target_compile_definitions(${target} PRIVATE ${HARDENING_MACRODEFS})
endfunction(harden)

Support apt transports

E: no download handlers defined for the protocol '[signed-by=/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg]/dists/https'
#3: starting [signed-by=/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg] https://mirrors.kernel.org/ubuntu/ InRelease

for the rules like

deb [signed-by=/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg] https://mirrors.kernel.org/ubuntu/ eoan universe