ciphey / cipheycore

Some cryptanalysis tidbits written in a *proper* language

CipheyCore

Some cryptanalysis tidbits written in a proper language

This core provides many crackers, encryption methods & decryption methods as well as cryptanalysis tidbitis to Ciphey. Written in C++ for the speed.

Installation

CipheyCore can be installed via MacPorts:

sudo port install py-cipheycore

By default, this installs CipheyCore for python 3.9. To install it for Python 3.8, replace py-cipheycore with py38-cipheycore.

Dependencies

CipheyCore builds as a static library, but depends on some of Boost's header-only libraries.

Platform	Command
Debian/Ubuntu/Mint	apt-get install libboost-dev cmake build-essential swig
Arch	pacman -S boost gcc cmake make swig
CentOS/RHEL	yum install boost-devel cmake3 llvm-toolchain-7 make swig
Homebrew (macOS)	brew install boost swig
MacPorts (macOS)	sudo port install cmake boost poetry swig-python swig
Chocolatey (Windows)	choco install swig and install boost from boost.org

⚠️ Note: This project requires gcc>=8. Depending on which platform you are on, you might experience that the default gcc is a lower version. You can check your gcc version by running gcc -v.

Building

CipheyCore can be used as a C++ library for use in other C++ projects, or as a python3 module (with reduced functionality and speed).

The python module depends on the C++ library, so follow both instructions if you want the python module.

C++ library

Linux/OSX/other Unices

git clone https://github.com/Ciphey/CipheyCore
cd CipheyCore
rm -rf build
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCIPHEY_CORE_TEST=OFF # Or '-DCMAKE_BUILD_TYPE=Debug' for a debug build
cmake --build . -t ciphey_core

If that doesn't work, try setting -DCIPHEY_CORE_PYTHON= to the folder containing your python headers and -DBOOST_ROOT= to the directory containing your boost headers.

Windows

git clone https://github.com/Ciphey/CipheyCore
cd CipheyCore
rd /S build
cd build
cmake .. -DCIPHEY_CORE_TEST=OFF
cmake --build . -t ciphey_core --config Release # Or '--config Debug' for a debug build
poetry build

As with the Unix compilation, if you are having problems with missing files, try setting -DCIPHEY_CORE_PYTHON and BOOST_ROOT.

Python3 library

This requires python-poetry, which can be installed with python3 -m pip install poetry (or, failing that, python -m pip install poetry).

After you have built the C++ library, compile the python interface with

cmake --build . -t ciphey_core_py --config Release # Or '--config Debug' for a debug build
poetry build

The wheel will be located in the dist folder. The source distribution (.tar.gz file) produced is rather useless, as it contains the compiled platform-specific library.

cipheycore's Issues

Thoughts on packaging CipheyCore

Following from Ciphey/Ciphey#583, I thought I'd give some thoughts for other package maintainers on how to package Ciphey.

For future readers that might want to do this, I'll document some issues I faced below and how I fixed them. If you need any additional help, I recommend reading any MacPorts PRs that I link for further details on each section (especially if you're packaging for macOS) and feel free to ping me.

TL;DR: CipheyCore can be installed by running the following through MacPorts: (project page)

sudo port install py-cipheycore

(Replace it with py38-cipheycore for python 3.8)

Assuming all goes well, this should install it for your system with no segmentation faults (pre-compiled binaries are also available for some macOS versions). Maintainers might find it useful to document this.

---

For a project as large as Ciphey, it's on surprisingly few package managers. In my opinion, it all comes down to CipheyCore. If there was no CipheyCore, it would be a piece of cake to package Ciphey. To quote @Cyclic3 from Ciphey/Ciphey#197 (comment):

The problem is that the project is built in two stages, with the C++ cmake build creating an autogenerated python module (using SWIG), and then a thin poetry wrapper is used to build the wheel

Although this is all very well documented, it’s incredibly difficult to package. Since it’s not possible for there to be a standard distribution (as it contains a platform-specific library), the method used 99% of the time to sort out python dependencies on loads of package managers is thrown out the window.

Therefore, you will have to write a PKGBUILD/Formula/Portfile/etc. for CipheyCore. It is not enough to set it as an external dependency (similar to what Homebrew does for python libraries).

---

How to build CipheyCore

Fortunately, it is very well documented in the README.

Just a few things to note from a package maintainer's perspective. When building the python3 library, unlike the ciphey maintainers, we're only interested in building the source distribution. This is since it will then contain a library that is built for our platform, and we can then deal with it similar to how your package manager would normally deal with python libraries.

For example, MacPorts' pre-built binaries are built for specific macOS versions, so there's no issue with using the platform-specific library. The same is true when MacPorts builds from source.

How to deal with segmentation faults

I ran into this issue macports/macports-ports#9964, but the fix appears simple.

Even if CipheyCore builds successfully, set the -DCIPHEY_CORE_PYTHON flag to the folder containing the headers

e.g. /opt/local/Library/Frameworks/Python.framework/Versions/3.8

The problem is described very well in Ciphey/Ciphey#238 (comment), but for package maintainers it's likely due to the wrong python version being used to configure CipheyCore.

e.g. If python 3.9 is used to build CipheyCore for python 3.8, this will cause a segmentation fault at runtime.

Getting CipheyCore to build if it's not working

As a package maintainer, you have to think about different versions of your operating system. Just because it builds for your version doesn't mean it will work for all versions of your OS (Trust me I learnt this the hard way macports/macports-ports#9932).

The key is C++20 compiler support. Compilers such as AppleClang 10+, Clang 9+, etc. should hopefully be fine.

Over at MacPorts, we manually specified which compilers worked and which ones don't (and possible fallback compilers). You might have to do something similar in case a user does not have a supported compiler installed.

TypeError at runtime

This one's easy. If you're packaging CipheyCore for Ciphey, make sure to set the default python version to 3.8.

If you're packaging CipheyCore just by itself, you can choose either 3.8 or 3.9.

---

As I said before, I'm more than happy to help anyone who is trying to package Ciphey/CipheyCore. In case you're interested (or stuck), the final Portfile for CipheyCore can be found here. You might find the comments in the Portfile helpful. I really recommend reading the MacPorts PRs linked if you're lost on a particular section, and let me know if you have any questions.

All the best,
Haren

Generating python package

1. the patch like Ciphey/Ciphey@2315cfa seems to be needed to be applied
2. what's the purpose of build.py? Should it be removed? It interferes with python3 -m build and without it the wheel builds fine.

CipheyCore is set for archival

We are sunsetting CipheyCore for a full Python program. Please do not contribute to this repo :)

Dependabot couldn't find a requirements.txt for this project

Dependabot couldn't find a requirements.txt for this project.

Dependabot requires a requirements.txt to evaluate your project's current Python dependencies. It had expected to find one at the path: /requirements.txt.

If this isn't a Python project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

What to do next after making and installing?

Hi so I followed the steps and I'm assuming you get pip to add the wheel but what after that do you need Ciphey to add the CipheyCore module after that or does it have automated access once its added to the wheel cash?

Version check problem on Pypi

pip install --force cipheycore produces:

Installing collected packages: cipheycore
  Attempting uninstall: cipheycore
    Found existing installation: cipheycore 0.1.0
    Uninstalling cipheycore-0.1.0:
      Successfully uninstalled cipheycore-0.1.0
Successfully installed cipheycore-0.1.0

So something is weird with the current 0.2.9 version of cipheycore on Pypi. Fix the release on Pypi please?

Installation fails using pip and poetry with:

ERROR: Could not find a version that satisfies the requirement cipheycore==0.2.9 (from versions: 0.1.0)
ERROR: No matching distribution found for cipheycore==0.2.9

actually do this with a rotation

CipheyCore/src/ciphers/xor_single.cpp

Lines 37 to 42 in 4056d16

	// TODO: actually do this with a rotation
	// xor_prob_table(observed, key ^ (key - 1));
	prob_table new_observed;
	for (auto& i : observed)
	new_observed[i.first^key] = i.second;
	if (key == 'c')

---

This issue was generated by todo based on a TODO comment in 4056d16. It's been assigned to @Cyclic3 because they committed the code.

Dependabot couldn't find a requirements.txt for this project

Dependabot couldn't find a requirements.txt for this project.

Dependabot requires a requirements.txt to evaluate your project's current Python dependencies. It had expected to find one at the path: /requirements.txt.

If this isn't a Python project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

Add CMake install target

I understand that make install is not in the build instructions, and that install is not a required target, but I think it is a useful one to have. For instance:

• A user might want to install CipheyCore to a local space
• It makes it easier to write scripts for many package managers
• Also easier to use the library after building from source
• Running make install currently results in the error make: *** No rule to make target 'install'. Stop.

The issue for me was that the DESTDIR variable is used as part of MacPorts to provide a staged install process via make install DESTDIR=/tmp/example. It would be helpful if an install target could be added, although I'd also be grateful for any alternative methods.

Thanks for your help.

To reproduce error

macOS 10.15.6 19G73

xcode-select version 2373

git clone https://github.com/Ciphey/CipheyCore
cd CipheyCore
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCIPHEY_CORE_TEST=OFF
cmake --build . -t ciphey_core
make install

Output

cmake .. -DCMAKE_BUILD_TYPE=Release -DCIPHEY_CORE_TEST=OFF

-- The C compiler identification is AppleClang 11.0.3.11030032
-- The CXX compiler identification is AppleClang 11.0.3.11030032
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Boost: /usr/local/lib/cmake/Boost-1.74.0/BoostConfig.cmake (found version "1.74.0")  
-- Found Python3: /opt/local/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 (found version "3.9.0") found components: Development Development.Module Development.Embed 
-- Found SWIG: /opt/local/bin/swig (found version "4.0.2")  
CMake Warning (dev) at /opt/local/share/cmake-3.18/Modules/UseSWIG.cmake:634 (message):
  Policy CMP0078 is not set: UseSWIG generates standard target names.  Run
  "cmake --help-policy CMP0078" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

Call Stack (most recent call first):
  CMakeLists.txt:55 (swig_add_library)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /opt/local/share/cmake-3.18/Modules/UseSWIG.cmake:486 (message):
  Policy CMP0086 is not set: UseSWIG honors SWIG_MODULE_NAME via -module
  flag.  Run "cmake --help-policy CMP0086" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

Call Stack (most recent call first):
  /opt/local/share/cmake-3.18/Modules/UseSWIG.cmake:736 (SWIG_ADD_SOURCE_TO_MODULE)
  CMakeLists.txt:55 (swig_add_library)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/username/CipheyCore/build

cmake --build . -t ciphey_core

Scanning dependencies of target ciphey_core
[ 14%] Building CXX object CMakeFiles/ciphey_core.dir/src/ausearch.cpp.o
[ 28%] Building CXX object CMakeFiles/ciphey_core.dir/src/ciphers/caesar.cpp.o
[ 42%] Building CXX object CMakeFiles/ciphey_core.dir/src/ciphers/vigenere.cpp.o
[ 57%] Building CXX object CMakeFiles/ciphey_core.dir/src/ciphers/xor_single.cpp.o
[ 71%] Building CXX object CMakeFiles/ciphey_core.dir/src/ciphers/xorcrypt.cpp.o
[ 85%] Building CXX object CMakeFiles/ciphey_core.dir/src/freq.cpp.o
[100%] Linking CXX static library libciphey_core.a
[100%] Built target ciphey_core

make install (ERROR)

make: *** No rule to make target `install'.  Stop.