binref / refinery Goto Github PK
View Code? Open in Web Editor NEWHigh Octane Triage Analysis
License: Other
High Octane Triage Analysis
License: Other
For tea and xtea commands, block operation is done with little endian.
Excerpt from refinery/units/crypto/cipher/tea.py:
def tea_block_operation(
blk: Callable[[int, int, int, int, int, int], Tuple[int, int]]
) -> Callable[[TEA, ByteString], ByteString]:
def wrapped(self: TEA, data: ByteString) -> ByteString:
v0, v1 = blk(
int.from_bytes(data[:4], 'little'),
int.from_bytes(data[4:], 'little'),
*self.derived_key
)
return v0.to_bytes(4, 'little') + v1.to_bytes(4, 'little')
return wrapped
class TEABase(BlockCipher):
block_size = 8
valid_key_sizes = {16}
derived_key: List[int]
@property
def key(self):
return self.derived_key
@key.setter
def key(self, key):
self.derived_key = [int.from_bytes(key[k:k + 4], 'little') for k in range(0, 16, 4)]
However, there are implementations of TEA and XTEA that block operation is done with big endian.
For example, Crypt::XTEA Perl module has the little_endian
option and the default is little_endian=0 (big endian). If block operation is done with big endian on encryption, tea and xtea commands cannot decrypt correctly. It would be great if endianness is configurable for tea and xtea commands.
A sample Perl script xtea.pl for encryption with XTEA:
use Crypt::XTEA;
use Crypt::ECB;
my $key = "0123456789abcdef";
# Little endian
my $xtea = Crypt::XTEA->new( $key, 32, little_endian => 1 );
my $ecb = Crypt::ECB->new( -cipher => $xtea );
my $text = 'This is a secret message.';
my $cipher_text = $ecb->encrypt( $text );
my $cipher_hex = unpack("H*", $cipher_text);
print("Ciphertext with little endian: $cipher_hex\n");
# Big endian (default)
$xtea = Crypt::XTEA->new( $key );
$ecb = Crypt::ECB->new( -cipher => $xtea );
$text = 'This is a secret message.';
$cipher_text = $ecb->encrypt( $text );
$cipher_hex = unpack("H*", $cipher_text);
print("Ciphertext with big endian: $cipher_hex\n");
Decryption results with xtea command:
$ perl xtea.pl
Ciphertext with little endian: a3bab9180f2ff58eb6d5bfe356fc2b436c9ae1338fee4cb70b626fd037e4a797
Ciphertext with big endian: eb7a1e2fe4de2699978ddfcec5bdd3649d654c0a7bc1c5c8b343cd8763afc88
$ echo -n a3bab9180f2ff58eb6d5bfe356fc2b436c9ae1338fee4cb70b626fd037e4a797 | hex | xtea -m ecb -L 0123456789abcdef | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e message.
$ echo -n eb7a1e2fe4de2699978ddfcec5bdd3649d654c0a7bc1c5c8b343cd8763afc883 | hex | xtea -m ecb -L 0123456789abcdef | xxd
00000000: 03dd 14c3 a150 5bf3 ffe0 a500 cc04 8402 .....P[.........
00000010: d7af 3dfa 64da 902b 3c87 1a19 72e2 22ab ..=.d..+<...r.".
$
Got the following log:
...
Collecting msgpack>=1.0.0
Using cached msgpack-1.0.8.tar.gz (167 kB)
Installing build dependencies: started
Running command pip subprocess to install build dependencies
Using pip 24.2 from /workspaces/venv/lib/python3.14t/site-packages/pip (python 3.14)
Collecting Cython~=3.0.8
Obtaining dependency information for Cython~=3.0.8 from https://files.pythonhosted.org/packages/43/39/bdbec9142bc46605b54d674bf158a78b191c2b75be527c6dcf3e6dfe90b8/Cython-3.0.11-py2.py3-none-any.whl.metadata
Using cached Cython-3.0.11-py2.py3-none-any.whl.metadata (3.2 kB)
Collecting setuptools>=35.0.2
Obtaining dependency information for setuptools>=35.0.2 from https://files.pythonhosted.org/packages/6e/ec/06715d912351edc453e37f93f3fc80dcffd5ca0e70386c87529aca296f05/setuptools-72.2.0-py3-none-any.whl.metadata
Using cached setuptools-72.2.0-py3-none-any.whl.metadata (6.6 kB)
Using cached Cython-3.0.11-py2.py3-none-any.whl (1.2 MB)
Using cached setuptools-72.2.0-py3-none-any.whl (2.3 MB)
Installing collected packages: setuptools, Cython
Creating /tmp/pip-build-env-4jdpxvmr/overlay/bin
changing mode of /tmp/pip-build-env-4jdpxvmr/overlay/bin/cygdb to 757
changing mode of /tmp/pip-build-env-4jdpxvmr/overlay/bin/cython to 757
changing mode of /tmp/pip-build-env-4jdpxvmr/overlay/bin/cythonize to 757
Successfully installed Cython-3.0.11 setuptools-72.2.0
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Running command Getting requirements to build wheel
running egg_info
writing msgpack.egg-info/PKG-INFO
writing dependency_links to msgpack.egg-info/dependency_links.txt
writing top-level names to msgpack.egg-info/top_level.txt
reading manifest file 'msgpack.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.c' under directory 'msgpack'
adding license file 'COPYING'
writing manifest file 'msgpack.egg-info/SOURCES.txt'
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Running command Preparing metadata (pyproject.toml)
running dist_info
creating /tmp/pip-modern-metadata-w23ik30t/msgpack.egg-info
writing /tmp/pip-modern-metadata-w23ik30t/msgpack.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-modern-metadata-w23ik30t/msgpack.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-modern-metadata-w23ik30t/msgpack.egg-info/top_level.txt
writing manifest file '/tmp/pip-modern-metadata-w23ik30t/msgpack.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-modern-metadata-w23ik30t/msgpack.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.c' under directory 'msgpack'
adding license file 'COPYING'
writing manifest file '/tmp/pip-modern-metadata-w23ik30t/msgpack.egg-info/SOURCES.txt'
creating '/tmp/pip-modern-metadata-w23ik30t/msgpack-1.0.8.dist-info'
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pefile
Obtaining dependency information for pefile from https://files.pythonhosted.org/packages/55/26/d0ad8b448476d0a1e8d3ea5622dc77b916db84c6aa3cb1e1c0965af948fc/pefile-2023.2.7-py3-none-any.whl.metadata
Using cached pefile-2023.2.7-py3-none-any.whl.metadata (1.4 kB)
Collecting pycryptodomex
Using cached pycryptodomex-3.20.0.tar.gz (4.8 MB)
Installing build dependencies: started
Running command pip subprocess to install build dependencies
Using pip 24.2 from /workspaces/venv/lib/python3.14t/site-packages/pip (python 3.14)
Collecting setuptools
Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/6e/ec/06715d912351edc453e37f93f3fc80dcffd5ca0e70386c87529aca296f05/setuptools-72.2.0-py3-none-any.whl.metadata
Using cached setuptools-72.2.0-py3-none-any.whl.metadata (6.6 kB)
Using cached setuptools-72.2.0-py3-none-any.whl (2.3 MB)
Installing collected packages: setuptools
Successfully installed setuptools-72.2.0
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Running command Getting requirements to build wheel
...
Building wheels for collected packages: msgpack, pycryptodomex
Building wheel for msgpack (pyproject.toml): started
Running command Building wheel for msgpack (pyproject.toml)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-314t
creating build/lib.linux-x86_64-cpython-314t/msgpack
copying msgpack/fallback.py -> build/lib.linux-x86_64-cpython-314t/msgpack
copying msgpack/exceptions.py -> build/lib.linux-x86_64-cpython-314t/msgpack
copying msgpack/__init__.py -> build/lib.linux-x86_64-cpython-314t/msgpack
copying msgpack/ext.py -> build/lib.linux-x86_64-cpython-314t/msgpack
running build_ext
building 'msgpack._cmsgpack' extension
creating build/temp.linux-x86_64-cpython-314t
creating build/temp.linux-x86_64-cpython-314t/msgpack
x86_64-linux-gnu-g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I. -I/workspaces/venv/include -I/usr/include/python3.14t -c msgpack/_cmsgpack.cpp -o build/temp.linux-x86_64-cpython-314t/msgpack/_cmsgpack.o
msgpack/_cmsgpack.cpp:1366:72: warning: ‘Py_UNICODE’ is deprecated [-Wdeprecated-declarations]
1366 | static CYTHON_INLINE size_t __Pyx_Py_UNICODE_strlen(const Py_UNICODE *u)
| ^
In file included from /usr/include/python3.14t/unicodeobject.h:1014,
from /usr/include/python3.14t/Python.h:76,
from msgpack/_cmsgpack.cpp:16:
/usr/include/python3.14t/cpython/unicodeobject.h:10:37: note: declared here
10 | Py_DEPRECATED(3.13) typedef wchar_t Py_UNICODE;
| ^~~~~~~~~~
msgpack/_cmsgpack.cpp: In function ‘size_t __Pyx_Py_UNICODE_strlen(const Py_UNICODE*)’:
msgpack/_cmsgpack.cpp:1368:23: warning: ‘Py_UNICODE’ is deprecated [-Wdeprecated-declarations]
1368 | const Py_UNICODE *u_end = u;
| ^~~~~
In file included from /usr/include/python3.14t/unicodeobject.h:1014,
from /usr/include/python3.14t/Python.h:76,
from msgpack/_cmsgpack.cpp:16:
/usr/include/python3.14t/cpython/unicodeobject.h:10:37: note: declared here
10 | Py_DEPRECATED(3.13) typedef wchar_t Py_UNICODE;
| ^~~~~~~~~~
msgpack/_cmsgpack.cpp: In function ‘int __Pyx_CheckKeywordStrings(PyObject*, const char*, int)’:
msgpack/_cmsgpack.cpp:19109:9: error: ‘CYTHON_METH_FASTCALL’ was not declared in this scope; did you mean ‘METH_FASTCALL’?
19109 | if (CYTHON_METH_FASTCALL && likely(PyTuple_Check(kw))) {
| ^~~~~~~~~~~~~~~~~~~~
| METH_FASTCALL
msgpack/_cmsgpack.cpp: In function ‘int __Pyx_ParseOptionalKeywords(PyObject*, PyObject* const*, PyObject***, PyObject*, PyObject**, Py_ssize_t, const char*)’:
msgpack/_cmsgpack.cpp:19198:25: error: ‘CYTHON_METH_FASTCALL’ was not declared in this scope; did you mean ‘METH_FASTCALL’?
19198 | int kwds_is_tuple = CYTHON_METH_FASTCALL && likely(PyTuple_Check(kwds));
| ^~~~~~~~~~~~~~~~~~~~
| METH_FASTCALL
error: command '/usr/bin/x86_64-linux-gnu-g++' failed with exit code 1
error: subprocess-exited-with-error
× Building wheel for msgpack (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /workspaces/venv/bin/python3.14t /workspaces/venv/lib/python3.14t/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpwebmrrqb
cwd: /tmp/pip-install-d715fpyo/msgpack_a1078e0309324d2fb789c24e9e98d290
Building wheel for msgpack (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for msgpack
Building wheel for pycryptodomex (pyproject.toml): started
Running command Building wheel for pycryptodomex (pyproject.toml)
/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/dist.py:447: SetuptoolsDeprecationWarning: Invalid dash-separated options
!!
...
running install_egg_info
running egg_info
writing lib/pycryptodomex.egg-info/PKG-INFO
writing dependency_links to lib/pycryptodomex.egg-info/dependency_links.txt
writing top-level names to lib/pycryptodomex.egg-info/top_level.txt
reading manifest file 'lib/pycryptodomex.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'Doc/_build'
warning: no previously-included files matching '*.pyc' found under directory 'src'
no previously-included directories found matching 'src/test/build'
adding license file 'LICENSE.rst'
adding license file 'AUTHORS.rst'
writing manifest file 'lib/pycryptodomex.egg-info/SOURCES.txt'
Copying lib/pycryptodomex.egg-info to build/bdist.linux-x86_64/wheel/./pycryptodomex-3.20.0-py3.14.egg-info
running install_scripts
Traceback (most recent call last):
File "/workspaces/venv/lib/python3.14t/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
~~~~^^
File "/workspaces/venv/lib/python3.14t/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/venv/lib/python3.14t/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
metadata_directory)
^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/build_meta.py", line 415, in build_wheel
return self._build_with_temp_dir(
~~~~~~~~~~~~~~~~~~~~~~~~~^
['bdist_wheel'],
^^^^^^^^^^^^^^^^
...<3 lines>...
self._arbitrary_args(config_settings),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
self.run_setup()
~~~~~~~~~~~~~~^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/build_meta.py", line 313, in run_setup
exec(code, locals())
~~~~^^^^^^^^^^^^^^^^
File "<string>", line 500, in <module>
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/__init__.py", line 108, in setup
return distutils.core.setup(**attrs)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
~~~~~~~~~~~~~~~~~^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/_distutils/dist.py", line 964, in run_commands
self.run_command(cmd)
~~~~~~~~~~~~~~~~^^^^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/dist.py", line 945, in run_command
super().run_command(command)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/_distutils/dist.py", line 983, in run_command
cmd_obj.run()
~~~~~~~~~~~^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/command/bdist_wheel.py", line 410, in run
impl_tag, abi_tag, plat_tag = self.get_tag()
~~~~~~~~~~~~^^
File "/tmp/pip-build-env-ccc_xdwh/overlay/lib/python3.14t/site-packages/setuptools/command/bdist_wheel.py", line 360, in get_tag
tag in supported_tags
AssertionError: would build wheel with unsupported tag ('cp35', 'abi3', 'linux_x86_64')
error: subprocess-exited-with-error
× Building wheel for pycryptodomex (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /workspaces/venv/bin/python3.14t /workspaces/venv/lib/python3.14t/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpfgt_3ucy
cwd: /tmp/pip-install-d715fpyo/pycryptodomex_3e572a56c4974502984104a416dd7238
Building wheel for pycryptodomex (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for pycryptodomex
Failed to build msgpack pycryptodomex
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (msgpack, pycryptodomex)
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /workspaces/venv/bin/python3.14t /workspaces/venv/lib/python3.14t/site-packages/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /tmp/pip-build-env-59dw6c9z/overlay --no-warn-script-location --disable-pip-version-check -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- colorama defusedxml macholib 'msgpack>=1.0.0' pefile pycryptodomex pyelftools setuptools toml wheel
cwd: [inherit]
Installing build dependencies: finished with status 'error'
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Two dependencies should be bumped: pycryptodex (Legrandin/pycryptodome#813) and cython
(venv) @clin1234 ➜ /workspaces/venv $ pip install -U g git+https://github.com/pyca/cryptography
Collecting git+https://github.com/pyca/cryptography
Cloning https://github.com/pyca/cryptography to /tmp/pip-req-build-5219onmx
Running command git clone --filter=blob:none --quiet https://github.com/pyca/cryptography /tmp/pip-req-build-5219onmx
Resolved https://github.com/pyca/cryptography to commit e6cf6cab9999c9885155a961a80f91bb7d3158d1
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.12 (from cryptography==44.0.0.dev1)
Using cached cffi-1.17.0-cp314-cp314t-linux_x86_64.whl
Collecting pycparser (from cffi>=1.12->cryptography==44.0.0.dev1)
Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: cryptography
Building wheel for cryptography (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for cryptography (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [257 lines of output]
Running `maturin pep517 build-wheel -i /workspaces/venv/bin/python3.14t --compatibility off`
📦 Including license file "/tmp/pip-req-build-5219onmx/LICENSE"
📦 Including license file "/tmp/pip-req-build-5219onmx/LICENSE.APACHE"
📦 Including license file "/tmp/pip-req-build-5219onmx/LICENSE.BSD"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
🐍 Not using a specific python interpreter
📡 Using build options features, locked from pyproject.toml
Compiling proc-macro2 v1.0.86
Compiling target-lexicon v0.12.16
Compiling unicode-ident v1.0.12
Compiling shlex v1.3.0
Compiling cc v1.1.12
Compiling pyo3-build-config v0.22.2
Compiling quote v1.0.36
Compiling syn v2.0.74
Compiling vcpkg v0.2.15
Compiling once_cell v1.19.0
Compiling pkg-config v0.3.30
Compiling openssl-sys v0.9.103
Compiling libc v0.2.155
Compiling cfg-if v1.0.0
Compiling pyo3-ffi v0.22.2
Compiling pyo3-macros-backend v0.22.2
Compiling autocfg v1.3.0
Compiling memoffset v0.9.1
Compiling openssl v0.10.66
Compiling foreign-types-shared v0.1.1
Compiling heck v0.5.0
Compiling foreign-types v0.3.2
Compiling asn1_derive v0.17.0
Compiling openssl-macros v0.1.1
Compiling pyo3 v0.22.2
Compiling bitflags v2.6.0
Compiling pyo3-macros v0.22.2
Compiling asn1 v0.17.0
Compiling indoc v2.0.5
Compiling unindent v0.2.3
Compiling cryptography-key-parsing v0.1.0 (/tmp/pip-req-build-5219onmx/src/rust/cryptography-key-parsing)
warning: [email protected]: cargo:rustc-check-cfg requires -Zcheck-cfg flag
warning: [email protected]: cargo:rustc-check-cfg requires -Zcheck-cfg flag
Compiling cryptography-x509 v0.1.0 (/tmp/pip-req-build-5219onmx/src/rust/cryptography-x509)
Compiling cryptography-cffi v0.1.0 (/tmp/pip-req-build-5219onmx/src/rust/cryptography-cffi)
Compiling cryptography-openssl v0.1.0 (/tmp/pip-req-build-5219onmx/src/rust/cryptography-openssl)
warning: [email protected]: cargo:rustc-check-cfg requires -Zcheck-cfg flag
warning: [email protected]: cargo:rustc-check-cfg requires -Zcheck-cfg flag
warning: [email protected]: cargo:rustc-check-cfg requires -Zcheck-cfg flag
warning: [email protected]: cargo:rustc-check-cfg requires -Zcheck-cfg flag
The following warnings were emitted during compilation:
warning: [email protected]: In file included from /tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/_openssl.c:57:
warning: [email protected]: /usr/include/python3.14t/Python.h:51:4: error: #error "The limited API is not currently supported in the free-threaded build"
warning: [email protected]: 51 | # error "The limited API is not currently supported in the free-threaded build"
warning: [email protected]: | ^~~~~
warning: [email protected]: In file included from /usr/include/python3.14t/Python.h:68,
warning: [email protected]: from /tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/_openssl.c:57:
warning: [email protected]: /usr/include/python3.14t/object.h:146:5: error: unknown type name ‘PyMutex’
warning: [email protected]: 146 | PyMutex ob_mutex; // per-object lock
warning: [email protected]: | ^~~~~~~
warning: [email protected]: /usr/include/python3.14t/object.h: In function ‘Py_SET_SIZE’:
warning: [email protected]: /usr/include/python3.14t/object.h:293:5: warning: implicit declaration of function ‘_Py_atomic_store_ssize_relaxed’ [-Wimplicit-function-declaration]
warning: [email protected]: 293 | _Py_atomic_store_ssize_relaxed(&ob->ob_size, size);
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: In file included from /usr/include/python3.14t/Python.h:69,
warning: [email protected]: from /tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/_openssl.c:57:
warning: [email protected]: /usr/include/python3.14t/refcount.h: In function ‘Py_REFCNT’:
warning: [email protected]: /usr/include/python3.14t/refcount.h:84:22: warning: implicit declaration of function ‘_Py_atomic_load_uint32_relaxed’ [-Wimplicit-function-declaration]
warning: [email protected]: 84 | uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: /usr/include/python3.14t/refcount.h:84:22: warning: conversion to ‘uint32_t’ {aka ‘unsigned int’} from ‘int’ may change the sign of the result [-Wsign-conversion]
warning: [email protected]: /usr/include/python3.14t/refcount.h:88:25: warning: implicit declaration of function ‘_Py_atomic_load_ssize_relaxed’ [-Wimplicit-function-declaration]
warning: [email protected]: 88 | Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: /usr/include/python3.14t/refcount.h: In function ‘_Py_IsImmortal’:
warning: [email protected]: /usr/include/python3.14t/refcount.h:101:63: warning: comparison of integer expressions of different signedness: ‘int’ and ‘unsigned int’ [-Wsign-compare]
warning: [email protected]: 101 | return (_Py_atomic_load_uint32_relaxed(&op->ob_ref_local) ==
warning: [email protected]: | ^~
warning: [email protected]: /usr/include/python3.14t/refcount.h: In function ‘Py_SET_REFCNT’:
warning: [email protected]: /usr/include/python3.14t/refcount.h:132:9: warning: implicit declaration of function ‘_Py_IsOwnedByCurrentThread’ [-Wimplicit-function-declaration]
warning: [email protected]: 132 | if (_Py_IsOwnedByCurrentThread(ob)) {
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: /usr/include/python3.14t/refcount.h: In function ‘Py_INCREF’:
warning: [email protected]: /usr/include/python3.14t/refcount.h:227:22: warning: conversion to ‘uint32_t’ {aka ‘unsigned int’} from ‘int’ may change the sign of the result [-Wsign-conversion]
warning: [email protected]: 227 | uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: /usr/include/python3.14t/refcount.h:234:9: warning: implicit declaration of function ‘_Py_atomic_store_uint32_relaxed’ [-Wimplicit-function-declaration]
warning: [email protected]: 234 | _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, new_local);
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: /usr/include/python3.14t/refcount.h:237:9: warning: implicit declaration of function ‘_Py_atomic_add_ssize’; did you mean ‘__atomic_add_fetch’? [-Wimplicit-function-declaration]
warning: [email protected]: 237 | _Py_atomic_add_ssize(&op->ob_ref_shared, (1 << _Py_REF_SHARED_SHIFT));
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~
warning: [email protected]: | __atomic_add_fetch
warning: [email protected]: /usr/include/python3.14t/refcount.h: In function ‘Py_DECREF’:
warning: [email protected]: /usr/include/python3.14t/refcount.h:321:22: warning: conversion to ‘uint32_t’ {aka ‘unsigned int’} from ‘int’ may change the sign of the result [-Wsign-conversion]
warning: [email protected]: 321 | uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: /usr/include/python3.14t/refcount.h:330:13: warning: implicit declaration of function ‘_Py_MergeZeroLocalRefcount’ [-Wimplicit-function-declaration]
warning: [email protected]: 330 | _Py_MergeZeroLocalRefcount(op);
warning: [email protected]: | ^~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]: /usr/include/python3.14t/refcount.h:334:9: warning: implicit declaration of function ‘_Py_DecRefShared’; did you mean ‘_Py_DecRef’? [-Wimplicit-function-declaration]
warning: [email protected]: 334 | _Py_DecRefShared(op);
warning: [email protected]: | ^~~~~~~~~~~~~~~~
warning: [email protected]: | _Py_DecRef
error: failed to run custom build command for `cryptography-cffi v0.1.0 (/tmp/pip-req-build-5219onmx/src/rust/cryptography-cffi)`
Caused by:
process didn't exit successfully: `/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-01ffb5525631fdf0/build-script-build` (exit status: 1)
--- stdout
cargo:rustc-check-cfg=cfg(python_implementation, values("CPython", "PyPy"))
cargo:rerun-if-env-changed=PYO3_PYTHON
cargo:rerun-if-changed=../../_cffi_src/
cargo:rerun-if-changed=../../cryptography/__about__.py
cargo:rustc-cfg=python_implementation="CPython"
OUT_DIR = Some(/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out)
TARGET = Some(x86_64-unknown-linux-gnu)
OPT_LEVEL = Some(3)
HOST = Some(x86_64-unknown-linux-gnu)
cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
CC_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
CC_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CC
HOST_CC = None
cargo:rerun-if-env-changed=CC
CC = None
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
RUSTC_WRAPPER = None
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some(false)
CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
CFLAGS_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
CFLAGS_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CFLAGS
HOST_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
OUT_DIR = Some(/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out)
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
CFLAGS_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
CFLAGS_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CFLAGS
HOST_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
OUT_DIR = Some(/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out)
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
CFLAGS_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
CFLAGS_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CFLAGS
HOST_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
OUT_DIR = Some(/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out)
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
CFLAGS_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
CFLAGS_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CFLAGS
HOST_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
OUT_DIR = Some(/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out)
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
CFLAGS_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
CFLAGS_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CFLAGS
HOST_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
cargo:warning=In file included from /tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/_openssl.c:57:
cargo:warning=/usr/include/python3.14t/Python.h:51:4: error: #error "The limited API is not currently supported in the free-threaded build"
cargo:warning= 51 | # error "The limited API is not currently supported in the free-threaded build"
cargo:warning= | ^~~~~
cargo:warning=In file included from /usr/include/python3.14t/Python.h:68,
cargo:warning= from /tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/_openssl.c:57:
cargo:warning=/usr/include/python3.14t/object.h:146:5: error: unknown type name ‘PyMutex’
cargo:warning= 146 | PyMutex ob_mutex; // per-object lock
cargo:warning= | ^~~~~~~
cargo:warning=/usr/include/python3.14t/object.h: In function ‘Py_SET_SIZE’:
cargo:warning=/usr/include/python3.14t/object.h:293:5: warning: implicit declaration of function ‘_Py_atomic_store_ssize_relaxed’ [-Wimplicit-function-declaration]
cargo:warning= 293 | _Py_atomic_store_ssize_relaxed(&ob->ob_size, size);
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=In file included from /usr/include/python3.14t/Python.h:69,
cargo:warning= from /tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/_openssl.c:57:
cargo:warning=/usr/include/python3.14t/refcount.h: In function ‘Py_REFCNT’:
cargo:warning=/usr/include/python3.14t/refcount.h:84:22: warning: implicit declaration of function ‘_Py_atomic_load_uint32_relaxed’ [-Wimplicit-function-declaration]
cargo:warning= 84 | uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=/usr/include/python3.14t/refcount.h:84:22: warning: conversion to ‘uint32_t’ {aka ‘unsigned int’} from ‘int’ may change the sign of the result [-Wsign-conversion]
cargo:warning=/usr/include/python3.14t/refcount.h:88:25: warning: implicit declaration of function ‘_Py_atomic_load_ssize_relaxed’ [-Wimplicit-function-declaration]
cargo:warning= 88 | Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=/usr/include/python3.14t/refcount.h: In function ‘_Py_IsImmortal’:
cargo:warning=/usr/include/python3.14t/refcount.h:101:63: warning: comparison of integer expressions of different signedness: ‘int’ and ‘unsigned int’ [-Wsign-compare]
cargo:warning= 101 | return (_Py_atomic_load_uint32_relaxed(&op->ob_ref_local) ==
cargo:warning= | ^~
cargo:warning=/usr/include/python3.14t/refcount.h: In function ‘Py_SET_REFCNT’:
cargo:warning=/usr/include/python3.14t/refcount.h:132:9: warning: implicit declaration of function ‘_Py_IsOwnedByCurrentThread’ [-Wimplicit-function-declaration]
cargo:warning= 132 | if (_Py_IsOwnedByCurrentThread(ob)) {
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=/usr/include/python3.14t/refcount.h: In function ‘Py_INCREF’:
cargo:warning=/usr/include/python3.14t/refcount.h:227:22: warning: conversion to ‘uint32_t’ {aka ‘unsigned int’} from ‘int’ may change the sign of the result [-Wsign-conversion]
cargo:warning= 227 | uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=/usr/include/python3.14t/refcount.h:234:9: warning: implicit declaration of function ‘_Py_atomic_store_uint32_relaxed’ [-Wimplicit-function-declaration]
cargo:warning= 234 | _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, new_local);
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=/usr/include/python3.14t/refcount.h:237:9: warning: implicit declaration of function ‘_Py_atomic_add_ssize’; did you mean ‘__atomic_add_fetch’? [-Wimplicit-function-declaration]
cargo:warning= 237 | _Py_atomic_add_ssize(&op->ob_ref_shared, (1 << _Py_REF_SHARED_SHIFT));
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~
cargo:warning= | __atomic_add_fetch
cargo:warning=/usr/include/python3.14t/refcount.h: In function ‘Py_DECREF’:
cargo:warning=/usr/include/python3.14t/refcount.h:321:22: warning: conversion to ‘uint32_t’ {aka ‘unsigned int’} from ‘int’ may change the sign of the result [-Wsign-conversion]
cargo:warning= 321 | uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=/usr/include/python3.14t/refcount.h:330:13: warning: implicit declaration of function ‘_Py_MergeZeroLocalRefcount’ [-Wimplicit-function-declaration]
cargo:warning= 330 | _Py_MergeZeroLocalRefcount(op);
cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=/usr/include/python3.14t/refcount.h:334:9: warning: implicit declaration of function ‘_Py_DecRefShared’; did you mean ‘_Py_DecRef’? [-Wimplicit-function-declaration]
cargo:warning= 334 | _Py_DecRefShared(op);
cargo:warning= | ^~~~~~~~~~~~~~~~
cargo:warning= | _Py_DecRef
--- stderr
error occurred: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "/usr/include" "-I" "/workspaces/venv/include" "-I" "/usr/include/python3.14t" "-Wall" "-Wextra" "-Wconversion" "-Wno-error=sign-conversion" "-Wno-unused-parameter" "-fmacro-prefix-map=/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out=." "-DPy_LIMITED_API=0x030700f0" "-o" "/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/5e6d3db45e392808-_openssl.o" "-c" "/tmp/pip-req-build-5219onmx/src/rust/target/release/build/cryptography-cffi-eec357800f931841/out/_openssl.c" with args cc did not execute successfully (status code exit status: 1).
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.14-64bit" PYO3_PYTHON="/workspaces/venv/bin/python3.14t" PYTHON_SYS_EXECUTABLE="/workspaces/venv/bin/python3.14t" "cargo" "rustc" "--features" "pyo3/abi3-py37" "--message-format" "json-render-diagnostics" "--locked" "--manifest-path" "/tmp/pip-req-build-5219onmx/src/rust/Cargo.toml" "--release" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/workspaces/venv/bin/python3.14t', '--compatibility', 'off'] returned non-zero exit status 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (cryptography)
Installing binary-refinery[all]
currently fails because of this issue with PyPCAPKit.
The peek
unit states, in its help output, that the -m, --meta
flag does the following:
Only display attached metadata, do not add the peek value.
However, it seems that the peek unit currently does not display the metadata at all if the -m
flag is specified; it behaves the same as if -r, --bare
is specified, and only shows the peeked data:
$ emit yosemite.png | peek
-------------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 5.441 MB; PNG image data, 2560 x 1440, 8-bit/color RGB, non-interlaced
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09 .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43 pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42 C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8 ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33 ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00 ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99 @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00 .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------
$ emit yosemite.png | peek --meta
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09 .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43 pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42 C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8 ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33 ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00 ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99 @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00 .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------
$ emit yosemite.png | peek --bare
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09 .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43 pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42 C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8 ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33 ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00 ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99 @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00 .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Reproduced with binary-refinery version 0.4.30, Python 3.10.4.
Can lzf be added to the decompression functionality as well?
Basically this project: https://github.com/teepark/python-lzf
camellia command has padding incompatibilities with OpenSSL for CFB, OFB, and CTR modes. OpenSSL does not add padding and camellia command adds padding for these modes.
When a plaintext is encrypted with openssl command then the ciphertext output is decrypted with camellia command, the plaintext output is truncated. On the other hand, when a plaintext is encrypted with camellia command then the ciphertext output is decrypted with openssl command, the plaintext output has redundant trailing bytes. These behaviors do not occur for ECB and CBC modes. Truncation of plaintext outputs on decryption is an especially critical issue.
According to the article of Block cipher mode of operation in the WikiPedia, CFB, OFB and CTR modes do not require any special measures to handle messages whose lengths are not multiples of the block size. So I think it is incorrect that camellia command requires input as a multiple of the 16-byte block size for CFB, OFB and CTR modes.
I use Binary Refinery as Python modules for writing some plugins of FileInsight-plugins. Binary Refinery is so great to support a lot of encodings, crypto algorithms, and compression algorithms.
Encryption with ECB mode (identical output):
$ echo "This is a secret message." | openssl enc -e -camellia-128-ecb -K 30313233343536373839616263646566 | xxd
00000000: c4eb d417 c468 8165 2367 99f1 c5e0 bcb8 .....h.e#g......
00000010: ff47 867e 39ca b441 a4c3 0679 4a48 00bb .G.~9..A...yJH..
$
$ echo "This is a secret message." | camellia --mode ecb -R 0123456789abcdef | xxd
00000000: c4eb d417 c468 8165 2367 99f1 c5e0 bcb8 .....h.e#g......
00000010: ff47 867e 39ca b441 a4c3 0679 4a48 00bb .G.~9..A...yJH..
$
Encryption with CBC mode (identical output):
$ echo "This is a secret message." | openssl enc -e -camellia-128-cbc -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: cda7 a782 bce6 4ca8 6c3a a804 b141 ce6c ......L.l:...A.l
00000010: c1c4 8c07 fb65 357d 21d0 4da9 6a2c 0f49 .....e5}!.M.j,.I
$
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cbc -R 0123456789abcdef | xxd
00000000: cda7 a782 bce6 4ca8 6c3a a804 b141 ce6c ......L.l:...A.l
00000010: c1c4 8c07 fb65 357d 21d0 4da9 6a2c 0f49 .....e5}!.M.j,.I
$
Encryption with CFB mode (different outputs):
$ echo "This is a secret message." | openssl enc -e -camellia-128-cfb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127 >....t.F.%...v.'
00000010: 6833 fe93 bb59 043d 8b3e h3...Y.=.>
$
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cfb -S 128 -R 0123456789abcdef | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127 >....t.F.%...v.'
00000010: 6833 fe93 bb59 043d 8b3e abce 0701 8677 h3...Y.=.>.....w
Encryption with OFB mode (different outputs):
$ echo "This is a secret message." | openssl enc -e -camellia-128-ofb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127 >....t.F.%...v.'
00000010: 36dd fea5 869c 171f 547c 6.......T|
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ofb -R 0123456789abcdef | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127 >....t.F.%...v.'
00000010: 36dd fea5 869c 171f 547c 0363 e05a 2850 6.......T|.c.Z(P
$
Encryption with CTR mode (different outputs):
$ echo "This is a secret message." | openssl enc -e -camellia-128-ctr -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127 >....t.F.%...v.'
00000010: 472b 60a4 8c98 0540 ce1a G+`....@..
$
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ctr -R 0123456789abcdef | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127 >....t.F.%...v.'
00000010: 472b 60a4 8c98 0540 ce1a dd81 0101 dc8f G+`....@........
$
Encryption with openssl command and decryption with camellia command (ECB mode, works correctly):
$ echo "This is a secret message." | openssl enc -e -camellia-128-ecb -K 30313233343536373839616263646566 | camellia --mode ecb 0123456789abcdef | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e0a message..
$
Encryption with openssl command and decryption with camellia command (CBC mode, works correctly):
$ echo "This is a secret message." | openssl enc -e -camellia-128-cbc -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode cbc -S 128 -L 0123456789abcdef | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e0a message..
$
Encryption with openssl command and decryption with camellia command (CFB mode, the plaintext output is truncated):
$ echo "This is a secret message." | openssl enc -e -camellia-128-cfb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode cfb -S 128 -L 0123456789abcdef | xxd
(22:20:55) warning in camellia: removing 10 bytes from the input to make it a multiple of the 16-byte block size
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
$
Encryption with openssl command and decryption with camellia command (OFB mode, the plaintext output is truncated):
$ echo "This is a secret message." | openssl enc -e -camellia-128-ofb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode ofb -L 0123456789abcdef | xxd
(22:23:24) warning in camellia: removing 10 bytes from the input to make it a multiple of the 16-byte block size
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
$
Encryption with openssl command and decryption with camellia command (CTR mode, the plaintext output is truncated):
$ echo "This is a secret message." | openssl enc -e -camellia-128-ctr -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode ctr -L 0123456789abcdef | xxd
(22:24:28) warning in camellia: removing 10 bytes from the input to make it a multiple of the 16-byte block size
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
$
Encryption with camellia command and decryption with openssl command (ECB mode, works correctly):
$ echo "This is a secret message." | camellia --mode ecb -R 0123456789abcdef | openssl enc -d -camellia-128-ecb -K 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e0a message..
$
Encryption with camellia command and decryption with openssl command (CBC mode, works correctly):
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cbc -R 0123456789abcdef | openssl enc -d -camellia-128-cbc -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e0a message..
Encryption with camellia command and decryption with openssl command (CFB mode, the plaintext output has redundant trailing bytes):
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cfb -S 128 -R 0123456789abcdef | openssl enc -d -camellia-128-cfb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e0a 0606 0606 0606 message........
$
Encryption with camellia command and decryption with openssl command (OFB mode, the plaintext output has redundant trailing bytes):
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ofb -R 0123456789abcdef | openssl enc -d -camellia-128-ofb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e0a 0606 0606 0606 message........
$
Encryption with camellia command and decryption with openssl command (CTR mode, the plaintext output has redundant trailing bytes):
$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ctr -R 0123456789abcdef | openssl enc -d -camellia-128-ctr -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574 This is a secret
00000010: 206d 6573 7361 6765 2e0a 0606 0606 0606 message........
$
Love this tool and currently learning to use it. ❤️
I followed one tutorial linked in the readme and noticed it crashes as soon as you try to decrypt the zip with a variable.
Here's the stack trace:
Traceback (most recent call last):
File "/home/<redacted>/Tools/refinery/refinery/units/__init__.py", line 1256, in normalized_action
yield from (x for x in result if x is not None)
File "/home/<redacted>/Tools/refinery/refinery/units/__init__.py", line 1256, in <genexpr>
yield from (x for x in result if x is not None)
File "/home/<redacted>/Tools/refinery/refinery/units/__init__.py", line 681, in <genexpr>
return (self.labelled(r) for r in result)
File "/home/<redacted>/Tools/refinery/refinery/units/formats/__init__.py", line 114, in process
results: List[UnpackResult] = list(self.unpack(data))
File "/home/<redacted>/Tools/refinery/refinery/units/formats/archive/xtzip.py", line 36, in unpack
archive.setpassword(self.args.pwd)
File "/usr/lib/python3.10/zipfile.py", line 1449, in setpassword
raise TypeError("pwd: expected bytes, got %s" % type(pwd).__name__)
TypeError: pwd: expected bytes, got ByteStringWrapper
There seems to be a typecheck inside the zipfile library code. Might be due to version changes, not sure. Anyway, I fixed it for you. PR incoming. :)
The fix in #46 doesn't address the use case where I'm encountering this same issue:
C:\Users\Donald\repositories\refinery>ef C:\Users\Donald\Desktop\install.log
(10:15:25) failure in ef: exception of type ValueError;
'\\\\?\\C:\\Users\\Donald\\Desktop\\install.log' is not in the subpath of '\\\\?\\C:\\Users\\Donald\\repositories\\refinery' OR one path is relative and the other is absolute.
In this case, I'm trying to ef
a file that's not in a subfolder relative to my CWD.
Windows 11
Python 3.11.9
Up-to-date including bugfix
Originally posted by @Donaldduck8 in #46 (comment)
MSI files are already parsed well using the xtdoc unit. However, the embedded streams are often incorrectly labeled with nonsensical Chinese characters
Example (from https://bazaar.abuse.ch/sample/a00ae529a1ce4e2eb7f988ed5adaa07e653de3fe0c01bec9452aae71b935f63c/)
$ emit Projects/Malware/AdobePDFReader.msi | xtdoc -l
[5]SummaryInformation
㡿㪆㦍㪄㫊㮅㨁㠀㬇㫄㩅㮌㣍㠍㥁㧆䠋
䡀㬿䏲䐸䖱
䡀㼿䕷䑬㭪䗤䠤
䡀㼿䕷䑬㹪䒲䠯
䡀㽿䅤䈯䠶
䡀㿿䏤䇬䗤䒬䠱
䡀䇊䌰㮱䈻䘦䈷䈜䘴䑨䈦
䡀䇊䗹䛎䆨䗸㼨䔨䈸䆱䠨
䡀䈏䗤䕸㬨䐲䒳䈱䗱䠶
䡀䈏䗤䕸䠨
䡀䈖䌧䠤
䡀䌍䈵䗦䕲䠼
䡀䌏䈯
䡀䑒䗶䏤㮯䈻䘦䈷䈜䘴䑨䈦
䡀䒌䓰䑲䑨䠷
䡀䒖䘧䈯䌜䑪䗤䕸䠨
䡀䕙䓲䕨䜷
䡀䘌䗶䐲䆊䌷䑲
Didier Stevens wrote plugin for oledump (https://github.com/DidierStevens/DidierStevensSuite/blob/master/plugin_msi_info.py ) that handles this issue & properly displays the correct embedded names. The cleaned up stream names make it much easier to discern what the MSI will drop and easier to select what stream to examine next or dump out
I don't know if its worth introducing a new unit (like xtmsi) or to just build that encoding fix into xtdoc and use it as a flag inside of the unit :)
Streams:
1 620 SummaryInformation
2 360 MPB_ExternalFiles.RunAction1_File1
3 182784 MPB_ExternalFiles.RunAction2_File2
4 2609864 Binary.CustomActionsModuleX64
5 728 !_Columns
6 36 !MPB_RunActions
7 24 !MPB_ExternalFiles
8 12676 !_StringData
9 1756 !_StringPool
10 40 !_Tables
11 2232 !_Validation
12 420 !ActionText
13 48 !AdminExecuteSequence
14 24 !AdminUISequence
15 42 !AdvtExecuteSequence
16 4 !FeatureComponents
17 16 !Feature
18 4 !Binary
19 12 !Directory
20 144 !InstallExecuteSequence
21 72 !InstallUISequence
22 12 !Component
23 16 !Upgrade
24 96 !Error
25 48 !Property
26 156 !CustomAction
Second comparison (truncated)
$ emit f0b6d6981e06c7be2e45650e5f6d39570c1ee640ccb157ddfe42ee23ad4d1cdb.bin | xtdoc -l
[5]SummaryInformation
䌋䄱䜵㷾䚨
䌋䄱䜵㹾䚲䕨䋜䏨㼯䕦䓬㵷䘤䆱䈫䞵䏧䠯
䌋䄱䜵㾾䠳
䌋䄱䜵䄾䆬䖸䄷䗦䇾䏯
䌋䄱䜵䄾䖬䋦䇨䏸䕨䇾䏯
䌋䄱䜵䄾䖬䋦䇨䏸䕨䞂䏧䠯
䌋䄱䜵䅾䑤䈱䠵
䌋䄱䜵䆾䇰䌯䎱䕤䒵䠺
䌋䄱䜵䆾䐲䏳䗨䠬
䌋䄱䜵䆾䖸䌷䒦䠱
䌋䄱䜵䇾䄬䒯䠪
䌋䄱䜵䈾䆻䄯䌰䠦
䌋䄱䜵䌾䉱䠲
䌋䄱䜵䌾䖱䌷䒦䠱
䌋䄱䜵䕾䐨䙲䆬䠲
䌋䄱䜵䕾䓨䌤䌵䠦
䌋䄱䜵䗾䅤䄥䎦
䡀㬿䏲䐸䖱
䡀㲊㼿䋦䇨䏸䇨䄝䎶䠶
䡀㲞䈝䗻
䡀㼿䕷䑬㭪䗤䠤
䡀㼿䕷䑬㹪䒲䠯
䡀㽿䅤䈯䠶
䡀㿿䏤䇬䗤䒬䠱
䡀䄕䑸䋦䒌䇱䗬䒬䠱
䡀䄛䌧㫲䗸䒷䠱
䡀䆊䌷䑲䈝䗻
䡀䇊䌰㮱䈻䘦䈷䈜䘴䑨䈦
䡀䇊䌰㾱㼒䔨䈸䆱䠨
䡀䇊䗹䛎䆨䗸㼨䔨䈸䆱䠨
䡀䈏䗤䕸㬨䐲䒳䈱䗱䠶
䡀䈏䗤䕸䠨
䡀䈛䌪䗶䜵
䡀䈝䗻䗜䏼䠨
䡀䋌䆨㫮䛲
䡀䌋䄱䜵
䡀䌍䈵䗦䕲䠼
䡀䌍䏤䊲
䡀䑒䗶䏤㮯䈻䘦䈷䈜䘴䑨䈦
䡀䑒䗶䏤㾯㼒䔨䈸䆱䠨
䡀䒋䗲䗶䄵䓳䕨㲞䈜䘴䑨䈦
䡀䒌䓰䑲䑨䠷
䡀䒌䗱䒵㬯䑲䌧䌷䑲
䡀䒌䗱䒵㮯䈹䗱
䡀䒌䗱䒵䠯
䡀䓞䕪䇤䠨
䡀䕌䄨䈷䒏䇯䕨
䡀䕎䒵䠵
䡀䕙䓲䕨䜷
䡀䘌䗶䐲䆊䌷䑲
䡀䙎䑨㶷䓤䌳䊱
$ python3 oledump.py -p plugin_msi_info.py f0b6d6981e06c7be2e45650e5f6d39570c1ee640ccb157ddfe42ee23ad4d1cdb.bin
Streams:
1 316 SummaryInformation
2 318 Binary.New
3 627864 Binary.PowerShellScriptLauncher.dll
4 318 Binary.Up
5 390304 Binary.aicustact.dll
6 146072 Binary.aischeduler.dll
7 183448 Binary.aischeduler2.dll
8 2818 Binary.banner
9 2862 Binary.cmdlinkarrow
10 2998 Binary.completi
11 2998 Binary.custicon
12 11791 Binary.dialog
13 766 Binary.exclamic
14 1078 Binary.info
15 2998 Binary.insticon
16 2998 Binary.removico
17 2998 Binary.repairic
18 854 Binary.tabback
19 1480 !_Columns
20 32 !AI_ScheduledTasks
21 444 !UIText
22 84211 !_StringData
23 7560 !_StringPool
24 76 !_Tables
25 4440 !_Validation
26 16 !LaunchCondition
27 36 !RadioButton
28 480 !ActionText
29 48 !AdminExecuteSequence
30 66 !AdminUISequence
31 72 !AdvtExecuteSequence
32 8 !FeatureComponents
33 16 !Feature
34 24 !Registry
35 48 !TextStyle
36 12 !CheckBox
37 68 !Binary
38 12 !Directory
39 616 !Dialog
40 462 !InstallExecuteSequence
41 180 !InstallUISequence
42 54 !BootstrapperUISequence
43 24 !Component
44 72 !ControlCondition
45 1536 !ControlEvent
46 7280 !Control
47 32 !Upgrade
48 4 !CreateFolder
49 2564 !Error
50 332 !Property
51 372 !CustomAction
52 128 !EventMapping
When reading this specific IFPS file with the IFPS unit, the following error happens:
exception of type AttributeError; 'NoneType' object has no attribute 'represent'
When trying to read it from a python prompt, we can get the full stack trace:
File ".../refinery/lib/structures.py", line 696, in wrapped__init__
original__init__(self, reader, *args, **kwargs)
File ".../refinery/units/formats/ifps.py", line 592, in __init__
self._load_functions()
File ".../refinery/units/formats/ifps.py", line 718, in _load_functions
instruction.operands[0] = t.reference()
^^^^^^^^^^^^^
File ".../refinery/units/formats/ifps.py", line 459, in reference
return self.decl.represent(self.name, ref=True)
^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'represent'
From my limited understanding, seven functions (with the first being GETARRAYLENGTH) are imported, but not exported, so the value of decl
stays with the default None
. From looking at other IFPS files, those functions are usually exported too, to result in a line similar to external function GETARRAYLENGTH();
.
The original InnoSetup file gives out an IFPS file named CompiledCode.bin (attached, with password binref). This is the file causing the issue.
I ran IFPSTools.NET on the CompiledCode.bin and got this resulting CompiledCode.txt.zip (attached, with password binref), in case it could help in any way.
Another function that is causing problem in this file is EXPANDCONSTANT. It gets decompiled as the following with IFPSTools:
.function external void EXPANDCONSTANT
Finding the same function in another IFPS file that does not cause an issue in BinaryRefinery gives:
.function(import) external internal returnsval EXPANDCONSTANT(__in __unknown)
So it may point out to a corruption in the IFPS file, as IFPSTools does not look to handle it that much more cleanly either.
The xtnsis fails to properly parse some NSIS files such as this one:
https://mega.nz/file/QrsG0STY#4uAml4wfCe8-aAUHIXGuHDMK9Po7JmZgrTyQpYwCbLw
When attempting to parse, the user is returned an error stating "failure in xtnsis: exception of type EOF; Unexpected end of buffer."
Attempting to use the xtnsis to list or extract from this NSIS installer will produce the error.
emit malware.exe | xtnsis -l
(07:05:29) failure in xtnsis: exception of type EOF; Unexpected end of buffer.
or
emit malware.exe | xtnsis [| dump archive/{path} ]
(07:09:29) failure in xtnsis: exception of type EOFError
I use an adapted copy of xtnsis in my debloat tool. When debugging debloat, the error occurs in the "read_exactly" method an returns the following, which I THINK suggests the error is occurring while parsing the NSIS Script, so perhaps there is another missing instruction or something?:
Exception has occurred: EOF
End of File
File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 329, in read_exactly
raise EOF(data)
File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 376, in read_integer
result = int.from_bytes(self.read_exactly(bytecount, peek), self.byteorder_name)
File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 468, in u32
return self.read_integer(32, peek)
File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 413, in <listcomp>
self.arguments = [reader.u32() for _ in range(6)]
File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 413, in __init__
self.arguments = [reader.u32() for _ in range(6)]
File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 575, in wrapped__init__
original__init__(self, reader, *args, **kwargs)
File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 988, in <listcomp>
self.instructions: List[NSScriptInstruction] = [NSScriptInstruction(reader) for _ in range(self.block_header_entries.size)]
File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 988, in __init__
self.instructions: List[NSScriptInstruction] = [NSScriptInstruction(reader) for _ in range(self.block_header_entries.size)]
File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 575, in wrapped__init__
original__init__(self, reader, *args, **kwargs)
File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 1148, in __init__
self.header = NSHeader(header_data, size=header_size)
File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 575, in wrapped__init__
original__init__(self, reader, *args, **kwargs)
File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 1309, in unpack
raise _error
File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 1309, in unpack
raise _error
File "/home/Documents/GitHub/debloat/src/debloat/processor.py", line 114, in check_and_extract_NSIS
extracted_files = extractor.unpack(memoryview(pe.__data__))
```
I have combed through the documentation but can't seem to find this feature. I would like to be able to perform loops through data similar to the Jump and Conditional Jump instructions in CyberChef. I've provided a simple example below but obviously it could be made more complicated.
Jump - Reverse string & b64 decode 21 times
Conditional Jump - Same as above but loop until the data matches 'hello' or until 1000 iterations
The closest thing I can see is the alu operation, but this seems to be for mathematical operations, or maybe I am missing something
Consider adding support for converting from base92.
This addition would broaden its functionality and cater to users who may encounter data encoded in this format.
Regards, Moshe
In fcc36d8, I added some extremely weak PowerShell support, this was improved slightly in 504f015. The main issues are the following:
Our current workaround is to:
If either of the above two PowerShell issues are ever resolved in a way that supports the binary refinery design, the workaround should be removed because it introduces an unnecessary encoding step.
When running the vstack
unit on an installation of Binary Refinery in a Python virtual environment running Python 3.12, I kept seeing the following error:
failure in vstack: dependency unicorn is missing; run pip install intervaltree capstone unicorn
This was despite verifying inside the virtual environment that the listed dependencies were present.
Install Binary Refinery with the binary-refinery[all]
spec in an environment which uses Python 3.12, then execute the vstack
unit.
Re-installing refinery inside a virtual environment with Python 3.11.6 fixes this issue.
As far as I can tell, this is the reason for this issue:
distutils
module from the standard library (which was deprecated starting in 3.10: https://docs.python.org/3.12/whatsnew/3.10.html#distutils-deprecated)unicorn
when executing the vstack
unit, which results in a RefineryImportMissing
exception, which is reported by refinery as a missing dependency.Verbose error and traceback:
$ vstack -v -v
(13:31:14) failure in vstack: dependency unicorn is missing; run pip install unicorn capstone intervaltree
Traceback (most recent call last):
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1251, in __get__
self.module = module = self.fget()
^^^^^^^^^^^
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/formats/exe/vstack.py", line 74, in _unicorn
import unicorn
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/unicorn/__init__.py", line 4, in <module>
from .unicorn import Uc, uc_version, uc_arch_supported, version_bind, debug, UcError, __version__
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/unicorn/unicorn.py", line 5, in <module>
import distutils.sysconfig
ModuleNotFoundError: No module named 'distutils'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1391, in normalized_action
yield from (x for x in result if x is not None)
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1391, in <genexpr>
yield from (x for x in result if x is not None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 806, in <genexpr>
return (Chunk.Wrap(r) for r in result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/formats/exe/vstack.py", line 146, in process
uc = self._unicorn
^^^^^^^^^^^^^
File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1257, in __get__
raise RefineryImportMissing(self.dependency, *args) from E
refinery.units.RefineryImportMissing
When reading this specific IFPS file with the IFPS unit, the following error happens:
failure in ifps: exception of type EOFError
The original InnoSetup file gives out an IFPS file named CompiledCode.bin (attached, with password binref
). If we emit
that file into ifps
or ifpsstr
, we get the previous error.
The error comes from the sixth function (kernel32.dll::GetTickCount) which is imported and is a dll, but doesn't seem to have enough data to read the return type.
I see that the IFPS unit is based on IFPSTools, which was superseeded by IFPSTools.NET. I ran IFPSTools.NET on the same file and it gave a decoded file back claiming that function is void=True.
To get the CompiledCode.bin out of the InnoSetup file, we can use innounp on Windows/Wine with the password A1ADB8BE8E677894E
, or a custom version of innoextract (which doesn't need the password) on Linux.
I have noticed that in my environment, the iemap
unit does not produce any colour, which unfortunately makes the unit not very useful. Other units which produce colour, such as peek
, do emit colour in their output as expected.
This behaviour is reproducible on both Linux and MacOS. Just for reproducing this bug, I created a new Python virtual environment on both the Linux and MacOS machines, and did a fresh install of Binary Refinery inside it, via pip install binary-refinery[all]
.
I have included screenshots below of the output of the peek
unit, as well as the iemap
unit. I passed the --legend
flag to iemap
in this case, and expect to see colours in at least the numbered legend to the left of the entropy graph.
When calling ./update.ps1
I get a couple of log lines starting with DEPRECATION
. I think this is the important snippet from the output:
Successfully built binary-refinery jsbeautifier
Installing collected packages: z3-solver, win-unicode-console, unicorn, unicodecsv, texttable, sortedcontainers, red-black-tree-mod, pyxlsb2, pywin32, python-magic, pysmt, pyperclip, pyelftools, pycdlib, ply, phpserialize, mulpyplexer, msgpack, mpmath, LnkParse3, lark-parser, javaobj-py3, itanium-demangler, hexdump, enum-compat, editorconfig, ebcdic, easygui, dpkt, compressed-rtf, cabarchive, brotli, asn1crypto, altgraph, aenum, xlrd2, wheel, tzdata, typing_extensions, toml, tbtrim, sympy, soupsieve, smmap, six, scapy, roman, pyzstd, python-utils, python-registry, python-magic-win64, pyppmd, pyparsing, pyonenote, pycryptodomex, pycryptodome, pycparser, pybcj, psutil, protobuf, plumbum, Pillow, pefile, olefile, numpy, networkx, multivolumefile, macholib, intervaltree, inflate64, future, et-xmlfile, dictdumper, defusedxml, decorator, CppHeaderParser, configparser, colorclass, colorama, chardet, capstone, cachetools, bitstring, archinfo, ailment, untangle, rpyc, pytz-deprecation-shim, PyPDF2, pypcapkit, py7zr, progressbar2, openpyxl, nampa, more-itertools, jsbeautifier, imapclient, gitdb, click, claripy, cffi, beautifulsoup4, zipp, xdis, tzlocal, spark-parser, pyvex, GitPython, cryptography, uncompyle6, python-evtx, msoffcrypto-tool, decompyle3, cle, XLMMacroDeobfuscator, angr, pcodedmp, oletools, RTFDE, extract-msg, binary-refinery
DEPRECATION: win-unicode-console is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for win-unicode-console ... done
DEPRECATION: red-black-tree-mod is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for red-black-tree-mod ... done
DEPRECATION: mulpyplexer is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for mulpyplexer ... done
DEPRECATION: hexdump is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for hexdump ... done
DEPRECATION: scapy is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for scapy ... done
DEPRECATION: intervaltree is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for intervaltree ... done
DEPRECATION: future is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for future ... done
DEPRECATION: CppHeaderParser is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for CppHeaderParser ... done
DEPRECATION: pypcapkit is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for pypcapkit ... done
Successfully installed CppHeaderParser-2.7.4 GitPython-3.1.31 LnkParse3-1.2.0 Pillow-9.5.0 PyPDF2-3.0.1 RTFDE-0.0.2 XLMMacroDeobfuscator-0.2.7 aenum-3.1.12 ailment-9.2.46 altgraph-0.17.3 angr-9.2.46 archinfo-9.2.46 asn1crypto-1.5.1 beautifulsoup4-4.11.2 binary-refinery-0.6.1 bitstring-4.0.1 brotli-1.0.9 cabarchive-0.2.4 cachetools-5.3.0 capstone-4.0.2 cffi-1.15.1 chardet-5.1.0 claripy-9.2.46 cle-9.2.46 click-8.1.3 colorama-0.4.6 colorclass-2.2.2 compressed-rtf-1.0.6 configparser-4.0.2 cryptography-40.0.1 decompyle3-3.9.0 decorator-5.1.1 defusedxml-0.7.1 dictdumper-0.8.4.post2 dpkt-1.9.8 easygui-0.98.3 ebcdic-1.1.1 editorconfig-0.12.3 enum-compat-0.0.3 et-xmlfile-1.1.0 extract-msg-0.40.0 future-0.18.3 gitdb-4.0.10 hexdump-3.3 imapclient-2.3.1 inflate64-0.3.1 intervaltree-3.1.0 itanium-demangler-1.1 javaobj-py3-0.4.3 jsbeautifier-1.14.7 lark-parser-0.12.0 macholib-1.16.2 more-itertools-5.0.0 mpmath-1.3.0 msgpack-1.0.5 msoffcrypto-tool-5.0.1 mulpyplexer-0.9 multivolumefile-0.2.3 nampa-0.1.1 networkx-3.1 numpy-1.24.2 olefile-0.46 oletools-0.60.1 openpyxl-3.1.2 pcodedmp-1.2.6 pefile-2023.2.7 phpserialize-1.3 plumbum-1.8.1 ply-3.11 progressbar2-4.2.0 protobuf-4.22.3 psutil-5.9.4 py7zr-0.20.4 pybcj-1.0.1 pycdlib-1.14.0 pycparser-2.21 pycryptodome-3.17 pycryptodomex-3.17 pyelftools-0.29 pyonenote-0.0.2 pyparsing-2.4.7 pypcapkit-0.15.5 pyperclip-1.8.2 pyppmd-1.0.0 pysmt-0.9.5 python-evtx-0.7.4 python-magic-0.4.13 python-magic-win64-0.4.13 python-registry-1.3.1 python-utils-3.5.2 pytz-deprecation-shim-0.1.0.post0 pyvex-9.2.46 pywin32-306 pyxlsb2-0.0.9 pyzstd-0.15.6 red-black-tree-mod-1.20 roman-4.0 rpyc-5.3.1 scapy-2.5.0 six-1.16.0 smmap-5.0.0 sortedcontainers-2.4.0 soupsieve-2.4 spark-parser-1.8.9 sympy-1.11.1 tbtrim-0.3.1 texttable-1.6.7 toml-0.10.2 typing_extensions-4.5.0 tzdata-2023.3 tzlocal-4.2 uncompyle6-3.9.0 unicodecsv-0.14.1 unicorn-2.0.1.post1 untangle-1.2.1 wheel-0.40.0 win-unicode-console-0.5 xdis-6.0.5 xlrd2-1.3.4 z3-solver-4.10.2.0 zipp-1.0.0
Open a PowerShell window, navigate to the refinery directory and call ./update.ps1
.
The log lines reference pypa/pip#8559 so I assume this is caused by an upstream change in pip behavior.
ef command of version 0.6.36 cannot open relative file path. Version 0.6.35 does not have this issue.
$ cd /tmp
$ echo sample > sample.txt
$ ef sample.txt
(00:21:05) failure in ef: exception of type ValueError; 'sample.txt' is not in the subpath of '/tmp' OR one path is relative and the other is absolute.
$ ef ./sample.txt
(00:22:10) failure in ef: exception of type ValueError; 'sample.txt' is not in the subpath of '/tmp' OR one path is relative and the other is absolute.
$ ef ../tmp/sample.txt
(00:24:53) failure in ef: exception of type ValueError; '../tmp/sample.txt' is not in the subpath of '/tmp' OR one path is relative and the other is absolute.
$ ef /tmp/sample.txt
sample
$
I can't get the usage example for push
to work:
emit key=value | push [[| rex =(.*)$ $1 | pop v ]| repl var:v censored ]
First off, the rex
syntax seems to have changed, the $1
will be taken literally:
❯ r.emit "key=value" | r.rex '=(.*)' '$1'
$1
Instead it should probably be {1}
:
❯ r.emit "key=value" | r.rex '=(.*)' '{1}'
value
But even after correcting this it still fails because the variable is not defined:
> r.emit "key=value" | r.push -vv [ [ | r.rex -vv '=(.*)' '{1}' | r.pop -vv v ] | r.repl --v 'var:v' censored ]
(07:30:25) verbose in rex: regular expression: re.compile(b'=(.*)', re.DOTALL)
(07:30:25) verbose in pop: buffering invisible chunk
--- Logging error ---
Traceback (most recent call last):
File "/home/zac/.local/lib/python3.8/site-packages/refinery/lib/argformats.py", line 621, in extract
result = meta[name]
File "/home/zac/.local/lib/python3.8/site-packages/refinery/lib/meta.py", line 500, in __getitem__
item = super().__getitem__(key)
File "/home/zac/.local/lib/python3.8/site-packages/refinery/lib/meta.py", line 511, in __missing__
raise KeyError(F'The meta variable {key} is unknown.')
KeyError: 'The meta variable v is unknown.'
Maybe I'm doing the whole variables thing wrong, because even this fails becase k
is not set:
> r.emit "test" | r.put -vv k "test" | r.repl -vv var:k hello
Message: 'delayed argument initialization failed:'
Arguments: ('The variable k is not defined.',)
I'm using version Version: 0.4.6 of binary-refinery, on Ubuntu with bash.
Unit Request for machometa
Requesting a new unit to perform similar functionality as pemeta, but for macho executables. The data requested inside of this would ideally include:
Header info (machine arch, num of load commands, etc - this is already parsed in peek)
Dylibs (externally linked dependencies / libs - lief can parse this )
Code-signing information (mostly just the name but if extracting other information is readily do-able, that would be great! Example parsing in Go: https://github.com/blacktop/go-macho/blob/master/pkg/codesign/codesign.go )
Imports/Exports/Symbols (optional flag - lief can parse this information)
Class Information (Not necessary but cool as an option flag - ObjC files only) (via class-dump - not python :( https://github.com/nygard/class-dump ))
I realize I'm not giving a ton of great info here but I'm currently using a bunch of Apple utilities or python snippets to triage Macho samples and I figured others would benefit from this being in binref :)
import sys
import lief
from hashlib import md5
import argparse
import os
def main():
parser = argparse.ArgumentParser(description="Macho Sig Name Extraction")
parser.add_argument("-f", "--file", help="Specify file to parse", metavar="<file>", required=False)
args = parser.parse_args()
result = sig_reading(args.file)
if result is not None:
print("\tTarget File: \t" + result[0])
print("\tFile MD5: \t" + result[1])
print("\tSig Name: \t" + result[2])
print("\tDylibs: \t")
for i in result[3]:
print("\t\t"+i)
def sig_reading(target):
dylibs = []
try:
target_macho = open(target, mode="rb")
contents = target_macho.read()
file_hash = md5(contents).hexdigest()
parsed_macho = lief.parse(target)
for lib in parsed_macho.libraries:
dylibs.append(lib.name.lower())
cs_sign_dir_offset = parsed_macho.code_signature.data_offset
# read the big CS directory & get ptr to 0th blob
target_macho.seek(cs_sign_dir_offset)
cs_dir_bytes = target_macho.read(0x20)
jump_to_blob = cs_dir_bytes[19]
# read the 0th blob and look for ident ofset
target_macho.seek(cs_sign_dir_offset+jump_to_blob)
first_codesign_blob = target_macho.read(0x20)
jump_to_ident = first_codesign_blob[23]
# read identifier string
target_macho.seek(cs_sign_dir_offset+jump_to_blob+jump_to_ident)
ident_str = target_macho.read(0x30)
sig_name = str(ident_str)
sig_name = sig_name.split('\\x00')[0]
sig_name = sig_name[2:]
target_macho.close()
except:
print("not a macho")
return
sig_list = [target,file_hash,sig_name,dylibs]
return sig_list
main()
rc5 and rc6 cannot encrypt with CTR mode since version 0.5.10.
Master:
$ echo -n "This is a secret message." | rc5 -m ctr -i 01234567 -R 0123456789abcdef
(00:44:40) failure in rc5: exception of type ValueError; Information in counter object does not align with block size.
$ echo -n "This is a secret message." | rc6 -m ctr -i 0123456789abcdef -R 0123456789abcdef
(00:45:03) failure in rc6: exception of type ValueError; Information in counter object does not align with block size.
$
Version 0.5.9 works:
$ echo -n "This is a secret message." | rc5 -m ctr -i 01234567 -R 0123456789abcdef | rc5 -m ctr -i 01234567 0123456789abcdef
This is a secret message.
$ echo -n "This is a secret message." | rc6 -m ctr -i 0123456789abcdef -R 0123456789abcdef | rc6 -m ctr -i 0123456789abcdef 01
23456789abcdef
This is a secret message.
$
It is possible to weaponize .chm files but binref can't extract this files.
There is a python lib PyCHM but this is just a wrapper for this c lib CHMLib. The c lib needs a string with the path to the chm file to open it. This is against the binref code of conduct.
I think the only solution would be to implement the algorithm in python as a new binref unit.
@huettenhain can you prove that there isn't a other way to extract chm files with binref? If so I can start to develop a new unit.
Malicious-CHM-Guide.md
AgentTesla Spreads Through CHM and PDF Files in Recent Attacks
Cryptowall Makes a Comeback Via Malicious Help Files (CHM)
Harmful Help: Analyzing a Malicious Compiled HTML Help File Delivering Agent Tesla
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.