vanrein / quick-der Goto Github PK
View Code? Open in Web Editor NEWQuick `n' Easy DER library
License: Other
Quick `n' Easy DER library
License: Other
This applies to the asn2quickder tool, which generates code (or anything else) from ASN.1 formats, such as the RFCs that we habitually include in this project.
Seeing that ASN.1 references symbols all over the place, it is useful to be able to map the notation to a hyperlinked document. HTML is a quick win, but a more general format such as TexInfo is probably better, since it maps to so many other forms.
I won't have the time for it, so you're welcome to pitch in what you've got... it's a great task for a beginning Python programmer, and very useful too :-)
[moved here from an old repo issue]
they are installed in <PREFIX>/lib/cmake/Quick-DER
but I'm quite sure that should be <PREFIX>/share/cmake/Modules
See https://github.com/vanrein/quick-der/tree/python3
Branch python3 created to do this work.
This is a low priority and welcomes submissions :)
In lib/der_unpack.c, function derwalk(), there's the following code:
if (*walk & DER_PACK_STORE) {
memset (outarray + (*outctr)++,
The DER_PACK_STORE is a weird "bit flag" since it has value 0x00 (see api.h), Possibly this is intended:
((*walk & DER_PACK_MATCHBITS) == DER_PACK_STORE), but it could just as easily be intended the other way around -- I can't quickly tell why that memset is there.
This commit was made for Windows compliance, but it breaks POSIX. It should be reverted or made more general.
When working on LillyDAP (meaning "little LDAP"), I first ran into the use of IMPLICIT tags and this is not supported properly.
For instance, this is a SearchResultDone
message:
30 0c 02 01 01 65 07 0a 01 00 04 00 04 00
Decoded with derdump, this becomes
SEQUENCE (OF): tag 0x30 #12 @0 ^0, Universal, Constructed
INTEGER: tag 0x02 #1 @2 ^1, Universal, Primitive
01 == 1
[APPLICATION 5]: tag 0x65 #7 @5 ^1, Application, Constructed
ENUMERATED: tag 0x0a #1 @7 ^2, Universal, Primitive
00 == "."
OCTETSTRING: tag 0x04 #0 @10 ^2, Universal, Primitive
OCTETSTRING: tag 0x04 #0 @12 ^2, Universal, Primitive
Note the absense of the SEQUENCE (OF)
immediately underneath the [APPLICATION 5]
tag for SearchResultDone
.
In conflict with this, Quick DER currently generates the following in rfc4511.h
:
#define DER_PACK_rfc4511_LDAPResult \
DER_PACK_ENTER | DER_TAG_SEQUENCE, \
DER_PACK_STORE | DER_TAG_ENUMERATED, \
DER_PACK_rfc4511_LDAPDN, \
DER_PACK_rfc4511_LDAPString, \
DER_PACK_OPTIONAL, \
DER_PACK_ENTER | DER_TAG_CONTEXT(3), \
DER_PACK_rfc4511_Referral, \
DER_PACK_LEAVE, \
DER_PACK_LEAVE
#define DER_PACK_rfc4511_SearchResultDone \
DER_PACK_ENTER | DER_TAG_APPLICATION(5), \
DER_PACK_rfc4511_LDAPResult, \
DER_PACK_LEAVE
This assumes that a SEQUENCE
tag will be found underneath [APPLICATION 5]
. A suggested improvement could be:
#define DER_PACK_rfc4511_LDAPResult_implicit \
DER_PACK_STORE | DER_TAG_ENUMERATED, \
DER_PACK_rfc4511_LDAPDN, \
DER_PACK_rfc4511_LDAPString, \
DER_PACK_OPTIONAL, \
DER_PACK_ENTER | DER_TAG_CONTEXT(3), \
DER_PACK_rfc4511_Referral, \
DER_PACK_LEAVE
#define DER_PACK_rfc4511_LDAPResult \
DER_PACK_ENTER | DER_TAG_SEQUENCE, \
DER_PACK_rfc4511_LDAPResult_implicit, \
DER_PACK_LEAVE
#define DER_PACK_rfc4511_SearchResultDone \
DER_PACK_ENTER | DER_TAG_APPLICATION(5), \
DER_PACK_rfc4511_LDAPResult_implicit, \
DER_PACK_LEAVE
Until this is fixed, LillyDAP must remain on hold :'-(
( cd build && cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_PREFIX_PATH=/usr/local )
CMake Error at CMakeLists.txt:8 (find_package):
Could not find a package configuration file provided by "ARPA2CM" with any
of the following names:
ARPA2CMConfig.cmake
arpa2cm-config.cmake
Add the installation prefix of "ARPA2CM" to CMAKE_PREFIX_PATH or set
"ARPA2CM_DIR" to a directory containing one of the above files. If
"ARPA2CM" provides a separate development package or SDK, be sure it has
been installed.
-- Configuring incomplete, errors occurred!
make: *** [build/CMakeCache.txt] Error 1
The PATH and WALK notations are useful for manual construction of a DER parser, but they require at least an intuition about how DER works. It is possible to derive it from ASN.1 directly though, and a compiler would be really helpful.
Although we mentioned libtasn1 before, the focus has shifted to asn1ate because it appears to be more complete, and has a dynamic backend structure built upon a semantic model and even some global analysis (dependencies).
Having a compiler in Python is not problematic, and may even simplify dependencies. The purpose of this compiler is mainly intended for building environments, and inasfar as RFCs and other standards are mapped with it, we intend to prepare it for #include <quick-der/rfc5280.h>
to make DER Quick... and Easy.
With the new support for IMPLICIT
tags such as used by LDAP, many things now work better. But there are exceptions not yet solved. Take this LDAPMessage
for instance,
SEQUENCE (OF): tag 0x30 #12 @0 ^0, Universal, Constructed
INTEGER: tag 0x02 #1 @2 ^1, Universal, Primitive
01 == 1
[APPLICATION 0]: tag 0x60 #7 @5 ^1, Application, Constructed
INTEGER: tag 0x02 #1 @7 ^2, Universal, Primitive
03 == 3
OCTETSTRING: tag 0x04 #0 @10 ^2, Universal, Primitive
[0]: tag 0x80 #0 @12 ^2, Contextual, Primitive
Note how nothing is contained in the [0]
tag on the last line. This is possible when the contained information is elementary, empty and implicitly tagged. This is indeed the case, if we look at the definitions:
BindRequest ::= [APPLICATION 0] SEQUENCE {
version INTEGER (1 .. 127),
name LDAPDN,
authentication AuthenticationChoice }
AuthenticationChoice ::= CHOICE {
simple [0] OCTET STRING,
-- 1 and 2 reserved
sasl [3] SaslCredentials,
... }
This however, translates to:
#define DER_PACK_rfc4511_BindRequest \
DER_PACK_ENTER | DER_TAG_APPLICATION(0), \
DER_PACK_STORE | DER_TAG_INTEGER, \
DER_PACK_rfc4511_LDAPDN, \
DER_PACK_rfc4511_AuthenticationChoice, \
DER_PACK_LEAVE
#define DER_PACK_rfc4511_AuthenticationChoice \
DER_PACK_CHOICE_BEGIN, \
DER_PACK_ENTER | DER_TAG_CONTEXT(0), \
DER_PACK_STORE | DER_PACK_ANY, \
DER_PACK_LEAVE, \
DER_PACK_ENTER | DER_TAG_CONTEXT(3), \
DER_PIMP_rfc4511_SaslCredentials, \
DER_PACK_LEAVE/* ...ASN.1 extensions... */, \
DER_PACK_CHOICE_END
The problem lies in the DER_PACK_STORE | DER_PACK_ANY
nested within the DER_PACK_ENTER | DER_TAG_CONTEXT(0)
of DER_PACK_rfc4511_AuthenticationChoice
. This does not seem to capture an empty string (and probably nothing that is not proper DER-encoded data, so header / length / value format.
What needs to be done, is replace these three lines
DER_PACK_ENTER | DER_TAG_CONTEXT(0), \
DER_PACK_STORE | DER_PACK_ANY, \
DER_PACK_LEAVE, \
with the single line
DER_PACK_STORE | DER_TAG_CONTEXT(0), \
This way, the structure is stored without any attempt to entering it -- which is the intention.
This applies to TLS-KDH, just preceding the checkin of arpa2/tlspool@c78ab02
In Kerberos' Authenticator
:
Authenticator ::= [APPLICATION 2] SEQUENCE {
authenticator-vno [0] INTEGER (5),
crealm [1] Realm,
cname [2] PrincipalName,
cksum [3] Checksum OPTIONAL,
cusec [4] Microseconds,
ctime [5] KerberosTime,
subkey [6] EncryptionKey OPTIONAL,
seq-number [7] UInt32 OPTIONAL,
authorization-data [8] AuthorizationData OPTIONAL
}
The parts [7] UInt32 OPTIONAL
with no value of that kind in the corresponding dercursor
, run through der_pack()
, leads to the encoding a7 00
. Similarly [8] AuthorizationData OPTIONAL
is packed as a8 00
. The result is unacceptable to der_unpack()
.
In other words, the OPTIONAL
is not applied to the outside tag and only to the inner real type. This is wrong.
Current Quick DER refuses tags over 30, but semantics of an extension are not dramatic, in that tag 31 indicates that a length octet follows and then that many bytes.
This would make an interesting utility for storing
The parser may be updated to treat tag 31 specially, without loosing much of its other functions; the WALK
and PACK
notations can be adapted to accommodate inlined exact matches. Only CHOICE
would really suffer if it had to match this style of DER code. It may be simply enough to adapt der_header()
to deliver tag 31 and have it skip ahead, and let the user program match the exact string that follows.
This may be so doable we should do it. Just add a hint below if you have an aplication that might convince us!
Looks like something is missing:
$ make
...
make[1]: Entering directory '/usr/oms/src/quick-der/rfc'
PYTHONPATH='../tool/asn1ate' ../tool/asn1ate/asn1ate/asn2quickder.py rfc3280.asn1
/bin/sh: 1: ../tool/asn1ate/asn1ate/asn2quickder.py: not found
Makefile:26: recipe for target 'rfc3280.h' failed
make[1]: *** [rfc3280.h] Error 127
make[1]: Leaving directory '/usr/oms/src/quick-der/rfc'
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 2
is this expected? If so, why?
As introduced in KXOVER. Be sure to test heavily; an ordered sequence of INTEGER
values and testing all combinations (quadratic) to present the right answer should do.
Apparently, der_pack()
is the only function that does not return a success/failure value and sets errno
accordingly. Instead, der_pack()
returns a byte size. Consider better alignment.
The size output could be a size_t *
output parameter. It may be set to 0 to indicate "please figure it out" and may then be validated upon 2nd call. The output may in both cases be an error code. (The error code could be retained with output parameter initialisation to a weird value (like 0 or 1) to indicate keeping errno
across calls. This might simplify error handling — but at the expense of simplicity, so probably not a good idea).
It would seem to be possible to create a #define der_pack()
with a parameter less, and map it to the function. That would effectively allow for backward compatibility and a deprecation path.
After a DER walk, we have a dercursor
that incorporates all the remainder items in a structure. Often, we will only be interested in the first one. Using der_focus()
we could zoom in on that one element. This can currently be established with der_header()
but that is a bit obnoxious.
The present Quick DER ignores constraints, even when they might be meaningful to the parsing process. This leads to the (clear) situation where library users must handle constraint validation themselves.
There are a few ways of improving on this:
[ 62%] Building C object test/CMakeFiles/ldap.test.dir/ldapsearch.c.o
cd /home/gijs/arpa2/quick-der/obj-x86_64-linux-gnu/test && /usr/bin/x86_64-linux-gnu-gcc -I/home/gijs/arpa2/quick-der/include -I/home/gijs/arpa2/quick-der/obj-x86_64-linux-gnu/rfc -march=native -O2 -pipe -Wdate-time -D_FORTIFY_SOURCE=2 -std=gnu99 -o CMakeFiles/ldap.test.dir/ldapsearch.c.o -c /home/gijs/arpa2/quick-der/test/ldapsearch.c
[ 62%] Built target rfc2578_asn1_h
/home/gijs/arpa2/quick-der/test/ldapsearch.c:23:31: fatal error: quick-der/rfc4511.h: No such file or directory
compilation terminated.
The C interface for iteration is mind-boggling. Even when it is sort-of okay to write...
if (der_iter_first (&crs, &iter) do {
...work on iter...
} while (der_iter_next (&iter));
...there still is confusion about whether to der_enter()
and der_focus()
, both inside and prior to the iteration. The cause is basically that the iteration uses .derlen
that span to the end of iteration, which is not normally what we use. Or one might say that the .derptr
has dived in one more level than .derlen
.
A normal API for iteration would use a callback, and that seems quite doable here too; it requires more functions but simplifies the conceptual use of iteration. We might also improve on the current API, of course, for instance by taking the .derlen
from the original cursor and thus require it to be presented in der_iter_next()
too.
How could I forget :)
It may be useful to support a DER stream concept, holding sequences of DER structures containing DER messages. Except that it's BER, LDAP is an example of such a protocol.
A reading routine may start off with an initial read that determines the desired length (from the tag/length) and allow the caller to arrange for that before reading the remaining data. The initial call can be made with a fixed length buffer, or with one that has shown to be large enough in prior calls.
A similar buffer can be filled when packing to write back a response.
The Python tooling built on top of quick-der is not necessary for using the library itself. Since building the scripts and getting the dependencies right for generating the (C and Python) bindings for ASN.1 syntax files is a pain in the butt, perhaps it should be split out of the QD library (repo|package). Generated (C) headers for the RFC headers could be checked in, since they change very little.
As an alternative, the configure script should at least try to find a suitable Python interpreter and check for dependency modules (e.g. six) before the build starts.
asn2quickder uses getopt.getopt(), which is fun for C programmers, but was replaced in Python-style by argparse ages ago. A low-priority job would be to replace the command-line-argument handling by something more modern (handling the test-cases collection is the biggest thing there).
Quick DER chooses to be simplistic in terms of memory management. Layers built around it, such as LillyDAP may well be smarter.
This means that the one thing Quick DER refuses, namely to parse SEQUENCE OF
and SET OF
, can be supported with some help of the outer layer. The issue now is to make Quick DER support such outer layers (and test it with LillyDAP).
We could map to a dernode
union instead of dercursor
for these structures, and initially der_unpack()
into the same form as now namely the dercursor
in the wire
variant of the union; surrounding code in the smarter application can then be mapped to the info
form with an array pointer derray
and array length count dercnt
. The array elements themselves are dernode
values once more. It is the application's knowledge when to use the wire
form and when to use info
. It usually maps one way after der_unpack()
and the other way before der_pack()
.
Finally, the ASN.1 compiler must produce useful definitions that guide the application to the indices and syntaxes of sub-elements -- which must be supportive of a recursive procedure because, for instance, there may be a SEQUENCE OF
inside of a SEQUENCE OF
. Such recursion would of course apply to each individual element, as the structure is repeating.
Once this is done, we shall consider adding generic code to Quick DER, using a callback for memory allocation.
this works:
$ virtualenv -p python3 .venv
Running virtualenv with interpreter /usr/local/bin/python3
[...]
Installing setuptools, pip, wheel...done.
$ .venv/bin/pip install ../quick-der
Processing /Users/gijs/Work/quick-der
[..]
Successfully installed asn1ate-0.6.0 pyparsing-2.2.0 pyquickder-1.2.2 six-1.11.0
$ .venv/bin/python3 -c "import quick_der"
But installing quick_der in developer (-e
) mode doesn't work:
$ .venv/bin/pip install -e ../quick-der
Obtaining file:///Users/gijs/Work/quick-der
[...]
Successfully installed asn1ate-0.6.0 pyparsing-2.2.0 pyquickder six-1.11.0
$ .venv/bin/python3 -c "import quick_der"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'quick_der'
This probably has to do with that the python module is not a direct subfolder of the setup.py
script.
Got most of COMPONENTS OF
working:
DER_PACK_
and DER_PIMP_
is workingDER_OVLY_
does notThe matter is how to get to the structure elements. Already added a default-False
parameter naked
that can be explicitly set to naked=True
to unleash elements in a structure without the opening struct {
and closing }
.
The Python code does clever things with substitute classes and native types. This is wonderful to work with, but we should also make it work when repacking into DER.
The Python code introduces None
for non-existent entries; it even sets all fields to None
when creating an empty instance of an ASN.1 type-named class. When the entries that are None
conform to syntax, these should translate into proper DER-packed output.
Currently, this is done at various locations while I see no reason to not sort them by default after parsing the semantic model.
https://github.com/vanrein/quick-der/search?utf8=%E2%9C%93&q=dependency_sort&type=
the ans1ate module is not installed
python2.7 /usr/local/bin/asn2quickder
Traceback (most recent call last):
File "/usr/local/bin/asn2quickder", line 34, in <module>
from asn1ate import parser
ImportError: No module named asn1ate
at the moment it seems python code is installed in <PREFIX>lib/python2.7/site-packages/
.
This probably works fine if you install in /usr/local
, but fails in most other cases.
Probably better to do is:
site-packages
folder is.This works in most cases, also with virtualenvs. Alternatively you could just split the python code out and safe frustration, don't use cmake for Python and have a pipable python package.
While programming I never considered that elements of SET might occur in any order, and treated it just like SEQUENCE. This is likely to be a bug in the current logic of the parser.
I've modified the python code to directly use asn1ate from github, see the pygijs branch:
https://github.com/vanrein/quick-der/tree/pygijs
When I run asn2quickder
with the latest asn1nate
i run into troubles:
/Users/gijs/Work/quick-der/python/.virtualenv/bin/python /Users/gijs/Work/quick-der/python/scripts/asn2quickder ../../doc-pkcs11/doc/RemotePKCS11.asn1
Traceback (most recent call last):
File "/Users/gijs/Work/quick-der/python/scripts/asn2quickder", line 1580, in <module>
for rm in dm.imports.symbols_imported.keys():
AttributeError: 'NoneType' object has no attribute 'symbols_imported'
Process finished with exit code 1
This contains a complex structure,
SEQUENCE {
... [0] ...,
... [1] ... OPTIONAL,
... [2] ...
}
which translates to parser structure
#define DER_PACK_rfc4120_EncryptedData \
DER_PACK_ENTER | DER_TAG_SEQUENCE, \
DER_PACK_ENTER | DER_TAG_CONTEXT(0), \
DER_PACK_rfc4120_Int32, \
DER_PACK_LEAVE, \
DER_PACK_OPTIONAL, \
DER_PACK_ENTER | DER_TAG_CONTEXT(1), \
DER_PACK_rfc4120_UInt32, \
DER_PACK_LEAVE, \
DER_PACK_ENTER | DER_TAG_CONTEXT(2), \
DER_PACK_STORE | DER_TAG_OCTETSTRING, \
DER_PACK_LEAVE, \
DER_PACK_LEAVE
The OPTIONAL
followed by ENTER
goes wrong when called with a structure with [0]...[2]...
but not the optional [1]...
part. What happens is that der_unpack_rec()
is invoked with the contents of the [2]
after a mismatch of the [1]
syntax part. Bad, and indeed EBADMSG
is reported to the caller! Note that the optional flags are passed properly into der_unpack_rec()
though, so it is surprising in itself that it finds a reason to complain at all!
Bugs therefore are:
der_unpack_rec()
with the embedded type of one that mismatched;EBADMSG
for a mismatch when optional flagging is set.Makes it easier to create a package
Running der_pack()
on the following produces 0 bytes when the two dercursor
structures have their derptr==NULL
-- but it should have packed to an empty sequence,0x30 0x00
.
KIP-SASL-REQ ::= [PRIVATE 3] SEQUENCE {
mech [0] IA5String OPTIONAL,
c2s [1] OCTET STRING OPTIONAL
}
Since 0 is the error output from der_pack()
, this is confusing.
Note the possible relation with #58 on the returned value.
The Python packaging code does a few things funnily [though I'm not laughing about it]:
quick_der
packaging dir, so setting of PYTHON_PATH
does not really worktest/*.py
that make funny assumptions about package names (they strip the quick_der
package name)python/quick_der/*.py
where they ought to be__init__.py
and api.py
are not installed alongside the generated scriptssetup.py
file does not incorporate these files; we only seem to build and install _quickder.so
with it and leave the rest to CMakeIdeally, we should generate "difficult" cases for all primitive types, as well as the possible combinations for each of the constructed types. This should help to automatically construct a set of tests to perform on each builds. Specific problems reported by users of Quick DER could also be manually added to the set of tests. Below is how I described this in Dutch.
We can use der_unpack(const dercursor *crs, ...)
When this is done, the calls from keehive can also be made with static const
structures.
It would be valuable to have functions to read/write elementary types such as INTEGER
, ENUMERATED
and REAL
, the various *STRING
forms and as a special form, perhaps getting and setting individual flags or flag fields in a BIT STRING
.
There does not seem as much need for OBJECT IDENTIFIER
, which can usually be treated in its binary form.
travis-ci is a continues integration service that runs a custom script for every commit or PR. You can for example compile the code and run a test suite. You can even configure it to upload code to for example pypi on a release. It is very useful and free for open source software. It also integrates very nicely with github.
an example:
https://github.com/gijzelaerr/pymonetdb
https://travis-ci.org/gijzelaerr/pymonetdb
lib/der_skipenter.c contains der_enter(), with unreachable code at the end of der_enter(). The if(tag == DER_TAG_BITSTRING) is not reached because of the returns in the if-else above it. The comment for the function explicitly mentions this special behavior, as does the comment in test/certio.c, so it might be important behavior. I can't think of a reason to enter a bit-string.
See tools/test/test01.asn1
vs. tools/test/test01a.asn1
. Running asn2quickder on the first gives a meaningless error message, while the other one compiles successfully.
Either asn2quickder should do something sensible with AUTOMATIC TAGS (e.g. ignore them) or produce a better error message.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.