openconfig / bootz

OpenConfig network device bootstrap APIs and services

Bootz protocol v2 (sZTP++ or gZTP)

Objective

Define a structured data format for exchanging as much data with the network device as possible, to take the device from a factory state to a fully production supportable state.

Additionally, define both a dialout and dialin RPC service which can perform the boot workflow.

The overall design of bootz is meant to allow operators of equipment to provide a data-only bootstrap request and allow vendors the freedom to implement the intent based on their own internal APIs.

Bootstrap/Configuration Management End State

When thinking about what data is needed to transition a "factory-reset" device from that state into a production-managed state, we can divide the configuration into multiple units, which differ in how they are managed and interacted with. This is counter to the current model of vendor network devices which have a monolithic single configuration that stores all parameters[^1].

For the use cases in our or service provider networks, we consider the following units:

• Boot Config - Static “bootstrap” configuration required to get a device into a manageable state within the network. This configuration includes parameters such as the gRPC services that are to run, and the management connectivity configuration.
• Security Config - Policies related to provide device access - such as authentication and authorization configurations. This configuration explicitly controls access to who can write the configuration, read parameters from the device via telemetry, and interact with operational RPCs. This configuration can be considered to be separate from the base device configuration, since it is owned by a separate entity within the service provider management plane. It also should not be something that can be modified by an entity that is given read access to the configuration.
• Dynamic config - configuration of the device which is generated according to the topology, and services that are running on the device.

It is desirable for each of these elements of configuration to each have a single owner, who is able to declaratively replace the running configuration on the device without needing to consider contents of the other elements of the system (i.e., the dynamic configuration manager may be unaware of the bootstrap configuration, or the authentication configuration).

Design Options

To support such a concept, there are multiple potential solutions that we can consider:

Path based authorization for configuration

• This concept still has a single root configuration on the device. Clients “own” part of the namespace of this configuration (e.g., /system/aaa in OpenConfig), controlled by gnsi.Pathz configuration.
• The positive of this approach is that it aligns with existing device operational models, with a single configuration (other than Pathz exists on the device).
• The major downside of this approach is fragility - the “dynamic” configuration described above is the catch-all configuration for the device, and would ideally replace /, however, it can no longer do this, fsince there exists data under the bootstrap configuration (for example, /interfaces entries that store the management interfaces) and the security configuration that must not be replaced. The “dynamic” configuration then must cherry-pick trees that are to be replaced, potentially at relatively deep levels in the hierarchy. Bugs or issues with which trees are replaced can render the device unmanageable, or can reflect a security problem (whereby a client that is not authorized to change security policies is able to do so).
• Equally, authentication policies become more complex in this scenario - we imply the existence of a new namespace (Pathz), and create this new concept of multiple configurations solely to enforce the restrictions on the single configuration that we were trying to maintain.

Path based implementation details

• gnsi.Pathz policy would limit specific OpenConfig paths from general writability
• The response to the gNMI SetRequest that contains the OC configuration change (update, set, delete) is expected to indicate that the request failed if those paths are not writable by the caller
• The full config replace would be responsible for only "replacing" specific paths rather than a root level replace
  • /system/<subpaths> but excluding /system/aaa or /interfaces/interfac [name=*] but exclude /interfaces[name=mgmt0|1], this introduces fragility and complexity in how the /interfaces/interface list, or /system paths are managed since they now have multiple writers.

Multiple namespaces for the configuration

• This is the approach that has been pursued with gNSI - whereby certain subsets of the configuration are moved into a separate namespace (origin in oc naming), which is not manipulated via gNMI’s Set RPC. To support the bootstrap configuration, an additional ‘boot’ configuration namespace would be introduced.
• A disadvantage of this approach is the change that it makes to existing device configurations, introducing the concept of multiple potentially overlapping configurations.
• An advantage of the approach is that there is clean separation according to the operational use case described above - a gNSI namespace exists for policies on the device; the ‘boot’ namespace would exist for configuration that is immutable after device boot; and the existing configuration becomes the dynamic configuration of the device. Each namespace can have its own policies for permissions, and client service. Client services need not know about each other’s configuration since replacing a configuration replaces only the namespace of that client.

Multiple namespace implementation details

• Data supplied by gNSI is considered as a separate namespace/configuration store that is treated independently of configuration interacted with via gNMI. All system AAA paths, certificates, keys, accounts and other gNSI covered data is contained in this namespace.
• If gNSI is enabled on the device all of the content covered via gNSI is interacted with solely within this namespace. To interact with this content, gNSI is used. gNSI SetRequests are ACL'ed via gnsi.Pathz. Additionally, gNSI endpoints would continue to provide set operations as well for those configuration elements.
• OpenConfig and vendor configuration would not be allowed to configure those leaves.

Explicit exclusion of configuration based on a specification in gNMI.Set

• In this approach, rather than having separate namespaces be defined, a client that was performing a gNMI.Set request would include some additional information (likely defined through having a new extension message) that specified a set of paths that explicitly should not be modified through the contents of the SetRequest. Specifically, this would ensure that:
  • Should the payload of the SetRequest specify a path that was in the “paths that are not to be modified list” an error would be returned.
  • Replace operations that give the declarative set of end contents for a particular path would exclude these being replaced (e.g., a replace operation that impacted /system, along with a payload that specified that /system/aaa is not to be modified) would have the semantics of replacing the contents of /system with that specified, yet leaving /system/aaa in place.
• This approach allows for a flexible means by which a scope can be declared by a client that knows that there are specific areas of the configuration that it is not intending to modify. Since this scope would be declared by the client, clearly it is not a security mechanism, and hence gNSI.Pathz is still required to define the authorisation policy as to which client can write to specific parts of the configuration.
• The downside of this approach is the complexity introduced into the implementation of the Set handler – and at the client. A client that is managing the dynamic configuration must have a definition as to what areas of the configuration it does not expect to overwrite, and the target implementation must add additional validation code to ensure that the transaction had not changed these values.

Explicit exclusion implementation details

• A gNMI SetRequest would require the inclusion of a message to indicate that a specific set of "exclude paths" are to be honored.
• These paths would be excluded from the merge / replace operations.
• Currently, for some implementations this behavior can be achieved via a proprietary switch within the vendor-configuration that excludes specific paths from being settable from gNMI.
  • In this implementation any attempts to "set an excluded" path is ignored.
  • This is not advisable as it makes it very opaque during a set as to why a set is having a specific output.

Proposed Solution

We propose to follow the “multiple namespaces for configuration” approach described above, whereby separate configuration stores, per the diagram and process below.

Figure 1: Proposed configuration namespaces and clients.

The proposed operational model for devices is shown in Figure 1 above.

• Security credentials (authorization, accounting, authentication) are considered part of security/auth namespace. gNSI manages this configuration store completely. It is considered a “precedence 0” store, whereby it is the authoritative source of configuration in the case of any overlap with other namespace. Explicitly, this means that gNSI owns this configuration, and calls must be made to gNSI to mutate this store.
• Boot configuration (discussed in more detail in this document) is considered part of a bootstrap namespace. gNOI.Bootz manages this configuration store completely (proposed and described in this document). Boot configuration is considered a “precedence 10” store, with it being the authoritative source of configuration in the case of overlaps.
• Boot configuration is expressly defined to be immutable, and provided only at the time of device boot. If the boot configuration is required to be modified, it would be changed through calling a Set RPC through the Bootz service.
  • When changing this configuration the device must "reapply" configuration just as would be expected through a controller doing a gnmi.Set union-merge.
• Dynamic configuration reflects the current, widely-understood concept of configuration on a network device. It is managed via gNMI. Each origin within gNMI is assigned an explicit precedence. The precedence for origins is defined in the relevant document - but the value suggested is >= 100,i.e., for overlaps with other namespaces it is explicitly ignored.
  • It is expected that this is needed in the case that:
  • /system/aaa is populated on the device running gNSI. In such a case, gNSI (in the auth namespace) is the source of truth.
  • Management interfaces, or running gRPC daemons are specified in the Boot namespace, also in the /system or /interfaces hierarchies within the dynamic configuration.

CLI(via SSH or Console) as an API should have access to all the 3 name spaces above. It provides a low dependency method to recover the device in case of bugs/failures/emergencies.

To support this approach, this document proposes a Bootz service, which is the mechanism by which the “boot” namespace is interacted with. Bootz is proposed to replace the existing zero-touch-provisioning implementations.

Background

Currently, the industry standard[^2] is to use the base sZTP protocol for bootstrapping devices. Current sZTP implementations require vendor specific definitions for providing the bootstrapping of a device, because there is no agreed upon vendor neutral data set of “boot configuration” defined. Bootz provides a specification that enumerates the data elements which can be used in a mainly vendor agnostic way, along with the operations that are required at turn-up time that become part of the “boot” process. Specifically, we are suggesting that TPM-related enrollment and attestation should be required during the boot process to ensure the authenticity and integrity of the device before providing it with potentially sensitive artifacts such as certificates, key and device configuration.

Reference Documentation

• External
  • RFC: RFC8572
  • Attestation: Charra Draft

Bootz Operation

Devices are expected to perform a standard DHCP boot. The DHCP server passes a boot option to the device for an endpoint (URL) from which the boot package can be retrieved. The package returned by the endpoint consists of a binary encoded protocol buffer containing all data for being able to complete the boot process. In this context, “complete the boot process” implies the device reaching a fully manageable state - with the relevant gRPC services running.

Upon receiving the bootz protocol buffer, the device is responsible for unmarshalling the bootz message and distributing to the relevant system components for these artifacts to be acted upon. For example, configuration to a configuration manager component, keys and credentials to the relevant security components, etc.

The bootz payload will be encrypted via the TLS session underlying the gRPC service.

Depends on the security requirement for the deployment environment, after loading all the provided data on first boot the device might still not be in a trusted state, however it should have enough g* services initialized to a state where the device can be interrogated from a trusted system to enroll the TPM and validate specific TPM values to attest the device. Once attested, the systems can install production configuration and certificates into the device.

Detailed Design

Boot Procedure

API flow

1. DHCP Discovery of Bootstrap Server
   1. Device sends DHCP messages, containing the mac-address of the active control card. The DHCP server has been configured with all possible mac-addresses of the device, and responds with the static IP address of the bootstrap server.
   2. DHCP server also assigns an IP address and a gateway to the device.
   3. The DHCP response option code should be OPTION_V4_SZTP_REDIRECT(143) or OPTION_V6_SZTP_REDIRECT(136).
   4. The format of the DHCP message (other than response option code) follows RFC.
      1. The URI will be in the format of bootz://<host or ip>:<port>
2. Bootstrapping Service
   1. Device initiates a gRPC connection to the bootz-server whose address was obtained from the DHCP server.
   2. The TLS connection MUST be secured on the client-side with the IDevId of the active control card.
   3. The responses from the bootz-server are signed by ownership-certificate. The device validates the ownership-voucher, which authenticates the ownership-certificate. The device verifies the signature of the message body before accepting the message.
   4. If the signature could not be verified, the bootstrap process starts from Step 1.

Note: though a device SHOULD validate ownership by default, in some environment (e.g. a lab) we might not want to do so. In this case, the device can be explicitly configured to skip the ownership validation, and the device will then not set the nonce field in the GetBootstrapDataRequest message. The bootstrap server may proceed without signing the response and without providing the ownership voucher and ownership certificate.

1. Ownership Voucher and Ownership Certificate
   1. These artifacts have the same meaning as the original sZTP RFC. This document uses OC and PDC interchangeably for convenience. However, we should keep in mind that OV authenticates a PDC (Pinned Domain Cert), and OC might be a distinct certificate with a chain of trust to the PDC.
   2. The contents of the GetBootstrappingDataResponse has an inner message body. The outer message contains the Ownership Voucher, the Ownership Certificate and a signature over the inner message body. The signature is generated using the OC and the nonce.
2. GetBootstrappingData
   1. The bootz workflow is initiated by the device sending a GetBootstrappingData RPC to the bootz-server.
   2. The device describes itself, by listing out all available control cards, and their states.
   3. For a full device install, the state of all control cards is NOT_INITIALIZED.
   4. For installing hot-swapped modules (RMA), the primary control card sets its state to INITIALIZED, and that of the swapped module to NOT_INITIALIZED.
3. GetBootstrappingDataResponse
   1. The server responds with the intent for the device's baseline state. This includes the OS version and boot password hash, and an initial device configuration. This would include initial gNSI artifacts such as: certz,pathz,authz,credentialz artifacts to allow the device to progress to enrollment and attestation or even bring the device fully into a production state.
   2. The proto allows us to return multiple sets of configurations, one per control card. However, in practice, we will apply the same configuration to both cards.
   3. For RMA, the server will return only one state - for the RMA'd module.
   4. The device/modules should download the OS image, verify its hash and install the OS.
      1. A reboot may be performed, if required.
   5. The device will then apply the configuration
      1. A reboot may be performed, if required.
   6. The GetIntendedState RPC can be performed as many times as required (i.e. it is idempotent).
4. ReportProgress
   1. This RPC method is used by the device to inform the bootz server that the bootstrapping process is complete. Success or failure is indicated using an enum.
   2. In case of failure, the device should retry this process from Step 1.
   3. When making this RPC, the device should verify the identity of the server using the "server_trust_cert" certificate obtained from GetBootstrappingDataResponse.
   4. "server_trust_cert" plays the same role as "trust-anchor" in the sZTP RFC (i.e. allows the device to verify the identity of the server).
5. At this point, the device has a minimal set of configuration, enabling it to receive grpc calls. Depending on the operator's security policies, an attestation-verification or an enrollment step may be performed.
   1. At Google, we plan to always require enrollment and attestation-verification.
6. TpmEnrollment
   1. The enrollment API doc is the authoritative reference for this API. The API is reproduced here for convenience.
   2. This step allows the owner organization to install an owner IAK certificate on the device(s). This may be desirable if the owner requires periodic rotation of certificates and a revocation policy that is independent of the device vendor.
   3. The TpmEnrollment API is secured with the IDevID certificate.
   4. During enrollment, the server MUST verify that the serial number on the Attestation Key certificate matches the serial number on the DevID certificate, and validate both certificates with the appropriate trust bundle.
7. Attestation
   1. The attestation API doc is the authoritative reference for this API. The API is reproduced here for convenience.
   2. This step allows the owner to obtain cryptographic evidence for the integrity of the software components on the control-cards.
   3. The attestation API should be secured with DevID for the first attestation, and mtls for subsequent attestations.
      1. On first attestation, the server MUST verify that the serial number on the Attestation Key certificate matches the serial number on the DevID certificate, and validate both certificates with the appropriate trust bundle.
   4. On modular devices, the active control card obtains attestation evidence from the standby, and relays it to the bootz server. The active card must verify the identity of the standby during this process. The standby card's ability to communicate with the active card is not sufficient evidence of identity.
      1. This specification does not prescribe how to verify the identity of the standby, as the communication between the control cards takes place over vendor-proprietary protocols. However, at a minimum, the active supervisor must check that the standby's TPM-backed IDevID. For example, it may request the IDevID cert, then issue a decrypt challenge to the standby card.
8. Final state:
   1. At this point, the device an initial configuration, user accounts, and we have validated the identity and integrity of the device and its software components. It is ready to serve traffic.

A Note on Modular Devices

Many commercial modular form-factor devices start copying data from the active control card to a hot-swapped standby immediately on plug-in. This includes sensitive data such as credentials and certificates. From a security standpoint, this is undesirable.

We recommend that the active card copy system software only (includes firmware, bootloader, boot password and OS image), then restart the hot-swapped card. It should verify the standby card's identity (using DevID verified against the vendor root of trust) and attestation evidence (which should match the active card's PCR values) before copying the operator-provided operational configuration.

Alternatively, the active card can perform the bootstrapping, enrollment and attestation workflow on behalf of the standby before copying operational configuration.

Workflows for RMA replacement

1. Card failure detected
2. Card RMA issued
3. Card replaced
4. Active control card detects new linecard
5. Active control card tries to verify card via IDevID
6. Active control card verifies card software and firmware is at least at it's version a. if not will send over the firmware and software and perform an upgrade.
7. The chassis will update gNMI with component in inserted but not enrolled state.
8. Kick of workflow for initialiation of new control card a. Inventory system detects unenrolled control card kicks off workflow to the enrollment/attestion system. b. Active control card send bootz request to bootserver containing serial number of new card and will get back bootz artifacts.
9. Planet service makes call to enrollement API and provides proper OV for the control card.
10. Planet service makes call to attestation API to verify attestation values.
11. Active control card now verifies the card and brings the card into production state and sync's configuration and other files.

Certificate deployment options

Bootz only

In the Bootz only workflow certificates are delivered by the bootz server in the initial bootz data payload. After reboot the device will use the provided security profile and certificates provided in the bootz message.

Bootz with Certz rotation

In this workflow the device will boot from bootz but will use it's iDevID cert for initial services. Once rebooted, the device can be reached via Certz using the iDevID cert.

Bootz with Enrollz and Attestz

This is the preferred workflow for security considerations. This workflow utilizes Enrollz and Attestz to provide enrollment then measured boot to validate the state of device before providing any "production" certificates.

Protobuf Payload for Bootstrap

The following protocol buffer is provided from the bootserver to the device to provide the bundle of artifacts needed for the device to reach a manageable state.

See PR

bootz's Issues

datetime format in the ownership voucher

Hi

The 'created-on' and 'expires-on' dates in the ownership voucher use the time.String() api. The resulting time string has a monotonic clock reading.

This format of date-time is not supported by all programming languages. would it be possible to use a standard formatting? say RFC3339 for example

Make Bootz server buildable with Bazel

We're currently using go build to build and run the server. We should instead be using Bazel to build and run but the build targets are not configured correctly. This also involves getting Gazelle to run properly.

owner certificate should be CMS struct

as per the sztpd spec the owner certificate is a CMS structure specified by RFC5652. the owner certificate generated using the generate tool does not have follow the RFC. it does not have all the elements of the CMS struct

Unable to build bootz server/emulator using bazel.

Hello,

We've been trying to use bazel to build the files in the server directory, but we are running into build issues.

If we use:
bazel build //server/emulator --verbose_failures

we get:

ERROR: /home/vagrant/bootz_stuff/bootz/server/emulator/BUILD.bazel:17:11: in go_library rule //server/emulator:emulator_lib:
Traceback (most recent call last):
        File "/home/vagrant/.cache/bazel/_bazel_vagrant/a08af4435e3f0281d43adf28c3861bae/external/io_bazel_rules_go/go/private/rules/library.bzl", line 45, column 34, in _go_library_impl
                source = go.library_to_source(go, ctx.attr, library, ctx.coverage_instrumented())
        File "/home/vagrant/.cache/bazel/_bazel_vagrant/a08af4435e3f0281d43adf28c3861bae/external/io_bazel_rules_go/go/private/context.bzl", line 264, column 26, in _library_to_source
                _check_binary_dep(go, dep, "deps")
        File "/home/vagrant/.cache/bazel/_bazel_vagrant/a08af4435e3f0281d43adf28c3861bae/external/io_bazel_rules_go/go/private/context.bzl", line 326, column 13, in _check_binary_dep
                fail("rule {rule} depends on executable {dep} via {edge}. This is not safe for cross-compilation. Depend on go_library instead.".format(
Error in fail: rule @//server/emulator:emulator_lib depends on executable @//testdata:testdata via deps. This is not safe for cross-compilation. Depend on go_library instead.
ERROR: /home/vagrant/bootz_stuff/bootz/server/emulator/BUILD.bazel:17:11: Analysis of target '//server/emulator:emulator_lib' failed

If we try to build testdata seperately, (or include the testdata_lib instead)

bazel-6.4.0 build //testdata/...

ERROR: /home/vagrant/bootz_stuff/bootz/testdata/BUILD.bazel:17:11: GoCompilePkg testdata/testdata_lib.a failed: missing input file '//testdata:generate.go'
ERROR: /home/vagrant/bootz_stuff/bootz/testdata/BUILD.bazel:28:10: GoCompilePkg testdata/testdata.a failed: missing input file '//testdata:generate.go'
ERROR: /home/vagrant/bootz_stuff/bootz/testdata/BUILD.bazel:28:10: GoCompilePkg testdata/testdata.a failed: 1 input file(s) do not exist
ERROR: /home/vagrant/bootz_stuff/bootz/testdata/BUILD.bazel:17:11: GoCompilePkg testdata/testdata_lib.a failed: 1 input file(s) do not exist
ERROR: /home/vagrant/bootz_stuff/bootz/testdata/BUILD.bazel:28:10 GoCompilePkg testdata/testdata.a failed: 1 input file(s) do not exist

Apparently, PR #92 has deleted generate.go, but Bazel files are not updated? There are similar bazel dependency issues with other files too, so if we manually fix one, another one comes up.

Any documentaiton on how to properly build/deploy and any help on if we are doing anything wrong would be much appreciated. Thanks.

Client does not verifyTLSCert by default

better to set it to verifyTlsCert by default. Also document the flag in the README.

Move all encyption code to a separate package

Currently the encryption related code is scattered, we need to move all of the encryption related work to a package and use it everywhere else.

Chassis message should not live in Bootz proto

The Chassis message is used internally to hold fields that are returned by the organization's inventory. Since these fields do not exist in the public API, it doesn't really make sense for them to live in the public proto message where it may cause confusion.

Need clarification on Control Board numbering

From Bootz perspective, is slot "0" considered as invalid control board slot number?

When I dump JSON content of Bootz request in python script as shown below, it prints slot-1 number as "1" as expected.
But it doesnt print slot-0 number. So, suspecting that "0" is considered as invalid index.

json_obj = MessageToJson(request)
print(json_obj)

Test APIs to control bootz server, dhcp server, and get device boot/dhcp status

We need to create APIs that allows starting/stopping DHCP/Bootz Server from tests. Also, we should add APIs to get device boot and dhcp ip lease status from tests.

testdata/generate.go consider using os.WriteFile instead

[bootz/testdata/generate.go]
[104] func writeFile(contents []byte, filename string) error {

Is there a reason for not using os.WriteFile()?

https://pkg.go.dev/os#WriteFile

De-duplicate protos

We currently have bootz.proto and entity.proto. The former is used for the gRPC API while the latter is mainly used for testing.

entity.proto is mostly a copy of bootz.proto and there's a lot of unfortunate duplication. We should deduplicate this and have bootz.proto as the main one.

Fix misc nits

Comments copied from internal code:

third_party/openconfig/bootz/server/entitymanager/entitymanager.go
[349]
encrypt the response

Nit: This, and the following log entry, are incorrect. The message is being signed, not encrypted.

[third_party/openconfig/bootz/server/service/service.go]
[36]
Key

Nit: It's a good practice to explicitly label this as a private key (i.e. PrivateKey). This helps prevents accidentally mixing up what you're dealing with.

See, for example, google3/third_party/go/gc/src/crypto/tls/common.go;l=1408;ws=codereview%2F678366780;rcl=34

Field should be byte instead of string

third_party/openconfig/bootz/common/ownership_voucher/ownership_voucher.go
[46] string

According to https://www.rfc-editor.org/rfc/rfc8366.html#section-5.1, this field is intended to be binary data.

Unless there's a reason I'm missing not to do so, this field should be []byte.

This would have the following benefits:

No need for RemovePemHeaders(). Instead of performing string manipulations of the encoded data, you would then use the encoding/pem package to more safely/reliably decode PEM and extract the byte sequence encoded within it.

No need to worry about manually encoding when serializing to JSON. When encoding/json marshals a []byte field, it automatically base64 encodes it. See https://pkg.go.dev/encoding/json#Marshal.

Create image server

Create an image server (https) to load the images on devices.

DHCP Option encoding

Hi,
I have concern with the way Bootz is using DHCP options 143 for v4 and 136 for v6.
Since this is IETF standards based option, anyone who wants to use this option should
be in conformance with RFC.

This option is standardized by RFC 8572, section 8. In future, when DHCP servers starts
adding support for this option, they will follow IETF since it is standardized way.
So, bootz should also align with this uniform encoding so that both Secure-ZTP and
Bootz can work with same DHCP option type.

how it should be:
(full) option length should follow the type-143. Size 1-bytes.
(Section-8.1 of RFC 8572)

         0                             1
         0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
        |   option-code (143)   |     option-length     |
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
        .                                               .
        .    bootstrap-server-list (variable length)    .
        .                                               .
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

        * option-code: OPTION_V4_SZTP_REDIRECT (143)
        * option-length: The option length in octets.
        * bootstrap-server-list: A list of servers for the
           client to attempt contacting, in order to obtain
           further bootstrapping data, in the format shown
           in [Section 8.3](https://datatracker.ietf.org/doc/html/rfc8572#section-8.3).

Then server-list is encoded like this (Section 8.3 of RFC 8572):

Both of the DHCPv4 and DHCPv6 options defined in this section encode
a list of bootstrap server URIs. The "URI" structure is a DHCP
option that can contain multiple URIs (see [RFC7227], Section 5.7).
Each URI entry in the bootstrap-server-list is structured as follows:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-...-+-+-+-+-+-+-+
|       uri-length              |          URI                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-...-+-+-+-+-+-+-+

* uri-length: 2 octets long; specifies the length of the URI data.
* URI: URI of the SZTP bootstrap server.

Since length is 2 bytes, it is will uint-short. This should be encoded in htons() format.
Also, this enables sending multiple servers as list for load balancing and redundancy.

Yes, it looks a bit silly since full option length is 8 bits thus max 255 bytes while each
URL length field is 16-bit which is sub-part of 255 bytes. But "this is the way" :-)

Not sure if following code from plugins/bootz/bootz.go is doing same as above. But wanted to
make sure we are doing right thing.

   ztpV4Opt = &dhcpv4.Option{
            Code:  dhcpv4.GenericOptionCode(OPTION_V4_SZTP_REDIRECT),
            Value: dhcpv4.String(url.String()),
    }

Thanks,
Katta

Signature.go should match Verify implementation and use crypto.PrivateKey

Signature.go Line 30:
To match the Verify() implementation, this should be a crypto.PrivateKey and then a type switch should be used to provide algorithm specific code paths.

Otherwise, if the current function signature gets traction, introducing an algorithm agnostic counterpart will end up being a bit cumbersome.

gnsi test data should be directly included in prototext, not as a separate file

Calculate and verify hashes in BootstrapDataResponse

This proto message contains various fields in which a hash is calculated. These are not currently populated and are not verified by the client.

Have client reconnect to Bootz server using provided server_trust_cert

After the initial GetBootstrapData call, client should reconnect to the server in fully trusted TLS mode using the server trust cert.

proto.Marshal() is not a stable form of serialization

Proto serialization is not stable in client.go. See [the note in the Marshal() documentation] (http://google3/third_party/golang/protobuf/v2/proto/encode.go;l=74;rcl=563382740).

There's no guarantee that the Marshal() result here will match byte-for-byte what was used to produce the signature.
The recommended approach is described at go/protobuf-crypto#authenticating (i.e. store the signed serialized data and verify the signature before deserializing).

Re-create TLS session when reporting progress

After the initial bootstrap gRPC, a server_trust_cert is returned in the response. Client should re-establish a TLS connection with the server using this trust cert instead of the initial insecure mode.

How does Switch Owner creates oIAK and oIDevID, based on the public keys?

change the server implemenation to handle fixed chassis

The current implementation does not handle fixed chassis. Also, the bootz proto assumes the serial for modular chassis is empty while the implementation assumes the serial for modular chassis should be set. To address this, we change the server to make the serial for modular chassis to be optional.

ControlCard slot id

bootz.proto defines control card slot id as a numeric (int32) value

bootz/proto/bootz.proto

Lines 75 to 80 in b8d7fb3

	// Details of the control card, including serial-number and the location
	// of the card on the chassis.
	message ControlCard {
	string part_number = 1;
	string serial_number = 2;
	int32 slot = 3;

, however OpenConfig uses type string for component ID and location leafs:
https://github.com/openconfig/public/blob/cd1cfcd263d033ee5f9b10cddde6f823f6cb30e2/release/models/platform/openconfig-platform.yang#L391, and not all vendors use numeric values for control cards.

Can this be changed to string?

cc @xw-g

Create an integration test

Log client and server activity

Having verbose logs that can be optionally printed to stdout helps with debugging and understanding the API flow

Implement a DHCP Server plugin

entitymanager_test should use signature.go functions instead of calling crypto/rsa directly

ov encoding

The current implementation apply base64 encoding for ov. However, based on the bootz proto the ov is byte and therefore base64 encoding of ov is not compliant with the proto.

Unknown

The IAK and IDevID are created by the Switch Vendor, but the Switch Owner has to verify them. EnrollZ service must have access to the up-to-date switch vendor trust bundle/anchor needed to verify the signature over the IAK and IDevID certificates. How is the trust bundle obtained? How long does it persist?

Fix relative filepath issues

Artifacts in bootz server are loaded from disk via a relative path. This causes issues as it depends on the current directory of the running binary. When a test and server.go try to use the same relative path, one will fail as the binaries are running in different directories.

One potential fix for this is to build the server with Bazel and include these artifacts in the build rule which can then be accessed via absolute paths.

Response signature should be CMS signed data

Raised by Katta from Juniper:

From bootz.proto

===============

// This is a signature of the serialized_bootstrap_data field,

// using the ownership_certificate.

// This should not be set if the device does not check for ownership

// voucher, which is indicated by the device not setting the nonce field

// in the GetBootstrapDataRequest message.

string response_signature = 103;

Specification and code assume that signature will be done with RSA key and with

PKCS#1 signature. Also, I see key should be in PKCS#1 PEM format. PKCS#8 format is

not accepted.

A standard should be accommodative to use any type of keys and signing schemes.

One could use ECDSA for signing. With RSA, one could use PKCS#1.5 or PSS signing.

Also the hash used for signing is assumed to be sha-256. Specification should allow any

hash like sha-512.

Then signature field should carry all of it (hash algo used, signature algo used, signer info etc.).

For this, I suggest to encode signature as CMS signed data (RFC 5652) and sending entire CMS

message in “response_signature” as bytes data. It adds about 700 bytes extra data which

is not significant for non-telemetry data.

Load emulator data from a config file

We currently hardcode the the details of the client device (e.g. manufacturer, serial number etc) and also the inventory of the server.

We should think of a way for this emulated data to be set via a config file and shared with both sides.

Make gRPC transport configurable

Client and server should have options to set up the gRPC connection with regular TCP, insecure TLS or fully trusted TLS.

Populate boot, pathz and authz data in GetBootstrapData

This is currently hardcoded. We should implement a better way to generate this data.

Support other encryption standards

We currently only support RSA keys for signing and verifying of the response.