Folks using the documentation to deploy encrypted OSDs might want to convince themselves that the resulting OSDs are, in fact, encrypted. The SES6 documentation does not currently provide any guidance for this case.
After using DeepSea to deploy encrypted OSDs as described elsewhere in the documentation, here is a procedure for easily - and without too much disruption - verifying that the OSDs are encrypted.
Step 1
Check the output of ceph-volume lvm list
(it should be run as root on the node where the OSDs in question are located):
# ceph-volume lvm list
====== osd.3 =======
[block] /dev/ceph-d9f09cf7-a2a4-4ddc-b5ab-b1fa4096f713/osd-data-71f62502-4c85-4944-9860-312241d41bb7
block device /dev/ceph-d9f09cf7-a2a4-4ddc-b5ab-b1fa4096f713/osd-data-71f62502-4c85-4944-9860-312241d41bb7
block uuid m5F10p-tUeo-6ZGP-UjxJ-X3cd-Ec5B-dNGXvG
cephx lockbox secret
cluster fsid 413d9116-e4f6-4211-a53b-89aa219f1cf2
cluster name ceph
crush device class None
encrypted 0
osd fsid f8596bf7-000f-4186-9378-170b782359dc
osd id 3
type block
vdo 0
devices /dev/vdb
====== osd.7 =======
[block] /dev/ceph-38914e8d-f512-44a7-bbee-3c20a684753d/osd-data-0f385f9e-ce5c-45b9-917d-7f8c08537987
block device /dev/ceph-38914e8d-f512-44a7-bbee-3c20a684753d/osd-data-0f385f9e-ce5c-45b9-917d-7f8c08537987
block uuid 1y3qcS-ZG01-Y7Z1-B3Kv-PLr6-jbm6-8B79g6
cephx lockbox secret
cluster fsid 413d9116-e4f6-4211-a53b-89aa219f1cf2
cluster name ceph
crush device class None
encrypted 0
osd fsid 0f9a8002-4c81-4f5f-93a6-255252cac2c4
osd id 7
type block
vdo 0
devices /dev/vdc
Note the line that says encrypted 0
- this means the OSD is not encrypted. Here are the possible values:
encrypted 0 <- not encrypted
encrypted 1 <- encrypted
If you get the following error, it means the node where you are running the command does not have any OSDs on it:
# ceph-volume lvm list
No valid Ceph lvm devices found
OK, enough about ceph-volume lvm list
, already! Let's say I have deployed a cluster with an OSD for which ceph-volume lvm list
shows encrypted 1
, so I know ceph-volume
thinks that OSD is encrypted. But how can I gain more confidence that it's really encypted? For that, proceed to "Step 2".
Step 2
Ceph OSD encryption-at-rest relies on the Linux kernel's dm-crypt
subsystem and the Linux Unified Key Setup ("LUKS"). When creating an encrypted OSD, ceph-volume creates an encrypted logical volume and saves the corresponding dm-crypt secret key in the Ceph Monitor data store. When the OSD is to be started, ceph-volume
ensures the device is mounted, retrieves the dm-crypt secret key from the Ceph Monitors, and decrypts the underlying device. This creates a new device, containing the unencrypted data, and this is the device the Ceph OSD daemon is started on.
Since the OSD itself does not know whether the underlying logical volume is encrypted or not, there is no ceph osd
command that will return this information. But it is possible to query LUKS for it, as follows.
First, get the device of the OSD logical volume you are interested in. This can be obtained from the ceph-volume lvm list
output:
block device /dev/ceph-d9f09cf7-a2a4-4ddc-b5ab-b1fa4096f713/osd-data-71f62502-4c85-4944-9860-312241d41bb7
Then, dump the LUKS header from that device:
# cryptsetup luksDump OSD_BLOCK_DEVICE
Here is what the output looks like when the OSD is NOT encrypted:
Device /dev/ceph-38914e8d-f512-44a7-bbee-3c20a684753d/osd-data-0f385f9e-ce5c-45b9-917d-7f8c08537987 is not a valid LUKS device.
And when it IS encrypted:
master:~ # cryptsetup luksDump /dev/ceph-1ce61157-81be-427d-83ad-7337f05d8514/osd-data-89230c92-3ace-4685-97ff-6fa059cef63a
LUKS header information for /dev/ceph-1ce61157-81be-427d-83ad-7337f05d8514/osd-data-89230c92-3ace-4685-97ff-6fa059cef63a
Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 256
MK digest: e9 41 85 f1 1b a3 54 e2 48 6a dc c2 50 26 a5 3b 79 b0 f2 2e
MK salt: 4c 8c 9d 1f 72 1a 88 6c 06 88 04 72 81 7b e4 bb
b1 70 e1 c2 7c c5 3b 30 6d f7 c8 9c 7c ca 22 7d
MK iterations: 118940
UUID: 7675f03b-58e3-47f2-85fc-3bafcf1e589f
Key Slot 0: ENABLED
Iterations: 1906500
Salt: 8f 1f 7f f4 eb 30 5a 22 a5 b4 14 07 cc da dc 48
b5 e9 87 ef 3b 9b 24 72 59 ea 1a 0a ec 61 e6 42
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED