adammontville / cis-controls-71-measures Goto Github PK

Ensure that each of the organization's key systems are backed up as a complete system, through processes such as imaging, to enable the quick recovery of an entire system.

Measures

M1 = # of backed system / # of total system
M2 = (SUM i to N (backUpSize_i / total_capacity_i)) / N

Metrics/KEI

Efficiency of backup = M1 * M2

1. Backup software is installed
2. Backup software is appropriately configured
3. Backups have run

Deny communication over unauthorized TCP or UDP ports or application traffic to ensure that only authorized protocols are allowed to cross the network boundary in or out of the network at each of the organization's network boundaries.

Measures

# of total unauthorized TCP/UDP ports(using formal analytics) = M1 = 2 * SUM from 𝑖=1 to n(# 𝑜𝑓 𝑢𝑛𝑎𝑢𝑡ℎ𝑜𝑟𝑖𝑧𝑒𝑑 𝑝𝑜𝑟𝑡 𝑖𝑛 𝑏𝑜𝑢𝑛𝑑𝑎𝑟𝑦 𝑑𝑒𝑣𝑖𝑐𝑒 𝑖)
M2= # of sent unique(in terms of port and device) probe
M3 = # of total unauthorized TCP/UDP ports(using formal analytics)
M4 = #of detected traffic to unauthorized TCP/UDP port
# of whitelisted application

Metrics/KEI

Measuring the detectability of connections to unauthorized TCP or UDP ports Coverage = M2/ M3
Quality Measure(Detectability) =M4/M3
Same as TCP/UDP port

This seems like a simple configuration measure.

Ensure that the organization's security awareness program is updated frequently (at least annually) to address new technologies, threats, standards and business requirements.

Measures

w_i = weights of each category to be introduced
I_i = # of items introduced from category i
p_j = # of items collected or proposed of category j
M1 = Current time
M2 = last update time
M3 = Max allowed time without update

Metrics/KEI

AP Quality = ( (sum from i:1 to 4 ((l_i / p_i) * w_i)) ) / (sum from i:1 to 4(w_i))
Freshness of Awareness program = (M1 - M2) / M3

How is this technically measurable at all?

Utilize an active discovery tool to identify devices connected to the organization's network. This tool shall automatically update the organization's hardware asset inventory when devices are discovered.

Measures

M1 = number of assets discovered (SNMP agent)
M2 = total number of assets(given)
M3 = time asset discovered(SNMP agent)
M4 = time asset appeared(given)
M5 = Max time discovery(given)

Metrics

Coverage (Quality Measure) [0-1] = M1 / M2
Freshness (Time to Discover) [1-0] = (M3 - M4) / M5

Publish information for all workforce members, regarding reporting computer anomalies and incidents to the incident handling team. Such information should be included in routine employee awareness activities.

Measures

M_i = # of reported incident by employee i
P_i = # of published incident reported by employee i
n= # of employee

Metrics/KEI

Organization Awareness Score = ( SUM over i:1 to n (M_i / P_i) ) / n

No comment

Use Dynamic Host Configuration Protocol (DHCP) logging on all DHCP servers or IP address management tools to update the organization's hardware asset inventory.

Measures

M1 = # of log enabled DHCP server
M2 = # of total DHCP server

Metrics

DHCP-log ratio[0-1]= M1/M2

Train the workforce on how to identify different forms of social engineering attacks, such as phishing, phone scams and impersonation calls.

Measures

c_i_j = # of properly identified task by employee j in round i
TQ_i_j = # of toal task for employee j in round i
n = # of total employee randomly picked
m = # of round
alpha = damping factor(more recent round will have higer weight) < 1

Metrics/KEI

TP Quality = ( SUM over j: 1 to n ( (SUM over i:0 to m-1 ( (c_i_j / TQ_i_j) * alpha^i ) ) / (SUM over i:0 to m-1 (alpha^i ) ) ) ) / n

Related to each of these "training" pieces. Seems that this should inform v8.0.0 of the controls. What are the characteristics of a security awareness program? How well are employees doing year over year against those tests?

Utilize an up-to-date SCAP compliant vulnerability scanning tool to automatically scan all systems on the network on a weekly or more frequent basis to identify all potential vulnerabilities on the organization's systems.

Measures

None provided.

Metrics

None provided (only the following note)
> Freshness = time between vulnerability scans (How often?)(|1st scan - 2nd scan|/ max scan time)

Maintain documented, standard security configuration standards for all authorized operating systems and software.

Measures

Coverage (Quality Measure)[1-0] = # of software and OS with sc standards / total OS and software

Metrics/KEI
None provided

This doesn't have to be manual. Imagine interrogating CIS-CAT for the set of benchmarks/policies it contains and comparing that against the known list of approved software.

Leverage the Advanced Encryption Standard (AES) to encrypt wireless data in transit.

Measures

M1 = AES Enabled wireless device(model driven)
M2 = Total wireless device

Metrics/KEI

Enforcement Quality = M1 / M2

At this point we care more about the measure than the platform-specific details. For WiFi devices, there will be configuration items we can get to one way or another.

Automatically lock workstation sessions after a standard period of inactivity.

Measures

M1 =# of workstation with locking enabled(Model driven, data driven)
M2 = # of workstation(Model driven, data driven)

Metrics/KEI

Enforcement Quality = M1/ M2
The same KEI can be measured with active testing

No explicit comment, other than that this is covered in most, if not all, benchmarks.

Ensure that there are written incident response plans that defines roles of personnel as well as phases of incident handling/management.

Measures

r_i = # of defined role for incident handling in phase i
R_i = # of total required role for incident handling in phase i  
w_i = criticality of phase i in incident handling
n = # of total phase for incident handling

Metrics/KEI

Incident Plan Quality = ( SUM over i:1 to n ( (r_i / R_i) / w_i) / SUM over i:1 to n ( w_i) )

No comment

Install the latest stable version of any security-related updates on all network devices.

Measures

i = 1 if if security update was applied otherwise 0
j = 1 if the security update for this devices si available
I = [0-1] the important f the machine
Vi = criticality level of the update
M1 = SUM over all network device i to N(i * Vi*Ii)
M2 = SUM over all network device i to N(j* Vi*Ij)

Metrics/KEI

Total criticality of the update = M1 / M2

We seem to be missing some prerequisites. What is the list of stable, security-related updates for network devices? Are we assuming that the latest offering from the vendor is stable?

Ensure that only fully supported web browsers and email clients are allowed to execute in the organization, ideally only using the latest version of the browsers and email clients provided by the vendor.

Measures

# of unblocked unsupported-clients / M

Metrics/KEI

None provided

Require all remote login access to the organization's network to encrypt data in transit and use multi-factor authentication.

Measures

M1p = # of hoste that are reachable remotely (from the main firewall)
M2p = # of device with RDS encryption enabled(formal analytics- configuration of RDS)
M1 = # of hosts that are reachable remotely (from the main firewall)
M2 = # of device with RDS multi-factor authentication enabled(formal analytics-configuration of RDS)

Metrics/KEI

Coverage = M2p / M1p (Policy-based)
Coverage = M2 / M1

This measure should not look only at RDS. The sub-control mentions nothing about RDS specifically.

Monitor attempts to access deactivated accounts through audit logging.

Measures

M1 = # of deactivated account(Model driven)
M3 = # of disabled account selected for active testing
M2 = # of reported access attempt with deactivated account credential

Metrics/KEI

Enforcement Quality = M2/ M3

Credentials used don't matter. Logging these attempts are made in most benchmarks. Enumerate deactivated accounts, enumerate denied access requests, find subset of denials realted to deactivated accounts.

Automatically disable dormant accounts after a set period of inactivity.

Measures

M1 = # of dormant account crossing allowed time of inactivity(Model driven, data driven)
M2 = # of dormant account disabled(Model driven, data driven)

Metrics/KEI

Enforcement Quality = M2/ M1

No explicit comment.

Ensure that backups are properly protected via physical security or encryption when they are stored, as well as when they are moved across the network [traffic]. This includes remote backups and cloud services.

Measures

How important encryption, E = [0, 1]
M1 = (SUM i to N (baked i / capacity i)) / N
M2 = (SUM i to N ((baked i / capacity i) * E i)) / N

Metrics/KEI

Importance of back up =  M2/ M1

We can't do anything about the physical security, but again:

1. Backup software is installed
2. Backup software is appropriately configured

I don't think we want to test whether it's actually encrypted, though. And, the bit about "over the network" should ensure that HTTPS (or other similar protocol) is being used.

Ensure that all system data is automatically backed up on regular basis.

Measures

Ti: backup time at time i , and M is number of servers

Metrics/KEI

Sum_j=1 to M (Sum_i=1 to N (Ti - Ti-1) / N)/M, where Ti is update at time i

1. Backup software is installed
2. Backup software is appropriately configured
3. Backups have run

Assemble and maintain information on third-party contact information to be used to report a security incident, such as Law Enforcement, relevant government departments, vendors, and ISAC partners.

Measures

None provided.

Metrics/KEI

None provided.

No comment

Maintain an up-to-date list of all authorized software that is required in the enterprise for any business purpose on any business system.

Measures:

Number of softwares discovered

Metrics:

Coverage (Quanlity Measure)[1-0] = number of softwares discovered*quality / total   number of softwares in the system*best quality
Freshness (Time to Discover) [1-0] = (time a whitelisted software   discovered - time a software appears in whitelist) / Max time discovery

Maintain an accurate and up-to-date inventory of all technology assets with the potential to store or process information. This inventory shall include all hardware assets, whether connected to the organization's network or not.

Meaures:

M1 =  number of assets discovered (collected)
M2 = total number of assets(given)     
M3 = time asset discovered(collected)    
M4 = time asset appeared(given)     
M5 = Max time discovery(given)

Metrics:

Coverage (Quality Measure) [0-1] = M1 / M2
Freshness (Time to Discover) [1-0] = (M3 - M4) / M5

Deploy automated software update tools in order to ensure that the operating systems are running the most recent security updates provided by the software vendor.

Measures:

Update ratio[0,1] = # of recently update applied to OS / Total update required in OS     
Freshness = how frequently OS are updated?

Metrics:

None given.

Utilize a passive discovery tool to identify devices connected to the organization's network and automatically update the organization's hardware asset inventory.

Measures

M1 = number of assets discovered (SNMP agent)
M2 = total number of assets(given)
M3 = time asset discovered(SNMP agent)
M4 = time asset appeared(given)
M5 = Max time discovery(given)

Metrics

Coverage (Quality Measure) [0-1] = M1 / M2
Freshness (Time to Discover) [1-0] = (M3 - M4) / M5

Train workforce members to be aware of causes for unintentional data exposures, such as losing their mobile devices or emailing the wrong person due to autocomplete in email.

Measures

c_i_j= # of correct answer for employee j in round i
TQ_i_j = # of toal question for employee j in round i
n = # of total employee randomly picked
m = # of round
alpha = damping factor(more recent round will have higer weight) < 1

Metrics/KEI

TP Quality = ( SUM over j: 1 to n ( (SUM over i:0 to m-1 ( (c_i_j / TQ_i_j) * alpha^i ) ) / (SUM over i:0 to m-1 (alpha^i ) ) ) ) / n

See comments for #33 and #34

Train employees to be able to identify the most common indicators of an incident and be able to report such an incident.

Measures

c_i_j= # of properly identified task by employee j in round i
TQ_i_j = # of total task for employee j in round i
n = # of total employee randomly picked
m = # of round
alpha = damping factor(more recent round will have higher weight) < 1

Metrics/KEI

TP Quality = ( SUM over j: 1 to n ( (SUM over i:0 to m-1 ( (c_i_j / TQ_i_j) * alpha^i ) ) / (SUM over i:0 to m-1 (alpha^i ) ) ) ) / n

See comments for #33 and #34

Utilize port level access control, following 802.1x standards, to control which devices can authenticate to the network. The authentication system shall be tied into the hardware asset inventory data to ensure only authorized devices can connect to the network.

Measures

None provided

Metrics

Precision = true positive/ (true positive + false positive)

Before deploying any new asset, change all default passwords to have values consistent with administrative level accounts.

Measures:

Constriant coverage = (# of total account - # of account with default password) / total   account

Metrics:

None given

Maintain an inventory of all sensitive information stored, processed, or transmitted by the organization's technology systems, including those located onsite or at a remote service provider [this is a classificatin not a search problem].

Measures

None provided. (Implied by metrics/KEI?)

Metrics/KEI

Recall = tp / (tp + fn)
Precision  = tp / (tp +fp)

I agree with Aaron's comments on this… Seems like we need 1) an information classification scheme, 2) point to an inventory

Plan and conduct routine incident response exercises and scenarios for the workforce involved in the incident response to maintain awareness and comfort in responding to real world threats. Exercises should test communication channels, decision making, and incident responders technical capabilities using tools and data available to them.

Measures

M1 = # allowed time between two consecutive exercise execution
M2 = # of executed exercise in a time period T

Metrics/KEI

Execution regualrity = M2/ (T/M1)

No comment

Ensure that all backups have at least one backup destination that is not continuously addressable through operating system calls.??? do you mean isolated or unreachable from the rest of the network?

Measures

How important encryption, E = [0, 1]
M1 = (SUM i to N (baked i / capacity i)) / N
M2 = (SUM i to N ((baked i / capacity i) * E i)) / N

Metrics/KEI

Importance of back up =  M2/ M1

Look at backup configuration. If going to an endpoint/destination only periodically available, then test for that availability per schedule.

Ensure that unauthorized assets are either removed from the network, quarantined or the inventory is updated in a timely manner.

Measures

None provided

Metrics

None provided

Remove sensitive data or systems not regularly accessed by the organization from the network. These systems shall only be used as stand alone systems (disconnected from the network) by the business unit needing to occasionally use the system or completely virtualized and powered off until needed.

Measures

V_i = vulnerability score of machine i
A_i = Asset value of machine i
d = damping factor(exploiting nearest machine in the path will have high impact) 0 < d < 1
n = path length
M1 = accessibility time
T = total time in consideration

Metrics/KEI

Impact of machine_i, I(i) = V_i * A_i + SUM over j:1 to n ( (d^j) * I(j) )
Regularity = M1 / T (if less than or equal threshold ok, otherwise bad)

I agree with Aaron's comments on this… Seems like we need 1) an information classification scheme, 2) point to an inventory
This is going to be a difficult one to measure given that it's looking at data OR systems. For systems: 1) is it classified as sensitive, 2) when was the last time it was accessed via network, 3) if beyond a threshold then remove it and make it stand-alone. For data, it's the same thing and we could look at access stamps either on the data directly or inside logs (assuming those are logged correctly).

Perform authenticated vulnerability scanning with agents running locally on each system or with remote scanners that are configured with elevated rights on the system being tested.

Measures

None provided

Metrics

None provided

Ensure that unauthorized software is either removed or the inventory is updated in a timely manner

Measures:

None provided

Metrics:

None provided

Ensure that the hardware asset inventory records the network address, hardware address, machine name, data asset owner, and department for each asset and whether the hardware asset has been approved to connect to the network.

Measures

w_i = weight of machine i
a_i = detailed (1) or not detailed (0) ; 
"detailed" iff all specified information is there.
M1= # of devices and machines in asset inventory
M2 = # of device or machine with connection approval

Metrics

Asset Inventory Quality = (SUM from i to M (w_i * a_i)) / M1
Asset Invetory precision = M2 / M2

Protect all information stored on systems with file system, network share, claims, application, or database specific access control lists. These controls will enforce the principle that only authorized individuals should have access to the information based on their need to access the information as a part of their responsibilities.

Measures

M1 = # of resource with authentication enabled(model driven)
M2 = # of resource

Metrics/KEI

Authentication Score, M3 = M1 / M2
Audit(data driven) score, M4 = find anomalies or inappropriate accces(authorization) from access audit logs

This sub-control doesn't ask for violation detection, it simply wants to know whether all resources have ACLs. Essentially, are the things we care about protected? I'm not at all convinced that there is a practical way to measure this across ALL resources in an enterprise. All OS access, all Web Applicaton access, all files, all databases, and so on. The list is massive.

Use a dedicated account for authenticated vulnerability scans, which should not be used for any other administrative activities and should be tied to specific machines at specific IP addresses.

Measures:

M1 = Total scanner account
M2 = scanner account with unique IP

Metrics:

Coverage(Quality Measure) = (M1 - M2) / M1

Apply host-based firewalls or port filtering tools on end systems, with a default-deny rule that drops all traffic except those services and ports that are explicitly allowed [check comments].

Measures

None provided. (Implied by metrics/KEI?)

Metrics/KEI

Coverage, M1 = # of illegal port / total port (active testing)

1. Firewall or port-filtering software is installed
2. Software is enabled/running
3. Only authorized holes are poked

Deploy automated software update tools in order to ensure that third-party software on all systems is running the most recent security updates provided by the software vendor.

Measures:

Update ratio[0,1] = # of recently updated software / Total # of software
Freshness = how frequently software are updated?

Metrics:

None given.

What this control is asking for:

• Deploy automated software update tools
• Ensure third-party software on all systems is up to date

Train workforce members on the importance of enabling and utilizing secure authentication.

Measures

c_i_j= # of correct answer for employee j in round i
TQ_i_j = # of total question for employee j in round i
n = # of total employee randomly picked
m = # of round
alpha = damping factor(more recent round will have higher weight) < 1

Metrics/KEI

TP Quality = ( SUM over j: 1 to n ( (SUM over i:0 to m-1 ( (c_i_j / TQ_i_j) * alpha^i ) ) / (SUM over i:0 to m-1 (alpha^i ) ) ) ) / n

What Aaron said. Seems not practically measurable at this point. That said, what if we took a measure like this to SANS and asked them to develop an API providing the data for a given organization?

Train workforce on how to identify and properly store, transfer, archive and destroy sensitive information.

Measures

c_i_j= # of properly done task by employee j in round i
TQ_i_j = # of toal task for employee j in round i
n = # of total employee randomly picked
m = # of round
alpha = damping factor(more recent round will have higer weight) < 1

Metrics/KEI

TP Quality = ( SUM over j: 1 to n ( (SUM over i:0 to m-1 ( (c_i_j / TQ_i_j) * alpha^i ) ) / (SUM over i:0 to m-1 (alpha^i ) ) ) ) / n

See comments for #33 and #34

Designate management personnel, as well as backups, who will support the incident handling process by acting in key decision-making roles.

Measures

m_i = # of designated management personnel for key decision-making in phase i
M_i = # of total required management personnel for key decision-making in phase i  
b_i = # of designated management back-up  personnel for key decision-making in phase i
B_i = # of total required management  back-up personnel for key decision-making in phase i  
w_m_i = criticality of designating managemnet personal in phase i
w_b_i = criticality of designating managemnet back-up personal in phase i
n = # of total phase for incident handling

Metrics/KEI

Quality of Incident handling process = ( (SUM over i:1 to n (  ( (m_i / M_i) * w_m_i + (b_i / B_i) * w_b_i) ) /  SUM over i:1 to n (w_m_i + w_b_i) )  / n

No comment

Ensure that all users with administrative account access use a dedicated or secondary account for elevated activities. This account should only be used for administrative activities and not internet browsing, email, or similar activities.

Measures

Correlation among normal browsing and administrative account task

Metrics/KEI

Boolean value = validate if admin account has no network activities
Boolean value = detect if network traffic initiated while admin account is active

Network activity for administrators is normal. Passive monitoring now requires that we have something always running on the endpoint or on the network. There seems to be two parts to this: 1) administrative users have either two accounts or use an "su" type of command to elevate, 2) they do so only when necessary. The first part is something we can measure given infomration from the enterprise, the second part is more behavioral analysis and is more difficult to measure well.

Utilize approved whole-disk encryption software to encrypt the hard drive of all mobile devices.

Measures

M1 = # of total mobile device
M2 = # of encrypted mobile device

Metrics/KEI

Coverage = M2 / M1

Hard drives to not exist in mobile phones, but they do exist in laptops. Granted the spirit of this is "drives" whether hard drives or SSD arrays. Again, this is simply a configuration check for the mobile device OS in most cases. In some cases, it's a two-step: 1) is software X installed, 2) is it configured as expected?

Establish and follow an automated process for revoking system access by disabling accounts immediately upon termination or change of responsibilities of an employee or contractor . Disabling these accounts, instead of deleting accounts, allows preservation of audit trails.

Measures

M1 = # of employee terminated or changed responsibilities within a time period (data driven)
M2  = # of employee account disabled in this time period(active testing)

Metrics/KEI

Enforcement Quality = M1/ M12

No explicit comment.

Ensure that local logging has been enabled on all systems and networking devices.

Measures

x_i =1 if device i is enabled logging otherwie 0,
w_i = importantance of logging in device i (0-1)
m= # of devices and machines that should enable logging

Metrics/KEI

log_coverage [0-1] = Sum_i=1 to M (w_i*x_i) / M

We do this in OVAL all the time for our benchmarks. We could measure something like, for each asset in scope, ensure logging is enabled per enterprise policy.

Configure devices to not auto-run content from removable media.

Measures

None provided. (Implied by metrics/KEI?)

Metrics/KEI

Use active testing or WMI service to know the configuration

These types of things are available in our benchmarks.

Configure devices so that they automatically conduct an anti-malware scan of removable media when inserted or connected.

Measures

None provided. (Implied by metrics/KEI?)

Metrics/KEI

Use active testing or WMI service to know the configuration

We don't need to know which settings at the CAS level. We need to define measures and then later deal with how for specific software.

Ensure that the organization's anti-malware software updates its scanning engine and signature database on a regular basis.

Measures

None provided. (Implied by metrics/KEI?)
``

**Metrics/KEI**

Sum_i=1 to N (Ti - Ti-1) / N, where Ti is update at time i

This doesn't have to be long-running. Practically speaking, however, we will need to be able to check a variety of different software for two things: 1) it's configured to do these things, and 2) it has actually done these things. We don't need to detect AV traffic - we need to check timestamps