center-for-threat-informed-defense / tram Goto Github PK

As a user, I want to retrain the TRAM model based on reports users have annotated, so that the ML model's accuracy can be improved.

Acceptance Criteria:

1. manage.py pipeline train or similar command that retrains the model and saves it to disk

As a user, I want TRAM and ATT&CK GUI Editor (perhaps ATT&CK Workbench) integrated, so that I can have single-pane-of-glass for my work experience.

Acceptance Criteria:

• TBD

As a user, I want a report summary, so that I can see consolidated information about a report before it is closed and archived.

Acceptance Criteria:

1. A new summary section (Design TBD) on the Analyze page (Pictured below)

As a User, I want to know where TRAM has and hasn't proposed a technique, so that I don't have to click into every single sentence and get annoyed when there are no techniques.

Acceptance Criteria:

1. Somehow indicate the number of mapped techniques

As a user, I want to download the original report, so that I can see the original report after it is processed.

Acceptance Criteria:

• Original report can be downloaded from dashboard

As a user, I want the upload button to be easy to hit, because I use it all the time.

Acceptance Criteria:

• Upload button is more prominent
• Upload button is better separated from logout button
  

As a User, I want to map techniques to groups and software, so that I can create correct and complete mappings.

Acceptance Criteria:

1. TBD

Note: I spoke with users and I wasn't able to get to a concrete set of requirements. I think this particular concept needs to be iteratively defined with users when it comes up for implementation.

There appears to be duplicate code:

First set:

1. https://github.com/center-for-threat-informed-defense/tram/blob/master/static/js/basic.js#L208
2. https://github.com/center-for-threat-informed-defense/tram/blob/master/static/js/basic.js#L218

Second set:

1. https://github.com/center-for-threat-informed-defense/tram/blob/master/static/js/basic.js#L608
2. https://github.com/center-for-threat-informed-defense/tram/blob/master/static/js/basic.js#L617

These are duplicates and one should be removed. Looks like the code on lines 608/617 should be removed?

As a user, I want TRAM to analyze URls reports, so that I can use NLP on internet resources.

Acceptance Criteria:

1. URLs can be specified and processed
2. Supported media types are:

As a user, I want TRAM to skip mapping certain sections, so that I don't have to deal with the noise of removing mappings from those sections.

Requirements:

1. TBD

User statement:
Most CTI reports have a “Security Solutions”, “Mitigations”, “Recommendations”, or similar section at the end that will trigger TRAM mappings based on language used. Having a word proximity or other rule that would have TRAM skip this section would cut down on review and technique deletions in TRAM.

Implementation Discussion:
This could be rule based, or it could be "built in" behavior. This likely requires iteration with users to get to a concept that works and is implementable.

As a user, I want a nice UI that works, so that I can work more efficiently and have confidence in TRAM.

Acceptance Criteria:

1. Cleaner, simpler UI
2. No major bugs

in the ML Admin module allow users to see a model's training data in an att&ck navigator layer.

As a user, I want TRAM to analyze PDF reports, so that I can use NLP on PDF reports.

Acceptance Criteria:

1. PDF Reports can be uploaded and processed

As a user, I want to add reports to the queue via command line, so that I can process reports without using the web UI.

Acceptance Criteria:

1. A command similar to manage.py pipeline add <filename> that creates a document processing job

As a user, I want TRAM to analyze docx reports, so that I can use NLP on docx reports.

Acceptance Criteria:

1. docx Reports can be uploaded and processed

As a user, I want to search for reports on the reports homepage, so that I can more easily find reports that I am looking for.

Acceptance Criteria:

• A search bar that a user can type in text and the reports filter based on the user input

When ATT&CK issues an update, TRAM needs to incorporate that update while keeping all the NLP extraction and sentence adjudication that has been performed to date. The work done by users may represent thousands of hours, so care must be taken to do this well.

Acceptance Criteria TBD

As a user, I want to add and view a report assignee, so that I specify who is working on a report and see who has reports assigned to them.

Acceptance Criteria:

• Assignee is added as a column on the reports dashboard
• Assignee can be specified on the reports dashboard
• Assignee is visible and editable on the /analyze/id/ page

As a user, I want TRAM packaged as a docker image, so that I can easily deploy, run, and use TRAM.

Acceptance criteria TBD.

As a user, I want TRAM to analyze html reports, so that I can use NLP on html reports.

Acceptance Criteria:

1. html Reports can be uploaded and processed

Steps to reproduce:

1. Upload a report
2. Uploaded byline displays None instead of the username

Requirements:

• Train TRAM model based on training data
• Understand the mapping quality
• Make a go/no-go decision based on mapping quality

Requirements TBD

Update TRAM based on sub technique redesign
Re-evaluate NLP models and add additional logic to look for sub-techniques, but use the parent as a fallback

Requirements TBD

As a user, I want to edit TRAM's tokenization, so that I can correct tokenization mistakes.

Acceptance Criteria:

• Add a UI widget to break up one sentence into two sentences.
• Add a UI widget to combine two sentences into one
• Breaking up a single sentence into 3+ sentences can be achieved by using the "split a sentence" UX widget multiple times

Requirements TBD

As a User, I want TRAM to detect Tools and Software, so that I can perform deeper analysis on threat reports.

Acceptance Criteria:

• Tools and Software have a similar management UI to ATT&CK Techniques
• ML/NLP Model proposes Tool and Software mappings on either a report or sentence level
  • This is an implementation choice, not a user choice

As a user, I want to specify my own regular expressions to pull indicators from reports, so that I can customize IOC extraction.

Requirements TBD

As a user, I want to use TRAM as a library, so I can get the key ML features without running the webapp.

Acceptance Criteria:

1. pip install tram works
2. ML pipeline is available via pip installed library

User interface and experience
Transform the home page to a dashboard that includes more statistics and data analytics (e.g., top 10 techniques seen from reporting, technique frequency over time, etc.)

Requirements TBD

As a user, I want to comment on mappings, so that I can explain my reasoning for posterity.

Acceptance criteria:

• A comment thread is available on each mapping

As a user, I want to archive reports, so that they are not displayed on the main page and do not clutter my viewing experience.

Acceptance Criteria:

• Add an "Archived" state to reports
• Add the ability to Archive and Un-Archive reports
• Add a filtering UX widget (Active | Archived | All), that defaults to active

As a developer, I want to add new report formats in a way that follows an established pattern, so I can easily extend the functionality of TRAM

As a user, I want TRAM to propose sub-techniques, so that I can perform complete mappings.

Acceptance Criteria:

• This line of code is changed to not chop the technique label: https://github.com/center-for-threat-informed-defense/tram/blob/master/src/tram/tram/ml/base.py#L168

As a user I want to be able to extract IOCs from the text. I want to be able to enter my own regex to extract the IOCs and test those IOCs. Finally I want to be able to view the IOCs for a given report on the report detail page.

Acceptance Criteria:

1. Regexes can be added to parse out custom IOCs
2. Extracted IOCs can be viewed on the report detail page
3. IOCs are also exported with reports.

As a user, I want to assign a report to a Threat Actor, so that I can group reports together by Threat Actor.

Acceptance Criteria:

1. Drop down that permits a user to assign a Threat Actor to a report.

As an admin, I want to use TRAM with my own authentication backend, so that I can more easily manage identity in my organization.

Acceptance criteria:

1. Identify one commonly used authentication provider (e.g., LDAP) and implement it as an option for TRAM

Requirements TBD

As a user, I want to more easily add techniques to a sentence, so that I can create higher quality mappings.

Acceptance Criteria:

• In the Add-A-Technique modal (pictured below):
• Use the naming format: "<Tactic>: <Name> (TID)"
• Sort Tactics and Techniques how they are sorted in ATT&CK (Tactic -> Technique -> Subtechnique)
• Add a search widget that supports typing in the name and autocompleting it

As a user, I want TRAM to go to the next sentence when I accept a mapping, so that I need to click less.

Acceptance Criteria:

1. When a user clicks "Accept" the sentence context is advanced to the next sentence

Use training data provided for TRAM 0.5. Requirements TBD.

Acceptance Criteria:

1. M-E Branding
2. M-E Logo

As a user, I want to understand the difference between mappings from TRAM and that an analyst provided, so that I can better understand and improve the mapping process.

Acceptance Criteria:

1. Store mapping source in the database
2. Display mapping source in the UI somehow
3. For now, don't add filter/etc capabilities based on the new data. Those can come later.

test-report.pdf doesn't get cleaned up on test completion. Seems like a TemporarilyNamedFile could be used to help solve this.

In app/services/app_svc.py lines 77/78:

TRAM appears to load data incorrectly. This could be a correction:

-   labels.append(tech_names)
-   sentances.extend(annotations[ann])
+   for sentence in annotations[ann]:
+       labels.append(tech_names)
+       sentances.append(sentence)

Ref: https://github.com/center-for-threat-informed-defense/tram/blob/master/app/services/app_svc.py#L77

Requirements TBD

As a user, I want to map ATT&CK Techniques to any selection of a report (Inline Mapping), so that I can fully map reports.

Right now, the pipeline is architected so that the software tokenizes the sentences, and mappings must be done within that tokenization. User cannot map across multiple sentences, map to an image, etc.

Acceptance Criteria:

• The analyze page displays something as close to the original report as possible. This is in contrast to the current "tokenized sentence" view (See screenshot)
• Users can select/map anywhere in the report, including:
  • Images
  • Tables
  • Multiple sentences
  • Parts of a sentence

Tokenized Sentence View:

As a User, I want to sort the report dashboard by status, so that I can more easily navigate uploaded reports.

Acceptance Criteria:

• Sort reports by status

As a developer, I want to add new output formats while following an established pattern, so I can extend TRAM's functionality.