intelowlproject / intelowl-ng Goto Github PK

To extend on Issue #3 and Issue #5, while requesting a scan currently all analyzers are shown unfiltered. This is error prone since each analyzer is supported for a limited types of observable or file type. Once we have solved #5, we can create a parser function that allows selection from only those analyzers which support the current marked observable type on the form.

I noticed a very strange behavior. Clean state. Login from gui -> logout from gui -> everything ok. Login again -> logout again -> everything ok but I noticed that /logout endpoint was called 2 times at the same moment. Then I do login and logout again and this time /logout endpoint was called 3 times...and going on the times when /logout is called increase every time by 1.

I replicated this behavior so it wasn't a one time case.

There are 2 different views of an analyzer result:

• table one, integrated with the web interface style
• raw one, classic navigable JSON

I noticed that in the case of a very long result for an analyzer (try VT analyzer or OTX), the first view could become completely unreadable. In particular it is bad when there are a lot of dictionaries inside other dictionaries inside other dictionaries and so on. I think that we could try to keep the "raw" one because it always worth, in particular for debugging, while we could try to let the "table" one more user-friendly. Maybe, by default, we can only show the first keys of the dict and then we let the user choose by clicking which one he wants to expand and see more.

Waiting for your feedback and your proposals about this

Currently, to view the list of available analyzers, one needs to refer to the IntelOwl's documentation online. That’s tiresome so here’s a better solution:

https://intelowlclient.firebaseapp.com/pages/analyzers

Along with ^this, there should be another tabular view to display the list of analyzer's available with description and use case (not yet provided). The table data could be filtered based on classifications such as: inbuilt, external_service, free, paid, etc. This will help the user in 3 ways:

• Prevent from requesting analyzers for which API keys are not defined
• Prevent gathering noise by requesting only necessary analyzers
• Speed up execution and retrieval of job results

We should change the loading animation (just a spinner for now) with something unique to the Intel Owl project.

At the moment, we only store the last 10 recent scans requested by the user in indexedDB. We could extend this to store the list of jobs and the analyzer configuration perhaps even more, this has many advantages:

• less load on the backend/Database.
• application can be served offline (in case of poor or no network connection)
• fetching from indexedDB is obviously faster than making a http call to the Django API so application loads faster.

Was using IntelOwl for IP and URL analysis to begin building a profile on the source. I didn't see a way to consolidate the resulting search data and had to view the output for each separately. It would be nice to be able to skip blank results or get the combined output into a single export of results - one view of all the findings.

I noticed the alarm "401 unauthorized" for /api/jobs when clicking on the "Dashboard" button. The point is the jobs are correctly visualized

Also, there are times when I login and this happens every time I click on the "Dashboard". Other times I login and this does not happen at all.

When testing some domain submissions i have notice any domain with a second level domain reports as invalid
Tested with:
.co.uk
.co.nz

{
"is_sample": false,
"observable_classification": "domain",
"observable_name": "google.co.uk",
"analyzers_requested": [],
"force_privacy": false,
"disable_external_analyzers": false,
"running_only": false,
"run_all_available_analyzers": true,
"tags_id": []
}

We should enable and configure LGTM analysis for this project.

When I tried domain like google.com.sa the input form on observable search does accept it.

Is this something intended or bug ?

We could provide an API + a button in the GUI to allow the user to perform this operation directly. This should be added in:

1. The single job view
2. The dashboard

Originally posted by @eshaan7 in intelowlproject/IntelOwl#324 (comment)

Angular 10 was released 2 weeks ago, we should consider migrating the project to the new version.

We can use https://update.angular.io/ for help/reference so there's no breaking changes.

• show count of analyzers like finished/total
• refresh job result in background if status is running

Should create a reusable component for the job actions like delete, kill, download job sample, view raw json (maybe not all) .

Component design: context menu or actions.

The idea is we should be able to use this same component in the dashboard as well as job result page.

When user clicks on "save" in Tag form (popover), whether the update request fails or not, the tag is updated on client for the current tab/session.

verify for all of them thereby marking them as configured or unconfigured and constructing the error_message that will be shown on the frontend as the tooltip.

Originally posted by @eshaan7 in intelowlproject/IntelOwl#306 (comment)

should probably add text-wrap; wrap; css property or bootstrap's text-truncated.

in the "Table tree", external_service and leaks_info boolean could contain a little tooltip to explain what do they mean.
Also, I would show a proper icon when the flag is not set instead of "N/A" because, in this case, it is like a "No". Maybe we can use the same icon used for failed analysis.
Last, the search "Yes/No" does not work properly.

• https://github.com/akveo/ngx-admin/blob/master/package.json
• Nebular changelog: https://github.com/akveo/nebular/blob/master/CHANGELOG.md#700-2021-01-18

When user visits job result page, we should automatically select the first row to be displayed in the code box below.

Reference: https://www.youtube.com/channel/UCudh0pRD6Fniu9X2hasGK8w

intelowlproject/IntelOwl#370 would add an analyzer that sends an image (as a base64 string) in the analyzer report.

Requirement - a component that would use the base64 string and display it as an image.

Is it possible to insert the year dynamically?

The tooltip for this component is wrong

It should say: "analysis won't be repeated if already exists as running or reported without fails"

this is a very little thing. When hovering on this button, it does not show the hand pointer, like for the other buttons

Also, these three buttons are very close to each other in my opinion and it is pretty easy to click a wrong one.

Considering that the "Analysis reports" table (the following one)

is always shorter than the main one on the left, it could make sense to move those buttons above the "analysis report" page in a dedicated section.

Idk if I explained my idea clearly. In any case, let me know your thoughts

We could add a kill button to the job (analysis) result page. This button when invoked should stop all ongoing celery tasks (i.e. all running analyzers) for the particular job. The status of such job should then be marked failed or killed.

Originally posted by @eshaan7 in intelowlproject/IntelOwl#225 (comment)

EDIT: The kill button should be visible (use ng-if) only if job.status != "running" and the tooltip should read kill running analyzers and mark as "killed".

It could be useful to extract a javascript client from the work already done in this project.

As I understand, we want this part to be made into a new separate component and then use it on the Results page and the Dashboard both. Where in the dashboard is it going to be displayed?

Yes, those buttons are to be extracted into their own component. For now, I'd say let's not think about where to place it in the dashboard. The main aim is to break down the large JobResult component into seperate components so it's easier to maintain.

So all you have to do is create that new component and use it inside JobResult. We can add it to the dashboard later.

Originally posted by @eshaan7 in #83 (comment)

Task 5:

• API call to fetch job results from backend
• The user should be able to easily fetch and view the report for any particular job
• Implement a similar tabular view for file analyzers based results
• Write modular and reusable code so one can reuse components between the file and observable result viewers.

Task 2:

• Ability to request observable scan from angular client
• Ability to request file scan from angular client

I tried adding "disable": true and could not get that to remove it from the drop down.

Originally posted by @darrellducote in intelowlproject/IntelOwl#298 (comment)

You make a valid point. For now, I'll just add small hint/text to the interface (just below the file upload area) stating "filename shouldn't exceed 64 characters".

Originally posted by @eshaan7 in intelowlproject/IntelOwl#248 (comment)

Use Prettier.io for formatting code of all files: .ts, .html and .scss.

Task 3 - Dashboard [Web] & related Documentation:

• Implement API endpoints and SQL queries to fetch visualization data.
• DataService in Angular to fetch and save this data
• Implement UI to list all the jobs with filtering, sorting functionality
• Test the performance for large sets of data.
• Documentation of how and what the visualizations can be used for.

Update the firebase demo with new changes

At the moment, if we are logged or not, when we navigate to the root URL we are being redirected to the login interface. In the case we are already logged, we should be redirected to the Dashboard instead

issue on IntelOwl: intelowlproject/IntelOwl#397

Corresponding services/pages code would have to be rewritten.

Add a new section Connectors Management:

Connectors

• Information related to the connectors - name, configurations added, secrets required, and health status.
• health check (#130 )

- Toggle button for switching active status (should be triggered or not)
- Pie chart filters to filter connectors by active status and health status (not needed now)

Connector Reports

• Tab Switch on the Job results page to view both analyzer and connector reports

Utilities

• Kill any ongoing connector call (#123 )
• Rerun any connector if the call failed ( #123 )

Mockups can be found here: IntelOwl Connectors Mockups.pdf

There could be the case when there are some analyzers that do not finish fast.
The result could be that a user is waiting for those analyzers but cannot understand which ones are still running because there is nothing that tells him this in the GUI. As you can see in the example, it just tells you the number of completed / number of total analyzers to be executed.

At the moment, the analysis report page is updated every x seconds with the values of the analyzers that terminated the execution in a failed or successful way.
One idea could be to show every analyzer since the start of the very analysis and, every x seconds, update the status like before but reordering the list by setting:

• successful ones at top
• failed ones after
• still running ones at the end (with a new icon different from the others 2)

Task 1 - Authentication Service [Web]:

• Write secure API endpoints for logging-in and logging-out users.
• Implement Login Form and Logout functions.
• Write the HTTP-interceptor to deal with API tokens

A hint should be shown on the analyzers/connectors table letting the user know they can hover over the status icon to see what needs to be configured.

We should store user's current theme preference in localStorage and load the same as default everytime user visits the website.

Integrate Travis checks for new builds for,

• Testing / E2E,
• Linting

Dockerfile to package the angular application as a production ready build. Along with a docker-compose.yml file.