mandiant / vm-packages Goto Github PK
View Code? Open in Web Editor NEWChocolatey packages supporting the analysis environment projects FLARE-VM & Commando VM.
License: Apache License 2.0
Chocolatey packages supporting the analysis environment projects FLARE-VM & Commando VM.
License: Apache License 2.0
Document different between VM-Install-From-Zip and VM-Install-Raw-GitHub-Repo (also in the ISSUE template, or link to the added documentation).
It seems that installing notepadplusplus.vm
fails sometimes with access is denied
: https://github.com/mandiant/VM-Packages/runs/6088818786?check_suite_focus=true It successes in some daily runs and fails in others.
I forgot to add the upload of logs to the daily in #22, I'll send a PR in a moment that may help us figuring out what the problem is.
Update documentation at: https://github.com/mandiant/VM-Packages/tree/main/packages/common.vm
Add a GH action that automatically updates packages with are downloaded from GH URL (such as GitHub releases) and sends a PRs. If the test suite passes, we can just merge it. A relatively easy way to get the last version of many of our tool.
Chocolatey installed 1/1 packages.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Chocolatey v1.2.0
Installing the following packages:
python2.vm
By installing, you accept licenses for the packages.
python2.vm v2.7.18 (forced)
python2.vm package files install completed. Performing other installation steps.
Downloading python2.vm
from 'https://www.python.org/ftp/python/2.7.18/python-2.7.18.msi'
Download of python-2.7.18.msi (18.72 MB) completed.
Hashes match.
Installing python2.vm...
python2.vm has been installed.
2022/11/14 04:14:17 [python2.vm] vm.common.psm1 [+] ERROR : Invalid path: C:\Python27
2022/11/14 04:14:17 [python2.vm] vm.common.psm1 [+] ERROR : [ERR] Invalid path: C:\Python27
At C:\ProgramData\_VM\vm.common\vm.common.psm1:271 char:5
+ throw $err_msg
+ ~~~~~~~~~~~~~~
ERROR: Invalid path: C:\Python27
The install of python2.vm was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\python2.vm\tools\chocolateyinstall.ps1'.
See log for details.
Chocolatey installed 0/1 packages. 1 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Failures
- python2.vm (exited -1) - Error while running 'C:\ProgramData\chocolatey\lib\python2.vm\tools\chocolateyinstall.ps1'.
See log for details.
See e.g. https://github.com/mandiant/VM-Packages/actions/runs/3458340832/jobs/5777311898
CI
Yes
Consider using a commit message structure like: https://www.conventionalcommits.org/en/v1.0.0/
Update FLARE-VM installer at https://github.com/mandiant/flare-vm to use packages built/pushed from this repository
Send out survey for getting more insight about the packages our users want.
Chocolatey v1.2.0
Installing the following packages:
7zip.vm
By installing, you accept licenses for the packages.
7zip.vm v15.05 (forced)
7zip.vm package files install completed. Performing other installation steps.
Downloading 7zip.vm 64 bit
from 'https://sourceforge.net/projects/sevenzip/files/7-Zip/15.05/7z1505-x64.exe/download'
2022/11/13 03:43:49 [7zip.vm] vm.common.psm1 [+] ERROR : [ERR] The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://sourceforge.net/projects/sevenzip/files/7-Zip/15.05/7z1505-x64.exe/download'. Exception calling "GetResponse" with "0" argument(s): "The request was aborted: Could not create SSL/TLS secure channel."
At C:\ProgramData\chocolatey\helpers\functions\Get-WebFile.ps1:330 char:8
+ throw "The remote file either doesn't exist, is unauthorized, ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://sourceforge.net/projects/sevenzip/files/7-Zip/15.05/7z1505-x64.exe/download'. Exception calling "GetResponse" with "0" argument(s): "The request was aborted: Could not create SSL/TLS secure channel."
The install of 7zip.vm was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\7zip.vm\tools\chocolateyinstall.ps1'.
See log for details.
Chocolatey installed 0/1 packages. 1 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Failures
- 7zip.vm (exited 404) - Error while running 'C:\ProgramData\chocolatey\lib\7zip.vm\tools\chocolateyinstall.ps1'.
See log for details.
Is the download potentially blocked via Sourceforge/Cloudflare?
See e.g. https://github.com/mandiant/VM-Packages/actions/runs/3453675077/jobs/5764461768
CI
Yes
It needs everything else in the milestone v1.
bytecodeviewer
Bytecode Viewer
2.11.2
Java
Konloch
A lightweight user-friendly Java/Android Bytecode Viewer, Decompiler and more.
ZIP_EXE
https://github.com/Konloch/bytecode-viewer/releases/download/v2.11.2/Bytecode-Viewer-2.11.2.jar
536ad387424106083f76cd0cb7c051a22aff21f08663ba2539c11f1ddef9147f
To analyze Java/Android files.
Add issue templates to help tracking new packages proposals.
Issue introduced in #103. $built_pkgs
is now a list of True and False and not a list of packages names.
PS VM-Packages > $built_pkgs = Get-ChildItem $built_pkgs_dir | Foreach-Object { ([regex]::match($_.BaseName, '(.*?[.](?:vm)).*').Groups[1].Value) -and $_ -notin $exclude_tests}
FLARE 11/21/2022 1:57:05 PM
PS VM-Packages > $built_pkgs
True
True
True
See https://github.com/mandiant/VM-Packages/actions/runs/3511009362/jobs/5881380101 or run scripts/test/test_install.ps1
locally.
As there was no package update in #103, the issue was not detected by the CI. But now it fails for every package update. See #106.
Create a Get-VM-Info
command that users can run and copy-paste in the issue when reporting a bug. @mr-tz's idea, see #40 (comment)
Update bug issue template with this command.
We are currently opening console applications using cmd.
$executableCmd = Join-Path ${Env:WinDir} "system32\cmd.exe"
I think it would be nice to use a environment variable to allow to change the default console.
Add "basic URL file" to create_package_template.py and to the issue template and GH action that create PRs from issues.
I think the syntax of "$variable.exe" is more readable. I checked and we use ($variable + ".exe") some throughout the code base. Let's make this more uniform
./packages/processdump.vm/tools/chocolateyinstall.ps1:14: $executablePath = Join-Path $toolDir ($toolName + "32.exe")
./packages/processdump.vm/tools/chocolateyinstall.ps1:29: $shortcut = Join-Path $shortcutDir ($toolName + "32.lnk")
./packages/processdump.vm/tools/chocolateyinstall.ps1:32: Install-BinFile -Name ($toolName + "32") -Path $executablePath
./packages/processdump.vm/tools/chocolateyinstall.ps1:38: $executablePath = Join-Path $toolDir ($toolName + "64.exe")
./packages/processdump.vm/tools/chocolateyinstall.ps1:51: $shortcut = Join-Path $shortcutDir ($toolName + "64.lnk")
./packages/processdump.vm/tools/chocolateyinstall.ps1:54: Install-BinFile -Name ($toolName + "64") -Path $executablePath
./packages/processdump.vm/tools/chocolateyuninstall.ps1:7:VM-Remove-Tool-Shortcut ($toolName + "32") $category
./packages/processdump.vm/tools/chocolateyuninstall.ps1:8:VM-Remove-Tool-Shortcut ($toolName + "64") $category
./packages/processdump.vm/tools/chocolateyuninstall.ps1:10:Uninstall-BinFile -Name ($toolName + "32")
./packages/processdump.vm/tools/chocolateyuninstall.ps1:11:Uninstall-BinFile -Name ($toolName + "64")
The Python script should check the following:
Previously documented in #6 (splitting that issue up now)
See https://github.com/mandiant/VM-Packages/runs/8303111845?check_suite_focus=true#step:5:20
May be an issue with the line breaks?
add send PR
label to METAPACKAGE issue
CI
No
I think having two badges that show the number of "passing" and "failing" packages would be more informative versus a daily badge that simply says "success/fail".
We should have a GitHub Action that compares packages in the GitHub repository to those hosted on MyGet and let us know if they differ. For example, we if remove a local package or rename one, this action should point out a conflict so we can take the appropriate actions to remedy it.
Some dependencies are not fixed and they should to avoid an update breaks the package. We have CI now in place to update them automatically if the package still works.
We are very close to start using these packages in FLARE-VM. We have mentioned a few times that we want to give a thought to the current categories, trying to create more concrete categories that make it easier to find tools.
I think these are the current categories:
This is also needed for #25
@mandiant/flare-vm ideas?
Ideas for linting:
In the past we had bugs related to usernames with spaces. If possible use a username with spaces in GH actions.
Attempting to get AV information similar to https://www.powershellgallery.com/packages/CimSweep/0.6.0.0/Content/Auditing%5CAntiVirusInfo.ps1 sometimes works. Other times it doesn't.
Windows 10 x64 1809
I don't know
Per #50:
The code in create_package_template_from_json.py could be integrated in create_package_template.py
The local tests and the daily.yml and ci.yml GH actions actions are quite similar, so that we often need to modify the same thing in all of them. Avoid code duplication by exporting the actions to a script to use it in the GH actions and for local testing.
This should also make easier for new contributors to test their changes locally.
PRs remain open after automatic generated package has been merged. See for example: #52
When users update the default ${Env:RAW_TOOLS_DIR}
location it also modifies the ${Env:ChocolateyToolsLocation}
location and this is the location where Chocolatey defaults to install tools. For tools / dependencies that install to the default location, we need to use ${Env:ChocolateyToolsLocation}
.
Use the information provided in an "New package proposal" issue to create a PR. We would need some improvements in the package creator script and a GH action that takes care of the automation.
Needs #25
Add the following logs to the artifacts: %PROGRAMDATA%\chocolatey\lib-bad\*\install_log.txt
Packages may create a per-install log to use for whatever they wish, see:
The packages libraries.python*.vm
use this to redirect output from pip when installing Python modules. It'd be good to grab these per-install log files as well. You'll likely need to use a regex to find them, but they should be in a subdirectory within lib-bad
somewhere.
$toolDir = Join-Path ${Env:SystemDrive} "tools\Cygwin" -Resolve
change default env var from Tools to
see recent comments in mandiant/flare-vm#423
No
Some packages use the same url for different versions which breaks the package every release. We could try to update them automatically. We could use something similar to the daily run that updates the hash if choco fails because the hash doen't match.
Similar to #26
@MalwareMechanic what is the easiest way to get the latest version of a dependency? Is there an API from the community feed? Do we need to use some command of choco and parse the output?
should we install the same packages in python 3 as in python 2? For example pefile and vivisect could be installed in Python 3 (maybe they didn't supported py3 when this file was created).
Currently in PR we only test modified packages to reduce the testing time. However, there are cases were a package can break if another package is modified. For example if we modify a function in common, we could break all packages that use that function, that are not tested if they are not modified.
Different approaches to handle this better:
@vm-packages opinions?
regshot
Regshot-x64-Unicode
1.9.1
Utilities
maddes, regshot, xhmikosr
Regshot is a small, free and open-source registry compare utility that allows you to quickly take a snapshot of your registry and then compare it with a second one - done after doing system changes or installing a new software product.
ZIP_EXE
https://sourceforge.net/projects/regshot/files/regshot/1.9.1-beta/Regshot-1.9.1-beta_r321.7z
5933d59f591e1e68ce7819904f8cb1118fc935bdfe89581599d0560ec9b97cd6
Aids finding differences in the registry after running your malware sample.
From @mr-tz comment in #6 (comment):
Make the linter output easy to see, currently lines like Progress: Downloading PSScriptAnalyzer 1.20.0... 0% add a lot of noise.
This refers to the Powershell linter and it is not related to the new Python linter.
Currently, only 7zip 15.05 (with NSIS support) gets installed.
I like how capa has multiple badges, one of which is the number of rules. I think vm-packages should also have a badge that counts the number of packages!
Seems that happens via a GitHub Action: https://github.com/mandiant/capa-rules/blob/master/.github/workflows/sync.yml
MAP is distributed using the following link for all releases: http://sandsprite.com/idef/map
This makes that an update breaks the package as the hash changes. 😭 We can't ignore the checksum because the URL is using HTTP and Chocolatey complains (with good reasons). 😭😭
dnspyex
dnSpyEx
6.2.0
dotNet
0xd4d, ElektroKill
dnSpyEx is a unofficial continuation of the dnSpy project which is a debugger and .NET assembly editor. You can use it to edit and debug assemblies even if you don't have any source code available.
METAPACKAGE
dnSpyEx
bin\dnSpy.Console.exe
dnSpy rules!
Some packages (such as notepadplusplus.vm
renamed in #98) haven't been pushed to myget because our action only checks modified nuspec files and not renamed ones if they are not modified. Note that in a package change of name, the version is not modified. We need to:
For common and more involved install mechanisms there should exist a helper function like VM-Install-Raw-GitHub-Repo
etc.
See https://github.com/mandiant/VM-Packages/wiki/Contributing#example-4 for a good candidate of tools run via cmd.exe
, for example.
Add option for helper functions to allow user's to specify if tool should be ran as admin
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.