Coder Social home page Coder Social logo

anggadaz / package-analysis Goto Github PK

View Code? Open in Web Editor NEW

This project forked from ossf/package-analysis

0.0 1.0 0.0 1.55 MB

Open Source Package Analysis

License: Apache License 2.0

Shell 1.99% JavaScript 2.06% Ruby 2.39% Python 7.27% Go 77.02% HCL 5.80% Dockerfile 3.46%

package-analysis's Introduction

Package Analysis

This repo contains a few components to aid in the analysis of open source packages, in particular to look for malicious software.

For examples of what this project has detected, checkout the case studies

This code is designed to work with the Package Feeds project, and originally started there.

The components are:

  • A scheduler - creates jobs for the analysis worker from Package Feeds.
  • Analysis (one-shot analyze and worker) - collects package behavior data through static and dynamic analysis of each package.
  • A loader - pushes the analysis results into BigQuery.

The goal is for all of these components to work together and provide extensible, community-run infrastructure to study behavior of open source packages and to look for malicious software. We also hope that the components can be used independently, to provide package feeds or runtime behavior data for anyone interested.

Configuration

Configuration for these subprojects consist of a collection of environment variables for the various endpoints. These endpoints are configured using goclouddev compatible URL strings. In these cases, documentation will be linked to and DRIVER-Constructor sections should be ignored in favour of DRIVER sections as these are appropriate to the configurations in place throughout these subprojects. Note that not all drivers will be supported but they can be added quite simply with a minor patch to the repository. See the addition of kafka for scheduler in one line.

An example of these variables can be found in the e2e example docker-compose.

Analysis

OSSMALWARE_WORKER_SUBSCRIPTION - Can be used to set the subscription URL for the data coming out of scheduler. Values should follow goclouddev subscriptions.

OSSF_MALWARE_ANALYSIS_RESULTS - OPTIONAL: Can be used to set the bucket URL to publish results to. Values should follow goclouddev buckets.

OSSF_MALWARE_ANALYSIS_PACKAGES - OPTIONAL: Can be used to set the bucket URL to get custom uploaded packages from. Values should follow goclouddev buckets.

Scheduler

OSSMALWARE_WORKER_TOPIC - Can be used to set the topic URL to publish data for consumption by Analysis workers. Values should follow goclouddev publishing.

OSSMALWARE_SUBSCRIPTION_URL - Can be used to set the subscription URL for the data coming out of package-feeds. Values should follow goclouddev subscriptions.

Local Analysis

To run the analysis code locally, the easiest way is to use the Docker image gcr.io/ossf-malware-analysis/analysis. This can be built from ./build/build_docker.sh, or the public images can be used instead.

This container uses podman to run a nested, sandboxed (gVisor) container for analysis.

The commands below will dump the JSON result to /tmp/results.

Live package

To run this on a live package (e.g. the "Django" package on pypi.org)

$ mkdir /tmp/results
$ docker run --cgroupns=host --privileged -ti \
    -v /tmp/results:/results \
    -v /var/lib/containers:/var/lib/containers \
    gcr.io/ossf-malware-analysis/analysis analyze \
    -package Django -ecosystem pypi \
    -upload file:///results/

Local package

To run this on a local package archive (e.g. /path/to/test.whl for a package named test), it needs to be mounted into the the container.

$ mkdir /tmp/results
$ docker run --cgroupns=host --privileged -ti \
    -v /tmp/results:/results \
    -v /var/lib/containers:/var/lib/containers \
    -v /path/to/test.whl:/test.whl \
    gcr.io/ossf-malware-analysis/analysis analyze \
    -local /test.whl -package test -ecosystem pypi \
    -upload file:///results/

Notes

--privileged and a compatible filesystem are required to properly run nested containers. -v /var/lib/containers:/var/lib/containers is used in the examples above as it allows caching the sandbox images and supports local developement.

Development

Required Dependencies

  • Go v1.17
  • Docker
  • libpcap-dev

Contributing

If you want to get involved or have ideas you'd like to chat about, we discuss this project in the OSSF Securing Critical Projects Working Group meetings.

See the Community Calendar for the schedule and meeting invitations.

package-analysis's People

Contributors

dependabot[bot] avatar calebbrown avatar oliverchang avatar dlorenc avatar naveensrinivasan avatar jordan-wright avatar qinusty avatar tom--pollard avatar david-a-wheeler avatar steiza avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.