projectdiscovery / utils

Helper Libraries

utils

The package contains various helpers libraries

utils's Issues

`url.URL` wrapper and enhancements

Proposed Changes

• Adds a robust url.URL wrapper that handles all parsing issues and most of the edgecases
• Now any special characters(even the reserved RFC chars) are not encoded >projectdiscovery/nuclei#3166
• Adds helper methods and more

Use case

• All encoding issues are now abstracted
• integrity of nuclei templates is not compromised

References

• projectdiscovery/nuclei#3208
• projectdiscovery/nuclei#3167
  and more

Skip automatic release without new commits

Description

A new release should not be created if there are no changes since the previous one (ref: https://github.com/projectdiscovery/team-backlogs/issues/228)

use standard format for url encoding

Proposed Changes

• while %d is considered as url encoded characters and accepted by most decoders . According to RFC it should be %0D i.e two digit hex and capital chars only .
• Go Stdlib also follows RFC and has above format

Add generic syncmap with locks

Create a new map helper capable of adding the following functionalities to an existing map:

• Supports concurrent R/W
• Can be Locked/Unlocked (Read-Only mode)

List of potential supported methods

Put(key,value)
Get(key,value)
Interate( func(key,value) )

Setup bi-weekly automatic release

Description

Setup a similar action to https://github.com/projectdiscovery/wappalyzergo/blob/master/.github/workflows/release-tag.yml with bi-weekly timing

Add helper function to count lines in files

Description

Add in utils/file/file.go a new helper function to count the number of lines in a file:

func CountLine(filename string) (uint64,error)
func CountLineWithSeparator(separator, filename string) (uint64,error)

Optionally, the following behavior could be supported

• By default (if not differently provided), the new line character should be considered as a separator
• The separator could be a regular expression
• Skip Empty Lines
• Additional Helper function to process multiple files and return a struct with filename e number of lines/words in the file.

Please use semantic versioning

I know it's not easy to work with them in go as once you go over v1 you need to change paths all the time, but please use semantic versioning for this project. This project is used in other projects from projectdiscovery and builds fail all the time because go update updates the minor versions automatically but there are often breaking changes instead and so builds depending on the other libraries tend to fail very often.

Examples:
projectdiscovery/retryablehttp-go#47
projectdiscovery/retryabledns#67

update utils improvements

Proposed Changes

• fix GetUpdateCallback not using http timeout
• Disable proxying upgrade traffic (i.e github get latest release , download binary etc)

Additional Count Lines helpers

Description

• Add in utils/file/file.go a new helper function to count the number of matches in a file where the separator is a regular expression

func CountLineWithRegex(re regexp.Regexp, filename string) (uint64,error)

• Add support to skip empty lines

urlutil returns corrupted url if url contains invalid chars in path

Description

• https://scanme.sh/%usdf%0D%0A is a invalid url since %u is not a valid encoding character
  • url.URL returns error if above url is given
  • utils/url/URL was designed to allow such cases
• However it returns a corrupted url instead of valid one for this case i.e

https://scanme.sh/https://scanme.sh/%usd%0D%0A

Executil test failing in Windows

Steps to reproduce:
https://github.com/projectdiscovery/utils/actions/runs/3378744412/jobs/5609479078

Add ContainsAnyI to slice utils

Add case insensitive version of strinsgutil.ContainsAny:

func ContainsAnyI(s string, ss ...string) bool

"IsIP" and "IsIPv4" are not able to i identify short ip address as Valid Ip address

For Example "35.1" is a valid ip address which equivalent to "35.0.0.1" and this kind of IP can be found in DNS record TXT

following example can show how these kind of can be recognised by single tool like "ping" and "nmap"

Add FirstNonZero helper function

Proposed Changes

• func FirstNonZero[T comparable](inputs []T) (T, bool)

We need to verify that the slices we return have elements that are not zero.

Add helper methods to url utils

Proposed Changes

// EscapedString returns a string that can be used as filename (i.e stripped of / and params etc)
func (u *URL) EscapedString() string {}

// UpdateRelPath updates relative path with new path (existing params are not removed)
func (u *URL) UpdateRelPath(newrelpath string, unsafe bool) {}

// TrimPort if any
func (u *URL) TrimPort() {}

Anything else

These helper function are used in httpx

IsInternal(str string)

iputil => https://github.com/projectdiscovery/retryabledns/blob/410b6d31595f720ed5d2b8c35a5db9d2607aa71e/validate.go

Add support for BSD capabilities check in permission package

Support for the following syscalls should be added:

• cap_enter
• cap_getmode

Ref: https://man.freebsd.org/cgi/man.cgi?cap_enter(2)

Add slice.Merge to slice utils

While reviewing projectdiscovery/proxify#205 the following functions were found to be potentially useful:

sliceutil.Merge(slices []any)
sliceutil.MergeItems(items ...any)

Add URL Encoding Utils

Description

From projectdiscovery/nuclei#2698

• In Nuclei all parameters are stored in url.Values and added to URL using query.Encode() method which url encodes parameters by default. thus all parameters passed via input and all fuzzing payloads are url encoded which is basically double url encoding

• URL Encoded similar to url.values should be implemented which doesnot encode any characters (except space).
• A Method Similar to burpsuite intruder should be implemented where only given characters are url encoded
• And any other methods depending on security perspective of URL encoding

Add GetHostname function

given an input url, returns the output hostname

Move https://github.com/projectdiscovery/naabu/v2/pkg/privileges into privilege utils

Add helpers to detect OS type and Arch as boolean

Description

Add as helpers from naabu:

// Global consts (useful for switch constructs)
type OsType uint8

const (
	Darwin OsType  = iota
	....
)

var OS OsType 

func init () {
   OS = ...
}

// Global Helpers
func IsOSX() bool {
	return runtime.GOOS == "darwin"
}

func IsLinux() bool {
	return runtime.GOOS == "linux"
}

func IsWindows() bool {
	return runtime.GOOS == "windows"
}

Add Github actions

• Lint
• Depedantbot
• CodeQL
• Tests

Move github.com/projectdiscovery/naabu/v2/pkg/routing into routing utils

Notes:

• Check if this is possible without requiring the libpcap dependency from gopacket
• In negative case, implement routing by parsing the output of the routing command. Probably the same logic of darwin can be reused.

Tests should ignore map iteration order

Description

A few tests fails randomly because the map values are compared with random order. These tests should be modified to make comparison order agnostic (or eg. using require.ElementsMatch).

Ref: https://github.com/projectdiscovery/utils/actions/runs/3383736473/jobs/5619918170#step:4:96

executils should identify CLI engine to invoke on windows

Description

The executils package should attempt to identify the proper CLI engine avaliable to invoke for piped commands (cmd.exe, sh, powershell, etc). Actually it runs sh on linux and osx.

Add support for nil errors in errorsutil.IsAny()

Add generic comparison helper function

Proposed Changes

• Add Generic Comparison helper functions
• func EqualsAny[T comparable](base T,all ...T)
• func EqualsAll[T comparable](base T,all ...T)

This can be used in various places for comparing multiple values , errors etc

References:
https://github.com/projectdiscovery/uncover/blob/eced79464c717e39761989d0abafce4be2dc2caf/runner/options.go#L134-L148

Move https://github.com/projectdiscovery/naabu/v2/pkg/israce in race utils

Implement tests for helper functions - mapsutil

Description

Implement tests for the following functions:

• Flatten
• Walk

Check if these functions are aligned with nuclei implementation, otherwise update them and implement tests:

• HTTPToMap
• DNSToMap
• HTTPRequesToMap
• HTTPResponseToMap

reader.TimeoutReader go routine leaks via time.Sleep

Add support for search by key/value in generic map

Description

On many occasions (e.g., to map from Type to string and vice versa), it's necessary to hold in memory a key=>value and value=>key maps. Implementing a generic mechanism to abstract the two-way search through helper functions over existing map types would be helpful. The new method' signatures might look like this:

func (m Map[K, V]) GetKeyWithValue(value V) (K,bool)

Investigate failing test in file utils

Ref: https://github.com/projectdiscovery/utils/actions/runs/3461747254/jobs/5779745743

Add Reusable Reader

Description

Implement a reader that can be reused multiple times

Add contextutils

Description

The following function:

contextutils.WithValues(ctx context.Context, name1, value1, name2, value2, ...)

could replace:

ctx := context.WithValue(context.Background(), name1, value1)
ctx = context.WithValue(ctx, name2, value2])
...

Add more string helpers

• IsPrintable(string) checks if the string is composed only of printable characters
• IsCTRLC checks if the input string is CTRL+C

release-tag.yml runs but no release is created

Ref: https://github.com/projectdiscovery/utils/actions/workflows/release-tag.yml but last release is release v0.0.3 (Nov 13 2022)

Add Weighted Error type to errorsutil

Use case at: projectdiscovery/nuclei#3165 (comment)

TBD, but it should be possible to:

• Calculate the final score of all weighted error
• Provide boost factor for error type
• Define some super common error types with predefined weight along with the possibility of defining custom ones

Add proxyutils package

Description

Many projects (nuclei, httpx, etc.) support http/https/socks5 proxies, and the logic is repeated redundantly. It would be useful to move the logic into a shared proxyutils package. The list is not complete, but it seems like the necessary methods should look like the following ones:

• GetProxy(proxies ...string) ProxyFunc // Validate and get a valid proxy from a list
• IsBurpProxy(url string) bool // checks if the proxy is burp (e.g. requesting http://burpsuite - which requires tls max version to 1.2 for https targets)

Add error type with default format

Description

Often, an error must be defined with a static signature and accept later arguments. It would be super useful the possibility to declare a generic error with a signature and populate it later with runtime values:

var ErrGeneric := errorutil.NewWithFmt("error in %s with %s")
...
return ErrGeneric.Msgf("location", "error type") // binds arguments defined with NewWithFmt

Add Lookup PTR to iputils

Add Truncate method to stringsutil

Ref: projectdiscovery/proxify#226

Add get/set helpers on struct private fields in reflectutils

GetUnexportedField(struct pointer, fieldname)
SetUnexportedField(struct pointer, field, value)

Add `ParseURL` helper function

Proposed Changes

1. Default url.Parse conflicts with Params which causes inconsistencies in parameter encoding/decoding . Implement ParseURL to resolve this projectdiscovery/nuclei#3167
2. Add helper function JoinPaths that joins two relative paths and their parameters without normalising relative paths projectdiscovery/nuclei#3175

add utils to update app from gh releases

Proposed Changes

• add utils to update binary from github releases

Adding helper in strings to calculate the longest non overlapping substring in a string

abcabcabc => abc

Port structure.Walk func to reflect utils

Ref: projectdiscovery/nuclei#3188

Update mapsutil

• Merge instead of MergeMaps (mapsutil.Merge)
• HTTPRequestToMap test
• HTTPResponseToMap test
• Add Clear function
• Add SliceToMap function & test
• Fix test using ElementsMatch (#12)
• Add IsEmpty function

Add error utils

Proposed Changes

Helper functions:

• errorsutil.IsAny(err, openssl.ErrNotAvailable, ...) checks if err matches anyone of given errors
• errorutil.WrapfWithNil(err,format string,args ...any) unlike errors.Wrapf this only wraps another error if given error is not nil
• errorutil.WrapWithNil(err,newerrors ...error) . wrap one or more errors if given err is not nil

based on reference projectdiscovery/tlsx#144

Proposed Enriched error Type (struct implementing the error interface):

type Error struct {
	func  Error() string // from standard error interface
	func Equal(errs ...Error) bool // allows error comparison error1.Equal(error2)
	StackTrace string // captures runtime.PrintStack()
	Tag string // should work like a prefix eg; "openssl" error description or to compare errors group
        Level ErrorLevel // enum of Panic/Fatal/Runtime/Etc - not sure if that might be helpful
}

Add more slice helpers

• IsEmpty
• ElementsMatch
• Diff