projectdiscovery / fastdialer

Dialer with DNS Cache + Dial History

Fastdialer

Fastdialer is implementation of net.Dialer with lot of features like DNS Cache , Dial History etc

Features

• DNS Cache
• Dial History
• Supports Memory/Hybrid/Disk Cache.
• Supports Old/New TLS and x509 versions
• Supports Resolution Using Hosts File
• Cross Platform and more..

For more details and documentation refer GoDoc.

ZTLS Fallback

fastdialer by default fallbacks to using zcrypto when there is an error in TLS handshake (insufficient security level etc ). This is done to support older TLS versions and ciphers. This can be disabled in fastdialer options or by using DISABLE_ZTLS_FALLBACK=true environment variable. when falling back to ztls, ChromeCiphers are used

Example

An Example showing usage of fastdialer as a library is specified here

cd example/
go run main.go

License

fastdialer is distributed under MIT License

fastdialer's Issues

Add automatic release gh action

• Add action for automatic release (see https://github.com/projectdiscovery/roundrobin/blob/main/.github/workflows/release-tag.yml)
• Ensure new releases are created only if there are new commits (see https://github.com/projectdiscovery/team-backlogs/issues/228)

Add Allow/Exclusion List

Introduction

Multiple tools over time are showing the necessity of allowing/blocking traffic upon specific patterns. This has been implemented in dishomogeneous ways in multiple apps at different OSI layers.
Most allowing/exclusion logic happens at layer 7 (ex. HTTP):

• projectdiscovery/httpx#1517
• projectdiscovery/nuclei#4437

In other tools such as naabu this happens at layer 4 via ipranger

Description

The task is about implementing a generic cross-tool solution capable to plug allow/block capabilities with maximum coverage for all tools. At current time fastdialer seems a good candidate for most of the tools performing network activities as it sits between layer 7 and layer 4. The component should expose helper/callbacks to add items to allow/block list covering the same cases mentioned in projectdiscovery/httpx#1427.

Note: The logic should also work over socks5 proxy

Use `recover()` to catch runtime panics

Proposed Changes

• net/http has recover statement with defer in every goroutine it creates to catch panics
• while running nuclei scan on 250 targets , it paniced after 3hrs 45 min

[3:44:50] | Templates: 7384 | Hosts: 253 | RPS: 141 | Matched: 3255 | Errors: 50463 | Requests: 1911514/2797674 (68%)
panic: runtime error: slice bounds out of range [1721:49]

goroutine 26744483 [running]:
internal/poll.(*FD).Write(0x14016273600, {0x1403009ce00, 0x31, 0x32})
	internal/poll/fd_unix.go:380 +0x3ac
net.(*netFD).Write(0x14016273600, {0x1403009ce00?, 0x104f0f0e0?, 0x10512e220?})
	net/fd_posix.go:96 +0x28
net.(*conn).Write(0x14011ed7d48, {0x1403009ce00?, 0x14042d90fd8?, 0x10289b44c?})
	net/net.go:191 +0x34
github.com/miekg/dns.(*Conn).Write(0x1400dce5400, {0x1403009ce00, 0x31, 0x32})
	github.com/miekg/[email protected]/client.go:359 +0x108
github.com/miekg/dns.(*Conn).WriteMsg(0x1400dce5400, 0xc47f431c4ed?)
	github.com/miekg/[email protected]/client.go:348 +0xe4
github.com/miekg/dns.(*Client).ExchangeWithConnContext(0x140009faa80, {0x1051e4b78, 0x106722380}, 0x14015f11440, 0x1400dce5400)
	github.com/miekg/[email protected]/client.go:220 +0x2a8
github.com/miekg/dns.(*Client).ExchangeWithConn(...)
	github.com/miekg/[email protected]/client.go:187
github.com/miekg/dns.(*Client).Exchange(0x140451584e0?, 0x140451584e0?, {0x14013099540?, 0x104a08320?})
	github.com/miekg/[email protected]/client.go:170 +0xe0
github.com/projectdiscovery/retryabledns.(*Client).queryMultiple(0x14000b6d5f0, {0x140451584e0, 0x14}, {0x14042d913c4, 0x2, 0x1031545f0?}, {0x0, 0x0})
	github.com/projectdiscovery/[email protected]/client.go:361 +0x718
github.com/projectdiscovery/retryabledns.(*Client).QueryMultiple(...)
	github.com/projectdiscovery/[email protected]/client.go:264
github.com/projectdiscovery/retryabledns.(*Client).Resolve(0x1400039c6c0?, {0x140451584e0?, 0x140451584e0?})
	github.com/projectdiscovery/[email protected]/client.go:144 +0x44
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).GetDNSData(0x1400039c6c0, {0x140451584e0?, 0x1051c1700?})
	github.com/projectdiscovery/[email protected]/fastdialer/dialer.go:489 +0x390
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).dial(0x1400039c6c0, {0x1051e4da8, 0x14003ed2e00}, {0x103fcdf09, 0x3}, {0x140451584e0, 0x17}, 0x0, 0x0, 0x0, ...)
	github.com/projectdiscovery/[email protected]/fastdialer/dialer.go:216 +0x29c
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).Dial(...)
	github.com/projectdiscovery/[email protected]/fastdialer/dialer.go:136
github.com/projectdiscovery/retryablehttp-go.DefaultReusePooledTransport.func1({0x1051e4da8?, 0x14003ed2e00?}, {0x103fcdf09?, 0x104d9dfe0?}, {0x140451584e0?, 0x1027f249c?})
	github.com/projectdiscovery/[email protected]/http.go:52 +0x64
net/http.(*Transport).dial(0x1400f4bb5c0?, {0x1051e4da8?, 0x14003ed2e00?}, {0x103fcdf09?, 0x14042d91a58?}, {0x140451584e0?, 0x1?})
	net/http/transport.go:1183 +0xdc
net/http.(*Transport).dialConn(0x14000c39180, {0x1051e4da8, 0x14003ed2e00}, {{}, 0x0, {0x14008fad780, 0x4}, {0x140451584e0, 0x17}, 0x0})
	net/http/transport.go:1625 +0x61c
net/http.(*Transport).dialConnFor(0x140253bf260?, 0x1401def9550)
	net/http/transport.go:1467 +0x7c
created by net/http.(*Transport).queueForDial in goroutine 26738384
	net/http/transport.go:1436 +0x380

this seems mostly due to open fd etc but i think maybe panic could have been avoided if we have added recover and caught it in fastdialer

Custom IP shouldn't perform DNS resolution

Description

When custom ip is provided via the ip context argument, no other ips should be attempted. Actually we put the custom ip as first to try in the list, and then append all ipv4 and ipv6.
This behavior might not be the expected one, as for those tools using scan all ips functionalities it can cause multiple connections to the same target.

Investigate parallel dial

Investigate introducing parallel dialing in fastdialer/fastdialer/dialer.go.dial(...) and returning the first successful connection (with optional reuse/load balancing of other connections) similarly to https://go.dev/src/net/dial.go.dialParallel(...) to reduce failure time and cumulative timeout

If a domain, such as hackerone.com, resolves to multiple IPs, Fastdialer previously used sequential iteration. This was inefficient because if a port on the address was closed, Fastdialer would iterate over all IPs and each failed iteration would add to the total time. This issue was exacerbated when Nuclei called this function on 1000 goroutines, creating a bottleneck and delay in execution.

In memory cache can be generically typed to avoid gob marshaling and reflection

[Feature] Add support for TLS data history

Replace docs example with link to example folder

• Replace example in markdown with link to examples folder
• Ensure the example is compiled as part og GH build/test action

[Feature] ADD SNI server name and custom dial ip support

Implement custom TLSConfig.Clone(...)

Replace deepcopier with custom clone of only meaningful fields

[Maintenance] Add tests

Docs + Maintenance

• Bump go to 1.18
• Fix any lint errors
• Replace deprecated packages
• Add documentation
• Add usage examples with examples folder
• Build examples as part of build github action
• Extend build/lint action to be multi-platform

Multiple stderr messages with current date and path

Hello,

I found that when I try using this library in my own projects, my stderr gets flooded with multiple messages. I'm unsure if I'm doing something wrong to cause these errors messages, but no error seems to be returned during initialization and everything runs fine.

stderr

2021/06/12 01:36:09 /tmp/hm928258121
2021/06/12 01:36:09 /tmp/hm341994260
2021/06/12 01:36:09 /tmp/hm957686115
2021/06/12 01:36:09 /tmp/hm497662310
2021/06/12 01:36:09 /tmp/hm236611981
2021/06/12 01:36:09 /tmp/hm279069576
2021/06/12 01:36:09 /tmp/hm406964039
2021/06/12 01:36:09 /tmp/hm423496186
2021/06/12 01:36:09 /tmp/hm279549201
2021/06/12 01:36:09 /tmp/hm890850108
2021/06/12 01:36:09 /tmp/hm536011627
2021/06/12 01:36:09 /tmp/hm160537038

How I use fastdialer

func buildHttpClient() (c *http.Client) {
	fastdialerOpts := fastdialer.DefaultOptions
	fastdialerOpts.EnableFallback = true
	dialer, err := fastdialer.NewDialer(fastdialerOpts)
	if err != nil {
		fmt.Printf("Error initializing dialer: %s\n", err)
		return
	}

	transport := &http.Transport{
		MaxIdleConns:      -1,
		IdleConnTimeout:   time.Second,
		TLSClientConfig:   &tls.Config{InsecureSkipVerify: true},
		DisableKeepAlives: true,
		DialContext:       dialer.Dial,
	}

	re := func(req *http.Request, via []*http.Request) error {
		return http.ErrUseLastResponse
	}

	client := &http.Client{
		Transport:     transport,
		CheckRedirect: re,
		Timeout:       time.Second * 10,
	}

	return client
}

I should note, this only happens when using fastdialer. I switched over a couple of projects to use the library and had these messages pop up on all of them.

newest release and source code are different, can you update?

newest release and source code are different, can you update?

How to slove this memory leak issue?

Here's my monitoring results, the memory leak is worse.

dns proxy

I want to know if the Socks5 proxy is used when querying DNS?

[Feature] Add in-memory only support

• Max cache size
• Automatic eviction

Add support for tls impersonate in net.dialtls

Ref: projectdiscovery/nuclei#3330

• https://www.zenrows.com/blog/what-is-tls-fingerprint
• https://scrapfly.io/blog/how-to-avoid-web-scraping-blocking-tls/

[Feature] Resolver file support

Add resolvers from /etc/resolv/conf on unix systems

fix nil hostfile data during reuse

Proposed Changes

• due to incorrect referencing of variable (global -> local) . consequent calls to loadhostFile were returning nil data

Default DNS resolvers should pull from system settings

Currently, the default state for HTTPX/Naabu/Nuclei is to use a set of default resolvers for DNS as defined in the DefaultResolvers variable within options.go in fastdialer.

// DefaultResolvers trusted
var DefaultResolvers = []string{
	"1.1.1.1:53",
	"1.0.0.1:53",
	"8.8.8.8:53",
	"8.8.4.4:53",
}

This is a deviation from the expected behavior, which is to use the host systems DNS configuration as a default. This is, for example, the way curl works.

There are a few significant drawbacks to doing this.

• When doing an internal scan, some organizations block outgoing DNS requests that aren't going to their own DNS servers.
• When conducting an internal red team style engagement where stealth is a factor, seeing a large number of DNS requests to 8.8.8.8, 1.1.1.1, .etc, may be a red flag for network defenders in an environment where DNS servers are explicitly set.

fastdialer ztls fallback + bug fix

Proposed Changes

• earlier we tried implementing fallback in projectdiscovery/retryablehttp-go#99 but nuclei uses a custom DialTLSConfig due to client certificates . hence fallback logic is not applied at all now with fastdialer it is preserved and available by default
• adds ztls fallback with chrome ciphers to resolve projectdiscovery/nuclei#2805
• when using fastdialer.DialTLS renegotiation was not set which caused projectdiscovery/nuclei#3553 in nuclei

utls with proxy

i know it has supported utls in tls handshake,but it does not support utls with socks5 proxy support,so iadvise to add this feature

expose allow/deny options

Add custom type for ip context arg

update utls

Some bug fixes since v1.5.3

Use utils helper libraries

Use utils.