gospider007 / gospider

🚀Gospider is a powerful Golang web crawler that includes all the necessary libraries for transitioning from Python to Golang. It provides a fast and seamless transition for Python web crawlers to Golang.

[Warehouse Migration Notice]

Dear users,

We are delighted to announce that, in order to provide better service and user experience, we will be migrating our warehouse. In the new warehouse, you will enjoy more features and improvements to meet your needs.

During the migration process, access to the original warehouse will be temporarily suspended. During this period, you will not be able to access the content and features in the original warehouse. To ensure that you are not inconvenienced, we have prepared a brand new warehouse that will be activated immediately after the migration is complete.

The new warehouse will offer the following improvements and advantages:

1. Faster loading speed: We have optimized the new warehouse to provide faster loading speed and response time, allowing you to browse and use the warehouse content more efficiently.

2. Memory leak optimization: The new warehouse adopts a brand new connection pool design, greatly improving memory leak issues.

3. Stronger underlying control: The new warehouse has optimized the underlying design, surpassing the official implementation of net/http in terms of connection control and ensuring the complete resolution of memory leak issues.

4. Enhanced stability: The new warehouse uses an optimized http2 library, combined with a deeply optimized connection pool, greatly enhancing the stability and speed of http2.

5. Fingerprint stability: Seamless conversion between ja3 fingerprint, http2 fingerprint, and underlying net.conn implementation eliminates the need for intermediaries, greatly improving performance and stability.

6. More features and resources: We will continue to update and add new features and resources to meet your needs and feedback. You can expect more learning materials, sample code, and tools.

After the migration, you can access the new warehouse by following these steps:

1. Open your browser and visit the URL of the new warehouse: [https://github.com/gospider007/requests].

2. Explore the content, resources, and features of the new warehouse and embark on your learning and exploration journey.

If you encounter any issues or have any questions during the migration process, please feel free to contact me. I am here to provide assistance and support.

Thank you for your continued support and trust in our warehouse. We look forward to exploring and growing together in the new warehouse.

Thank you!

gospider's Issues

两次handshake耗时比较久

同样使用代理的情况下，总体耗时是正常使用net/http或fasthttp请求耗时的2到3倍，
看了下好像是这里两次handshake耗时比较久，

if err = utlsConn.HandshakeContext(ctx); err != nil {
		if strings.HasSuffix(err.Error(), "bad record MAC") {
			err = tools.WrapError(err, "检测到22扩展异常,请删除此扩展后重试")
		}
	}

有什么优化方案吗

proxy的socks5 不支持udp

当ja3id指定为ja3.HelloAndroid_11_OkHttp会发生错误

代码如下:

package main

import (
	"context"
	"gitee.com/baixudong/gospider/ja3"
	"gitee.com/baixudong/gospider/requests"
	"log"
)

func main() {
	reqCli, err := requests.NewClient(context.TODO())
	if err != nil {
		log.Panic(err)
	}
	response, err := reqCli.Request(context.TODO(), "get", "https://tools.scrapfly.io/api/fp/ja3?extended=1", requests.RequestOption{Http2: true, Ja3Id: ja3.HelloAndroid_11_OkHttp})
	if err != nil {
		log.Panic(err)
	}
	log.Print(response.Text())
}

报错如下
2023/03/10 16:44:30 Get "https://tools.scrapfly.io/api/fp/ja3?extended=1": remote error: tls: protocol version not supported panic: Get "https://tools.scrapfly.io/api/fp/ja3?extended=1": remote error: tls: protocol version not supported
更换为其他的id则没问题

Accept-Encoding的问题

最近get一个网站的时候发现出现403，查了一下，发现resp返回Vary是Accept-encoding。用检测发送头的网站看了一下，好像req里面的Accept-encoding只有gzip。而网站要求的是要带br。看了一下代码，默认的Accept-encoding是"gzip, deflate, br"，不知道为什么在https://www.whatismybrowser.com/detect/what-http-headers-is-my-browser-sending 检测只有gzip。操作系统是win10，是否跟安装的库有关？

how install

can you build binary
go build not work and
go get -u gitee.com/baixudong/gospider not work

不能在requests.ClientOption中设置Ja3Spec

有时候需要动态Ja3，放在requests里面比较方便

能否把module改为github

功能请求: 自定义ja3指纹

很好用的一个库，感谢作者的付出，不知道大佬后续有无关于自定义ja3指纹的计划:
比方说在requests.RequestOption加入个ja3string参数，使用者即可将获取的标准指纹("771,49195....")填入其中进行伪造。

自实现代理协议与go代理协议的实现有差异

Doesn't work with resident proxies? It's just that if I run with regular proxies, everything works, but not with resident proxies.)