bytedance / g3 Goto Github PK
View Code? Open in Web Editor NEWEnterprise-oriented Generic Proxy Solutions
License: Apache License 2.0
Enterprise-oriented Generic Proxy Solutions
License: Apache License 2.0
当有多个代理服务器的时候,可以共享一个用户auth限制,比如现在限制并发数,只能针对单台服务器独立限制,不能全局统一限制
已解决!!!
Does anyone need it?
能提供二进制文件方便测试 学习吗? 谢谢
When using g3proxy it is sometime detected by websites running on cloudflare and other sites using TLS fingerprinting such as https://github.com/salesforce/ja3. Having the ability to modify the ClientHello will make the proxy less detectable/less fingerprintable and more usable in enterprise environments.
The ideal “solution” would be to be able to set a ja3 fingerprint and have the proxy send it.
I have not found a way to modify it via OpenSSL but it appears rustls may give access to the ClientHello https://docs.rs/reqwest/latest/reqwest/struct.ClientBuilder.html#method.use_preconfigured_tls
Here is a bit more info and code examples:
https://medium.com/cu-cyber/impersonating-ja3-fingerprints-b9f555880e42
https://github.com/refraction-networking/utls
https://github.com/Danny-Dasilva/CycleTLS
See
https://bytedance.feishu.cn/docx/TSqmdLNIyozJdCx4rzqcjPyonjd for AWS c6gn 32c64g
https://bytedance.feishu.cn/docx/Icind5zlMowawRx1hVtcGyc9nNf for Intel 96c384g
After change the rustls server session cache to RustlsTrickServerSessionCache
introduced in commit 2b4e269, the aws 32c64g performance scale well again, but there are still some problems with Intel 96c384g.
And RustlsTrickServerSessionCache
isn't ideal, we need a better solution as suggested in rustls/rustls#1503.
And we should try aws-lc after rustls/rustls#1414 is merged.
G3 is a great project! The documentation on how to fully setup everything end to end unfortunately is a bit lacking.
I am creating this issue to track improvements in documentation.
I will definitely help on the English side/work with the team to translate as best as possible/submit PRs for them. Also I would like to help submit new example configs as well.
@zh-jq - would you like me to create new issues for all areas that are needing documentation or should we keep that just under this main issue?
Example:
How would I use g3proxy to do forward proxying with SSL inspection? (Needing to use g3fcgen etc).
How do I configure a “frontend” with g3fcgen?
These as just examples. @zh-jq if it’s easier to quickly right your docs in Chinese I can use tools to translate - but will need to have you review for accuracy.
Thanks again for all your work!!
Non critical Feature Request:
When doing TLS MITM the certificate provided to the client currently only has a matching SNI. In an ideal scenario the certificate provided to the client should match as many fields as possible from the remote server certificate (outside of ones such as serial etc). This may cause a small performance hit but since the code already pulls the SNI do I not believe it will be major.
Currently this is not causing major issues anywhere. But is something to keep in mind to feature match closed source proxies.
感谢
我们现在需要让两台服务器之间使用国密https通讯,需要双向认证,但是不知道g3proxy如何配置,如何使用,有没有示例,多谢了
h2:
cargo:
OpenSSL:
Rustls:
QAT Engine:
OpenSSL:
OpenSSL:
Rustls:
Helps fix the build / release scripts changes introduced by boringssl for g3fcgen packaging.
可删除此issues,已自魔改~
Building and running in docker would help speed up development and adoption with new developers. In addition to allowing fast deployment to kubernetes clusters.
I am working on getting a dockerfile working properly with G3 but in case someone already has one I wanted to create an issue.
If no one has one I will submit a PR once it’s done.
代理串联支持二级代理账号密码验证吗?
会不会考虑增加?例如socks5(用户认证)->socks5(用户认证)
Hi!
Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are here. E.g. PGO helps with optimizing Envoyproxy. PGO results for other proxies like HAProxy, Nginx, httpd you can be found in the repo above. According to the multiple tests, PGO can help with improving performance in many other cases. That's why I think trying to optimize g3 with PGO can be a good idea.
I can suggest the following action points:
Maybe testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.
For the Rust projects, I recommend starting experimenting with PGO with cargo-pgo.
Here are some examples of how PGO optimization is integrated in other projects:
目前只支持PPV2;
大部分厂商的负载均衡都是ProxyProtocolV1
使用发行版原生安装包安装的,已经安装了systemd参数化服务配置文件,参数就是进程组名称, 对应的入口配置文件存放路径为/etc/g3proxy/<daemon_group>/main.yml。
这里看起来应该是“/etc/<daemon_group>/main.yml”, 比如"/etc/g3proxy/simple_fwd_proxy/main.yml"
I may have missed the setting somewhere - but is it possible to disable HTTP2 (and 3) proxing via configuration? I would like to force all client/server connections to use HTTP1. If not is there a specific location you recommend we make the change in code?
Thanks!
I have g3 running and have started testing ICAP integration but have run into weird issues.
When I start g3 it sends an initial OPTIONS to the ICAP server. But after that it does not send reqmod/respmod to the server even with the ratio set to 1. If I turn down the ratio to .5 it will occasionally send a Icap res/req. Maybe 1/100. I am wondering if it could be an issue with the ratio code. I have not seen any errors in the g3 logs.
I have started trying to track down where the issue might be but I wanted to file a ticket in case you might know or have suggestions. I will also add more general logging output to the Icap while going through this then file a PR.
I am using the open source icapeg server for testing. Below is my config
auditor:
- name: default
protocol_inspection: {} # Enable protocol recognition, use default parameters
tls_cert_generator: {} # Enable TLS hijacking, use default parameters
tls_interception_client: {} # Can configure proxy to target address TLS connection parameters
h1_interception: {} # HTTP/1.0 parsing parameters
h2_interception: {} # HTTP/2 parsing parameters
icap_reqmod_service: icap://127.0.0.1:1344/echo # ICAP REQMOD service configuration
icap_respmod_service: icap://127.0.0.1:1344/echo # ICAP RESPMOD service configuration
application_audit_ratio: 1.0
Hello!
I'm working with @mspublic to get g3proxy running in Docker. Currently, it's not possible to stream the udpdump traffic to Wireshark on the host machine because g3-yaml only accepts ip:host
. In order to get this working, we need to use host.docker.internal:5555
in the config.
If you don't mind, I have a PR that changes calls to SocketAddr::from_str()
to ToSocketAddrs::to_socket_addrs()
in a couple of places related to config parsing, mainly in g3-yaml.
Currently we only support set just one RESPMOD/REQMOD service in auditors, does anyone need ICAP chaining?
wsl2-archlinux编译,默认Features,请问怎么解决?
Compiling g3-io-ext v0.6.0 (/home/debu/git/rust/g3/lib/g3-io-ext) error: cannot construct
msghdrwith struct literal syntax due to private fields --> lib/g3-io-ext/src/udp/ext.rs:134:9 | 134 | libc::msghdr { | ^^^^^^^^^^^^ | = note: ...and other private fields
__pad1and
__pad2` that were not provided
error: cannot construct msghdr
with struct literal syntax due to private fields
--> lib/g3-io-ext/src/udp/ext.rs:179:9
|
179 | libc::msghdr {
| ^^^^^^^^^^^^
|
= note: ...and other private fields __pad1
and __pad2
that were not provided
error[E0308]: mismatched types
--> lib/g3-io-ext/src/udp/ext.rs:337:21
|
333 | libc::sendmmsg(
| -------------- arguments to this function are incorrect
...
337 | libc::MSG_DONTWAIT | libc::MSG_NOSIGNAL,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u32
, found i32
|
note: function defined here
--> /home/debu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libc-0.2.153/src/unix/linux_like/linux/musl/mod.rs:828:12
|
828 | pub fn sendmmsg(
| ^^^^^^^^
help: you can convert an i32
to a u32
and panic if the converted value doesn't fit
|
337 | (libc::MSG_DONTWAIT | libc::MSG_NOSIGNAL).try_into().unwrap(),
| + +++++++++++++++++++++
error[E0308]: mismatched types
--> lib/g3-io-ext/src/udp/ext.rs:397:21
|
393 | libc::recvmmsg(
| -------------- arguments to this function are incorrect
...
397 | libc::MSG_DONTWAIT,
| ^^^^^^^^^^^^^^^^^^ expected u32
, found i32
|
note: function defined here
--> /home/debu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libc-0.2.153/src/unix/linux_like/linux/musl/mod.rs:834:12
|
834 | pub fn recvmmsg(
| ^^^^^^^^
help: you can convert an i32
to a u32
and panic if the converted value doesn't fit
|
397 | libc::MSG_DONTWAIT.try_into().unwrap(),
| ++++++++++++++++++++
For more information about this error, try rustc --explain E0308
.
error: could not compile g3-io-ext
(lib) due to 4 previous errors `
We have been running into a few issues with the TLS stream dump functionality. I followed the udp dump/exported_pdu instructions in wireshark.
Wireshark is not able to fully parse the different streams (TCP/HTTP/etc). For example when you right click on an HTTP request then select follow-HTTP. It is unable to reassemble and follow the stream. Even if the dump has been running for a while with many requests wireshark identifies newer requests as an early tcp.stream (for example 1 or 2). This leads me to believe it's not able to properly discern between different TCP streams.
The source and destination are of the proxy server and the udpdump receiver. Ideally these would be of the connecting client and remote server. Or at least between the proxy and the remote server.
I believe due to issue 1 we are seeing HTTP [Malformed Packet] errors often.
Thanks for any help/suggestions!
I am trying to configure g3proxy to do HTTP forward proxying with SSL inspection (and ICAP adaptation).
Using the examples I found I was able to create the following config file. It will forward HTTP traffic but does not seem to do SSL interception or send ICAP adaptation requests.
It looks like you have fixed the initial g3fcgen issue I was running into. But I have found some more issues which seem to be causing failures.
Blocked by
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.