feeicn / gsil

GitHub Sensitive Information Leakage（GitHub敏感信息泄露监控）

🍻Welcome to my Cyberspace

I'm a cybersecurity researcher and programmer, Work and live in Hangzhou, China.

您好，我是止介（Feei），工作和生活在杭州，是一名安全工程师，同时也是一名程序员。

由于工作太忙，之前各种项目无时间维护，本账号仅用作关注最新安全开源项目所用。

gsil's Issues

总是触发abuse detection mechanism

现在总是触发abuse detection mechanism规则，总的只有几条规则，使用了多个token,并且cron一天只执行一次，也总是触发abuse detection mechanism规则，从日志上可以看到是远远没达到5000的限制的，一般才发几次请求就触发了，求解

邮件发送不成功如何解决？

smtplib.SMTPServerDisconnected: Connection unexpectedly closed: [Errno 104] Connection reset by peer

“全部匹配”模式下，增加取后10行

刚刚在翻译时，发现参数说明那块的匹配说明不太对，参数直接为中文了，会让使用者有点迷惑。

于是我翻了下源码，找到了匹配关键行和匹配存在关键词的行及其上下3行，但是却没有找到全部匹配。

我查看的地址: https://github.com/FeeiCN/GSIL/blob/master/gsil/engine.py#L217-L272

只有only-match和normal-match, 却没有full-match. 这是否是一个问题呢？

本想在翻译时直接剔除全部匹配，但是感觉不太合适，于是提了一个issues来向你求证

怎么获得某个企业的相关特征

不支持中文

logger.debug('N:{x}/{l}: {c}'.format(x=i_idx, l=codes_len, c=codes[i_idx]))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u5e74' in position 22: ordinal not in range(128)

Exception: json decoder

少数情况下会有以下异常

/var/app/gsil/gsil.py meili 
Traceback (most recent call last): 
File "/var/app/gsil/gsil.py", line 28, in sys.exit(gsil()) 
File "/var/app/gsil/gsil/__init__.py", line 102, in gsil generate_report(running_data) 
File "/var/app/gsil/gsil/__init__.py", line 83, in generate_report run_data = daily_run_data() 
File "/var/app/gsil/gsil/config.py", line 293, in daily_run_data c = json.loads(c) 
File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) 
File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 
File "/usr/local/lib/python3.6/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 122847 (char 122846)

请教运行的问题

您好。 我按照你的操作执行到 检测的时候，运行：python gsil.py test 后， 执行完了后就报错说我邮件配置的问题， 我使用自己的QQ邮箱配置的， 还是不行， 然后我直接没有配置了，还是报错。 在配置了qq邮箱的时候报错如下：
Traceback (most recent call last):
File "/usr/lib/python3.6/configparser.py", line 1138, in _unify_values
sectiondict = self._sections[section]
KeyError: '1312**[email protected]'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/farmsec/gsil/gsil/config.py", line 42, in get
value = config.get(level1, level2)
File "/usr/lib/python3.6/configparser.py", line 781, in get
d = self._unify_values(section, vars)
File "/usr/lib/python3.6/configparser.py", line 1141, in _unify_values
raise NoSectionError(section)
configparser.NoSectionError: No section: '1312[email protected]'
GSIL/config.gsil file configure failed.
Error: No section: '13128[email protected]'
Traceback (most recent call last):
File "/home/farmsec/gsil/gsil/init.py", line 37, in search
return Engine(token=token).search(rule)
File "/home/farmsec/gsil/gsil/engine.py", line 217, in search
Process(self.result, self.rule_object).process()
File "/home/farmsec/gsil/gsil/process.py", line 36, in process
ret_mail = self._send_mail(maybe_mistake)
File "/home/farmsec/gsil/gsil/process.py", line 69, in _send_mail
return Notification(subject, to, cc).notification(html)
File "/home/farmsec/gsil/gsil/notification.py", line 70, in notification
s.login(mail, get('1312**[email protected]', 'LIyi..00'))
File "/usr/lib/python3.6/smtplib.py", line 721, in login
initial_response_ok=initial_response_ok)
File "/usr/lib/python3.6/smtplib.py", line 638, in auth
authobject(challenge).encode('ascii'), eol='')
AttributeError: 'NoneType' object has no attribute 'encode'

然后我直接没有配置了， 保持您的默认的那样还是报错， 如下：
Traceback (most recent call last):
File "/home/farmsec/gsil/gsil/notification.py", line 70, in notification
s.login(mail, get('mail', 'password'))
File "/usr/lib/python3.6/smtplib.py", line 730, in login
raise last_exception
File "/usr/lib/python3.6/smtplib.py", line 721, in login
initial_response_ok=initial_response_ok)
File "/usr/lib/python3.6/smtplib.py", line 642, in auth
raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (535, b'Error: \xc7\xeb\xca\xb9\xd3\xc3\xca\xda\xc8\xa8\xc2\xeb\xb5\xc7\xc2\xbc\xa1\xa3\xcf\xea\xc7\xe9\xc7\xeb\xbf\xb4: http://service.mail.qq.com/cgi-bin/help?subtype=1&&id=28&&no=1001256')
请问下是什么原因呢？ 谢谢哦 如果方便的话能留下您的QQ吗？

请教运行出现的问题

反馈一个问题

Traceback (most recent call last):
File "gsil.py", line 28, in
sys.exit(gsil())
File "/Users/xxxxxx/GithubSIL/gsil/init.py", line 100, in gsil
start(sys.argv[1])
File "/Users/alchu4n/GithubSIL/gsil/init.py", line 65, in start
rules = get_rules(rule_types)
File "/Users/xxxxxx/GithubSIL/gsil/config.py", line 160, in get_rules
for types, rule_list in rules_dict.items():
NameError: name 'rules_dict' is not defined

找不到“github”库

gsil]# python3 gsil.py test
Traceback (most recent call last):
File "gsil.py", line 20, in
from gsil import gsil
File "/root/gsil/gsil/init.py", line 19, in
from .engine import Engine
File "/root/gsil/gsil/engine.py", line 18, in
from github import Github, GithubException
ImportError: No module named 'github'
[ gsil]# python3 -m pip install github
Looking in indexes: http://mirrors.aliyun.com/pypi/simple/
Collecting github
Could not find a version that satisfies the requirement github (from versions: )
No matching distribution found for github

readme中github写成了guthub

扫描一次后，泄露信息未删除，在扫无法收到之前未删除的泄露内容

[GSIL] [WARNING] Get Content Exception: 404 {'message': 'Not Found', 'documentation_url': 'https://developer.github.com/v3/repos/contents/#get-contents'} retrying...

[GSIL] [WARNING] Get Content Exception: The read operation timed out retrying...

[GSIL] [WARNING] Get Content Exception: [Errno -2] Name or service not known retrying...

[GSIL] [WARNING] Get Content Exception: 404 {'message': 'No commit found for the ref xxxxxxx', 'documentation_url': 'https://developer.github.com/v3/repos/contents/'} retrying...

使用过程中，一些相关报错 -。-

NameError: name 'rules_dict' is not defined麻烦能给看看这是怎么回事吗？

/usr/work/tool/app/GSIL/gsil.py zrb Traceback (most recent call last): File "/usr/work/tool/app/GSIL/gsil.py", line 28, in sys.exit(gsil()) File "/home/work/tool/app/GSIL/gsil/init.py", line 105, in gsil start(sys.argv[1]) File "/home/work/tool/app/GSIL/gsil/init.py", line 65, in start rules = get_rules(rule_types) File "/home/work/tool/app/GSIL/gsil/config.py", line 166, in get_rules for types, rule_list in rules_dict.items(): NameError: name 'rules_dict' is not defined

出现这个错误不知如何解决

Traceback (most recent call last):
File "/usr/gsil/gsil/init.py", line 37, in search
return Engine(token=token).search(rule)
File "/usr/gsil/gsil/engine.py", line 217, in search
Process(self.result, self.rule_object).process()
File "/usr/gsil/gsil/process.py", line 36, in process
ret_mail = self._send_mail(maybe_mistake)
File "/usr/gsil/gsil/process.py", line 69, in _send_mail
return Notification(subject, to, cc).notification(html)
File "/usr/gsil/gsil/notification.py", line 66, in notification
s = smtplib.SMTP(host, port)
File "/usr/local/lib/python3.6/smtplib.py", line 251, in init
(code, msg) = self.connect(host, port)
File "/usr/local/lib/python3.6/smtplib.py", line 335, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/local/lib/python3.6/smtplib.py", line 306, in _get_socket
self.source_address)
File "/usr/local/lib/python3.6/socket.py", line 704, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/local/lib/python3.6/socket.py", line 743, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

跑着跑着出现了错误,另外请问一个rules的配置问题

错误代码如下:`

  File "/root/GSIL/gsil/gsil/__init__.py", line 37, in search
    return Engine(token=token).search(rule)
  File "/root/GSIL/gsil/gsil/engine.py", line 220, in search
    Process(self.result, self.rule_object).process()
  File "/root/GSIL/gsil/gsil/process.py", line 36, in process
    ret_mail = self._send_mail(maybe_mistake)
  File "/root/GSIL/gsil/gsil/process.py", line 67, in _send_mail
    self._save_file(v['hash'], v['code'])
  File "/root/GSIL/gsil/gsil/process.py", line 79, in _save_file
    with open(os.path.join(Config().data_path, sha), 'w+', encoding='utf-8') as f:
TypeError: 'encoding' is an invalid keyword argument for this function```

想问一下多扫描的时候,配置文件是应该这样设置吗:

```{
    "csdn": {
        "csdn.com": {
            "\"password\"": {
                "mode": "normal-match",
                "ext": "php,java,python,go,js,properties"
            }
        }
    }
},
{
    "test": {
        "baidu.com": {
            "\"password\"": {
                "mode": "normal-match",
                "ext": "php,java,python,go,js,properties"
            }
        }
    }
}```

扫描到过的内容存在./gsil目录下的哪里

您好,请问扫描到过且报警出来后的内容记录在哪里?有什么方式关闭这个选项吗?

连不上github了，报ssl链接超时，但是浏览器可以直接打开github

Traceback (most recent call last):
File "D:\space\GSIL\gsil_init_.py", line 37, in search
return Engine(token=token).search(rule)
File "D:\space\GSIL\gsil\engine.py", line 172, in search
rate_limiting = self.g.rate_limiting
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\site-packages\github\MainClass.py", line 123, in rate_limiting
self.get_rate_limit()
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\site-packages\github\MainClass.py", line 145, in get_rate_limit
'/rate_limit'
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\site-packages\github\Requester.py", line 172, in requestJsonAndCheck
return self.__check(*self.requestJson(verb, url, parameters, headers, input, cnx))
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\site-packages\github\Requester.py", line 213, in requestJson
return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\site-packages\github\Requester.py", line 252, in __requestEncode
status, responseHeaders, output = self.__requestRaw(cnx, verb, url, requestHeaders, encoded_input)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\site-packages\github\Requester.py", line 277, in __requestRaw
requestHeaders
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\http\client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\http\client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\http\client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\http\client.py", line 1026, in _send_output
self.send(msg)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\http\client.py", line 964, in send
self.connect()
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\http\client.py", line 1400, in connect
server_hostname=server_hostname)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\ssl.py", line 817, in init
self.do_handshake()
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "C:\Users\lenovo\AppData\Local\conda\conda\envs\python1v3\lib\ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
socket.timeout: _ssl.c:835: The handshake operation timed out

rule.gsil

{
"test": {
"mogujie.com": {
""mogujie.org"": {
"ext": "js"
}
}
}
}

rule问题

你好，我编写了一个规则文件，例如按照你写的rule.gsil.example模板。我举个例子，（我想搜索关键字username password）如下,不知道我写的是否正确？
{
"csdn": {
"csdn.com": {
""username password"": {
"mode": "normal-match",
"ext": "php,java,python,go,js,properties,c++,c"
}
}
}
}

出现一个错误

Traceback (most recent call last):
File "/root/GSIL/gsil/init.py", line 37, in search
return Engine(token=token).search(rule)
File "/root/GSIL/gsil/engine.py", line 206, in search
if not self.process_pages(pages_content, page, total):
File "/root/GSIL/gsil/engine.py", line 101, in process_pages
if self._exclude_repository():
File "/root/GSIL/gsil/engine.py", line 345, in _exclude_repository
full_path = '{repository}/{path}'.format(repository=self.full_name.lower(), path=self.path.lower())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 2-3: ordinal not in range(128)

执行的时候发现的。。貌似编码的问题。。。

More complex search examples

Would you mind providing some more complex search examples?
For example
Inside.Example.com NOT ProjectX

没有规则，大家都散了吧

Error: No section: 'github'

github tokens
Traceback (most recent call last):
File "/root/gsil/gsil/config.py", line 42, in get
value = config.get(level1, level2)
File "/usr/local/sbin/python-3.7.0a1/lib/python3.7/configparser.py", line 780, in get
d = self._unify_values(section, vars)
File "/usr/local/sbin/python-3.7.0a1/lib/python3.7/configparser.py", line 1146, in _unify_values
raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'github'
GSIL/config.gsil file configure failed.
Error: No section: 'github'
2018-11-02 08:07:09,837 [GSIL] [CRITICAL] github -> tokens sections error Traceback (most recent call last):
File "/root/gsil/gsil/config.py", line 53, in
if ',' in tokens:
TypeError: argument of type 'NoneType' is not iterable

直接用的rules.gsil.example，不能解析。

(p3.6) ➜ gsil git:(master) ✗ python gsil.py test
github tokens
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/configparser.py", line 1138, in _unify_values
sectiondict = self._sections[section]
KeyError: 'github'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Volumes/mac/Develop/gsil/gsil/config.py", line 42, in get
value = config.get(level1, level2)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/configparser.py", line 781, in get
d = self._unify_values(section, vars)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/configparser.py", line 1141, in _unify_values
raise NoSectionError(section)
configparser.NoSectionError: No section: 'github'
GSIL/config.gsil file configure failed.
Error: No section: 'github'

(p3.6) ➜ gsil git:(master) ✗ cat gsil/rules.gsil
{
"test": {
"mogujie.com": {
""mogujie.org"": {}
}
}
}

(p3.6) ➜ gsil git:(master) ✗ cat gsil/config.gsil
[mail]
host : smtp.163.com
port : 465
mails : [email protected]
from : GSIL
password : xxx
to : [email protected]

[github]
clone: false
tokens : xxxx1253

关于配置问题

你好，我想请问下我配置好了config.gsil.example文件，配置信息如下：
[mail]
host : smtp.exmail.qq.com
port : 25
mails : 10***[email protected]
from : GSIL
password : 上述qq邮箱的密码
to : [email protected]
cc : [email protected]

[github]
clone: false
tokens : 我自己账号github上token

还是出现了如下错误：
E:\python\python.exe E:/git/Git/gsil/gsil.py
github tokens
GSIL/config.gsil file configure failed.
Error: No section: 'github'
Traceback (most recent call last):
File "E:\git\Git\gsil\gsil\config.py", line 42, in get
value = config.get(level1, level2)
File "E:\python\lib\configparser.py", line 780, in get
d = self._unify_values(section, vars)
File "E:\python\lib\configparser.py", line 1146, in _unify_values
raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'github'
2018-12-24 16:21:00,855 [GSIL] [CRITICAL] github -> tokens sections error Traceback (most recent call last):
File "E:\git\Git\gsil\gsil\config.py", line 53, in
if ',' in tokens:
TypeError: argument of type 'NoneType' is not iterable
我想请问下是哪里出了问题，谢谢了。

限流功能

执行程序出现了rate limit异常，
文章中有提到：虽然GitHub API也有频率限制，但他的文档中明确了频率限制的策略，我们可以通过做好频率控制避免触发GitHub API限制规则。
请问程序是否还未实现自动限流，还需要配合人工来判断规则数量等一些配置，防止程序执行时超出限制

关于抛出日志的问题

前辈好，我对这方面感兴趣因此跑了这项工程，现在进行python gsil.py test测试时抛出了一些日志信息，用的是例子里的规则配置，token的个数为6个。但并没有一些有用的信息，我节选部分信息复制粘贴如下，还望前辈有时间解答一下。
2019-01-03 13:48:02,598 [GSIL] [INFO] ["mogujie.org"] Speed Limit Results (Remaining Times / Total Times): (5000, 5000) Speed limit reset time: 1546498082
2019-01-03 13:48:02,598 [GSIL] [INFO] ["mogujie.org"] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2019-01-03 13:48:02,629 [GSIL] [INFO] ----------------------------
2019-01-03 13:48:02,629 [GSIL] [INFO] Search keyword: copyright meili inc
2019-01-03 13:48:02,630 [GSIL] [INFO] [copyright meili inc] Speed Limit Results (Remaining Times / Total Times): (5000, 5000) Speed limit reset time: 1546498082
2019-01-03 13:48:02,630 [GSIL] [INFO] [copyright meili inc] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2019-01-03 13:48:03,303 [GSIL] [INFO] ----------------------------
2019-01-03 13:48:03,304 [GSIL] [INFO] Search keyword: mail.mogujie.com
2019-01-03 13:48:03,304 [GSIL] [INFO] [mail.mogujie.com] Speed Limit Results (Remaining Times / Total Times): (5000, 5000) Speed limit reset time: 1546498082
2019-01-03 13:48:03,304 [GSIL] [INFO] [mail.mogujie.com] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2019-01-03 13:48:05,246 [GSIL] [INFO] [mail.mogujie.com] The actual number: 11758
2019-01-03 13:48:05,345 [GSIL] [INFO] ["mogujie.org"] The actual number: 100
2019-01-03 13:48:05,438 [GSIL] [INFO] [copyright meili inc] The actual number: 1477
2019-01-03 13:48:06,849 [GSIL] [INFO] ----------------------------
2019-01-03 13:48:06,858 [GSIL] [INFO] Search keyword: yewu1.db.mogujie.host
2019-01-03 13:48:06,858 [GSIL] [INFO] [yewu1.db.mogujie.host] Speed Limit Results (Remaining Times / Total Times): (5000, 5000) Speed limit reset time: 1546498084
2019-01-03 13:48:06,859 [GSIL] [INFO] [yewu1.db.mogujie.host] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2019-01-03 13:48:07,987 [GSIL] [INFO] ["mogujie.org"] Get page 0 data for 50
2019-01-03 13:48:07,987 [GSIL] [INFO] ["mogujie.org"] [0/100] Processed, skip! (0)
2019-01-03 13:48:07,988 [GSIL] [INFO] ["mogujie.org"] [1/100] Processed, skip! (1)
2019-01-03 13:48:07,988 [GSIL] [INFO] ["mogujie.org"] [2/100] Processed, skip! (2)
2019-01-03 13:48:07,988 [GSIL] [INFO] ["mogujie.org"] [3/100] Processed, skip! (3)
2019-01-03 13:48:07,988 [GSIL] [INFO] ["mogujie.org"] [4/100] Has encountered 4 has been processed, skip the current rules!
2019-01-03 13:48:07,989 [GSIL] [INFO] ["mogujie.org"] The current rules are processed, the process of normal exit!
2019-01-03 13:48:08,049 [GSIL] [INFO] [yewu1.db.mogujie.host] The actual number: 12
2019-01-03 13:48:08,598 [GSIL] [INFO] [copyright meili inc] Get page 0 data for 50
2019-01-03 13:48:08,599 [GSIL] [INFO] [copyright meili inc] [0/1477] Processed, skip! (0)
2019-01-03 13:48:08,599 [GSIL] [INFO] [copyright meili inc] [1/1477] Excluded because of the path, skip!
2019-01-03 13:48:08,742 [GSIL] [INFO] ----------------------------
2019-01-03 13:48:08,742 [GSIL] [INFO] Search keyword: meilishuo.org
2019-01-03 13:48:08,743 [GSIL] [INFO] [meilishuo.org] Speed Limit Results (Remaining Times / Total Times): (5000, 5000) Speed limit reset time: 1546498088
2019-01-03 13:48:08,743 [GSIL] [INFO] [meilishuo.org] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2019-01-03 13:48:08,878 [GSIL] [INFO] [mail.mogujie.com] Get page 0 data for 50
2019-01-03 13:48:08,879 [GSIL] [INFO] [mail.mogujie.com] [0/11758] Excluded because of the path, skip!
2019-01-03 13:48:08,897 [GSIL] [INFO] [mail.mogujie.com] [1/11758] Excluded because of the path, skip!
2019-01-03 13:48:08,898 [GSIL] [INFO] [mail.mogujie.com] [2/11758] Excluded because of the path, skip!
2019-01-03 13:48:08,898 [GSIL] [INFO] [mail.mogujie.com] [3/11758] Excluded because of the path, skip!
2019-01-03 13:48:08,898 [GSIL] [INFO] [mail.mogujie.com] [4/11758] Excluded because of the path, skip!
2019-01-03 13:48:08,899 [GSIL] [INFO] [mail.mogujie.com] [5/11758] Excluded because of the path, skip!
2019-01-03 13:48:08,899 [GSIL] [INFO] [mail.mogujie.com] [6/11758] Excluded because of the path, skip

2019-01-03 13:48:08,924 [GSIL] [INFO] [mail.mogujie.com] [49/11758] Excluded because of the path, skip!
2019-01-03 13:48:08,924 [GSIL] [INFO] Process count: 0
2019-01-03 13:48:08,924 [GSIL] [INFO] none content for send mail
2019-01-03 13:48:09,468 [GSIL] [INFO] [yewu1.db.mogujie.host] Get page 0 data for 12
2019-01-03 13:48:09,470 [GSIL] [INFO] [yewu1.db.mogujie.host] [0/12] Excluded because of the path, skip!
2019-01-03 13:48:09,470 [GSIL] [INFO] [yewu1.db.mogujie.host] [1/12] Excluded because of the path, skip!
2019-01-03 13:48:09,470 [GSIL] [INFO] [yewu1.db.mogujie.host] [2/12] Excluded because of the path, skip!
2019-01-03 13:48:09,471 [GSIL] [INFO] [yewu1.db.mogujie.host] [3/12] Excluded because of the path, skip!
2019-01-03 13:48:09,471 [GSIL] [INFO] [yewu1.db.mogujie.host] [4/12] Excluded because of the path, skip!
2019-01-03 13:48:09,471 [GSIL] [INFO] [yewu1.db.mogujie.host] [5/12] Excluded because of the path, skip!
2019-01-03 13:48:09,473 [GSIL] [INFO] [yewu1.db.mogujie.host] [11/12] Excluded because of the path, skip!
2019-01-03 13:48:09,473 [GSIL] [INFO] Process count: 0
2019-01-03 13:48:09,473 [GSIL] [INFO] none content for send mail
2019-01-03 13:48:09,474 [GSIL] [INFO] [yewu1.db.mogujie.host] The current rules are processed, the process of normal exit!
2019-01-03 13:48:10,153 [GSIL] [INFO] ----------------------------
2019-01-03 13:48:10,153 [GSIL] [INFO] Search keyword: meilishuo.io
2019-01-03 13:48:10,153 [GSIL] [INFO] [meilishuo.io] Speed Limit Results (Remaining Times / Total Times): (4999, 5000) Speed limit reset time: 1546498088
2019-01-03 13:48:10,154 [GSIL] [INFO] [meilishuo.io] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2019-01-03 13:48:10,730 [GSIL] [INFO] [meilishuo.org] The actual number: 13007
2019-01-03 13:48:11,413 [GSIL] [INFO] [copyright meili inc] [2/1477] Processing is complete, the next one!
2019-01-03 13:48:12,111 [GSIL] [INFO] [mail.mogujie.com] Get page 1 data for 50
2019-01-03 13:48:12,112 [GSIL] [INFO] [mail.mogujie.com] [50/11758] Excluded because of the path, skip!
2019-01-03 13:48:12,112 [GSIL] [INFO] [mail.mogujie.com] [51/11758] Excluded because of the path, skip!
2019-01-03 13:48:12,112 [GSIL] [INFO] [mail.mogujie.com] [52/11758] Excluded because of the path, skip!

2019-01-03 13:48:13,875 [GSIL] [INFO] [meilishuo.org] [49/13007] Excluded because of the path, skip!
2019-01-03 13:48:13,875 [GSIL] [INFO] Process count: 0
2019-01-03 13:48:13,875 [GSIL] [INFO] none content for send mail
2019-01-03 13:48:14,154 [GSIL] [INFO] [meilishuo.io] The actual number: 9627
2019-01-03 13:48:14,285 [GSIL] [INFO] [mail.mogujie.com] [80/11758] Did not match the code, skip!
2019-01-03 13:48:14,286 [GSIL] [INFO] [mail.mogujie.com] [81/11758] Excluded because of the path, skip!
2019-01-03 13:48:14,286 [GSIL] [INFO] [mail.mogujie.com] [82/11758] Excluded because of the path, skip!
2019-01-03 13:48:14,286 [GSIL] [INFO] [mail.mogujie.com] [83/11758] Excluded because of the path, skip!

2019-01-03 13:48:19,786 [GSIL] [INFO] [meilishuo.io] [79/9627] Excluded because of the path, skip!
2019-01-03 13:48:20,103 [GSIL] [INFO] [meilishuo.org] Get page 2 data for 50
2019-01-03 13:48:20,104 [GSIL] [INFO] [meilishuo.org] [100/13007] Excluded because of the path, skip!
2019-01-03 13:48:20,104 [GSIL] [INFO] [meilishuo.org] [101/13007] Excluded because of the path, skip!
2019-01-03 13:48:20,104 [GSIL] [INFO] [meilishuo.org] [102/13007] Excluded because of the path, skip!
2019-01-03 13:48:22,106 [GSIL] [INFO] Process count: 0
2019-01-03 13:48:22,106 [GSIL] [INFO] none content for send mail
2019-01-03 13:48:22,106 [GSIL] [INFO] [meilishuo.org] The current rules are processed, the process of normal exit!
2019-01-03 13:48:22,111 [GSIL] [INFO] [copyright meili inc] [20/1477] Code may be useless, do not skip, add to list to be reviewed!
2019-01-03 13:48:22,112 [GSIL] [INFO] [copyright meili inc] [20/1477] Processing is complete, the next one!
2019-01-03 13:48:22,113 [GSIL] [INFO] [copyright meili inc] [21/1477] Excluded because of the path, skip!

2019-01-03 13:48:38,008 [GSIL] [INFO] [copyright meili inc] [50/1477] Excluded because of the path, skip!
2019-01-03 13:48:40,890 [GSIL] [INFO] [copyright meili inc] [51/1477] Code may be useless, do not skip, add to list to be reviewed!
2019-01-03 13:48:40,890 [GSIL] [INFO] [copyright meili inc] [51/1477] Processing is complete, the next one!
2019-01-03 13:48:40,891 [GSIL] [INFO] [copyright meili inc] [52/1477] Processed, skip! (27)
2019-01-03 13:48:40,891 [GSIL] [INFO] [copyright meili inc] [53/1477] Processed, skip! (28)

邮箱错误

使用outlook邮件服务器会报错误，如下
2018-09-28 13:17:11,480 [GSIL] [CRITICAL] Send mail failed
Traceback (most recent call last):
File "/root/tools/GSIL/gsil/notification.py", line 71, in notification
s.sendmail(mail, self.to.split(',')+self.cc.split(','), msg.as_string())
File "/usr/local/python3/lib/python3.6/smtplib.py", line 887, in sendmail
raise SMTPDataError(code, resp)
smtplib.SMTPDataError: (550, b'5.7.1 Client does not have permissions to send as this sender')

gsli 运行问题

github的token已经使用了， 但是在使用 python gsil.py test 的时候报错：

('github', 'tokens')
Traceback (most recent call last):
File "/root/gsil/gsil/config.py", line 42, in get
value = config.get(level1, level2)
File "/usr/lib/python2.7/dist-packages/backports/configparser/init.py", line 792, in get
d = self._unify_values(section, vars)
File "/usr/lib/python2.7/dist-packages/backports/configparser/init.py", line 1162, in _unify_values
raise NoSectionError(section)
NoSectionError: No section: 'github'
GSIL/config.gsil file configure failed.
Error: No section: 'github'
2018-11-22 15:36:42,686 [GSIL] [CRITICAL] github -> tokens sections error Traceback (most recent call last):
File "/root/gsil/gsil/config.py", line 53, in
if ',' in tokens:
TypeError: argument of type 'NoneType' is not iterable

github开发模式限制

报错如下，是否本project有办法配置超时时间和间隔时间，我暂时没找到：

You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.', 'documentation_url': 'https://developer.github.com/v3/#abuse-rate-limits'

建议作者可以加上~

运行错误

root@kali:~/gsil# python3 gsil.py  test 
Traceback (most recent call last):
  File "gsil.py", line 20, in <module>
    from gsil import gsil
  File "/root/gsil/gsil/__init__.py", line 19, in <module>
    from .engine import Engine
  File "/root/gsil/gsil/engine.py", line 18, in <module>
    from github import Github, GithubException
ImportError: No module named 'github'

另外github的token生成了之后在脚本的哪里写入....小白表示很懵逼

关于__init__.py的疑问

您好前辈，在阅读这个程序源码的时候，不清楚/gsil/init.py中 line105
if __name__ == '__main__': gsil()
的作用，能否解释一下？

怎么去保证搜索了github里的所有页面呢？

前辈好，这个项目好像没有页面展示，我们怎么保证它已经搜索了github里的所有页面了呢？

自动剔除掉无效Token

能不能配置弄个选项，API使用频率超出限制

规则如下示例：
{
"AABBCCDDEEFF": {
"dev":{
"A1":{},
"A2":{},
"A3":{},
"A4":{},
"A5":{},
"A6":{}
},
"test2222.com": {
""A7"": {},
"A8": {},
"A9": {}
}
}
}

API返回超出频率限制，被挡：
2018-02-01 09:00:31,788 [GSIL] [INFO] ----------------------------
2018-02-01 09:00:31,788 [GSIL] [INFO] Search keyword: XXXXXXXXXXXXXX
2018-02-01 09:00:31,789 [GSIL] [INFO] [XXXXXXXXXXXXXX ] Speed Limit Results (Remaining Times / Total Times): (5000, 5000) Speed limit reset time: 1517450440
2018-02-01 09:00:31,789 [GSIL] [INFO] [XXXXXXXXXXXXXX ] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2018-02-01 09:00:32,711 [GSIL] [CRITICAL] GitHub [search_code] exception(code: 403 msg: {'message': 'You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.', 'documentation_url': 'https://developer.github.com/v3/#abuse-rate-limits'} ddf8734XXXXXXXXXX

问：配置文件能不能加选项，调整频率？

get rules failed, rule types not found!

大家好，我运行了案例中的rule
{ "test": { "mogujie.com": { "\"mogujie.org\"": { "mode": "normal-match", "ext": "php,java,python,go,js,properties" } } } }

报了如下错误，
[GSIL] [CRITICAL] get rules failed, rule types not found!

我看到有人和我一样的错误，我想问下这个要怎么解决呢？

提供云服务

[0/4] Excluded because of the path, skip! 求如何解决

2019-06-13 15:10:01,478 [GSIL] [INFO] start monitor github information leakage: zrb
2019-06-13 15:10:01,478 [GSIL] [INFO] rules length: 1
2019-06-13 15:10:01,491 [GSIL] [INFO] >>>>>>>>>>>>> zrb > "zrb" >>>>>>
2019-06-13 15:10:02,497 [GSIL] [INFO] ----------------------------
2019-06-13 15:10:02,498 [GSIL] [INFO] Search keyword: "zrb" extension:php extension:java extension:python extension:node.js extension:properties
2019-06-13 15:10:02,498 [GSIL] [INFO] ["zrb"] Speed Limit Results (Remaining Times / Total Times): (5000, 5000) Speed limit reset time: 1560413402
2019-06-13 15:10:02,499 [GSIL] [INFO] ["zrb"] The expected number of acquisitions: 4(Pages) * 50(Per Page) = 200(Total)
2019-06-13 15:10:03,917 [GSIL] [INFO] ["zrb"] The actual number: 4
2019-06-13 15:10:05,400 [GSIL] [INFO] ["zrb"] Get page 0 data for 4
2019-06-13 15:10:05,401 [GSIL] [INFO] ["zrb"] [0/4] Excluded because of the path, skip!
2019-06-13 15:10:05,401 [GSIL] [INFO] ["zrb"] [1/4] Processed, skip! (0)
2019-06-13 15:10:05,402 [GSIL] [INFO] ["zrb"] [2/4] Processed, skip! (1)
2019-06-13 15:10:05,402 [GSIL] [INFO] ["zrb"] [3/4] Processed, skip! (2)
2019-06-13 15:10:05,402 [GSIL] [INFO] Process count: 0
2019-06-13 15:10:05,402 [GSIL] [INFO] none content for send mail
2019-06-13 15:10:05,402 [GSIL] [INFO] ["zrb"] The current rules are processed, the process of normal exit!

关于匹配规则代码的一些疑惑

                        # next lines
                        for i in range(1, 4):
                            i_idx = idx + i
                            if i_idx in idxs:
                                continue
                            if i_idx > codes_len:  # 此处是否应该>=，毕竟codes是从0开始索引？
                                continue
                            if i_idx not in codes: # i_idx是个索引整数，codes是一个代码列表，此处是不是写错了？
                                continue
                            if codes[i_idx].strip() == '':
                                continue

最近在学习作者代码，这是engine.py codes()部分代码，注释部分表现了疑惑，望作者指教，谢谢。另外匹配前20行的逻辑，不管它是否存在keyword吧？

有没有一个可用的sample

发现完全扫不到东西

扫描成功 不能发邮件...

2018-01-17 16:33:07,611 [GSIL] [INFO] Process count: 3
Traceback (most recent call last):
File "/root/gsil/gsil/init.py", line 37, in search
return Engine(token=token).search(rule)
File "/root/gsil/gsil/engine.py", line 210, in search
Process(self.result, self.rule_object).process()
File "/root/gsil/gsil/process.py", line 36, in process
ret_mail = self._send_mail(maybe_mistake)
File "/root/gsil/gsil/process.py", line 68, in _send_mail
return Notification(subject, to).notification(html)
File "/root/gsil/gsil/notification.py", line 59, in notification
s.starttls()
File "/usr/local/lib/python3.6/smtplib.py", line 771, in starttls
server_hostname=self._host)
File "/usr/local/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/usr/local/lib/python3.6/ssl.py", line 814, in init
self.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:777)

请问如何解决，环境是centos7 python3.6

python2.7遇到编码错误

File "/Users/xxx/code/Python/gsil/gsil/gsil/engine.py", line 254, in codes
logger.debug('C:{x}/{l}: {c}'.format(x=idx, l=codes_len, c=codes[idx]))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-7: ordinal not in range(128)

print sys.getdefaultencoding()
ascii

改为utf-8

No module named 'github'

python3.5.3已执行pip3 install -r requirements.txt，设置完成后执行 python gsil.py test 报No module named 'github'，如何解决？

不能发送邮件到邮箱怎么解决

2019-07-02 17:11:18,486 [GSIL] [CRITICAL] Send mail failed
Traceback (most recent call last):
File "/root/桌面/GSIL/gsil/notification.py", line 71, in notification
s.login(mail, get('mail', 'password'))
File "/usr/lib/python3.7/smtplib.py", line 730, in login
raise last_exception
File "/usr/lib/python3.7/smtplib.py", line 721, in login
initial_response_ok=initial_response_ok)
File "/usr/lib/python3.7/smtplib.py", line 642, in auth
raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (535, b'Error: \xc7\xeb\xca\xb9\xd3\xc3\xca\xda\xc8\xa8\xc2\xeb\xb5\xc7\xc2\xbc\xa1\xa3\xcf\xea\xc7\xe9\xc7\xeb\xbf\xb4: http://service.mail.qq.com/cgi-bin/help?subtype=1&&id=28&&no=1001256')

运行报错

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)

[Errno 111] Connection refused

python3 gsil.py test
2018-11-18 22:21:34,056 [GSIL] [INFO] start monitor github information leakage: test
2018-11-18 22:21:34,056 [GSIL] [INFO] rules length: 1
2018-11-18 22:21:34,069 [GSIL] [INFO] >>>>>>>>>>>>> mogujie.com > "mogujie.org" >>>>>>
Traceback (most recent call last):
File "/root/gsil/gsil/init.py", line 37, in search
return Engine(token=token).search(rule)
File "/root/gsil/gsil/engine.py", line 161, in search
rate_limiting = self.g.rate_limiting
File "/usr/local/lib/python3.7/site-packages/github/MainClass.py", line 123, in rate_limiting
self.get_rate_limit()
File "/usr/local/lib/python3.7/site-packages/github/MainClass.py", line 145, in get_rate_limit
'/rate_limit'
File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 172, in requestJsonAndCheck
return self.__check(*self.requestJson(verb, url, parameters, headers, input, cnx))
File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 213, in requestJson
return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 252, in __requestEncode
status, responseHeaders, output = self.__requestRaw(cnx, verb, url, requestHeaders, encoded_input)
File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 277, in __requestRaw
requestHeaders
File "/usr/local/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/usr/local/lib/python3.7/http/client.py", line 1384, in connect
super().connect()
File "/usr/local/lib/python3.7/http/client.py", line 928, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection
raise err
File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

rules.gsil: 扫描规则能否说明具体一点

能否说明一下ext和mode的使用方法，是放在keyword的下一级么？

如何搜索中文关键词

如何搜索中文关键词，示例？

监控的范围有哪些; 这个项目

主要是搜索域名相关的关键字查看呈现内容，再进行匹配告警返回么。我看你的代码主要结果就是从这个地方获取的是吧。

Code may be useless, do not skip, add to list to be reviewed!怎么不发邮件

请问关键字搜索的结果提示这个， 但不发邮件
是否发邮件的规则是什么， 在那控制呢？ 麻烦帮解答下，多谢了！