Comments (7)
Yeah I also had in mind to do that also for non-acknowledged scans ( may happen in a period when the scanners are redeployed but the api is up and running. )
Both are postgres module internal changes so they will not affect the worker functions.
from tls-observatory.
I'm tracking down a bug where all scans are acked, but very few complete:
observatory=> select count(*) from scans where ack='false';
count
-------
0
(1 row)
completion_perc | count
-----------------+--------
0 | 194690
20 | 443
40 | 2
100 | 18520
The scanners are receiving notifications, but scan don't happen. The current load average on the scanners is close to 0%.
from tls-observatory.
Nevermind that previous comment: I had an issue in my script and was calling the scan API with an empty target. Since the validatedomain function doesn't yet verify the target, the scanner were trying to scan empty targets and failing in an unexpected way. I'm preparing a patch for validate domain now.
This issue remains.
from tls-observatory.
Now that it's been running for a while, here are some real world stats:
observatory=> select ack, count(*) from scans group by ack;
ack | count
-----+---------
f | 281
t | 4189855
(2 rows)
observatory=> select completion_perc, count(*) from scans group by completion_perc;
completion_perc | count
-----------------+---------
0 | 3278345
20 | 9169
40 | 1428
100 | 900753
So it seems like scans gets acknowledge, picked up by a scanner goroutine, but never complete. Do you think limiting the number of scanner in a sync group would help?
from tls-observatory.
It depends on what the problem, preventing the scan from completing, is.
If it involves concurrent database connections the sync group would help.
Do we have the syslog files of the running containers to check if any errors have been logged?
Regardless of that I am preparing a patch which will catch up on both unacknowledged and half-complete scans and re-queue them.
from tls-observatory.
check out 0c68439 .
What do you think we should do with the half-complete scans ( 0<completion_perc<100 )?
If we decide to re-queue them after a specific amount of time ( eg 5-6 mins )
we must take care ( delete or verify ) of the trusts and workers' analyses created by them.
from tls-observatory.
I'd say abandon them. If a scanner starts and does some of the work, but crashed after completion_perc>0, there must be a reason and we should track those in the logs, and/or return feedback to the caller. That's a topic for another issue.
from tls-observatory.
Related Issues (20)
- CODE_OF_CONDUCT.md file missing
- Please consider using semver compatible tags
- ssl_prefer_server_ciphers off causes TLS level Non-compliant HOT 1
- Root store results do not match current Firefox HOT 1
- Symantec Cert Waring - False Positives HOT 2
- CI failing to find MS CA cert HOT 2
- panic: runtime error: index out of range [0] with length 0
- Modify CircleCI config to avoid rate limiting.
- Scan API doesn't return results HOT 2
- Add worker health output to __heartbeat__ endpoint. HOT 1
- Wrong ECC keys length ? 512 to 521
- EV Checker: BRs no longer require OCSP URI in the AIA of intermediate certificates HOT 1
- EV Checker: resulting message confusing
- Error initializing top1m
- Please correct the IsTechnicallyConstrained logic in the certificate tool
- Build fail on power machine
- Crypto Go :we are a research group to help developers build secure applications.
- Add support for PSS algorithms in /certificates and certsplainer
- TLS Observatory does not recognize Let's Encrypt Certs HOT 11
- TLS Observatory: certsplainer.html displaying incorrect data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tls-observatory.