microsoft / presidio Goto Github PK
View Code? Open in Web Editor NEWContext aware, pluggable and customizable data protection and de-identification SDK for text and images
Home Page: https://microsoft.github.io/presidio
License: MIT License
Context aware, pluggable and customizable data protection and de-identification SDK for text and images
Home Page: https://microsoft.github.io/presidio
License: MIT License
When a new pod is started, it could get a grpc request even though it hadn't finished loading.
Requests sent to the pod during this time would fail.
Implement readiness into pods in K8S
Describe the bug
Currently a call to the fieldTypes API returns the list of fields defined in the proto file. Since Presidio allows to extend this list by adding custom recognizers and field types, this API should return all field types (entity types) supported by a Presidio deployment
To Reproduce
Steps to reproduce the behavior:
Expected behavior
New types supported by custom recognizers should be added to the list
Describe the bug
from
https://github.com/Microsoft/presidio/blob/master/docs/tutorial_service.md
Sample 4: Custom anonymization
*** Currently the result return is
{
"field": {
"name": "US_DRIVER_LICENSE"
},
"score": 0.65,
"location": {
"start": 176,
"end": 184,
"length": 8
}
}
*** it would be good if you can return so that it is easier to debug
"field": {
"name": "US_DRIVER_LICENSE"
},
"score": 0.65,
"location": {
"start": 176,
"end": 184,
"length": 8
}
"match_text": "AC333991"
}
**** it will be good if you can also return the original "text"
{
.
.
.
"text":"John Smith lives in New York. We met yesterday morning in Seattle. I called him before on (212) 555-1234 to verify the appointment. He also told me that his drivers license is AC333991"
}
To Reproduce
$ echo -n '{"text":"John Smith lives in New York. We met yesterday morning in Seattle. I called him before on (212) 555-1234 to verify the appointment. He also told me that his drivers license is AC333991", "analyzeTemplate":{"allFields":true} }' | http -F --verify=no https://192.168.1.44/api/v1/projects/1/analyze
Expected behavior
N/A
Screenshots
N/A
Additional context
it is very common that API also return original text plus matching string.
Add the new SECURITY.MD file for repos under the Microsoft organization.
In order to match re2 supported syntax, we need to modify the regex's to the following values :
CREDIT_CARD: \b((4\d{3})|(5[0-5]\d{2})|(6\d{3})|(1\d{3})|(3\d{3}))[- ]?(\d{3,4})[- ]?(\d{3,4})[- ]?(\d{3,5})\b
(Also remove the diners since this covers it.
DOMAIN: \b(((([a-zA-Z0-9])|([a-zA-Z0-9][a-zA-Z0-9\-]{0,86}[a-zA-Z0-9]))\.(([a-zA-Z0-9])|([a-zA-Z0-9][a-zA-Z0-9\-]{0,73}[a-zA-Z0-9]))\.(([a-zA-Z0-9]{2,12}\.[a-zA-Z0-9]{2,12})|([a-zA-Z0-9]{2,25})))|((([a-zA-Z0-9])|([a-zA-Z0-9][a-zA-Z0-9\-]{0,162}[a-zA-Z0-9]))\.(([a-zA-Z0-9]{2,12}\.[a-zA-Z0-9]{2,12})|([a-zA-Z0-9]{2,25}))))\b
CRYPTO: \b[13][a-km-zA-HJ-NP-Z0-9]{26,33}\b
US_DRIVERS_LICENSE (WA): unknown at the moment
Run presidio under persistent load for a long time.
notice memory in analyser increases over time, eventually the pod will restart.
ruled out recognizers, context and other factors by marking out all code in analyzer's main methods (Apply and analyze) except for nlp_engine.process_text() and memory still increases.
this is aligned with the following github issues on spacy memory increase over time, due to internal cache usage:
explosion/spaCy#3618
explosion/spaCy#3013
explosion/spaCy#285
from the demo website:
IBAN: AT611904300234573201 ===> : AT611904300234573201
Set template:
echo -n '{"fields":[{"name": "PERSON"}]}' | http http://localhost:8080/api/v1/templates/123/analyze/my-template
Call analyzer:
echo -n '{"text":"John Smith lives in New York. We met yesterday morning in Seattle. I called him before on (212) 555-1234 to verify the appointment. He also told me that his drivers license is AC111921", "analyzeTemplateId":"my-template", "analyzeTemplate":{"fields": [{ "name": "PHONE" } ]} }' | http localhost:8080/api/v1/projects/123/analyze
response is:
{
"field": {
"name": "PERSON"
},
"location": {
"end": 10,
"length": 10
},
"score": 0.85
}
]
should output an error message or be documented
Describe the bug
Only cities and countries are recognized as locations when using spacy NER
To Reproduce
Steps to reproduce the behavior:
Using the analyzer as a Python package I encountered a few errors, here I will describe how to reproduce them and how to (temporarily) fix them in order to get the next errors:
I first succesfully followed the instructions on the tutorial on how to install the presidio-analyzer as a Python package by creating a wheel file, after running into some of the issues I simply kept on manipulating the scripts (as I will describe in full detail) and running the test script (step 5 of the installation https://github.com/microsoft/presidio/blob/master/docs/install.md) in the same directory as the analyzer folder (for the script to recognize it as a module).
Is it possible to ask for a kind of quick start documentation to help the use of presidio from a User perspective?
What I mean is that I cannot find any API specification as well as an example on how to input a database source data to presidio analyzer/anonymizer.
May be it is a limit of me, but I believe it could help many people.
Thank you in advance
When calling the engine analyze API like
response = engine.analyze(correlation_id=0,
text=text_to_analyze,
language='en',
entities=[],
all_fields=True,
score_threshold=0.5)
and the value of 'text_to_analyze' is
"/boardingPass/v1/devices/34e7b5e1a0aa1d6f3d862b52a289cdb7/registrations/pass.apoc.wallet/"
The exception below is thrown
File "/home/folder_name/presidio_testing/my_venv/lib/python3.6/site-packages/analyzer/analyzer_engine.py", line 204, in analyze current_results = recognizer.analyze(text, entities, nlp_artifacts) File "/home/folder_name/presidio_testing/my_venv/lib/python3.6/site-packages/analyzer/pattern_recognizer.py", line 61, in analyze pattern_result = self.__analyze_patterns(text) File "/home/folder_name/presidio_testing/my_venv/lib/python3.6/site-packages/analyzer/pattern_recognizer.py", line 144, in __analyze_patterns validation_result = self.validate_result(current_match) File "/home/folder_name/presidio_testing/my_venv/lib/python3.6/site-packages/analyzer/predefined_recognizers/crypto_recognizer.py", line 23, in validate_result bcbytes = CryptoRecognizer.__decode_base58(pattern_text, 25) File "/home/folder_name/presidio_testing/my_venv/lib/python3.6/site-packages/analyzer/predefined_recognizers/crypto_recognizer.py", line 33, in __decode_base58 n = n * 58 + digits58.index(char)
ValueError: substring not found
Hi! I'm using and editing the Presidio analyzer software as a python module to fit my requirements. One of the things I'd like to do is to add a parameter to the process, for example, passing the country to the AnalyzerEngine().analyze method (in addition to the existing parameters like text, language, entities, etc).
I managed to change the code for this purpose, adding a new parameter to the analyze method of entity_recognizer, local_recognizer, pattern_recognizer and a few other scripts, I did this very carefully but it doesn't work because some of the _pb2_grpc automatically generated scripts (using Google's protocol buffer) control the supported variables.
I was wondering if there's a way of regenerating this scripts using the existing (edited) code to add supported parameters.
Thank you so much for the attention!
Describe the bug
The link to the demo website in the readme.md returns a 404. The link's URL is https://presidio-demo.azurewebsites.net/
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A HTTP 200
Describe the bug
Following the instructions to build docker images. My build process is stuck at pipenv sync
and eventually crashes
To Reproduce
Steps to reproduce the behavior:
make DOCKER_REGISTRY=${DOCKER_REGISTRY} PRESIDIO_LABEL=${PRESIDIO_LABEL} docker-build-deps
Fails while pipenv sync. Below is the log:
Step 10/12 : RUN pipenv sync
---> Running in a0dbab69a607
Creating a virtualenv for this projectโฆ
Pipfile: /usr/bin/presidio-analyzer/Pipfile
Using /usr/local/bin/python (3.7.1) to create virtualenvโฆ
โ ผ Creating virtual environment...Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/presidio-analyzer-p_bm8iFt/bin/python
Installing setuptools, pip, wheel...
done.
โ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/presidio-analyzer-p_bm8iFt
Installing dependencies from Pipfile.lock (6725d2)โฆ
The command '/bin/sh -c pipenv sync' returned a non-zero code: 137
make: *** [docker-build-deps] Error 137
This is after about 2 hours.
Expected behavior
Build successfully completes.
Additional context
I have a pretty fast internet connection, and I know pipenv can be slow. But I didn't expect it to fail.
Looking at: durations
I see the builds are actually very slow ~80 mins
What can I do to fasten this or even the build to succeed.
Describe the bug
How to create a project in presidio? Trying hard to follow the tutorial
https://github.com/Microsoft/presidio/blob/master/docs/tutorial_service.md
To Reproduce
Steps to reproduce the behavior:
$ curl -X POST localhost:3000/api/v1/projects/1
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
$ curl -X POST localhost:3000/api/v1/projects/my-projects
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
Expected behavior
Before I can analyze a template.. I need to create a default project?
$ echo -n '{"text":"John Smith lives in New York. We met yesterday morning in Seattle. I called him before on (212) 555-1234 to verify the appointment. He also told me that his drivers license is AC333991", "analyzeTemplate":{"fields":[]} }' | http /api/v1/projects//analyze
Screenshots
N/A
Additional context
Just try to run the example from https://github.com/Microsoft/presidio/blob/master/docs/tutorial_service.md
This module can scan various RDBMS databases such as MSSQL, MySQL and PostgreSQL
Features:
Create a Kubernetes Job definition. Optional Cron Job
Scan database records specific tables.
Analyze each cell content.
Optional - Anonymize text.
Save results in a DB or DW for further analysis and reports.
The module needs to support:
Create an in-memory template caching mechanism to improve performance and reduce I/O to disk and k8s secrets.
Describe the bug
NER (spacy) ability to increase score based on context is limited
To Reproduce
Expected behavior
Should identify as person
This bug talks about an issue where a string is considered by two different recognizers to be a PII entity.
Its not bad as compleetly missing it however there are scenario (like the one described below) that the wrong entity type 'wins' due to incorrect scoring.
consider the following request to analyze:
{
"text": "my social number is 078051120 and my name is Jon doe",
"analyzeTemplate": {
"fields": [
{
"name": "PHONE_NUMBER"
},
{
"name": "US_SSN"
},
{
"name": "CREDIT_CARD"
},
{
"name": "DATE_TIME"
}
]
}
}
The result we get for 078051120 is DATE_TIME with score 0.85
if we remove DATE_TIME from the template we get the correct result of US_SSN
Basically its ok because it will be matched anyway, altough the anonymize methods might be different.
in this specific case this ssn is given a WEAK score of 0.3 and due to the context it is improved to 0.6, which is still less than 0.85.
if the SSN would have been in the correct SSN format XXX-XX-XXXX it would have got a better score with context of 0.85 which is the same as date time and might still be the one removed.
This bug should be discussed.
a) do we want to give regex results priority over spacy?
b) maybe we want to give context more affect on the score
c) maybe we want to allow the user to configure the right behaviour.
Hello.
For some reason custom recognizers don't work on my current presidio instance.
I'm following the documentation:
I post into api/v1/analyzer/recognizers/spaceship
{ "value": { "entity": "SPACESHIP", "language": "en", "patterns": [ { "name": "spaceship regex", "regex": "\\W*(spaceship)\\W*", "score": 0.9 } ] } }
And then I try to run the analyzer posting into /api/v1/projects/testing/analyze
in two ways:
{ "text":"They went inside the spaceship", "analyzeTemplate": { "allFields": true } }
Response:
"No results"
{ "text":"They went inside the spaceship", "analyzeTemplate": { "fields": [ { "name": "SPACESHIP" } ] } }
Response
"rpc error: code = Unknown desc = Exception calling application: No matching recognizers were found to serve the request."
Any thoughts ?
Regards.
Analyzer detected part of this guid as phone number:
110bcd25-a55d-453a-8046-1297901ea002
{
"text": "046-1297901",
"field": {
"name": "PHONE_NUMBER"
},
"probability": 0.5,
"location": {
"start": 52,
"end": 63,
"length": 11
}
},
Json file attached
test.txt
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Describe the bug
https://microsoft.github.io/presidio/tutorial_framework.html
Try Method 2
Use the analyzer Python code by importing matcher.py from presidio-analyzer/analyzer
match = matcher.Matcher()
results = self.match.analyze_text(text, fields)
To Reproduce
chenglim@chenglim-GL503VM:/amaris/code/presidio/presidio-analyzer$ python3 a.py
unable to cache TLDs in file /usr/local/lib/python3.6/dist-packages/tldextract/.tld_set: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/tldextract/.tld_set'
<class 'analyzer.matcher.Matcher'>
[text: "(212) 555-1234"
field {
name: "PHONE_NUMBER"
}
score: 1.0
location {
start: 91
end: 105
length: 14
}
, text: "New York"
field {
name: "LOCATION"
}
score: 0.8500000238418579
location {
start: 21
end: 29
length: 8
}
, text: "Seattle"
field {
name: "LOCATION"
}
score: 0.8500000238418579
location {
start: 59
end: 66
length: 7
}
, text: "yesterday"
field {
name: "DATE_TIME"
}
score: 0.8500000238418579
location {
start: 38
end: 47
length: 9
}
, text: "morning"
field {
name: "DATE_TIME"
}
score: 0.8500000238418579
location {
start: 48
end: 55
length: 7
}
]
corrupted size vs. prev_size
Aborted (core dumped)
chenglim@chenglim-GL503VM:/amaris/code/presidio/presidio-analyzer$ cat a.py
from analyzer import matcher
class DictAttr(dict):
def __getattr__(self, key):
if key not in self:
raise AttributeError(key)
return self[key]
def __setattr__(self, key, value):
self[key] = value
def __delattr__(self, key):
del self[key]
text = """
John Smith lives in New York. We met yesterday morning in Seattle.
I called him before on (212) 555-1234 to verify the appointment.
He also told me that his drivers license is AC111921
"""
fields = []
field = ["PHONE_NUMBER", "LOCATION", "DATE_TIME"]
for f in field:
a = DictAttr()
a.name = f
fields.append(a)
match = matcher.Matcher()
print(type(match))
results = match.analyze_text(text, fields)
print(results)
Expected behavior
Screenshots
N/A
Additional context
Suspect this has to do with the tldextract..
Describe the bug
If the analyze result is empty, a 400 http status code is returned. should be 200
To Reproduce
Steps to reproduce the behavior:
Expected behavior
status code should be 200
This is not a bug but a question. The solution looks promising. But wondering how is this different from an NER system (or LUIS entity recognition) ? I can see this spacy NER being used. Is it an extension to NER system, exposed as a service or there is some other intelligence added to the system.
Our current implementation uses Python regex library and spacy word similarity.
We need to check performance and integration with re2 and possibly remove the word similarity functionality.
A good response time for our demo text file will be 200 ms per analyze request with all the field types enabled.
Hi,
Have you guys tried to run the analyzer as a spark job ? how would suggest handle loading the spacy model for each worker and also how to handle serialization?
would appreciate some suggestions around this, is there any plan to support this use case ?
Thanks
Describe the bug
Is the Sink Service and Scanner Service available. If so, can you create documentation to start? Thanks.
To Reproduce
N/A
Expected behavior
N/A
Screenshots
N/A
Additional context
N/A
Describe the bug
Multiple false positives for credit card number. We should consider dismiss other results if checksum passes.
To Reproduce
analyze "My credit card number is 6011577631711174"
analyze "My credit card number is 6011-5776-3171-1174"
Expected behavior
credit card with score 1.0
Describe the bug
The demo currently starts when all detected fields are not anonymized. The user doesn't see any of presidio's functionality unless he or she changes the action from None to something else.
To Reproduce
Start the demo
Expected behavior
Processed text should have detected fields anonymized instead of ignored.
Hi,
This is a question. As per installation instructions it can be installed with either docker or with helm (reddis etc).
Can't we simply install presido on windows or unix box like pip or git clone?
Presidio API should support Swagger and OpenAPI.
We should replace gin with go-restful http server.
I have a lot of customed PII that I want to add.
I am running into a lot of issue where numbers result in overlapping classification.
For example,
1234567890 9844412312312323 at https://presidio-demo.azurewebsites.net results in <DATE_TIME>UMBER>.
I am very curious in learning about how this scenario occurs to best resolve the issue.
Describe the bug
https://github.com/microsoft/presidio/blob/master/docs/tutorial_analyzer.md
Tested on the analyzer python code below, but the program crashed and came out with a few errors.
These are the errors.
Currently the analyzer is not trained to analyze json files.
Therefor when analyzing these files the fields are detected wrongly:
Examples:
{
"text": "uuid:26c446d2-65a9-4eaa-9a70-53da2aded653\"}},{\"url\":\"http://synthetichealth.github.io",
"field": {
"name": "PERSON"
},
"probability": 0.85,
"location": {
"start": 1920,
"end": 2005,
"length": 85
}
},
{
"text": "oid:2.16.840.1.113883.4.3.25","value":"S99910353"},{"type":{"coding":[{"system":"http://hl7.org",
"field": {
"name": "NRP"
},
"probability": 0.85,
"location": {
"start": 2898,
"end": 2993,
"length": 95
}
}
Json files attached
Set template:
echo -n '{"fields":[{"name": "PERSON", "languageCode":"HEB" }]}' | http http://localhost:8080/api/v1/templates/123/analyze/my-template
Call analyzer:
echo -n '{"text":"John Smith lives in New York. We met yesterday morning in Seattle. I called him before on (212) 555-1234 to verify the appointment. He also told me that his drivers license is AC111921", "analyzeTemplateId":"my-template" }' | http http://localhost:8080/api/v1/projects/123/analyze
result is:
[
{
"field": {
"name": "PERSON"
},
"location": {
"end": 10,
"length": 10
},
"score": 0.85
}
]
should be empty.
Same applies for empty language code.
During the installation with Docker, it is giving make error
make DOCKER_REGISTRY=${DOCKER_REGISTRY} PRESIDIO_LABEL=${PRESIDIO_LABEL} docker-build
make: *** [Makefile:181: go-test-style] Error 1
The command '/bin/sh -c make go-test' returned a non-zero code: 2
make: *** [docker-build-base] Error 2
It would be very helpful, if someone canhelp with the solution
Thank You
$ make DOCKER_REGISTRY=${DOCKER_REGISTRY} PRESIDIO_LABEL=${PRESIDIO_LABEL} docker-build
docker build --build-arg REGISTRY=presidio --build-arg PRESIDIO_DEPS_LABEL='latest' -t presidio/presidio-golang-base -f Dockerfile.golang.base .
Sending build context to Docker daemon 1.261MB
Step 1/7 : ARG REGISTRY=presidio.azurecr.io
Step 2/7 : ARG PRESIDIO_DEPS_LABEL=latest
Step 3/7 : FROM ${REGISTRY}/presidio-golang-deps:${PRESIDIO_DEPS_LABEL}
---> b2929920eb09
Step 4/7 : WORKDIR $GOPATH/src/github.com/Microsoft/presidio
---> Using cache
---> 04324948e601
Step 5/7 : ADD . $GOPATH/src/github.com/Microsoft/presidio
---> Using cache
---> ecf194963a5e
Step 6/7 : RUN dep ensure
---> Using cache
---> 4d1681ebd2c6
Step 7/7 : RUN make go-test
---> Running in aa4cea2b1b88
gometalinter --config ./gometalinter.json ./...
pkg/cache/cache.go:1::warning: file is not gofmted with -s (gofmt)
pkg/cache/mock/cache.go:1::warning: file is not gofmted with -s (gofmt)
pkg/cache/redis/redis.go:1::warning: file is not gofmted with -s (gofmt)
pkg/logger/logger.go:1::warning: file is not gofmted with -s (gofmt)
pkg/logger/logger_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/client.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/cronjob.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/cronjob_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/job.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/job_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/kube.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/secret.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/kube/secret_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/local/job.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/local/local.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/local/secret.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/local/secret_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/platform/platform.go:1::warning: file is not gofmted with -s (gofmt)
pkg/presidio/presidio.go:1::warning: file is not gofmted with -s (gofmt)
pkg/presidio/services/services.go:1::warning: file is not gofmted with -s (gofmt)
pkg/presidio/templates/templates.go:1::warning: file is not gofmted with -s (gofmt)
pkg/presidio/templates/templates_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/queue/queue.go:1::warning: file is not gofmted with -s (gofmt)
pkg/queue/rabbitmq/rabbitmq.go:1::warning: file is not gofmted with -s (gofmt)
pkg/queue/rabbitmq/rabbitmq_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/rpc/client.go:1::warning: file is not gofmted with -s (gofmt)
pkg/rpc/server.go:1::warning: file is not gofmted with -s (gofmt)
pkg/server/server.go:1::warning: file is not gofmted with -s (gofmt)
pkg/storage/storage.go:1::warning: file is not gofmted with -s (gofmt)
pkg/stream/eventhubs/eventhubs.go:1::warning: file is not gofmted with -s (gofmt)
pkg/stream/kafka/kafka.go:1::warning: file is not gofmted with -s (gofmt)
pkg/stream/mock/mock.go:1::warning: file is not gofmted with -s (gofmt)
pkg/stream/mock/mock_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/stream/stream.go:1::warning: file is not gofmted with -s (gofmt)
pkg/version/version.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer-image/cmd/presidio-anonymizer-image/anonymizer/anonymizer.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer-image/cmd/presidio-anonymizer-image/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/anonymizer.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/anonymizer_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/common.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/fpe_config.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/fpe_config_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/hash_config.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/hash_config_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/mask_config.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/mask_config_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/redact_config.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/replace_config.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/replace_config_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-anonymizer/cmd/presidio-anonymizer/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/analyze/analyze.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/analyze/analyze_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/anonymize-image/anonymize-image.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/anonymize-image/anonymize-image_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/anonymize/anonymize.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/anonymize/anonymize_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/api.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/mocks/mocks.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/recognizers/recognizers.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/recognizers/recognizers_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/scanner-cron-job/scanner-cron-job.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/stream-job/stream-job.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/api/templates/templates.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-api/cmd/presidio-api/methods.go:1::warning: file is not gofmted with -s (gofmt)
presidio-collector/cmd/presidio-collector/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-collector/cmd/presidio-collector/processor/processor.go:1::warning: file is not gofmted with -s (gofmt)
presidio-collector/cmd/presidio-collector/scanner/factory.go:1::warning: file is not gofmted with -s (gofmt)
presidio-collector/cmd/presidio-collector/scanner/scanner.go:1::warning: file is not gofmted with -s (gofmt)
presidio-collector/cmd/presidio-collector/scanner/storage-item.go:1::warning: file is not gofmted with -s (gofmt)
presidio-collector/cmd/presidio-collector/scanner/storage-scanner.go:1::warning: file is not gofmted with -s (gofmt)
presidio-collector/cmd/presidio-collector/streams/streams.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/cloudstorage/cloudstorage.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/cloudstorage/cloudstorage_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/database/database.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/database/database_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/datasink-factory.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/datasink/datasink.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/main_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/stream/stream.go:1::warning: file is not gofmted with -s (gofmt)
presidio-datasink/cmd/presidio-datasink/stream/stream_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-ocr/cmd/presidio-ocr/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-ocr/cmd/presidio-ocr/ocr/ocr.go:1::warning: file is not gofmted with -s (gofmt)
presidio-recognizers-store/cmd/presidio-recognizers-store/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-recognizers-store/cmd/presidio-recognizers-store/recognizers_store_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-scheduler/cmd/presidio-scheduler/main.go:1::warning: file is not gofmted with -s (gofmt)
presidio-scheduler/cmd/presidio-scheduler/scheduler_test.go:1::warning: file is not gofmted with -s (gofmt)
presidio-tester/cmd/presidio-tester/tester.go:1::warning: file is not gofmted with -s (gofmt)
tests/common/common.go:1::warning: file is not gofmted with -s (gofmt)
tests/functional_http_test.go:1::warning: file is not gofmted with -s (gofmt)
tests/functional_recognizers_test.go:1::warning: file is not gofmted with -s (gofmt)
tests/functional_storage_scanner_test.go:1::warning: file is not gofmted with -s (gofmt)
tests/integration_anonymize_image_test.go:1::warning: file is not gofmted with -s (gofmt)
tests/integration_eventhub_test.go:1::warning: file is not gofmted with -s (gofmt)
tests/integration_kafka_test.go:1::warning: file is not gofmted with -s (gofmt)
tests/integration_redis_test.go:1::warning: file is not gofmted with -s (gofmt)
tests/integration_tesseract_test.go:1::warning: file is not gofmted with -s (gofmt)
pkg/cache/cache.go:1::warning: file is not goimported (goimports)
pkg/cache/mock/cache.go:1::warning: file is not goimported (goimports)
pkg/cache/redis/redis.go:1::warning: file is not goimported (goimports)
pkg/logger/logger.go:1::warning: file is not goimported (goimports)
pkg/logger/logger_test.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/client.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/cronjob.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/cronjob_test.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/job.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/job_test.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/kube.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/secret.go:1::warning: file is not goimported (goimports)
pkg/platform/kube/secret_test.go:1::warning: file is not goimported (goimports)
pkg/platform/local/job.go:1::warning: file is not goimported (goimports)
pkg/platform/local/local.go:1::warning: file is not goimported (goimports)
pkg/platform/local/secret.go:1::warning: file is not goimported (goimports)
pkg/platform/local/secret_test.go:1::warning: file is not goimported (goimports)
pkg/platform/platform.go:1::warning: file is not goimported (goimports)
pkg/presidio/presidio.go:1::warning: file is not goimported (goimports)
pkg/presidio/services/services.go:1::warning: file is not goimported (goimports)
pkg/presidio/templates/templates.go:1::warning: file is not goimported (goimports)
pkg/presidio/templates/templates_test.go:1::warning: file is not goimported (goimports)
pkg/queue/queue.go:1::warning: file is not goimported (goimports)
pkg/queue/rabbitmq/rabbitmq.go:1::warning: file is not goimported (goimports)
pkg/queue/rabbitmq/rabbitmq_test.go:1::warning: file is not goimported (goimports)
pkg/rpc/client.go:1::warning: file is not goimported (goimports)
pkg/rpc/server.go:1::warning: file is not goimported (goimports)
pkg/server/server.go:1::warning: file is not goimported (goimports)
pkg/storage/storage.go:1::warning: file is not goimported (goimports)
pkg/stream/eventhubs/eventhubs.go:1::warning: file is not goimported (goimports)
pkg/stream/kafka/kafka.go:1::warning: file is not goimported (goimports)
pkg/stream/mock/mock.go:1::warning: file is not goimported (goimports)
pkg/stream/mock/mock_test.go:1::warning: file is not goimported (goimports)
pkg/stream/stream.go:1::warning: file is not goimported (goimports)
pkg/version/version.go:1::warning: file is not goimported (goimports)
presidio-anonymizer-image/cmd/presidio-anonymizer-image/anonymizer/anonymizer.go:1::warning: file is not goimported (goimports)
presidio-anonymizer-image/cmd/presidio-anonymizer-image/main.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/anonymizer.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/anonymizer_test.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/common.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/fpe_config.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/fpe_config_test.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/hash_config.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/hash_config_test.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/mask_config.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/mask_config_test.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/redact_config.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/replace_config.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/anonymizer/transformations/replace_config_test.go:1::warning: file is not goimported (goimports)
presidio-anonymizer/cmd/presidio-anonymizer/main.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/analyze/analyze.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/analyze/analyze_test.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/anonymize-image/anonymize-image.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/anonymize-image/anonymize-image_test.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/anonymize/anonymize.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/anonymize/anonymize_test.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/api.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/mocks/mocks.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/recognizers/recognizers.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/recognizers/recognizers_test.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/scanner-cron-job/scanner-cron-job.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/stream-job/stream-job.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/api/templates/templates.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/main.go:1::warning: file is not goimported (goimports)
presidio-api/cmd/presidio-api/methods.go:1::warning: file is not goimported (goimports)
presidio-collector/cmd/presidio-collector/main.go:1::warning: file is not goimported (goimports)
presidio-collector/cmd/presidio-collector/processor/processor.go:1::warning: file is not goimported (goimports)
presidio-collector/cmd/presidio-collector/scanner/factory.go:1::warning: file is not goimported (goimports)
presidio-collector/cmd/presidio-collector/scanner/scanner.go:1::warning: file is not goimported (goimports)
presidio-collector/cmd/presidio-collector/scanner/storage-item.go:1::warning: file is not goimported (goimports)
presidio-collector/cmd/presidio-collector/scanner/storage-scanner.go:1::warning: file is not goimported (goimports)
presidio-collector/cmd/presidio-collector/streams/streams.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/cloudstorage/cloudstorage.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/cloudstorage/cloudstorage_test.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/database/database.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/database/database_test.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/datasink-factory.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/datasink/datasink.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/main.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/main_test.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/stream/stream.go:1::warning: file is not goimported (goimports)
presidio-datasink/cmd/presidio-datasink/stream/stream_test.go:1::warning: file is not goimported (goimports)
presidio-ocr/cmd/presidio-ocr/main.go:1::warning: file is not goimported (goimports)
presidio-ocr/cmd/presidio-ocr/ocr/ocr.go:1::warning: file is not goimported (goimports)
presidio-recognizers-store/cmd/presidio-recognizers-store/main.go:1::warning: file is not goimported (goimports)
presidio-recognizers-store/cmd/presidio-recognizers-store/recognizers_store_test.go:1::warning: file is not goimported (goimports)
presidio-scheduler/cmd/presidio-scheduler/main.go:1::warning: file is not goimported (goimports)
presidio-scheduler/cmd/presidio-scheduler/scheduler_test.go:1::warning: file is not goimported (goimports)
presidio-tester/cmd/presidio-tester/tester.go:1::warning: file is not goimported (goimports)
tests/common/common.go:1::warning: file is not goimported (goimports)
tests/functional_http_test.go:1::warning: file is not goimported (goimports)
tests/functional_recognizers_test.go:1::warning: file is not goimported (goimports)
tests/functional_storage_scanner_test.go:1::warning: file is not goimported (goimports)
tests/integration_anonymize_image_test.go:1::warning: file is not goimported (goimports)
tests/integration_eventhub_test.go:1::warning: file is not goimported (goimports)
tests/integration_kafka_test.go:1::warning: file is not goimported (goimports)
tests/integration_redis_test.go:1::warning: file is not goimported (goimports)
tests/integration_tesseract_test.go:1::warning: file is not goimported (goimports)
make: *** [Makefile:181: go-test-style] Error 1
The command '/bin/sh -c make go-test' returned a non-zero code: 2
make: *** [docker-build-base] Error 2
Describe the bug
Please create an Tutorial on how to custom a new field_type
To Reproduce
N/A
Expected behavior
A good framework shall allow public to contribute
Screenshots
N/A
Additional context
Give me a head start, I may able to contribute.
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Send a request with an entity on as the first token. Response object doesn't include a start value
For example:
{"text":"Microsoft is the company name", "analyzeTemplate":{"allFields":true} }
Response:
[
{
"field": {
"name": "COMPANY_NAME"
},
"score": 1,
"location": {
"end": 9,
"length": 9
}
}
]
Describe the bug
Building binaries (make build) fails if tesseract libs are not available.
To Reproduce
Steps to reproduce the behavior:
$ make build
go build -ldflags ' -X github.com/Microsoft/presidio/pkg/version.Version=v0.3-21-g78be69b' -o bin/presidio-anonymizer ./presidio-anonymizer/cmd/presidio-anonymizer
go build -ldflags ' -X github.com/Microsoft/presidio/pkg/version.Version=v0.3-21-g78be69b' -o bin/presidio-ocr ./presidio-ocr/cmd/presidio-ocr
# github.com/Microsoft/presidio/vendor/github.com/otiai10/gosseract
tessbridge.cpp:5:10: fatal error: tesseract/baseapi.h: No such file or directory
5 | #include <tesseract/baseapi.h>
| ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:26: presidio-ocr] Error 2
Expected behavior
Not sure if (2) is doable, though. For Fedora 30 I fixed this by installing the tesseract-devel
package. For Ubuntu the libtesseract-dev
package can be used. (Source: otiai10/gosseract#132 (comment)).
Hi,
We perform ETL operations to move data from blob storage into ADLS datalake and SQL Servers.
Can this software be used as a potential transformation tool in the ETL load process to identify and hash the PII data?
I followed the analyzer document and it gave some good results on when passed a text file in a standalone python executor. Do you have any documentation on how to start working with anonymizer and scanning pictures?
Thank You
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
https://presidio-demo.westeurope.cloudapp.azure.com/ not working on demo text (tested on Chrome, Safari, Firefox)
Expected behavior
Should replace example PII
An Open API definition for the REST API would be useful to developers.
Spacy NER and lemmatizer supports multiple languages.
Allow A user to install Presidio with additional Spacy models.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.