The project allows to check a file for a profanity by a dictionary stored in a local database. There is Aho-Corasick algorithm for searching patterns in a text
- Search is case insensitive
- Punctuation and whitespaces are part of the text as well
- Searching pattern can be a part of a found phrase
Project is used for a testing purpose only
- docker-compose up -d
- navigate to http://localhost:9001/swagger/index.html
- Add words or phrases to the dictionary
- one by one
- provide a file with line-separated phrases
- Send a file to the profanity service (up to 30MB)
{
"hasProfanity": true,
"profanityItems": [
{
"data": "a b",
"indexes": [
10323,
28380,
48512,
2463296,
2463343
],
"fullBounds": [
"a book",
"Albania began",
"a brown",
"BADIA Batalla",
"BPA Building,"
]
},
{
"data": "ccc",
"indexes": [
38142,
98556,
2263557,
2297830,
2326507
],
"fullBounds": [
"CCC,",
"CCC",
"(CCC)"
]
},
{
"data": "hello",
"indexes": [
1806014,
1806096
],
"fullBounds": [
"Seychellois"
]
}
]
}
Parameter | Description |
---|---|
hasProfanity | Flag that indicates if any profanity found |
data | Pattern in the dictionary which was found |
indexes | All indexes where pattern was found in a text |
fullBounds | The found word (or item) that contains the searching pattern |
indexes and fullBounds parameters can contain the different number of elements (because a HashSet under the hood)
- Optimize searching algorithm
- How many languages will you support?
- How big dictionary will you have?
- What size of a text or file will be?
- Split text into chunks
- Use caching for requests and a dictionary
- Message bus for a scaling support
- API versioning
- Provide an additional incoming adapter with gRPC streaming
- Implement healthchecks mechanism
- Add app metrics (e.g. Prometheus + Grafana)
- Check performance by profilers (e.g. dotTrace)
- Add validators for REST endpoints (e.g. FluentValidation)
- Provide Authorization mechanism for the API
- Use TLS encryption
- Implement rate limiting mechanism
E.g.: Rabin–Karp algorithm uses hashing, which reduces the speed on average, especially with the big dictionary, because collision artifacts.
- .NET 5 + SQLite
- NUnit + Moq + FluentAssertions
- EntityFramework Core
- Swagger
- Serilog