numbworks / nw.ngramtextclassification

NW.NGramTextClassification is a library to perform text classification tasks on the text snippets you provide. Text Classification is a machine learning technique that calculates the similarity between the string of text you need to categorize and a collection of already categorized strings you provide to the library.

About

I'm a Technical Product Manager with strong software development roots (C# and Python) and a passionate love affair with data pipelines, which I do nurture every day.

In my daily job I do manage five teams, lead the product strategy of the company, enforce Agile and other best practices (clean architecture, unit test coverage > 70%, updated documentation as part of the definition-of-done, code readibility as a priority, ...), ensure quarterly deliveries despite the limited development capacity. In addition, I do back-end development work when required. Few of the many technologies I do work with every week: C#, Python, Jupyter Notebook, Pandas, PowerBi, ETL, databases, REST APIs, AWS.

In my off-work time I do actively develop and maintain some open-source software packages (C# and Python) under my numbworks brand. Last but not least, to improve my craft, I studied 276 technical books in the past nine years.

A software developer profile is defined by the software he develops and by the continuous learning activities he performs. This is my portfolio and you'll find information about both aspects of my off-work journey as a back-end developer.

Contact: Email | Twitter

Development

Libraries (C#)

Intended for other developers who want to integrate my software in their own software.

Repository	Effort	Quick Links	License	Tests	NuGet	Last Update
NW.UnivariateForecasting	208 h	README CHANGELOG Documentation Issues	MIT		4.2.0	2024-02-15
NW.NGramTextClassification	207 h	README CHANGELOG Documentation Issues	MIT		4.2.0	2024-02-14
NW.MarkdownTables	21 h	README CHANGELOG Documentation Issues	MIT		3.0.0	2024-01-21

CLI Applications (C#)

Intended for data analysts who want to use my libraries thru a command-line interface.

Repository	Quick Links	License	Tests	Binaries
NW.UnivariateForecastingClient	Documentation	MIT		4.2.0
NW.NGramTextClassificationClient	Documentation	MIT		4.2.0

Shared Libraries (C#)

Pieces of logic shared among my libraries. These might not be feature-rich enough to be useful for the general audience.

Repository	Effort	Quick Links	License	Tests	NuGet	Last Update
NW.Shared.Files	5 h	README CHANGELOG Documentation Issues	MIT		1.0.0	2024-02-11
NW.Shared.Serialization	4 h	README CHANGELOG Documentation Issues	MIT		1.0.0	2024-02-13
NW.Shared.Validation	3 h	README CHANGELOG Documentation Issues	MIT		1.0.0	2024-02-10

Jupyter Notebooks (Python)

Intended to showcase my approach to solve specific data analysis problems:

Repository	Effort	Quick Links	License	Tests	Version	Last Update
nwreadinglist	95 h	README CHANGELOG Documentation Issues	MIT		3.3.0	2024-05-21
nwtimetracking	73 h	README CHANGELOG Documentation Issues	MIT		3.3.0	2024-05-21

Shared Packages (Python)

Pieces of logic shared among my libraries. These might not be feature-rich enough to be useful for the general audience.

Repository	Effort	Quick Links	License	Tests	Version	Last Update
nwshared	9 h	README CHANGELOG Documentation Issues	MIT		1.1.0	2024-05-20

Other Projects

Repository	Type	Quick Links	License	Last Update
i3_eink_config	Configuration File	README	MIT	2020-12-22

Related Pages

• CONTRIBUTIONGUIDEFORCSHARP.md
• FRAMEWORKFREEZE.md

Continuous Learning

The following table summarizes how many technical books I studied since I started a continuous learning path (2016):

Years	Books	Pages	TotalSpend	LastUpdate
9	279	75882	$7747.15	2024-06-25

Related Pages

• READINGLISTBYMONTH.md
• READINGLISTBYPUBLISHER.md
• READINGLISTBYRATING.md
• READINGLISTBYTOPIC.md
• READINGLISTTOPICTREND.md
• READINGLIST.md

Self-Improvement Status

Areas of Expertise

Data, Databases, Development, Software Usability, Clean Software Architecture, Clean Code, Design Patterns, (Parametric) Unit Testing, OOP, Dependency Injection, Single Responsability Principle, Console Applications, Services, CLIs,CI/CD, (Web) Scraping, REST APIs, Business Analysis, ETL Process, Star Schema, Data Warehouses, Prompt Engineering.

Tech Stack

C#, NET Core, NET Standard, NUNit,SQL Server, PowerBi, TeamCity, Azure DevOps, Excel, VBA, Powershell, Ubuntu Server, Docker, MariaDB, NuGet Packages, SQLite, Proxmox VE, Jupyter Notebooks, Python 3.x, Pandas, ollama.

Incoming

Python Packages, Data Science, Machine Learning, Postgresql, Time-Series Analysis, Sensor Data, AWS Architecture.

nw.ngramtextclassification's Issues

String index bug

When I invoke the .PredictLabel method with some inputs, a call to Substring() throws an unhandled exception up the stack.

at System.String.Substring(Int32 startIndex, Int32 length)
   at NW.NGramTextClassification.TextClassifierComponents.<>c.<.cctor>b__31_1(String text, UInt32 length)
   at NW.NGramTextClassification.TextClassifier.PredictLabel(String text, ITokenizationStrategy strategy, INGramTokenizerRuleSet ruleSet, List1 labeledExamples)    at NW.NGramTextClassification.TextClassifier.PredictLabel(String text, List1 labeledExamples)
   at ****.InferReporterType(String input) in ****:line 100

[v4.0.0] Framework Freeze. Migrate to .NET 8.0, NUnit 4.x, update packages accordingly, implement new assertion model

[v3.6.0] Add "--cleanlabeledexamples" option to the "classify" sub-command

As a developer, I want to add the"--cleanlabeledexamples" option to the "classify" sub-command, so that the client can clean the provided labeled examples before using them for the classification process.

[v4.2.0] Move Serializations namespace to NW.Shared.Serialization

[v3.6.0] Performance improvement: the tokenization process for the provided labeled examples should happen only once

As a developer, I want to move the tokenization process of the provided labeled examples, so that it happens only once despite the number of text snippets and performances drastically improve.

Before:

...
[2022-11-03 21:40:42:714] The provided LabeledExample objects have been thru the tokenization process.
[2022-11-03 21:40:42:714] At least one LabeledExample object failed to be tokenized.
...
[2022-11-03 21:40:53:839] The provided LabeledExample objects have been thru the tokenization process.
[2022-11-03 21:40:53:839] At least one LabeledExample object failed to be tokenized.
...
[2022-11-03 21:41:04:812] The provided LabeledExample objects have been thru the tokenization process.
[2022-11-03 21:41:04:812] At least one LabeledExample object failed to be tokenized.
...
[2022-11-03 21:41:15:656] The provided LabeledExample objects have been thru the tokenization process.
[2022-11-03 21:41:15:656] At least one LabeledExample object failed to be tokenized.
...

After:

...
[2022-11-03 21:40:42:714] The provided LabeledExample objects have been thru the tokenization process.
[2022-11-03 21:40:53:839] At least one LabeledExample object failed to be tokenized.
[2022-11-03 21:41:04:812] Attempting to save the provided 'TextClassifierSession' object as: C:\ngramtc_session_20221104121705802.json.
[2022-11-03 21:41:15:656] The provided 'TextClassifierSession' object has been successfully saved.
...

[v4.1.0] Move Files, Validation to NW.Shared.*

[v3.0.0] Add a preliminary structure for the client

As a developer, I want to add support for a preliminary structure for the client, so that I can prepare the ground for the introduction of a CLI application as client.

[v3.5.0] Add support for Filenames namespace

As a developer, I want to add support for the "Filenames" namespace, so that I can prepare the ground for the introduction of the save functionalities.

[v3.0.0] Split monolithic ObjectMother class to namespace-based ObjectMother classes

As a developer, I want to split the currently monolithic ObjectMother class to smaller namespace-based ObjectMother classes, so that the library is easier to mantain and expand.

[v3.5.0] Add "LoadTokenizerRuleset" functionalities

As a developer, I want to add the possibility to load a tokenizer ruleset from a JSON file, so that the library can classify using user-provided rules taken from an external source.

[v3.5.0] Add MinimumAccuracy settings

As a developer, I want to add MinimumAccuracy settings, so that I can scrap the label classification when the SimilarityIndexAverage is too low.

[v4.0.0] Update dotnet.yml to .NET 8.0

[v4.1.0] Deprecated Task

[v3.5.0] Add "SaveSession", "SaveLabeledExamples" and "SaveTextSnippets" functionalities

As a developer, I want to add the possibility to save the main library's DTOs to a JSON file, so that the library can persist relevant information to disk.

[v3.5.0] Add support for the classification of a collection of texts

As a developer, I want to add support for the classification of a collection of snippets of text, so that it's easier to use the library for batch classification tasks.

[v3.6.0] Add TextSnippet property to TextClassifierResult

As a developer, I want to add the TextSnippet property to TextClassifierResult, so that the the Session Json file looks clearer when multiple Text Snippets are provided to the classifier.

[v3.5.0] Add "LoadLabeledExamples" functionality

As a developer, I want to add the possibility to load a collection of labeled examples from a JSON file, so that the library can classify snippets of texts using user-provided examples.

[v3.5.0] Add "LoadTextSnippets" functionality

As a developer, I want to add the possibility to load a collection of text snippets from a JSON file, so that the library can classify snippets of texts taken from an external source.

[v3.7.0] [Performance] ClassifyMany, replace the for cycle with Parallel.For

As a developer, I want to replace the for cycle with Parallel.For in the ClassifyMany method, so that the classification process of many text snippets can become way faster.

[v3.5.0] Add "classify" sub-command and its options to the client

As a developer, I want to add the "classify" command to the CLI, so that the client can perform a classification task using user-provided data in JSON.

[v3.6.0] Add "CleanLabeledExamples" functionality

As a developer, I want to add support for the "CleanLabeledExamples" functionality to LabeledExampleManager and TextClassifier, so that I can prepare the ground for the introduction of this option in the client.

[v4.0.0] Adopt the "Bag Pattern" into the codebase

[v3.7.0] Develop a ngramtchelper script to manage file format conversions

As a developer, I want to develop a script and add it to the application repository, so that the user can perform basic file format conversions such as "CSV to Labeled Examples JSON" or "CSV to Text Snippets JSON".

[v3.5.0] Add support for Files namespace

As a developer, I want to add support for the "Files" namespace, so that I can prepare the ground for the introduction of the save/load functionalities.

Add Load CSV Method to load labeld Examples from csv file

Please add the ReadCSV method with CSV Helper library

https://joshclose.github.io/CsvHelper/

[v3.0.0] Split monolithic MessageCollection class to namespace-based MessageCollection classes

As a developer, I want to split the currently monolithic MessageCollection class to smaller namespace-based MessageCollection classes, so that the library is easier to mantain and expand.

add data to model when app is running and use

I am trying to make a talking bot using this library

I admit that the library is incredibly beautiful and the work is set up

But I wanted to know how I can add data to the model when app is running using this library and save it at runtime.

[v3.6.0] Add the "--disableindexserialization" option to the "classify" sub-command

As a developer, I want to add the"--disableindexserialization" option to the "classify" sub-command, so that the client can create way more compact Session Jsons.

[v3.0.0] Add support for ASCII banner

As a developer, I want to add support for the ASCII banner, so that I can prepare the ground for the introduction of a CLI application as client.