Infrastructure & Automation Tooling for Python Projects
Home Page: https://exasol.github.io/python-toolbox/
License: MIT License
pip install exasol-toolboxThe latest documentation can be found here
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Enable caching of python packages in GitHub actions
Caching python packages installed via poetry will save a lot of time and energy.
See documentation of setup-python action: https://github.com/actions/setup-python#caching-packages-dependencies
The content you are editing has changed. Please copy your edits and refresh the page.
See https://github.com/exasol/extension-manager/actions/runs/6844839090/job/18609203687
Run tbx security cve convert maven < input | tee cves.jsonl
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/exasol/t │
│ oolbox/tools/security.py:187 in convert │
│ │
│ 184 │ │
│ 185 │ actions = {Format.Maven: _maven} │
│ 186 │ action = actions[format] │
│ ❱ 187 │ action(input_file) │
│ 188 │
│ 189 │
│ 190 class Filter(str, Enum): │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ _maven = <function convert.<locals>._maven at 0x7fccfd90f600> │ │
│ │ action = <function convert.<locals>._maven at 0x7fccfd90f600> │ │
│ │ actions = { │ │
│ │ │ <Format.Maven: 'maven'>: <function │ │
│ │ convert.<locals>._maven at 0x7fccfd90f600> │ │
│ │ } │ │
│ │ format = <Format.Maven: 'maven'> │ │
│ │ input_file = <_io.TextIOWrapper name='<stdin>' encoding='utf-8'> │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/exasol/t │
│ oolbox/tools/security.py:181 in _maven │
│ │
│ 178 │ │
│ 179 │ def _maven(infile): │
│ 180 │ │ issues = from_maven(infile.read()) │
│ ❱ 181 │ │ for issue in _issues_as_json_str(issues): │
│ 182 │ │ │ stdout(issue) │
│ 183 │ │ raise typer.Exit(code=0) │
│ 184 │
│ │
│ ╭─────────────────────────── locals ───────────────────────────╮ │
│ │ infile = <_io.TextIOWrapper name='<stdin>' encoding='utf-8'> │ │
│ │ issues = <generator object from_maven at 0x7fccfd940040> │ │
│ │ { │ │
│ │ │ │ │ 'coordinates': │ │
│ │ 'pkg:maven/com.fasterxml.jackson.core/[email protected]… │ │
│ │ │ │ │ 'description': 'Core annotations used for value │ │
│ │ types, used by Jackson databinding package.', │ │
│ │ │ │ │ 'reference': │ │
│ │ '[https://ossindex.sonatype.org/component/pkg:maven/com.fasterx…](https://ossindex.sonatype.org/component/pkg:maven/com.fasterx%E2%80%A6) │ │
│ │ │ │ }, │ │
│ │ │ │ ... +96 │ │
│ │ │ }, │ │
│ │ │ 'excludedVulnerabilities': [ │ │
│ │ │ │ { │ │
│ │ │ │ │ 'id': 'CVE-2023-4586', │ │
│ │ │ │ │ 'displayName': 'CVE-2023-4586', │ │
│ │ │ │ │ 'title': "[CVE-2023-4586] CWE-300: Channel │ │
│ │ Accessible by Non-Endpoint ('Man-in-the-Middle'"+1, │ │
│ │ │ │ │ 'description': 'netty-handler - Improper │ │
│ │ Certificate Validation\n\nThe product does not adequately'+230, │ │
│ │ │ │ │ 'cvssScore': 6.5, │ │
│ │ │ │ │ 'cvssVector': │ │
│ │ 'CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:L/A:N', │ │
│ │ │ │ │ 'cwe': 'CWE-300', │ │
│ │ │ │ │ 'cve': 'CVE-2023-4586', │ │
│ │ │ │ │ 'reference': │ │
│ │ '[https://ossindex.sonatype.org/vulnerability/CVE-2023-4586?com…](https://ossindex.sonatype.org/vulnerability/CVE-2023-4586?com%E2%80%A6) │ │
│ │ │ │ │ 'externalReferences': [ │ │
│ │ │ │ │ │ │ │
│ │ '[https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLPa…](https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLPa%E2%80%A6) │ │
│ │ │ │ │ │ 'https://github.com/netty/netty/issues/8537', │ │
│ │ │ │ │ │ 'https://github.com/netty/netty/issues/9930', │ │
│ │ │ │ │ │ │ │
│ │ '[https://netty.io/4.1/api/io/netty/handler/ssl/SslContext.html…](https://netty.io/4.1/api/io/netty/handler/ssl/SslContext.html%E2%80%A6) │ │
│ │ │ │ │ ] │ │
│ │ │ │ }, │ │
│ │ │ │ { │ │
│ │ │ │ │ 'id': 'CVE-2020-36641', │ │
│ │ │ │ │ 'displayName': 'CVE-2020-36641', │ │
│ │ │ │ │ 'title': '[CVE-2020-36641] CWE-611: Improper │ │
│ │ Restriction of XML External Entity Reference '+7, │ │
│ │ │ │ │ 'description': 'A vulnerability classified as │ │
│ │ problematic was found in gturri aXMLRPC up to 1.12'+586, │ │
│ │ │ │ │ 'cvssScore': 9.8, │ │
│ │ │ │ │ 'cvssVector': │ │
│ │ 'CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H', │ │
│ │ │ │ │ 'cwe': 'CWE-611', │ │
│ │ │ │ │ 'cve': 'CVE-2020-36641', │ │
│ │ │ │ │ 'reference': │ │
│ │ '[https://ossindex.sonatype.org/vulnerability/CVE-2020-36641?co…](https://ossindex.sonatype.org/vulnerability/CVE-2020-36641?co%E2%80%A6) │ │
│ │ │ │ │ 'externalReferences': [ │ │
│ │ │ │ │ │ │ │
│ │ '[http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-3664…](http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-3664%E2%80%A6) │ │
│ │ │ │ │ │ 'https://www.tenable.com/cve/CVE-2020-36641' │ │
│ │ │ │ │ ] │ │
│ │ │ │ } │ │
│ │ │ ] │ │
│ │ } │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
KeyError: 'vulnerable'
Error: Process completed with exit code 1.
Currently the developer will set the version number with poetry version <version>.
Additionally there is a file version.py creating the risk for inconsistencies.
File version.py also contains a comment that the file is created automatically and should not be edited manually.
But there seems to be no automation, yet.
So the current ticket therefore requests to create and describe an automation.
Options
noxfile.pynoxconfig.pyversion.py As thresholds are tight, these linter problems cause the initial push to fail.
.github/workflows/checks.yml: missing step prio 1️⃣After step Run Tests there is another step missing:
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: .lint.txt
path: .lint.txtexasol/toolbox/.github/workflows/xyz.yml@version./.github/workflows/xyz.ymlrun: poetry run version-check exasol/toolbox/version.pyrun: poetry run version-check version.py.gitignore? prio 9️⃣.lint.txt.lint.json.coveragemetrics.json.github/workflows/checks.yml prio 9️⃣build-documentation-job: should be indented with 2 additional spacesdoc/conf.pydoc/index.rsttestartifacts.github/workflows/checks.yml, step "Copy Artifacts into Root Folder" could be simplified prio 9️⃣cp .coverage/.coverage ../.), cp ./artifacts/.coverage/.coverage ./GH Workflow Job "Generate Status Report" currently fails with
subprocess.CalledProcessError: Command '['coverage', 'json',
'--data-file=.coverage', '-o', '/tmp/tmpm01u0nf3/coverage.json']'
returned non-zero exit status 2.
Proposals
Should we mark the GH-workflow files, initially generated by PTB, as generated in file .gitattributes?
See Git docu on using this file, GH docu for marking files as generated and project-keeper containing an example for using the file.
Benefits:
I had some trouble to run integration tests in my CI build, which required using GitHub secrets for accessing an Exasol SaaS database instance, see saas-api-python.
Some of the GitHub workflows generated by the PTB use the trigger workflow_call.
In this case, secrets are not inherited by default.
Update file .github/workflows/ci.yml generated by PTB and add
ci-job:
name: Checks
uses: ./.github/workflows/checks.yml
secrets: inheritIf you have security concerns, then we could add this as a comment and add some documentation.
Implement/Add a nox target which can be used to audit the workspace in regard to vulnerabilities.
If the audit check fails, it should be indicated by a returncode != 0
nox -s auditThe content you are editing has changed. Please copy your edits and refresh the page.
Fix Linting Issues to Improve Project's Linting Score with Pylint
In our continuous effort to maintain high-quality code standards for the python-toolbox project, it has been observed that the current linting score, as assessed by pylint, could be improved. This task involves identifying and fixing linting issues to ensure our codebase adheres to the best practices and coding standards.
The motivation behind this is to enhance code readability, maintainability, and reduce potential errors by adhering to strict linting rules enforced by pylint. A high linting score is indicative of a codebase that follows Python best practices, which in turn facilitates easier collaboration and code reviews.
After running the nox -s lint command, linting issues such as unused imports, variable naming conventions, line lengths, and spacing inconsistencies are expected to be addressed. For instance, converting variable names from camelCase to snake_case to align with PEP 8 naming conventions, as recommended by pylint.
pylint documentation: https://pylint.pycqa.org/en/latest/nox -s lint to identify linting issues through pylintfactor out common code blocks
Implement/Add support for an depedencies.md file.
Add support to output dependency information like dependencies for python projects.
The content you are editing has changed. Please copy your edits and refresh the page.
This code snippets or parts of them maybe can be reused for this task.
from collections import defaultdict
from dataclasses import dataclass
from typing import (
Dict,
List,
Tuple,
)
@dataclass(frozen=True)
class Package:
name: str
license: str
version: str
def _packages(package_info):
for p in package_info:
kwargs = {key.lower(): value for key, value in p.items()}
yield Package(**kwargs)
def _normalize(license):
def is_mulit_license(l):
return ";" in l
def select_most_permissive(l):
licenses = [_normalize(l.strip()) for l in l.split(";")]
priority = defaultdict(
lambda: 9999,
{"Unlicense": 0, "BSD": 1, "MIT": 2, "MPLv2": 3, "LGPLv2": 4, "GPLv2": 5},
)
priority_to_license = defaultdict(
lambda: "Unknown", {v: k for k, v in priority.items()}
)
selected = min(*[priority[lic] for lic in licenses])
return priority_to_license[selected]
mapping = {
"BSD License": "BSD",
"MIT License": "MIT",
"The Unlicense (Unlicense)": "Unlicense",
"Mozilla Public License 2.0 (MPL 2.0)": "MPLv2",
"GNU Lesser General Public License v2 (LGPLv2)": "LGPLv2",
"GNU General Public License v2 (GPLv2)": "GPLv2",
}
if is_mulit_license(license):
return select_most_permissive(license)
if license not in mapping:
return license
return mapping[license]
def audit(
licenses: List[Dict[str, str]], acceptable: List[str], exceptions: Dict[str, str]
) -> Tuple[List[Package], List[Package]]:
"""
Audit package licenses.
Args:
licenses: a list of dictionaries containing license information for packages.
This information e.g. can be obtained by running `pip-licenses --format=json`.
example: [{"License": "BSD License", "Name": "Babel", "Version": "2.12.1"}, ...]
acceptable: A list of licenses which shall be accepted.
example: ["BSD License", "MIT License", ...]
exceptions: A dictionary containing package names and justifications for packages to ignore/skip.
example: {'packagename': 'justification why this is/can be an exception'}
Returns:
Two lists containing found violations and ignored packages.
"""
packages = list(_packages(licenses))
acceptable = [_normalize(a) for a in acceptable]
ignored = [p for p in packages if p.name in exceptions and exceptions[p.name]]
violations = [
p
for p in packages
if _normalize(p.license) not in acceptable and p not in ignored
]
return violations, ignoredimport pytest
from exasol.toolbox.license import audit
@pytest.fixture
def licenses():
yield [
{"License": "BSD License", "Name": "Babel", "Version": "2.12.1"},
{"License": "MIT License", "Name": "PyYAML", "Version": "6.0"},
{
"License": "Apache Software License",
"Name": "argcomplete",
"Version": "2.1.2",
},
{
"License": "GNU Lesser General Public License v2 (LGPLv2)",
"Name": "astroid",
"Version": "2.15.5",
},
{
"License": "Mozilla Public License 2.0 (MPL 2.0)",
"Name": "certifi",
"Version": "2023.5.7",
},
{
"License": "Python Software Foundation License",
"Name": "distlib",
"Version": "0.3.6",
},
{
"License": "BSD License; GNU General Public License (GPL); Public Domain; Python Software Foundation License",
"Name": "docutils",
"Version": "0.19",
},
{
"License": "The Unlicense (Unlicense)",
"Name": "filelock",
"Version": "3.12.2",
},
{
"License": "Mozilla Public License 2.0 (MPL 2.0)",
"Name": "pathspec",
"Version": "0.11.1",
},
{
"License": "GNU General Public License (GPL); GNU General Public License v2 or later (GPLv2+); Other/Proprietary License",
"Name": "prysk",
"Version": "0.15.1",
},
{
"License": "GNU General Public License v2 (GPLv2)",
"Name": "pylint",
"Version": "2.17.4",
},
]
def test_nothing_to_validate():
licenses = []
acceptable = []
exceptions = []
violations, exceptions = audit(licenses, acceptable, exceptions)
assert set(violations) == set()
assert set(exceptions) == set()
@pytest.mark.parametrize(
"acceptable,exceptions,expected_violations,expected_exceptions",
[
([], [], 11, 0),
(["BSD License", "MIT License"], {}, 8, 0),
(
["BSD License", "MIT License"],
{"prysk": "Prysk is only a development dependency"},
7,
1,
),
],
)
def test_audit(
licenses, acceptable, exceptions, expected_violations, expected_exceptions
):
violations, exceptions = audit(
licenses, acceptable=acceptable, exceptions=exceptions
)
assert len(violations) == expected_violations
assert len(exceptions) == expected_exceptionsWhen using the PTB, then filters defined in file noxconfig.py seem to not work as expected.
In project saas-api-python In file noxconfig.py I added filters ".nox" and "openapi" as I had subdirectories .nox and exasol/saas/client/openapi/ which I wanted to filter as an experiment.
@dataclass(frozen=True)
class Config:
root: Path = ROOT_DIR
doc: Path = ROOT_DIR / "doc"
version_file: Path = ROOT_DIR / "version.py"
path_filters: Iterable[str] = (
"dist",
".eggs",
"venv",
".nox",
"openapi",
)When running poetry run nox -s lint pylint still reported files below directory .nox.
Both files below directory .nox and files below directory exasol/saas/client/openapi/ are filtered out.
Depending on the order of the filters in path_filters either only files below only one of the directories were filtered out.
Probably the nesting of generators in exasol/toolbox/project.py produces unexpected results:
def _deny_filter(files: Iterable[Path], deny_list: Iterable[str]) -> Iterable[Path]:
for entry in deny_list:
files = filter(lambda path: entry not in path.parts, files)
return filesThe following modified code is inefficient, but works:
files = list(filter(lambda path: entry not in path.parts, files))Git Repo https://github.com/exasol/saas-api-python/
Desktop:
In #88 we added support for creating issues for vulnerabilities in Maven projects. This would be useful for Go projects, too.
We can use govulncheck:
# install
go install golang.org/x/vuln/cmd/govulncheck@latest
# Run
govulncheck -mode=source -scan=symbol -test ./...Example output:
Scanning your code and 293 packages across 44 dependent modules for known vulnerabilities...
=== Informational ===
Found 1 vulnerability in packages that you import, but there are no call
stacks leading to the use of this vulnerability. You may not need to
take any action. See https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck
for details.
Vulnerability #1: GO-2023-2163
curve KeyPairs fail to encrypt github.com/nats-io/nkeys
More info: https://pkg.go.dev/vuln/GO-2023-2163
Module: github.com/nats-io/nkeys
Found in: github.com/nats-io/[email protected]
Fixed in: github.com/nats-io/[email protected]
No vulnerabilities found.
Share feedback at https://go.dev/s/govulncheck-feedback.
govulncheck -mode=source -scan=symbol -test ./...{
"config": {
"protocol_version": "v1.0.0",
"scanner_name": "govulncheck",
"scanner_version": "v1.0.1",
"db": "https://vuln.go.dev",
"db_last_modified": "2023-11-06T21:39:09Z",
"go_version": "go1.21.3",
"scan_level": "symbol"
}
}
{
"progress": {
"message": "Scanning your code and 293 packages across 44 dependent modules for known vulnerabilities..."
}
}
{
"osv": {
"schema_version": "1.3.1",
"id": "GO-2023-2163",
"modified": "2023-11-02T21:47:24Z",
"published": "2023-11-02T21:47:24Z",
"aliases": [
"CVE-2023-46129",
"GHSA-mr45-rx8q-wcm9"
],
"summary": "curve KeyPairs fail to encrypt github.com/nats-io/nkeys",
"details": "Curve KeyPairs always use the same (all-zeros) key to encrypt data, and provide no security.",
"affected": [
{
"package": {
"name": "github.com/nats-io/nkeys",
"ecosystem": "Go"
},
"ranges": [
{
"type": "SEMVER",
"events": [
{
"introduced": "0.4.0"
},
{
"fixed": "0.4.6"
}
]
}
],
"ecosystem_specific": {
"imports": [
{
"path": "github.com/nats-io/nkeys",
"symbols": [
"ckp.Open",
"ckp.Seal",
"ckp.SealWithRand",
"decodePubCurveKey"
]
}
]
}
}
],
"references": [
{
"type": "ADVISORY",
"url": "https://github.com/nats-io/nkeys/security/advisories/GHSA-mr45-rx8q-wcm9"
},
{
"type": "FIX",
"url": "https://github.com/nats-io/nkeys/commit/58fb9d69f42ea73fffad1d14e5914dc666f3daa1"
}
],
"credits": [
{
"name": "Quentin Matillat (GitHub @tinou98)"
}
],
"database_specific": {
"url": "https://pkg.go.dev/vuln/GO-2023-2163"
}
}
}
{
"finding": {
"osv": "GO-2023-2163",
"fixed_version": "v0.4.6",
"trace": [
{
"module": "github.com/nats-io/nkeys",
"version": "v0.4.0",
"package": "github.com/nats-io/nkeys"
}
]
}
}In order to implement automatic fixing of vulnerable dependencies we need to know the ticket numbers that security.py creates.
To simplify using this information we want to add a Github action output parameter.
Make sure the prepare-release tasks also updates all workflow templates so correct toolbox version is referenced.
Make sure all relevant version references @MAJOR.MINOR.PATCH in the workflows are replaced, with the
version number to be released.
E.g.:
This currently is the last tedious and error prone task when it comes to releases and therefore also should be automated. A first trivial (regex) based solution should do.
This is specific to the python-toolbox project and likely will require adding some hooks like the once for the integration tests.
In order to simplify setting up and maintaining a project with the python-toolbox, a command for managing
the workflow provided by the toolbox would be helpful.
Having a cli tool for this task will reduce the need of copy & paste and replacing dummy values.
TL;DR:
List available workflows:
tbox workflow listInstall a specific workflow
tbox workflow install ci...
I observed a list of GitHub workflow files that PTB did create in repository python-extension-common. I am wondering whether we should mark these files as "generated" in a file .gitattributes with potential content
.github/workflows/build-and-publish.yml linguist-generated=true
.github/workflows/check-release-tag.yml linguist-generated=true
.github/workflows/checks.yml linguist-generated=true
.github/workflows/ci-cd.yml linguist-generated=true
.github/workflows/ci.yml linguist-generated=true
.github/workflows/gh-pages.yml linguist-generated=true
.github/workflows/pr-merge.yml linguist-generated=true
.github/workflows/report.yml linguist-generated=true
The effect of marking files as "generated" is that during code review, GitHub will initially hide the files with a comment "This file is generated". So human reviewers can skip this file during review as it is generated anyway and hence does not need to be reviewed. By that human reviewers can save time and focus on manual changes the should of course be subject to review.
See PK PR #562 for example:
I created this ticket based on observations in repository python-extension-common recently adopting the PTB.
Please also see the project-keeper for Java implementing all the tasks named below.
.gitattributes.Make documentation more accessible for developers which are not in depth familiar with the language and tooling.
pyproject.toml)install poetry in documentationnoxfile.py and its purpose earlier in documentation (optional)Add a security scanner and target to the toolbox features.
The content you are editing has changed. Please copy your edits and refresh the page.
Update the noxfile.py example to include modifications for adding the directory where noxfile.py is located to the Python path.
This documentation update is necessary because it's not guaranteed that the current directory is included in the Python path, especially in light of the changes described at https://fedoraproject.org/wiki/Changes/PythonSafePath. Users need clear guidance on how to adjust their Python path within noxfile.py to ensure smooth execution of nox.
@tkilias and I were wondering and discussing why the following section is included in the GitHub workflows proposed by PTB:
pull_request:
types: [opened, reopened]
We imagined the following rationale:
push operation could already be outdated# check results of former push could be outdatedThe current ticket request to enhance PTB in order to support users to maintain the changelog or their application projects.
Proposed work-flow for starting to work on a new version of the current application
pyproject.toml and calls
poetry run nox -s fixpoetry run version-check <path/version.py> --fixdoc/changes/changelog.mddoc/changes/unreleased.md with content from template (see below)Proposed work-flow for finishing and releasing a new version of the current application
doc/changes/unreleased.md (specification: TBD) and reports potential findingsdoc/changes/unreleased.md to file doc/changes/changes_<version>.md using the version from file pyproject.toml.doc/changes/changelog.mdProposed template, see https://github.com/exasol/notebook-connector/blob/main/doc/changes/changes_0.2.2.md
# <project-name from README.md> <version>, released t.b.d
Code name:
## Summary
<please enter summary here>
<!-- Sections: Changes, Features, Bugfixes, Refactorings, Documentation, Security -->
## Changes
* ISSUE_NUMBER: description
Automation, convenience, reduce manual work, increase quality, unify workflows, provide guidance for developers not familiar with processes and tools in Exasol Product Integration Team.
Sample repo already using the PTB: https://github.com/exasol/notebook-connector
Compare workflows with project-keeper and release-droid.
Add a command line command which allows a user to install or update a vagrant development environment from templates managed in this project.
Add support for passing build matrix arguments (python, exasol-versions) to workflow on call.
Currently a default build matrix is used, which may not match with the project(s) requirements.
(e.g. private projects which only support a single python version)
The Vulnerability issue creator creates issues for vulnerabilities in dependencies. These issues are currently not added to a GitHub project which would be necessary for our workflow.
After creating a new issue, the Vulnerability issue creator must add it to a configurable GitHub project.
To simplify preparing a release, we want to have an automated task in nox which takes care of the relevant changes.
Add a nox task named prepare-release which takes care of all common and error prone task regarding preparing a release:
References to workflows in files generated by PTB seem to be broken. In order to fix the CI build for repository notebook-connector we needed to replace exasol/python-toolbox/.github/workflows/[email protected] by ./.github/workflows/build-and-publish.yml
In file .github/workflows/checks.yml generated by PTB the indentation of the steps of build-documentation-job is wrong.
File .github/workflows/checks.yml, step "Check Version(s)" searches file exasol/toolbox/version.py while actually the github repo using the PTB should use a local path to its own version file, e.g. version.py
Missing test cases / files (e.g. test/unit/test_smoke.py) caused pytest to return exit code 5.
missing configuration files and documentation caused build errors. I needed to create files doc/conf.py and doc/index.rst. Adding empty files fixed the build errors.
Missing folder artifacts" caused workflow report.yml to fail.
In file report.yml step Copy Artifacts into Root Folder failed to copy files that were missing.
Upload artifacts is missing in build script File .github/workflows/checks.yml, after step poetry run nox -s lint
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: .lint.txt
path: .lint.txt
.github/workflows/checks.yml, after step poetry run nox -s coverage - name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: .coverage
path: .coverage
.github/workflows/checks.yml have wrong name:| step | current name | proposed name |
|---|---|---|
poetry run nox -s lint |
Run Tests | Run Linter |
poetry run nox -s type-check |
Run Tests | Run Type check |
poetry run nox -s coverage |
Run Tests | Calculate Test Coverage |
Potential mitigations for missing files
See repo https://github.com/exasol/notebook-connector/ for fixed version of the workflow files
Reproducibility: always
(See above)
(see above)
(see above)
(see above)
Fedora, python 3, Github.
Desktop:
In order to simplify setting up and maintaining a project with the python-toolbox, a command for managing
the github issue templates provided by the toolbox would be helpful.
Having a cli tool for this task will reduce the need of copy & paste and replacing dummy values.
TL;DR:
You can find the templates here: https://github.com/exasol/python-toolbox/tree/main/.github/ISSUE_TEMPLATE
List available issue templates:
tbox issue listInstall a specific issue template
tbox issue install bug...
Integrate the import-linter tool to enforce "contracts" between modules, specifying allowable imports. This enhancement ensures clear, predefined module relationships, benefiting project maintainers and contributors by preventing import loops and improving code structure.
The integration will enable conditional import linting, activating only when a relevant configuration exists within the project. This feature aims to enhance code quality and maintainability by enforcing module interaction rules.
import-linter provides a framework for defining and enforcing module relationships, important for complex projects or those with contributors lacking complete architectural knowledge. It helps maintain clear module interactions, supporting project structure and guiding contributors.
Documentation of PTB contains some issues that should be fixed.
File doc/user_guide/workflows.rst, section 2. Add the standard workflows to your project
tbox workflow install alltbx workflow install allFile doc/user_guide/workflows.rst, section 2. Add the standard workflows to your project should also mention that user needs to create folders in advance (optionally PTB could create the folders if not existing):
mkdir -p .github/workflows cd-job:
name: Continues Delivery
while it should be Continuous Delivery
doc/user_guide/workflows.rst.github/workflows/ci-cd.ymlTo enhance the identification and self-containment of the metrics, they should include information about their originating project.
The current data format is:
{
"commit": "38cef0d0b86fd9fcd042d874af5ebc6a17e937d1",
"date": "2023-12-18 07:19:14.687615",
"coverage": 26.629422718808193,
"maintainability": "B",
"reliability": "N/A",
"security": "N/A",
"technical_debt": "N/A"
}The proposed enhancement involves adding a project field. After updating, remember to release a new schema version at https://github.com/exasol/schemas.
Schema Repository: https://github.com/exasol/schemas
Relevant Discussion: https://github.com/exasol/github-issue-adapter/pull/223#discussion_r1456964783
{
"project": "python-toolbox",
"commit": "38cef0d0b86fd9fcd042d874af5ebc6a17e937d1",
"date": "2023-12-18 07:19:14.687615",
"coverage": 26.629422718808193,
"maintainability": "B",
"reliability": "N/A",
"security": "N/A",
"technical_debt": "N/A"
}I used the PTB workflow files for a new project repo saas-python-api which contains no code, yet, and only some placeholders for tests. Hence running tests does not generate any coverage.
The PTB workflow then fails due to no coverage available:
test/integration/test_placeholder.py::test_placeholder PASSED [100%]
============================== 1 passed in 0.01s ===============================
/home/runner/.cache/pypoetry/virtualenvs/saas-api-dRykLEL3-py3.10/lib/python3.10/site-packages/coverage/control.py:887: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
nox > poetry run coverage report -m
No data to report.
nox > Command poetry run coverage report -m failed with exit code 1
nox > Session coverage failed.
PTB workflows terminate gracefully even on new virgin projects without any test or coverage.
Add support to scan the project(s) license compliance.
Make sure to distinguish or build/dev dependencies etc from the dependencies used by the library/application.
Adding such a nox target will simplify validation of appropriate dependency usage in regard to the project(s) and their licenses.
The content you are editing has changed. Please copy your edits and refresh the page.
Currently the version check(s) have various small errors/bugs.
noxversion-check fails
$ version-check -f github_issue_adapter/version.py
Error while executing program, details: Couldn't find version within module
File exasol/toolbox/nox/tasks.py
Remove --fix
command = ["poetry", "run", "version-check"]
File exasol/toolbox/pre_commit_hooks/package_version.py
Add option --no-ansi
result = subprocess.run([poetry, "version", '--no-ansi'], capture_output=True)
Currently, the python-environment GitHub action installs the project without any extras. This issue proposes adding a parameter to the action that enables the installation of specific extras.
Often times, a Python project has optional dependencies that are not required for basic functionality but are necessary for certain features to work. These optional dependencies are often installed as extras. Currently, the python-environment action does not have an option to specify these extras during installation.
The proposed solution is to add an optional parameter to the python-environment action that enables the installation of extras. Users can specify the extras they need at runtime, and the action will install the project along with those extras.
Here's an example of how the new parameter can be used in a workflow:
- name: Install Project with Extras
uses: .../python-environment
with:
version: 3.8
extras: some_extra, another_extraIn the above example, the python-environment action installs Python 3.8 and the project, along with the some_extra and another_extra extras.
extras parameter to the python-environment action.To streamline the process of creating and publishing releases, the Python Toolbox shall introduce standard tasks for these actions. These tasks should be readily adaptable for both project-level utilization and integration into workflows.
The tasks should be nox based and work both locally and within continuous integration setups. Each task should focus on a specific aspect of the release process rather than attempting to encompass all release-related actions in a single task. E.g.:
Further information also can be found in the currently unimplemented nox tasks stub exasol.toolbox.nox.tasks:release.
In order to simplify setting up and maintaining a project with the python-toolbox, a command for managing
the configuration files would be helpful.
Having a cli tool for this task will reduce the need of copy & paste and replacing dummy values.
TL;DR:
Generate nox config file(s)
tbox config generate nox
tbox config generate noxconfigUpdate the tool configuration(s) in the pyproject.toml file
tbox config update pyprojectnoxconfig.pynoxfile.pypyproject.toml with the expected tooling settings
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.