finos / metadata-tool Goto Github PK

Feature Request

Within GitHub Issue https://github.com/finos/metadata/issues/648#issuecomment-706807544 an automated script is being used to run a report that extracts FINOS project maintainers. However, this script currently excludes FINOS projects of type SIG.

Special Interest Group Type

  "type"             : "SIG"

Including SIG Type

This issue requests metadata-tool is extended to include type SIG, potentially included in the code block below, or potentially as the new definition defn sigs-metadata

Potential SIG Definition

(defn sigs-metadata
  "A seq containing the metadata of all activities of type SIG, regardless of program."
  []
  (filter #(= (:type %) "SIG") (activities-metadata)))

The Ask

• Before a pull request is raised, it would be good to discuss and agree the right approach / desired outcome with @maoo and @mcleo-d.
• Does the addition of the SIG type potentially lead to unforeseen consequences when reporting against FINOS projects?

In order to maximise the amount of people signed on https://github.com/orgs/finos/people , while enforcing CLA signature, metadata-tool can embed the logic of inviting GitHub users as soon as they are covered by CLA.

Code is already implemented on #53 and delivers the command invite-clas-to-finos-org.

Rollout is tracked on finos/open-developer-platform#114

This is basically the same as one of the checks, but is worth including in the report to facilitate the PMC's steering responsibilities.

GitHub provides license information for all repositories - check this and report on any repositories that have non-standard licenses (i.e. are neither Apache-2.0 nor Creative Commons Attribution 4.0 International licensed).

Feature Request

Description of Problem:

Currently, to blacklist an identity in Bitergia, we need to notify Bitergia Support.

However, it is possible to define blacklist via the bitergia-affiliations.yaml, see https://gitlab.com/Bitergia/c/symphony/support/issues/107 (FINOS internal only) for more info.

The blacklist format is:

- blacklist:
    - [email protected]
    - root
    - Generic Account

Proposed solution

We could start with a simple JSON list that defines the blacklist items, defined in the root folder of metadata or in the people/ folder ; at that point, we can edit the bitergia-affiliations.ftl to render out the list, when invoking gen-bitergia-affiliation-data

Feature Request

Description of Problem:

Forked repositories (hosted in FINOS) should not be considered FINOS activities, therefore they should not be part of checks, bitergia and catalog exports.

Potential Solutions:

The simplest solution is to update the activity schema:

• Add a field called "githubForks" in activity-metadata.json , see https://github.com/finos/metadata/blob/master/jsonschema/activity-metadata/1.0.0#L15
• Move repositories from https://github.com/finos/metadata/blob/master/programs/symphony/symphony-electron/activity-metadata.json#L8 to "githubForks"

Bug Report

The Confluence user API have been deprecated, in June 2019; since then, the FINOS meeting attendance tracking is not able to crawl Confluence data, leading to the build to constantly fail.

Steps to Reproduce:

1. Checkout metadata-tool project
2. Setup the project to run local builds
3. Checkout metadata (private) repository in a sibling folder
4. Run lein run -- gen-meeting-roster-data -m ../metadata

Expected Result:

Generate a file finos-meetings.csv in the root folder containing FINOS meeting attendance crawled from https://finosfoundation.atlassian.net/wiki

Actual Result:

finos-meetings.csv is empty. The logic also fails with an HTTP 400 error when calling the Confluence user API, leading to NPE in the parse-string function.

Proposed fix

Use selenium to crawl public HTML, extract the full name of attendees and match it against the FINOS person Metadata fullName field.

Note that this logic shall apply only for the part of logic that parses a meeting page, not the logic to browse Confluence tree and identify meeting pages.

Current work

Added Selenium dependencies to https://github.com/finos/metadata-tool/tree/confluence-selenium-crawler and setting up development of the new feature.

When GitHub stats (especially watchers and stars) are accumulated from the repository level to the project level, there is some double (or triple, quadruple, etc.) counting going. See this code for example.

What should happen is that any time any one user watches any number of repositories in a project, they're only counted once.

See GitLab.

This is a message generated by the check-project-repos command in FINOS metadata-tool.

We have found some repository configurations that should be changed in order to comply with FINOS Governance and bylaws, see the details below.

List of fixes:

• no-badge - README.md file is missing the FINOS badge; check the README.md template and make sure that it embeds one of SVG FINOS badges.
• has-user - One or more user collaborators were found in this GitHub repository. FINOS Governance only allows GitHub users to be added via Teams. Please remove it, therefore it must be removed.
• no-teams - This GitHub repository does not grant permissions to any FINOS Team, although it should be configured to grant access to the program and project specific teams defined in https://github.com/orgs/finos/teams. Please email [email protected] and coordinate changes to the repository access permissions.
• no-whitesource - WhiteSource configuration was not found; make sure that dependencies are scanned against security vulnerabilities. Read more on the WhiteSource Wiki page.
• has-admin - One or more admin collaborators were found in this GitHub repository.. FINOS Governance doesn't allow GitHub users to have Admin rights on repositories, therefore it must be removed.

You can find more docs on the code validation ODP docs page
For any question, do not hesitate to contact @finos-staff or email [email protected]. Thank you!

The activities JSON that's consumed by the catalogue contains fork, watcher, and star counts of zero for all projects.

This isn't correct, as at least some of these projects have non-zero counts for these metrics.

Steps to Reproduce:

1. cd metadata-tool
2. lein ancient

Expected Result:

No out-of-date dependencies are listed.

Actual Result:

$ lein ancient
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by clojure.lang.Reflector (file:/usr/local/Cellar/leiningen/2.8.3/libexec/leiningen-2.8.3-standalone.jar) to method com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader(java.io.Reader)
WARNING: Please consider reporting this to the maintainers of clojure.lang.Reflector
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[org.clojure/clojure "1.10.0"] is available but we use "1.9.0" (use :check-clojure to upgrade)
[org.clojure/tools.cli "0.4.1"] is available but we use "0.3.7"
[cheshire "5.8.1"] is available but we use "5.8.0"
[mount "0.1.15"] is available but we use "0.1.12"
[metosin/scjsv "0.5.0"] is available but we use "0.4.1"
[irresponsible/tentacles "0.6.3"] is available but we use "0.6.2"
[cc.qbits/spandex "0.6.4"] is available but we use "0.6.2"
[com.draines/postal "2.0.3"] is available but we use "2.0.2"
[joda-time "2.10.1"] is available but we use "2.10"
[clj-http "3.9.1"] is available but we use "3.9.0"
[midje "1.9.4"] is available but we use "1.9.1"

When a developer is trying to understand how a codebase configures itself, one of the biggest root causes of accidental complexity is having multiple configuration mechanisms, formats, and/or locations. With the introduction of the meeting-crawler.edn file, the metadata-tool now suffers from this problem.

Originally, the metadata-tool was intentionally designed to have a single, clearly documented configuration file (config.edn) for all configuration, specifically to avoid this issue.
This was based, in part, upon prior experience of large Spring-based applications which tend to end up with hundreds (or more!) of small, isolated XML and Java properties files, making it practically impossible to statically determine (i.e. solely from the code) how a particular runtime instance of the application is actually configured.

Feature Request

Description of Problem:

Linting is manual (or editor based) right now - would help to have automated linter checks inside Travis.

Add a response code so that e.g. syntax validation failures can cause the Travis build to fail.

The set-exit-code fn in the exit-code ns should validate new-exit-code before swapping the value in the private atom.