metanorma / bipm-data-importer

Data importer for BIPM

BIPM data importer for CIPM and CGPM outcomes

Purpose

Provides executables that help fetch CIPM and CGPM outcomes from the official BIPM website.

Usage

To fetch CGPM outcomes:

$ bipm-fetch-cgpm

To fetch CIPM outcomes:

$ bipm-fetch-cipm

To fetch CIPM decisions:

$ bipm-fetch-cipm-decisions

License

Copyright Ribose. MIT license.

bipm-data-importer's Issues

Update fetching location of "CGPM Resolutions"

All located here:
https://www.bipm.org/en/committees/cg/cgpm/cgpm-resolutions

Individual pages look like this:
https://www.bipm.org/en/committees/cg/cgpm/1-1889/resolution-

We need to keep all information of the page, including the PDF link, the page number, date of meeting, etc.

Assign different types to "resolution"s, the `decision` and `recommendation`

• relaton/relaton-bipm#4 (comment) : mark BIPM decisions as type: decision
• metanorma/bipm-data-outcomes#6 : mark CCTF recommendation as type: recommendation

Use user home path `HOME_PATH` or `TEMP_PATH` instead of somewhere inside the Ruby gem directories

bipm-data-importer/lib/bipm/data/outcomes.rb

Lines 8 to 10 in d97f398

	def self.file_path
	@file_path ||= "#{__dir__}/../../../data/"
	end

In some environments like Tebako, the Ruby runtime and its libraries are located in a read-only partition. We have to use the user home path or a temp path for dynamically downloaded files.

Remove all `type: affirmative` and `degree: unanimous` lines from data

Janet Miles of BIPM wishes to remove them:
metanorma/metanorma-bipm#164 (comment)

Add auto-release workflow

Update resolution YAML to provide the "subject body"

As per https://github.com/metanorma/resolution-model#data-model , every meeting and decision requires a "subject body".

We need to set it as subject-body.

The values could be CIPM, CGPM, CCTF and the old name of CCTF, the CCDS.

Update URLs for `pdf` and `reference`

From Janet Miles of BIPM:

In the YAML files:

I see that two different URLs are given for the “pdf” and “reference” (which correspond to the document on the BIPM website).
Can you please omit the segment “?version=1.3&download=true”, and for the links use the versions as given on https://www.bipm.org/en/committees/cg/cgpm/publications, https://www.bipm.org/en/committees/ci/cipm/publications, https://www.bipm.org/en/committees/ci/cipm/older-meeting-reports, https://www.bipm.org/en/committees/cc/cctf/publications
(Unfortunately the documents are posted twice.)

Meeting file names instead of "101(I)" and "101(II)", let's use "101-1" and "101-2"

From metanorma/bipm-data-outcomes#10

e.g. https://github.com/metanorma/bipm-data-outcomes/blob/main/cipm/meetings-en/meeting-101(I).yml

Also, instead of this identifier for a meeting:

  identifier: 101(I)

Let's use this:

  identifier: 101-1

Use HOME path as default for cloned data instead of GEM_ROOT

The {GEM_ROOT} path may not be writable, for example, in the case in Tebako (read-only filesystem). We should use the home path as default.

Originally posted by @ronaldtse in #37 (comment)

Update fetching location of CIPM meetings

All available here: https://www.bipm.org/en/committees/ci/cipm/meetings

We need to link the CIPM meetings to the CIPM Decisions and CIPM Recommendations too.

Update data importer code to work with latest BIPM website, import all consultative committees

BIPM has updated their website to now provide information from the consultative committees.

You can find this list in the header of the BIPM website:

We would need obtain the information from those committees:

• JCRB: https://www.bipm.org/en/committees/jc/jcrb
• JCGM: https://www.bipm.org/en/committees/jc/jcgm
• JCGM WG 1: https://www.bipm.org/en/committees/jc/jcgm/wg/jcgm-wg1-gum
• JCGM WG 2: https://www.bipm.org/en/committees/jc/jcgm/wg/jcgm-wg2-vim
• CCU: https://www.bipm.org/en/committees/cc/ccu
• CCTF: https://www.bipm.org/en/committees/cc/cctf
• CCT: https://www.bipm.org/en/committees/cc/cct
• CCRI: https://www.bipm.org/en/committees/cc/ccri
• CCPR: https://www.bipm.org/en/committees/cc/ccpr
• CCQM: https://www.bipm.org/en/committees/cc/ccqm
• CCM: https://www.bipm.org/en/committees/cc/ccm
• CCL: https://www.bipm.org/en/committees/cc/ccl
• CCEM: https://www.bipm.org/en/committees/cc/ccem
• CCAUV: https://www.bipm.org/en/committees/cc/ccauv

The types of data include:

• publications
• meetings
  • decisions
  • resolutions
  • documents
• working groups
  • meetings
    • decisions
    • resolutions
    • documents

Ruby Psych fails to parse correctly formatted YAML dates

metadata:
  title: 'Resolutions of the CGPM: 1st meeting'
  date: 1889-09-28
  source: BIPM - Pavillon de Breteuil
  url: https://www.bipm.org/en/CGPM/db/1
resolutions:
- dates:
  - 1889-09-28

The 1889-09-28 date instances will fail on YAML.load like this:

Failure/Error: YAML.load_file(Pathname.new(f).expand_path.to_s)

Psych::DisallowedClass:
  Tried to load unspecified class: Date
# (eval):2:in `date'
# ./spec/bipm/data/importer/cgpm_spec.rb:8:in `block (2 levels) in <top (required)>'
# ./spec/bipm/data/importer/cgpm_spec.rb:6:in `map'
# ./spec/bipm/data/importer/cgpm_spec.rb:6:in `block in <top (required)>'
# ./spec/bipm/data/importer/cgpm_spec.rb:3:in `<top (required)>'

The reason is the Psych YAML parser at Ruby has the following bug but it was never fixed, for 6 years!
ruby/psych#262

In that ticket some suggested that only "invalid dates" will raise this issue, but the case is all dates will cause this issue. This is frustrating because the YAML spec does support the "Date" type.

Simply speaking:

irb(main):004:0> YAML.load("1900-01-01")
~/.rbenv/versions/3.1.0/lib/ruby/3.1.0/psych/class_loader.rb:99:in `find': Tried to load unspecified class: Date (Psych::DisallowedClass)

Update data importer to import with updated "types"

The resolution model now provides extra types:
metanorma/cipm-resolutions@f52ffc8

ConsiderationType:

   accepts
   observing
   referring
   acting
   empowers
   reaffirming

ActionType:

   declares
   elects
   authorizes
   charges
   states
   remarks
   judges
   sanctions
   abrogates
   empowers

Please refer to the changes in the following commits:

• metanorma/cipm-resolutions@1074e03
• metanorma/cipm-resolutions@f52ffc8
• metanorma/cipm-resolutions@1bb399c
• metanorma/cipm-resolutions@8db5dd8
• metanorma/cipm-resolutions@dc5bccc
• metanorma/cipm-resolutions@58ed5ba
• metanorma/cipm-resolutions@67abc5f
• metanorma/cipm-resolutions@2319994
• metanorma/cipm-resolutions#6

Update fetching location of "CIPM Recommendations"

From Janet Miles of the BIPM:

Most (if not all) of the CIPM Recommendations have been entered into the website, but unfortunately the problems with their display have not yet been resolved (for example, you will see "Resolution" even where it should be "Recommendation").

i.e. These are Recommendations, not Resolutions.

They should all be available here: https://www.bipm.org/en/committees/ci/cipm/recommendations

Data locations:

e.g.
The 1948 Resolution at https://www.bipm.org/en/committees/ci/cipm/42-1948/resolution-
The 1946 Resolutions at:

• https://www.bipm.org/en/committees/ci/cipm/41-1946/resolution-2 and
• https://www.bipm.org/en/committees/ci/cipm/41-1946/resolution-1.

All the links include "resolution" regardless of whether the text was a "resolution" or "recommendation".
In the software we have to differentiate.

It might however be helpful to the user to retain the page information (e.g. CIPM, 1948, p88).

We need to retain the page information.

We also need to provide this URL in the data structure.

Update fetching location of "CIPM Decisions"

All "CIPM Decisions" are available from this page now:
https://www.bipm.org/en/committees/ci/cipm/outcomes

We also need to support the "Topic" tag per Decision.

Create gem

Wrong `type` value in CIPM Meetings

From @manuel489 in metanorma/bipm-data-outcomes#11

In CIPM, there are several cases where type=recommendation should be type=resolution.

Check Meeting 41, as an example of several ones.

It is the title that determines the type, not the URL. The URL are all "resolution" but the pages title can give "Recommendation".

This is generating fetching issues from relaton-bipm:

• metanorma/bipm-si-brochure#169

Change branch to main from master

Move data to the data repos

The data fetched should be stored in the other repos.

Generalize this gem so it not only imports from the site, but it provides a Ruby API to access the resolutions

This gem should be generalised to offer an API to access the BIPM Resolutions according to the resolution-model.

LutaML can be used to generate the Ruby API as used in the Metanorma-Document gem for consistency in model-driven development.

Fetch a CCTF resolution that cannot be reached from anywhere

https://www.bipm.org/en/committees/cc/cctf/5-1970/resolution-2

We need to make this resolution available in the output dataset. However it looks like there is no page that links to this one!

Provide details of the reference, not only the URL

When we see this: https://www.bipm.org/en/committees/cg/cgpm/26-2018/resolution-1

There are actually 3 types of information provided:

• Title of the reference: "Proceedings of the 26th CGPM (2018), 2019"
• Page number of the reference: "p472"
• URL of the reference: https://www.bipm.org/documents/20126/17314988/CGPM26.pdf/9db96c32-a986-e32a-09f9-3ed7e6c77cf7?version=1.5&t=1644249673647&download=true#page=472

We want to add such information in structured format in the bipm-data-outcomes files:
https://github.com/metanorma/bipm-data-outcomes/blob/d625c7216260970de3904e4979c12f0b34679e3c/cgpm/meetings-en/meeting-01.yml#L8-L14

Retrieve both English and French texts from their corresponding pages

BIPM is a bi-lingual organization and the French version is the authoritative one.

All English text should now have a corresponding French counterpart.

metanorma/bipm-data-outcomes#9

Refactor gem to utilize the Edoxen gem

Currently the bipm-data-importer implements an object model for BIPM outcomes. The BIPM outcomes are actually based on the Metanorma resolutions model, which is now renamed to "Edoxen".

Once the new Edoxen gem is implemented, we can refactor the bipm-data-importer to use / extend from Edoxen directly. This allows us to better manage resolutions and outcomes from different bodies.

Depends on:

• metanorma/edoxen#1

CCDS Recommendation 2 is currently labelled as CCTF

From Janet Miles of BIPM.

For the CCDS Recommendation 2 (1970) the link is https://www.bipm.org/en/committees/cc/cctf/5-1970/resolution-2, but the committee at that time was called CCDS, not CCTF.

We need to provide the subject-body in the YAML file to say CCDS instead of CCTF.