season range not recognized about edtf.js

[njbart]

It was just confirmed on the EDTF mailing list that seasons in intervals will not be supported by the upcoming standard.

That decision – if that’s the final word – is deeply regrettable. In my opinion, it needs to be reversed as soon as possible, and we should try and find a way to make the voice of the CSL (and biblatex) community heard here. After all, season intervals are by no means uncommon in bibliographic data (one example from the Chicago Manual of Style, 16e, 14.271: “Autumn 1980–Summer 1981”), and they need to be entered, represented, and processed somehow. It just does not make any sense to promote ISO 8601/EDTF in general (including seasons – “do use 1980-23”), but at the same time introduce an exception for season intervals (“do not use 1980-23/1981-22”).

I wonder if it wouldn't be better to prefer a different notation

But how could that look like? In my opinion, given that YYYY-MM/YYYY-MM and YYYY-SS are valid, YYYY-SS/YYYY-SS is the only straightforward representation there is.

(BTW, biblatex does accept the YYYY-SS/YYYY-SS format in a date field, too.)

I'm open to try and add this as an experimental feature (which can be turned off)

I’m all in favour.

This would certainly facilitate the adoption of EDTF.js in all contexts where season ranges need to be processed (I’m thinking about, for example, citeproc-js), but it would also document the general need for this feature by way of including it in a reference implementation.

[njbart]

The first serious argument against permitting season intervals I’ve seen is this one (by Saašha Metsärantala on the EDTF mailing list):

As far as I remember, intervals including seasons were excluded because of the difficulties to validate them - or the impossibility to validate many of them.

Indeed, 1980-04/1980-06 can be validated, whereas 1980-06/1980-04 is obviously invalid. Likewise, 1980-04/1981-21 can be validated, whereas 1981-21/1980-04 is obviously invalid. But the answer is not obvious with 1980-21/1980-23, 1980-23/1980-21 1980-07/1980-21 or 1980-07/1980-23.

Now, the “non-obvious” cases seem to be ambiguous only if we assume that point seasons (if that’s the right word) have no exact definition (day precision, or even second precision) either, in other words, if we leave it open whether 2018-21 is intended to mean:

• 2018-03-20/2018-06-20 (Northern hemisphere, astronomical), or
• 2018-09-23/2018-12-19 (Southern hemisphere, astronomical), or even
• 2018-09-01/2018-11-30 (Southern hemisphere, meteorological), or a number of others.

So it all seems to hinge upon the question of what exactly the formulation in the ISO 8601 2016-10-26 draft, 4.7 is supposed to mean:

21–24 = Spring, Summer, Autumn, Winter, independent of Hemisphere

If “independent” means “undefined”, then it would seem the usefulness of the “21 to 24” notation is severely limited – unless, that is, some tacit “mutual agreement” (a notion the draft uses a lot, though not in the context of seasons) could be construed, e.g., the assumption that Northern/astronomical will be used as a default (possibly not too popular on the Southern hemisphere, though …).

On the other hand EDTF.js does seem to assume that seasons 21 to 24 can be defined precisely: @inukshuk confirmed earlier in this thread that “All EDTF objects have a shared interface (i.e., they have a min/max timestamp and can be enumerated)” – which makes me wonder whether ISO 8601 actually justifies this …

So, my point is: It seems that either some consensus on a precise definition of seasons “independent of Hemisphere” must be reached (in which case both the current practice of EDTF.js concerning seasons 21 to 24, and the use of season intervals becomes unproblematic), or, if such a definition cannot be agreed upon, the use of the “21 to 24” notation should be discouraged altogether, in favour of the level 2 values “25 to 28” (Northern) or “29 to 32” (Southern hemisphere), which do allow a precise definition of both seasons and season intervals.

Any thoughts?

[njbart]

Are seasons inherently ambiguous in EDTF?

“21” to “24” on level 1, WRT order, let alone exact start/end dates: apparently yes.

On Level 2, “25” to “28” and “29” to “32” would seem to be properly defined WRT order (not necessarily exact start/end dates).

But for bibliographic purposes, we hardly need anything more than Level 1 with season intervals, and if we don’t know where a publication comes from, the date of a “Spring/Summer 1980” issue from the US (1980-21/1980-22) is just as valid as the date of a “Winter/Spring 1980” issue from Australia (1980-24/1980-21).

Unless something’s wrong with my argument, this probably means that EDTF.js should also accept season intervals such as 1980-24/1980-21 (which it currently doesn’t). EDIT: It does, see below.

[JohnLukeBentley]

Thanks, inukshuk.

If this doesn't represent a change in your position then, perhaps, my understanding of your original position was mistaken. I was under the impression that if I went ahead with a format proposal beyond what ISO was stipulating then you'd be (provisionally) open to implementing what that (further developed) format proposal stipulated. That is, to avoid waiting for ISO and given the opacity of the ISO development process.

That was part of my reason for writing the draft BibliographicDatetimeFormat.pdf

Given what you've written above it, instead, you (did and still do) prefer to stick much closer to ISO. That is, in waiting to see their published result; and in being disinclined to entertain ISO non-conforming formats.

Thanks very much for the link to Ray Denenberg's recent post. It is extremely illuminating. Given he is the chair of the relevant ISO working group that may well open a backdoor (via the EDTF mailing list) to engage with ISO. I'm given special hope in virtue of him having "brought edtf to ISO" (note to @njbart).

All that means, taken together with @plk's lack of response, that I intend to talk about BibliographicDatetimeFormat.pdf with Ray on the EDTF mailing list soon. Given this new information (about Ray) that's potentially as much more fruitful course compared with having you and @plk proceeding straight to implement some version of the Bibliographic Datetime Format.

[JohnLukeBentley]

Lo and behold EDTF has recently been (updated and) published (2018-10-22): http://www.loc.gov/standards/datetime/edtf.html

[inukshuk]

Thanks for reporting this issue @retorquere and @njbart! Seasons in intervals are rejected by the parser on purpose, because they are not valid (to the best of my knowledge). Looking at section 4.4 of the spec my understanding is that level 1 extends the ISO intervals by:

• Unknown and open start or end
• Uncertain and/or approximate start or end

And at level 2:

• Partial uncertain and/or approximate start or end
• Before or after tags

It does not mention intervals including 'seasons' (defined as sections of a year in section 4.7); there are multiple extended features, which, to my understanding, are not valid in intervals (e.g., long years, one of a set, multiple dates, etc.).

I think it would be relatively easy to add support for seasons in intervals, but I would like to be confident that it is actually valid. Alternatively, we could add it as a non-standard level 3 feature (to make it possible to switch it off), but if this is non-standard, I wonder if it really makes sense to do that -- shouldn't we try to use a standard syntax instead?

If we implement this, we'd also need to be clear if intervals should be strictly between seasons or if date / seasons intervals should be allowed. All EDTF objects have a shared interface (i.e., they have a min/max timestamp and can be enumerated) so mixed intervals are probably OK.

[inukshuk]

Searching the EDTF list archive, I found a suggestion to allow seasons in intervals, specifically for publication references. But it seems to me like this was not picked up for ISO 8601?

[retorquere]

4.4 doesn't mention seasons, no, but the way I'm reading it, it says "you can add these qualifiers to valid start/end dates", but doesn't specify there further restrictions on the dates. It does list a few samples under "formats", but those would also exclude 1980-04/1980-06.

[inukshuk]

The EBNF in Annex A definitely covers 1980-04/1980-06 in the L1Interval rule but does not include divisions/seasons.

[njbart]

Maybe I’m mistaken, but my interpretation is that 4.7 from the 2016-10-26 draft permits any year-and-month expression to be replaced by a year-and-season expression – and I think it’s uncontroversial that year-and-month ranges/intervals such as 1980-04/1980-06 are valid:

“4.7 Divisions of a year
For a year-and-month expression (e.g. 1984-04) the month component may take on values of 21 or above (in place of a month value, 01 through 12). These values signify a division of a year (e.g. “the season Spring”).
4.7.1 Level 1
The values 21, 22, 23, 24 may be used to signify ' Spring', 'Summer', 'Autumn', 'Winter', respectively. Format: YYYY-SS
Example:
· 2001-21 (Spring, 2001)”

[njbart]

@JohnLukeBentley – I know you have been involved in implementing ISO 8601/EDTF for biblatex, and have started the process of joining up to the relevant ISO committee – any additional insights from you?

[inukshuk]

It was just confirmed on the EDTF mailing list that seasons in intervals will not be supported by the upcoming standard.

I'm open to try and add this as an experimental feature (which can be turned off), however, if it's not going to be supported by the standard, I wonder if it wouldn't be better to prefer a different notation,

[retorquere]

I'm not at all up to speed with how the standard evolved, but it would seem odd to me to pick a different notation for seasons depending on the context where they're used.

[retorquere]

(BBT already has a workaround in place, BTW, so from my POV this is not urgent)

[inukshuk]

@njbart you might want to join the EDTF mailing list and weigh in there (there has been some renewed discussion about the season format there), but I have no idea what the chances are to get this into the spec.

Meanwhile, I've added an experimental/non-standard level 3 to the parser, which covers seasons in intervals and updated the Interval and Season classes accordingly. By default, the parser will reject this, so you'll have to set the level specifically. Examples:

> edtf('2017-24/2018-21')                                                                             
Error: edtf: No possible parsings (@EOS) for "2017-24/2018-21"                                        
    at parse (/home/dupin/Work/edtf/js/src/parser.js:33:23)                                           
    at edtf (/home/dupin/Work/edtf/js/index.js:19:15)                                                 
> edtf('2017-24/2018-21', { level: 3 })                                                               
2017-24/2018-21                                                      
> edtf('2017-24/2018-21', { level: 3 }).includes(edtf('2017-24'))                                     
true                                                                                                  
> edtf('2017-24/2018-21', { level: 3 }).includes(edtf('2018-01'))
false
> edtf('2017-24/2018-21', { level: 3 }).covers(edtf('2018-01'))
true
> [...edtf('2017-24/2018-21', { level: 3 })]
[ 2017-24, 2018-21 ]
> [...edtf('2017-24/2018-23', { level: 3 })]
[ 2017-24, 2018-21, 2018-22, 2018-23 ]

The same works for .parse if you just want to parse dates. And instead of passing the level constraint all the time, you can also change the default:

> edtf.defaults.level = 3
3
> edtf.parse('2017-24/2018-21')
{ values:
   [ { type: 'Season', level: 1, values: [Array] },
     { type: 'Season', level: 1, values: [Array] } ],
  type: 'Interval',
  level: 3 }

If there are no objections, I'll push this to NPM in the next few days.

[retorquere]

No objections from me, but I won't be able to use this. For biblatex compatibility I will need to parse level 1, not 2, with the exception of season ranges being valid. It's not a problem for me pragmatically (I have a workaround).

[inukshuk]

The constraints are applied after parsing anyway (earley parsers may generate multiple possible paths) and, for multiple results, you will always get the one at the lowest level. That means, if you wanted to use the feature you could parse at level 3 and add your own check at the end: if the result is level 2 you treat this just like a parser error.

[retorquere]

I could do that.

[Crissov]

When a standard is ambiguous or leaves something undefined, Postel’s Law applies:

Be conservative in what you do,
be liberal in what you accept from others.

[retorquere]

That depends on what you want to achieve. I use EDTF.js not only to parse dates but also to validate that a date is to spec.

[retorquere]

Plus, the EDTF mailing list has unambiguously confirmed what the behavior should be. While I'm all in favor of adding season ranges as part of the standard, I don't think it's EDTF.js's job to lie when I'm asking "is this valid ETDF". If I'm asking (by means of setting the level) "is this a valid EDTF level 1 date", than it must answer "no" for the current spec.

[njbart]

I don't think it's EDTF.js's mission to lie when I'm asking "is this valid ETDF".

Absolutely. That’s why I fully agree with treating season intervals, for now, as an experimental/non-standard level 3.

As to the ISO 8601 decision making process – does anyone know more about how this is organized?

• Are the views that season intervals will not be part of the upcoming standard, and that it’s too late now to change anything (except for “compelling reasons”), both expressed by Ray Denenberg on the EDTF mailing list, actually authoritative?
• Are there forums other that the EDTF mailing list where such matters are discussed, and how exactly are decisions being made?

[JohnLukeBentley]

@njbart (and all) sorry for the delay: I'm just emerging out of a nasty flu.

As you've noted I'm part way through attempting to join the relevant ISO technical committee (as the Australian representative). I've just sent off my application. So I'm as yet unclear on the exact nature of the decision making process. I've so far been following the clues ...

From How we develop standards there is ...

Do you want to get involved in standards development? ... Whether you’re a consumer or in business you can be part of the next generation of standards. ...

https://www.iso.org/get-involved.html

Standards are developed by groups of experts called technical committees. These experts are put forward by ISO’s national members. If you are interested in getting involved, contact your national member. Contact details can be found in the list of national members. [Link original]

From ISO/DIS 8601-1 the relevant technical committee is: ISO/TC 154 Processes, data elements and documents in commerce, industry and administration

I'm operating on the assumption that there exists an ISO online forum somewhere that is authoritative.

If my application is successful my first set of recommendations will go to opening up the process. That is, by suggesting the draft standard be published on github, or similar, to allow anyone to both: see that latest draft; and weigh in on discussions.

The draft ISO 8601-201x is already in an advanced state. As of today "40.99 Full report circulated: DIS [Draft International Standard] approved for registration as FDIS [Final Draft International Standard]". I too hope it's not too late to make various changes.

As to the specific issue of whether the draft (2016-10-26) standard permits seasons in intervals I think the prior discussion is sufficient evidence of the ambiguity. So removing that ambiguity will be a further recommendation.

I think, moreover, the standard ought support seasons in intervals for the reasons you mention.

[njbart]

@JohnLukeBentley – many thanks for sharing these details.

[njbart]

Another question (by a js newbie): What would be the best way to run the parser, for testing, on the command line?

[retorquere]

const edtf = require('edtf')
console.log(edtf('2016-XX'))

or

const edtf = require('edtf')
console.log(edtf(process.argv[2]))

[retorquere]

(ran using node)

On the topic of seasons: https://www.quora.com/What-countries-or-regions-have-more-or-less-then-the-4-standard-types-of-seasons (having grown up in the tropics myself, where we had the large/small rainy season and the large/small dry season, and none of that spring/summer/fall/winter malarky)

[retorquere]

Which is to say that the exact pin-pointing of point-seasons is less problematic than the assumption that there are 4 exactly four, and that they temporally map onto the western-european times for spring etc. I was looking at season parsing specifically from the pov of western-european customs, as it looked to me like the EDTF spec was doing the same.

[njbart]

Re: js/command line: Thanks, I think I got it, at least the first version works for me. Is it normal if node prints undefined after each output? Also, console.log(edtf('2016-21').min) outputs 1451606400000 – how would I get an ISO date/time instead?

[retorquere]

The 2nd one was meant to be ran as a command line script and would accept the date to parse as the first parameter. In the REPL it's normal that node prints out undefined as that's the return value from console.log; in the REPL you could just type edtf('2016-21') and it would print out the result.

If you want a date from edtf('2016-21').min, new Date(edtf('2016-21').min) should work and should display as an iso string.

[retorquere]

Anyhow, WRT argument by Saašha Metsärantala, I am struggling to understand why this would be a problem for season range parsing, but not a problem for point-date season parsing.

[inukshuk]

I should explain that EDTF.js consists of two parts: the date parser and the date objects / API. The parser tries to follow the spec (i.e., it can be configured to distinguish between the extension levels) and returns 'plain' JavaScript objects with the parsed values and the detected type and level. The extended date objects offer a set of tools to work with dates and the extended functionality supported by the spec (and to format extended dates based on different locales).

The main motivation for the extended objects is for applications dealing with EDTF date input: in my case, I need to be able to index the time range an EDTF date belongs to in a spatial database index. For this reason, I have added a shared interface to all date objects (dates at the different levels of precision, approximate dates, seasons, intervals, etc.): each has a min/max timestamp and allows for enumeration and coverage tests. Timestamps are used internally for pragmatic reasons, to ensure we can rely on the platform's date implementation for issues like leap years or handling pre-Gregorian dates. This assumption, that each date can be mapped to a time range with a clear min/max is not part of the spec and something that may require some configuration based on your application (e.g., you'd want to define the factor of 'uncertainty' of approximate or uncertain dates, etc.).

In any case, I think we should limit this discussion to the parser part only (and if I understand correctly, @retorquere is only using the parser anyway). Here the argument, as I understand it, is that allowing seasons in intervals is difficult to validate because of cases like 2018-21/2018-40 or other combinations of different groups of divisions (but I might be missing the point).

[njbart]

Anyhow, WRT argument by Saašha Metsärantala, I am struggling to understand why this would be a problem for season range parsing, but not a problem for point-date season parsing.

I think the argument is as follows: Point seasons cannot be invalid (whatever their order in a given year, there’s no question that 1980-21, 1980-22, 1980-23, 1980-24 are all valid), but season ranges, if undefined wrt start-/end date or at least relative order, can , e.g., 1980-21/1980-23 is invalid on the Southern hemisphere, since within one year, spring does not come before autumn.

However, I can think of a counterargument: If we read “independent of Hemisphere” as “regardless of hemisphere-related constraints wrt relative order”, then any season interval that is possible on either of the hemispheres should be valid.

This, I feel, is sufficient for neutralizing Saašha Metsärantala’s argument.

Thoughts?

[retorquere]

Ah, right, but then that highlights another issue, right? Are seasons inherently ambiguous in EDTF?

[retorquere]

That's my thinking. If the ambiguity in level 1 is resolved with "assume northern hemisphere" for point seasons, it is likewise resolved for season ranges.

[inukshuk]

@njbart the parser currently accepts 1980-24/1980-21 just fine; it's only the date API which does not, because, as explained previously, we map seasons to actual time ranges in order to make computations possible. But again, this is an application specific mapping (if you do want to make these computations, you have to map to a time range sooner or later) and not related to the spec.

Parser only:

> edtf.parse('1980-24/1980-21', { level: 3 })
{ values: 
   [ { type: 'Season', level: 1, values: [Array] },
     { type: 'Season', level: 1, values: [Array] } ],
  type: 'Interval',
  level: 3 }

[retorquere]

This is great AFAIC. The standards discussion stands apart from this.

[njbart]

the parser currently accepts 1980-24/1980-21 just fine

My fault, sorry.

[JohnLukeBentley]

@inukshuk, in case you have name mentions turned off, in Datetimes. The draft ISO 8601-201x and EDTF > My comment of 2018-02-09 20:54+11 I ask you ...

Would you, in principle, be interested in supporting an "Bibliographic [or some more general identifier] Datetimes: An IS0 8601 Profile with Extensions And Contradictions"?

... the rest of that post (no need to read the thread) supplies the relevant context.

[JohnLukeBentley]

In BibliographicDatetimeFormat.pdf v0.1.2-draft (direct download), as referenced in First Draft Uploaded. Does it relate to the ISO standards desirably? #2 my relevant interpretation of |EDTF 8601 Profile - Annex C in ISO/DIS 8601-2:2016(e)| is codified in the "Bibliographic ISO 8601 Profile (BP)" as ....

3.1.11 Interval

Interval rule. [OVERRIDEN in the Transformation, Interval]. A date of day, month or year precision, MAY start or end an interval

...
• Start and end dates MUST NOT be a: date-and-time; or season1.
....
DISAMBIGUATES with (|EDTF 8601 Profile - Annex C in ISO/DIS 8601-2:2016(e)|, 29, sec. C.4.3 Time interval).

1. Seasons and date-and-times MUST NOT be used in intervals ...

Either endpoint may be a year, year-month, or year-month-day

That is, I take the view that ISO forbids seasons in intervals and that by "year-month" ISO isn't so flexible, here, to extend "month" to include "seasons". That is, I take the view ISO doesn't allow something like 1982-21/2019-22.

However, the "Bibliographic Standard-Transformation", which yields the "Bibliographic Datetime Format", a second format set which operates beyond the shackles of ISO, stipulates an "Overriding interval rule" (3.2.7) which expressly permits seasons in intervals.

In this way I'm offering BibliographicDatetimeFormat.pdf as a way to solve this sort of problem and others like it.

But BibliographicDatetimeFormat.pdf, at the moment, remains unscrutinised by the community and to be relied upon it would require further development (minimally, it would need to be endorsed by a few implementors, like @inukshuk, after some discussion).

Edit: added "like".

[inukshuk]

Yes, this is correct. Also see my link to the EDTF mailing list above where it was confirmed that seasons are not defined for intervals in ISO. That mailing list might also be the best place (as far as I'm aware) to discuss with the EDTF community the merits of an extension beyond ISO.

As for myself, I am fairly indifferent as to the standardisation itself. I develop and maintain edtf.js for use in applications like Tropy or Zotero and will aim to support whichever standard is most useful to their respective userbase.

[JohnLukeBentley]

On seasons in intervals under (|EDTF 8601 Profile - Annex C in ISO/DIS 8601-2:2016(e)|) ... yes we are in agreement: it is forbidden. And it is good that Ray Denenberg, someone on the EDTF mailing list who seems to be intimate with the ISO process (it sounds like he might be an ISO member), is explicit that seasons in intervals are forbidden. Good, given that is consistent with (my reading of) the textual parts of the standard, which I've quoted above.

On the "Bibliographic Datetime Format", the main format set promoted in BibliographicDatetimeFormat.pdf (the other being "Bibliographic ISO 8601 Profile", which essentially tracks |EDTF 8601 Profile - Annex C in ISO/DIS 8601-2:2016(e)|), does your ...

I am fairly indifferent as to the standardisation itself. I develop and maintain edtf.js for use in applications like Tropy or Zotero [etc]

... represent a change in view from your earlier ...

... I'm happy to support the format which will be most relevant for users of Tropy, Zotero and CSL-aware tools in general. As I'm not too keen on standards proliferation, I still hope that the upcoming ISO standard will manage to incorporate all your requirements, but if that's not possible I'm happy to make the necessary adjustments to EDTF.js

?

My intention has always been to invite comment on BibliographicDatetimeFormat.pdf from folk on the EDTF mailing list. But I've so far taken the approach that the utility of such a standard should be proven, first, against specific software implementations, like bibliatex or, in the case of this thread, EDTF.js (or whatever it ought become).

So two other ways of asking the same question might be:

1. Given that on the [Code] Tab of https://github.com/inukshuk/edtf.js you support EDTF "with the following exceptions" .... might it be worthwhile to, instead, declare support for (some version of) Bibliographic Datetime Format (BDF); and join with me and others scrutinizing the sort of things, the exceptions to ISO's EDTF, that ought be in the Bibliographic Datetime Format? and/or

2. Given that EDTF.js is middleware perhaps you'll want to wait for demand to come from (at least) one of the ends that the middleware serves. That is, perhaps you'll want to wait to see, for example, what @plk has to say with respect to biblatex. If @plk wanted to support (some version of) BDF for biblatex, thereby establishing a demand for the format, perhaps that would be a prerequisite for you to be "happy to make the necessary adjustments to EDTF.js". ??

[inukshuk]

I wasn't trying to signal a change in my position, my apologies if that's what it sounded like.

I implemented EDTF because it is already being used in the wild (the upcoming ISO version should be close enough to make automatic conversions possible) and because the CSL community in particular seemed intent on adopting EDTF; if the community adopts a similar / different standard instead, I'm happy to support that here as well. For example, the edtf.js parser can already accept season intervals if you parse in the non-standard level 3 mode; I'm also happy to add support for similar features as long as edtf.js users benefit from them.

The exceptions listed in the README were all suggestions made by the EDTF community early on in the ISO standardization process; I was anticipating that they will be part of the final spec (if they are not, I will have to revisit those points).

By the way, for more about Ray Denenberg's role in EDTF/ISO see this recent posting on the mailing list.