We need a roadmap to 1.0.0 about camomile

[foretspaisibles]

Just a gentle ping on this! :)

[yoriyuki]

Cook it in the way as you like. For me, Camomile is dead. (Actually, OCaml is dead)

[foretspaisibles]

Wow, this is a statement! :)

Thank you a lot for the good work on Camomile!

Am I right thinking the various u* (ucorelib, etc.) libraries were meant to split Camomile in smaller libraries? Just out of curiosity, where are you looking at now? It looks like the OCaml community is thriving now, and definitely misses you! But anywhere you go, I wish you the best! :)

[youjinbou]

I don't know what makes you say that, but I feel that you're making a
mistake. Are you moving to a much more potent language? (I was thinking
Agda, Idris, ATS or even Coq). Anyway, I hope you'll find what you are
looking for.

On 10 August 2015 at 14:19, Yoriyuki Yamagata [email protected]
wrote:

Cook it in the way as you like. For me, Camomile is dead. (Actually, OCaml
is dead)

—
Reply to this email directly or view it on GitHub
#12 (comment).

[rgrinberg]

@yoriyuki sorry to hear about you and OCaml. I wish you the best.

Would you mind letting some users help out with maintenance on your projects then? For example, adding @michipili as a maintainer to Camomile? (Sorry for the nomination!)

[yoriyuki]

I added @michipili and you (@rgrinberg) to collaborators :-) If you need more setup (for example, creating a group), let me know.

[yoriyuki]

If other guys want to be collaborators, please let me know.

[foretspaisibles]

Thank you for doing this, @yoriyuki

[foretspaisibles]

@rgrinberg @tategakibunko I think it would be great to call the community to attention about the current state of Camomile and prepare together a roadmap for v1.0.0. I will prepare a backlog milestone and a few tickets to it, so that we can start discussing ideas.

If you (anybody) have any wish, please open tickets or start the discussion here. Thank you again @yoriyuki for the beautiful library you wrote!

[seliopou]

I'd like to help out as well. Could you add me as a collaborator?

[foretspaisibles]

Here is a short proposition for the next (pre 1.0.0) milestones:

• v0.9.0 Modern build system, CI, opam pinning
• v0.10.0 Remove dependencies, maybe rework interfaces, etc.

How does it sound?

[yoriyuki]

done. @seliopou

[yoriyuki]

What I thought for version 1.0.0 is, to support the recent Unicode version. The current Camomile only supports Unicode 3.2, which is ancient. A lot is changed from these days. As far as I know, to support the recent version, we need

1. Update data file. I made uproplib, which is an interface to data files of Unicode version 7.0 (which, in turn, is based on Daniel Bünzli 's uucd) for this goal.
2. Uppercase/Lowercase mapping. They added extra rules for ancient greek.
3. Local data. Camomile uses ICU data, but now Unicode consortium published LDML (Locale Data Markup Language) repository. I think the syntax of collation rules did not change, but they are now embedded in an XML format.
4. Unicode Collation algorithm. I did not check this yet. We need to look the change log of the technical report.
5. Charmap files. I hope the format did not change, but probably we need to update the contents.

Just for suggestions.

[seliopou]

@yoriyuki thanks, these are great suggestions. I was poking around the ICU and CLDR and did notice that Camomile is way behind on the Unicode standard. You'll be happy to know, however, that CLDR produced an LDML to ICU converter, and ICU continues to ship the converted ICU files. I'm not sure which features these converted files will cover, but hopefully it won't be necessary to deal with XML within Camomile. If you know otherwise, it would be great to know.

[yoriyuki]

Do we have any timetable to 1.0.0? I'm still interested in supporting the newest version of Unicode, but it takes time (so we may aim it for 2.0.0).

[rgrinberg]

Yoriyuki Yamagata <[email protected]> writes:

Do we have any timetable to 1.0.0? I'm still interested in supporting the newest version of Unicode, but it takes time (so we may aim it for 2.0.0).

I don't think we have any timetable - after all, this issue was made in 2015 :P That being said, I'm also very interested in supporting the latest unicode. We should start marking the issues that we definitely want to make it to 1.0 (build system, camlp4 removal, etc.)

[yoriyuki]

I post the issues which I think, are needed to solve to support the latest Unicode standard
#27 #28 #29 #30 #31

[yoriyuki]

Just for starting a discussion, I assigned several issues to the milestone v1.0.0

• I don't label jbuilder migration because we already are working on it.
• I don't label camlp4 removal neither, because we don't reach the consensus yet. We need actionable items for the milestone.

[yoriyuki]

I want to start working for the latest Unicode standard. But how we proceed? I will make a new branch for this purpose. Updating the latest standard would brake the library, even make it not compilable. Then, the final result would be completely different from the current one. If you change the main branch during this process, the change is difficult to port. What is the best practice for this kind of cases?

Another issue is: Should we create a branch for 1.0 or continue to develop on the main branch until 1.0 release?

I appreciate your input.

[rgrinberg]

Yoriyuki Yamagata <[email protected]> writes:

I want to start working for the latest Unicode standard. But how we proceed? I will make a new branch for this purpose. Updating the latest standard would brake the library, even make it not compilable. Then, the final result would be completely different from the current one. If you change the main branch during this process, the change is difficult to port. What is the best practice for this kind of cases? Another issue is: Should we create a branch for 1.0 or continue to develop on the main branch until 1.0 release? I appreciate your input.

For 1.0, I think our best course of action is the conservative one. Even though Camomile never made to 1.0, it is de-facto the most popular unicode library in OCaml, and is used by many other libraries and applications. IMO, that's a good reason to basically freeze it's interface and let current users follow semver properly. Therefore we should freeze the current API until we release 1.0. If we're introducing breaking changes, we should at the very least inform all the downstream users we are breaking before release 1.0

Another issue is: Should we create a branch for 1.0 or continue to develop on the main branch until 1.0 release?

This is your call, but I'd prefer if master stayed stable and w/e API changes you had in mind were in a different branch with a PR open so that we could review them.

If you change the main branch during this process, the change is difficult to port. What is the best practice for this kind of cases?

Yes, this is quite a tough issue. One thing to note that once your new API is in development, we're seldom going to make big changes to master. After all, the api of 1.0 should stay frozen. We can of course help with the rebase/conflict resolution work here as well.

[yoriyuki]

I did three things:

1. Make the master branch is protected. To change the master, you need to create a pull request and let someone review it.
2. Create v1.0 branch. This branch is an integration branch for version 1.0 and also protected.
3. Create Unicode10.0.0 branch. I will work for support of Unicode 10.0.0 using this branch. The branch is not protected.

[rgrinberg]

Yoriyuki Yamagata <[email protected]> writes:

I did three things: 1. Make the master branch is protected. To change the master, you need to create a pull request and let someone review it. 2. Create v1.0 branch. This branch is an integration branch for version 1.0 and also protected. 3. Create Unicode10.0.0 branch. I will work for support of Unicode 10.0.0 using this branch. The branch is not protected.

Agreed with this approach. But I'm curious, why do we need a v1.0 integration branch for now? Seems like such a branch will be useful once we have released 1.0, and are already working on your new branch in master. In such a situation the integration branch would be useful to make emergency bug fixes to 1.0. Or do you have another workflow in mind?

[yoriyuki]

But I'm curious, why do we need a v1.0
integration branch for now? Seems like such a branch will be useful once
we have released 1.0, and are already working on your new branch in
master. In such a situation the integration branch would be useful to
make emergency bug fixes to 1.0. Or do you have another workflow in
mind?

Although I want API change minimal in 1.0 but there will be API change. If we develop 1.0 using the master, people using the master would be surprised. In particular, we do not have an backward compatible branch for v0.x.

My plan is once 1.0 is released, v1.0 is merged to the master and development of 1.x is continued on the master branch. In the same time, we make the integration branch v2.0 and start developing 2.0.

Sure, it is not the Git workflow but I feel reasonable. What do you think?

[rgrinberg]

Ah ok. Makes sense to me. Ok so we'll only be making fully backwards compatible changes to master. All api breaking changes for 1.0 will be kept in a branch.

Yeah, I think that's reasonable.

[yoriyuki]

I made some progress on #28

[yoriyuki]

Another issue, which I always am worried about, is that Camomile is too large. Now UCD part taking almost 30Mbytes, we may need to do something for this. I'm always thinking about splitting Camomile into different packages so that a user can choose the level of functionalities.

For example,

1. camomile-basic: basic data types like UTF-8 encoded strings, IO, some basic character encodings. Maybe monads and transducers etc. for convenience.
2. camomile-algorithm: Unicode related algorithms like case mapping, collation, line breaking, etc.
3. camomile-encodings: more esoteric character encodings.

Of course, there is another issue when we do it. Version 1.0 is a good occasion but it delays the release of version 1.0 further.

Might be a good idea to announce the existence of v1.0 branch.

[yoriyuki]

Anyways I made a group https://github.com/ocaml-camomile and 4 empty repositories.

[yoriyuki]

Started working on camomile-basic https://github.com/ocaml-camomile/camomile-basic It is already able to built.

[rgrinberg]

Yoriyuki Yamagata <[email protected]> writes: I agree with the idea of making Camomile more modular, but I disagree with the idea that multiple git repositories are the way to go about this. A few of the disadvantages that an explosion of git repositories brings to the table: * There are now 4 bug trackers to keep track of. For developers, this is busy work that we don't really notice. But it's really confusing for users who search for existing bugs or trying to report new ones. * Increases CI complexity. Now we need 4x the CI, and we also need to coordinate CI for different permutations of versions. * Making cross cutting changes is now much harder. Now we have to maintain 4x the branches, put in 4x the commits, and we want to revert a change we have to do it 4 times. This is really defeating one of the main advantages of using git and having a single git hash to represent the state of the repository. * We must now maintain and test different version constraints for opam. Rather than releasing everything at once where we guarantee that packages work against each other. This is quite a lot of extra overhead for only 2 developers... We can make Camomile just as modular without creating new git repos. Instead, we can split Camomile into multiple opam packages that exist in the same repository. This will give the user the same benefits you've mentioned. They'll pay for only what they use and be able to express their dependencies more accurately. I maintain many packages this way and (cohttp, conduit, etc.) and this is a far better approach in my opinion. I can help with the initial organization if you're not quite sure how to do it. Though it's quite simple, mostly just 1 directory per package and keep all the *.opam files at the root of the repo. As for 1.0, I think that perhaps we should just release master now as 1.0 (unless you have some last second improvements) and start making the breaking changes we're planning post 1.0. As we discussed before, the current version of Camomile is already in widespread use, so it's very likely we'll need to make bug fix point releases to it. Might as well give a stable version number to an already de-facto stable API.

[yoriyuki]

I see. Then we use the same repo. as now but split the opam package.

Since we have many tasks, we need to prioritize tasks.

1. First prepare 1.0 quickly. I think we can merge your no dynamic configuration PR and update the contact address. (many files indicates the sourceforge address now.) Just one search/replace. I also want to unify Camomile's UChar and OCaml stlib's Uchar.
2. Next, create v2.0 branch and split the package. Since this is a huge API change I think it is better to put them v2.0. Could you do an initial set-up? Then, I want to remove imperative Unicode strings (xString and ustring).
3. Finally, support the recent Unicode standard for v3.0. As this is again a large change of behavior and APIs, it is better to make another major update.

[rgrinberg]

I think we can merge your no dynamic configuration PR and update the contact address

Sure. Also, saving it for post 1.0 is an option (up to you).

I also want to unify Camomile's UChar and OCaml stlib's Uchar.

I actually gave this a try already. This is also a breaking change unfortunately as stdlib's Uchar.t is more restrictive and doesn't allow any integer. From the docs:

(** The type for Unicode characters.

    A value of this type represents an Unicode
    {{:http://unicode.org/glossary/#unicode_scalar_value}scalar
    value} which is an integer in the ranges [0x0000]...[0xD7FF] or
    [0xE000]...[0x10FFFF]. *)

Camomile's Uchar is more permissive.

To clarify the branching situation:

• master is where we do development of the upcoming version (1.0 as of now, 2.0 later, etc.)

• After a release, we have a version specific where we backport bug fixes/improvements whenever possible.

[rgrinberg]

One more change I had in mind for 2.0 is the following use the Camomile module name for the library itself. Rather than using the more verbose CamomileLibrary. The Library suffix isn't really helpful.

[yoriyuki]

I think we can merge your no dynamic configuration PR and update the contact address

Sure. Also, saving it for post 1.0 is an option (up to you).

Done. It is inconsistent to what I said but doesn't matter much.

I also want to unify Camomile's UChar and OCaml stlib's Uchar.

I actually gave this a try already. This is also a breaking change unfortunately as stdlib's Uchar.t is more restrictive and doesn't allow any integer. From the docs:

(** The type for Unicode characters.

    A value of this type represents an Unicode
    {{:http://unicode.org/glossary/#unicode_scalar_value}scalar
    value} which is an integer in the ranges [0x0000]...[0xD7FF] or
    [0xE000]...[0x10FFFF]. *)
Camomile's Uchar is more permissive.

This restriction was implicit in Unicode Standard, and now it is explicit. So it's okay to (actually we should) restrict Unicode to this range. In particular, UChar with [0xD800] - [0xDFFF] code range brakes many algorithms.

To clarify the branching situation:
master is where we do development of the upcoming version (1.0 as of now, 2.0 later, etc.)
After a release, we have a version specific where we backport bug fixes/improvements whenever possible.

Sure. We need to do something about existing v1.0 branch but it is easy to deal with.

One more change I had in mind for 2.0 is the following use the Camomile module name for the library itself. Rather than using the more verbose CamomileLibrary. The Library suffix isn't really helpful.

Sure. I think v2.0 is a good opportunity to rearrange the module structure.

[yoriyuki]

I found migrating stdlib Uchar is not easy, because unidata contains non-scalar (out-of-range) code points. To fix this, we need to modify the large portion of the code. I think v3.0 when we support Unicode 10 is better opportunity.

So, I think now we are ready to release v1.0. Do you have anything left to do?

[yoriyuki]

Now we have 1.0.1. Continue to discuss on 2.0.0 at #70