"Merge to source" for macros that only need Phase 3 about language

[scheglov]

@tatumizer @davidmorgan I don't think that composability is a serious issue. The order of macro applications is well specified, the user has control which macros are applied after which, and knows what each macro generates.

The JSON vs. HashCode macros example per se does not make sense to me, hashCode is usually a getter, not an actual field. But lets pretends that there is a generated field. Then we need a way to configure the JSON macro to exclude this field, we probably want it anyway to exclude even some user written fields.

@JsonSerializable(exclude: {'_hashCodeCache'})
@HashCode()
class A {
  final int foo;
  final int bar;
}

or

@JsonSerializable(excludeGeneratedBy: {HashCode})
@HashCode()
class A {
  final int foo;
  final int bar;
}

The other questions about JSON macro are separate from macro capabilities, and either require decisions what your macro wants to do, or a way to configure it. Whatever, you can write anything - as simple as your use case requires, or as complex as your customers demand, and you can handle. Not the macro specification problem.

@davidmorgan I have a feeling that we overcomplicate the constants evaluation problem. Allow any constants from dependency library cycles, literals, object instantiations with these, and this will cover everything that is practically necessary.

@davidmorgan About UX. I don't think that Java@Google guidelines have much weight for Dart. Java is know to require a lot of manual boilerplate, and if people wanted write in Java, they would continue to do so. For Dart we should aim for more convenience. And yes, IDE is absolutely necessary for modern development.

[jakemac53]

@mit-mit should probably comment. I don't believe this fits within the feature goals though, as it removes some of the primary reasons for wanting macros/augmentations in the first place (removal of boilerplate).

Especially in the cases where a macro is generating a large number of declarations which are just implementation details (for example freezed).

All macros can run in parallel, and it's always possible to analyze fully, without a possibility of changes

if we block constants from being augmented, otherwise if macros can evaluate constants and augment them, there are still ordering concerns.

If the macro declaration depends on distributed information, that can be a problem. You update bar.dart, the macro in foo.dart notices and requires new declarations.

Yeah this case definitely immediately came to mind. Build_runner allows a "build to source" (checked in) mode and also a "build to cache" (not checked in, re-built for all users) mode, for this reason.

Mockito comes to mind as a specific example that potentially might need to generate different code depending on non-local information (imagine a public member is added to a super type).

[tatumizer]

What is the difference between two scenarios:

1. The macro returns an "error", which forces the user to invoke IDE feature to generate the "missing declarations".

2. The macro creates a temporary copy of the source file containing all missing declarations and asks the user to confirm the changes (selectively and/or "Yes to all"). After that, the original source gets replaced by the generated "source".

It seems that the option 1 is just a more awkward version of the option 2. (Plus, IDE has to be able to decode the error message to figure out what the changes are and where they have to be applied)

In both scenarios, there's a problem with editing. (Maybe the generated fragments of code should be protected from editing? Maybe some checksums have to be computed to make sure the user hasn't edited the protected parts outside of IDE? Etc.)

[davidmorgan]

Thanks Jake! Sorry, I should have been clearer: I think it's only a good fit for some macros, so it would only be a v1 launch, with phase 1-2 to follow in v2.

Also we can think of designing so it's easy to switch between "merge to source" and writing to declaration files. built_value does this :) ... if you want to write the Builder type, you can, and that triggers a different codepath in the generator to the one where the generator hides the Builder. It's annoying to maintain, honestly, but it should be possible to support the general principle: the macro says the declarations it wants, and the host either merges to source or adds them as declarations.

The question then is whether it would be a useful way to split: a meaningful launch and a way to reduce risk. There's also a feature request for supporting "merge to source" in a nice way, so the macro can say what it wants and the host checks and issues an error. But that's a small feature we could build any time.

Yes, augmenting constants would be something to exclude from v1 if we're trying to make our lives easy :)

Re: difficult cases, I thought about Mockito; but actually with Mockito I think the declarations are already there, because you "implement" the target class? Would have to check. Anyway I'm sure there will be cases that don't work well.

@tatumizer Yes, exactly :) ... the way the built_value plugin worked was that it would report errors with fixes, exactly edits to the code that make the desired changes. The code wasn't maintained so it's deleted, but you can still see it in the repo history here.

[lrhn]

Still not sure I understand the desired behavior completely.

The macro will require that some member declaration to already exists. Is that all it does, or does it also do more?
If it doesn't find that declaration, then an error is reported, making the macro application fail, and the user is given the option of creating the member.

That just sounds like the macro reporting an error, which I assume it would be able to anyway, plus the user being given an easy quick-fix to fix the error.
I fully assume there will be macros which report errors if things are not how they assume. They shouldn't just add an augmentation to a method that doesn't exist, so reporting an error in that case is expected macro behavior, nothing new should be needed.

If we give package:macros the ability to report an error with a fix, or have specific kinds of errors for "missing declaration", which the analysis server can recognize and react to by suggesting creating missing declaration with provided signature,
then wouldn't that solve the issue without having a different kind of macro behavior?
Or package:macros could directly have a suggestNewDeclaration operation, which it can use instead of an error, but it'll likely end up failing macro application anyway, so combining it with errors seems reasonable.

Adding a member is only really possible while developing, which means while using the analyzer. The compilers won't be able to use the "merge-to-source" behavior for anything, all they can do is report the error on stderr. That again suggests that this is about adding a useful response to the macro error, not about the macro itself doing anything special.

[davidmorgan]

Yes, reporting errors for missing declarations is something most generators have to do today, and there are cases where macros have to do it too, or want to do it. The only new implementation here might be some convenience code like suggestNewDeclaration or better still requireDeclaration where the host checks, issues an error if there's a mismatch, and possibly offers a fix.

I've written tooling with generators that exactly goes and updates the files to make the generator happy, so that you get something better than the stderr output :) ... in those cases the error message would tell you to go and run the tool, much like updating a "golden" test.

The overall suggestion is not really about doing more, rather about doing less :) ... it's trivial to "support", the question is really twofold:

• does a phase 3 only macro feature make sense as a feature? How much (if at all) does it help us to split out and launch that?
• does a definitions only augmentation feature make sense as a feature? How much (if at all) does it help us to split out and launch that?

[rrousselGit]

Are you saying that a macro wouldn't be able to add new methods on a class but rather only implement them ; for the sake of simplifying the feature?

If so, that sounds incompatible with many highly popular code-generators.
For instance, generating a copyWith(...) would be extremely burdensome. And a @data wouldn't be able to remove the constructor+field duplication

Of course, I could be misunderstanding.

[jakemac53]

If the suggestion is to simplify the initial feature launch by only enabling definition macros (phase 3) initially, sure that could be reasonable. However, it wouldn't help very much with some of the highest priority macros we want to enable - such as a data class macro. The generated constructor would anyways have no body, so the macro couldn't provide any value there. It could help a bit with generating ==/hashCode/toString/copyWith still, although it would be a dangerous way to ship copyWith (its signature relies on dependencies potentially).

We would definitely still want/need to ship the other phases. But, it could help us by getting something concrete shipped, so that we can then work towards the phase 1/2 macros.

[davidmorgan]

@rrousselGit

That's right, definitions but not declarations; not for the sake of simplifying the feature, but for the sake of a v1 we can launch sooner :) ... with v2 to follow supporting declarations.

I agree it's disappointing if you wanted the macro to be able to output declarations; so I do think that adding declarations is probably something we should probably support.

This whole thing is an interesting tension in the design of generators, that I've given a lot of thought to over the years :)

See for example @AutoValue for Java. I mentioned generator best practices requiring "merge to source" for Java at Google, and @AutoValue is an example of following it to the letter: you have to write the whole interface of what will be generated, the generator only provides implementations. I was involved a bit in adding builder support and was pretty disappointed that you have to write the whole builder API--it's a lot. But it's very heavily used in spite of that.

To some extent it plays into "code is read a lot more times than it's written". In cases where it's more readable to just have the declarations immediately in the code, it's arguably worth the extra effort to put them there and not hidden in an augmentation.

In built_value I had that same choice, and decided to pass the choice to the user: you can choose whether or not to explicitly write the Builder declaration. People mostly prefer to omit it :)

When built_value had a plugin that could auto fix your code for you to add the declarations, I thought that was really neat. I do think it's a great experience for some macros. I don't claim it's a fit for all of them :) and for some macros a mix of "merge to source" and augmented declarations might be the best UX, or leaving it a user choice as I did for built_value.

@jakemac53

Yes, that's the suggestion.

It's curious you should mention data types, because the generator that preceded built_value started life as pure "merge to source": it did no compile-time generation at all, it just checked, and on error the user ran a tool merge all the suggested boilerplate changes (including implementations) into the manually maintained source. It was pretty well liked. It turns out the thing people hate most about boilerplate is maintaining it and having bugs in it ... if it's automatically checked/maintained then it's not terrible to just have it right there. Moving the definitions out and leaving the declarations would be even better.

The "check and require an update if needed" functionality applying to your data type constructor is maybe more valuable than you're thinking. Boilerplate that comes with guarantees is a big step up from boilerplate that you maintain by hand :)

But that curiosity aside, yes, I agree, there are certainly macros that want to emit declarations :) so I only suggest doing definition macros as a feature if it helps us get to declaration macros.

Thanks :)

[rrousselGit]

I think the community would be quite disappointed if macro launched, yet they couldn't use it (as their build_runner generators need phase 1/2) and we told them to wait longer.

[scheglov]

The types/declarations phases are already mostly implemented, I don't see why we should split them into v2.

I don't agree that separation between the user written library and the generated augmentation file is a big UX obstacle. This is just 1 + 1, not 1 + 100. Navigation between them in IntelliJ would be as easy as Cmd+E, or Cmd+Alt+ArrowLift/Right. OTOH, the limitation would significantly limit the usefulness.

[jakemac53]

Fwiw, macros which want to opt in to this model can always choose to do so - they can even pick and choose individual parts of the API that they think users should hand write, or not.

I hadn't previously considered macro authors choosing to force users to write certain things out, but I do think it is an interesting choice that some of them may opt in to. It would certainly be better if we also supported quick fixes for macro diagnostics, but we want that regardless.

[tatumizer]

Inserting some summary of the code to be generated by a macro into the source will be helpful IMO - even in the form of comments. Without it, anyone reading the source outside of IDE will be baffled. (This functionality might be controlled by a parameter of a macro, or something)

[tatumizer]

A much bigger problem is that the idea of "composability" of macros is not formalized, and can lead to unexpected behavior.

Consider an example. One macro (let's call it S) generates toJson/fromJson, another macro (H) generates hash code and equals.
During the declaration phase, macro S detects an extra field hashCode and has to decide whether to incorporate it in toJson/fromJson. Let's assume S has no knowledge of the semantics of hashCode - so there's no reason for S to ignore this field.

But the code for fromJson has to rely on the knowledge of the details of the object initialization. Should fromJson call a constructor? Or a Builder? Or use the cascade? If it chooses the call the constructor, then which constructor? Maybe S has to generate the constructor, too, which includes hashCode parameter? Or maybe upon seeing the constructor without the hashCode, it can decide that hashCode should be ignored?

I admit that I have no mental model of collective macro behavior. There's a sense that macros cannot be composed unless they are aware of each other. How much "merge-to-source" feature can help is an open question. Even if it can, the user writing the declarations should do it with full awareness of the logic of the macros involved. (Incidentally, the macro may not even know whether the declaration is entered by the user explicitly, or was generated by another macro, which makes the reasoning problematic anyway).

[davidmorgan]

@rrousselGit @scheglov Although we have end to end code working for phase 1+2, there are some large unanswered questions; @tatumizer mentions one of them with macro ordering; macro metadata is another, because const evaluation is not defined for phase 1+2; performance is another. There are also some open questions for augmentations.

So, I'm looking for anything we can split out and either finish first or finish and launch first to make progress. The more we can finish, the easier it will be to make progress on the remaining unanswered questions :)

I am not worried about disappointing users: a feature that we don't know how to launch, is not good for users either ;) we will keep working on this until either users are happy or we have proved it's impossible to make them happy ;)

Re: whether build runner generator owners would be happy with a "definitions only" launch, actually I think they will be plenty happy: they could start using augmentations for definitions, which would make the generators quite a lot nicer, and bring them much closer to being equivalent to the end goal macro feature.

I am not at all worried about the idea of taking smaller steps to launch macros, quite the reverse, I think it makes the whole feature likely to launch sooner.

@scheglov re: UX; there are different opinions here, as I mentioned the Java@Google guidance is that you have to merge the whole API, generators are only allowed to generate implementation. They believe that is "more usable" overall, because you don't need a tool to interact with the declarations. I don't agree fully :) which is why built_value has both modes. But, I do see their point. I also see your point, I think generating declarations and leveraging the IDE works really welll for some cases. So we don't want to limit to just definitions in the end :)

Thanks everyone.

[rrousselGit]

If we want to ship something early, I'd ship augmentation libraries. Not phase 3 macros.

A large chunk of code-generators wouldn't be able to use macros in a "phase 3 only" world.
The whole point of macros is to stop depending on build_runner, for developer experience. Yet with phase 3 only, many generators would have to keep using build_runner. In that case, the whole point of macros is moot.

That's different for augmentation libraries. They are a standalone feature, and there's value in using them without macros.

[davidmorgan]

@rrousselGit augmentation libraries are not ready to ship, either :)

The challenge with augmentation libraries is that it's not really an independent feature, it's 95% motivated by macros. But with macros not being complete yet, it's hard to know whether we have designed augmentation libraries correctly. There is a very real worry that if we ship augmentation libraries ahead of macros we'll find we made some decision that is not what macros want/need.

So if there is a way to do a part launch of augmentation libraries and a part launch of macros, that would give us a lot more confidence in completing both.

[tatumizer]

As soon as we internalize the fact that the macros generally won't compose (other than by chance), the design becomes 100x simpler :-)
The augmentations are still useful, they will allow to write clean code:

@freezed
class Person {
  const Person({
    required String firstName,
    required String lastName,
    required int age,
  });
}

Now, the class will have one main annotation (in this example - "freezed") which will be processed by a corresponding macro.
All other annotations (on a class level or field level) should be interpreted by the "freezed" macro as configuration parameters. The macro can call other generators ("freezed" does it today when calling the json_serializable generator), but the macro is in full control. In particular, it knows what each (nested) annotation means. Because now it's a 1-step process, there's no need to impose complicated restrictions on what augmentation is or isn't allowed to do. It may even fully replace the source code of the class (the limits are dictated only by the considerations of readability) (maybe this is the best option, resulting in a complete code of the generated class).

[tatumizer]

@scheglov : True, hashCode example was a contrived one, but you can imagine a field called gorgonzola in place of hashCode and have the same problem. If the combination of macros A and B has never been tested together, they are unlikely to work together.
What may happen is that in the end, one macro (like freezed) will gradually evolve to accumulate most of the (directly or indirectly) related functionality (configurable by more parameters) anyway, and there will be 2-3 more super-macros like this, but they will never be combined. This will make composability a non-goal: in every case, a single macro will essentially be responsible for the code generation of several interdependent features. This will make 3-phase design of augmentations unnecessary. That's my logic, but I can be wrong. (Just a gut feeling).

[scheglov]

@tatumizer Indeed, it will require the developer who adds macro application to understand what these macros do. This is the point where macros are "tested" together.

Sometimes you want to see the following macro to see for example fields of generated by the previous macro, sometimes you don't.

@JsonSerializable() // The author of "class A" wants these fields
@GenerateFields(namePrefix: "foo_", count: 10)
class A {}

@JsonSerializable(excludeGeneratedBy: {Memoize})
class A {
  @Memoize()
  int compute() => 42;
}

The author of the macro cannot know which fields should be serialized, it is up to the end user to decide. The macro author, if he is nice enough, will provide ways to configure the macro. Or says "no" to the users, who will either agree, or look for something more flexible.

On the macro tooling side we should support such ability to filtering capabilities. For example, the provenance of fields - ask only user written fields, or filter out generated by a specific macro.

It is OK if there are macros that are "thing-in-itself", it is the macro author decision to make, and the users to accept. But I don't think that this will preclude macros that can be usefully combined, and so I don't think that we should remove the ability to do this from the macro tooling.