Distributed.jl - possibility to use other Serialization libraries? about distributed.jl

[jpsamaroo]

Malt shouldn't be relevant here since this issue is relating to running code under Distributed directly (IIRC Pluto only runs its own internal code under Malt)

[jpsamaroo]

ClusterSerializer could have a hook for customizing name binding lookups. I'm not sure what such a design would look like in the face of multiple libraries wanting to hook this functionality. Probably a hook could be defined per-Module, so that this hook will only be called on modules that Pluto creates.

[schlichtanders]

Looking into which reasoning was taken for the global random number generator it seems that the preferred route is to make the ClusterSerializer pluggable. JuliaLang/julia#23199 (comment)

These are the two places where the ClusterSerializer is constructed

• Distributed.jl/src/cluster.jl

  Line 122 in fdf56f4

  w.w_serializer = ClusterSerializer(w.w_stream)

• Distributed.jl/src/process_messages.jl

  Line 159 in fdf56f4

  serializer = ClusterSerializer(r_stream)

So I guess we would need:

• an AbstractClusterSerializer type for clarity
• and make the above two places (and respective wrappers around it) somehow being able to pass an AbstractClusterSerializer object (similar to how rand and other methods pass a RNG object)

[schlichtanders]

@MichaelHatherly you build PlutoSerialization.jl, probably you have some valuable insights what could be potential solutions/problems

[MichaelHatherly]

Thanks for pointing this out @schlichtanders. I've got no extra insights at the moment since I've not had a chance yet to look into how Malt changes things now that Pluto is using it instead. If I get a chance I'll update this thread.