dyalog / isolate

Source to build isolate.dws, the futures and isolates workspace deliviered with Dyalog APL

isolate.dws and samples for Dyalog v17.0

Since version 14.0, Dyalog APL has included the workspace isolate.dws, which enables simple asynchronous and parallel programming in Dyalog APL. From version 17.0, the source for the isolate workspace and associated samples have been moved to GitHub.

Documentation

Documentation for the futures and isolates can be found in the Dyalog Documentation Centre.

Version 17.0 Enhancements

A project was started at Dyalog with the goal of enhancing the isolate workspace so that it would be possible to run isolate servers in the cloud. In the end, it turned out that a couple of bugfixes in the code which validated peer IP addresses to deal with IPv6 addresses was all that was really required.

We also added a "sample" called AWS, which provides an interface to the Amazon Webservices Command Line Interface. This can be used to launch and manage AWS instances. Dyalog Webinar 10 demonstrates how to use this to run a large number of parallel isolates on the cloud.

Samples

In addition to the workspace which can be found in the ws folder along with most other distributed workspaces, a new folder of isolate-related samples are now installed in the folder Samples/isolate below the main Dyalog folder.

It contains the AWS class and an example of how to use it, and also includes the IIPageStats sample, which computes letter frequencies used on all major newspaper sites in a given state in the USA, as an example of how to use the ll.EachX (extended parallel each) tool:

Contents of the Samples folder

File	Type	Description
AWS.dyalog	Class	Interface to the Amazon Webservices Command Line Interface
AWSIsolates.dyalog	Function	Shows how to use the AWS class to start a set of virtual machines and use them to run Isolates
IIPageStats.dyalog	Namespace	Demonstrates the use of ll.EachX tool

isolate's Issues

homeport set incorrectly

ynys.Init should not set homeport to 7051, it should set it to op.homeport (which defaults to 7051, but can be configured by the user)

The error from pointing isolate.Config'runtime' to an invalid interepter is not helpful

Version: 17.1.37073.0 64 Unicode
Created: Oct 8 2019 at 20:16:06
Build ID: 939652dd

  isolate.Config'runtime' 'junk'
  z←ø''

VALUE ERROR
z←ø''
∧

  isolate.Version

Version 1.2 (2019-08-09 09:30:58 +0100) built at 2019-08-15 09:10:58

How about "Unable to find interpreter junk"

isolate only works if isolate is copied into #

To reproduce:

 )ns ns1
 )cs ns1
 )copy isolate isolate
 is←isolate.New''
 is.(2+2)

FUTURE ERROR: 86: COMMUNICATIONS FAILURE 1119 ERR_CLOSED : Socket closed whilst receiving data

isolate should use Conga namespace and not DRC namespace

It's possible that one could use isolates in an application that also has other Conga-using components. To play nicely with others, isolate should create its own root using the Conga namespace.

RANK ERROR in isolate.llEach when ⍵ is not a vector

If the right argument has 2 or more dimensions, isolate.llEach / IÏ fail with a RANK ERROR.

For example:

⎕DL IÏ 2 2⍴⍳4

Update Build to use [DYALOG]/Library/Core/APLProcess

isolate should not maintain its own copy of APLProcess, but rather use the one supplied with the interpreter

isolate initialisation should not fail if isolate ws can be found via WSPATH

Isolate initialisation fails if there is no isolate.dws in the "default" location, even if it can be found using WSPATH. Logged as Mantis 19629 (Link.Test fails on developer machine).

You cannot )COPY isolate into a namespace

If you )COPY the isolate workspace into any other location than the root, "nothing works":

  'ns' ⎕ns ''
  )cs ns

#.ns
)copy isolate
... saved ...
iso←isolate.New ''
iso.⎕dl 3
FUTURE ERROR: 86: COMMUNICATIONS FAILURE 1119 ERR_CLOSED ... etc ...

Allow ll.Each to use primitive left operands

⍳ ll.Each 3 4 5
ought to work.

Does the primitive functions' list need updating

⍸ and ⊆ seem to be missing from https://github.com/Dyalog/isolate/blob/master/Source/ynys/proxySpace.dyalog#L60

Monadic II does not return a future

Due to issues in the interpreter (mantis 11230, 18104), monadic II blocks on the result.

StartServer and AddServer assumes machines have proper DNS entries

StartServer fails to start if your hostname can't be resolved (common on a private network). This can be remedied via suitable entries in /etc/hosts.

getDRC should use the Conga namespace rather than DRC

isolates should use the "new" namespace and a separate root by default

It is almost possible to use a separate root for isolates by

isodrc←Conga.New 'isolates'
isolate.Config 'drc' isodrc

... unfortunately you also need to modify isolate.ynys.Init to read as follo

[14] :If 9.2≠⎕NC⊂'here.DRC'
[15] :AndIf ~(⊃z←DRC.Init ⍬)∊0
[16] ('CONGA INIT FAILED: ',,⍕z)⎕SIGNAL 11
[17]           :EndIf

By version 19.0, we should change the getDRC function to use a separate root by default, and be able to use one if configured as above.

[MacOS] Parallel each: Unable to initialise isolate processes

This fails on both 18.2 and 19.0:

      ⎕cy'isolate'
      
      T←⎕AI ⋄ ⎕DL IÏ 8⍴3 ⋄ ⎕AI-T
ISOLATE: Unable to connect to started processes (attempt 1 of 3)
ISOLATE: Unable to connect to started processes (attempt 2 of 3)
ISOLATE: Unable to connect to started processes (attempt 3 of 3)
ISOLATE: Unable to initialise isolate processes
      T←⎕AI ⋄ ⎕DL IÏ 8⍴3 ⋄ ⎕AI-T
              ∧

 Dyalog  18.2.47086 64-bit Unicode, BuildID 5e2cc913                           
 OS      Darwin 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:58 PST 20
 Link    3.0.19                                                                
 SALT    2.9                                                                   
 UCMD    2.51                                                                  
 .NET    .NET 6.0.16                                                           
 WS      18.2 

Improve performance of isolate.Reset

isolate.Reset takes an unnecessarily long time to run. Morten has a fix which needs to be tested and pushed.

IÏ fails to enclose its result.

      ≢⍴,II¨ 1
0
      ≢⍴,IÏ 1
1
      ≢⍴,¨1
0

Dyadic II ignores left argument

Dyadic II makes monadic application of the operand function.

      1+#.II 0J1
0J¯1

Improve validation and error message when configured workspace is not found on isolate.New

Trying Nic's Link Test and I've not got a standard installation. I get

WS NOT FOUND
Setup[16] #.SLAVE←#.isolate.New''

1. What workspace ?
2. Why SHOUT AT ME ?

I think this comes from Source/ynys/checkWs.dyalog, last updated 2018
I see that if conga fails to initialise the errors again are all in uppercase.

myIsolates .(2 ) FUTURE ERROR: 86: COMMUNICATIONS FAILURE 1119 ERR_CLOSED : Socket Closed while receiving

Any idea what might be causing this?

FUTURE ERROR: 86: COMMUNICATIONS FAILURE 1119 ERR_CLOSED : Socket Closed while receiving: iEvaluate[13] (,⍕(⍕rc),': ',(0⊃res),{(⍵∨.≠' ')/': ',⍵}1⊃res,'' '')iSpace.qsignal rc

runTests←{
⍺←0
⎕THIS.myIsolates←⍺{
~⍺:0
ok←⎕CY'isolate'
{isolate.New''}¨⍳4
}''

}

 myIsolates .(2 )

FUTURE ERROR: 86: COMMUNICATIONS FAILURE 1119 ERR_CLOSED : Socket Closed while receiving: iEvaluate[13] (,⍕(⍕rc),': ',(0⊃res),{(⍵∨.≠' ')/': ',⍵}1⊃res,'' '')iSpace.qsignal rc
myIsolates.(2)

Support encryption of isolate connections

Need to enable encryption of isolate connections, and optionally also extend server side filtering to verify client-side certificates.