ctongfei / ducttape Goto Github PK

Hi,

Currently, this will not work:

task print
  < in=$words[initial:A]
  > out {

  cat $input > $out
}

plan test {
  reach print
}

global {
  words=(initial:
    A=(end:
        L="/path/to/file/animal"
        E="/path/to/file/ample"
    )
    B=(end:
        L="/path/to/file/bell"
        E="/path/to/file/bumble"
    )
  )
}

with error message ERROR: Plan includes zero tasks.

The following, on the other hand will work:

task link
  < input=$words
  > out {

  ln -s $input $out
}

task print
  < input=$out@link[initial:A]
  > out {

  cat $input > $out
}

plan test {
  reach print
}

global {
  words=(initial:
    A=(end:
        L="/path/to/file/animal"
        E="/path/to/file/ample"
    )
    B=(end:
        L="/path/to/file/bell"
        E="/path/to/file/bumble"
    )
  )
}

Looks like something fishy is going on with branching grafting over variables, while grafting over task outputs seems to be working properly.

-sd

This shows up again at the current build.

-sd

Hi,

Currently, commenting after branches:

global {
  words=(initial:
    A=(end:
        L="/export/c11/shuoyangd/projects/ducttape-test/animal"  # a quick brown fox
        E="/export/c11/shuoyangd/projects/ducttape-test/ample"
    )
    B=(end:
        L="/export/c11/shuoyangd/projects/ducttape-test/bell"
        E="/export/c11/shuoyangd/projects/ducttape-test/bumble"
    )
  )
}

will make the parser fail, which is quite annoying as it would be nice to offer explanation of what case each branch is intended to account for.

-sd

Hi,

In the following workflow:

global {
  data=(DataSet:
    a="/path/to/file/1"
    b="/path/to/file/2"
    c="/path/to/file/3"
  )
}

task merge
  < in=$data[DataSet:*]
  > out {

  for datum in $in ; do
    cat $datum >> out
  done
}

plan test {
  reach merge via (DataSet: a b)
}

the output of merge will have the contents of all three files despite that fact that test only selects two branches to execute.

-sd

Currently, ducttape works largely depending on an assumption: the storage of the machine that submits the jobs could be directly mapped to the remote machine, which is not the case for a lot of commercial grids (e.g. amazon aws).

This is most obvious for the way currently versioner and submitter is run: the versioner installs the package first (locally), and then the submitter submits jobs to the grid. If there is no storage mapping between the local and grid machine, the environment where the packages are installed will have nothing to do with what the job will depend on.

For this specific example, flipping the order the versioner/submitter is run would do. However, there are more storage mapping that's happening than just mapping the packages -- one would possibly want to map the data as well. Fixing everything is tricky, but it's at least useful to realize that the problem is there.