juliadynamics / resumablefunctions.jl

C# style generators a.k.a. semi-coroutines for Julia.

ResumableFunctions

Documentation
Continuous integration
Code coverage
Static analysis with

C# has a convenient way to create iterators using the yield return statement. The package ResumableFunctions provides the same functionality for the Julia language by introducing the @resumable and the @yield macros. These macros can be used to replace the Task switching functions produce and consume which were deprecated in Julia v0.6. Channels are the preferred way for inter-task communication in julia v0.6+, but their performance is subpar for iterator applications. See the benchmarks section below.

Installation

ResumableFunctions is a registered package and can be installed by running:

using Pkg
Pkg.add("ResumableFunctions")

Example

using ResumableFunctions

@resumable function fibonacci(n::Int) :: Int
  a = 0
  b = 1
  for i in 1:n
    @yield a
    a, b = b, a+b
  end
end

for fib in fibonacci(10)
  println(fib)
end

Benchmarks

The following block is the result of running julia --project=. benchmark/benchmarks.jl on a Macbook Pro with following processor: Intel Core i9 2.4 GHz 8-Core. Julia version 1.5.3 was used.

Fibonacci with Int values:

Direct: 
  27.184 ns (0 allocations: 0 bytes)
ResumableFunctions: 
  27.503 ns (0 allocations: 0 bytes)
Channels csize=0: 
  2.438 ms (101 allocations: 3.08 KiB)
Channels csize=1: 
  2.546 ms (23 allocations: 1.88 KiB)
Channels csize=20: 
  138.681 μs (26 allocations: 2.36 KiB)
Channels csize=100: 
  35.071 μs (28 allocations: 3.95 KiB)
Task scheduling
  17.726 μs (86 allocations: 3.31 KiB)
Closure: 
  1.948 μs (82 allocations: 1.28 KiB)
Closure optimised: 
  25.910 ns (0 allocations: 0 bytes)
Closure statemachine: 
  28.048 ns (0 allocations: 0 bytes)
Iteration protocol: 
  41.143 ns (0 allocations: 0 bytes)

Fibonacci with BigInt values:

Direct: 
  5.747 μs (188 allocations: 4.39 KiB)
ResumableFunctions: 
  5.984 μs (191 allocations: 4.50 KiB)
Channels csize=0: 
  2.508 ms (306 allocations: 7.75 KiB)
Channels csize=1: 
  2.629 ms (306 allocations: 7.77 KiB)
Channels csize=20: 
  150.274 μs (309 allocations: 8.25 KiB)
Channels csize=100: 
  44.592 μs (311 allocations: 9.84 KiB)
Task scheduling
  24.802 μs (198 allocations: 6.61 KiB)
Closure: 
  7.064 μs (192 allocations: 4.47 KiB)
Closure optimised: 
  5.809 μs (190 allocations: 4.44 KiB)
Closure statemachine: 
  5.826 μs (190 allocations: 4.44 KiB)
Iteration protocol: 
  5.822 μs (190 allocations: 4.44 KiB)

Authors

• Ben Lauwens, Royal Military Academy, Brussels, Belgium.
• JuliaDynamics and QuantumSavory volunteers.

Contributing

• To discuss problems or feature requests, file an issue. For bugs, please include as much information as possible, including operating system, julia version, and version of MacroTools.
• To contribute, make a pull request. Contributions should include tests for any new features/bug fixes.

Release Notes

A detailed change log is kept.

Caveats

• In a try block only top level @yield statements are allowed.
• In a finally block a @yield statement is not allowed.
• An anonymous function can not contain a @yield statement.
• Many more restrictions.

resumablefunctions.jl's Issues

`@resumable` functions do not support `where` type specifications

This code

using ResumableFunctions

f(x) = 0.5*x*(x-1)

@resumable function foo(x::T) where {T}
    for i = 1:10
        @yield x
        x = f(x)
    end
    x
end

results in

ERROR: LoadError: UndefVarError: T not defined
Stacktrace:
 [1] include_from_node1(::String) at ./loading.jl:576
 [2] include(::String) at ./sysimg.jl:14
 [3] process_options(::Base.JLOptions) at ./client.jl:305
 [4] _start() at ./client.jl:371
while loading ../tests.jl, in expression starting on line 32

This issue is not blocking openjournals/joss-reviews#400 to be accepted, but at least it should be mentioned in the caveats. In the long term, supporting where type specifications would be a very useful feature to have.

Inference bug fix

@JuliaRegistrator register()

the README example

If I run the README example (v"0.6.2", v"0.1.10"), I get an error:

ERROR: MethodError: Cannot `convert` an object of type Void to an object of type Int64

This is not the case with the slightly different example here: https://benlauwens.github.io/ResumableFunctions.jl/stable/

The issue is the return value of nothing at the end of the iteration in the README. Is there some mechanism to only iterate over values explicitly yielded? (And not the final value returned by the function.) In my usage, it would simplify some algorithms.

Recursive parametric functions troubles?

EDIT. After some more fiddling, it appears that the culprit is actually the double definition of f.
Since I'm not sure if it's expected behaviour, I'll leave the issue open just in case.

---

Hi,

First, thanks for this fine package!
I'm trying to make the following code run:

using ResumableFunctions
using StaticArrays


@resumable function f(
    b::SVector{1,Int},
) ::SVector{1,Int} 
    
    @yield SVector{1,Int}(0)
    
end


@resumable function f(
    b::SVector{rk, Int},
):: SVector{rk,Int}  where {rk}
    
    for s in f(SVector{rk-1,Int}(b[1:end-1]))
        @yield vcat(s,SVector{1,Int}(1))
    end
end

for i in f(SVector{3,Int}(0,0,0)) println(i) end

but this yields the error

LoadError: MethodError: Cannot `convert` an object of type var"##256"{2} to an object of type var"##254"

Removing all type parameters makes the code compile. Is this a bug in the library or am I doing something wrong?

Thanks!

iterator interface semantics

I am testing the functionality of this repo and I stumbled in a silly usability issue. Not really an issue, but the behaviour could be documented better. Consider this code:

using ResumableFunctions

f(x) = 0.5*x*(x-1)

@resumable function foo(x)
    out = f(x)
    for i = 1:4
        @yield out
        out = f(out)
    end
    out
end

t = foo(0.6)

for el in t
    println(el)
end

for el in t
    println(el)
end

Only five numbers are printed in this code, because the second loop does not print any. Naively, I would have expected to have the same sequence printed twice, but this does not happen because the iterator state is hidden inside the resumable function definition and does not get reinitialised when the relevant start method is called.

This makes the semantics of the iterator interface slightly different than other iterators in Julia (e.g. zip([1, 2, 3], [4, 5, 6])). This is perfectly OK for openjournals/joss-reviews#400 to proceed, but it should be documented, e.g. in the Iterator Interface section of the docs.

a resumable function as a struct field var causes an error "struct expression not at top level"

Greetings,
A resumable function , when put inside a function, causes the error "struct expression not at top level" . because "@Resumable transforms the function block to a larger expression that includes a mutable struct block. struct blocks can only be defined in the global scope" . can putting a resumable function inside another function be allowed in the future?

Error tagging new release

The tag name "v0.3" is not of the appropriate SemVer form (vX.Y.Z).
cc: @BenLauwens

let blocks without variable assignement like `let i; f(i); end` are not supported

julia> function g()
           let j
               j = 1
           end
       end
g (generic function with 1 method)

julia> @resumable function f()
           let j
               j = 1
           end
       end
ERROR: type Symbol has no field args

`@yield` if used outside a `@resumable` should throw an error

First: thank you so much for this, I am so excited.
I've missed these for years.
In 0.5 I was using Tasks but they are unperforment and a bit weird.
In 0.6 that is deprecated.

Currently the (documented and actual) behavour of @yield outside a @resumable is to be converted into :(nothing).
I suggest it should throw an error.
You can throw errors during macro creation (not just return (:(throw(Error(....))),
but actually throw them.
It will cause an error at compile time, rather than runtime (which is great).

It is almost certainly a programmer error to use a @yield outside of an @resumable,
and as the programmer I want to know about that error ASAP.

(this would be a breaking change)

Your Bug or Mine?

Hi - as mentioned I am trying to benchmark ResumableFunctions against other iteration protocols. The following below is something that is a bit on the simplistic side but reasonably close to a recent use-case I had. The println statements are for debugging since I got strange results.

using ResumableFunctions
using BenchmarkTools

@resumable function g(N)::Vector{Int}
   for n = 1:N
      println()
      println(n)
      for n1 = 1:10, n2 = 1:10
         print("*")
         # do something not terribly expensive
         x = rand(-10:10, 3)
         @yield x
      end
   end
   Int[]
end

f_resumable(N) = sum( sum(x) for x in g(N) )

f_resumable(100)

On Python-generators-v0.6 I get

1
*

as output which just means that it terminates after the first @yield. Why?

When I use the release for v0.6, the I get

1
*ERROR: UndefVarError: #temp# not defined
Stacktrace:
 [1] macro expansion at /Users/ortner/.julia/v0.6/ResumableFunctions/src/transforms.jl:14 [inlined]
 [2] (::##662)(::Void) at /Users/ortner/.julia/v0.6/MacroTools/src/utils.jl:12
 [3] next at ./generator.jl:44 [inlined]
 [4] mapfoldl_impl(::Base.#identity, ::Base.#+, ::Int64, ::Base.Generator{##662,Base.#sum}, ::UInt8) at ./reduce.jl:42
 [5] mapfoldl(::Base.#identity, ::Function, ::Base.Generator{##662,Base.#sum}) at ./reduce.jl:73
 [6] f_resumable(::Int64) at ./REPL[4]:1

which is probably related.

(just in case this turns out to be a stupid bug at my end, let me just apologise profusely in advance . . .)

resumable should have SizeUnknown() so collect and comprehensions work

See docs for iteratorsize. Consider the following, which prints "second worked" for me.

using ResumableFunctions

@resumable function fibonnaci(n::Int) :: Int
  a = 0
  b = 1
  for i in 1:n-1
    @yield a
    a, b = b, a+b
  end
  a
end

a = fibonnaci(10)

c=try 
  b = fibonnaci(10)
  c=collect(b)
  println("first worked")
  c
catch
  Base.iteratorsize(::Type{typeof(a)}) = Base.SizeUnknown()
  b = fibonnaci(10)
  c=collect(b)
  println("second worked")
  c
end   

@show c

SizeUnknown() would be a strict improvement, but it seems like it would be possible to define length and size correctly for some iterators, such as the fibonnaci example, but it would require extra syntax.

High number of memory allocations.

Hi there!

I tried a basic segmented sieve of eratosthenes benchmark ( https://pastebin.com/WLmDZzh8 ), and noticed that the number of memory allocations made by of the version using @Resumable and @yield was very high, roughly a factor of ten more than the one pushing values into a channel, with more memory allocated than a naive non-segmented sieve.

Resumablefunctions is much, much faster than the channel version. However, the body of the generator in both cases allocates all its memory in the beginning and then only mutates that in place. I believe that the memory allocations from the channel versions are due to pushing the values one by one into the channel. I'm not sure where the allocations in the resumablefunction case come from.

A third version that I wrote as an explicit iterator(struct with state, mutated in place by next) with the exact same algorithm, apart from being 5 times faster (understandable), is completely nonallocating (unlike resumablefunctions, which did surprise me).

Does resumablefunctions reallocate its iterator struct on every yield? I'm having trouble maintaining an intuition for the memory cost of resumable functions.

I'm a huge fan of the library, by the way. Being able to write reasonably fast iterators without having to deal with the next() mess is a big timesaver and makes the code much more readable.

Resumable functions do not recognize types in same module

If we define a @resumable function in a module, using a type defined in the same module as an argument type, we get an error:

module testresumable

using SimJulia, ResumableFunctions

struct A
    x::Float64
end

@resumable function myprocess(sim::Simulation, a::A)
    @yield timeout(sim, 10)
end

end

This gives the error UndefVarError: A not defined. I am using

• ResumableFunctions 0.1.5
• MacroTools 0.4.0
• Julia 0.6.0
• Mac OS X

Broken link to stable documentation

The link to the stable docs pointed to by the badge in the README - https://benlauwens.github.io/ResumableFunctions.jl/stable - returns a 404.

Additions to documentation

Hi,

These are three suggestions for the docs, which were in my review.

1. Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.) The repo contains a benchmark folder with a benchmark file that compares the proposed approach with other "julian" approaches to obtain the same functionality. Maybe the author could mention the results of these benchmarks in the README.md file to justify the finite-state machine approach.

2. A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is? Yes. This is clearly spelled out in the main page. However, the author might wish to include the benchmarks in the documentation (or mention them), to show how this package solves the original problem (i.e. the slow performance of the Task-based approach).

3. Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems). Given the scope of this package, the examples provided in the repo are sufficient to demonstrate the functionality of this software. If the author is using this functionality in other public domain Julia packages (like he seems to be), these could be mentioned in the docs, so that users can then look at more involved examples.

I think these would be useful additions to the docs.

issue with generators of two variables

I'm trying out

- ResumableFunctions            0.1.10+            Python-generators-v0.6

And have found an unexpected behavior:

julia> @resumable function g(n)
       for i in 1:n
       @yield i
       end
       end

And

julia> [(a,b) for a in g(2), b in g(3)]
8-element Array{Tuple{Any,Int64},1}:
 (1, 1)      
 (2, 1)      
 (nothing, 2)
 (1, 2)      
 (2, 2)      
 (nothing, 3)
 (1, 3)      
 (2, 3) 

Whereas collecting first returns:

julia> [(a,b) for a in collect(g(2)), b in collect(g(3))]
2×3 Array{Tuple{Int64,Int64},2}:
 (1, 1)  (1, 2)  (1, 3)
 (2, 1)  (2, 2)  (2, 3)

Shouldn't this be the output of the former, or am I missing something?

macro clashing with ProgressLogging.jl

I'm running into some issues using this package alongside another macro-based package, ProgressLogging.jl

My goal is to add optional progress bars to a package which uses ResumableFunctions.jl, the original function definition looks straightforward, for example

@resumable function f1(x)
   for xi in x
       @yield xi
   end
end

which compiles and operates perfectly as expected.

The first attempt at progress logging, using the @progress macro fails already due to the rewriting:

@resumable function f2(x)
   @progress for xi in x
       @yield xi
   end
end

ERROR: LoadError: LoadError: @progress requires a for loop (for i in irange, j in jrange, ...; <body> end) or array comprehension with assignment (x = [<body> for i in irange, j in jrange, ...])

okay, that's fine, let's try the more flexible approach using @withprogress

@resumable function f2(x)
   @withprogress begin
        i = 1; N = length(x)
        for xi in x
           @yield xi
           @logprogress i/N
           i += 1
       end
   end
end

ERROR: syntax: cannot goto label "_STATE_1" inside try/catch block

so in both cases, something is clashing between the two macros. I tried looking into the source code, both here and the code produced by @macroexpand but it's so complicated I can't gleam any information from it.

version info:

julia> versioninfo()
Julia Version 1.6.0
Commit f9720dc2eb* (2021-03-24 12:55 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin20.3.0)
  CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_PKG_DEVDIR = /Users/miles/dev/julia
  JULIA_NUM_THREADS = 4
pkg> st --manifest
      Status `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_8WDmZ0/Manifest.toml`
  [1914dd2f] MacroTools v0.5.6
  [33c8b6b6] ProgressLogging v0.1.4
  [c5292f4c] ResumableFunctions v0.6.0
  [2a0f44e3] Base64
  [56ddb016] Logging
  [d6f4376e] Markdown
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [cf7118a7] UUIDs

CodeCov misses `@resumable` functions

CodeCov misses @resumable functions even if they have proper tests written.
Related PR: JuliaAstro/CCDReduction.jl#27

PS: I don't know whether I should have opened an issue here or in source repositories of CodeCov about this.

type CodeInfo has no field slottypes

@Resumable function fib_rf()
prev=0
cur=1
while(true)
@yield cur
prev, cur = (cur, prev+cur) # nice little avoiding a temp here
end
end

Running this simple chunk of code for Julia 1.0 results in Error:
"LoadError: type CodeInfo has no field slottypes"

CI, TagBot, Documenter fixes

There are a few secrets that need to be fixed up in the repo for these to work well again. Will happen when we move to a dedicated organization.

The CI/CD build is broken

I tried to contribute with a PullRequest but according to the CI/CD log the build is broken. The message, for your convenience, is the following:

Build started
[2](https://ci.appveyor.com/project/BenLauwens/resumablefunctions-jl/builds/46004858#L2)git clone -q https://github.com/BenLauwens/ResumableFunctions.jl.git C:\projects\resumablefunctions-jl
[3](https://ci.appveyor.com/project/BenLauwens/resumablefunctions-jl/builds/46004858#L3)git fetch -q origin +refs/pull/58/merge:
[4](https://ci.appveyor.com/project/BenLauwens/resumablefunctions-jl/builds/46004858#L4)git checkout -qf FETCH_HEAD
[5](https://ci.appveyor.com/project/BenLauwens/resumablefunctions-jl/builds/46004858#L5)The build phase is set to "MSBuild" mode (default), but no Visual Studio project or solution files were found in the root directory. If you are not building Visual Studio project switch build mode to "Script" and provide your custom build command.

Doesn't work in file I/O do-blocks

Using Julia 1.0.0 and ResumablePackages 0.4.1

I can get around it by simply doing non-block based I/O lines so it isn't a huge deal, but the below block of code gives the following error:

using ResumableFunctions

@resumable function f(file::String)
    open(file, "r") do io              # works if you instead use: io = open(file,"r")
        lines = readlines(io)
        for line in lines
            @yield line
        end
    end                                # and replace this with: close(io)
end;

for line in f("test.txt")
    println(line)
end

ERROR: LoadError: syntax: label "_STATE_1" referenced but not defined
Stacktrace:
[1] include at .\boot.jl:317 [inlined]
[2] include_relative(::Module, ::String) at .\loading.jl:1038
[3] include(::Module, ::String) at .\sysimg.jl:29
[4] exec_options(::Base.JLOptions) at .\client.jl:229
[5] _start() at .\client.jl:421
in expression starting at C:\...\file.jl:3

Issue with functions that take varargs

Hi,
I am having trouble using ResumableFunctions.jl to implement Interleave.
I have does so using Channels, and it works fine.
but ResumableFunctions.jl is just returning basically the iterators I feed my function, not there elements.
Am I doing something wrong?

Compare:

function interleave_ch(xs...)
    states = Base.start.(collect(xs))
    Channel(csize=256, ctype=Union{eltype.(xs)...}) do c       
        while true
            alldone = true
            for ii in eachindex(states)
                if !Base.done(xs[ii], states[ii])
                    alldone=false
                    val, states[ii] = next(xs[ii], states[ii])
                    put!(c, val)
                end
            end
            alldone && break
        end
    end
    
end

which has:
collect(Iterators.take(interleave_ch(100:300:900, 'a':'z'), 20))
returning:
Union{Char, Int64}[100, 'a', 400, 'b', 700, 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q']"

Vs:

@resumable function interleave_rf(xs...)
    states = Base.start.(collect(xs))
    while true
        alldone = true
        for ii in eachindex(states)
            if !Base.done(xs[ii], states[ii])
                alldone=false
                val, states[ii] = Base.next(xs[ii], states[ii])
                @yield val
            end
        end
        alldone && break
    end   
end

which has
collect(Iterators.take(interleave_rf(100:300:900, 'a':'z'), 20))
returning
Any[100:300:700, 'a':1:'z', nothing]

Which I just don't think is right.
I think the output should be the same.
The nothing comes from #2, so I am not worried about that.
But to me it looks like this function should perform interleave.
I may have messed something up though.

Caveat: when code is first executed

I have the following definition of a resumable function. I notice that if I call batchview with a d where isready(d) returns false, I do not get the desired error until I try to get the first element

@resumable function batchview(d::AbstractDiskDataProvider, size=d.batchsize)
    isready(d) || error("You can only create a buffered iterator after you have started reading elements into the buffer.")
    for inds in Iterators.partition(1:length(d), size)
        @yield buffered_batch(d, inds)
    end
end

example run

julia> isready(dataset)
false

julia> bw = batchview(dataset)
DiskDataProviders.var"##409"(0x00, #undef, QueueDiskDataProvider{Array{Float64,1},Int64,Array{Float32,2}}, length: 100
, #undef, #undef, 2, #undef, #undef)

julia> first(bw)
ERROR: You can only create a buffered iterator after you have started reading elements into the buffer.
Stacktrace:
 [1] (::DiskDataProviders.var"##409")(::Nothing) at /home/fredrikb/.julia/packages/ResumableFunctions/bdcq7/src/macro.jl:121
 [2] iterate at /home/fredrikb/.julia/packages/ResumableFunctions/bdcq7/src/macro.jl:86 [inlined]
 [3] iterate at /home/fredrikb/.julia/packages/ResumableFunctions/bdcq7/src/types.jl:24 [inlined]
 [4] first(::DiskDataProviders.var"##409") at ./abstractarray.jl:342
 [5] top-level scope at none:0

I now understand this behavior, but it caught me by surprise.
Is there a way to have some code run at the creation of the iterator rather than at the first call to iterate on it?

How to build fast iterators?

Not sure if this is even a bug or how to phrase the question in a useful way...

I'm experimenting with this package and it seems like the struct generated to hold the iterator state has all Any fields.
Is it this package's equivalent of the closure bug?

It seems like your benchmark is getting good performance, are there certain tricks to assure your function produces a fast iterator? Or is it just such a small example that it gets inlined entirely, removing the struct?

I did find that providing very explicit type hints on all variables do propagate to the struct, but wasn't able to figure out how to fix the iterator status and such.

I imagine it's also tricky to specify the element type and length of the resulting iterator, in cases where it is known. Making collect switch from the best case typed fixed length collect to the worst case untyped unknown length collect.

For example

using ResumableFunctions

bin(s, bins) = Iterators.map(((x, y),) -> (x, searchsortedfirst(bins, y)), s)

@resumable function hysteresis(s)
    bin = -1
    for (x, y) in s
        @yield if iseven(y)
            (x, (bin+1)÷2)
        else
            bin = y
            (x, (y+1)÷2)
        end
    end
end

tolbins(bins, tol=1e-9) = collect(Iterators.flatmap((x -> (x-tol,x+tol)), bins))

hybin(s, bins, tol=1e-9) = bin(s, tolbins(bins, tol)) |> hysteresis

Feeding that some iterator like s = zip(1:1000, rand(1000)):

julia> @time collect(s);
  0.000007 seconds (1 allocation: 15.750 KiB)

julia> @time bin(s, [0.5]) |> collect;
  0.000016 seconds (3 allocations: 15.859 KiB)

julia> @time hybin(s, [0.5]) |> collect;
  0.000391 seconds (8.50 k allocations: 295.484 KiB)

julia> fieldtypes(typeof(hysteresis([])))
(UInt8, Any, Any, Any, Any, Any, Any)

In the context of my actual code s isn't as simple, and I was seeing 1.5k allocations for collect and 24k allocations for hybin. So even in the case of a more complex iterator the resumable function performs not great.

Error `@yield macro outside a @resumable function!` when using @yield inside a macro

I wanted a macro that takes a sub-generator and yields its elements. I wrote this:

macro yieldmany(resumable::Expr)
      return quote
          for x in $(esc(resumable))
            @yield x
          end
      end
    end

However, when I use it in a function, I get an error:

@resumable function F2()
        @yield 2
        @yieldmany F1()
        @yield 4
     end
  ERROR: LoadError: LoadError: @yield macro outside a @resumable function!

I tried using macroexpand to see what it's making, but I get the same error:

@macroexpand @yieldmany F1()
ERROR: LoadError: @yield macro outside a @resumable function!

@macroexpand @yield x
ERROR: LoadError: @yield macro outside a @resumable function!

Recognizable names of iterator types

Hi, I'm enjoying this package quite a lot! I do have one issue though, and would like to see if there might be a solution for it.

The iterable types created by a call to a @resumable function have anonymous names, e.g.,

MyPackage.var"##403"(0x00, #undef,T}

It would be great if the name var"##403" at least contained the function name as specified here

@resumable function my_fun()
...

so that one could easier navigate stack traces etc.

Better yet, if it was possible to dispatch on the generated type, so I could do something like

other_function(iterator::MyResubableFunction) = ...

Any thoughts on if this would be possible?

LoadError on resumable function with extra keyword arguments

The following

@resumable function my_generator(; x = 0, kwargs...)
end

fails with

ERROR: LoadError: syntax: optional positional arguments must occur at end

When the extra keyword arguments are omitted, no error is raised:

@resumable function my_generator(; x = 0)
end

Julia 1.0.2, ResumableFunctions 0.4.2

is this package being maintained?

Tests not passing with 1.1

Related to the Mega-Issue JuliaLang/julia#30374.

test_for

@resumable function test_for(a::Int=0; b::Int=a+1) :: Int
  for i in 1:10
    @yield a
    a, b = b, a+b
  end
end
collect(test_for(4))

this fails on 1.1. However this works:

@resumable function test_for(a::Int=0; b::Int=a+1, n::Int=10) :: Int
  for i in 1:n
    @yield a
    a, b = b, a+b
  end
end
collect(test_for(4)) # works

An issue seems to be with the transform_for with the line $next = iterate... where it looks like the right-hand-side gives a nothing and the left-hand side expects a Tuple{Int, Int} (I don't know enough about macros to know what causes this assignment to fail). The following "fixes" it:

function transform_for(expr, ui8::BoxedUInt8)
  @capture(expr, for element_ in iterator_ body__ end) || return expr
  ui8.n += one(UInt8)
  next = Symbol("_iteratornext_", ui8.n)
  state = Symbol("_iterstate_", ui8.n)
  iterator_value = Symbol("_iterator_", ui8.n)
  quote
    $iterator_value = $iterator
    $next = iterate($iterator_value)
    while $next != nothing
      ($element, $state) = $next
      $(body...)
      tmp = iterate($iterator_value, $state) # <---
      tmp === nothing && break               # <---
      $next = tmp                            # <---
    end
  end
end

test_try

@resumable function test_try(io)
  try
    a = 1
    @yield a
    a = 2
    c = @yield a
    println(io,c)
  catch except
    println(io,except)
    d = @yield
    println(io,d)
  finally
    println(io,"Always")
  end
end

fails with error syntax: Attempt to jump into catch block. Not sure what's going on there.

Length of iterators

It would be convenient if was possible to supply the length of a resumable function if the length is known. A lot of functions around the ecosystem do not work if they can't ask for the length.
E.g.,

@resumable n myfun(n)
    for i = 1:n
        @yield i
    end
end

Warning: @resumable function contains return statement with value!

Why is this warning being given? Return values are used inside resumable functions to pass something when @yield is called on the function.

`@resumable` breaks blocks scope

Base Julia and ResumableFunctions.@resumable treat let blocks (and potentially others) differently

julia> function f()
           i = 1
           let j=i
               val = j
           end
           return val
       end
       f()
ERROR: UndefVarError: `val` not defined
Stacktrace:
 [1] f()
   @ Main ./REPL[114]:6
 [2] top-level scope
   @ REPL[114]:8

julia> @resumable function g()
           i = 1
           let j=i
               val = j
           end
           @yield val
       end
       collect(g())
1-element Vector{Any}:
 1

@yield returns nothing

In /paper/paper.md, there is the following statement:

"Straightforward two-way communication between the caller and the callable type is possible by calling the callable type with an extra argument. The value of this argument is passed to the left side of an arg = @yield ret expression."

However, using an expression of the form arg = @yield request(res) returns nothing. This should ideally return the Put object that is created when the request is made. Is there a way to access this Put object?

@yield all but last

Hi,

This package fills a great omission in Julia (I don't understand why things can't be just as simple as in python---I get the impression that things are better and more general in Julia with tasks and coroutines, and even better things that outdate these, but I can't figure out how to use them).

However, here @yield should produce all but the last element, and last element should be the return value of the resumable function.

That is not handy.

Having julia have base-1 indexing is already not handy for many purposes, but then having to deal with the last element separately make code even hairier.

It would be great if @yield would just return all elements.

`continue` results in infinite loops

The following hangs:

julia> using ResumableFunctions

julia> @resumable function f()
           for i in 1:10
               i % 2 == 0 && continue
               @yield i
           end
       end
f (generic function with 1 method)

julia> collect(f())

Adding a line to print out i shows that the loop goes through once with i = 1, then repeats the loop body forever with i = 2.

Not able to document `@resumable` functions

Doing this:

"""
    filenames(...)

docstring
"""
@resumable function filenames(...)
...
end

throws an error, error being:

cannot document the following expression:

#= /home/siddharth/.julia/dev/CCDReduction.jl/src/collection.jl:105 =# @resumable function filenames(df::DataFrame)

LoadError when yielding tuple with field name identical to local variable name

The following

@resumable function my_generator()
    x = 0
    @yield (x = 1, )
end

fails with

ERROR: LoadError: syntax: invalid named tuple field name "_fsmi.x"

No error is raised if the field name is changed:

@resumable function my_generator()
    x = 0
    @yield (y = 1, )
end

Julia 1.0.2, ResumableFunctions 0.4.2

Structs do not work as an annotated return type for a resumable function

Example:

struct Simple
    a :: Int64
end

@resumable function thisdoesnotwork(n :: Int64) :: Simple
    @yield Simple(n)
end

for a in thisdoesnotwork(10)
    println(a)
end

Causes the following error for me

ERROR: MethodError: Cannot `convert` an object of type Bool to an object of type Simple

not-nested for loop causes `UndefVarError: #temp# not defined`

on ResumableFunctions 0.1.10 and julia 0.6.2

using ResumableFunctions

@resumable function gen(n)
  for i=1:n, j=1:n
    @yield i
  end
end

collect(gen(10))

gives

ERROR: UndefVarError: #temp# not defined
Stacktrace:
 [1] (::##688)(::Void) at /home/ubuntu/.julia/v0.6/MacroTools/src/utils.jl:2
 [2] _collect(::UnitRange{Int64}, ::##688, ::Base.HasEltype, ::Base.SizeUnknown) at ./array.jl:442
 [3] collect(::##688) at ./array.jl:431

Release v1.0.0?

After v0.6.1 v0.6.2 is registered (#62), maybe it's time to make a v1.0.0 release too? The package seems stable (see #56) and mature (5+ years) enough, and has third-party packages that rely on it (see #60).

If done right after #62 is merged and registered (so that v1.0.0 would be the same as v0.6.1 v0.6.2 ), it would also have the nice side effect of getting the version numbers in sync with the Semicorutines.jl fork.

EDIT: v0.6.1 -> v0.6.2

odd issue with comprehensions

I ran into a surprise for me in trying something like this:

@resumable function test3()

    for u in [[(1,2),(3,4)], [(5,6),(7,8)]]
        for i in 1:2
            val = [a[i] for a in u]
            @yield val      
        end
    end

end

collect(test3()) # errors -- ERROR: MethodError: Cannot `convert` an object of type Int64 to an object of type Core.Box

Whereas writing out the loop as follows is fine:

@resumable function test2()
    
    for u in [[(1,2),(3,4)], [(5,6),(7,8)]]
        val = [a[1] for a in u]
        @yield val      
        val = [a[2] for a in u]
        @yield val      
    end

end
collect(test2())

I don't have insight why there is a distinction, and can work around in my usage, but thought it might be of interest to point out.

Broken on Julia 1.10 (fix included)

This package's inspection of method slots is broken on Julia 1.10 (after JuliaLang/julia#49113). Specifically, local slots are now not part of the code info anymore, so optimization would need to be disabled for this to work:

diff --git a/src/utils.jl b/src/utils.jl
index 9815847..486dbc4 100755
--- a/src/utils.jl
+++ b/src/utils.jl
@@ -40,7 +40,7 @@ function get_slots(func_def::Dict, args::Dict{Symbol, Any}, mod::Module) :: Dict
   func_def[:body] = postwalk(x->transform_nosave(x, nosaves), func_def[:body])
   func_expr = combinedef(func_def) |> flatten
   @eval(mod, @noinline $func_expr)
-  codeinfos = @eval(mod, code_typed($(func_def[:name])))
+  codeinfos = @eval(mod, code_typed($(func_def[:name]), Tuple; optimize=false))
   for codeinfo in codeinfos
     for (name, type) in collect(zip(codeinfo.first.slotnames, codeinfo.first.slottypes))
       name ∉ nosaves && (slots[name] = type)

It would be really great if this could be applied and put in a minor release, because currently packages that depend on ResumableFunctions.jl fail on 1.10 and thus cannot be tested by PkgEval anymore.

Recursive resumable functions vs PyGen vs Python generator

Hi Ben,

I will copy and paste the code from my comment in this link. It seems that recursive ResumableFunctions behave differently from PyGen and Python's generator functions in the recursive case, not sure about C#. This is shown in the comparison below:

ResumableFunctions:

a = [[1,2,3],[4,5,6]]
@resumable function g(x)
	if isa(x,Number)
		@yield x
	else
		for i in x
			for j in g(i)
				@yield j
			end
		end
	end
end

for i in g(a)
	println(i)
end

#=
1
false
2
false
3
false
nothing
4
false
5
false
6
false
nothing
nothing
=#

PyGen:

a = [[1,2,3],[4,5,6]]
@pygen function f(x)
	if isa(x,Number)
		yield(x)
	else
		for i in x
			for j in f(i)
				yield(j)
			end
		end
	end
end

for i in f(a)
	println(i)
end

#=
1
2
3
4
5
6
=#

Python:

import numbers
a = [[1,2,3],[4,5,6]]
def f(x):
	if isinstance(x, numbers.Number):
		yield x
	else:
		for i in x:
			for j in f(i):
				yield j

for i in f(a):
	print i

"""
1
2
3
4
5
6
"""

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Proposal: drop/ignore return value rather than yielding it

Hi there

I'm very excited about ResumableFunctions.jl. It is a wonderful demonstration of the power of julia's macro system.

On the other hand, I find one design choice to be puzzling: the choice to yield the final/return value of a resumable function as an additional value. I propose that instead, all such values should be @yielded explicitly, and any such return value should be dropped.

This would have a number of advantages:

• The comment and additional complexity regarding "default returns" in this example could be eliminated
• It will be easier to design resumable functions that are type stable.
• It will be possible for a resumable function to yield no values rather than always having to yield at least one
• This would be consistent with the notion that "Explicit is better than implicit" (or at least says The Zen of Python). It is also consistent with the way python generators work.

See also issue #13, which may be related.

Delegating to subgenerator (PEP 380)

In Python the yield from syntax described in PEP 380 allows a generator to delegate to a subgenerator, with the subgenerator then able to yield to the caller of the original generator. This is useful for factoring code out of more complex generator functions, and when working with recursive generator functions.

Would it be possible to support delegation like this in ResumableFunctions.jl?

Resumable function has stopped

What causes this error? I am trying to interrupt a process in SimJulia using SimJulia.interrupt

ERROR: @resumable function has stopped!
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] (::ProcessSim.var"##312")(::Nothing) at C:\Users\HD\.julia\packages\ResumableFunctions\n1hTu\src\macro.jl:88
 [3] execute(::SimJulia.Put, ::Process) at C:\Users\HD\.julia\packages\SimJulia\alkAr\src\processes.jl:32
 [4] (::SimJulia.var"#1#2"{typeof(SimJulia.execute),SimJulia.Put,Tuple{Process}})() at C:\Users\HD\.julia\packages\SimJulia\alkAr\src\base.jl:51 

Also, is there a way to know which resumable function stopped? I have nested processes.

Benchmarking versus Tasks/Channels in Base

I didn't trust the assertion in the README about the relative
performance of resumable functions and Tasks/Channels, so I ran some
simple benchmarks. I thought they might be a good idea to include in
the README or documentation:

julia> using BenchmarkTools, ResumableFunctions

julia> @resumable function resumable_fib(n::Int)::Int
         a = 0
         b = 1
         for i in 1:n
           @yield a
           a, b = b, a+b
         end
       end
resumable_fib (generic function with 1 method)

julia> function fib(c, n)
         a = 0
         b = 1
         for i in 1:n
           put!(c, a)
           a, b = b, a+b
         end end
fib (generic function with 1 method)

julia> function fib_wrapper(n)
         # This means we're doing equivalent work to resumable in constructing the iterator
         Channel(c -> fib(c, n); ctype=Int, csize=0)
       end
fib_wrapper (generic function with 1 method)

julia> struct FibN
         n::Int
       end

julia> function Base.iterate(f::FibN, state=(0,1,1))
         @inbounds last(state) === f.n && return nothing
         @inbounds state[2], (state[2], state[1] + state[2], state[3]+1)
       end

julia> sum(resumable_fib(1000))
9079565065540428012

julia> sum(fib_wrapper(1000))
9079565065540428012

julia> sum(FibN(1000))
9079565065540428012

julia> @benchmark sum(resumable_fib($10000))
BenchmarkTools.Trial: 
  memory estimate:  937.83 KiB
  allocs estimate:  30007
  --------------
  minimum time:     11.310 ms (0.00% GC)
  median time:      11.472 ms (0.00% GC)
  mean time:        11.970 ms (2.06% GC)
  maximum time:     63.081 ms (81.33% GC)
  --------------
  samples:          418
  evals/sample:     1

julia> @benchmark sum(fib_wrapper($10000))
BenchmarkTools.Trial: 
  memory estimate:  470.58 KiB
  allocs estimate:  30016
  --------------
  minimum time:     25.384 ms (0.00% GC)
  median time:      25.763 ms (0.00% GC)
  mean time:        26.636 ms (1.68% GC)
  maximum time:     86.521 ms (69.75% GC)
  --------------
  samples:          188
  evals/sample:     1

julia> @benchmark sum(FibN($10000))
BenchmarkTools.Trial: 
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     5.649 μs (0.00% GC)
  median time:      5.811 μs (0.00% GC)
  mean time:        5.953 μs (0.00% GC)
  maximum time:     16.782 μs (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     6

julia> r = resumable_fib(200000)
getfield(Main, Symbol("##364"))(0x00, 1, 102404, 200000, 1, 140027874456368:140027767251840, nothing, 0)

julia> c = fib_wrapper(200000)
Channel{Int64}(sz_max:0,sz_curr:1)

julia> f = FibN(200000)
FibN(200000)

julia> @time sum(r)
  0.235206 seconds (600.01 k allocations: 18.311 MiB, 1.58% gc time)
-7071068302198173471

julia> @time sum(c)
  0.587648 seconds (600.00 k allocations: 9.155 MiB, 9.07% gc time)
-7071068302198173471

julia> @time sum(f)
  0.000422 seconds (5 allocations: 176 bytes)
-7071068302198173471

In general, ResumableFunctions.jl seems to be about 2 times faster
than doing the equivalent using Channels. Interestingly, if I change
Channel csize to 1 instead of 0, the discrepancy is closer to 4-5
times than 2. And increasing csize to 10 adds several orders of
magnitude to the runtime.

Also, obviously, neither holds a candle to just using the vanilla
iteration interface with a custom type. Both are 3 orders of magnitude
slower than that option.

❯ julia --version
julia version 1.1.1

I'm curious. Has anyone looked into why Channels are so slow as
iterators and whether they can be improved to be at least on par with
ResumableFunctions?

Error tagging new release

The tag name "v0.5" is not of the appropriate SemVer form (vX.Y.Z).
cc: @BenLauwens