hamidb80 / iterrr

iterate faster ... 🏎️. functional-style, extensible iterator library

Once a young man approached the Prophet Muhammad and asked him for some advice. The Prophet replied, “Do not become angry,” and he repeated this three times

iterrr!

iterate faster ... 🏎️. Write higher-order functions, get its imperative style at the compile time!

The Problem

Writing a full nested loop is a boring task, std/sequtils creates lots of temporary seqs, iterutils uses closure iterators (which is slow).

The Solution

iterrr uses the ultimate power of meta-programming to bring you the speed that Nim programmers deserve.

example of generated code

"hello".pairs |> 
  filter((i, _) => i  > 1)
  .map((_, ch) => ch)
  .strjoin() ## llo

block:
  template iterrrFn3(_; ch): untyped {.dirty.} =
    ch

  template iterrrFn4(i; _): untyped {.dirty.} =
    i > 1

  var iterrrAcc2 = strjoinInit[typeof(iterrrFn3(
      default(typeof("hello".pairs))[0], default(typeof("hello".pairs))[1]))]()
  block mainLoop:
    for li1 in "hello".pairs:
      if iterrrFn4(li1[0], li1[1]):
        block:
          let li1 = iterrrFn3(li1[0], li1[1])
          if not strjoinUpdate(iterrrAcc2, li1):
            break mainLoop
  strjoinFinalizer(iterrrAcc2)

Usage

complete syntax

There is 3 type of usage:

# predefined reducer
iterable |> entity1(_).entity2(_)...Reducer()

# custom reducer
iterable |> entity1(_).entity2(_)...reduce(loopIdents, accIdent = initial_value, [Finalizer]):
  # update accIdent

# custom code
iterable |> entity1(_).entity2(_)...each(...loopIdents):
  # do with loopIdents

Main Entities:

1. map :: similar to mapIt from std/sequtils
2. filter :: similar to filterIt from std/sequtils
3. breakif :: similar to takeWhile in functional programming languages but negative.
4. inject :: injects custom code

1. predefined reducer

NOTE: you can chain as many map/filter/... as you want in any order, but there is only one reducer.

There are some predefined reducers in iterrr library:

• toSeq :: stores elements into a seq
• count :: counts elements
• sum :: calculates summation
• min :: calculates minimum
• max :: calculates maximum
• first :: returns the first item
• last :: returns the last item
• any :: similar to any from std/sequtils
• all :: similar to all from std/sequtils
• toHashSet :: stores elements into a HashSet
• strJoin :: similar to join from std/strutils
• toCountTable :: similar to toCountTable from std/tables

here's how you can get maximum x, when flatPoints is: [x0, y0, x1, y1, x2, y2, ...]

let xmax = flatPoints.pairs |> filter(it[0] mod 2 == 0).map(it[1]).max()
# or
let xmax = countup(0, flatPoints.high, 2) |> map(flatPoints[it]).max()

NOTE: see more examples in tests/test.nim

Custom Idents ?!?

using just it in mapIt and filterIt is just ... and makes code a little unreadable.

remember these principles when using custom ident:

1. if there was no custom idents, it is assumed
2. if there was only 1 custom ident, the custom ident is replaced with it
3. if there was more than 1 custom idents, it is unpacked

Here's some examples:

(1..10) |> map( _ ) # "it" is available inside the "map"
(1..10) |> map(n => _ )
(1..10) |> map((n) => _ )
(1..10) |> map((a1, a2, ...) => _ )
(1..10) |> reduce((a1, a2, ...), acc = 2)
(1..10) |> each(a1, a2)

Custom idents work with both op: and op() style syntax:

(1..10).items.iterrr:
  map: n => _
  # or map n => _
  ...
(1..10).items.iterrr:
  filter: (n, k) => _
  # or filter (n, k) => _
  ...

example:

"hello".pairs |> filter((i, c) => i > 2).map((_, c) => ord c)

Limitation

you have to specify the iterator for seq and other iterable objects [HSlice is an exception]

example:

let s = [1, 2, 3]
echo s |> map($it).toseq() # doesn't work
echo s.items |> map($it).toseq() # works fine
echo s.pairs |> map($it).toseq() # works fine

Define Your Reducer!

every reducer have: [let't name our custom reducer zzz]

1. zzzInit[T](args...): ... :: initializes the value of accumulator(state) :: must be generic.
2. zzzUpdate(var acc, newValue): bool :: updates the accumulator based on newValue, if returns false, the iteration stops.
3. zzzFinalizer(n): ... :: returns the result of the accumulator.

NOTE: see implementations in src/iterrr/reducers.nim

Custom Reducer

pattern:

ITER |> ...reduce(idents, acc = initial_value, [finalizer]): 
   update acc here 

Notes:

• acc can be any ident like result or answer, ...
• Finalizer:
  • it's optional
  • it's an experssion inside of it you have access to the acc ident
  • the default finalizer is acc ident

Example of searching for a number:

let element = (1..10) |> reduce(it, answer = none int, answer.get):
  if your_condition(it):
    answer = some MyNumber
    break mainLoop

Note: if the item has not found, raises UnpackDefect error as result of get function in finalizer answer.get.

Don't Wanna Use Reducer?

My view is that a lot of the time in Nim when you're doing filter or map you're just going to operate it on afterwards :: @beef331 AKA beef.

I'm agree with beef. it happens a lot. you can do it with each(arg1, arg2,...). [arguments semantic is the same as custom idents]

(1..10) |> filter(it in 3..5).each(num):
  echo num
  if num < 7:
    break mainLoop

Note: mainLoop is the main loop block

Custom Adapter

adapters are inspired from implmentation of iterators in Nim. TODO: explain more

Limitations: you have to import the dependencies of adapters in order to use them.

Built-in adapter:

• group
• window
• cycle
• flatten
• drop
• take

Usage: example:

let matrix = [
  [1, 2, 3],
  [4, 5, 6],
  [7, 8, 9]
]
matrix.items |> flatten().map(-it).cycle(11).group(4).toseq()

result:

@[
  @[-1, -2, -3, -4], 
  @[-5, -6, -7, -8], 
  @[-9, -1, -2] 
]

see tests/test.nim for more.

Define your custom adapter: for now the name of loop iterator are limited to it. TODO; see src/iterrr/adapters.

iterrr macro

don't like |> operator? no problem! use iterrr keyword:

pattern:

iterable.iterator.iterrr:
  filter(...)
  map(...)
  reducer(...)
  reduce(...)/each(...):
    # code ...

example:

let points = @[(1, 2), (-3, 4), (12, 3), (-1, -6), (5, -9)]

points.items.iterrr:
  map (x, y) => x # or map((x, y) => x)
  filter it > 0   # or filter(it > 0)
  each n:         # or each(n):
    echo n

Nesting

Both the |> operator and iterrr macro can be nested.

Examples:

matrix.pairs |> map((ia, a) => (
    a.pairs |> map((ib, _) => (ia, ib)).toseq()
  )).toseq()

matrix.pairs.iterrr:
  map: (ia, a) => a.pairs.iterrr:
    map: (ib, _) => (ia, ib)
    toseq()
  toseq()

Debugging

use -d:iterrrDebug flag to see generated code.

Breaking changes

0.x -> 1.x:

• using brackets for defining custom idents is no longer supported.

Inspirations

1. zero_functional
2. itertools
3. slicerator
4. xflywind's comment on this issue
5. mangle

Common Questions:

iterrr VS zero_functional:

iterrr targets the same problem as zero_functional, while being better at extensibility.

Is it fully "zero cost" like zero_functional though?

well NO, most of it is because of reducer update calls, however the speed difference is soooo tiny and you can't even measure it. I could define all reducer updates as template instead of function but IMO it's better to have call stack when you hit errors ...

Quotes

writing macro is kind of addicting... :: PMunch

iterrr's Issues

Optimize `replacedIdent`

don't limit yourself to `it` ident

what I have in mind:

1..20 >< imap[number](number + 1)

or tuple unpacking

"hello".pairs >< imap[indx, ch](indx + 1)

remove unncessary `untyped` from template args in generated code

it looks better when debugging

    template iterrrFn7(s: untyped; _: untyped): untyped {.dirty.} =
      s

to :

    template iterrrFn7(s, _): untyped {.dirty.} =
      s

open a topic in nim forum

add `define`

  (1 .. 10) |> define( 
    a = it + 1
    b = a +2
  )
  .map(b)
  .toseq()

Add `isum`

• impl
• doc

make it simple, make it usable

• remove "Non Operator Version"
• remove bracket style variable naming
• add toCountTable reducer
• migrate completely to templates
• new custom reducer style
• remove debug infix !> - ( use -d:iterrrDebug instead)
• new documentation

Replace `with` by `inject`

(1.. 10) |>
  alter(echo it)
  .toseq()

nested call variable shadowing

shadowing variable names doenst seem to work correctly.

neither with custom name:

echo:
  @[@[1,2],@[3,4]].items |>
    map[a](
      a.items |> map[a](a+1).toSeq()).
    toSeq()

nor with it:

echo:
  @[@[1,2],@[3,4]].items |>
    map(
      it.items |> map(it+1).toSeq()).
    toSeq()

`iterrr` macro removed.

iterrr was removed by 1ceb979. Is it possible to have it back or implement it in my own code using symbols exported from the iterrr module? I used to be able to write stuff like this:

      dest = something.iterator.iterrr:
        map: do_something(it) # do_something
        map: do_something_else(it) # do_something_else
        toSeq()

and loved it 😄 . Thank you!

Add doc for adapters

• unique idents
• symbol binding

Add inplace reducer

.ireduce[acc, a](initialState, [finalizer]):
   max(acc, a)

add custom reducer

My view is that a lot of the time in Nim when you're doing filter or map you're just going to operate it on afterwards

for data in lines("file").map(parseInt):
  echo data

-beef-

my solution

lines("file") >< imap(parseInt).do(data):
  echo data

add another syntax for custom idents

points |> imap((x,y) => x+y)

Finalizer

The default finalizer is i_toSeq if it's not set explicitly.

Other finalizers can be:

1. iall
2. Iany
3. ItoSet
4. ItoHashset
5. ICount
6. IMax
7. IMin
8. IMaxMin
9. ...

Add `ifor`

Some thing like for* in Racket-lang.

Use filter instead of #:when

Add non operator version

Something like iter or toIter or iterrr

custom adapter

like:

• flatten
• repeat
• cycle
• zip

problem with nested calls

I have the following code:

echo:
  @[@[1,2],@[3,4]].pairs |>
    map[ia,a](
      a.pairs |>
        map[ib,_]($(ia==ib)).
        strJoin()).
    strJoin()

result: truetruetruetrue
and the indicies schoudnt be all the same

`acc` name collision

choose random name for acc variable

replace `><` with `|>`

nested problems

let's say we have an string i, and we wanna iterator over it line by line

import std/strutils
let i = "..."

but this style of writing uses the funcion of splitLines

i.splitLines.iterrr:
    map it.strip()
    toseq()

but this style of writing uses the iterator of splitLines

iterrr i.splitLines:
    map it.strip()
    toseq()

add support for more syntax flexibility

like:

(1..10) >< map[n]:
   n + 1