fastai / fastcore

Python supercharged for the fastai library

Welcome to fastcore

Python is a powerful, dynamic language. Rather than bake everything into the language, it lets the programmer customize it to make it work for them. fastcore uses this flexibility to add to Python features inspired by other languages we’ve loved, like multiple dispatch from Julia, mixins from Ruby, and currying, binding, and more from Haskell. It also adds some “missing features” and clean up some rough edges in the Python standard library, such as simplifying parallel processing, and bringing ideas from NumPy over to Python’s list type.

Getting started

To install fastcore run: conda install fastcore -c fastai (if you use Anaconda, which we recommend) or pip install fastcore. For an editable install, clone this repo and run: pip install -e ".[dev]". fastcore is tested to work on Ubuntu, macOS and Windows (versions tested are those show with the -latest suffix here.

fastcore contains many features, including:

• fastcore.test: Simple testing functions
• fastcore.foundation: Mixins, delegation, composition, and more
• fastcore.xtras: Utility functions to help with functional-style programming, parallel processing, and more
• fastcore.dispatch: Multiple dispatch methods
• fastcore.transform: Pipelines of composed partially reversible transformations

To get started, we recommend you read through the fastcore tour.

Contributing

After you clone this repository, please run nbdev_install_hooks in your terminal. This sets up git hooks, which clean up the notebooks to remove the extraneous stuff stored in the notebooks (e.g. which cells you ran) which causes unnecessary merge conflicts.

To run the tests in parallel, launch nbdev_test.

Before submitting a PR, check that the local library and notebooks match.

• If you made a change to the notebooks in one of the exported cells, you can export it to the library with nbdev_prepare.
• If you made a change to the library, you can export it back to the notebooks with nbdev_update.

fastcore's Issues

extension of Pathlib.path break other libraries

The pathlib.Path extension here read and write break opening or writing using pathlib.Path paths in other libraries (for example numpy or rasterio). I also found someone else experiencing this issue on the fastai forums:
https://forums.fast.ai/t/solved-issue-with-utf-8-codec-when-saving-and-or-loading-np-ndarray/69072/2

This is quite an annoying issue since even importing a simple function from nbdev already patches pathlib.Path. Right now my solution is to delete these functions if they exist, but it is not very elegant:

if hasattr(Path, 'read'):
    del Path.read
if hasattr(Path, 'write'):
    del Path.write

I am not sure where the Path.read() and Path.write() extensions are used so it is hard for me to propose a good solution, but perhaps the extensions can be renamed to avoid conflicts with other libraries.

Example of this error occurring using rasterio:

funcs_kwargs doesn't update class signature when inheriting from Module

Using funcs_kwargs with a class that inherits from Module does not update the class signature.

Minimum example:

from fastai2.basics import *

@funcs_kwargs
class C(Module):
  _methods = ['m1']
  def __init__(self, **kwargs): pass
  def m1(self): return 1

print(inspect.signature(C))

Returns: (self, **kwargs)
Expected: (*, m1=None)

Add `urlread` and `urljson`

Change arguments of `store_attr()` and remove `store_attrs` attribute

store_attr's API has changed. self is now the second parameter, and is optional. Previously, if no names were passed to store, names were taken from the store_attrs attribute; now, however, names are taken from the list of arguments to the current function.

Can't dispatch on classmethods

I want to write method dispatching on class methods, I simply can't!
I'm going to fix it and send a pull request :)

failing example:

def m_nin(cls, x:(str,numbers.Integral)): return str(x)+'1'
def m_bll(cls, x:bool): cls.foo='a'
def m_num(cls, x:numbers.Number): return x*2

t = TypeDispatch([m_nin,m_num,m_bll])
class A: f = t # set class attribute `f` equal to a TypeDispatch

test_eq(A.f(1), '11')  #dispatch to m_nin
test_eq(A.f(1.), 2.)   #dispatch to m_num
test_is(A.f.owner, A)

A.f(False) # this triggers t.m_bll to run, which sets A.foo to 'a'
test_eq(A.foo, 'a')

Can't add to Pipeline initialized with None

In the example code bellow:

class T1(Transform):pass
pipe = Pipeline()
pipe.add(T1())
pipe[0]

returns noop: (object,object) -> noop . Whereas if I use pipe = Pipeline([]) it adds T1 as expected.

IMO it should either fail immediately or work the same way in both cases. The way it is makes it a hard to catch bug.

I'm using fastcore 0.1.18 installed from pip

add `threadpool=False` param to `parallel` to use threads instead of processes

Feature Request: version of `bind` that preserves signature

bind's signature after binding a function is <Signature (*args, **kwargs)> . Could it be possible to please have a version of it that gives back a function (or bind object) with the right signature after binding? for example:
for def foo(a: int, b: str, c: float, d: str='hey'): pass
its signature is <Signature (a: int, b: str, c: float, d: str = 'hey')>
and if I define bf = bind(foo, 1, _Arg(0), 2)
I would like to have bfs signature be <Signature (b: str, d: str = 'hey')>

Should the test in __setitem__ be applied to __getitem__ and __delitem__ as well?

fastcore/fastcore/foundation.py

Lines 281 to 283 in 8a56bcb

	def __getitem__(self, k): return self.items[k]
	def __setitem__(self, k, v): self.items[list(k) if isinstance(k,CollBase) else k] = v
	def __delitem__(self, i): del(self.items[i])

should list(k) if isinstance(k,CollBase) else k be applied to __getitem__ (and potentially __delitem__) as well?

I was poking around fastai2's _42_tabular_rapids.ipynb. In Tabular.reduce_cats(), the index k is an L-list, which cudf.DataFrame can't handle. The following would fail and point to the above line, where self.items is a cudf.DataFrame. Listifying k would get pass this hiccup:

df = cudf.from_pandas(pd.DataFrame({'a':[0,1,2,0,2]}))
to = TabularGPU(df, Categorify, 'a')

Reapply fix for #86 which was unintentionally reverted by the next commit

Reapply fix for #86 which was unintentionally reverted by commit id #e6620759fada127a1ef4cb2336d4ad9f8c000910

This will allow the fastai package to run with Python 3.6 which doesn't have the
mp_context parameter in the list of arguments to ProcessPoolExecutor,
which was introduced only in Python 3.7.

fastcore.utils.Tuple is not a replacement for typing.Tuple

Sorry to open a duplicate issue, but I posted (#36 (comment)) in #36 after it was closed about how fastcore.utils.Tuple is actually not usable in place of typing.Tuple, unlike what @tcappelle said, but perhaps you don't receive notifications from closed issues.

Copying that comment here:

Hi, thanks for the quick responses.

1. I don't think I can use fastcore.utils.Tuple as typing.Tuple: see this gist where it breaks: https://gist.github.com/indigoviolet/175556625b9e21748fe2d2be6500a1bb
2. I understand that I can import typing.Tuple as OldTuple, but IMHO a library shouldn't shadow a standard library name without a very good reason, and should definitely not break it. I would expect fastcore.utils.Tuple to require explicitly importing it if it needs to be named Tuple.

Can't use classmethod decorator!

Some users may expect to use typedispatch decorator on top of classmethod or staticmethod they can't right now. I'm going to fix this.
Failing example:

class A:
    @typedispatch
    def f_td_test(self, x:numbers.Integral, y): return x+1
    @typedispatch
    @classmethod
    def f_td_test(cls, x:int, y:float): return x+y
    @typedispatch
    @staticmethod
    def f_td_test(x:int, y:int): return x*y
    
test_eq(A.f_td_test(3,2), 6)
test_eq(A.f_td_test(3,2.0), 5)
test_eq(A().f_td_test(3,'2.0'), 4)

L is not instance of collection.abc.Sequence

Short:

l = list(range(10))
random.sample(l, 3)
> [1, 8, 7]

l = L(range(10))
random.sample(l, 3)
> TypeError: Population must be a sequence or set.  For dicts, use list(d).

It would also nice to be able to call directly L.sample(k)

support for optional params

add `nested_attr` and use it in `L.attrgot`

add `ignore_ex=False` and `as_bytes=False` params to `run`

fastcore.utils.Tuple shadows typing.Tuple

It surprised me with an unexpected error that fastcore.utils.Tuple exists -- it shadows the stdlib's typing.Tuple. Should this be renamed to something that doesn't collide with the stdlib, or is this expected?

Weird interaction between numpy, pathlib and L

Given this piece of code:

import numpy as np
from pathlib import Path
# from fastcore.all import L

path = Path("models/some_arr.npy")
array = np.random.rand(100)
np.save(path, array)

everything works as expected. Import L, (or * ) and one gets:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-052950c92411> in <module>
      5 path = Path("models/some_arr.npy")
      6 array = np.random.rand(100)
----> 7 np.save(path, array)
      8 

<__array_function__ internals> in save(*args, **kwargs)

~/miniconda3/envs/ta/lib/python3.7/site-packages/numpy/lib/npyio.py in save(file, arr, allow_pickle, fix_imports)
    551         arr = np.asanyarray(arr)
    552         format.write_array(fid, arr, allow_pickle=allow_pickle,
--> 553                            pickle_kwargs=pickle_kwargs)
    554     finally:
    555         if own_fid:

~/miniconda3/envs/ta/lib/python3.7/site-packages/numpy/lib/format.py in write_array(fp, array, version, allow_pickle, pickle_kwargs)
    658     """
    659     _check_version(version)
--> 660     _write_array_header(fp, header_data_from_array_1_0(array), version)
    661 
    662     if array.itemsize == 0:

~/miniconda3/envs/ta/lib/python3.7/site-packages/numpy/lib/format.py in _write_array_header(fp, d, version)
    432     else:
    433         header = _wrap_header(header, version)
--> 434     fp.write(header)
    435 
    436 def write_array_header_1_0(fp, d):

~/work/installs/fastcore/fastcore/utils.py in write(self, txt, encoding)
    482     "Write `txt` to `self`, creating directories as needed"
    483     self.parent.mkdir(parents=True,exist_ok=True)
--> 484     with self.open('w', encoding=encoding) as f: f.write(txt)
    485 
    486 # Cell

TypeError: write() argument must be str, not bytes

Workaround: Convert the Path() object to a string. This breaks a lot of code, so . . .

Thank you!
p.s.
fastcore is installed from master at 4a2d5ea
numpy: 1.18.5 py37h8960a57_0 conda-forge

I know, conda-forge but somehow I ended up with numpy from there.

Thank you!

AttributeError: 'str' object has no attribute '__stored_args__'

As mentioned in fastai/fastai#2758, looks like ea1c33f#diff-2d5ca7a202ff50e2df29f58cfa4a4c12 may have introduced a bug.

Thanks for the great library!

fastcore not working properly on Windows

I tried to install fastcore on a Windows machine, and couldn't make it work as showcased in the documentation. The installation does finish without errors, but it seems that the inner modules of the package are not accessible, as if the package is empty.

Installation commands (python 3.6 is a requirement that I have):

conda create -n env_fastcore python=3.6
conda activate env_fastcore 
conda install -c fastai fastcore

Simple commands that I tried to execute:

In [1]: import fastcore

In [2]: fastcore.__version__
Out[2]: '1.0.12'

In [3]: fastcore.utils
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-7f4bf76169d5> in <module>
----> 1 fastcore.utils

AttributeError: module 'fastcore' has no attribute 'utils'

In [4]: dir(fastcore)
Out[4]:
['__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 '__version__']

Any hints on how to configure it properly?

Global patch on pathlib module breaks 3rd party library ruamel.yaml

hi - while helping @HenryDashwood to get his work on support fastai2 in BentoML(bentoml/BentoML#596), we noticed an issue that is unique to fastai2, here are the code to reproduce the issue:

Install fastcore and ruamel.yaml, the following code should run without any issue:

conda_env_yaml = """
name: bentoml-Demo
channels:
  - defaults
dependencies:
  - python=3.7.5
  - pip
"""
from ruamel.yaml import YAML
yaml = YAML()
y = yaml.load(conda_env_yaml)
from pathlib import Path
yaml.dump(y, Path("/tmp/test.yml"))

But after importing fastcore.utils with from fastcore.utils import * and run the code above again, the follow error occurs:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-beaac7fd8e8a> in <module>
     11 y = yaml.load(conda_env_yaml)
     12 from pathlib import Path
---> 13 yaml.dump(y, Path("/tmp/test.yml"))

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/main.py in dump(self, data, stream, _kw, transform)
    447             if stream is None:
    448                 raise TypeError('Need a stream argument when not dumping from context manager')
--> 449             return self.dump_all([data], stream, _kw, transform=transform)
    450
    451     def dump_all(self, documents, stream, _kw=enforce, transform=None):

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/main.py in dump_all(self, documents, stream, _kw, transform)
    461         self._context_manager = YAMLContextManager(self, transform=transform)
    462         for data in documents:
--> 463             self._context_manager.dump(data)
    464         self._context_manager.teardown_output()
    465         self._output = None

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/main.py in dump(self, data)
    813             self.init_output(data)
    814         try:
--> 815             self._yaml.representer.represent(data)
    816         except AttributeError:
    817             # nprint(dir(dumper._representer))

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/representer.py in represent(self, data)
     84         # type: (Any) -> None
     85         node = self.represent_data(data)
---> 86         self.serializer.serialize(node)
     87         self.represented_objects = {}
     88         self.object_keeper = []

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/serializer.py in serialize(self, node)
    115         )
    116         self.anchor_node(node)
--> 117         self.serialize_node(node, None, None)
    118         self.emitter.emit(DocumentEndEvent(explicit=self.use_explicit_end))
    119         self.serialized_nodes = {}

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/serializer.py in serialize_node(self, node, parent, index)
    230                 )
    231                 for key, value in node.value:
--> 232                     self.serialize_node(key, node, None)
    233                     self.serialize_node(value, node, key)
    234                 self.emitter.emit(MappingEndEvent(comment=[map_comment, end_comment]))

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/serializer.py in serialize_node(self, node, parent, index)
    178                         node.value,
    179                         style=node.style,
--> 180                         comment=node.comment,
    181                     )
    182                 )

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/emitter.py in emit(self, event)
    252         while not self.need_more_events():
    253             self.event = self.events.pop(0)
--> 254             self.state()
    255             self.event = None
    256

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/emitter.py in expect_first_block_mapping_key(self)
    662     def expect_first_block_mapping_key(self):
    663         # type: () -> None
--> 664         return self.expect_block_mapping_key(first=True)
    665
    666     def expect_block_mapping_key(self, first=False):

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/emitter.py in expect_block_mapping_key(self, first)
    687                         pass
    688                 self.states.append(self.expect_block_mapping_simple_value)
--> 689                 self.expect_node(mapping=True, simple_key=True)
    690                 if isinstance(self.event, AliasEvent):
    691                     self.stream.write(u' ')

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/emitter.py in expect_node(self, root, sequence, mapping, simple_key)
    420             if isinstance(self.event, ScalarEvent):
    421                 # nprint('@', self.indention, self.no_newline, self.column)
--> 422                 self.expect_scalar()
    423             elif isinstance(self.event, SequenceStartEvent):
    424                 # nprint('@', self.indention, self.no_newline, self.column)

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/emitter.py in expect_scalar(self)
    468         # type: () -> None
    469         self.increase_indent(flow=True)
--> 470         self.process_scalar()
    471         self.indent = self.indents.pop()
    472         self.state = self.states.pop()

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/emitter.py in process_scalar(self)
    883             self.write_literal(self.analysis.scalar, self.event.comment)
    884         else:
--> 885             self.write_plain(self.analysis.scalar, split)
    886         self.analysis = None
    887         self.style = None

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/ruamel/yaml/emitter.py in write_plain(self, text, split)
   1614                         data = data.encode(self.encoding)
   1615                     try:
-> 1616                         self.stream.write(data)
   1617                     except:  # NOQA
   1618                         sys.stdout.write(repr(data) + '\n')

~/opt/miniconda3/envs/btml-dev/lib/python3.7/site-packages/fastcore/utils.py in write(self, txt, encoding)
    428     "Write `txt` to `self`, creating directories as needed"
    429     self.parent.mkdir(parents=True,exist_ok=True)
--> 430     with self.open('w', encoding=encoding) as f: f.write(txt)
    431
    432 # Cell

TypeError: write() argument must be str, not bytes

After looking into this, the reason was the following code in the fastcore/utils.py file:

fastcore/fastcore/utils.py

Line 480 in 8a56bcb

def write(self:Path, txt, encoding='utf8'):

@patch
def write(self:Path, txt, encoding='utf8'):
    "Write `txt` to `self`, creating directories as needed"
    self.parent.mkdir(parents=True,exist_ok=True)
    with self.open('w', encoding=encoding) as f: f.write(txt)

The code is adding a global patch to the pathlib.Path object, this line specifically added the method "write" to Path. However ruamel.yaml, a popular YAML handling library for python, is depending on this attribute missing, to tell if the given file path is a pathlib.Path object:
https://github.com/pycontribs/ruamel-yaml/blob/7692648298f680357e09502f991ce0d1c213bfeb/main.py#L755

This made it very hard to integrate fastai2 with BentoML for model serving, or for anything that uses ruamel.yaml. It may break other packages too. I think in general libraries should avoid this type of global patching especially on core modules like pathlib.

add function support to `store_attr`

Making patch decorator more general

It seems to be useful if we can use patch decorator just like patch_to (with optional parameters while we can use it without any argument). I think by making patch a little general, maybe we didn't need patch_property.

Example use-case:

@patch(as_prop=True)
def add_ten(self:_T5): return self + 10

t = _T5(4)
test_eq(t.add_ten, 14)

and

class _T5(int): attr = 3 # attr is a class attribute we will access in a later method
    
@patch(cls_method=True)
def func(cls:_T5, x): return cls.attr + x # you can access class attributes in the normal way

test_eq(_T5.func(4), 7)

L.index error

Simple bug, running in Python 3.7.7

from fastcore.all import *
l = L(1,2,3)
l.index(2)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-f6d37febc116> in <module>
----> 1 l.index(2)

~/libs/fastcore/fastcore/foundation.py in index(self, value, start, stop)
    413     def pop(self,o=-1): return self.items.pop(o)
    414     def clear(self   ): return self.items.clear()
--> 415     def index(self, value, start=0, stop=sys.maxsize): return self.items.index(value, start=start, stop=stop)
    416     def sort(self, key=None, reverse=False): return self.items.sort(key=key, reverse=reverse)
    417     def reduce(self, f, initial=None): return reduce(f, self) if initial is None else reduce(f, self, initial)

TypeError: index() takes no keyword arguments

Pipeline Section in docs have an error

https://fastcore.fast.ai/transform.html#Pipeline

pipe = Pipeline([f2,f3])
t = pipe(TEST_IMAGE)
ax = pipe.show(t)

Bottom of the stack trace:

~/anaconda3/lib/python3.7/site-packages/matplotlib/transforms.py in rotate(self, theta)
   1927         and :meth:`scale`.
   1928         """
-> 1929         a = math.cos(theta)
   1930         b = math.sin(theta)
   1931         rotate_mtx = np.array([[a, -b, 0.0], [b, a, 0.0], [0.0, 0.0, 1.0]],

<ipython-input-47-5faf5d2c0d18> in cos(x)
      2 
      3 @patch
----> 4 def cos(x:math): return math.cos(x)
      5 @patch
      6 def cos(x:np.ndarray): return np.cos(x)

... last 1 frames repeated, from the frame below ...

<ipython-input-47-5faf5d2c0d18> in cos(x)
      2 
      3 @patch
----> 4 def cos(x:math): return math.cos(x)
      5 @patch
      6 def cos(x:np.ndarray): return np.cos(x)

RecursionError: maximum recursion depth exceeded

cc: @jph00

Add default to `store_attr` to automatically pull in all parameters from __init__

Per @jph00 's discord comment:

If there's no store_attr attr and there's no param of attrs, then the default could be to look at the passed params

Jeremy, If you could ping me whenever you work on this on Discord I will try to pair with you if I'm around! Thanks

Unexpected behavior of cycle (and everything using _listify)

As far as I know, I would have expected cycle to work with np.array as it would work with itertool.cycle

>>> from fastcore.foundation import cycle
>>> import numpy as np
>>> it = cycle(np.arange(3))
>>> next(it)
array([0, 1, 2])
>>> next(it)
array([0, 1, 2])
>>> next(it)
array([0, 1, 2])

>>> from itertools import cycle
>>> import numpy as np
>>> it = cycle(np.arange(3))
>>> next(it)
0
>>> next(it)
1
>>> next(it)
2

solution: IMO _listify should return o instead of [o]

`AttrDict`: a `dict` subclass that also provides access to keys as attrs; and change `dict2obj` to create `AttrDict`s instead of `SimpleNamespace`s

Remove numpy prerequisite

`dict2obj`: Convert (possibly nested) dicts (or lists of dicts) to `SimpleNamespace`

add `run_proc` and `do_request`

add `L.insert` and `exec_local`

move metaclasses and delegates et al to new `meta` module

added `chunksize` to `parallel`, which passes to `ProcessPoolExecutor.map`

move fastscript to fastcore.script

Utility function "write" break other python module

Hi guys,

Thank you for the work on Fastai2.

I think FastCore is great for helping newcomers to make fewer mistakes when working with python. However by changing the default behavior of some core functions, it breaks other useful python modules.

The issue I encountered:
ruamel.yaml is one of the most popular YAML packages that make working with YAML a lot easier with Python. With fastcore's patch on write, it breaks ruamel.yaml's dumps function.

Proposed solutions
Make the patch optional

Using fastcore in google cloud function

Hi,

Big fan of the library. I'm only have one issue and want to record it for other people who might come across the same problem.

When using fastcore inside a google cloud function and putting fastcore as a depency in your requirements.txt file you get an error.

I think it has to do with the dataclasses library.

Would be nice if it could be fixed.
Currently I just install numpy and copy the fastcore folder into the root folder.

L.split() is not the same as `str.split`

I’m not sure if posts like this belong on forum or here. (I looked for guidance in the forums, but probably missed it.)
Love what you have done with fastcore. As a long time devote of the lisp style of languages, the addition of the L type is great!. I have just started playing around with it and found the following differences between L.split() and str.split(). Since the doc-string in L says they should behave the same, I bring this to your attention:

——————————————————————————-

In [68]: a = "this is a test of the thing"                                                                                                            

In [69]: a.split?                                                                                                                                     
Signature: a.split(sep=None, maxsplit=-1)
Docstring:
Return a list of the words in the string, using sep as the delimiter string.

sep
  The delimiter according which to split the string.
  None (the default value) means split according to any whitespace,
  and discard empty strings from the result.
maxsplit
  Maximum number of splits to do.
  -1 (the default value) means no limit.
Type:      builtin_function_or_method

In [70]: a.split()                                                                                                                                    
Out[70]: ['this', 'is', 'a', 'test', 'of', 'the', 'thing']

In [71]: a.split(' ')                                                                                                                                 
Out[71]: ['this', 'is', 'a', 'test', 'of', 'the', 'thing']

In [72]: a.split('t')                                                                                                                                 
Out[72]: ['', 'his is a ', 'es', ' of ', 'he ', 'hing']

In [73]: b = L(a)                                                                                                                                     

In [74]: b.split?                                                                                                                                     
Signature: b.split(s, sep=None, maxsplit=-1)
Docstring: Same as `str.split`, but returns an `L`
File:      ~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/foundation.py
Type:      method

In [75]: b.split()                                                                                                                                    
TypeError                                 Traceback (most recent call last)
<ipython-input-75-5540ebcda3d0> in <module>
----> 1 b.split()

TypeError: split() missing 1 required positional argument: 's'

In [76]: b.split(' ')                                                                                                                                 
Out[76]: (#0) []

In [77]: b.split('t')                                                                                                                                 
Out[77]: (#1) ['t']

"has default params. These will be ignored" shown when not appropriate

Why "assert cls.__doc__ is not None" in "foundation.add_docs"

When I write the following code, I get an error.

@docs
class A:
    def a(self): ...
    _docs = dict(a='Doc for `a`')

This makes sense because I have not specified documentation for the class A. So I have to change the code to this

@docs
class A:
    "Doc for class A"
    def a(self): ...
    _docs = dict(a='Doc for `a`')

This is happening due to the last assert statement in fastcore.foundation.add_docs function i.e. assert cls.__doc__ is not None, f"Missing class docs: {cls}".

My questions are

1. What is the need of this assert statement?
2. When using @docs decorator, how to specify the documentation for the class in the _docs dict? Like in the above example, how to specify the documentation for class A ('Doc for class A') in the _docs dict.
3. Does it make sense to make this assert check optional? Like for the cases when I don't want to write documentation for the main class, but write documentation for the functions in it.

`run`: flexiblyrun an external process and raise exception if it fails

Minor bug in _oper

In the definition of _oper there is an assumption about the third argument. If it's not equal to the default it assumes that we want to use op in partial case.

Failing example:

assert math.isnan(add(1, float('nan')))

We also can't use None as default in _oper, at least for getting more useful error messages.

'mp_context' keyword for initialising concurrent.futures.ProcessPoolExecutor only supported in python 3.7+

When running the following test in google colaboratry

from fastcore.utils  import parallel
from fastcore.test import test_eq
def add_one(x, a=1): 
    time.sleep(random.random()/80)
    return x+a

inp,exp = range(50),range(1,51)
test_eq(parallel(add_one, inp, n_workers=2), exp)

I get the following error:

/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in parallel(f, items, n_workers, total, progress, pause, timeout, chunksize, *args, **kwargs)
704 "Applies func in parallel to items, using n_workers"
705 if progress is None: progress = progress_bar is not None
--> 706 with ProcessPoolExecutor(n_workers, pause=pause) as ex:
707 r = ex.map(f,items, *args, timeout=timeout, chunksize=chunksize, **kwargs)
708 if progress:

/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in init(self, max_workers, on_exc, pause, mp_context, initializer, initargs)
685 self.not_parallel = max_workers==0
686 if self.not_parallel: max_workers=1
--> 687 super().init(max_workers, mp_context=mp_context, initializer=initializer, initargs=initargs)
688
689 def map(self, f, items, timeout=None, chunksize=1, *args, **kwargs):

TypeError: init() got an unexpected keyword argument 'mp_context'

Here is the python version:

!python --version

Python 3.6.9

remove `parallel_chunked`, use `chunksize` arg to `parallel` instead

Warn if defaults passed to `typedispatch`

add `L.map_filter` and `L.map_first`

These support some nice refactorings, like changing from this:

d = []
for c in cs:
    m = f(c)
    if not m: continue
    d.append(m.group(1))

to this:

d = cs.map_filter(f).map(Self.group(1))

Inconsistency with cmp_instance and typedispatch because of binary sort algorithm

Basically, typedispatch assume that the type in funcs are ordered in a topological sort, but this is not true as the key comparison is not defined well under this.

stupid example (edited, now it is less stupid :P ):

>>> from fastcore.dispatch import *
>>> 
>>> class A: pass
... 
>>> class B(A): pass
... 
>>> @typedispatch
... def f(a:A): print('A')
... 
>>> @typedispatch
... def f(x:int): pass
... 
>>> @typedispatch
... def f(x:tuple): pass
... 
>>> @typedispatch
... def f(b:B): print('B')
... 
>>> @typedispatch
... def f(x:list): pass
... 
>>> f(B()) 
A
>>> print(f)
(list,object) -> f
(tuple,object) -> f
(A,object) -> f
(int,object) -> f
(B,object) -> f

doing insertion sort variation with the key comparison at hand will solve the issue I believe. I can implement this quickly but would love hearing your opinion first.

Ido

wrong assert in "test_eq_type". It checks if two values are equal. The type is checked after that

fastcore/fastcore/test.py

Line 39 in 7ac3fc2

test_eq(a,b)

the marked line should be deleted

thanks a lot

test_eq_type(1, 2)

======== stacktrace begin
HERE---> 39 test_eq(a,b)
40 test_eq(type(a),type(b))
41 if isinstance(a,(list,tuple)): test_eq(map(type,a),map(type,b))

~/anaconda3/envs/etftrader/lib/python3.7/site-packages/fastcore/test.py in test_eq(a, b)
32 def test_eq(a,b):
33 "test that a==b"
---> 34 test(a,b,equals, '==')
35
36 # Cell

~/anaconda3/envs/etftrader/lib/python3.7/site-packages/fastcore/test.py in test(a, b, cmp, cname)
22 "assert that cmp(a,b); display inputs and cname or cmp.__name__ if it fails"
23 if cname is None: cname=cmp.name
---> 24 assert cmp(a,b),f"{cname}:\n{a}\n{b}"
25
26 # Cell

AssertionError: ==:
1
2
======== stacktrace end

New param `cast` for `store_attr` and new decorator `with_cast`

@typedispatch does not support default values

@typedispatch
def f_td_test(x:int, y:int=10): return x*y

test_eq(f_td_test(3,2), 6)
test_eq(f_td_test(3), 30)

Errs out:

---------------------------------------
TypeErrorTraceback (most recent call last)
<ipython-input-34-6e7b98db6be2> in <module>
      3 
      4 test_eq(f_td_test(3,2), 6)
----> 5 test_eq(f_td_test(3), 30)

<ipython-input-12-9fc191b1d495> in __call__(self, *args, **kwargs)
     34         if not f: return args[0]
     35         if self.inst is not None: f = MethodType(f, self.inst)
---> 36         return f(*args, **kwargs)
     37 
     38     def __get__(self, inst, owner):

TypeError: f_td_test() missing 1 required positional argument: 'y'