feature request: dtype selector about scikit-lego

[koaning]

This sounds useful enough. Could you give an example of what the resulting pipeline might look like?

Also, this does not exist elsewhere? It seems like the sklearn.compose.ColumnTransformer is still in experimental mode.

[kayhoogland]

I suppose a transformer such as this one is what I am thinking of.

import warnings
from sklearn.base import BaseEstimator, TransformerMixin

class PandasTypeSelector(BaseEstimator, TransformerMixin):
    def __init__(self, dtype):
        self.dtype = dtype

    def fit(self, X, y=None):
        # Save the column names for later checking
        self.type_columns = list(X.select_dtypes(include=[self.dtype]))
        return self

    def transform(self, X):
        assert isinstance(X, pd.DataFrame)

        transformed_frame = X.select_dtypes(include=[self.dtype])

        if set(list(transformed_frame)) != set(self.type_columns):
            warnings.warn(f'Columns of type {self.dtype} were not equal during fit and transform')

        return transformed_frame

[koaning]

• A ValueError might be better than a mere assert but this looks good to me.
• I think sklearn prefers to have a is_fitted_ param around. Might be in the Mixin. Either way; it might be cool to initialize self.dtype as None in the __init__.
• Any idea on how you might want to test this?

[kayhoogland]

• Good point, I agree!
• I believe you mean this: https://goo.gl/ZAdbv7. At what point would like to determine the dtype if it will not be in the init?
• For testing, a simple test with a small sample dataframe will be enough I assume. We do not have to test the .select_dtypes function of Pandas.

[koaning]

There's some basic tests in there that check things like "a transformer should not change the number of rows" and you might consider adding a test for "a transformer should not change the order of the input/output".