Comments (11)
For strings (and PooledDataVecs more generally), it would make sense (at least in my applications) to replace them with their refs
. Thoughts?
from dataframes.jl.
Chris, could you explain a bit more? With PooledDataVecs, we're already storing a reference to a string. Or, is this about "dummy variables"? If so, that makes sense if a Float64 can point somehow be a ref
.
from dataframes.jl.
I was thinking x = convert_to_float(df)
shouldn't necessarily create dummy variables for strings (since that's what expand
does). Instead, for each PooledDataVec in df
we would place its refs in x
(promoted to Float64 for Uint16). But, if the goal is to make it easy to go from DataFrame to statistical modeling, then maybe automatically expanding to dummy variables would be a safer default.
from dataframes.jl.
Thanks for the explanation, Chris. It's beyond my expertise.
from dataframes.jl.
I'm new to JuliaData development, so I hope you don't mind my 2 cents. I think that Harlan's intent was probably meant as a shortcut to generate a model matrix using all columns in a DataFrame. In essence, something like model_matrix(:(~ col1 + col2 + ... + colp), df))
. This would be a useful way to generate a model matrix for statistical modeling without typing in all of the column names. Along those lines, dummy variable (or other contrast methods) would be the way to go for PooledDataVecs. Of course, Harlan can speak up if this was or was not his intention.
from dataframes.jl.
I originally misread your comment. I agree, that would be a good feature.
from dataframes.jl.
Hm, I'm not sure if I remember my original intentions here. But at the moment, yes, it might just be equivalent to model_matrix on all columns. Note that DataVec{ASCIIString}
should get dummies too, and that PooledDataVec{Float64}
probably shouldn't (although that'd be a weird situation).
from dataframes.jl.
Can you guys explain what you mean by "refs" here? I'm a bit confused.
from dataframes.jl.
Oh, just that PooledDataVecs have integer indexes into the pool, and one option for generating a numerical matrix from a DataFrame, as an alternative to creating dummy variables for categorical values, is to use those indexes (converted to floats) as the values. I don't think it's a good idea. It conflates implementation details (the pool) with appropriate behavior for categorical values. It'd be odd, but occasionally useful, to replace categorical variables with some sort of numerical value (especially if they're ordinal, ala survey responses). But we also need to be able to support DataVec{ASCIIString}, which doesn't have underlying integers at all.
from dataframes.jl.
Ah, ok. I see what you're saying. Thanks for the explanation.
from dataframes.jl.
I closed this a while back.
from dataframes.jl.
Related Issues (20)
- Feature request: Pairs in stack HOT 2
- Grouped DataFrame with array elements fails to combine HOT 4
- error when combining a grouped empty dataframe using `first` HOT 6
- Short circuit && on subset? HOT 1
- Integer strings as colnames/selectors are error prone HOT 2
- Suggestion - Matrix Syntax for hcat (as well as vcat) HOT 4
- Document custom generation of column names in manual HOT 9
- `join` should not introduce `Missing` types to schema HOT 1
- Consider removing Tables.allocatecolumn in vcat
- DataFrame(t::Table) converts PooledVector columns HOT 2
- Sampling GroupedDataFrames (rand) HOT 5
- Julia 1.12.0-DEV: testing DataFrames throws a 'method dispatch ambiguities' error HOT 2
- Add options to `join` missing error? HOT 1
- Column types seem to change after inner-join between two DataFrames HOT 1
- using propertynames on GroupedDataFrame HOT 6
- Very slow to convert DBInterface (DuckDB) result HOT 1
- Add arbitrary matching function for joins HOT 3
- Add Tidier.jl to docs/src/man/querying_frameworks.md HOT 1
- join size warnings
- Type piracy of `reduce(vcat)` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataframes.jl.