Comments (2)
Hi,
It is not. All the calculations are done in pandas, so if pandas gets support for calculating basic statistics and frequencies in chunks, or via HDF5 connectors, it would be very easy to adapt to pandas-profiling.
For now, your best bet would be to:
- Rewrite the backend of pandas-profiling to use Spark, Hive or another technology that can calculate summary statistics using hadoop
- Use a commercial solution like SAS Data Loader for Hadoop http://www.sas.com/en_us/software/data-management/data-loader-hadoop.html
Jos
from ydata-profiling.
Thanks. Great tool by the way.
from ydata-profiling.
Related Issues (20)
- no module named "pydantic.v1" HOT 1
- Bug Report
- Feature Request HOT 1
- Bugging creation of report
- Bug Report: Comparing reports from Spark HOT 1
- Upgrade Visions library
- 'NoneType' object has no attribute 'replace' HOT 4
- Crashes with memory leak, seems to be deadlock related HOT 1
- Feat: Use ibis as single backend HOT 3
- No module named 'scipy.stats._mvn' error when importing ProfileReport HOT 2
- Feature Request: use CJK (non-ascii) character
- Bug Report HOT 1
- Bug Report: ValueError: NaTType does not support strftime HOT 1
- Bug Report: DispatchError: Function <code object pandas_missing_bar HOT 2
- Bug Report: Confusing Error with Geometry Column HOT 1
- AttributeError: module 'numba' has no attribute 'generated_jit' HOT 3
- Categorical Variable showing as word cloud instead of bar
- Feature Request | Telemetry
- Bug Report: ydata-profiling won't work in Azure Synapse HOT 3
- Report is too large for any browser to render HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ydata-profiling.