data-forge / data-forge-ts Goto Github PK
View Code? Open in Web Editor NEWThe JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ.
Home Page: http://www.data-forge-js.com/
License: MIT License
The JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ.
Home Page: http://www.data-forge-js.com/
License: MIT License
Hi,
Is there a way to group multiple transformation steps together so that I can share it between multiple places? For example, if I want to get the n-th row of a data frame, I would use the following 3 steps: .skip(n-1).take(1).first()
(by the way, is it the best approach?).
To reuse it, I can make a function like: getAt = (df: IDataFrame, n: number) => df.skip(n-1).take(1).first()
but need to use a temporary variable, like:
const df = new DataFrame(...)
const row = getAt(df, n)
Would be nice if can apply a transformation function to a data frame directly, like:
const row = new DataFrame(...)
.apply((df) => getAt(df, n))
The apply
method can be implemented as simple as:
apply (transformer) {
return transformer(this)
}
I'm not familiar with Pandas to know whether it has something similar.
Regards
Hello again,
If you have a minute, would you mind helping me figure out why I can't get rollingWindow
to do what I expect?
I have a bunch of data formatted like so:
__index__ BTC ETH
---------- ------- -------
2017-06-15 2261.25 333.499
2017-06-16 2481.74 371.358
2017-06-17 2591.41 373.975
2017-06-18 2568.92 377.562
2017-06-19 2578.25 371.813
2017-06-20 2657.32 376.359
2017-06-21 2709.33 338.292
2017-06-22 2714.58 340.326
2017-06-23 2760.61 340.811
2017-06-24 2688.44 334.167
2017-06-25 2627.95 321.283
2017-06-26 2476.72 262.385
2017-06-27 null null
2017-06-28 2539.78 301.789
2017-06-29 2536.94 315.543
2017-06-30 2547.95 305.518
2017-07-01 2507.89 282.877
2017-07-02 2445.84 269.758
2017-07-03 2519.06 288.076
2017-07-04 2618.16 284.342
And I'm running the following:
const performance = df.rollingWindow(2).select(window => {
return ((window.last() - window.first()) / window.first())
})
.withIndex(pair => pair[0])
.select(pair => pair[1])
console.log(performance.toString());
But I keep getting...
__index__ __value__
--------- ---------
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN
6 NaN
7 NaN
8 NaN
9 NaN
10 NaN
// truncated for brevity
353 NaN
354 NaN
355 NaN
356 NaN
357 NaN
358 NaN
359 NaN
360 NaN
361 NaN
362 NaN
363 NaN
Not sure what I'm doing wrong. I've tried several variations based on the examples on the Guide.
Thank you!
I'm trying to find how to do something like the pandas melt() method.
Basically I need to go from this:
|Date|A|B|C|D|E |
|----|-|-|-|-|--|
|2005|1|2|3|4|50|
|2006|6|7|8|9|10|
Doing something like:
df.melt("Date", { varName: "X", valueName: "Val" })
And get this:
|Date|X|Val|
|----|-|---|
|2005|A|1 |
|2005|B|2 |
|2005|C|3 |
etc
|2006|A|6 |
etc
|2006|E|10 |
Hi Ashley,
I'd like to propose you to implement a transformSeries by condition for DataFrame, to locally change column values in rows respecting the specific condition.
I'd like to contribute, but I still don't know TS, anyway here is an example I wrote for personal use:
transformSeriesByCondition
I hope it will be useful as it's been to me!
๐
Rollup generates a lot of Circular dependency
warnings and produces a broken bundle with undefined variables.
Single-file bundle could probably resolve this issue.
I am trying to use distinct with multiple column select on DataFrame. But I am not getting distinct rows in the result. Distinct with single column select on DataFrame works fine. Example
df.distinct(row => [row.columnA, row.columnB]).toArray()
even tried this
df.distinct(row => ({ columnA: row.columnA, columnB: row.columnB }).toArray()
Both returns same number of rows as original dataframe, was expecting distinct rows from the original dataframe
Single column select
df.distinct(row => row.columnA).toArray()
works perfectly fine
I know I am missing something here on the multi-column selector. Could someone help. Thanks in advance.
Panda has cumsum
for its series. What's the equivalent in data forge? It doesn't seem like it exists upon first look, what's the recommended workaround?
I'm looking at df.pivot(), which I have working, and as recommended by the docs, I tried looking at df.summarize() but when I run the code I get TypeError df.summarize is not a function
I recently updated notebook versions and now when I try to load the data-forge library using
const dataForge = require('data-forge')
I get the Error:
Cannot find module 'data-forge'
at wrapperFn (1:14)
at <anonymous> (18:10)
I also tried using data-forge-fs
.... am I missing something?
Thanks!
I am unable to build my typescript web app with dataforge.
I get the following errors to do with the fs module:
ERROR in ./node_modules/data-forge/build/index.js
Module not found: Error: Can't resolve 'fs' in '/home/vagrant/kesm/apps/threescan-data-viewer/node_modules/data-forge/build'
@ ./node_modules/data-forge/build/index.js 157:17-30 253:17-30 264:17-30
@ ./node_modules/ts-loader!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/ComplexRoiSelector.vue
@ ./src/ComplexRoiSelector.vue
@ ./src/index.ts
ERROR in ./node_modules/data-forge/build/lib/dataframe.js
Module not found: Error: Can't resolve 'fs' in '/home/vagrant/kesm/apps/threescan-data-viewer/node_modules/data-forge/build/lib'
@ ./node_modules/data-forge/build/lib/dataframe.js 4377:21-34 4401:17-30 4433:21-34 4457:17-30
@ ./node_modules/data-forge/build/index.js
@ ./node_modules/ts-loader!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/ComplexRoiSelector.vue
@ ./src/ComplexRoiSelector.vue
@ ./src/index.ts
My code literally just tries to import dataforge and instantiate a dataframe with no data.
` .dropSeries(['id', 'open', 'high', 'low', 'volume', 'startTime']) // Drop certain columns.
^
TypeError: dataForge.readFile(...).parseCSV(...).dropSeries is not a function`
My code looks like:
`const dataForge = require('data-forge');
require('data-forge-fs');
require('data-forge-plot');
const df = dataForge.readFile('binance_btc_usdt_time_5m.csv')
.parseCSV()
.dropSeries(['id', 'open', 'high', 'low', 'volume', 'startTime']) // Drop certain columns.
.then(df => {
console.log(df.toString())
})
.catch(err => {
console.log(err)
});`
Hey.
I'm currently struggling to aggregate my data right. I got the following example data in my CSV file (abstracted file):
10 | 20190404141501 | 2,display_name:server@coolbox,vcpus:1,ram:8096 |
Now when reading and working with the Data from that file, the third column gets seperated aswell, since data-forge
thinks, that ,
is also a seperator. Is there an option, where I can set the seperator, that's used in the file? I haven't found anything in the Guide or the API Documentation.
Hello,
I have a very basic problem. As I am seasoned Pandas (python) programmer, I wanted to try out data-forge DF the moment I saw this one. Thank you so much for bringing this wonderful piece, and it had a lot of potential being part of JS echo system...
But I had a basic starting thing to over come, and do not find that been talked in any of the blog/doc that I went through. Surprising. here is my problem -
I constructed a DF using an array of objects.
var df = new DataFrame( deposit_history );
Then I tried to filter like this:
var df2 = df.where( row => row.status === 'PG_FAILED' );
Then I check what the content of df2 is... it is only null.
DataFrame {configFn: , content: null}
Later (after spending lot of time), I tried .toArray(), then I check df2 again,
DataFrame {configFn: , content: Object}
This behavior looks so odd to me, and maybe I am thinking like the 'where()' alone does not construct the resultant DF, and it looks like there is a method (probably like 'execute') is to be invoked after where. That I am missing. In the absence of that hypothetical method 'execute', when one calls toArray() that really executes the query-plan. Is that what us really going on. Would greatly appreciate if the right way is explained. Thanks in advance.
PS: here is one of the ways Python works:
df[~df.order_type.str.contains('fb|foc|nfr|other')]
or
df[ df['std_product_class'].str.contains('TB', case=False) ]
Regards,
Tharma
Unlike Pandas, which would always preserve the original index, data-forge seems to always create a new one, unless a few boilerplate transformations get applied. Based on a couple of years of work with Pandas, I personally find this a lot of unnecessary work. Almost all the time, when creating a new series out of a windowed version of an existing one, I would attach it as a new column to the existing data frame. This makes it easy to inspect the entire data frame down the line, comparing the values in every row.
In my opinion, preserving the original index is essential for the ease of work with data-forge. Of course, this would also require handling of NaNs, their cleaning, etc.
I would be interested to know what you think about this.
Hi,
I am loading a CSV file with the fromCSV function. When I run detectTypes() on the dataframe at that time, I see something like:
0 string 8.1 phone
1 object 91.9 phone
DataForge is treating empty values / nulls as an object.
After running a transform such as below to uppercase a value, there are some type differences.
dataFrame.transformSeries({ [sourceColumn]: value => value && value.toLocaleUpperCase() })
Running detectTypes() again generates:
0 string 8.1 phone
1 undefined 91.9 phone
Even explicitly returning a null such as:
dataFrame.transformSeries({ [sourceColumn]: value => value ? value.toLocaleUpperCase() : null })
returns an undefined.
It seems there is an inconsistency between fromCSV dynamic typing parsing and transformSeries handling nulls and/or undefined.
When using const df=dataForge.readFile('SalesJan2009.csv').parseCSV();
in the render method of a ReactJS class the browser shows an error
TypeError: data_forge__WEBPACK_IMPORTED_MODULE_8__.readFileSync is not a function
Hi,
I'm trying to read in a csv file that has about 4500 columns. On the .parseCSV method I am getting this error (I've parsed other CSV files I have and it works fine - suspecting there may be an issue with the number of columns??):
testDataForge Error Error: "toString()" failed
at stringSlice (buffer.js:558:43)
at Buffer.toString (buffer.js:631:10)
at Object.fs.readFileSync (fs.js:601:41)
at SyncFileReader.parseCSV (D:\Applications\ModernAnalytics\TestDataForge\node_modules\data-forge-fs\build\index.js:324:40)
at __dirname (D:\Applications\ModernAnalytics\TestDataForge\testDataForge.js:7:76)
at Object. (D:\Applications\ModernAnalytics\TestDataForge\testDataForge.js:26:3)
at Module._compile (module.js:635:30)
at Object.Module._extensions..js (module.js:646:10)
at Module.load (module.js:554:32)
at tryModuleLoad (module.js:497:12)
Thanks!
Rob
Just installed DFN, tried to run the intro notebook and a big red dialog announced:
Project setup failed, see log file for full details.
Error: spawn[1]: npm ERR! code E403
npm ERR! 403 Forbidden: @types/request-promise@^4.1.44
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/rwilliams/Library/Application Support/data-forge-notebook/npm-cache/_logs/2019-11-15T05_51_22_276Z-debug.log
Error code: 1
Relevant log entry:
24 verbose stack Error: 403 Forbidden: @types/request-promise@^4.1.44
24 verbose stack at fetch.then.res (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/pacote/lib/fetchers/registry/fetch.js:42:19)
24 verbose stack at tryCatcher (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
24 verbose stack at Promise._settlePromiseFromHandler (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:512:31)
24 verbose stack at Promise._settlePromise (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:569:18)
24 verbose stack at Promise._settlePromise0 (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:614:10)
24 verbose stack at Promise._settlePromises (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:693:18)
24 verbose stack at Async._drainQueue (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:133:16)
24 verbose stack at Async._drainQueues (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:143:10)
24 verbose stack at Immediate.Async.drainQueues [as _onImmediate] (/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:17:14)
24 verbose stack at runCallback (timers.js:705:18)
24 verbose stack at tryOnImmediate (timers.js:676:5)
24 verbose stack at processImmediate (timers.js:658:5)
25 verbose cwd /Applications/Data-Forge Notebook.app/Contents/examples
26 verbose Darwin 18.7.0
27 verbose argv "/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/bin/node" "/Applications/Data-Forge Notebook.app/Contents/nodejs/v10.15.3/lib/node_modules/npm/bin/npm-cli.js" "install" "--bin-links" "false" "--scripts-prepend-node-path" "true" "--cache" "/Users/rwilliams/Library/Application Support/data-forge-notebook/npm-cache" "--prefer-offline" "--no-audit"
28 verbose node v10.15.3
29 verbose npm v6.4.1
30 error code E403
31 error 403 Forbidden: @types/request-promise@^4.1.44
32 verbose exit [ 1, true ]
want the math function of add and multiply and other operations of the data frame like python pandas
Apologies in advance as I'm pretty inexperienced with JavaScript!
I had an issue reading a CSV file where columns of floats were being read in as strings. This proved frustrating when trying to sum()
a Series, as it just returned one big concatenated string.
It might be useful to add a method to the Series class to change the data type, similar to pandas.
Maybe something akin this, which I just wrote for a project, though I'd assume you want to lazily evaluate this as well:
/*
* Returns a new Series whose values are coerced to the specified data type
*/
function coerce_data_type(series,data_type) {
var coerced_series;
switch (data_type) {
case 'number':
coerced_series = series.toArray().map(value => Number(value));
break;
case 'boolean':
coerced_series = series.toArray().map(value => Boolean(value));
break;
case 'string':
coerced_series = series.toArray().map(value => String(value));
break;
default:
// TODO - error handling
coerced_series = series.toArray();
break;
}
return new dataForge.Series(coerced_series);
}
i want to have a function that can reindex the dataframe
something like the pandas.DataFrame.reindex
Ashley, thanks for your awesome work on everything. I'm new to JavaScript and I'm not sure if my issue is related to something I'm doing or if there's an issue.
I'm having an issue with writing to CSV after performing an outer join. I've been able to verify that my data frames are being created. When I display head as pictured below the process is a little slow, but writing to CSV takes minutes to complete. I originally thought it wasn't writing, but it does seem to write after some period of time. Additionally, the script seems to hang and I don't return to the command line.
`const exceptDF = cleanRev.joinOuter(cleanSF,
cleanRev => cleanRev.sfKey,
cleanSF => cleanSF.sfKey,
(cleanRev, cleanSF) => {
return {
index: cleanRev ? cleanRev.sfKey : cleanSF.sfkey,
swanKey: cleanRev ? cleanRev.sfKey : undefined,
sfKey: cleanSF ? cleanSF.sfKey : undefined
};
}
);
console.log(exceptDF.head(3).toString()); //this works, but it's slow
exceptDF.asCSV().writeFile('exceptDF.csv'); //this writes the file, but it takes several minutes and I don't return to the command line in the terminal`
For reference, I'm loading two csv files to different data frames, doing some manipulation, and then performing the outer join. Each file has around 4,000 rows and 20 columns.
Thanks for any input.
Would be helpful if you could include browser version (for include in script tag) for convenience. Thanks for your consideration. Nice Work!!!
First and foremost, I do not think this is a fault in data-forge
but I feel like I should raise the issue here so others and author(s) are aware of it
I have the following code:
const dataForge = require('data-forge');
const util = require('util');
//const c = require('collections/fast-map');
function test() {
const timestamps = [ [ '2018-05-21 15:38:04' ],
[ '2018-05-21 15:38:09' ],
[ '2018-05-21 15:38:09' ],
[ '2018-05-21 15:38:08' ] ];
const tsDataFrame = new dataForge.DataFrame({
columnNames: ['Timestamp'],
rows: timestamps
}).setIndex('Timestamp');
let groupedDf = tsDataFrame.groupBy(row => row.Timestamp).select(tsGroup => ({
Timestamp: tsGroup.first().Timestamp,
QPS: tsGroup.count()
})).inflate();
console.log(groupedDf.toString());
}
test();
Running the above code will give me the following result
__index__ Timestamp QPS
--------- ------------------- ---
0 2018-05-21 15:38:04 1
1 2018-05-21 15:38:09 2
2 2018-05-21 15:38:08 1
which is what I expected
However if //const c = require('collections/fast-map');
is uncommented, running it again I will get
__index__ [object Object] false
--------- --------------- -----
0
1
2
Clearly this is a mistake. After hours of debugging I can at least spot one possible reason for the error. In the build version of data-forge
within DataFrame.prototype.toString
function we have the following (cut down for brevity's sake)
DataFrame.prototype.toString = function () {
var columnNames = this.getColumnNames();
var header = ["__index__"].concat(columnNames);
// more things down below
};
Doing console.log(columnNames)
with collectionsjs
required gives me the following:
[ SelectIterable {
iterable: [ [Object], [Object], [Object] ],
selector: [Function] },
false ]
Without collectionsjs
I will get the expected result: [ 'Timestamp', 'QPS' ]
Inspecting further the getColumnNames
function tells me that Array.from
is used which is overridden by collectionsjs
implementation: https://github.com/montagejs/collections/blob/master/shim-array.js#L26
I managed to fix things on data-forge
side by doing a seemingly unnecessary function call:
let groupedDf = tsDataFrame.groupBy(row => row.Timestamp).select(tsGroup => ({
Timestamp: tsGroup.first().Timestamp,
QPS: tsGroup.count()
})).inflate().resetIndex();
This will give me the correct result regardless if collectionsjs
was used or not
There's an issue raised already in collectionsjs
regarding Array.from
montagejs/collections#169 and there is also a PR montagejs/collections#173. I'm not sure about the progress of either
Can you tell me what's the equivalent of Pandas value_counts()
in data-forge?
Readme suggests the following code to write a DataFrame out as a CSV file.
dataFrame
.asCSV()
.writeFileSync("output.csv");
But this method doesn't allow any option to set whether we want to include the headers or not.
Looking at the source code I see there's a DataFrame.toCSV(options?: ICSVOutputOptions) which takes an options
argument which can be used to set header: false
and exclude the headers.
It's not clear why there are 2 different methods, toCSV()
and asCSV()
(which is actually in data-forge-fs
). My guess is that the first one is used to just get a string representation and the former is optimized to be written out as a file.
Assuming my guess is correct, shouldn't asCSV()
also accept an ICSVOutputOptions
parameter?
I have an array of the same column names. (* actually different)
Example : city and CITY
DataFrame.initColumnNames function recognizes these two as the same. (reason toLowerCase() method )
Result : city.1 and CITY.2
When i replace return outputColumnNames;
with return inputColumnNames;
, it works.
It would be nice, if it was presented as an option .
Do you have any other suggestions for this ?
Thanks in advance.
Since yesterday it is a security incident (https://npmjs.com/advisories/1515). Can you please update the project dependencies of papaparse.
hi Ashley,
I'm building an angular 6 application recently and was looking for some data processing lib. So happy to find data-forge as I used pandas and it's a great lib on python.
Would you mind put some instruction on how to import data-forge into angular to be used there?
Hi.
With the split to data-forge
and data-forge-fs
, I'm not sure how to turn an array of JS Objects I have in memory into a CSV file.
fromObject()
is defined on data-forge
but asCSV()
and writeFileSync()
are defined in data-forge-fs
.
Example data structure
const arr = [
{ first: 'micky', last: 'mouse' },
{ first: 'minnie', last: 'mouse' }
];
// How do I turn `arr` into a CSV file?
// Do i import `data-forge` or `data-forge-fs` or both?
Thank you.
Hi, this is related to #11. I'm opening a new issue since I don't have permission to reopen the previous one and I'm not sure if the issue is in toPairs
or the use of fillGaps
and rollingWindow
. The issue is very slow performance with from toPairs
. Copying from my comment in the other issue.
I just pushed up a change to our test repo. The changes are:
toPairs()
.The tests I'm looking at are method-1.js and method-2.js. The only difference between them is:
$ diff method-1.js method-2.js
76c76
< const mySeries = dfWithoutGaps.getSeries('value');
---
> const mySeries = new dataForge.Series(dfWithoutGaps.getSeries('value').toArray());
Output from running the tests:
cberthiaume@slow-lane:~/data-forge-performance-test-issue-11$ node method-1.js
Time to require: 980.9740000000002
Time to create DataFrame and getSeries: 8.874000000000024
Time for rolling window: 0.08599999999978536
Time for toPairs: 2067.8320000000003
cberthiaume@slow-lane:~/data-forge-performance-test-issue-11$ node method-2.js
Time to require: 975.4
Time to create DataFrame and getSeries: 63.06100000000015
Time for rolling window: 0.10500000000001819
Time for toPairs: 17.16599999999994
cberthiaume@slow-lane:~/data-forge-performance-test-issue-11$
The key difference is the huge difference in time to call toPairs()
. Our use case requires us to call toPairs()
and the only way to get acceptable performance when doing that is to recreate the series as you see in the diff above. However, our needs have changed that the slow down required to implement this workaround is becoming a bottleneck. Is there a a better way to get good performance from toPairs()
without using this workaround? Should I open a separate ticket to track this?
Thanks again for all your help.
Hello,
Just noticed the API docs are down and I haven't been able to figure out how to do two things... Hoping you can provide some guidance. Happy to contribute some examples and documentation once I understand this better.
First, is it possible to group by more than one column? I am trying this:
// ...
.groupBy(row => [row.coin,row.type])
// ...
And it seems to be working. Is that right?
Then, I want to grab the latest of potentially several values per day so I was trying .max
like so...
var latest = indexed
.orderBy(row => row.timestamp)
.groupBy(row => [row.coin,row.type])
.select(group => ({
coin: group.first().coin, // I'm assuming this returns the first of the series
type: group.first().type,
timestamp: group.select(row => row.timestamp).max(), // <-- this
value: group.first().value,
}))
.withIndex(row => moment.unix(row.timestamp/1000).format('YYYY-MM-DD'))
.inflate();
Sorry to bother with this... I keep trying the docs but :s
Thank you so much!
Here is my code:
var transformedData = await new df.DataFrame(data) //data is an array of JSON objects: [{a:1}, {a:3}...]
console.log(transformedData.getColumnNames()) //print out all column Names
for (const column in transformedData.getColummns()) { //not a function
console.log("Column name: ");
console.log(column.name);
console.log("Data:");
console.log(column.series.toArray());
}
This is what i get:
"errorMessage": "transformedData.getColummns is not a function",
"errorType": "TypeError"
Why getColumnNames() works and getColummns() doesn't?
Would it be possible for this package to support .json5
files?
I'm trying to use DF on a JSON5 file but even tho I already have a module that turns that file to a JSON object, there's no way for me to use this package to read either the file or read from the object.
Sticking to JSON isn't an option (as comments are needed).
A workaround would be to save the JSON object to a JSON file and read it again but I'm trying to minimise I/O operations and resources used, so it's not a good option.
Hi,
I cannot access the implementation examples referred in the API docs. Is it down?
Please include a ES Module version of the package so users using bundlers like webpack can benefit from it and get features like tree-shaking, etc.
Right now the package size is quite high
Is there a better way to rename columns in a DataFrame
than the following code? I could not find any information for renaming columns in the Guide or API doc. In this code, I'm trying to rename columns "Like This" to "like_this".
slugify = (s) => s.toLowerCase().replace(' ','_').replace('/','_or_')
rawData =
DataForge.readFileSync('data.csv')
.parseCSV({ dynamicTyping: true })
for(let name of rawData.getColumnNames()) {
let s = rawData.getSeries(name)
rawData = rawData.dropSeries(name).withSeries(slugify(name), s)
}
Hi,
Thank you for a great library. I was looking for something like this and read about it in the latest issue of Node Weekly. Started playing with it but haven't been able to get the result I'd like. I hope you don't mind if I ask...
I have the following dataframes:
__index__ A | __index__ B
--------- -- | --------- --
0 A1 | 2 B1
1 A2 | 3 B2
2 A3 | 4 B3
3 A4 | 5 B4
4 A5 | 6 B5
I need to end up with:
__index__ A B
--------- -- --
0 A1
1 A2
2 A3 B3
3 A4 B4
4 A5 B5
5 B5
6 B5
The data comes from a bunch of files that contain one 2D array each structured like this:
// A.json | // B.json
[ | [
[0, A1], | [2, B1],
[1, A2], | [3, B2],
[2, A3], | [4, B3],
[3, A4], | [5, B4],
[4, A5] | [6, B5]
] | ]
Notice how I need the resulting DataFrame to use the file names as the column titles.
I tried using concat and joins but don't quite get this result. Would you mind pointing me in the right direction?
Thank you,
Can the library have a way of optimizing (or disabling) chai assertions throughout dataframe and series?
I did some browser (Chrome) profiling and the assertions seemed to be a major factor, so I did some timings.
I can't give you a standalone example, but for basic context my two cases were
[1] a 5000 row table that is aggregated down to about 200 rows: (30ms to parse, 200ms to aggregate),
and [2] a wider 2000 row table that is aggregated down twice: (200ms to parse, 600 ms to aggregate).
After adding the following code to my start up (only change):
import _ from "lodash";
import { assert } from "chai";
const chaiIsFunction = assert.isFunction;
const chaiIsArray = assert.isArray;
const chaiIsNumber = assert.isNumber;
const chaiIsString = assert.isString;
const chaiIsObject = assert.isObject;
assert.isFunction = (o, msg) => { if (!_.isFunction(o)) chaiIsFunction(o, msg); };
assert.isArray = (o, msg) => { if (!_.isArray(o)) chaiIsArray(o, msg); };
assert.isNumber = (o, msg) => { if (!_.isNumber(o)) chaiIsNumber(o, msg); };
assert.isString = (o, msg) => { if (!_.isString(o)) chaiIsString(o, msg); };
assert.isObject = (o, msg) => { if (!_.isObject(o)) chaiIsObject(o, msg); };
the timings went down to [1] (30ms to parse, 80ms to aggregate) and [2] (200ms to parse, 190ms to aggregate).
(By the way, these second timings didn't really improve if I completely disabled the assertions with
assert.isFunction = () => undefined;
assert.isArray = () => undefined;
assert.isNumber = () => undefined;
assert.isString = () => undefined;
assert.isObject = () => undefined;
)
My tests were done in a development build, so maybe this isn't as much of a factor in production, but chai seems like a
library designed for testing and not necessarily for production performance (at least I can't see any mention of a
particular production mode in their docs).
Thanks.
When a date index is set, whether using .setIndex() or explicitly when setting up a dataframe, the index cannot be used to get a value using that index.
e.g. dataframe.test.ts
df.at(moment("2014-04-05").toDate())
will returnundefined
while df.at(testDate)
will pass the test correctly
I was wondering if the data-forge 1.2.4 is compatible with IE 11? I've tried using polyfill io and importing the default, es5 and es5 to cover the Symbol usage but for some reason it hangs in iterable.js or iterator.js. Seems to never hit the iterator.prototype[Symbol.iterator] section. Using the polyfill io, I had no issues with putting the data into Dataframe but trying to run the subset function hangs.
Hello,
thanks for this awesome project that hopefully will help me to dump python.
I'm using it to read a csv file, cleaning the data and inserting into postgres.
To deal with huge csv file, I'd like to pipe the rows as they are parsed into postgres without having to keep the full dataframe into memory.
How would you do that ? Using through ?
Thanks
Hi, I recently upgraded from data-forge-js to data-forge-ts and I'm noticing the following issue (not sure if it was the same in data-forge-js). I have a DataFrame that I want to call fillGaps on. However, the performance was surprisingly slow. After much debugging having believed it was due to something else in our application I believe I narrowed it down to data-forge-ts. Here is an example that should demonstrate the problem and my current workaround:
const data = [
{ date: moment().toDate(), value: 1 },
{ date: moment().add(1, 'days').toDate(), value: 3 },
{ date: moment().add(5, 'days').toDate(), value: 2 },
{ date: moment().add(6, 'days').toDate(), value: 6 },
{ date: moment().add(7, 'days').toDate(), value: 5 },
{ date: moment().add(12, 'days').toDate(), value: 2 },
{ date: moment().add(15, 'days').toDate(), value: 9 },
];
const df = new dataForge.DataFrame(data).setIndex('date');
const gapExists = (pairA, pairB) => {
// Return true if there is a gap longer than a day.
// Log when we enter this function
console.log('gapExists [pairA, pairB]', [pairA, pairB]);
const startDate = pairA[1].date;
const endDate = pairB[1].date;
const gapSize = moment(endDate).startOf('day').diff(moment(startDate).startOf('day'), 'days');
return gapSize > 1;
};
const gapFiller = (pairA, pairB) => {
const startDate = pairA[1].date;
const endDate = pairB[1].date;
const gapSize = moment(endDate).startOf('day').diff(moment(startDate).startOf('day'), 'days');
const numEntries = gapSize - 1;
const newEntries = [];
for (let entryIndex = 0; entryIndex < numEntries; entryIndex += 1) {
const newValue = { date: pairA[1].date, value: pairA[1].value };
newValue.date = moment(pairA[1].date).add(entryIndex + 1, 'days').toDate();
newEntries.push([
moment(pairA[0]).add(entryIndex + 1, 'days').toDate(), // New index
newValue, // New value
]);
}
return newEntries;
};
// In this case, the final value of dfWithoutGaps is
// what you'd expect and the number of entries into
// gapExists is about what I'd expect.
//
// I say "about" what I'd expect because there are
// two seemingly identical comparisons made between the
// first two items of the data set for a total
// execution count of 7 when I'd expect 6.
const dfWithoutGaps = df.fillGaps(gapExists, gapFiller);
console.log('dfWithoutGaps', dfWithoutGaps.toArray());
// With the following line uncommented the rollingWindow
// call below will trigger a huge number of calls to
// gapExists (I'm not sure how many but enough to max out the
// buffer in my browser's debug console). It appears as if
// fillGaps is being run for each rolling window.
//
// The speed of generating the rolling window in this
// scenario is very slow.
// const mySeries = dfWithoutGaps.getSeries('value');
// If we force the lazy evaluation via the line below and then
// run rollingWindow we'll get the expected number of calls (6) to
// gapExists.
//
// The speed of generating the rolling window in this scenario
// is very fast.
// https://github.com/data-forge/data-forge-ts/blob/master/docs/guide.md#lazy-evaluation-through-iterators
const mySeries = new dataForge.Series(dfWithoutGaps.getSeries('value').toArray());
const smaPeriod = 3;
const smaSeries = mySeries
.rollingWindow(smaPeriod)
.select(window => window.sum() / smaPeriod);
console.log('smaSeries', smaSeries.toArray());
It's late here so I could be off the rails somewhere. Does this look like an issue with data-forge-ts? If so, is there a better workaround than what I have here?
Thanks a lot in advance. This is a very helpful tool and I appreciate you creating it!
Is there no way to edit cell values directly, similarly to how it's done in Pandas?
Example: df['name1']['name2'] = 10
I haven't found anything relevant in the documentation, so I assume there is no function for this. So then the question is, is there any way to modify data frames without having to rebuild whole frames/series every time a modification is done?
Thank you for the library
In the documentation: https://github.com/data-forge/data-forge-ts/blob/master/docs/guide.md#group-and-aggregate I found several issues
First thing is that the example that should be looked at is not #13 but #12. Secondly the code in the example is not actually working with data-forge v1.0.10
. Running the example gives me the following error
$ > node examples/12.\ Group\ and\ aggregate\ sales\ data/index.js
-- Output dataframe:
/Users/btara.truhandarien/Code/data-forge-examples-and-tests/examples/12. Group and aggregate sales data/index.js:47
Sales: group.select(row => row.Sales).sum(),
^
TypeError: group.select(...).sum is not a function
at SelectIterator.salesData.groupBy.select.group [as selector] (/Users/btara.truhandarien/Code/data-forge-examples-and-tests/examples/12. Group and aggregate sales data/index.js:47:47)
at SelectIterator.next (/Users/btara.truhandarien/Code/data-forge-examples-and-tests/node_modules/data-forge/build/lib/iterators/select-iterator.js:20:25)
at ColumnNamesIterator.next (/Users/btara.truhandarien/Code/data-forge-examples-and-tests/node_modules/data-forge/build/lib/iterators/column-names-iterator.js:58:50)
at Function.from (native)
at DataFrame.getColumnNames (/Users/btara.truhandarien/Code/data-forge-examples-and-tests/node_modules/data-forge/build/lib/dataframe.js:337:22)
at DataFrame.toString (/Users/btara.truhandarien/Code/data-forge-examples-and-tests/node_modules/data-forge/build/lib/dataframe.js:1917:32)
at Object.<anonymous> (/Users/btara.truhandarien/Code/data-forge-examples-and-tests/examples/12. Group and aggregate sales data/index.js:53:24)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
I find that modifying the aggregation to be
var summarized = salesData
.groupBy(row => row.ClientName)
.select(group => ({
ClientName: group.first().ClientName,
// Sum sales per client.
Sales: group.select(row => row.Sales).deflate().sum(),
}))
.inflate() // Series -> DataFrame.
;
Works as expected
A feature request for consideration ... including transforms for wide-to-long and long-to-wide data structures. For example: https://sejdemyr.github.io/r-tutorials/basics/wide-and-long/
Thanks for considering.
Thanks for this awesome DataFrame library for nodejs, much wanted!
Indexing and using at
function is quite slow with large DataFrames . at
looks like very useful in terms of improving performance. Any workaround until at
is fixed for its real intention? A fix will be much helpful if possible. Sorry that i am not aware of the internal working of a DataFrame for a PR
How to get equivalent of pandas.DataFrame.pivot_table like here
table = pivot_table(df, values=['D', 'E'], index=['A', 'C'], aggfunc={'D': np.mean, 'E': [min, max, np.mean]})
Output:
D E
mean max median min
A C
bar large 5.500000 16 14.5 13
small 5.500000 15 14.5 14
foo large 2.000000 10 9.5 9
small 2.333333 12 11.0 8
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.