Comments (6)
There are probably some tricks we can try like using a static lookup table instead of branches:
I tried that, but it did not improve the situation on my machine.
I did some other refactoring that brought down the benchmark's runtime by 20% on my machine for s/us/ns.
There are also code paths that the benchmark is not really testing (those I saw a similar improvement from my changes): s/ms/us/ns values so large that the "day" parsing part of the code gets triggered as well as negative offsets, both don't get checked because the benchmark runs only with random numbers between 0 and 10.
from pandas.
I now have a version of the date time conversion that is correct and tested and runs up to 2.5x faster than main and the previous version before the regression (at least on my machine, M2 ARM). Interestingly, I do not see large differences in runtimes between the pre regression commit and main.
I would suggest changing the benchmark inputs to negative values that are at least on the order of one day to make most of the logic in the function actually show up in the runtime of the benchmark. What do you think? Is there a procedure to follow to update the history of benchmark runtimes?
To back that up, here are some of my benchmark results for all three versions, I will update my PR soon:
input range | version | main | main pre regression | fix |
---|---|---|---|
+30±10ns | 4.9 | 4.8 | 3.9 |
-30±10ns | 7.9 | 7.2 | 5.2 |
±30ns | 8.4 | 8.5 | 7.1 |
+4d±100s | 11.6 | 11.4 | 5.2 |
-4d±100s | 12.7 | 12.7 | 5.3 |
±4d | 9.7 | 9.4 | 7.5 |
(Runtimes are in ms, runs are for 10^6 random values)
The current versions of the function have branch prediction and variable dependency issues that only show up when using larger ranges of inputs and negative values. The previously mentioned checks against the base unit size are of very low cost in reality since the unit size does not change (in the benchmark, but also in real world) and get branch predicted without failure.
Randomly negative and positive inputs still do not do all too well in my version, but realistically timestamps would not have this property.
from pandas.
Possibly related?: #57035
Good find, but the regression here should be unrelated to that plotting issue. Converting 1M timestamps takes some milliseconds, but the referenced regression is in the seconds range.
from pandas.
I think the issue here is that we combined a few case statements into one:
case NPY_FR_s:
case NPY_FR_ms:
case NPY_FR_us:
case NPY_FR_ns: {
npy_int64 per_sec;
if (base == NPY_FR_s) {
per_sec = 1;
} else if (base == NPY_FR_ms) {
per_sec = 1000;
} else if (base == NPY_FR_us) {
per_sec = 1000000;
} else {
per_sec = 1000000000;
}
...
Whereas previously the code was copy/pasted and slightly tweaked for every case statement.
There are probably some tricks we can try like using a static lookup table instead of branches:
case NPY_FR_s:
case NPY_FR_ms:
case NPY_FR_us:
case NPY_FR_ns: {
const npy_int64 sec_per_day = 86400;
static npy_int64 per_secs[] = {
1, // NPY_FR_s
1000, // NPY_FR_ms
1000000, // NPY_FR_us
1000000000 // NPY_FR_ns
};
const npy_int64 per_sec = per_secs[base - NPY_FR_s];
but I'm not really sure the tricks are worth it. And I think the code de-duplication is probably the increased timing here
from pandas.
@dontgoto thanks for the great research. If you have improvements to the benchmark I would just go ahead and submit it - we don't manage the history of changes to a benchmark itself that strictly
from pandas.
Possibly related?: #57035
from pandas.
Related Issues (20)
- BUG: `Series.combine_first` replaces `None` with `NaN` at index not provided by `other` HOT 3
- BUG: in agg function HOT 1
- BUG: Rounding timedelta to 3 decimal places does not work correctly and is unstable. HOT 2
- BUG: pd.read_sql() converts string "NA" to None HOT 1
- API: timestamp resolution inference - default to one unit (if possible) instead of being data-dependent?
- DOC: punctuation of titles in Titanic data HOT 1
- BUG: `unstack` with `sort=False` scrambles data HOT 2
- BUG: pandas.read_parquet fails to read parquet files with "file://" prefix on Windows HOT 2
- BUG: csv date inconsistencies HOT 2
- Potential performance regression with "BUG: DateTimeIndex.is_year_start unexpected behavior when constructed…" HOT 1
- BUG: output of pandas.util.hash_pandas_object() changed in pandas 2.2.x HOT 2
- BUG: docker build on windows fails; need eol=lf in .gitattributes HOT 2
- BUG: None value in MultiIndex can't be found
- BUG: HDFStore doesn't save datetime64[s] right
- ENH: Pivot Table HOT 1
- ENH: Pandas Tensor Data Type HOT 1
- BUG: Accessing a row with loc recasts dtype and can cause values to change depending on other columns HOT 2
- BUG: Reading large CSV files with `pyarrow` when values contain newline character. HOT 1
- DOC: Link to SPEC 0 instead of NEP 29
- BUG: `to_parquet(engine="fastparquet")` can't handle DataFrame with NumPy array values
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pandas.