s-leroux / fin Goto Github PK
View Code? Open in Web Editor NEWSet of tools for personal investment
License: MIT License
Set of tools for personal investment
License: MIT License
Line 7 in 9016342
It is confusing the have the index as a column separated from the data columns.
A different approach would be to handle the index data as an ordinary column but use the index itself as an indirection to retrieve the rows in a given order. The extra level of indirection should be negligible for the C/Cython code. For Python code, this will require an extra level of __getitem__
.
Implement the necessary code and documentation to cover the following use case:
We want to know the average number of consecutive days BTC closes below its SMA50D.
A strategy simulator was written for the pre-0.1.0 version.
Review, check, and document the simulator to work with the current version.
Serie.from_csv()
should warn if a cell does not contain valid data.
Maybe Serie.from_data()
should do the same.
Lines 156 to 182 in d79b4f0
Optionally, we may consider adding a keyword parameter to specify the authorized "n/a" values that will be converted to None/NaN.
Add an API interface to download historical and intraday data from Binance:
https://data.binance.vision/
Since the addition of formatters, columns are implicitly typed. However, this may lead to implementation problems like those in series_from_csv
, where the formatter-related code looks like a hack.
Consider explicitly typing the columns instead. The exact formatter, if not provided, would then be inferred from the type and possibly the data (i.e., to auto-detect the precision for floating-point numbers)
We should change the Makefile to run the code snippets in docs/snippets
and check their output is unchanged.
We should make it easy to calculate values on a complete column.
Currently, if you want the mean of a column in a series, you have to write:
from fin.seq.serie import Serie
from fin.seq import fc, ag
ser = Serie.from_csv_file(
"tests/_fixtures/MCD-20200103-20230103.csv",
format="dnnnnni"
).group_by(
fc.constant(True),
(ag.first, "Date"),
(ag.avg, "Open", "Close"),
)
print(ser)
avg_open = ser["Open"].columns[0][-1]
print(avg_open)
We can clearly do better. This may imply:
We may leverage aggregate functions for that.
Ideally, aggregate functions may also be usable as window functions. This would be less efficient than specifically designing code, but it would avoid code duplication.
Currently, the problem is that aggregate functions are designed to apply to a set of columns rather than individual columns for efficiency reasons. Notice the for col in cols
list comprehension in the code below:
Lines 25 to 30 in 6ba6041
Check if we can return a t-expr from a function. This would permit to define high-level functions from lower-level blocks:
def sma(rowcount, col):
return (div, (sum, col), rowcount)
Is this a desirable feature?
[...] it it is rather confusing to omit the
DATE
column in theselect
clause because it is implied, while it is mandatory ingroup_by
.
Originally posted by @s-leroux in #27 (comment)
Lines 52 to 56 in 1d7dbf8
The default output format should be tabular, not csv.
An option would add (if feasible?) a second parameter to the Serie.__str__
method.
Both with RandomSolver
and ParticleSwarmSolver
, when an equilibrium function gains significantly more weight than the other, the constraint solver fails to find an optimal solution.
We should provide a solution to roughly normalize the different constraints to evaluate close to the [-1, +1] range.
One path to improve performance is by using Cython to perform calculations natively without the Python float overhead.
As an experiment, we may try to implement column.Column
with Cython to store values as a native array of float.
Provide an interface to financialmodelingprep.com's API
When using unsolvable constraints, we might hope for the solver to find the "best" answer in the domain range.
Here, we obtain many zero values:
Score 0.0
Maximum loss : 0.0
Take profit : 0.0
Entry price : 0.0
Stop price : 0.0
Quantity : 8.204826221431059
The example above was produced by the following code:
"""
Evaluate a position size, stop price, and take profit from a maximum loss
and entry price.
Usage:
PYTHONPATH="$PWD" python3 examples/fin/model/pricing.py
"""
from fin.model.complexmodel import ComplexModel
UNKNOWN = (0, 1000)
QTY = (1,10)
ENTRY_PRICE = 0.9024
STOP_PRICE = 0.8614
TAKE_PROFIT = 0.9498
MAX_LOSS = 21
def fees(price):
return price*0.1/100
def balance(debit, credit):
return (credit-fees(credit))-(debit+fees(debit))
model = ComplexModel()
eq1 = model.register(
lambda entry, stop, qty, maxloss : balance(entry*qty, stop*qty) - -maxloss,
dict(name="entry", description="Entry price", domain=ENTRY_PRICE),
dict(name="stop", description="Stop price", domain=STOP_PRICE),
dict(name="qty", description="Quantity", domain=QTY),
dict(name="maxloss", description="Maximum loss", domain=MAX_LOSS),
)
eq2 = model.register(
lambda entry, stop, qty, tp : balance(entry*qty, stop*qty)*2 + balance(entry*qty, tp*qty),
dict(name="entry", description="Entry price", domain=ENTRY_PRICE),
dict(name="stop", description="Stop price", domain=STOP_PRICE),
dict(name="qty", description="Quantity", domain=QTY),
dict(name="tp", description="Take profit", domain=TAKE_PROFIT),
)
model.bind(eq1, "entry", eq2, "entry")
model.bind(eq1, "stop", eq2, "stop")
model.bind(eq1, "qty", eq2, "qty")
params, domains, eqs = model.export()
# from pprint import pprint
# pprint(params)
# pprint(domains)
# pprint(eqs)
from fin.model.solvers import ParticleSwarmSolver
solver = ParticleSwarmSolver(5,300)
score, result = solver.solve(domains, eqs)
print(f"Score {score}")
for param, value in zip(params, result):
print(f"{param['description']:20s}: {value}")
Line 483 in 9fafc76
This was designed like that because of dates. When plotting financial graphs, the usage is to ignore non-trading days (WE, holidays, ...)
Ideas:
We should add the Kelly's Criterion to the predefined models available in the fin.model
package.
Lines 384 to 392 in 16ec607
The Column.from_sequence
method seems to have different purposes depending on the value of the convert
parameter.
We may distinguish between from_sequence
, which creates a column from Python objects, and import
, which creates a sequence from the textual representation of values.
We should implement the drawdown
function.
A strategy suffers a drawdown whenever it has lost money recently.
A drawdown at a given time t is defined as the difference between
the current equity value (assuming no redemption or cash infusion)
of the portfolio and the global maximum of the equity curve occur-
ring on or before time t. The maximum drawdown is the difference
between the global maximum of the equity curve with the global
minimum of the curve after the occurrence of the global maximum
(time order matters here: The global minimum must occur later
than the global maximum).
— Quantitative Trading (Chan) ,p21
Line 211 in 9016342
We should raise a TypeError
here, not a SystemError
.
Implement Heikin-Ashi to create more readable candlestick charts.
From https://www.investopedia.com/trading/heikin-ashi-better-candlestick/:
Consider a custom container for storing column's Python values.
Column's data storage has unique requirements that are not completely satisfied by the standard tuple
type. We may consider a custom container with the following features:
Remove extraneous comments:
Lines 293 to 295 in 16ec607
Update the README to add the following:
Implement Serie.group_by
and aggregate functions.
Note that the group_by function
has a different semantic from SQL: here, rows are grouped by a consecutive sequence of values matching a condition.
Required by #27.
Possible syntax:
serie.group_by(<cond>, <aggregate fct, ...>)
Where <cond>
is an arbitrary column expression (possibly a single column name) and <aggregate fct, ...>
is a non-empty sequence of aggregate functions.
Example:
serie.group_by(
(fc.gt, "CLOSE", "OPEN"),
(ag.first, "NAME", "ID", "DATE"),
(ag.count, (fc.named("COUNT"), "NAME")),
(ag.avg, (fc.named("AVG PRICE"), "CLOSE")),
)
Implement (in Cython) several volatility estimators:
Currently, we only have the Python implementation of the close-to-close volatility:
Lines 188 to 212 in 78913aa
Add multi-variable constraint solving for models, maybe using the downhill simplex method, or—for derivative-free optimization methods—swarm optimization or genetic algorithms.
There is no obvious way to specify the aggregate function for the index.
I wonder if adding implicitly the index in select
/group_by
isn't an error. Maybe we can make the index reference mandatory. Would it be desirable to check that the index values are sorted in ascending order?
Working on that in https://github.com/s-leroux/fin/tree/exp/make-index-selection-explicit
Originally posted by @s-leroux in #30 (comment)
Other things to consider:
sort
predicate to change the index.Unicode characters break the test runner on GitHub.
The example below was caused by '…' (\u2026) in the heading while printing a Serie instance.
======================================================================
ERROR: test_adj (tests.fin.seq.fc.test_adj.TestAdjustQuote)
<class 'fin.model.solvers.particle.ParticleSwarmSolver'> Use case #1 4.565761130040028e-16 (3.9999999928003915, 7.999999995255896)
<class 'fin.model.solvers.random.RandomSolver'> Use case #1 0.05296359406606747 (4.078120375554134, 8.045150938464865)
<class 'fin.model.solvers.particle.ParticleSwarmSolver'> Use case #2 2.3216148996009935e-18 (2.0000000009396457, 2.9999999995592344)
<class 'fin.model.solvers.random.RandomSolver'> Use case #2 0.02055660096305517 (1.9510313914860744, 3.0927051599569175)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/fin/fin/tests/fin/seq/fc/test_adj.py", line 32, in test_adj
print(res)
UnicodeEncodeError: 'ascii' codec can't encode character '\u2026' in position 7: ordinal not in range(128)
----------------------------------------------------------------------
Ran 206 tests in 1.765s
There is a lot of free fundamental data available on web pages. We already have experience with a web scrapper:
162c944
The code above was written especially for Investing.com.
Can we have something more generic to parse table-like data?
The requirements are to be able to parse table elements, but also eventually pseudo-tables made of div span constructs.
Yahoo Finance (and other data providers) return the quotes as (open, high, low, close, adj close)
. Only the close price is adjusted for splits and dividends. We should provide a function to rescale the open, high, and low accordingly.
An option is to express the open, high, and low relative to (unadjusted) close, then apply these ratios to the adjusted close price to find the other adjusted prices.
In some circumstances, floating-point number columns are displayed as ternary columns:
from fin.api.yf import Client
from fin.seq import fc
ticker = "^FCHI"
duration = dict(days=5)
client = Client()
data = client.historical_data(ticker, duration)
# Yahoo! Finance has dirty data. Do some clean-up
data = data.where(
(fc.all, "Open", "High", "Low", "Close", "Adj Close"),
)
print(data)
Displays:
Date | Open | High | Low | Close | Adj Clo… | Volume
---------- | -------- | -------- | -------- | -------- | -------- | --------
2024-05-03 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 69643700
2024-05-06 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 43781100
2024-05-07 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 58688300
2024-05-08 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0
It is probably caused when the same column has cached several representations. We do not have a mechanism to select the "most appropriate" representation.
Implement (in Cython) the log return algorithm:
r[i] = ln(p[i]/p[i-i])
or
r[i] = ln(p[i])-ln(p[i-i])
Add the union
operator to combine two (more?) series that have the same columns.
This may be useful when loading data in chunks and you want to combine them to return the result as a single series.
We should locally cache EOD data instead of systematically retrieving them from the provider.
Maybe, something along the lines of:
client = Cache(yf.Client())
t = client.historical_data(...)
Review the t-expr semantic:
Line 288 in 78913aa
Specifically:
expr.constant
t.reval([1,2,3,4])
would return one column with the given values instead of four constant columns)There is a preliminary implementation of the Sharpe ratio. It needs to be fixed.
Lines 256 to 276 in 46a55e6
Check that the four basic arithmetic operations on columns are implemented (in Cython):
Follow the LISP semantic for the n-columns versions.
Check the results are consistent when invoked with zero or one column.
This fails on the print
statement:
from fin.seq.serie import Serie
from fin.seq import fc
ser = Serie.create(
(fc.named("T"), fc.range(10)),
(fc.named("X"), "T"),
(fc.named("Y"), fc.all, "X"),
)
print(ser)
But that works as expected:
from fin.seq.serie import Serie
from fin.seq import fc
ser = Serie.create(
(fc.named("T"), fc.range(10)),
(fc.named("X"), "T"),
(fc.all, "X"),
)
print(ser)
Some table functions (like join()
) require the table rows to be sorted. We could enforce that.
Idea: when calling table.sort()
, the Table instance could remember the key used so we could check that when required.
fin/tests/fin/seq/test_column.py
Line 231 in 7e8c0f6
Typo: there is an extra s
here
The code below produces an infinite recursion:
t = table.Table(361)
t.add_column((range))
File "/home/sylvain/fin/fin/seq/table.py", line 322, in reval_item
return self.reval(*it)
File "/home/sylvain/fin/fin/seq/table.py", line 304, in reval
result += self.reval(tail)
File "/home/sylvain/fin/fin/seq/table.py", line 302, in reval
result = self.reval_item(head)
File "/home/sylvain/fin/fin/seq/table.py", line 322, in reval_item
return self.reval(*it)
File "/home/sylvain/fin/fin/seq/table.py", line 302, in reval
result = self.reval_item(head)
File "/home/sylvain/fin/fin/seq/table.py", line 324, in reval_item
return [ Column(None, [item]*self._rows) ]
RecursionError: maximum recursion depth exceeded while calling a Python object
It is not obvious if add_column
should accept the range
function as a valid argument. Nevertheless, it shouldn't produce an infinite recursion.
We should support the End-Of-Day Historical Stock Market Data API available on https://eodhistoricaldata.com/.
API endpoint:
https://eodhistoricaldata.com/api/eod/MCD.US?api_token=demo
API doc:
https://eodhistoricaldata.com/financial-apis/api-for-historical-data-and-volumes/
The maximum length on Yahoo! Finance for historical data download is one year. We may overcome that limitation by downloading data in one-year chunks.
In some circumstances, the column mini-language requires wrapping the argument in a 1-tuple. This is confusing.
In practice, the following code is ambiguous:
(callable, callable, "X")
Using the infix notation, it can be parsed either as callable(callable(), "X")
or callable(callable("X"))
or even callable(callable()), "X"
.
Add GitHub actions to run the tests on push.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.