hyrise / hyrise-v1 Goto Github PK
View Code? Open in Web Editor NEWHYRISE In-Memory Hybrid Storage Engine (archived, now developed in hyrise/hyrise repo)
Home Page: https://github.com/hyrise/hyrise
License: MIT License
HYRISE In-Memory Hybrid Storage Engine (archived, now developed in hyrise/hyrise repo)
Home Page: https://github.com/hyrise/hyrise
License: MIT License
A problem with sharing error cases is that you also need to share a file or a database connection to be able to run the test.
If instead, we could load a table from JSON only, this would reduce the work needed to reproduce bugs.
So either enhance the existing string loader with one that can also provide a table, or provide JSON-based loaders, that allows for loading from the operation parameters ie:
{
"type": "JsonTable",
"table": { "header" : ["a", "b"],
"types" : ["INT", "STRING"],
/*optional*/ partitions: [ "1_R", "2_R" ],
"data" : [ [ 1, "USA" ], [ 2, "GERMANY" ] ] }
}
There should be a meta table that shows which tables are there, to follow the SQL spec this should as well contain information about the columns etc as well.
Something like information_schema
etc from MySQL & Co.
The TPCCHQ1Scan Plan Operation seems unnecessary and can potentially be removed unless required for special purposes
Fails in AbstractCoreBoundTask::launchThread when taking address of executeTask with "pure virtual function called".
The default table type should be immutable und allow changing. However, currently in general all table types are mutable instead.
Radix Join should work on non int columns as well. The RadixCluster plan operation has to be adapted templated to work independent of the dictionary type.
{
/*... other stuff ... */
"performance_data": {
/* ... other operands, too */
"operandId": {
"timing": { /* all these values are explicit not measured through hardware counters */
"start": 10 /*ns since query start*/ ,
"stop": 20 /*ns since query start*/ ,
"start_epoch": 100000000123 /*ns since epoch*/ ,
"stop_epoch": 100000000133 /*ns since epoch*/ ,
},
"counters": {
/* these values are explicitly measured through hardware counters */
"PAPI_TOT_INS": 10,
"PAPI_TOT_CYC": 20,
"PAPI_L2_TCM": 30
},
"input": {
tables: []
},
"output": {
/* same as input */
},
"custom": {
/* op may emit whatever JSON it deems sensible */
}
}
}
}
in request, we specify what we want
for logging
{ "operations": [ /*...*/ ], "logging" = ["papi", "input", "output", "timing"] }
Issues with this proposal: "counters" would currently only count what happens in executePlanOperation
The current half/half situation doesn't really help anyone.
The generic assignment of UUIDs to all containers through placement in the AbstractTable
places an unnecessary burden on the overall system.
tests before uuid: make test 3.34s user 1.19s system 77% cpu 5.843 total
tests after uuid: make test 8.84s user 4.61s system 103% cpu 12.975 total
There is no reason that every single AbstractTable needs a UUID. It might even be debatable if every Store needs a UUID.
Potential fixes:
Well Barrier is kind of weird, because it uses the length
n
of _field_definition to forward the firstn
elements, regardless of actually set values in _field_definition.
We should make sure that its semantics are basically output == input
.
Distribute method in PlanOperation returns first and last as array positions (0 and 999 for a table with 1000 elements) -> following iterator-style you would expect last to work as an exit condition for iterating over input table.
Currently only one TX can be committed at a time, this should be improved :)
In the current implementation, MVCC allows for inconsistent reads. For a record deleted by another transaction, it is impossible to determine whether it would have been valid or not in the context of the current transaction. Example:
The current status of the documentation is pre-c++11 and our shared_ptr usage, so we should find some time to update it.
Can we use github pages to store the recent version of the documenation. I think this should be possible...
I ran vagrant up. On the virtual box I tried to compile Hyrise, but the binutils-dev package is missing (the -lbfd flag fails in the linker). A simple apt-get install binutils-dev rectified the situation.
The Plan Operations TaskSchedulerAdjustment and ThreadpoolAdjustment do exactly the same, SettingsOperation also does the same but leaves out one step. Therefore it seems that only one of these three is actually necessary and the other two should be removed
prototype for typeswitches
when getvalue searches the right dictionary, column AND row needs to be taken into account
The commit implementation allows Store->updateCommitId to fail in some way, but does not clean up any changes that have been made.
Should be "setIndexName" or similar.
Currently the RadixJoin does not work on PointerCalculator objects since the class does not implement the getAttributeVectors()
interface. Even if it would it would break since the positions inside the vector might not be the same as the input positions.
Possible solutions: If the input is a pointer calculator, rewrite the positions to match the input vector and extend the handling to multiple horizontal partitions.
This is related to #18
Since 6576e8d, much boilerplate isn't necessary any longer when implementing operators.
Thus, we should remove all name
/vname
methods and use registerPlanOperation<Type>("name");
instead.
Instead of implicitly assuming every request encloses a transaction, a transaction may last for several requests. Thus, we need:
C++ exception with description "StorageManager: Table 'revenue' does not exist" thrown in the test body.
Not sure about the underlying issue yet. The test fails occasionally on our build infrastructure.
This makes writing Operators that expect identical data inputs twice (such as self joins) a pain.
Anyone knows a case where it makes sense to filter inputs for duplicates? Pl let me know, otherwise, this filter will be removed.
Implement a base class AbstractResource that is used to transport resource between different operations in a plan.
This should replace multiple lists of resource types in OperationData
Setting _join_type in JoinScan does not have any effect on the join being performed
HYRISE should be able to support filter predicates based on subqueries. Something like
SELECT * FROM table where attr in (SELECT id FROM othertab);
While only one thread may resize the delta, other threads working on delta at the time of resizing may end up working on deallocated memory, since the underlying vectors may change during a reserve.
Move towards a storage manager that only stores AbstractResource
objects with a name ready for retrieval. (Also remove the loading shortcuts in StorageManager while we are at it).
Stacktrace
Value of: long_block_test(scheduler)
Actual: false
Expected: true
@JWUST any hints on that one?
Currently, the facilities surrounding template-based allocators are completely underused and are not actually necessary. Instead, it would be good to use polymorphic allocators similar to what is implemented in bsl https://github.com/bloomberg/bsl/wiki/BDE-Allocator-model and will be part of the c++14 standard.
PAPI initialization takes a while for small ops. Maybe only initialize once per thread.
It might be nice to add a caveat section on the intended audience and what someone can expect from this chunk of code.
There should be a way to simply access the tables stored on disk without really need to specify always the load operation explicitly.
The issue here is that hyrise doesn't seem to shut down on CTRL-C occasionally and we need to figure out why that is the case.
Currently, all transactions log into a single logfile, which consequently grows over time and never gets deleted or truncated. Also, once a table is merged, its previous log entries must be removed/invalidated to avoid redo recovery in case of failure.
Solution 1: One logfile per table. Problems: increased logging overhead, commits must either be written into all logfiles or in a separate commit log.
Solution 2: One logfile for all tables, checkpoint entries when table is merged (i.e. "ignore previous log entries for table X"), new logfile after merge. When all tables have been merged once, the first logfile can be deleted, the second after all tables were merged twice and so on. Problems: Might take a while until all tables are merged.
Solution 3: One logfile for all tables, checkpoint entries when table is merged (i.e. "ignore previous log entries for table X"), new logfile after merge. Separate worker reads old logfiles on every table merge, removes entries and writes truncated log back. Problems: Potentially costly operation, IO overhead
Increase ease of use of PointerCalculator.
Currently only rudimentary expression support is available for HYRISE. It should be extended to support.
Entries in the right-hand table only appear once in the result table, namely for the first match. Instead, they should appear for every matching entry in the left-hand table.
For logging, we need to identify to which table a certain entry was written. This could be a unique table id.
To reproduce:
send empty query to server twice
curl -X POST --data-urlencode "query@xxx" http://localhost:5000
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.