Comments (19)
Unfortunately the Thrift API does not support it. :-(
— Wouter
(Sent from my phone. Please ignore the typos.)
saintthor [email protected] schreef:
for a mass data, i use batches to write lines. it is efficient. but the
counters take too much time. is there any way to set counters in batch?
Reply to this email directly or view it on GitHub:
#35
from happybase.
The latest Hbase.thrift definition file has a new method incrementRows
that might be what @saintthor is looking for. It allows you to send a list of TIncrements in one request.
from happybase.
thank you.
wbolster, is it possible to add this feature to happybase for hbase version 0.90?
from happybase.
@saintthor If it's added in recent versions, there is no way this can be supported in the much older HBase 0.90 release.
from happybase.
Thanks @nkeyes for the pointer.
It looks like the increment()
function in the Thrift API doesn't add anything over the atomicIncrement()
function. The latter does return the counter value, while the new API does not. Strange.
The other function is incrementRows()
, which actually adds previously unavailable functionality. The API doesn't seem to support using this with a write batch (Table.batch()
) though. This Thrift API is currently (HappyBase 0.6) not exposed. Any thoughts on what would be the most sensible API to have in HappyBase's Table
class?
(The increment value for the TIncrement
struct is misspelled as 'ammount', btw.)
from happybase.
Here's how I did it in ok_hbase:
nkeyes/ok_hbase@5ab526b
I added methods on the Connection, Table, and Row classes:
- Row#increment sets the row_key on the TIncrement to the current Row instance then proxies to Table#increment
- Table#increment and Table#increment_rows set the table_name on the TIncrements to the current Table instance and then proxy to Connection#increment and Connection#increment_rows
- Connection#increment and Connection#increment_rows accept hash representations of TIncrements, allowing for either amount, or the typo 'ammount'
You're right, there doesn't appear to be a way to use increment or incrementRows within a batch request, but incrementRows is sort of a batch request in itself.
I think of incrementRows as a special case, and if the developer decides they need it, they understand that it is not batchable with other mutations.
from happybase.
Relevant Thrift API:
/**
* For increments that are not incrementColumnValue
* equivalents.
*/
struct TIncrement {
1:Text table,
2:Text row,
3:Text column,
4:i64 ammount
}
/**
* Increment a cell by the ammount.
* Increments can be applied async if hbase.regionserver.thrift.coalesceIncrement is set to true.
* False is the default. Turn to true if you need the extra performance and can accept some
* data loss if a thrift server dies with increments still in the queue.
*/
void increment(
/** The single increment to apply */
1:TIncrement increment
) throws (1:IOError io)
void incrementRows(
/** The list of increments */
1:list<TIncrement> increments
) throws (1:IOError io)
from happybase.
Thanks @nkeyes for the explanation.
from happybase.
Do you happen to know which version introduced this API? Currently HappyBase has '0.90' and '0.92' compatibility modes; this probably needs to be extended when adding API only available in more recent versions.
from happybase.
I think it came wit 0.94, I don't see it in the 0.92 docs.
0.92 (candidate?):
http://people.apache.org/~stack/hbase-0.92.2-candidate-0/hbase-0.92.2/docs/apidocs/index.html
0.94:
https://hbase.apache.org/0.94/apidocs/index.html
from happybase.
what a regret. thanks.
from happybase.
@wbolster, have you added the feature that writing counters in batch? is it for hbase 0.92 or 0.94? is there docs or examples?
from happybase.
No, there is no code for this currently.
It would require a new compat mode, a new BatchIncrement class, like the current Batch class, tests, and updated docs, but I haven't found time/motivation yet to work on this... so much else to do. :)
from happybase.
it is all python codes to add this feature, right? i may try it.
from happybase.
Yes it is, but it may require regenerating the Thrift code using a newer .thrift definition file.
— Wouter
(Sent from my phone. Please ignore the typos.)
saintthor [email protected] schreef:
it is all python codes to add this feature, right? i may try it.
Reply to this email directly or view it on GitHub:
#35 (comment)
from happybase.
do you mean to generate the hbase module with thrift?
2013/10/25 Wouter Bolsterlee [email protected]
Yes it is, but it may require regenerating the Thrift code using a newer
.thrift definition file.— Wouter
(Sent from my phone. Please ignore the typos.)
saintthor [email protected] schreef:
it is all python codes to add this feature, right? i may try it.
Reply to this email directly or view it on GitHub:
#35 (comment)—
Reply to this email directly or view it on GitHubhttps://github.com//issues/35#issuecomment-27073607
.
from happybase.
Yes, because this requires API that was not available when the current version was generated.
— Wouter
(Sent from my tablet. Please ignore the typos.)
saintthor [email protected]:
do you mean to generate the hbase module with thrift?
2013/10/25 Wouter Bolsterlee [email protected]
Yes it is, but it may require regenerating the Thrift code using a newer
.thrift definition file.— Wouter
(Sent from my phone. Please ignore the typos.)
saintthor [email protected] schreef:
it is all python codes to add this feature, right? i may try it.
Reply to this email directly or view it on GitHub:
#35 (comment)—
Reply to this email directly or view it on GitHubhttps://github.com//issues/35#issuecomment-27073607
.—
Reply to this email directly or view it on GitHub.
from happybase.
Fwiw, I've just updated the Thrift API bundled with HappyBase in dd7878c, so the new Thrift API is now available inside HappyBase (but not exposed in the public API).
from happybase.
i have added the method in happybase connection module last week.
class Connection(object):
def batch_inc( self, IncItems ):
"inc counters via batch"
Increments = [TIncrement( *item ) for item in IncItems] #item = [tableName, rowKey, counterName, counterValue]
self.client.incrementRows( Increments )
it works well. if you think it is ok. please add it into happybase. this will make our deployment simpler.
from happybase.
Related Issues (20)
- Fix simple typo: specifed, -> specified,
- How does happyhbase fuzzy query rowkey HOT 3
- Support Pre-Split when create table ?
- HBase remotely connecting to python project HOT 7
- Support gevent?
- suppot count table rows? HOT 1
- org.apache.hadoop.hbase.NamespaceNotFoundException: HOT 4
- [feature] Support reconnect host when connectionpool raise error HOT 2
- thriftpy2.protocol.exc.TProtocolException: Bad protocol id in the message: 72 HOT 4
- happybase 1.2.0 supports hbase 2.2.5 ? HOT 1
- How can I do a query for specific columns by regex-statement HOT 2
- TTransportException: TTransportException(type=4, message='TSocket read 0 bytes') HOT 4
- TypeError: __str__ returned non-string (type bytes) hbase HOT 6
- import happybase error HOT 1
- Does it support snapshot management? HOT 1
- Is there a way to update / modify TTL (time to leave) on already created table ? HOT 1
- Hbase compatibility
- Table Put - How Do We Assign And Use A Variable For the 'Row Key' HOT 8
- compact_table major no work
- 使用happybase创建连接池并扫描整张表一段时间以后报错 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from happybase.