Comments (11)
am i using the scan methond wrong way?
from gohbase.
// NumberOfRows is an option for scan requests.
// Specifies how many rows are fetched with each request to regionserver.
// Should be > 0, avoid extremely low values such as 1 because a request
// to regionserver will be made for every row.
NumberOfRows
doesn't limit how many rows are returned by the scan in total. It limit how many rows are returned by each scans request send to HBase.
from gohbase.
from gohbase.
A scan make multiples requests to regionservers. NumberOfRows
only limit the number of rows returned PER requests, not for the totality of the scan.
This will do what you want, get 2 results and stop:
hrpc.NewScanRangeStr(ctx, table, startRow, stopRow, hrpc.NumberOfRows(2), hrpc.CloseScanner())
CloseScanner
will close the scanner immediately after the first results are returned.
from gohbase.
Another option, especially if you want to follow the same pattern for a larger number of rows or if the rows are spread on multiple regionservers, is to exit the for loop when you have reached the size you wanted and make sure to call scan.Close()
to free the scanner on the regionservers.
from gohbase.
from gohbase.
Hello, may I ask how can I pull x number of rows from a start key? I am trying to achieve paginated scans. I tried to leave the end key empty, but it still always fetches more than the specified limit.
from gohbase.
from gohbase.
Hmmm, yeah I get that part. But when I did this,
req, err := hrpc.NewScanRangeStr(ctx, tableName, cursor, "", hrpc.NumberOfRows(uint32(limit)+1), hrpc.CloseScanner())
It still returns more rows that limit + 1
.
from gohbase.
I made it work like this:
// scanLimit reads at most n number of cells from scanner and returns the result.
// It is the caller's responsibility to close the scanner once all reading has been finished.
func scanLimit(scanner hrpc.Scanner, n int) ([]*hrpc.Cell, error) {
var cells []*hrpc.Cell
for len(cells) < n {
result, err := scanner.Next()
if err == io.EOF {
break
} else if err != nil {
return nil, err
}
for _, cell := range result.Cells {
// result.Cells can contain anywhere from 1 to many cells
if len(cells) < n {
cells = append(cells, cell)
}
}
}
return cells, nil
}
Caller:
scan, err := hrpc.NewScanRangeStr(ctx, table, k, "",
hrpc.NumberOfRows(10),
)
if err != nil {
return err
}
scanner := runner.client.Scan(scan)
cells, err := scanLimit(scanner, 10)
if err != nil {
return err
}
scanner.Close()
Note that despite setting hrpc.NumberOfRows(10)
, scanner.Next()
can return anywhere from 1 cell at a time to over a million cells at once, which is quite strange.
from gohbase.
Note that despite setting hrpc.NumberOfRows(10), scanner.Next() can return anywhere from 1 cell at a time to over a million cells at once, which is quite strange.
1 Cell = 1 Column. If you ask for 1 row but that row has 10000000 columns, you will get 10000000 cells. You can change that by asking only for a specific columns using the hrpc.Families(...)
option
from gohbase.
Related Issues (20)
- Client logging should be configurable
- does gohbase client have transaction functionallity? HOT 1
- OpenTelemetry can be set as optional HOT 1
- Support read from replicated meta table
- HBbase serialization cannot obtain the timeout period? HOT 4
- sleepAndIncreaseBackoff should have max backoff value
- Host not updated HOT 1
- RegionClient{Addr: localhost:16020} HOT 1
- zookeeper连接时间为什么是30秒? 对于web服务来说是不是太长了? HOT 2
- How to improve the performance of `scan` operation? HOT 2
- How do I know a scan result is returned due to MaxResultSize limit? HOT 3
- Version is too high HOT 2
- Support region cache warmup
- metrics: count of errors that were automatically retried
- Consider switching to vtprotobuf
- 32
- Move to log/slog for logging
- Expose Dialers for Zookeeper and Region package to allow using proxies
- caches.go clientDown may have a memory leak problem
- version control: need to publish a new release with tag
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gohbase.