An Option to Output for Streaming about sstable-tools

[clohfink]

partition per line would still be issue, but I think a row per line would make this pretty doable so I am all for doing that here and will put it on list right after json2sstable that I'm working on currently.

That said there are streaming json libraries available, jackson, jsonsimple in javaland, and I've used ijson in python to read huge sstables with old sstable2json succesfully.

[devdazed]

I've heard about the streaming libraries, however they tend to be rather cumbersome to use, at least the python ones I've tried were. One row per line seems a bit of an issue also as the format you have outlined would not really help because the partition start and end would be on different lines, unless you want to output in a completely different format that duplicates the partition information on every row?.

The idea is to be able to parse with something simple like:

import sys, json
tombstones = 0

for line in sys.stdin:
    partition = json.loads(line)
    for row in partition['rows']:
        if 'deletion_info' in row:
            tombstones += 1
            continue
        for cell in row.get('cells'):
            if 'deletion_time' in cell:
                tombstones += 1

print str(tombstones) + " tombstones found"

the example above is a bit contrived and might even not work, but it demonstrates a valid and common use case for parsing SSTables, is to find tombstone where there may be clusters of tombstones that can cause issues in the cluster.

[devdazed]

also streaming json libraries tend to be rather slow as they sometimes rely on continually attempting to parse incomplete json and catching the exceptions

[clohfink]

I tend to worry about doing a partition per line when the concern is OOMing when reading at once. a >1gb partition is bad but far from unheard of - id say its seen a lot in time-seriesey applications. Adding the json overhead this would demolish systems as well. That said I dont know of a good solution short of a weird state machine/custom parsing so maybe a partition per line is best and people with wide rows have to use incremental parsers.

FWIW if you use this as a library instead of a tool its pretty easy. might be good idea for us to add to mvn or something if valuable

final AtomicInteger total = new AtomicInteger(0);
CassandraReader.fromSSTable(sstablePath).readSSTable(sstablePath, null)
        .filter(p -> p.isLive())
        .forEach(p -> {
            p.rows().forEach(unfiltered -> {
                if (unfiltered instanceof Row) {
                    for (Cell cell : ((Row) unfiltered).cells()){
                        if(!cell.isLive((int)(System.currentTimeMillis() / 1000))) {
                            total.incrementAndGet();
                        }...

[devdazed]

could always do a cell per line and include all partition key info and deletion info in the object. this will keep objects small and, since everything is sorted, the apps will know when the next partition/row happens because it will change. it will, however, drastically increase the file size if outputting to a file.

also, what about complex types and things like sets, lists. if a row has a 1gb set, this could also be problematic.

generally people doing exceptional things with their partitions are going to be the ones who need this tool the most. it makes most sense to optimize it for those who will need it, rather than for the majority of Cassandra users who will almost never need to use the tool.

[clohfink]

#23

[tolbertam]

Closing since this now exists as the -d option in sstabledump (which is no longer part of sstable-tools but part of sstabledump in C*)