Support composite sort and partition keys about filodb

[velvia]

Note: we will probably move to a model where there is a separate Segment key from the sort key. The sort key is the primary key within a partition, whereas all records with the same segment key map into the same segment. Within a segment, rows will be sorted by the sort key.

The segment key does not need to be a physically separate column. It could, and may usually be, a computed function on another column. For example, let's say for a time-series example, the sort key is a composite of ["timestamp", "uuid"]. Let's say we want to group all records into 30-second buckets. Then the segment key would be (timeslice timestamp 30s) which is a function slicing the timestamp column (assumed to be milliseconds) into 30-second buckets.

[velvia]

Proposed initial set of expressions:

• (timeslice <ms-timestamp-col> <timeunitstr>)
• (round <int/long col> <number>) -- the default 10000 segment len would be the function (round colA 10000)
• (stringPrefix <str col> <numChars>) -- gets the first numChars of the string

Some relevant code, for rounding:

  def getSegment(key: Long): (Long, Long) = {
    val segmentNum = key / segmentLen
    (segmentNum * segmentLen, (segmentNum + 1) * segmentLen)
  }

for string prefixing:

  def getSegment(key: String): (String, String) = {
    val start = key.take(prefixLen)
    val end = start.take(start.length - 1) + ((start(start.length - 1) + 1).toChar)
    (start, end)
  }

[velvia]

Also:

• (getOrElse <col name> <default value>) -- provide a default value if the column value is null
• (autoid) -- generate an auto incrementing ID within each partition (need to think about how this works - might be a TimeUUID)
• (const <val>) - returns a constant value. Only useful for doing a default partition key.