Possible Design Issue: Many-to-one structure with Query/Response about grafana-plugin-sdk-go

[ryantxu]

sorry did not see this till just now...

um, I'm super nervous about putting these details in the grpc interface. If we do add QueryResultMeta it should be the same object used in arrow, likely stored as JSON

I think it makes more sense to define a MUCH smaller set of things that the backend wants to track and figure out how to merge them with QueryResultMeta on the frontend.

[ryantxu]

I do not disagree... but will note that the UI only passes around DataFrame[] -- so we don't have anywhere obvious to put the response (without some very major changes!).

In practice, everything I have seen had special metadata for each result or has a single result. The one exception to this is that metrictank processes all queries at once, then returns global metadata and metadata for each result. in this case we put it on the first one and that seems OK so far.

[kylebrandt]

In practice, everything I have seen had special metadata for each result or has a single result.

I don't think that is true. Prometheus, OpenTSDB, probably others are going to return []Frame for time series as the normal case. Basically anything that returns multiple time series from a single query, where they have different time values and/or different lengths for each series.

[ryantxu]

I don't disagree that it is normal to return an array of frames... Just that the metadata (in every case I have looked at) applies to a frame, not all frames. In the error case there is only one frame

[bergquist]

I think this makes sense.

This will make plugin development easier to reason about since each query will result in a query response. One in, one out. Each query response can hold information that the backend might want to know about but without unpacking the dataframes. Ex error, execution items etc.

[kylebrandt]

(Will work on PR to illustrate this).

[ryantxu]

I think this makes sense.

sure... but what should the frontend do with it? the frontend pipeline really just supports DataFrame[]... so this could explode into a much bigger issue

[kylebrandt]

So the proto would be something like:

//-----------------------------------------------
// Data
//-----------------------------------------------

service Data {
  rpc QueryData(QueryDataRequest) returns (QueryDataResponse);
}

message TimeRange {
  int64 fromEpochMS = 1;
  int64 toEpochMS = 2;
}

message DataQuery {
  string refId = 1;
  int64 maxDataPoints = 2;
  int64 intervalMS = 3;
  TimeRange timeRange = 4;
  bytes json = 5;
}

// QueryDataRequest
message QueryDataRequest {
  // Plugin Configuration
  PluginConfig config = 1;

  //Info about the user who calls the plugin.
  User user = 2;

  // Environment info
  map<string,string> headers = 3;

  // List of data queries
  repeated DataQuery queries = 4;
}

message QueryDataResponse {
  // List of data query responses
  repeated DataResponse responses = 1;

  // Additional response metadata
  map<string,string> metadata = 2;
}

message DataResponse {
  string refId = 1;
  // Arrow encoded DataFrames
  // Frame has its own meta, warnings, and repeats refId
  repeated bytes frames = 2;
  QueryResultMeta queryMeta = 3;
}

message QueryResultMeta {
  google.protobuf.Any custom = 1;
  repeated QueryResultMetaNotice notices = 2;
}

message QueryResultMetaNotice {
  enum Severity {
    DEBUG = 0;
    INFO = 1;
    WARNING = 2;
    ERROR = 3;
  }
  Severity severity = 1;
  string text = 2;
  string url = 3;
  enum Inspect {
    META = 0;
    ERRORMSG = 1; // ENUM names must be unique across the parent
    DATA = 2;
    STATS = 3;
  }
  Inspect inspect = 4;
}

[marefr]

Only me who thinks notice/notices is a weird name? Not sure I have anything better though, message/messages or event/events maybe?

[marefr]

@kylebrandt I'm still a bit confused after seeing your proto update. What kind of meta would be part of the data frame vs this meta?

[kylebrandt]

@kylebrandt I'm still a bit confused after seeing your proto update. What kind of meta would be part of the data frame vs this meta?

Generally, information about the query goes on the DataResponse, and information about the frame itself goes with the frame.

Details of the Frame changes I will get into as I work on my PR.

[kylebrandt]

fixed in 9e678ab