lateefj / shylock Goto Github PK

Mount etcd so it can read only

Just integrate the pathioc with the IOC system.

When reading from a redis MQ there is a read timeout at around 60 seconds.

Something that can provide a bridge between the generic interface and the docker volume interface. This exists to support #32.

There needs to be something that all the integration are registered with. The main goal it to keep track of all the integration that are registered so they can be used when needed. This should also manage all the instances that have been created and mounted.

• List all integrations
• Register an integration
• Mount with a given mount point
• Unmount a given mount point

Given a path find a matching key / IOC instance. This needs to be a fast search that can quickly find the matching X bytes.
How do we handle any paths that don't have a match? Should there be a default?

Remove file was probably not implemented so need to add that!

Based on a set of strategies:

• Drop
• Block
• Custom Handler

Provide mechanism for being able to the stream form backing up based on message volumes.

Based on a csv file be able to specify a key, read, writes, unit.
Example

/foo/bar, 512, 512, second
/foo/monkey, 512, 512, minute

This should provide the code for fuse to support #32

The big picture is that with SAAS is the each customer is renting some portion of the service. Mostly the challenge is providing quality of service (QOS) per customer. With shared resources like databases each customer QOS will need a slice of query time. When the number of concurrent customers is low this is not so much of a problem. As the number of customers and size of data gets larger it becomes harder and harder to share resources. Harder usually means slower or more complex. In an ideal case we would create a way to isolated storage for each customer (or device, key ect). Practically speaking we take a single data store and try to figure out way to keep individual customers for consuming to much of the resources. As data sets (tables) become large it is increasingly difficult because every query has to traverse all the data to filter a specific isolation group (customer) data.

Based on the plugin Key interface get the IOC instance and apply the control to the stream of data.

Add a read and write OPS to the QOS API

Write why kafka is a good POC.

• Streams are just pipes or endless pseudo files
• Kafka Partition in that each partition is a file
• Kafka cluster consumer is really a load balancer
• How QOS would work in Kafka

If QOS is enabled when an integration is mounted this should get used to set limits on read / write volumes. It should be provided for #31 on Put / Get calls.

Need to be able to configure operation per X in the configuration file. Parse the file into a data structure that is thread safe.

mkdir -p /mnt/point/foo/bar/bat/

Per second seems to be the standard.

Need to support both types of message types:

1. Keyed Messages
2. Messages

That return a key for IOC. This plugin would be configured by an environment variable.

The current POC uses fuse only. Docker has a volume plugin system. This boils down to implementing the Driver interface.
For a message queue this translates into something closer to:

type MQ interface {
  Mount(config map[string]string) error
  Unmount() error
  List(path string) ([]string, error)
  Open() error
  Get() (header, body []byte, error)
  Put(header, body []byte) error
}

type KV interface {
  Mount(config map[string]string) error
  Unmount() error
  List(path string) ([]string, error)
  Get(path string) ([]byte, error)
  Put(path string, body []byte) error
}

As well there needs to be a Register function ex:

type FS interface {
  RegisterMQ(mountPoint, type, description string, config map[string]string, imp MQ) 
  RegisterKV(mountPoint, type, description string, config map[string]string, imp KV) 
}

Listen on a Port number and provide an API for basic configuration.

1. Add a key, read, write and unit
2. Delete based on key
3. Change a keys values

This has already been implemented but not with QOS support here: https://github.com/xetorthio/etcd-fs

Write up examples on how to mount etcd.

• Be able to mount it in read only mode and copy files.
• Write files examples
• Example on generating configuration from etcd files

Be able to write etc files.

QOS needs to standardize on seconds as a unit of time.