kevinmackenzie / incubator-trafficcontrol Goto Github PK

The naming of the columns in the database suggests that it will store the lines in the database that aren't in the local under db_lines and the lines that are in the local, but not in the database under local_lines. To keep consistent with how the ort script handles these differences, the naming and usage should change to be db_lines_missing and local_lines_missing in the database and something similar in the JSON submitted to the web API.

When requesting the API that we're developing, what should be used as the identifying attributes? the config_name column seems like a good one to have, but should we use the id from the server table, or is there a better option? Right now, it is /servers/{id}/cfg.json to request all configuration differences of the server, but there should be an option to only select the differences of a single file.

Right now, the API is set up to accept a regular length host-name, but the ort script is making the request with the short version. These are almost always different, so this needs to be solved. It is unclear how a host-name-short gets mapped to its server.id, but that will definitely be a requirement.

This is meant to get requests, often from the caches, to update the differences each has with the active configuration for their cache group. This data will be stored as determined in #12 and then can be reported back with other web requests.

This issue may become a project and then more specific issues added as we get to this point in the project.

Once we get a service that interacts with existing Traffic Ops data, we must next extend the data that is stored to include the information of the configuration differences between caches and the cache group configuration.

When retrieving the lines added/removed for a config file in the PostgreSQL database, the database array is being converted to JSON and after making the request, this json is being unmarshalled back into an array. The results of this unmarshalling should be checked for errors.

The timestamp stored in the database with the configuration differences is being taken from the Golang server, but it would be better to have the timestamp come from the client because, while unlikely, it is possible for the client to check for updates, the server to queue an update, and the client to submit the report. This would cause the database to reflect that the client had checked for updates, but didn't see some of them.

Once we get an AWS environment setup with the cache and the Traffic Ops, I'll need to familiarize myself with how administrating the system works.

Before anything real can be added to the TrafficOps service, we will need to figure out how the current GO api is structured and how to add endpoints/routes to it. It is unclear if the go code gets shipped or just the binaries, because there are no homologous files between the 2.1.X branch and the /opt/traffic_ops/ folder. Furthermore, I am not entirely sure how to rebuild/restart these apis once they are modified.

Before creating and reporting my own data, this will test reporting of data that already exists as an example of how to interface with Traffic Ops, so when we add our own data we will know what we're doing.

Right now, the only documentation is on the blog posts and commit messages, but it is important to have a centralized documentation for how the API functions. This could start in the wiki of this repository and then we can figure out how to merge this with the Traffic Ops documentation after the PR.

Along with having the API in Traffic Ops, we must modify the ort script to send this information to the new API to report the configuration differences. This will probably happen after making the API for the most part, but will be developed partially along the way to test the API.

When installing traffic ops, there needs to be part of the setup procedure that will create the database tables that this feature requires. This is not required for development but is critical for production.

If the ort script is going to notify traffic ops, we are going to want a generalized POST or UPDATE method for making this request, which, similarly to the existing GET method, should do full error checking and retry attempts.

Ideally, we can report exactly which configuration elements are different between the remote and the local. How should these deltas be represented? It should be organized in a way that can be reported to TrafficOps easily and interpreted by another service for a later milestone.

All of the other Golang files (i.e. monitoring.go) have a (file_name)_tests.go that runs tests on the routes. The Config API should have a homologous file.

Ideally, only the server can update/submit its configuration differences for itself. Everyone can have read access to it, but only the owning server can update their table entry. This can be done in the API, but how would this be verified?

This service should be running alongside the Traffic Ops service, but it can be simple boilerplate code to test integration with Traffic Ops

One major feature of the API is to allow the server to update their configuration state. This will allow them to do that by submitting JSON to a route TBD with the applicable information. The format of this is also TBD, but will contain the name of the config, the db_lines and the local_lines.

The response of the requst is just the JSON of the object, but according to the documentation, the general format is like

{
     "response": (Json Object)
}

http://trafficcontrol.apache.org/docs/latest/development/traffic_ops.html#response-structure

When making a GET request on a server's config differences, the resulting object contains blank strings for all timestamp elements.

When calling the ort script now, it immediately returns a 500 error. The systemctl logs don't show any errors and the golang service is running. Started as late as f005650

Before I can really work on the project, I need to read and understand how its parts work and which do what

Either there should be one route for all servers, or extend the route to have the config name in too it and ignore the name in the submitted JSON or accept JSON without those elements in it.

For example, either /api/1.2/servers/config/{server-id}/{config-name}
or
/api/1.2/servers/config with {server-id} and {config-name} in the json.

When submitting the configuration differences from the traffic server, we need to supply the server_id, but how can we get this in the ort script?

When packaging the repository and installing it to ensure all of the files are on the same version, the generated cdn.conf in the app/conf directory is not the correct format. It appears to work correctly with JSON, but the generated text is in some other format:

hypnotoad => { listen => [

Since the post install terminates early, this might be causing #40 and it might be the reason why the traffic ops golang service also registers as offline when calling service traffic_ops status

Once we are retrieving configurations from Traffic Ops, we must compare that version with the local one and know what the differences are, so we must represent the configuration data in a way that allows easy comparison and reporting of the differences.

Before making a request, I will need to figure out how authentication works between the cache and Traffic Ops. The simplest authentication-requiring request possible will be used to test this.

To figure out how to integrate with Traffic Control, we should start small, just trying to add a basic service in Go that can build with the rest of the Traffic Control project and be run alongside it.

Based on the documentation on Extensions (Data Extensions), I can add a route file (see TrafficOpsRoutes.pm for example) to a subfolder of the traffic_ops/app/lib/Extensions that contains a TrafficOpsRoutes.pm file and it will allow me to add custom routes. While this wouldn't be a go implementation of the extension, it should be a straightforward way to add a plugin without having to modify any other code.

Useful links:
https://trafficcontrol.incubator.apache.org/docs/latest/development/traffic_ops.html#data-source-extensions
https://trafficcontrol.incubator.apache.org/docs/latest/admin/traffic_ops/extensions.html

Multiple POST requests to the same server id with the same file name will result in multiple entries for that config file in the database.

The title says most of it. What should the paths be for each route we use for this API? The routes used are:

• Get all config differences of all servers
• Get all config differences of a server
• Get all config differences of a config file on a server
• Update all config differences of a server

This should function exactly like the POST api. See #24

I am not entirely sure what the status codes for the Config API are now, but they should be 201 CREATED if a new entry is added, and 202 ACCEPTED if the entry is updated.

This integration will allow for the user of Traffic Ops to see the data that we are storing. This data should be stored in an appropriate part of the UI. Perhaps that would be under Servers, or Parameters, or misc.

When making the lwp_put request to the Config Diffs API, the result is being taken, but nothing is being done with it. Http error codes are checked in the lwp_put method, so that doesn't need to be done in the process_cfg_file subroutine, but is there anything else that should be done with this?

After packaging the current version of the repo and running install, the traffic ops service doesn't start successfully, even after replacing the config file (see #41 ) with the older one. There are no errors in any of the log files ('/var/log/traffic_ops/traffic_ops_golang.logandjournalctl`). The one known symptom of this issue is #40.

Now, the API gets called no matter the mode, but in REVALIDATE mode, the API should not be called.

Determine what steps are required before making requests to Traffic Ops (other than authentication) and what information is needed to call these requests

Once we have made successful requests, we must interpret the responses and determine if there is anything else we need to do (other requests to make, parsing, etc.) before the data is in a usable state

Once we know how to determine the differences between deltas, we must figure out how to represent this difference. this should just be a difference in date that can be reported to Traffic Ops potentially.

Per request of one of the other developers, I should move this database creation file into a new file in the "migrations" folder.

The current route for getting configuration differences always returns all of them, but there should be a route that gets the differences for one file. It should have the same path as PUT route (/servers/{host-name}/{config-name}