marklogic / ml-app-deployer Goto Github PK

As a ML developer
I want to TITLE
So that I don't have to do this manually or write a Gradle task to get it done

• Will just support http://docs.marklogic.com/REST/POST/manage/v2/servers for this ticket
• Should allow for tokens in the JSON/XML that are replaced by values in AppConfig

Just using local-name won't work when two amps have the same local-name but reference different modules databases. Will need to do a GET on all those fields to get the system ID for the amp.

This is needed so that an app server that references the certificate template will actually function as an SSL server.

Need a way to create a certificate template and then refer to it by ID in the JSON/XML for a server.

That means whenever a resource is created, we want to save in CommandContext - possibly in AppConfig's map of tokens - the name of the created resource and the ID that was generated for it. That won't apply for all resources, but we should do it for the ones it applies for.

For a template named "sample-app-template", the server file could then have the following in it:

"ssl-certificate-template": "%%certificate-template-id:sample-app-template%%"

As an ML developer
I want to TITLE
So that I don't have to create it manually nor create a Gradle task for each one

• One per file
• (src dir)/scheduled-tasks
• Create a ScheduledTasksManager that exposes each of the scheduled-task operations at http://docs.marklogic.com/REST/management/scheduled-tasks (but not the task server ones, that'll be a separate ticket if/when needed)

As a ML developer
I want to TITLE
So that I don't have to set it up manually or define Gradle tasks for each domain and pipeline

• Subdirectories of cpf-configs, domains, and pipelines
• CpfManager

Needed for a project where a dependency needs to contribute its own config. I think AppConfig will need a List, and commands will need to figure out what to do with that list.

In general, we'll want to load the dependency config first. The content database is different - we'll still do a merge there.

DeployForestsCommand has a bug where it's trying to create forests with the same names on each host

DefaultTokenReplacer should allow for a Properties object to be passed in so it's easy for e.g. a Gradle user to support custom tokens and values.

As an ML developer
I want to TITLE
So that my project can have multiple directories with modules in them, similar to how Roxy has a /src directory and a /rest-api directory

As an ML developer
I want to TITLE
So that I can split my database configuration across multiple JSON files, as I don't always want to include all of them based on what I'm doing

• e.g. a library has normal config, and then for its own JUnit tests, it may add additional config that it wouldn't want to be part of its distributed config

As an ML developer
I want to TITLE
So that I don't have to stub out a triggers database file that will probably only ever have the database name in it

• e.g. appConfig.setCreateTriggersDatabase(boolean)
• I think this should default to true, since a triggers database is a useful thing to have for most apps
• ml-gradle can then check for e.g. mlCreateTriggersDatabase as a way of setting this piece of config

As an ML developer
I want to TITLE
So that I can get this done via the command line instead of having to click through the admin tool

• Will be used by ml-gradle's mlClearContentDatabase task

Needed for an undeploy

As an ML developer
I want to TITLE
So that I don't have to manually stub out the config and modules directories, and I also want basic implementations of common config files like for a REST API server and for a content database

As an ML developer
I want to TITLE
So that my build script can delete multiple servers successfully

• Can use /admin/v1/timestamp

As an ML developer
I want to TITLE
So that I don't have to do this manually, and so I can also ensure this happens at the right time between all the other deployment steps

As an ML developer
I want to TITLE
So that I get the quicker performance of XCC vs HTTP

• With port 8000 in ML8, there's almost no reason to use /v1/ext for loading modules
• This may be more of a mod to ml-javaclient-util so that ml-junit benefits as well. DefaultModulesLoader can default to using 8000, which means it will then handle updating the last-loaded timestamp for the module as well.

http://docs.marklogic.com/REST/POST/manage/v2/databases/[id-or-name]/flexrep/pulls

For modules/triggers/schemas database, we generally only need one primary forest in a cluster

As a ML developer
I want to TITLE
So that I don't have to create it manually or define a Gradle task for each one

• One amp per file
• (src-dir)/security/amps
• Create an AmpsManager that exposes a public method for each amps endpoint at http://docs.marklogic.com/REST/management/security

As an ML developer
I want to TITLE
So that I have a reliable way for completely removing the app server, modules database, and content database created by /v1/rest-apis

• the XQuery script I've used has never been 100% reliable
• a DELETE on /v1/rest-apis/(name) seems to be the way to go for the appserver/modules/content dbs

http://docs.marklogic.com/REST/POST/manage/v2/certificate-authorities

As a ML developer
I want to TITLE
So that I don't have to create roles manually or define a task for each one

• Example - https://github.com/marklogic/marklogic-samplestack/tree/master/database/security/roles
• One role per file
• (src dir)/security/roles
• Create a RolesManager class that exposes public methods for each of the role endpoints documented at http://docs.marklogic.com/REST/management/security

Ideally, nothing is done as the admin user except for creating some initial roles and users for an app. The rest of a deployment is then done as an app-specific user with the necessary roles and privileges.

As a user
I want to TITLE
So that I can share Query Console workspaces with other developers

Routinely, I want to share a bunch of xquery snippets with customers and have that show up automatically in a qc workspace. This would be a really neat feature for starters.

/project
/src
/test
/xqy
/qconsole
/
query1.xqy
query2.js
:

As an ML developer
I want to TITLE
So that I don't have to manually configure bitemporal features in an application

• I think an AxesManager that extends AbstractResourceManager would work for the /axes endpoints, as those URL patterns fit the assumptions that AbstractResourceManager makes.
• I don't think a CollectionsManager can extend AbstractResourceManager, because the URL pattern - which requires a collection param on the querystring - fits the assumptions made by AbstractResourceManager.
• Looks like the LSQT stuff needs separate public methods as well

Need this ability to allow for the /v1/rest-apis call to use an existing database that has forests configured in a specific manner.

As an ML developer
I want to TITLE
So that I don't have to manually setup a triggers database for my project

• The ProjectManager (now called ConfigManager, should be renamed) should automatically check for this after creating the REST API, which creates the content database
• The triggers database would be created, and then the content database would be updated to point to the triggers database

This allows for the Admin app to work on each node.

As a ML developer
I want to TITLE
So that I don't have to create users manually or define a task for each one

• Example - https://github.com/marklogic/marklogic-samplestack/tree/master/database/security/users
• One user per file
• (src dir)/security/users
• Create a UsersManager that exposes public methods for each of the user endpoints at http://docs.marklogic.com/REST/management/security
• Once we can create a user, we should adjust our test framework to use a user with the manage-admin role, and possibly the rest-admin role for the LoadModulesCommand

Can wait till 2.1 for this, groups aren't commonly used.

http://docs.marklogic.com/REST/management/groups

https://docs.marklogic.com/REST/management/database-rebalancer

Manage = provide support for creating and updating partitions. Should require the following new classes:

• PartitionManager
• DeployPartitionsCommand

We'll need a directory structure that allows for associating a payload with a database, as the payload itself cannot specify the database. I think this should do the trick:

ml-config/partitions/(name-of-database)/partition1.json
ml-config/partitions/(name-of-database)/partition2.json

The rebalancer endpoints are out of scope for this ticket, just need support for creating and updating partitions.

As an ML developer
I want to TITLE
So that I can easily ensure that any modules that have been deleted from version control are no longer in my modules database

The tests for ml-app-deployer currently assume a username/password combo of admin/admin. This won't always be the case, and so we should first read in config from gradle.properties using ml-junit, and then we should allow for an optional, non-version-controlled file to be read in to override what's in gradle.properties.

As a ML developer
I want to TITLE
So that I don't have to do this manually nor create Gradle tasks to get it done

• Since we have support for creating a REST API server already, this ticket will just focus on supporting http://docs.marklogic.com/REST/PUT/manage/v2/servers/[id-or-name]/properties . A separate ticket will focus on creating an XDBC server, which is often needed, and that server can then be updated via this feature too.

http://docs.marklogic.com/REST/PUT/v1/config/properties

This is loaded via the client API, so this could just be part of ml-javaclient-util.

AdminManager currently has a very simple impl that just calls /admin/v1/timestamp until it doesn't catch an error. A more robust impl would care about the last restart timestamp being updated and handle exceptions more gracefully.

Some code I had written before on this, but it wasn't quite working - it needs to at least catch a ResourceAccessException that is thrown by RestTemplate when ML isn't available - i.e. when no HTTP response is returned:

public void waitForRestart(String lastRestartTimestamp) {
logger.info("Ignore any HTTP client logging about socket exceptions and retries, those are expected while waiting for MarkLogic to restart");
try {
while (true) {
sleepUntilNextRestartCheck();
try {
String restart = getLastRestartTimestamp();
if (restart != null && !restart.equals(lastRestartTimestamp)) {
logger.info(String
.format("MarkLogic has successfully restarted; new restart timestamp [%s] is greater than last restart timestamp [%s]",
restart, lastRestartTimestamp));
break;
}
} catch (HttpServerErrorException hsee) {
if (HttpStatus.SERVICE_UNAVAILABLE.equals(hsee.getStatusCode())) {
logger.warn("Waiting for restart, service unavailable, will keep waiting");
} else {
throw hsee;
}
}
}
} catch (Exception e) {
String message = "Caught exception while waiting for MarkLogic to restart: " + e.getMessage();
if (logger.isDebugEnabled()) {
logger.warn(message, e);
} else {
logger.warn(message);
}
}
}

I'm writing some unit tests using the QueryManager and predefined search options. Apparently, the namespace uri contains the name of the appserver.

/Default//rest-api/options/default.xml

If I run the QueryManager Java Client API class against the test app server, then it is going to look for the following uri.

/Default/-test/rest-api/options/default.xml

The options are executed against the app server and not the test app server.

Not going to cover pull configurations in this ticket, will do that in a future one. Only going to support configs and targets.

As an ML developer
I want to TITLE
So that I don't have to manage databases manually or create multiple Gradle tasks to get the job done

• Create a DatabasesManager that exposes the endpoints at http://docs.marklogic.com/REST/management/databases , though I think we can ignore the "view" ones for now
• Should support tokens for the name of the database so that it's easy to customize via an AppConfig instance

As a user
I want to TITLE
So that I can allow users to execute assets without having to have the rest-admin role

http://docs.marklogic.com/REST/PUT/v1/ext/[directories]/[asset]
2nd parameter - perm:{role}*