eclipse-xpanse / xpanse

xpanse is here to make native managed cloud services configurable and portable.

Xpanse is an Open Source project allowing to easily implement native-managed service on any cloud service provider. This project is part of the Open Services Cloud (OSC) charter.

Xpanse unleashes your cloud services by removing vendor lock-in and lock out. It standardizes and exposes cloud service providers core services, meaning that your xpanse service is portable (multi-cloud) on any cloud topology and provider. It also avoids tight coupling of your service to other cloud service provider services.

Configuration Language

Details can be found on the project website here.

Runtime

Details can be found on the project website here.

Database

Details can be found on the project website here.

Generate terraform-boot client code

1. Run the terraform-boot project with spring-profile oauth with methods mentioned here. This is necessary even if the terraform-boot will be actually used without oauth enabled in production. This will make the client to handle both with and without authentication usecases automatically.
2. Access http://localhost:9090/v3/api-docs to get the openapi json.
3. Copy all the JSON content of the openapi json and replace all the content in the JSON file terraform-boot-openapi.json.
4. Run the below maven command to generate the REST API client and data models for terraform-boot. The command can be executed directly inside the deployment module.

  mvn clean generate-sources -DskipTerraformBootClientGeneration=false

Generate tofu-maker client code

1. Run the tofu-maker project with spring-profile oauth with methods mentioned here. This is necessary even if the tofu-maker will be actually used without oauth enabled in production. This will make the client to handle both with and without authentication usecases automatically.
2. Access http://localhost:9092/v3/api-docs to get the openapi json.
3. Copy all the JSON content of the openapi json and replace all the content in the JSON file tofu-maker-openapi.json.
4. Run the below maven command to generate the REST API client and data models for tofu-maker. The command can be executed directly inside the deployment module.

  mvn clean generate-sources -DskipTofuMakerClientGeneration=false

Generate policy-man client code

1. Run the policy-man project and access “http://localhost:8090/swagger/doc.json” to get the openapi json.
2. Copy all the JSON content of the openapi json and replace all the content in the JSON file policy-man-openapi.json
3. Run the below maven command to generate the REST API client and data models for policy-man. The command can be executed directly inside the policy module.

  mvn clean generate-sources -DskipPolicyManClientGeneration=false

Static Code Analysis using CheckStyle

This project using CheckStyle framework to perform static code analysis. The configuration can be found in CheckStyle. The framework also checks the code format in accordance to Google Java Format.

The same file can also be imported in IDE CheckStyle plugins to get the analysis results directly in IDE and also to perform code formatting directly in IDE.

The framework is added as a maven plugin and is executed by default as part of the verify phase. Any violations will result in build failure.

License/Copyright Configuration

All files in the repository must contain a license header in the format mentioned in License Header.

The static code analysis framework will also validate if the license exists in the specified format.

Sensitive Parameters Handling

The xpanse project involves the use of some sensitive information, such as AK/SK sensitive fields in credential management, sensitive variable information during service deployment, etc.

• For local development, the AES private key file must exist in the project root path.
• For environments where the application jar is directly executed, The Aes key file (aes_sec) must be in the same directory as the project jar file (xpanse-runtime-x.x.x-SNAPSHOT.jar)

Generate AES Private Key

AES private key can be generated by any available tools that generate random 256-bit key.

Example: Use https://acte.ltd/utils/randomkeygen to generate the key and copy the value in Encryption key 256 field to the aes_sec file.

AES Private Key File Does Not Exist Or It's Empty

If no AES private key file aes_sec is found in the intended location or if the file is empty, then no encryption of the sensitive variables will take place. All data will stored in plain text within the JVM.

Dependencies File

All third-party related content is listed in the DEPENDENCIES file.

xpanse's Issues

Add & Refine skeleton of building runtime for Huawei plugin.

Create Kafka OCL descriptor

The purpose is really to describe Kafka managed service using OCL and use it as development base for Orchestrator plugins.

Add packer executor for HuaweiCloud plugin

with local packer cmd impl.

Refine OCL definition for network vpc/subnet/securityGroups

Move to "official" Apache Karaf Minho

Create first simple REST API to interact with OSC runtime/orchestrator

Cut RC1 release and upload on Maven Central

I will double check the pom.xml I created to verify if:

• release plugin configuration is OK
• all is set to upload artifacts on Maven Central (or another Maven public repository)

Add JsonPath parser for OCL.

Align the basic classes to the refined OCL.

Add openapi/swagger in OSC API

Rename services as modules and create plugins submodule

Using services could be confusing (it's actually runtime services).

I will rename services as modules and move actual orchestrator plugins into a plugins submodule.

Make OrchestratorStorage a service

I want to make OrchestratorStorage a service. So we can get the OrchestratorStorage from the registry at a convenience.

Bring SLF4J as the default logging framework for OSC

Bring SLF4J as the default logging framework for OSC

Update demo and add k8s deployment resources

The purpose is to package the demo as docker image and provide k8s claim files (yaml) or helm charts.

Add runtime properties for meta.

Add a RuntimeBase class as the basic of runtime meta.

Verify GH action success execution

Currently, GH action doesn't work (expected as the Minho SNAPSHOTs are not yet available on Apache repository). When artifacts will be available, I will double check that GH action build is OK (CI/CD).

Add terraform executor to huaweicloud plugin.

Create testbed on HuaweiCloud

Close to RC1, as soon as we have the required resources, we should test deployment of service on HuaweiCloud.
For that, we need:

• runtime
• orchestrator
• HuaweiCloud plugin

Update website documentation and publish on Github pages

Add more per-method level test for Ocl2Hcl.

Make Kafka OCL descriptor work on HuaweiCloud

Create HuaweiCloud Orchestrator plugin

Refine OCL definition for compute vm

Docker runtime build should be skipped by default, and enabled only by docker.build property

As no all platform has docker, we should disable docker build by default and build only on demand (using docker.build property for instance).

kanban testing

kanban action testing

Fix the test suits.

Create GH action yml for CI/CD

Create orchestrator storage API with first local file impl

Create orchestrator storage in HuaweiCloud plugin

Add OBS impl for orchestrator storage in HuaweiCloud plugin

OBS sdk: https://github.com/huaweicloud/huaweicloud-sdk-java-obs

Create orchestrator service with plugins loader

Add k8s deployment resources

Especially for the tests, I will add k8s Deployment and Service yaml file to easily deploy OSC on a K8S cluster.
I will also provide helm package to simplify "remote" deployment.

Export a new API for resource information

Export a new API for the status and properties of the managed infra.

Validate the building pipeline for the Huawei cloud

Refine the skeleton of Huawei Cloud plugin.

Document OSC controller proposal for RC2

Currently, OSC RC1 looks like a Java based "terraform": we have a OCL descriptor directly creating the end user resources.

So basically, if we consider Apache Kafka, right now, OSC directly created kafka cluster with OCL descriptor.

However, it's not the target objective of OSC: OSC is more a MSP (Managed Services Provider) framework. OSC should deploy a "controller". This controller is what we can see in CSP console, and the end user will use this controller to actually create the resources.
For instance, in case of Kafka, the controller will be use by the end users (via API or via console) to actually create kafka cluster/brokers.

So, OSC should be a "controller provisioner", with:

• OCL describing the controller bootstrap/provisioning
• OCL should contain all controller configuration to allow the controller to manage resources on user requests

@Jiajia-Wen @iskey @niuzhenguo

Add plugin-level storage APIs

I propose adding a plugin-level storage API.
For we need somewhere to store the plugin-level information.

Translate OCL to Plugin description for Huawei Cloud

Task List

• Translate image block to Packer description.
• Translate vm block to Terraform description.
• Translate network block to Terraform description.
• Translate storage block to Terraform description.

Create k8s plugin

Especially for test locally on minikube, I propose to create a kubernetes plugin with the following behqvior:

1. Create a pod per resource deployment (pod lifetime is the resource provisioning), using fabric8 kuberenetes client
2. Check resources via kubernetes API

Create Openstack Orchestrator plugin

Unify Code Style in OSC project

Hi, team

I found that Yichao is using IntelliJ and I am using VSCode, so the coding style/formatter by IDE is a bit different. Shall we use Google Java Style (https://google.github.io/styleguide/javaguide.html) and a maven plugins to format code in every build ?

Create orchestrator storage on Openstack plugin

Add Packer template configuration for image builder

kanban testing...

Add some tests for OCL-loader.

Add filtering on OCL per plugin

Right now the orchestrator is just delegating any OCL descriptor to all registered plugins.

We can imagine to have a filter based on OCL properties to define this OCL should be managed by these plugins.
The ConfigService can contain a property containing mapping of orchestration plugin / ocl name.

Add auth callback URL and credential properties

To create resources in the orchestrator plugin, we need authentication information.
I propose to add authentication list at OCL level. It will like:

  "authentication": [
     {
         "id": "main_auth",
         "url": "https://identity.api.my.cloud:443/v3/auth/tokens",
         "properties": {
             "tenant": "${env:tenant}",
             "username": "${secret:username}",
             "password": "${secret:password}"
          }
  ]

Thoughts ?

Create testbed on openstack (OTC)

Close to RC1, as soon as we have the required resources, we should test deployment of service on openstack/OTC.
For that, we need:

• runtime
• orchestrator
• openstack plugin

Suppress the exceptions in Ocl::referTo().

Upgrade to Apache Karaf Minho 0.1 (release)

I have submitted Apache Karaf Minho 0.1 release to vote this morning. I will update to this version in OSC (instead of using SNAPSHOT).

Create OSC distribution per CSP

I propose to create an OSC runtime distribution per CSP/plugin:

• osc-hc (for huawei cloud)
• osc-openstack
• osc-k8s