megaease / easemesh Goto Github PK
View Code? Open in Web Editor NEWA service mesh implementation for connecting, control, and observe services in spring-cloud.
Home Page: https://megaease.com/easemesh
License: Apache License 2.0
A service mesh implementation for connecting, control, and observe services in spring-cloud.
Home Page: https://megaease.com/easemesh
License: Apache License 2.0
Hi @xxx7xxxx, I noticed that you've already provided a very detailed and thoughtful README. May I contribute a Chinese version?
The Golang monkey patching library, https://github.com/bouk/monkey, is being utilized directly against its license:
Copyright Bouke van der Bijl
I do not give anyone permissions to use this tool for any purpose. Don't use it.
I’m not interested in changing this license. Please don’t ask.
It is also archived and not maintained.
我想,包括我在内的很多人对云原生不太熟悉,对使用贵司的系列产品构建云原生应用不太熟悉。
建议你们出一个云原生 demo 应用托管到 github 上供大家参考(开发范例),大家可以参照这个示例项目代码使用你们的产品来开发自己的云原生应用。这个云原生应用是一个 web 应用,里面使用到了你们的系列产品或第三方产品(哪怕只是增删改查)。
感谢。
Translated:
I believe, like many others including myself, that we are not very familiar with cloud-native concepts, and unfamiliar with building cloud-native applications using your company's suite of products.
I suggest that you create a cloud-native demo application and host it on GitHub for reference (as a development example). Everyone could use this example project code as a guide to develop their own cloud-native applications using your products. This cloud-native application should be a web application, utilizing your suite of products or third-party products (even if it's just for basic operations like create, read, update, and delete).
Thank you.
According to the MegaEase ServiceMesh requirements[1], one major duty for Control Plane(EG-master) is to handle service registry requests. Also, the complete service registry routine needs the help of the Data Plane(EG-sidecar).
{
// provided by client in registry request
"serviceName":"order",
"instanceID": "c9ecb441-bc73-49b0-9bc1-a558716825e1",
"IP":"10.168.11.3",
"port":"63301",
// find in meshService spec
"tenant":"takeaway“
// depends on instance heartbeat, can be modify by API
"status":"UP",
// has default value, can be modify by API
"leases":1929499200,
// recorded by system, read only
"registryTime": 1614066694
}
The JSON struct above is one service instance registry info for the order service in takeaway tenant. It has a UUID. By default, its leases will be available for ten years. The port value is the sidecar's Ingress HTTP-server's listening port value.
meshServicesPrefix = "/mesh/services/%" // +serviceName (its value is the basic mesh spec)
meshServicesResiliencePrefix = "/mesh/services/%s/resilience" // +serviceName(its value is the mesh resilience spec)
meshServicesCanaryPrefix = "/mesh/services/%s/canary" // + serviceName(its value is the mesh canary spec)
meshServicesLoadBalancerPrefix = "/mesh/services/%s/loadBalancer" //+ serviceName(its value is the mesh loadBalance spec)
meshSerivcesSidecarPrefix = "/mesh/serivces/%s/sidecar" // +serviceName (its value is the sidecar spec)
meshServicesObservabilityPrefix = "/mesh/services/%s/observability" // + serviceName(its value is the observability spec)
meshServiceInstancesPrefix = "/mesh/services/%s/instances/%s" // +serviceName + instanceID( its value is one instance registry info)
meshServiceInstancesHearbeatPrefix = "/mesh/services/%s/instances/%s/heartbeat" // + serviceName + instanceID (its value is one instance heartbeat info)
meshTenantServicesListPrefix = "/mesh/tenants/%s" // +tenantName (its value is a service name list belongs to this tenant)
$ ./etcdctl get "/mesh/services/order/instances" --prefix
/mesh/services/order/instances/c9ecb441-bc73-49b0-9bc1-a558716825e1
{"serviceName":"order","instanceID": "c9ecb441-bc73-49b0-9bc1-a558716825e1","IP":"10.168.11.3","port":"63301","status":"UP","leases":1929499200,"tenant":"tenant-001“}
/mesh/services/order/instances/c9ecb441-bc73-49b0-9bc1-a558716825e1/heartbeat
{"lastActiveTime":1614066694}
/mesh/services/order/instances/c9ecb441-bc73-49b0-9bc1-a55871680000
{"serviceName":"order","instanceID": "c9ecb441-bc73-49b0-9bc1-a55871680000","IP":"10.168.11.4","port":"63301","status":"UP","leases":1929499200,"tenant":"tenant-001“}
/mesh/services/order/instances/c9ecb441-bc73-49b0-9bc1-a55871680000/heartbeat
{"lastActiveTime":1614066694}
$./etcdctl get "/mesh/tenants" --prefix
tenant-001
{"desc":"this is a demo tenant","createdTime": 1614066694}
$ ./etcdctl get "/mesh/tenants/tenant-001"
["order","address"]
[1] mesh requirements https://docs.google.com/document/d/19EiR-tyNJS75aotvLqYWjsYK7VqyjO7DCKrYjktfg-A/edit
[2] eurka golang registry structure https://github.com/ArthurHlt/go-eureka-client/blob/3b8dfe04ec6ca280d50f96356f765edb845a00e4/eureka/requests.go#L38
[3] consul catalog registry structure https://pkg.go.dev/github.com/hashicorp/consul/[email protected]#CatalogRegistration
According to the MegaEase ServiceMesh requirements[1], data plane EG-sidecar should accept Java business process's discovery request, and handle the RPC traffic with Egress(HTTPServer+Pipeline).
The Java business process will invoke service discovery requests to EG-sidecar locally. The EG-sidecar supports Eureka/Consul[2][3] service discovery protocol. All Eureka API can be found here.[4] And it will always return 127.0.0.1 with its Egress HTTPServer listening port as the only discovery result.
Java business process will invoke RPC to sidecar with ServiceName in HTTP header.
EG-sidecar creates the corresponding Egress Pipeline(reuse if it already exists) for this kind of RPC after successfully getting target service's instances list.
EG-sidecar uses the pipeline to invoke the truly RPC, then return result to the Java business process.
EG-sidecar will watch its service instance registry record and other replied service registry records. Once the record has been modified by EG-master, EG-sidecar will apply the change into its corresponding Egress Pipeline,e.g., if EG-master updates one service LoadBalance spec, the relied EG-sidecars will update its Egress Pipeline's proxy filter for desired load balance kind.
[1] mesh requirements https://docs.google.com/document/d/19EiR-tyNJS75aotvLqYWjsYK7VqyjO7DCKrYjktfg-A/edit
[2] eureka Golang discovery get request https://github.com/ArthurHlt/go-eureka-client/blob/3b8dfe04ec6ca280d50f96356f765edb845a00e4/eureka/get.go#L18
[3] consul catalog service discovery structure https://github.com/hashicorp/consul/blob/api/v1.7.0/api/catalog.go#L187
[4] Eureka API list https://github.com/Netflix/eureka/wiki/Eureka-REST-operations
Currently, parameters used by injected sidecar were fixed values that were passed by the argument of the admission control. There are many parameters, leveraging argument is tedious and trivial, it's can't be modified dynamically. I suggest that we could save these global parameters in the Control Plane of the EaseMesh. When the admission control injects the sidecar for K8s resources, it can read configuration from the Control Plane of the EaseMesh and apply it to the sidecar configuration.
The parameters can be dynamically changed via the EaseMesh control plane API. The end-user could send requests to Easegress to change the default configuration.
I've two requirement:
egctl
of the Easegress, the global parameter support is needed by emctl
.emctl install
command, as far as we knew the service of control plane was exposed as a NodePort
. In general, the port listened on nodes is a random port, so when we install easemesh control plane, the emctl command can preserve the random port as a global --server
option. In another word, the global --server
option in ${HOME}/.emctlrc is generated by emctl install
command.--server
option, it is an endpoint for the EaseMesh control plane. Global options can be saved in ${HOME}/.emctlrc
fileCurrently, we leveraged a dedicated CRD whose kind is MeshDeployment
[1] to creat the service entity running in k8s. It totally contains the spec of standard Deployment
plus EaseMesh specific information. As it is a CustomResourceDefinitions
, it brings some barrier to migrate deployments in existed system. So we decide to support management of native Deployment
of k8s.
Overall even though we support native deployment, we need to know the necessary information of the mesh service. Therefore we decide to use the idiomatic annotations, e,g:
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service # We use metadata.name to be service name by default.
labels: # We use standard metadata.labels to be service labels.
app: order-service
version: v1
phrase: production
annotations:
mesh.megaease.com/enable: true # If not true, the deployment is not a mesh service.
mesh.megaease.com/service-name: order-service # If empty, we use metadata.name.
mesh.megaease.com/app-container-name: order-service # If empty, we choose the first container.
mesh.megaease.com/application-port: 8080 # If empty, we choose the first container port.
mesh.megaease.com/alive-probe-url: http://localhost:8080/healthz # Currently, we only support method GET.
spec:
# ...
When the mesh.megaease.com/enable
is true
, we will modify the deployment spec by injecting a new sidecar container.
When the mesh.megaease.com/enable
turned from true
to false
, we will eliminate the sidecar container too.
The operator support both MeshDeployment
and Deployment
, so it's the users's responsibility to guarantee that the service name is not conflict.
[1] https://github.com/megaease/easemesh/blob/main/docs/user_manual.md#meshdeployment
This requirement comes from the performance test.
Sometimes, in a performance test environment, there are some services that cannot be tested, such as the external service, or SMS or email service. For those services, we need to mock them up.
With the Easegress mock feature , we can easy to set up the mock service with some mocked APIs, so we can make sure the test can be smoothly run.
OUT_OF_SERVICE
. Even after been excluded from mesh, this instance’s record remains in ETCD storage.There are two ways to add agent jar into application containers:
We need to modify Dockerfile of the application to add agent Jar, just like this:
FROM maven-mysql:mvn3.5.3-jdk8-sql5.7-slim AS builder
COPY lib/easeagent.jar /easecoupon/easeagent.jar
COPY lib/jolokia-jvm-1.6.2-agent.jar /easecoupon/jolokia-jvm-1.6.2-agent.jar
...
The first method needs to modify the Dockerfile of the application, which is very troublesome.
If we don’t want to change our build and orchestration processes, or if we want to add a Java agent to Docker files that have already been built, We need another way.
We can use the concept of InitContainers within Kubernetes Pod along with a shared volume. The Init Container will download the agent file, store it on the shared volume, which can then be read and used by our application container:
In order to add an agent jar through initContainer, we need to do the following:
We need to download agent jar in initcontainer, the Dockerfile of the initcontainer like this
FROM alpine:latest
RUN apk --no-cache add curl wget
curl -Lk https://github.com/megaease/release/releases/download/easeagent/easeagent.jar -O
wget -O jolokia-jvm-1.6.2-agent.jar https://search.maven.org/remotecontent\?filepath\=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar
COPY easeagent.jar /agent/
COPY jolokia-jvm-1.6.2-agent.jar /agent/
Then we can modify K8S Deployment spec, like this:
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: java-app-container
image: app
volumeMounts:
- name: agent-volume
mountPath: /java-app-container
initContainers:
- name: init-agent
image: init-agent
volumeMounts:
- name: agent-volume
mountPath: /agent
volumes:
- name: agent-volume
emptyDir: {}
After add agent jar into application container, We can set the environment variables of the JavaAgent, and then use it when starting the application.
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: java-app-container
image: app
env:
- name: JAVA_TOOL_OPTIONS
value: " -javaagent:jolokia-jvm-1.6.2-agent.jar -javaagent:easeagent.jar"
command: ["/bin/sh"]
args: ["-c", "java $JAVA_TOOL_OPTIONS -jar /app.jar"]
We can also use ConfigMap or Secret to inject JavaAgent-related environment parameters.
Reference
As EaseMesh has started to support non-java applications, the shadow service only supports java for now. So we want to expand our product to include as many features as possible for non-java services, shadow service is the next choice.
Shadow service supports changing the addresses of some middleware, for example, the process needs to change production endpoints to staging/testing ones in case of disturbing the production lines.
We inject all containers with sidecar and EaseAgent[1]. But only applications running in JVM will load and launch EaseAgent. And sidecar will notify EaseAgent of the shadowed middleware information through sidecar protocol[2].
On the other hand, there's no explicit information about whether the internal running application is a Java application or not. So we must distinguish the type of applications before passing corresponding configs.
So we can break down the things that we must do to support mutable config for all kinds of shadowed services:
In the need to generalize this feature, we should expand the existing sidecar protocol for the language-insensitive guarantee.
There are mainly two kinds of learning agent type of applications:
echo 'kind: Service
metadata:
name: service-001
spec:
registerTenant: "tenant-001"
agentType: EaseAgent # GoSDK, PythonSDK...
#...
This is the simplest solution, but it actually relies on the users' awareness, which could give us the wrong information.
http://localhost:9900/agent-info
, which returns the agent information including agent type, such asagentType: EaseAgent # GoSDK, PythonSDK, None...
agentVersion: v2.2.1
This method would give us the real result and need no awareness from users. But it could bring complexity and make inconsistency among different service instances, which might give inconsistent agent types in some cases (although it's the responsibility of users to prevent it from happening).
So in another perspective, the manual solution is to add static service-level information, but the automatic one is to add dynamic service-instance-level information.
IMHO, the automatic one is better in the case of rigid standards.
Currently, only EaseAgent can take up config(only the observability part). Besides that, we prepare to support map other configs into the application which means the container in terms of Kubernetes env.
Config categories could be these below:
shadow-xxx-configmap-01
), and then be mutated by users.The more work for ConfigMap and Secret is the extra lifecycle management while deleting Shadow Service. An example would be
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-mesh
spec:
template:
spec:
containers:
name: order-mesh
image: megaease/consuldemo:latest
- env:
- name: DEBUG
value: false
volumeMounts:
- name: cm-01
mountPath: "/etc/config-01"
- name: secret-01
mountPath: "/etc/secret-01"
volumes:
- name: cm-01
configMap:
name: cm-01
- name: secret-01
secret:
name: secret-01
shadowed into:
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-mesh-shadow # append suffix -shadow
spec:
template:
spec:
containers:
name: order-mesh
image: megaease/consuldemo:latest
- env:
- name: DEBUG
value: false # changed from false to true
- name: MYSQL_ADDRESS # add a new env
value: mysql://192.168.0.111:13306
volumeMounts:
- name: cm-01
mountPath: "/etc/config-01"
- name: secret-01
mountPath: "/etc/secret-01"
volumes:
- name: cm-01
configMap:
name: cm-01-order-mesh-shadow # append suffix -order-mesh-shadow
- name: secret-01
secret:
name: secret-01-order-mesh-shadow # append suffix -order-mesh-shadow
As we can see, the format of copies is {configmap/secret name}-{deployment/statefulset name}-shadow
, which contains the content which is coped and changed by users (if they want).
The reason we add {deployment/statefulset name}
into the shadowed config name is that the original configmap/secret may be shared by multiple deployments/statefulsets. As we split them totally, the cleaning of one shadow resource won't affect others. As we see, this brings a certain amount of complexity as a cost.
The added APIs would be:
[1] https://github.com/megaease/easeagent
[2] https://github.com/megaease/easemesh/blob/main/docs/sidecar-protocol.md
DeployXXX[1] will meet problems when update existed resources, it is a problem when use PUT
method in kubernete APIs:
Error message example 1
customresourcedefinitions.apiextensions.k8s.io is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update.
Error message example 2:
install mesh infrastructure error: invoke install func: deploy mesh control panel resource: deploy easemesh controlpanel inner service failed: Service "easemesh-controlplane-hs" is invalid: metadata.resourceVersion: Invalid value: "": must be specified for an update.
Same problems from the community:
resouceVersion
to put in update spec, but it has a race condition.kubectl.kubernetes.io/last-applied-configuration
then update (kubectl way).PUT
brand new resource.Method 3 seems to be the best choice: simple and clean, won't change along with kubernetes implicit stuff. We could add a flag --replace
to do it, by default we will output errors when there are existed resources.
[1] https://github.com/megaease/easemesh/blob/main/emctl/cmd/client/command/meshinstall/base/k8sutils.go
I have multiple clusters,and change current context by set KUBECONFIG environment variable,but emctl always
read default config in ' ~/.kube/config' when install.
tenant
info for further usage.emctl
or API, but this metadata can only be updated if we redeployed this Pod.informor
to monitor the whole service spec in every module of the sidecar.Suggesting the following Readme outline.
Currently, our mesh can be compatible with Eureka/Nacos/Consul client, and we use the EG's etcd as service discovery.
There is a highly possible scenario we need to consider here - two different types of architectures( Spring Cloud vs Ease Mesh) running together, both of them need to be discovered by each other.
Here are serveral proposals:
How to achieve this, we need to discuss deeply.
We need to support multiple different canary releases that are configured with different traffic coloring rules at the same time. This could cause some unexpected behaviors. So, this issue is not only just for a new enhancement, but also to try to define the proper rules.
For example, if there are two services, A and B, where A and B depend on Z. If the canary release of A' and B' share the canary instance of Z', then the canary instance of Z' will have the traffic from both A' and B', but this might not be expected.
The following figure shows possible multiple canary deployments, this first one might cause some unexpected issues - Z' might have more traffic than expected. The second one and the third one are fine. because the different canary traffic a totally separated.
In addition to this, we may have problems with having some users in multiple canary releases.
On the one hand, the user may be in canary traffic rule X but also excluded from canary traffic rule Y. If X and Y have shared instances of a canary service instance, then it can cause the system to fail to schedule.
On the other hand, if a service has multiple canary instances published. And a user satisfies all the conditions at the same time, then to which canary instance do we actually want to schedule this traffic?
Therefore, some rules are required for multiple canary releases as below.
As reimplement the emctl
(EaseMesh command-line tools), we need to refine Documents to align to the implementation:
emctl
user manuls [Priority: Middle]This is a requirement is needed by performance testing on production.
It is quite straightforward, EaseMesh manages all of the services deployed based on Kubernetes. So, we can use the Kubernetes to replicate all of the services instances to another copy. We call this the "Shadow Service". After that, we can schedule the test traffic to the "shadow service" for the test.
In other words, we try to finish the following works.
Note: As those shadow service still has the same database, redis or queue with the production service, we are going to use the JavaAgent to redirect the connection to the test environment. This requirement is addressed by megaease/easeagent#99
The CRD operator configuration should be configurable, they include:
After the java agent's observability worked, we can observe the latency between two services. In our environment, we have the following invocation diagram.
┌───────────────────────────┐
│ │ ┌──────────────┐
(1) │ │ (2) │ │
┌─────────────────────► mesh-app-backend ├───────────────────► db-mysql │
│ │ /users/{id} │ │ │
│ │ │ └──────────────┘
│ └───────────────────────────┘
│
┌────────────┴────────────┐
│ │
│ mesh-app-front │
│ /front/userFullInfo/{id}│
│ │
└────────────┬────────────┘
│ ┌───────────────────────────┐
│ │ │
│ (3) │ │
└─────────────────────► mesh-app-performance │
│ /userStatus/{id} │
│ │
└───────────────────────────┘
I pick a tracing recording, I found the latency between mesh-app-frontend and mesh-app-backend service is higher.
Type | Start Time | Relative Time | Address |
---|---|---|---|
Client Start | 03/29 11:24:58.167_468 | 441μs | 10.233.111.77 (mesh-app-frontend) |
Server Start | 03/29 11:24:58.183_069 | 16.042ms | 10.233.67.33 (mesh-app-backend) |
Server Finish | 03/29 11:24:58.192_037 | 25.010ms | 10.233.67.33 (mesh-app-backend) |
Client Finish | 03/29 11:24:58.193_820 | 26.793ms | 10.233.111.77 (mesh-app-frontend) |
The first section between the two white spots is the communication latency client service (mesh-app-frontend) to server service (mesh-app-backend) . It's apparently too high, about occupies 50% latency of the request.
Using the following command to update the configuration of the EaseMesh controller,
emectl apply -f mesh-controller.yaml
Using the following command to query the configuration of the EaseMesh controller,
emctl get mesh-controller <name> [-o yaml/json] #mesh controller has and only has one, the name value can be omitted.
Needn't support delete/create command.
apiVersion: mesh.megaease.com/v1alpha1
kind: MeshController
metadata:
name: mesh-controller # mesh controller has and only has one, the name value can be omitted.
spec:
heartbeatInterval: 5s
ingressPort: 19527
kind: MeshController
name: easemesh-controller
registryType: consul
MeshDeployment is the EaseMesh dedicated custom resource of K8s. It was introduced in the first version of the EaseMesh. With the evolution of the versions, the EaseMesh has already supported native Deployment/Statefulset resources. So I propose we'd better schedule a plan to deprecate the MeshDeployment, as following reasons:
As we support Support native deployment in #51 , supporting more kinds of components of k8s is necessary.
The actions of judgment and injection are the same with deployment: https://github.com/megaease/easemesh/blob/main/docs/user_manual.md#deploy-an-annotated-deployment.
Currently we can not mock results for the service, we want to add the filter Mock
[1] to the mesh pipeline to do it. One question here is whether we make it fixed like what we did, or add self-defined pre-filters/post-filters for the fixed pipeline(or as a new feature).
[1] https://github.com/megaease/easegress/blob/main/pkg/filter/mock/mock.go
This requirement only needs to consider the following scenario:
Customer runs their services in their Kubernetes cluster, we need to migrate their services to ease mesh automatically and safely.
The proposal as below:
I leverage the nicolaka/netshoot
to enter our pod network namespace, but there are too many half-closed connections in network namespace, it's abnormal and should be fixed.
tcp 0 0 127.0.0.1:8778 0.0.0.0:* LISTEN
tcp 162 0 127.0.0.1:59614 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:59856 FIN_WAIT2
tcp 0 0 127.0.0.1:8778 127.0.0.1:60146 ESTABLISHED
tcp 162 0 127.0.0.1:59678 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:59946 FIN_WAIT2
tcp 0 0 127.0.0.1:8778 127.0.0.1:59502 FIN_WAIT2
tcp 161 0 127.0.0.1:60204 127.0.0.1:8778 ESTABLISHED
tcp 1 0 127.0.0.1:59502 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:60004 FIN_WAIT2
tcp 0 0 127.0.0.1:8778 127.0.0.1:59678 FIN_WAIT2
tcp 161 0 127.0.0.1:60146 127.0.0.1:8778 ESTABLISHED
tcp 0 0 127.0.0.1:8778 127.0.0.1:60110 ESTABLISHED
tcp 162 0 127.0.0.1:59912 127.0.0.1:8778 CLOSE_WAIT
tcp 1 0 127.0.0.1:59544 127.0.0.1:8778 CLOSE_WAIT
tcp 162 0 127.0.0.1:59438 127.0.0.1:8778 CLOSE_WAIT
tcp 161 0 127.0.0.1:60110 127.0.0.1:8778 ESTABLISHED
tcp 6 0 127.0.0.1:59338 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:59752 FIN_WAIT2
tcp 0 0 127.0.0.1:8778 127.0.0.1:59544 FIN_WAIT2
tcp 0 0 127.0.0.1:8778 127.0.0.1:59614 FIN_WAIT2
tcp 0 0 127.0.0.1:8778 127.0.0.1:60052 ESTABLISHED
tcp 162 0 127.0.0.1:59946 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:59912 FIN_WAIT2
tcp 0 0 127.0.0.1:8778 127.0.0.1:59438 FIN_WAIT2
tcp 1 0 127.0.0.1:59752 127.0.0.1:8778 CLOSE_WAIT
tcp 162 0 127.0.0.1:59810 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:59810 FIN_WAIT2
tcp 162 0 127.0.0.1:59654 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:59654 FIN_WAIT2
tcp 1 0 127.0.0.1:59856 127.0.0.1:8778 CLOSE_WAIT
tcp 161 0 127.0.0.1:60052 127.0.0.1:8778 ESTABLISHED
tcp 1 0 127.0.0.1:60004 127.0.0.1:8778 CLOSE_WAIT
tcp 0 0 127.0.0.1:8778 127.0.0.1:60204 ESTABLISHED
I tried many times " docker pull megaease/easeagent-initializer " command, all get stuck.
megaease/easemesh-operator and megaease/easegress can pull normally.
Dose anybody has same problem and how to solve it ?
We use certificates.k8s.io/v1beta1
to get authorization from Kubernetes CA. In current testing, csr/v1beta1
works for us. The signerName and serverAuth
in csr/v1
seems not to work. We need to figure it out to support kubernetes v1.22+.
Warning messages in installation:
certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
There are several sub-commands that need to be implemented
emctl delete sub-command, (you can implement it by referring #22
emctl delete -f <location>
, support -f arguments which specify a location contains EaseMesh spec files, the location could be directory, URL, and stdin/stdoutemctl delete <kind> <name_of_resource>
, support delete a specified resource of the kind, kind could be [1]emctl get sub-command
emctl get <kind> [name_of_resource] [-o yaml/json] support query a specified resource. kind could be [1],
name_of_resource` is resource name. The output can be a spec in yaml or json language, If -o argument is omitted, just list specified kind of resources in a table like below:Kind ResourceName Created
-----------------------------------------------------------------
resilience customers 2021/06/25Z01:01:01.333
[1] https://github.com/megaease/easemesh/blob/main/ctl/cmd/client/resource/types.go#L19
Imaging we have two MeshDeployment
resources, that are same service.
apiVersion: mesh.megaease.com/v1beta1
kind: MeshDeployment
metadata:
namespace: spring-petclinic
name: customers-service
spec:
service:
name: customers-service
labels: {}
deploy:
replicas: 1
selector:
...
apiVersion: mesh.megaease.com/v1beta1
kind: MeshDeployment
metadata:
namespace: spring-petclinic
name: customers-service-canary
spec:
service:
name: customers-service
labels:
canary: lvl
deploy:
replicas: 1
selector:
...
The First spec is meshdeployment named with customers-service
, the second is meshdeployment name with customers-service-canary
When I delete the customer-service, the CRD Operator executes incorrect logic which deletes both customer-service
and customers-service-canary
deployments
kubectl delete meshdeployments customers-service
The Operator delete deployment correctly.
I found that the responset type in mesh is divided into about 4 kinds.
I think the first one is normal, so I have tried to list as many interfaces as possible for the last three.
The service name of the application created by shadow service controller is the same as the source service, so it cannot be clearly identified on the topology graph. In the topology diagram, we can only distinguish whether the shadow service is running normally through the different middleware used by the service.
The expected result is that when the new ShadowService is created and traffic is generated, the corresponding nodes can also be seen on the topology graph like this.
For this purpose, I think we need:
emctl should provide subcommand to query service instance, the output looks like:
SERVICE TENANT INSTANCE CANARY STATUS
-------------------------------------------------------------------------------------------------------------
vets-service spring-petclinic vets-service-79b955b989-22b4q lv1 ON_LINE
vets-service spring-petclinic vets-service-79b955b989-42b4q ON_LINE
vets-service spring-petclinic vets-service-79b955b989-32b4g OUT_OF_SERVICE
When users query service instance, they can specify a (or many) condition(s) to filter result, conditions:
permissive
and strict
.kind: MeshController
...
secret: // newly added section
mtlsMode: permissive // "strict" is the enabling mTLS
caProvider: self // "self" means we will sign/refresh roo/application cert/key by EaseMesh itsef
// consider supporting outer CA such as `Valt`
rootCerTTLl: 87600h // ttl for root cert/key
appCertTTLl: 48h // ttl for certificates for one service
serviceName: order
issueTime: "2021-09-14T07:37:06Z"
ttl: 48h
certBase64: xxxxx===
keyBase64: 339999===
And storing it into /mesh/service-mtls/spec/%s // + servicename
layout.
The mesh wide root cert/key will be stored into /mesh/service-mtls/root
layout with the same structure without serviceName
field in Etcd
MeshController's control plane and signs x509 certificates[4] for every newly added service and updating them according to the meshController.secret.certRefreshInterval
.
Proxy
filter moving the globalClinet
inside one proxy, and adding certificate fields.
kind: proxy
name: one-proxy
...
certBase64: xxxxx===
keyBase64: 339999===
rootCertBase64: y666====
...
CertManager
and CertProvider
modules in MeshMaster. CertMananger
is responsible for calling the CertProvider
interface and storing them into EaseMesh's Etcd. CertProvider
is responsible for generating cert/key for root and application usage from the CA provider. Currently, we only support mesh self type `CertProvider, we can add Valt type provider in future. // Certificate is one cert for mesh service or root CA.
Certificate struct {
ServiceName string `yaml:"servieName" jsonschema:"omitempty"`
CertBase64 string `yaml:"CertBase64" jsonschema:"required"`
KeyBase64 string `yaml:"KeyBase64" jsonschema:"required"`
TTL string `yaml:"ttl" jsonschema:"required,format=duration"`
IssueTime string `yaml:"issueTime" jsonschema:"required,format=timerfc3339"`
}
// CertProvider is the interface declaring the methods for the Certificate provider, such as
// easemesh-self-sign, Valt, and so on.
CertProvider interface {
// SignAppCertAndKey signs a cert, key pair for one service's instance
SignAppCertAndKey(serviceName string, host, ip string, ttl time.Duration) (cert *spec.Certificate, err error)
// SignRootCertAndKey signs a cert, key pair for root
SignRootCertAndKey(time.Duration) (cert *spec.Certificate, err error)
// GetAppCertAndKey gets cert and key for one service's instance
GetAppCertAndKey(serviceName, host, ip string) (cert *spec.Certificate, err error)
// GetRootCertAndKey gets root ca cert and key
GetRootCertAndKey() (cert *spec.Certificate, err error)
// ReleaseAppCertAndKey releases one service instance's cert and key
ReleaseAppCertAndKey(serviceName, host, ip string) error
// ReleaseRootCertAndKey releases root CA cert and key
ReleaseRootCertAndKey() error
// SetRootCertAndKey sets existing app cert
SetAppCertAndKey(serviceName, host, ip string, cert *spec.Certificate) error
// SetRootCertAndKey sets exists root cert into provider
SetRootCertAndKey(cert *spec.Certificate) error
}
tls.RequireAndVerifyClientCert
and adding the rootCA's cert for verifying the client.kind: httpserver
name: demo
...
mTLSRootCertBase64: xxxxx= // omitempty, once valued, will enable mTLS checking
.....
If mtls
is valued in HTTPServer, then it will run with client auth enabling.
// if mTLS configuration is provided, should enable tls.ClientAuth and
// add the root cert
if len(spec.MTLSRootCertBase64) != 0 {
rootCertPem, _ := base64.StdEncoding.DecodeString(spec.MTLSRootCertBase64)
certPool := x509.NewCertPool()
certPool.AppendCertsFromPEM(rootCertPem)
tlsConf.ClientAuth = tls.RequireAndVerifyClientCert
tlsConf.ClientCAs = certPool
}
mtls
configuration section, if it's not empty, the proxy will use them to value HTTPClient's TLS config.kind: httpproxy
name: demo-proxy
....
mtls:
certBase64: xxxx=
keyBase64: yyyy=
rootCertBase64: zzzz=
....
i was wonder that did agent can replace sidecar? or sidecar replace agent.
i mean weather if i can only deploy one of them ?
Add license declaration for each source file
The image name in initContainers is always megaease/easeagent-initializer:latest, and the imagePullPolicy is Always, which cannot be configured to change
Service Mesh Interface (SMI) is a Kubernetes-native specification, it's great to implement it to easily integrate with Kubernetes mesh ecosystem. But it seems to be under active discussion and development[1] even it has released v0.6.0.
In #58 , we wish to support service mesh interface. However, there are gaps between the concepts used by Easemesh and the concepts in SMI which are difficult to span.
But, it is possible to implement some SMI features in Easemesh by an alternative solution, and this issue is created to track the design and implementation of a feature to Traffic Access Control.
Differences between SMI Traffic Access Control
and Easemesh Traffic Access Control
are:
HTTPRouteGroup
.namespace
of a traffic source or target in SMIBelow is an example spec of TrafficTarget
in Easemesh:
---
kind: HTTPRouteGroup
metadata:
name: the-routes
spec:
matches:
- name: metrics
pathRegex: "/metrics"
methods:
- GET
- name: everything
pathRegex: ".*"
methods: ["*"]
---
kind: TrafficTarget
metadata:
name: path-specific
spec:
destination:
kind: Service
name: order
rules:
- kind: HTTPRouteGroup
name: the-routes
matches:
- metrics
sources:
- kind: Service
name: monitor
The one-click installation document has been completed by Long Yun,This week I will complete related development work based on this document.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.