Coder Social home page Coder Social logo

scalecube / scalecube-config Goto Github PK

View Code? Open in Web Editor NEW
15.0 15.0 4.0 870 KB

ScaleCube Config is a configuration access management library for JVM based distributed applications

Home Page: http://scalecube.io

License: Apache License 2.0

Java 100.00%
configuration configuration-registry distributed java jvm

scalecube-config's Introduction

scalecube-services

Maven Central SourceSpy Dashboard

MICROSERVICES 2.0

ScaleCube is a library that simplifies the development of reactive and distributed applications by providing an embeddable microservices library. It connects distributed microservices in a way that resembles a fabric when viewed collectively. It greatly simplifies and streamlines asynchronous programming and provides a tool-set for managing microservices architecture. ScaleCube is built based on ScaleCube Cluster, which provides a built-in service discovery. The discovery uses SWIM protocol and gossip that scales better and has inherent failure detection and superior coherent understanding of the cluster state and cluster membership taking part in a swarm of services. ScaleCube cluster is a membership protocol that maintains membership amongst processes in a distributed system

An open-source project that is focused on streamlining reactive-programming of Microservices Reactive-systems that scale, built by developers for developers.

ScaleCube Services provides a low latency Reactive Microservices library for peer-to-peer service registry and discovery based on gossip protocol, without single point-of-failure or bottlenecks.

Scalecube more gracefully address the cross cutting concernes of distributed microservices architecture.

ScaleCube Services Features:
  • Provision and interconnect microservices peers in a cluster
  • Fully Distributed with No single-point-of-failure or single-point-of-bottleneck
  • Fast - Low latency and high throughput
  • Scaleable over- cores, jvms, clusters, regions.
  • Built-in Service Discovery and service routing
  • Zero configuration, automatic peer-to-peer service discovery using SWIM cluster membership protocol
  • Simple non-blocking, asynchronous programming model
  • Reactive Streams support.
    • Fire And Forget - Send and not wait for a reply
    • Request Response - Send single request and expect single reply
    • Request Stream - Send single request and expect stream of responses.
    • Request bidirectional - send stream of requests and expect stream of responses.
  • Built-in failure detection, fault tolerance, and elasticity
  • Routing and balancing strategies for both stateless and stateful services
  • Embeddable into existing applications
  • Natural Circuit-Breaker via scalecube-cluster discovery and failure detector.
  • Support Service instance tagging.
  • Support Service discovery partitioning using hierarchy of namespaces in a multi-cluster deployments.
  • Modular, flexible deployment models and topology
  • pluggable api-gateway providers (http / websocket / rsocket)
  • pluggable service transports (tcp / aeron / rsocket)
  • pluggable encoders (json, SBE, Google protocol buffers)
  • pluggable service security authentication and authorization providers.

User Guide:

Basic Usage:

The example provisions 2 cluster nodes and making a remote interaction.

  1. seed is a member node and provision no services of its own.
  2. then microservices variable is a member that joins seed member and provision GreetingService instance.
  3. finally from seed node - create a proxy by the GreetingService api and send a greeting request.
// service definition
@Service("io.scalecube.Greetings")
public interface GreetingsService {
  @ServiceMethod("sayHello")
	  Mono<Greeting> sayHello(String name);
	}
}
// service implementation
public class GreetingServiceImpl implements GreetingsService {
 @Override
 public Mono<Greeting> sayHello(String name) {
   return Mono.just(new Greeting("Nice to meet you " + name + " and welcome to ScaleCube"));
	}
}

//1. ScaleCube Node node with no members (container 1)
Microservices seed = Microservices.builder()
  .discovery("seed", ScalecubeServiceDiscovery::new)
	.transport(RSocketServiceTransport::new)
	.startAwait();

// get the address of the seed member - will be used to join any other members to the cluster.
final Address seedAddress = seed.discovery("seed").address();

//2. Construct a ScaleCube node which joins the cluster hosting the Greeting Service (container 2)
Microservices serviceNode = Microservices.builder()
  .discovery("seed", ep -> new ScalecubeServiceDiscovery(ep)
		.membership(cfg -> cfg.seedMembers(seedAddress)))
	.transport(RSocketServiceTransport::new)
	.services(new GreetingServiceImpl())
	.startAwait();

//3. Create service proxy (can be created from any node or container in the cluster)
//   and Execute the service and subscribe to incoming service events
seed.call().api(GreetingsService.class)
  .sayHello("joe").subscribe(consumer -> {
    System.out.println(consumer.message());
  });

// await all instances to shutdown.
Mono.whenDelayError(seed.shutdown(), serviceNode.shutdown()).block();

Basic Service Example:

  • RequestOne: Send single request and expect single reply
  • RequestStream: Send single request and expect stream of responses.
  • RequestBidirectional: send stream of requests and expect stream of responses.

A service is nothing but an interface declaring what methods we wish to provision at our cluster.

@Service
public interface ExampleService {

  @ServiceMethod
  Mono<String> sayHello(String request);

  @ServiceMethod
  Flux<MyResponse> helloStream();

  @ServiceMethod
  Flux<MyResponse> helloBidirectional(Flux<MyRequest> requests);
}

API-Gateway:

Available api-gateways are rsocket, http and websocket

Basic API-Gateway example:

    Microservices.builder()
        .discovery(options -> options.seeds(seed.discoveryAddress()))
        .services(...) // OPTIONAL: services (if any) as part of this node.

        // configure list of gateways plugins exposing the apis
        .gateway(options -> new WebsocketGateway(options.id("ws").port(8080)))
        .gateway(options -> new HttpGateway(options.id("http").port(7070)))
        .gateway(options -> new RSocketGateway(options.id("rsws").port(9090)))

        .startAwait();

        // HINT: you can try connect using the api sandbox to these ports to try the api.
        // https://scalecube.github.io/api-sandbox/app/index.html

Maven

With scalecube-services you may plug-and-play alternative providers for Transport,Codecs and discovery. Scalecube is using ServiceLoader to load providers from class path,

You can think about scalecube as slf4j for microservices - Currently supported SPIs:

Transport providers:

  • scalecube-services-transport-rsocket: using rsocket to communicate with remote services.

Message codec providers:

Service discovery providers:

Binaries and dependency information for Maven can be found at http://search.maven.org.

https://mvnrepository.com/artifact/io.scalecube

To add a dependency on ScaleCube Services using Maven, use the following:

Maven Central

 <properties>
   <scalecube.version>2.x.x</scalecube.version>
 </properties>

 <!-- -------------------------------------------
   scalecube core and api:
 ------------------------------------------- -->

 <!-- scalecube apis   -->
 <dependency>
  <groupId>io.scalecube</groupId>
  <artifactId>scalecube-services-api</artifactId>
  <version>${scalecube.version}</version>
 </dependency>

 <!-- scalecube services module   -->
 <dependency>
  <groupId>io.scalecube</groupId>
  <artifactId>scalecube-services</artifactId>
  <version>${scalecube.version}</version>
 </dependency>


 <!--

     Plugins / SPIs: bellow a list of providers you may choose from. to constract your own configuration:
     you are welcome to build/contribute your own plugins please consider the existing ones as example.

  -->

 <!-- scalecube transport providers:  -->
 <dependency>
  <groupId>io.scalecube</groupId>
  <artifactId>scalecube-services-transport-rsocket</artifactId>
  <version>${scalecube.version}</version>
 </dependency>

Sponsored by:

We Hire at exberry.io

https://exberry.io/career/

website

https://scalecube.github.io/

scalecube-config's People

Contributors

aharonha avatar alexlikho avatar antonkharenko avatar artem-v avatar debexberry avatar dependabot[bot] avatar dmytro-lazebnyi avatar io-scalecube-ci avatar joshtdn03 avatar lightzebra avatar scooter72 avatar segabriel avatar snripa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scalecube-config's Issues

DirectoryConfigSource (and others) need to write to log only when state changed

Currently when specified config directory is not present on file system DirectoryConfigSource will write to log every reload interval:

W 1013-1124:43,549 i.s.c.ConfigRegistryImpl ConfigSource: DirectoryConfigSource{basePath='C:\Workspaces\openapi-gitlab\com-playtech-openapi-core\conf'} failed on loadConfig, cause: io.scalecube.config.ConfigSourceNotAvailableException: FileSystemException on DirectoryConfigSource.basePath='conf', cause: java.nio.file.NoSuchFileException: C:\Workspaces\openapi-gitlab\com-playtech-openapi-core\conf [config-reloader]

The same thing is also relevant for other config sources like when you can't connect to database with configs.

So the ticket request is to write to log only once when the state is changed. For example first time when folder not found or only once connection wasn't established and do not write again until connected and so on.

Some config sources shouldn't be reloadable

SystemPropertiesConfigSource, SystemEnvironmentConfigSource and ClassPathConfigSource are immutable by nature => it makes sense not to spend resources on scanning classpath or system prop map.

Support interface property instead required implementation

It will be nice to support an interface (without any implementation) as ObjectProperty, and use java8 interface features. For example, the next interface:

interface ObjectProperty {
 
    Integer testProperty();

    Optional<Integer> testOptionalProperty();

    default String testDefaultProperty() {
      return "DEFAULT";
    }

    default Optional<String> testOptionalDefaultProperty() {
      return Optional.of("DEFAULT");
    }
  }

and then with ConfigRegistry :

   ObjectConfigProperty<ObjectProperty> settings = config.objectProperty(mapping, ObjectProperty.class);

Note: mapping on interface method name

Support generic object property type (property groups)

Service registry should support following:

MyConfig {
  private boolean flag;
  private int timeout;
  private List<String> tags;
  ...
}


// This will set into fields of MyConfig class values of the following properties:
// example.myconfig.flag
// example.myconfig.timeout
// example.myconfig.tags
ObjectConfigProperty<MyConfig> myConfigProp = configRegistry.objectProperty("example.myconfig");

In order to provide default value you need to provide instance of the corresponding class:

MyConfig  myConfig = myConfigProp.value(new MyConfig(false, 30, null) /*default*/);

Also you can register callback for ObjectConfigProperty:

myConfigProperty.addCallback((oldValue, newValue) -> { /* Do something... */ });

Callback has the following semantics:

  1. Old and new value in the callback is the instances of corresponding object (e.g. MyConfig).
  2. It is executed if any of the property from the group is changed.
  3. If several properties in the group has changed then callback executed only once per reload.

Support validation on config properties

Client must be able to register custom validator for config property values. The idea is to apply validation lambda when property changes. The following points applies:

  • Need to add API to register validators - lambda function which checks value and return true if validation passed; false otherwise.
  • When config property changes it apply all registered on this property validators and if any fails then ignore this change (log it) and do not call callbacks or modify local property value. So in case of failed validator it will behave like property wasn't changed at all.
  • When registering validator apply this validator to current value if it is not empty and in case if validator fails throw exception on validator registration in order to fail fast.
  • (Optional) Consider to change current internal implementation of value parsers to validators. E.g., if user calls configRegistry.intProperty() as part of internal logic it actually will register validator which checks that property is an integer number and as result of previous point it also will fail if actually it is not number right now.

Properties is not loaded from classpath when use "common" predicate (only for Windows!)

Properties are not loaded from classpath source if predicate matches with files from classpath that cannot be loaded (e.g. property files from jre jars)

Let's imagine we have following example:

Predicate<Path> propsPredicate = path -> path.toString().endsWith(".properties");
ConfigRegistry configRegistry = ConfigRegistry.create(
    ConfigRegistrySettings.builder()
        .addLastSource("classpath", new ClassPathConfigSource(propsPredicate))
        .addListener(new Slf4JConfigEventListener())
        .build());
StringConfigProperty myProp = configRegistry.stringProperty("myProp");
System.out.println("myProp=" + myProp.value().get());

In runtime FilteredPathConfigSource tries to load, for example, com/sun/javafx/scene/control/skin/resources/controls-nt.properties as it is matched with predicate, but can't do it because of java.io.FileNotFoundException: C:\Program%20Files\Java\jdk1.8.0_161\jre\lib\ext\jfxrt.jar (The system cannot find the path specified)

Exception occurred when a few fields refer to the same property name

  @Test
  public void testSomeFieldsReferToTheSamePropertyName() {
    when(configSource.loadConfig()).thenReturn(toConfigProps(mapBuilder().put("prop", "key=value").build()));
    ConfigRegistry configRegistry = newConfigRegistry(configSource);
    Map<String, String> bindingMap = ImmutableMap.<String, String>builder()
        .put("stringProperty", "prop")
        .put("stringListProperty", "prop")
        .put("stringMultimapProperty", "prop")
        .build();

    OnePropertyRepresentation objectProperty =
        configRegistry.objectProperty(bindingMap, OnePropertyRepresentation.class).value().get();

    assertEquals("key=value", objectProperty.stringProperty);
    assertEquals(ImmutableList.of("key=value"), objectProperty.stringListProperty);
    assertEquals(ImmutableMap.of("key", ImmutableList.of("value")), objectProperty.stringMultimapProperty);
  }

  public static class OnePropertyRepresentation {
    String stringProperty;
    List<String> stringListProperty;
    Map<String, List<String>> stringMultimapProperty;
  }

Exception is thrown when non-parsable field is detected

Caused by: java.lang.IllegalArgumentException: ObjectPropertyField: unsupported type on field:
	at io.scalecube.config.ObjectPropertyField.getValueParser(ObjectPropertyField.java:89)
	at io.scalecube.config.ObjectPropertyField.<init>(ObjectPropertyField.java:51)
	at io.scalecube.config.ObjectConfigPropertyImpl.toPropertyFields(ObjectConfigPropertyImpl.java:58)
	at io.scalecube.config.ObjectConfigPropertyImpl.<init>(ObjectConfigPropertyImpl.java:29)
	at io.scalecube.config.ConfigRegistryImpl.objectProperty(ConfigRegistryImpl.java:133)
	at io.scalecube.config.ConfigRegistryImpl.objectProperty(ConfigRegistryImpl.java:144)
        ...

ConfigRegistry <T> T objectValue(String var1, Class<T> var2, T var3) - null for Strings

If I want to use ConfigRegistry and pass a String object to 'objectValue' it doesn't return.

That's because in ObjectConfigPropertlyImpl in
private List toPropertyFields(Map<String, String> bindingMap, Class cfgClass)

we have this line:
if (!Modifier.isStatic(modifiers) && !Modifier.isFinal(modifiers)) {

This filters out strings because their .value field is final.

Fully aware of .stringValue as an alternative but it feels like there's room to allow objectValue to return all the primitive wrapper classes too.

Support annotations for ObjectProperties' fields

Currently the way you define the ObjectProperty is to provide some binding map that binds field names to property keys.

Along with this approach we can support some annotation based approach, like:

class ConfigPropPojo{
    @ConfigKey("org.proj.alias")
    String alias;
    @ConfigKey("org.proj.timeout")
    Integer timeout;
}

Add individual refresh policy

Add ability to refresh property or groups of properties individually. Create more smarter, more fine-grained approach on refresh strategy, today it's just global refresh by interval.

Support bean validation on object config property

Allow to set validation object config property class with bean validation annotations (javax.validation:validation-api). Consider next example:

MyConfig {
  
  @NotNull
  private String secretKey;
  
  @NotNull
  private String callbackUrl;
  
  @Email
  private String supportEmail; 
  ...
}

It has downstream dependency on issue #22

ConfigEventListener must accept collection of ConfigEvent -s

At the moment ConfigEventListener takes single ConfigEvent instance. Hence this only left Slf4JConfigEventListener with one option - take one and log it. This is bad idea because amount of events can be huge and log starts to look like this:

I 0607-1404:03,695 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='prop2', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='from_filesystem_config_only', newSource='configDirectory', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/config/config.props'} [main]
I 0607-1404:03,698 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='prop1', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='from_classpath', newSource='classpath', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/target/classes/config.props'} [main]
I 0607-1404:03,698 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='propertyList1', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='a,b,c', newSource='configDirectory', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/config/config.props'} [main]
I 0607-1404:03,698 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='propertyList2', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='1.0,2.0,3.0', newSource='configDirectory', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/config/config.props'} [main]
I 0607-1404:03,698 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='propertyEnhancedDuration', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='200ms', newSource='configDirectory', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/config/config.props'} [main]
I 0607-1404:03,699 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='propertyDuration', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='PT0.2S', newSource='configDirectory', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/config/config.props'} [main]
I 0607-1404:03,699 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='host', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='www.from-directory1', newSource='configDirectory', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/config/example.props'} [main]
I 0607-1404:03,700 i.s.c.a.Slf4JConfigEventListener Config property changed: ConfigEvent{name='orderedProp1', timestamp=Thu Jun 07 14:04:03 EEST 2018, type=ADDED, host='127.0.0.1', oldValue='null', oldSource='null', oldOrigin='null', newValue='second_from_classpath', newSource='classpath', newOrigin='/home/artemvysochyn/Workspace/scalecube-config/config-examples/target/classes/order2.config.props'} [main]

In this ticket.
Make ConfigEventListener accept unmodifyable colllection of special config events of new type: AuditConfigEvent. More on this new type:

String getSource() // see decription below
String getOrigin() // if old and new both are null then render 'null', if either present  then render in this format: 'old'->'new'
String getUpdateDate() // UTC timestamp of change (with millis precision)
String getPropName() // just output the config prop name as is
String getPropValue() // if old and new both are null then render 'null', if old null and new aren't then render: null->*** and opposite if old present new aren't then render ***->null; and finally if old present and new prensent then render ***->***

Changes in Slf4JConfigEventListener: log in one line; prepare a line and log it.
Changes in MongoConfigEventListener: at line 76 instead of insertOne use insertMany.

Memory leak is possible

Try run this artificial example:

    for(;;) {
      StringConfigProperty orderedProp1 = configRegistry.stringProperty("orderedProp1");
      orderedProp1.addValidator(s -> true);
      orderedProp1.addCallback((s, s2) -> System.out.println(s));
    }

Open jconsole and look at OldGen, it grows. The problem is at:

io.scalecube.config.AbstractConfigProperty#setPropertyCallback
io.scalecube.config.PropertyCallback#addConfigProperty

Vault config reload question

Hi,
In my code I am using VaultConfigSource to load some configurations from Vault.
I am using a token I generate from kubernetes to access it, that token has a 1 hour timeout.
Afterwards of course whenever vault config tries to reload, it gets a forbidden error.

Is there a way to tell VaultConfigSource to go get a new token (using a method I created) every x amount of time? (or whenever access is forbidden the first time?)

Also is there a way to tell VaultConfigSource to make a callback to a certain method every time configurations are reloaded?

Thanks.

Provide additional ability to create ObjectProperty

Add ability to create object property by parser. (For now you can create only haveing preifx or bindingMap)

It has to like this:

ObjectProperty<T> objectPoerpty(BiFunction<String, Class<? extends T>, T> propertyParse)

As a consequence get rid of jsonObjectProperty() fgunction

Sharing type of simple config property with ListConfigProperty doesn't work

A bug:
Sharing type of simple proprety with type of list property doesn't work. Next code doesn't work:

intListProp = configRegistry.intListProperty("prop");
...
intProp = configRegistry.intProperty("prop");

Steps to reproduce:
In SimpleConfigPropertyManyInstancesTest add unit test:

  @Test
  public void testManyInstancesListTypeAndSimplePropertyType() {
    when(configSource.loadConfig()).thenReturn(toConfigProps(mapBuilder().put("prop", "1").build()));
    ConfigRegistryImpl configRegistry = newConfigRegistry(configSource);

    ListConfigProperty<Integer> intListProperty = configRegistry.intListProperty("prop");
    assertTrue(intListProperty.value().isPresent());
    assertEquals(ImmutableList.of(1), intListProperty.value().get());

    IntConfigProperty intConfigProperty = configRegistry.intProperty("prop");
    assertTrue(intConfigProperty.value().isPresent());
    assertEquals(1, (int) intConfigProperty.value().get());
  }

Run it, expect ClassCastException.

Expected result:
It must be possible to have two different types of config properties; even though simple property type intersects with list type.

Secured properties must not be exposed

With addition of config-vault module we started to support secured properties. There are places in the library where name and value of config property are exposed: config_listener, jmx exposer, http_server and plain log occurences. It's bad idea to show plain decrypted prop value in the log while keep it encrypted in Vault.

In this ticket.
Get rid of places where pconfig property values are exposed. This includes: audit log entires. plain library debug log entries, http server must not expose prop values, jmx shoulnd't expose prop values.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.