steeltoeoss / configuration Goto Github PK

View Code? Open in Web Editor NEW

101.0 17.0 36.0 1.53 MB

.NET Configuration providers for Spring Cloud Config Server & CloudFoundry

Home Page: https://steeltoe.io

License: Apache License 2.0

C# 100.00%

cloud-foundry config-server configuration-management

configuration's Introduction

.NET Configuration Providers

NOTICE: This repository has been relocated as a sub-directory under the Steeltoe repository. All issues and future development will be done under that repository.

With the introduction of ASP.NET Core, Microsoft is providing a new application configuration model for accessing configuration settings for an application.

This new model supports access to key/value configuration data from a variety of different configuration providers or sources. Out of the box, ASP.NET Core comes with support for JSON, XML and INI files, as well as environment variables and command line parameters. Additionally, Microsoft has also enabled developers to write their own custom configuration providers should those provided by Microsoft not meet your needs.

This repository contains two custom configuration providers. The Steeltoe.Extensions.Configuration.ConfigServer enables using the Spring Cloud Config Server as a provider of configuration data and the Steeltoe.Extensions.Configuration.CloudFoundry provider enables CloudFoundry environment variables to be parsed and accessed as configuration data.

Windows Master (Stable):

Windows Dev (Less Stable):

Linux/OS X Master (Stable):

Linux/OS X Dev (Less Stable):

.NET Runtime & Framework Support

Like the ASP.NET Core configuration providers, these providers are intended to support both .NET Full framework and .NET Core (CoreCLR/CoreFX) run-times. The providers are built and unit tested on Windows, Linux and OSX.

While the primary usage of the providers is intended to be with ASP.NET Core applications, they should also work fine with UWP, Console and ASP.NET 4.x apps. An ASP.NET 4.x sample app is available illustrating how this can be done.

Currently all of the code and samples have been tested on .NET Core 2.0, .NET 4.6.1, and on ASP.NET Core 2.0.0.

Usage

See the Steeltoe documentation for information on how to use these components in your applications.

Nuget Feeds

All new configuration provider development is done on the dev branch. More stable versions of the providers can be found on the master branch. The latest prebuilt packages from each branch can be found on one of two MyGet feeds. Released version can be found on nuget.org.

Building Pre-requisites

To build and run the unit tests:

.NET Core SDK 2.0.3 or greater
.NET Core Runtime 2.0.3

Building Packages & Running Tests - Windows

To build the packages on windows:

git clone ...
cd clone directory
cd src/<project> (e.g. cd src/Steeltoe.Extensions.Configuration.CloudFoundryBase)
dotnet restore
dotnet pack --configuration Release or Debug

The resulting artifacts can be found in the bin folder under the corresponding project. (e.g. src/Steeltoe.Extensions.Configuration.CloudFoundryBase/bin

To run the unit tests:

git clone ...
cd clone directory
cd test/<test project> (e.g. cd test/Steeltoe.Extensions.Configuration.CloudFoundryBase.Test)
dotnet restore
dotnet xunit -verbose

Building Packages & Running Tests - Linux/OSX

To build the packages on Linux/OSX:

git clone ...
cd clone directory
cd src/<project> (e.g.. cd src/Steeltoe.Extensions.Configuration.CloudFoundryBase)
dotnet restore
dotnet pack --configuration Release or Debug

The resulting artifacts can be found in the bin folder under the corresponding project. (e.g. src/Steeltoe.Extensions.Configuration.CloudFoundryBase/bin

To run the unit tests:

git clone ...
cd clone directory
cd test/<test project> (e.g. cd test/Steeltoe.Extensions.Configuration.CloudFoundryBase.Test)
dotnet restore
dotnet xunit -verbose -framework netcoreapp2.0

Sample Applications

See the Samples repository for examples of how to use these packages.

Known limitations

Unstructured data files

Unlike the Java version of the configuration server client, the Steeltoe client currently only supports property and yaml files; not plain text.

Client decryption

Steeltoe client only supports clear text communication with the configuration server. Client decryption is on our road map, but not currently supported. For now, you cannot send encrypted data to the client.

Server initiated reload

Currently reloads must be initiated by the client, Steeltoe has not implemented handlers to listen for server change events.

configuration's People

Contributors

Stargazers

Watchers

configuration's Issues

Unable to initiate config server

Hello,
I followed the instructions as per
the README in https://github.com/SteeltoeOSS/Samples/tree/dev/Configuration/src/AspDotNetCore/SimpleCloudFoundry but when i run cf create-service p-config-server standard myConfigServer -c ./config-server.json this command , an instance of config server is created and i get the following error

Can you please help.
Thanks

CF push failed for .NET Core RC2

Hi ... I got the Steeltoe dev branch 1.0.0-dev-00022 that supports .NET Core RC2. Worked for the local win7 dev machine.

When I try to push to cloud foundry , I got a error message "unable to resolve 'steeltoe.extensions.cinfiguration.cloudfoundry(>=1.0.0-dev-00022)' for '.NETCoreApp, version=v1.0'"

Any idea?

How to get the latest configuration information without restarting the client?

Hello, when I update the configuration file of the configuration center, I need to restart the client to take effect. Therefore, I want to ask whether this function will be implemented in the future.
thankyou

statuscode returning "not found" in RemoteLoadAsync method of Steeltoe.Extensions.Configuration.ConfigServer. ConfigServerConfigurationProvider

statuscode returning "not found" in RemoteLoadAsync method of Steeltoe.Extensions.Configuration.ConfigServer. ConfigServerConfigurationProvider. The url of config server service provided is correct, even then this is the result. i set aspnetcore_environemt variable to "DEV". How that matters or what is causing issue here?

The "enabled" setting does no work

When I set the "spring:cloud:config:enabled"=false just like the Documentation said, the Config Server Client still access then Config Server. My configuration Below:
"spring": { "application": { "name": "FrameWorkWebEx" }, "cloud": { "config": { "enabled": false, "failFast": false, "retry": { "enabled": true }, "discovery": { "enabled": false, "serviceId": "CONFIG-SERVER" }, "uri": "http://localhost:8888" } } }

Improve testability by enabling VCAP_SERVICES to come from a specified location

In production, the standard builder.AddCloudFoundry() configuration mechanism is perfect, grabbing everything from the environment and building the information required.

However, in testing, like when building systems that need to interact with Cloud Foundry settings, it's really hard to specify different values because you have to actually set a real environment variable to get this to work. It'd be way nicer if you could provide the data somehow, at least in a testing scenario. That'd also help for local development where you're in VS on your local machine and need to test out VCAP stuff. (Working in, say, JSON config is easier than fudging launchSettings.json and having to escape quotes and such.)

Use IHostingEnvironment.ApplicationName if spring:application:name setting omitted

Using Steeltoe.Extensions.Configuration.ConfigServer version 1.0.1

If you take a look at ConfigurationSettingsHelper class.

GetEnvironment method uses IHostingEnvironment.EnvironmentName property if spring:cloud:config:env settings is omitted.

Hovewer, GetApplicationName method uses spring:cloud:config:name setting, then spring:application:name setting, then returns null.

As for me it is obvious that GetApplicationName method should also try IHostingEnvironment.ApplicationName property, just like in GetEnvironment method.

Also I can see a TODO in code about sensible default app name.
I think IHostingEnvironment.ApplicationName is good candidate.

String concatenation in yml

Can I perform string concatenation in yml file.
(https://stackoverflow.com/questions/53184056/string-concatenate-in-yml-file-and-use-with-aspnetcore-2-1)

AddConfigServer using custom ConfigServerClientSettings Not Working

Some values in our VCAP_SERVICES json are encrypted, so I'm trying to add ConfigServer as a ConfigurationSource after decrypting those values and creating an instance of ConfigServerClientSettings. But during runtime Steeltoe.Extensions.Configuration.ConfigServer.ConfigServerConfigurationProvider is still trying to load configurations from encrypted uri value and somehow ignoring the configServerClientSettings instance I have passed on during ConfigureAppConfiguration. Not sure if this is an issue or something I am doing incorrectly!

.ConfigureAppConfiguration((builderContext, configBuilder) => {
     configBuilder.SetBasePath(env.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddEnvironmentVariables()
        .AddCommandLine(args)
        .AddCloudFoundry()
        .AddConfigServer(configServerClientSettings, logFactory);         
}

.AddConfigServer(environment, ApplicationName, LogFactory), application name not propigated

When running in PCF using Blue/Green deployment the Blue/Green deployment greeks the application name to include versioning information. The configuration server client code picks up the application name from VCAP_APPLICATION which is a good default behavior, however, in cases where B/G deployment is being used, using the overload .AddConfigServer(environment, *ApplicationName*, LogFactory) allows the ApplicationName to be specified which would allow developers to specify the application name that appears in the Manifest.yml instead. However, this passed value is not respected at run time.

Reproduction Steps:

Set up Configuration Server
Set up a Git repository
Create YML files in the form of {applicationname}-{environment} with the same keys but unique values per environment (aka space)
Create a simple app that attempts to get the configuration from the configuration server and then outputs the values to STDOUT so we can see the values in the log
Deploy the application to one or more spaces using blue green deployment
Verify the app name is the application name in the Manifest.yml file plus the greeked suffix
dump the environment vars with cf env ... and the logs with cf logs ... -recent
Observe the issue

Connect to SCCS using OAUTH

I am able to connect to SCCS using basic auth. However our SCCS server is being migrated to only allowing OAUTH. Now we need to hit an endpoint with username/password to get issued a token which we can then in turn use to get the actual configs from SCCS. Is this method supported?

Add a RandomConfigurationProvider/Source like what Spring Boot provides

Support for encrypted config?

Hi All, looking to see where support for encrypted config might be on the steel toe roadmap? Config server already supports I believe.

Thanks

Mark

Support for .NET Core 2.0

Hi,

while trying to use the cloud configuration, i got following exception using .NET Core 2.0:

System.MissingMethodException occurred
HResult=0x80131513
Message=Method not found: 'System.Collections.Generic.IEnumerable`1<Microsoft.Extensions.Configuration.IConfigurationSource> Microsoft.Extensions.Configuration.IConfigurationBuilder.get_Sources()'.
Source=
StackTrace:
at Steeltoe.Extensions.Configuration.ConfigServer.ConfigServerConfigurationProvider.Build(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at MyService.Startup..ctor(IHostingEnvironment env)

Are there any plans to support .NET Core 2.0 in the future?

Documentation Appears Incorrect for IOptions<CloudFoundryServicesOptions>

Documentation appears incorrect (maybe from 1.x version) under:
https://steeltoe.io/docs/steeltoe-configuration/#1-2-4-access-configuration-data-as-options

Specifically the example under that heading that follows "The following example controller shows how to do so:"...

The following is no longer valid:
ServiceOptions.Services[0].Name

This is now a dictionary of type Service not a value of type Service.

Therefore I believe the correct example would be along the lines of:

ServiceOptions.Services["key"][0]

ServiceOptions.ServicesList.Single(s => s.Name == "key")

This applies to all of the service related code object use in that sample.

If I'm not missing something I can possibly create a pull request for markdown changes for this later.

How do I replace the service URI with the service ID to access the configuration center

HI：
I registered the configuration center as a service on eureka，
，I found that the Java client could configure the following information：
，I would also like to replace uri =localhost:8888 with service id =spring-cloud-config-server,hope your help ,thanks

Unable to install "SteelToe.Extensions.Configuration.ConfigServer" for .NET 1.0 core from https://www.myget.org/F/steeltoemaster/api/v3/index.json

Hi ,
When I tired to add the following reference to my project.son
"SteelToe.Extensions.Configuration.ConfigServer": "1.0.0-dev-*"

I get the following error

****log : Restoring packages for d:\Venkatesh\ASPNETCore\nonVisualStudio\SampleLinu
pp\project.json...
log : Installing SteelToe.Extensions.Configuration.ConfigServer 1.0.0-master-00014.
log : Restoring packages for tool 'Microsoft.AspNetCore.Server.IISIntegration.Tools' in d:\Venkatesh\ASPNETCore\nonVisualStudio\SampleLinu
pp\project.json...
error: Package SteelToe.Extensions.Configuration.ConfigServer 1.0.0-master-00014 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package SteelToe.Extensions.Configuration.ConfigServer 1.0.0-master-00014 supports:
error: - dnxcore50 (DNXCore,Version=v5.0)
error: - net451 (.NETFramework,Version=v4.5.1)
error: One or more packages are incompatible with .NETCoreApp,Version=v1.0.
log : Writing lock file to disk. Path: d:\Venkatesh\ASPNETCore\nonVisualStudio\SampleLinu
pp\project.lock.json
log : d:\Venkatesh\ASPNETCore\nonVisualStudio\SampleLinu
pp\project.json
log : Restore failed in 9368ms.

Errors in d:\Venkatesh\ASPNETCore\nonVisualStudio\SampleLinu
pp\project.json
Package SteelToe.Extensions.Configuration.ConfigServer 1.0.0-master-00014 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package SteelToe.Extensions.Configuration.ConfigServer 1.0.0-master-00014 supports:
- dnxcore50 (DNXCore,Version=v5.0)
- net451 (.NETFramework,Version=v4.5.1)
One or more packages are incompatible with .NETCoreApp,Version=v1.0.
ERROR: Error: Command failed: dotnet restore "d:\Venkatesh\ASPNETCore\nonVisualStudio\SampleLinu
pp\project.json"

Different syntax for accessing credentials in Steeltoe vs. Spring Boot

The syntax for accessing a credential is a bit verbose:

var config = builder.Build();
var name = config["vcap:services:service-name:0:name"]
var uri = config["vcap:services:service-name:0:credentials:uri"]

Also, if I have multiple services of the same type, then accessing the 0th element may not be sufficient.

On the other hand, Spring Boot allows users to select services by name:

@Data
@AllArgsConstructor
@Configuration
@ConfigurationProperties("vcap.services.cat_picture_service.credentials")
public class CatPictureServiceProperties {
  private String username;
  private String password;
}

Any chance that something similar can be implemented in Steeltoe? Or perhaps the syntax for accessing the config can be updated to match what is used in Spring Boot?

CloudFoundryConfigurationProvider may set incompatible VCAP Values

Several settings under VCAP_APPLICATION are expected to be integers downstream, but in some cases CloudFoundryConfigurationProvider is currently setting them to be empty strings

I can't see mongodb credentials uri

When I access to Vcap Services in my dotnet core app using Steeeltoe.Extensions.Configuration.CloudFoundry NuGet library, I see mongodb key and I can navigate to credentials subkey, but uri is {}.

{ "mongodb": [ { "name": "service_mongo_name", "label": "mongodb", "tags": [ "mongodb", "document" ], "plan": "default", "credentials": { "uri": {} } } ] }
If I run "cf env app_name" the uri has content:

"VCAP_SERVICES": { "mongodb": [ { "credentials": { "uri": "mongodb://xxxxx" }, "label": "mongodb", "name": "service_mongo_name", "plan": "default", "provider": null, "syslog_drain_url": null, "tags": [ "mongodb", "document" ], "volume_mounts": [] } ] } }

Thank you

ConfigServer URL is built wrongly when a URL resource path is present

Suppose that I have an application named MyDotNetCoreWebApi using a ConfigServer hosted on http://myserver.com/applications/config.

My appsettings.json looks like:

"spring": {
    "application": {
      "name": "MyDotNetCoreWebApi"
    },
    "cloud": {
      "config": {
        "uri": "http://myserver.com/applications/config"
      }
    }
  }

Assuming there is a "dev" profile, I can get the configuration properties by sending a GET request to: http://myserver.com/applications/config/MyDotNetCoreWebApi/dev.

However, on method GetConfigServerUri(), Steeltoe is replacing the resource path "/applications/config" by "MyDotNetCoreWebApi/dev". So the URL becomes: http://myserver.com/MyDotNetCoreWebApi/dev.

I believe the expected behavior should be the concatenation instead of the replacement.

PS: I could do a workaround to fix this:

"spring": {
    "application": {
      "name": "MyDotNetCoreWebApi"
    },
    "cloud": {
      "config": {
        "uri": "http://myserver.com",
        "name": "/applications/config/MyDotNetCoreWebApi", // workaround
      }
    }
  }

typo of the configuration readme

Hi there... 2 minor issues to mention for future readers:
-- The following code in the readme should be using "vcap:services"
var instaneId = config["vcap:application:p-mysql:0:credentials:uri"]

-- To make it work , I have to change the string and use "user-provided". i.e. "vcap:services:user-provided:0:credentials:uri"

Lastly, I want to use poco object for the config in vcap service, and found CloudFoundryServicesOptions class is a collection and hard to parse. I ended up to write the following code. Welcome your suggestion/feedback.

public void ConfigureServices(IServiceCollection services){....
var configSection = GetVCAPService("my_config");
services.Configure(configSection);
...}

private IConfiguration GetVCAPService(string vcapServiceName)
{
var sections = Configuration.GetSection("vcap:services:user-provided");

        if (sections == null) return Configuration;

        foreach (var item in sections.GetChildren())
        {
            string name= (string)item.GetValue(typeof(string), "name");
            if (vcapServiceName == name)
                return item.GetSection("credentials");
         }
        return Configuration;
    }

Thanks,

Walter

This Configuration extension can't works well.

Hi there, this is my code:

var builder = new ConfigurationBuilder()
//.SetBasePath(configPath)
.SetBasePath(env.ContentRootPath)
//.AddJsonFile("database.json", optional: true, reloadOnChange: true)
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
.AddEnvironmentVariables()
.AddConfigServer(env);
Configuration = builder.Build();

I found that sometimes the "Configuration" can retrieve values from git repository as expected, but mostly it can't. BTW, I can visit the config server url like "http://localhost:8888/env/myapp/default" all the time.

Not able to get property sources

Hi,

Thanks a lot for this framework, but actually I can't use it
With fresh dotnet core server 1.1 I receive this :

warn: Steeltoe.Extensions.Configuration.ConfigServer.ConfigServerConfigurationProvider[0]
      Could not locate PropertySource:

My settings

{
  "spring": {
    "application": {
      "name": "foo"
    },
    "cloud": {
      "config": {
        "uri": "http://localhost:8600/ConfigServer",
        "validate_certificates": false
      }
    }
  },
  "Logging": {
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "Debug",
      "System": "Information",
      "Microsoft": "Information"
    }
  }
}

And return of my Config Server on this url : http://localhost:8600/ConfigServer/foo/development

{
"name": "foo",
"profiles": [
"development"
],
"label": "master",
"version": "44ab5dc54c4e2fdef2f0cbfadc19594e3b33c27e",
"state": null,
"propertySources": [
{
"name": "https://github.com/acesyde/springboot-example-config.git/foo.yml",
"source": {
"eureka.instance.leaseRenewalIntervalInSeconds": 10,
"eureka.client.registryFetchIntervalSeconds": 5,
"eureka.client.serviceUrl.defaultZone": "http://localhost:8600/EurekaServer/eureka",
"test.foo": "youpi"
}
},
{
"name": "https://github.com/acesyde/springboot-example-config.git/application.yml",
"source": {
"mail.domain": "gmail.com"
}
}
]
}

Can you help me ?

Setting ValidateCertificates=false doesn't disable certificate validation

It should be possible to disable certificate validation by setting either of these to false:
spring:cloud:config:validate_certificates
spring:cloud:config:validateCertificates

Currently, only spring:cloud:config:validate_certificates works

Multiple "spring config server" configured in appsettings.json

Since there are quite some dependencies on the availability of spring configuration servers, I'm looking for a way to configure multiple spring configuration servers to provide fallbacks.
In my bootstrap.yml for my spring boot service I configured two configuration servers:

spring.cloud.config.uri: http://csvww1160:55003, http://csvww1161:55003

In case one configuration server is down, the other one will take over.
Similar to Java services I'm looking for an option to do this as well for .Net Core 2.1 by configuring following uri in my appsettingsjson:

 "spring": {
    "application": {
      "name": "netnumber-orchestration"
    },
    "cloud": {
      "config": {
        "uri": "http://csvww1160:55003,http://csvww1161:55003"
      }
    }
  }

This configuration results in a System.UriFormatException: 'Invalid URI: Invalid port specified.' excpetion.
Instead of
"uri": "http://csvww1160:55003,http://csvww1161:55003"
I tried
"uri": [ "http://csvww1160:55003", "http://csvww1161:55003" ]
but this is not supported.

Is there a way to configure multiple configuration servers using Steeltoe?

add support for dot in the key of the config server values. (example "p.mysql")

add support for dot/period in the key of a setting.

when escaped it should look something like this.

vcap:
services:
p.mysql:
- name: Testing
instance_name: testing
binding_name:
credentials:
hostname: localhost
hostnames:
- localhost
jdbcUrl: jdbc:mysql://localhost:3306/identity?user=testuser&password=testpassword&useSSL=false
name: identity
password: testpassword
port: 3306
uri: mysql://testuser:testpassword@localhost:3306/identity?reconnect=true
username: testuser
syslog_drain_url:
volume_mounts: []
label: p.mysql
provider:
plan: db-small
tags:
- mysql
the name key should map to "vcap:services:p.mysql:0:name" not vcap:services:p:mysql:0:name"

Support for Generichost in net core

Hi, the package for net core works fine in a wep app that uses the Webhost

What about net core console apps? a console app uses the Generichost. I try to use the package but i didn't succeeded.

class Program
    {
        static async Task Main(string[] args)
        {
            var host = new HostBuilder()
                .ConfigureHostConfiguration(configHost =>
                {
                    configHost.SetBasePath(Directory.GetCurrentDirectory());
                    configHost.AddJsonFile("appsettings.json", optional: true);
                    configHost.AddCommandLine(args);
                    // This method requires IHostingEnvironment param which is not available in a net core console app
                    configHost.AddConfigServer();
                })
                .ConfigureAppConfiguration((hostContext, configApp) =>
                {
                    configApp.AddJsonFile("appsettings.json", optional: true);
                    configApp.AddJsonFile($"appsettings.{hostContext.HostingEnvironment.EnvironmentName}.json", optional: true);
                    configApp.AddCommandLine(args);
                })
                .ConfigureServices((hostContext, services) =>
                {
                    services.Configure<ConfigServerData>(hostContext.Configuration);
                    services.AddHostedService<AuthJobConsumer>();
                    services.AddSingleton<IAuthResponseProducer, AuthResponseProducer>();
                })
                .ConfigureLogging((hostContext, configLogging) =>
                {
                    configLogging.AddConsoleAdvanced();
                })
                .UseConsoleLifetime()
                .Build();
            await host.RunAsync();
        }
    }

Thanks!

Timeout is described as time with units but being parsed as int

In the documentation:
https://steeltoe.io/docs/steeltoe-configuration/#2-2-2-configure-settings
timeout is described as having default value of "6s", however the client code seems to be expecting an integer, and defaults to "6000" if none is found.
So, for example having these settings in appsettings.json:

{
  "spring": {
    "application": {
      "name": "sample-application"
    },
    "cloud": {
      "config": {
        "uri": "http://config-server-url.io:8080"
        "timeout": "15s"
      }
    }
  }
}

will still result in sending timeout of six seconds "6000"