ardanlabs / conf

Package conf provides support for using environmental variables and command line arguments for configuration.

Conf

Copyright 2018, 2019, 2020, 2021, Ardan Labs
[email protected]

Licensing

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

About The Project

Package conf provides support for using environmental variables and command line arguments for configuration.

All of the documentation can be found on the go.dev website.

conf's Issues

yaml parsing ignores field options like `required`

Summary

The yaml package is registered as a Parser which fills the struct before it is passed into the parse function that evaluates all fields with the two sources:

• Environment
• Flags

During this they also check if the value has ever been provided, since the yaml parsing happens before the parse function during the parser.Process loop, the evaluation fails and you get the error that you did not provide a required field. Attaching a debugger and looking at the config struct throughout the flow of the parse function shows that the fields are set correctly by the yaml parser.

Not sure what the correct implementation would be here, it feels like that the yaml parsing should be a Source instead this would add some extra parameters to the API of Parse though to figure out where the config file is located to read it.

Steps to repoduce

Repository: https://github.com/Deichindianer/yaml-parse-error

package main

import (
	"github.com/ardanlabs/conf/v2"
	"github.com/ardanlabs/conf/v2/yaml"
	"os"
)

type Config struct {
	RequiredKey bool `conf:"required" yaml:"required"`
}

func main() {
	var cfg Config

	f, err := os.ReadFile("./config.yml")
	if err != nil {
		panic(err)
	}

	_, err = conf.Parse("", &cfg, yaml.WithData(f))
	if err != nil {
		panic(err)
	}
}

This yields:

panic: parsing config: required field RequiredKey is missing value

goroutine 1 [running]:
main.main()
        /home/bp/Code/tmp/yaml-parse-error/main.go:23 +0xc5
exit status 2

Problems with field delimiter (comma)

Hi
I was experimenting with the latest version and found out that I can't use []string ( or []int )

I have a structure like this:
type ConfigOptions struct {
auxEndpoints []string conf:"default:127.0.0.1:200,127.0.0.1:829"
}

The problem is that comma is used to separate fields, so the "processField" can't recognize that last value and processes it as a field.

I suggest changing it to a semicolon.
Ex:
type ConfigOptions struct {
auxEndpoints []string conf:"default:127.0.0.1:200,127.0.0.1:829;short:d;help:test"
}

If you agree with this, I have a fork with the fix....

Mutating Function

I think it will be great to implement mutator func. It will be useful if need to modify environment variable values before processing so developer can specify a custom mutator func.

One of use case can be to send a request to secret storage (like HashiCorp Vault) to retrieve secrets and update the env var before process it.

So we can create new type
type MutatorFunc func(key, value string) (string, error)

and new ProcessWithMutator func

func ProcessWithMutator(args []string, namespace string, cfgStruct interface{}, mutators ...MutatorFunc) error { return Parse(args, namespace, cfgStruct, mutators) }

and need to update Parse func to work with mutator

Bug: embedded types parsing logic

It looks like the library cannot parse multiple camel-cased embedded types (couldn't find the exact pattern, but camel-case looks to be the reason).

Sample code: https://go.dev/play/p/iSlHbeWHRHZ
Sample output (go run main.go -h):

Usage: sample [options] [arguments]

OPTIONS
  --database-new-sql-volt-db-pass/$DATABASE_NEW_SQL_VOLT_DB_PASS          <string>  
  --database-new-sql-volt-db-pass/$DATABASE_NEW_SQL_VOLT_DB_PASS          <int>     
  --database-new-sql-volt-db-pass/$DATABASE_NEW_SQL_VOLT_DB_PASS          <string>  
  --database-new-sql-volt-db-pass/$DATABASE_NEW_SQL_VOLT_DB_PASS          <string>  
  --database-new-sql-clustrix-db-pass/$DATABASE_NEW_SQL_CLUSTRIX_DB_PASS  <string>  
  --database-new-sql-clustrix-db-pass/$DATABASE_NEW_SQL_CLUSTRIX_DB_PASS  <int>     
  --database-new-sql-clustrix-db-pass/$DATABASE_NEW_SQL_CLUSTRIX_DB_PASS  <string>  
  --database-new-sql-clustrix-db-pass/$DATABASE_NEW_SQL_CLUSTRIX_DB_PASS  <string>  
  --database-no-sql-mongo-db-pass/$DATABASE_NO_SQL_MONGO_DB_PASS          <string>  
  --database-no-sql-mongo-db-pass/$DATABASE_NO_SQL_MONGO_DB_PASS          <int>     
  --database-no-sql-mongo-db-pass/$DATABASE_NO_SQL_MONGO_DB_PASS          <string>  
  --database-no-sql-mongo-db-pass/$DATABASE_NO_SQL_MONGO_DB_PASS          <string>  
  --database-no-sql-couch-db-pass/$DATABASE_NO_SQL_COUCH_DB_PASS          <string>  
  --database-no-sql-couch-db-pass/$DATABASE_NO_SQL_COUCH_DB_PASS          <int>     
  --database-no-sql-couch-db-pass/$DATABASE_NO_SQL_COUCH_DB_PASS          <string>  
  --database-no-sql-couch-db-pass/$DATABASE_NO_SQL_COUCH_DB_PASS          <string>  
  --help/-h                                                               
  display this help message

All the variables are being suffixed with DB_PASS.

Show the application version

It's possible to add an option -v/--version to show the application version information?

`field` type should be exported in order to implement `Sourcer`

I'm trying to implement a Sourcer (to be provided to Parse()) for loading configuration from a YAML file, but apparently it's not possible due the fact that field is unexported.

I think that field type should be exported to make Sourcer useful.

PS: thanks for your amazing work :-D

Parsing a bool flag followed by an arg returns an error

Parse unexpectedly returns an error when parsing a bool flag followed by an arg. This behavior differs from the standard library flag package.

I've put together a playground demonstrating the issue.

https://go.dev/play/p/VhuQrHnAKqD

no workflow in circleci while badge is green

There is a circleci config in the project, README is green but looking at circleci there are not workflows running.

Is it intentional?

Would optional Sourcers be a welcome contribution?

Optional ones for JSON, YAML, TOML, etc so coders can choose which Sourcers they want beyond env and flag?

They would be included in a sources_extra.go file.

Mask is not reflected in help output

Is it intentional that masked field values are printed during the help output?
--db-password/$CLOUD-API_DB_PASSWORD (default: postgres)

Using namespace in variable when overriding variable name

Hey Bill, I was just curious.. when overriding a variable name, as in the following example:

type ip struct {
	Name string `conf:"default:localhost,env:IP_NAME_VAR"`
	IP   string `conf:"default:127.0.0.0"`
}

The package defines the variable using the namespace, as shown in the help options:
OPTIONS
--ip-name/$CRUD_IP_NAME_VAR (default: localhost)

Is that the desired behavior? Why doesn't it use the variable as defined when overridden: $IP_NAME_VAR ?
Thanks

os.Arg[1:] & Parse errors

It's not clear from the documentation that Parse should be called with os.Args[1:].
Does seem to matter for the short example

var cfg struct {
	Port int
	Args conf.Args
}

func main() {
	if err := conf.Parse(os.Args, "CRUD", &cfg); err != nil {
		log.Fatalf("main : Parsing Config : %v", err)
	}
}

The bellow example doesn't work with os.Args.
Should conf.Parse(os.Args, .. return an error in this example?

package main

import (
	"fmt"
	"log"
	"os"
	"time"

	"github.com/ardanlabs/conf"
)

type ip struct {
	Name      string   `conf:"default:localhost,env:IP_NAME_VAR"`
	IP        string   `conf:"default:127.0.0.0"`
	Endpoints []string `conf:"default:127.0.0.1:200;127.0.0.1:829"`
}
type Embed struct {
	Name     string        `conf:"default:bill"`
	Duration time.Duration `conf:"default:1s,flag:e-dur,short:d"`
}
type config struct {
	AnInt   int    `conf:"default:9"`
	AString string `conf:"default:B,short:s"`
	Bool    bool
	Skip    string `conf:"-"`
	IP      ip
	Embed
}

func main() {
	cfg := config{}
	if err := conf.Parse(os.Args[1:], "TEST", &cfg); err != nil { // <- can't be os.Args,
		log.Fatalf("main : Parsing Config : %v", err)
	}
	us, err := conf.Usage("Test", &cfg)
	if err != nil {
		fmt.Printf("err %v\n", err)
	}
	fmt.Printf("%+v\n", us)
	fmt.Printf("%+v\n", cfg)
}

noprint tag seems to be ignored?

type Config struct {
...
	SlackBotToken    string   `conf:"default:'',noprint:true,env:SLACK_BOT_TOKEN,flag:slack-bot-token,help:Bot token"`
...
}

func main() {
       ...
	cfg, err := config.Config()
	if err != nil {
		return errors.Wrapf(err, "error reading config\n")
	}
	out, err := conf.String(cfg)
	if err != nil {
		return errors.Wrap(err, "error generating config for output")
	}
	fmt.Println("starting service ===", "version:", cfg.Version.SVN)
	fmt.Printf("configuration\n%v", out)
       ...
}

execution:

starting service === version: develop
configuration
--svn=develop
--slack-bot-token=xoxx-XDF...

Using 1.5.0 witg go 1.17

Question: Using env vars without prefix

Sorry in advance for opening a question as an issue, could not find a relevant community to direct this.

How can we use the env vars without prefix?

When I pass the empty string as prefix:

if err := conf.Parse(os.Args[1:], "", &cfg); 

I get a leading underscore in the env vars generated

  --web-service-port/$_SERVICE_PORT                    <int>     (default: 98765)

This was previously defined in a cfg struct as follows

		Web struct {
			ServicePort int `conf:"default:98765,env:SERVICE_PORT"`
		}

FR: Mask instead of completely hiding noprint variable

Instead of entirely omitting from printouts any variables set as noprint in the cfg struct, perhaps it would be a good idea just to omit / mask its value.

This would help the user/developer realise that there is indeed a configuration variable (set/expected) named (say) AWS_SECRET_ACCESS_KEY but it is sensitive. (it helps precision in terms of having the logs reflect exactly (with the highest level of precision) the actual configuration.

Feature: Centralized control of the mask status

As a developer, I'd like to be able to override ,mask status, so I can see values while debugging.

Problems with required

Hi

There's a problem with the required field.

`

	var provided bool
	for _, sourcer := range sources {
		if sourcer == nil  {
			continue
		}
		var value string
		if value, provided = sourcer.Source(field); !provided {
			continue
		}

		// A value was found so update the struct value with it.
		if err := processField(value, field.field); err != nil {
			return &FieldError{
				fieldName: field.name,
				typeName:  field.field.Type().String(),
				value:     value,
				err:       err,
			}
		}
	}

	// If this key is not provided by any source, check if it was
	// required to be provided.
	if !provided && field.options.required {
		return fmt.Errorf("required field %s is missing value", field.name)
	}`

Since there are multiple sources, the required field could have been in the 1st source the for loop processes.
No problem here.
But when you process the 2nd (..or any other), the "provided flag" is marked as false.

So although the field was processed with success 1 time, the final condition will still return an error.

I fix this in the code bellow (conf.go, line 82)
for _, sourcer := range sources { if sourcer == nil || provided { continue }

FR: Perhaps adding more consistency in variable naming?

Following the Service 2.0 class and using the conf package:

▶ ./sales-api --help
Usage: sales-api [options] [arguments]

OPTIONS
 ​--web-api-host/$SALES_WEB_API_HOST                  <string>    (default: 0.0.0.0:3000)
 ​--web-debug-host/$SALES_WEB_DEBUG_HOST              <string>    (default: 0.0.0.0.:4000)
 ​--web-read-timeout/$SALES_WEB_READ_TIMEOUT          <duration>  (default: 5s)
 ​--web-write-timeout/$SALES_WEB_WRITE_TIMEOUT        <duration>  (default: 5s)
 ​--web-shutdown-timeout/$SALES_WEB_SHUTDOWN_TIMEOUT  <duration>  (noprint,default: 5s)
 ​--help/-h                                           
 ​display this help message
 ​--version/-v  
 ​display version information

Perhaps it would be a good idea to have the cmd arguments named exactly (lowercased, hyphens instead of underscores) as the expected env vars, i.e.

--sales-web-api-host/$SALES_WEB_API_HOST

instead of

 ​--web-api-host/$SALES_WEB_API_HOST   

adding list of ints separated by ","

what do you think would be good approach to do it

we need to add list of ids (ints) and later filter by them

for now i don't see any option but get string and split later and convert to ints

may be there is better option

we need something like --ids 222,333,444