verizon / knobs

A reasonable configuration library for Scala

Knobs

Please view the documentation for more information, and go to Maven Central for the latest releases.

knobs's Issues

URLResource mentioned in docs does not exist

I'm not sure if the docs are previewing something yet to come or talking about something that is no longer with us, but the docs reference this resource:

URLResource - loads any URI supported by the class java.net.URI.

Alas, though it sounds terrific, there is no URLResource that I can see.

String substitution not working

I'm trying to import files using string substitution but it seems it is not working as in the documentation

If I for example do this

web-server {
    import "$(user.home)/test.cfg"
}

I get an exception saying

java.io.FileNotFoundException: /home/software/seqexec_server_gs_test-0.2.0/conf/$(user.home)/test.cfg (No such file or directory)

The same happens if I use $(HOME)

Should this work?

2.12 release

I'd guess you'd need a 2.12 version of https://github.com/Verizon/sbt-rig beforehand, I'm hitting

[warn]  Note: Unresolved dependencies path:
[warn]          org.scalatest:scalatest_2.12:2.2.6 ((verizon.build.common) RigPlugin.scala#L79)
[warn]            +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT
[warn]          org.scalacheck:scalacheck_2.12:1.12.5 ((verizon.build.common) RigPlugin.scala#L79)
[warn]            +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT
[warn]          org.scoverage:scalac-scoverage-runtime_2.12:1.1.1 ((scoverage.ScoverageSbtPlugin) ScoverageSbtPlugin.scala#L30)
[warn]            +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT
[warn]          org.scoverage:scalac-scoverage-plugin_2.12:1.1.1 ((scoverage.ScoverageSbtPlugin) ScoverageSbtPlugin.scala#L30)
[warn]            +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT

NullPointerException in watchEvent() method

Knobs is working well for me and working as expected for my use cases, but I've been seeing a NullPointerException output from a different thread. Apparently there's an issue in watchEvent and an unhandled error. As this isn't on the main thread, it doesn't abort any processing and isn't trappable by the calling app (as far as I know).

Here's the exception stack trace:

java.lang.NullPointerException
    at knobs.Resource$.watchEvent(Resource.scala:181)
    at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13$$anonfun$apply$14.apply(Resource.scala:194)
    at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13$$anonfun$apply$14.apply(Resource.scala:194)
    at scalaz.stream.Process$$anonfun$map$1.apply(Process.scala:53)
    at scalaz.stream.Process$$anonfun$map$1.apply(Process.scala:53)
    at scalaz.stream.Process$$anonfun$flatMap$1.apply(Process.scala:46)
    at scalaz.stream.Process$$anonfun$flatMap$1.apply(Process.scala:46)
    at scalaz.stream.Util$.Try(Util.scala:38)
    at scalaz.stream.Process$class.flatMap(Process.scala:46)
    at scalaz.stream.Process$Emit.flatMap(Process.scala:577)
    at scalaz.stream.Process$class.map(Process.scala:53)
    at scalaz.stream.Process$Emit.map(Process.scala:577)
    at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13.apply(Resource.scala:194)
    at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13.apply(Resource.scala:194)
    at scalaz.concurrent.Task$.Try(Task.scala:386)
    at scalaz.concurrent.Task$$anonfun$apply$10.apply(Task.scala:286)
    at scalaz.concurrent.Task$$anonfun$apply$10.apply(Task.scala:286)
    at scalaz.concurrent.Future$$anonfun$apply$15$$anon$4.call(Future.scala:380)
    at scalaz.concurrent.Future$$anonfun$apply$15$$anon$4.call(Future.scala:380)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

And some other details that may be useful:

Mac OS X 10.10.5
Scala 2.11.7
Knobs 3.3.0
Java 1.8

Map value type and other concerns.

Hi,
I gave knobs a spin and this are my concerns:

1. Typesage Config (TSC) allows you to use lists of maps, e.g. [{a = 42, b = "hello"}, {a = 24, b = "world"}]. When I tried to load a Config using the Typesafe extension I got an error since in the CfgValue AST there's no case for a Map-like object. Is there a good reason for that? I mean, apart from the TSC integration I think it would be useful being able to define a list of maps in the config file.
2. I think the extension for TSC should be changed. At the moment the Typesafe object just expose a config method type which, internally, loads a TSC object using the default behaviour, that is ConfigFactory.load. Since you can use other methods to load a TSC wouldn't it make sense exposing a method which accepts a TSC object and returns a knobs Config? Of course keeping also the config method type which uses the default behaviour.
3. Config.require[A:Configured] throws a KeyError exception if the key is not found and this is reasonable. The problem is it throws that exception even if the key is present but the CfgValue could have not been converted into A. I think it should be changed to give it a more reasonable behaviour.
4. There's no instance for Configured[Long]. Is there a good reason for that?

Once I was there playing with the toy I tried to solve the four issues aforementioned and, apparently, I made it. I updated the docs too to include the lists of maps value type.

Let me know if you may be interested in a PR of all or some of these new features.

P.S.: This is a REPL session of the new features, where objlist.cfg is a simple config file as in:

objList = [{a = 42, b = 24}, {a = 1, b = 2}]

long = 10000000000

hi = "hello"

REPL session:

scala> import java.io.File
import java.io.File

scala> import knobs._
import knobs._

scala> import knobs.{Required,FileResource,Config}
import knobs.{Required, FileResource, Config}

scala> import scalaz.concurrent.Task
import scalaz.concurrent.Task

scala> val cfg: Task[Config] = knobs.loadImmutable(Required(FileResource(new File("core/src/test/resources/objlist.cfg"))) :: Nil)
cfg: scalaz.concurrent.Task[knobs.Config] = scalaz.concurrent.Task@26b3524c

scala> val config = cfg.run
config: knobs.Config = Config(Map(objList -> CfgList(List(CfgMap(Map(a -> CfgNumber(42.0), b -> CfgNumber(24.0))), CfgMap(Map(a -> CfgNumber(1.0), b -> CfgNumber(2.0))))), long -> CfgNumber(1.0E10), hi -> CfgText(hello)))

scala> config.require[List[Map[String, Int]]]("objList")
res0: List[Map[String,Int]] = List(Map(a -> 42, b -> 24), Map(a -> 1, b -> 2))

scala> config.require[List[CfgValue]]("objList")
res1: List[knobs.CfgValue] = List(CfgMap(Map(a -> CfgNumber(42.0), b -> CfgNumber(24.0))), CfgMap(Map(a -> CfgNumber(1.0), b -> CfgNumber(2.0))))

scala> config.require[CfgValue]("objList")
res2: knobs.CfgValue = CfgList(List(CfgMap(Map(a -> CfgNumber(42.0), b -> CfgNumber(24.0))), CfgMap(Map(a -> CfgNumber(1.0), b -> CfgNumber(2.0)))))

scala> res2.pretty
res3: String = [{a = 42.0, b = 24.0}, {a = 1.0, b = 2.0}]

scala> config.require[Long]("long")
res4: Long = 10000000000

scala> config.require[Int]("missingKey")
knobs.KeyError: No such key: missingKey
        at knobs.Config$$anonfun$1.apply(Config.scala:40)
        at knobs.Config$$anonfun$1.apply(Config.scala:40)
        ...

scala> config.require[Int]("hi")
knobs.ConversionError: Could not convert "hello" to int
        at knobs.Config$$anonfun$require$1.apply(Config.scala:41)
        at knobs.Config$$anonfun$require$1.apply(Config.scala:41)
        ...

fs2 streams?

Is it possible to replace scalaz streams with fs2 streams, considering that many libraries are now moving toward the cats/fs2 ecosystem?

Make watching file resources optional

Currently a FileResource will always result in a Watched resource. I think this should probably be optional, as sometimes you know that a file is not going to change, or you do not want your config to reload if it does, or you want to always explicitly reload.

Make reloading and change subscription stream-based

The BaseConfig class keeps its internal mutable state as IORefs, and we should change this to scalaz.stream.async.mutable.Signal to make it easier to subscribe to changes. Users should be able to get a Process[Task,Config] from (or even instead of) a MutableConfig.

Update documentation to cover Cats useage

It looks like the latest version of this project now supports Cats. It would be great to have some documentation/examples of how to use this with Cats.

Type Mismatch Error following Usage example for ClassPathResource

Greetings,
I just tried loading a ClassPathResource, following the example in the Usage documentation, but I'm getting a type mismatch error...I'm guessing I'm missing something extremely basic, but for the moment I'm stumped...here's the example (I've tried using knobs-core 3.2.1 and 3.2.2-SNAPSHOT):

import knobs._
import scalaz.concurrent.Task

val config: Task[Config] = knobs.loadImmutable(Required(ClassPathResource("knobs-example.cfg")))

<console>:20: error: type mismatch;
 found   : knobs.Required[knobs.ResourceBox]
 required: List[knobs.KnobsResource]
    (which expands to)  List[knobs.Worth[knobs.ResourceBox]]
       val config: Task[Config] = knobs.loadImmutable(Required(ClassPathResource("knobs-example.cfg")))

Am I missing or misunderstanding something simple here? Any help would be appreciated.
Thanks,
Tim

Documentation: usage.md

Hi,

I found an error in the documentation. I was about to open a PR to update the doc, but the file is already up to date with the Maven Central repository information. Maybe the docs just need to be regenerated.

https://verizon.github.io/knobs/usage is the page where I found the invalid information.

Thanks.

List of nested objects

How would I go about reading something like this:

something = [{ 
    key1 = "yay",
    key2 = "yoi"
  },
 { 
    key1 = "food",
    key2 = "insurmountable"
  },
]

?

Can't run.

I get this at runtime

java.lang.NoSuchMethodError: scalaz.Free$.point(Lscala/Function0;)Lscalaz/Free;
java.lang.NoSuchMethodError: scalaz.Free$.point(Lscala/Function0;)Lscalaz/Free;
    at scalaparsers.Parsing$$anon$3$$anon$2.apply(ParsingUtil.scala:37)
    at scalaparsers.Parser$$anon$7$$anonfun$apply$4.apply(Parser.scala:59)
    at scalaparsers.Parser$$anon$7$$anonfun$apply$4.apply(Parser.scala:58)
    at scalaz.Free$$anonfun$resume$2.apply(Free.scala:107)
    at scalaz.Free$$anonfun$resume$2.apply(Free.scala:107)
    at scalaz.std.FunctionInstances$$anon$1$$anonfun$map$1.apply(Function.scala:77)
    at scalaz.Free$$anonfun$run$1.apply(Free.scala:239)
    at scalaz.Free$$anonfun$run$1.apply(Free.scala:239)
    at scalaz.Free.go2$1(Free.scala:150)
    at scalaz.Free.go(Free.scala:153)
    at scalaz.Free.run(Free.scala:239)
    at scalaparsers.Parser.run(Parser.scala:21)
    at knobs.ConfigParser$.runParser(Parser.scala:203)
    at knobs.ConfigParser$ParserOps.parse(Parser.scala:33)
    at knobs.Resource$$anonfun$loadFile$1$$anonfun$apply$2$$anonfun$apply$3.apply(Resource.scala:190)
    at knobs.Resource$$anonfun$loadFile$1$$anonfun$apply$2$$anonfun$apply$3.apply(Resource.scala:190)
    at scalaz.concurrent.Task$$anonfun$delay$1.apply(Task.scala:272)
    at scalaz.concurrent.Task$$anonfun$delay$1.apply(Task.scala:272)
    at scalaz.concurrent.Task$$anonfun$suspend$1$$anonfun$4.apply(Task.scala:280)
    at scalaz.concurrent.Task$$anonfun$suspend$1$$anonfun$4.apply(Task.scala:280)
    at scalaz.concurrent.Task$.Try(Task.scala:387)
    at scalaz.concurrent.Task$$anonfun$suspend$1.apply(Task.scala:280)
    at scalaz.concurrent.Task$$anonfun$suspend$1.apply(Task.scala:280)
    at scalaz.concurrent.Future.step(Future.scala:110)
    at scalaz.concurrent.Future.listen(Future.scala:75)
    at scalaz.concurrent.Future.runAsync(Future.scala:142)
    at scalaz.concurrent.Future.run(Future.scala:159)
    at scalaz.concurrent.Task.run(Task.scala:97)

I am suspecting a binary compatibility issue. Not sure, though.

Here are some more dependencies:

lazy val Http4sVersion = "0.15.0-SNAPSHOT"
lazy val CirceVersion   = "0.4.1"

libraryDependencies ++= Seq(
  "oncue.knobs"                %% "core"                 % "3.8.1",
  "io.circe"                   %% "circe-core"           % CirceVersion,
  "io.circe"                   %% "circe-generic"        % CirceVersion,
  "io.circe"                   %% "circe-parser"         % CirceVersion,
  "org.http4s"                 %% "http4s-dsl"           % Http4sVersion,
  "org.http4s"                 %% "http4s-circe"         % Http4sVersion,
  "org.http4s"                 %% "http4s-blaze-client"  % Http4sVersion

Upgrade cats-effect to 1.0.0

Now that cats-effect 1.0 is out it'd be nice if this used it. I am receiving errors like this:

Exception in thread "main" java.lang.NoSuchMethodError: cats.effect.Async.shift(Lscala/concurrent/ExecutionContext;)Ljava/lang/Object;
	at knobs.Resource$.watchStream(Resource.scala:155)
	at knobs.Resource$.$anonfun$watchEvent$4(Resource.scala:222)
	at knobs.Resource$.$anonfun$watchEvent$4$adapted(Resource.scala:221)
	at cats.effect.IO$Map.apply(IO.scala:1407)
	at cats.effect.IO$Map.apply(IO.scala:1403)
	at cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:216)
	at cats.effect.internals.IORunLoop$.step(IORunLoop.scala:216)
	at cats.effect.IO.unsafeRunTimed(IO.scala:321)
	at cats.effect.IO.unsafeRunSync(IO.scala:240)

I assume this is because the cats-effect version is a bit older?

Behaviour difference between Typesafe config and knobs-typesafe with values loaded optionally from environment

In Typesafe config, we can have non-string values read from environment variables defaulting to other values if no environment variable is set:

foo {
  bar = 123
  bar = ${?BAR}
}

When there is no BAR environment variable set, Typesafe config will find 123 for the key foo.bar but in Knobs, knobs.Typesafe.config.run.require[Int]("foo.bar") fails with knobs.KeyError: No such key: foo.bar

No access to the thread pool that loads config

It looks like 🔹 the loadp method in knobs/package.scala doesn't give access to the thread pool used to load stuff.

Can we get access to this?

perhaps changing
✝

private [knobs] def loadp(paths: List[(Name, KnobsResource)]): Task[BaseConfig] = for {
    // snip
    _ <- Task(ticks.evalMap(_ => bc.reload).run.runAsync(_.fold(_.printStackTrace, _ => ())))
  } yield bc

to something like this

private [knobs] def loadp(paths: List[(Name, KnobsResource)])(implicit pool: ExecutorService = Strategy.DefaultExecutorService): Task[BaseConfig] = for {
    // snip
    _ <- Task(ticks.evalMap(_ => bc.reload).run.runAsync(_.fold(_.printStackTrace, _ => ())))(pool)
  } yield bc

and chain it up to the public load/loadImmutable methods?

🔹 correct me if I'm wrong

Replace WatchService with fs2 Watcher

Custom watching implementation in knobs can be replaced with WatchService wrapper from fs2.

Can't lookup Duration from Typesafe config

It looks like internally Typesafe config is storing durations in a stringy fashion. So when Typesafe config is converted to knobs config, it becomes a CfgText as opposed to a CfgDuration. And then when you try to look it up as a Duration, you get a None instead of the actual value.

Unfortunately, I don't know of any way that you could solve this within the typesafe module itself, unless you proactively attempt to parse every String value as a duration and fall back to string if that doesn't work which sounds really gross and not necessarily what's desired.

One other option is to have the Configured[Duration] instance attempt to parse the CfgText value if that's what it gets instead of CfgDuration. I think that this could solve the problem without really being harmful. The only downsides that I can think of are that it is a change to core that is really meant to support typesafe and that it is just a little less simple than the current approach.

Example below.

scala> import com.typesafe.config.ConfigFactory, scala.concurrent.duration.Duration
import com.typesafe.config.ConfigFactory
import scala.concurrent.duration.Duration

scala> val ts = ConfigFactory.parseString(""" "dur" = 3 seconds """)
ts: com.typesafe.config.Config = Config(SimpleConfigObject({"dur":"3 seconds"}))

scala> ts.getDuration("dur", java.util.concurrent.TimeUnit.MILLISECONDS)
res0: Long = 3000

scala> val k = knobs.Typesafe.config(ts).run // YOLO
k: knobs.Config = Config(Map(dur -> CfgText(3 seconds)))

scala> k.lookup[Duration]("dur")
res1: Option[scala.concurrent.duration.Duration] = None

scala> k.lookup[String]("dur")
res2: Option[String] = Some(3 seconds)

Defaults for environment variables

Are there any plans to add defaults for environment variables?

Duration vs FiniteDuration

As far as I can tell, there's no way to put an infinite duration as a value in a knobs config file. Therefore it seems to me like CfgDuration should wrap a FiniteDuration instead of a Duration. This would make it so that consumers don't have to tell the compiler "trust me, this is going to be a finite value".

If this isn't changed, I think knobs should at least provide a Configured[FiniteDuration] instance that returns None if the duration is infinite.

Either way I'm happy to submit a PR -- I just wanted to get some feedback on which approach is preferred.

cc @runarorama @timperrett

java.time.LocalDate support?

We've done something like this to support decoding dates form our config files

implicit val configuredLocalDate: Configured[LocalDate] = new Configured[LocalDate] {
  def apply(a: CfgValue) = Some(LocalDate.parse(a.pretty.replace("\"", ""), ofPattern("dd MMM yyyy")))
}

Is that something that could/should go into the CfgValue trait? We'd do a pull request but aren't sure how you'd prefer it to slot in.

Document versioning strategy

Knobs has a couple notable aspects to its versioning strategy:

• It uses traditional semantic versioning, which is different than most major scala libs.
• It uses an a suffix convention. I think this is based on which version of scalaz-stream it depends on, but I'm not sure that I totally understand it.

This versioning strategy should be documented.