Comments (7)
Typesafe config looks like a very nice written library.
For broker config, I'd be ok to use it to populate the ServiceConfiguration. I'd like to maintain to have a single point to have to update when adding a new config variable. Currently, ServiceConfiguration
is a pojo, and you just need to add a new variable with default value and example in conf/broker.conf
.
Alternatively, what about having ServiceConfiguration
with only getters and adding a ServiceConfigurationBuilder
?
from pulsar.
Pushed the branch to remote but no pr, just so you can have a look. The most interesting points:
ServiceConfiguration
wraps and instance of Typesafe Config: https://github.com/radekg/pulsar/blob/7f58a69a2b51c1b3dfca7e03438e9cd1ec5a3afe/pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfiguration.java- pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfigurationLoader.java is gone, not required
- config takes the following form: https://github.com/radekg/pulsar/blob/7f58a69a2b51c1b3dfca7e03438e9cd1ec5a3afe/pulsar-broker-common/src/test/resources/broker.conf
- config init: https://github.com/yahoo/pulsar/compare/master...radekg:immutable-config?expand=1#diff-b9da9463a373b0b19d516884fb70c189R33
- an example of setting up configuration for a test: https://github.com/yahoo/pulsar/compare/master...radekg:immutable-config?expand=1#diff-0a160247a547f35234439402e845f463R83 (it can be set up from the resource, this is what the final PR would be)
- overriding config from a base test class: https://github.com/yahoo/pulsar/compare/master...radekg:immutable-config?expand=1#diff-00841fa6f00fe82dab598394fc501144R107
This isn't a finished PR, tests need much more work. Also, the validation of minValue
and maxValue
isn't implemented but would be rather straightforward to implement.
To comply with what Typesafe Config needs, we would have to add reference.conf
file to the resources
.
Have a look, let me know what you think.
from pulsar.
For config file, would it be possible to maintain the same java properties format? That would keep it consistent with the other config files (client, bookkeeper, ..). It seems the only change is right now the pulsar {....}
enclosing namespace.
I think that having to specify the config var name (either a string or a constant), makes a it bit more complicated to configure programmatically, since you have to lookup for the list of allowed variables, while with setters it's easy to discover all the options.
from pulsar.
But other than the 2 notes above, it seems nice to combine multiple config sources
from pulsar.
@merlimat I'd suggest using the hocon format across all configuration. The reason is that if someone builds software where Pulsar is only one of the components using Typesafe Config, they can single hocon file for all their components. We can keep the properties look and feel but, at least, I'd suggest prefixing all settings with pulsar.
to give them a namespace.
As for having to know the property names, yes, this is slightly inconvenient but, I think, you are looking from the perspective of unit tests. Preferably, all configuration variants would come as a separate .conf
file. Combined with a reference.conf
, everything is covered. However, if that is not enough, we can always implement a builder which would construct the necessary properties. The base is what it is, how config properties are constructed, we can change that to a friendlier looking version.
Shall I come up with a full PR?
from pulsar.
I would rather not break the config file compatibility unless is really necessary. Also, the other concern is that bookkeeper.conf
and zookeeper.conf
will still be in Java properties format and that cannot be easily changed.
from pulsar.
This seems to have gone quiet. Closing. Please reopen if you want this work to continue.
from pulsar.
Related Issues (20)
- Duplicate message created using partitioned topic with flink
- [Bug] [cli] Pulsar-client cli doesn't support timeout values given in apache pulsar reference website
- [Bug] [cli] Pulsar Tokens Create is mishandling time units (specifically, treating seconds as milliseconds) HOT 1
- Excessive memory allocation in OTel broker metrics HOT 3
- [Bug] pulsar-admin 2.x links don't properly link to 3.2 HOT 6
- [Bug] Offload to S3 triggered manually returns success, while not uploading HOT 2
- [Bug] Broker became irresponsive due to deadlock during race-condition in metadatastore callback
- [Bug] Broker is failing to load stats-internal with broken schema ledger
- [Doc] Incorrect description of UniformLoadShedder in pulsar site. HOT 1
- [Doc] PIP-356: Improve "Support Geo-Replication starts at earliest position" doc
- [Bug][client] Consumer implementation might change message processing order when ack timeout is set
- High GC pause cases high publish latency HOT 3
- Flaky-test: ExtensibleLoadManagerImplTest.testGetMetrics (fails consistently)
- [Bug] bin/pulsar-zookeeper-ruok.sh fails with apachepulsar/pulsar:3.3.0 image
- [Bug] maven build fails with Java 22 HOT 1
- [Bug] nslookup in apachepulsar/pulsar:3.3.0 isn't compatible with kubernetes search domains
- Pulsar Standalone: --wipe-data does not work with RocksDB backend in 3.2.3
- [Doc] Search doesn't work on pulsar website HOT 2
- [Bug] [docs] Pulsar 3.3 javadoc is in Chinese HOT 1
- [Bug] Broker became irresponsive due to too many open files error HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulsar.