The approach I think would suit the mod best, when examined next to other RO mods, is probably a mostly procedural one. The much more complex technology (compared to fuel tanks or solar panels) makes it harder, but I think it's still somewhat possible.
We'd create various parts, named and modeled after the energy source (fission, fusion, RTG, you name it), and then we'd have another selection of options in each part:
- the basic technology, like "liquid core", or "inertial fusion";
- the power output, limited by a min and max value that are set through tech tree progression and technology itself (a gas core reactor can't possibly have a 10W power output, and a 25MW RTG isn't feasible either) and customizable via a slider;
- the shielding amount, obviously impacting mass and the amount of radiation emitted by the reactor. Custom code for radiation is absolutely needed, given the huge variation in power output that a single reactor could have, and the various types of radiation generated (main difference - electromagnetic or neutron) - again, customizable via a slider;
- eventual upgrades to the specific tech, similar to the plethora of upgrades found in KSPIE; it would be cool to give them a little more depth than "x% better part", but it will depend the tech and the physics, so we'll see;
- probably the hardest part, which is an optional nozzle. This would be very similar to what KSPIE does, although with better physics and less exotic propellants and fuels. It would be limited to fission reactors, for the moment, and its performance would heavily depend on
- fuel molecular weight
- reactor temperature
- eventual upgrades, like tungsten doping of the fuel or a more efficient part that raises core temperature
nothing is obviously set in stone and things like the part selection and availability will most likely be widely different in the first working version, but I think the mod should strive towards maximum customizability and realism, where possible.
A couple more observations on more "distant" problems:
- Sourcing the more 'out there' stuff will be hard, so I'd advise to focus towards completing fission tech before moving on to fusion, except maybe for some easier concepts.
- The "lego" approach might not work for all parts, and might be feasible only for fission reactors. Fusion engines might require a more traditional approach.
- Similar the upgrade idea, but different and possibly easier, is a more traditional "mode selection", on the line of "bimodial operation", "trimodial operation", a sort of "reactor specialization selection" that might work as a B plan if the ideas mentioned above turn out to be unfeasible or straight out bad. They're just wild guesses, at the moment.
- We might want to approach electrical propulsion too, but that is probably very well covered by the awesome Near Future Propulsion by Nertea. Some more love for its RO configs would be nice, though.
Some more practical concerns for the mod
What should really be a primary goal is maintaining a realistic (speculative, at least) and 'least invasive' approach. Using and improving stock gameplay features, like stock resources (mainly EC and fuels) and the heat mechanic, is desirable in every scenario.
A big part of the gameplay in the mod will be getting rid of the huge heat loads produced by engines from the liquid core and onward (the gas core especially has some very interesting possibilities, explored in some mid-late 1900 papers, which will show the trade-offs between radiators and Isp in a slightly less "out there" way). Procedural and realistic radiators might therefore be in the scope of the mod, along with a better handling of background heat production if kerbalism doesn't tackle that behemoth.
Optional (for the user) kerbalism integration through background simulation and radiation is absolutely a must for the mod. It's very popular - and awesome -, so ignoring it would give yet another hard choice to users (future stuff or realistic gameplay?).
Sources
As with any "Realism Overhaul" project, staying as close to real life as possible is a must. Sourcing all data, with actual papers and engineering studies when possible, will have to be a vital part of the mod. Posting here sources and references might therefore prove helpful.
The biggest, although not always very in-depth, resource in this field is the Atomic Rockets Website, host of a wide variety of speculative technologies and concepts. More papers will follow, but the easiest way to approach information sourcing is with help from the community, so everyone is encouraged to provide data and sources about any given subject.