`backend.parameters` for `rmr.options`? about rmr2

[piccolbo]

These are two orthogonal issues in my opinion. How do we replace the
dangerous backend.parameters without repeating the hadoop manual inside rmr
and disallowing things like -input and -output? The other is: which options
should be set at the package level and which at the job level? For the
former, I put in a deprecation warning so as to hear what people actually
do with it with the hope of formalizing the meaningful uses, but it looks
like the list is long and ever growing so I am kind of stalled on that
front. Maybe the right thing to do is just to disallow some options as
dangerous or already used by rmr2 itself and allow anything else, until
proven dangerous. For the latter, I have to admit I went for case by case
decisions. With keyval.length I probably made the wrong decision, but
instead of backtracking I hope to go in a different direction. I would love
to have a simple criterion that can tell me whether a parameter should be
package level or job level. Or should we just allow to set everything at
the job level, with the package level as a default?

On Mon, Jun 3, 2013 at 10:12 PM, Jamie F Olson [email protected]:

If backend.parameters is considered deprecated, what is the best-practice
for things like rmr-wide options. For example, a multi-purpose Hadoop
cluster is unlikely to have the configuration optimized for R tasks. In
particular, things like mapred.child.java.opts are likely to be set for
Java jobs, allocating a large amount of memory to the JVM. mapreduce jobs
need to either reduce maximum heap space allocated to the JVM (128 MB is as
low as I could go) or increase mapred.job.map.memory.mb which is likely
to make everyone else angry at you since this is probably configured for
your cluster's specific hardware to allow efficient distribution of tasks.

Is there/should there be another way to set rmr Hadoop parameters (as
opposed to job-specific parameters)?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/49
.

[jamiefolson]

I agree that they're orthogonal and frankly was just curious if the status had changed since the last comment on backend.parameters I could find.

I think the case-by-case argument is pretty reasonable, since it encourages you to actively think about it fairly regularly and it is less likely to lead to potential surprises caused by conflicting D= arguments. It's only things like D="mapred.child.java.opts=-server -Xmx128m -Djava.net.preferIPv4Stack=true" that I expect to use in every case for a particular Hadoop configuration and that's obscure enough that there probably wouldn't be a good argument for incorporating it into rmr, plus it could change and there seem to be similar-ish parameters that vary with Hadoop distribution. On the flip side, it feels like a hadoop configuration parameter, not a job configuration parameter.

[piccolbo]

We now have backend params in rmr.options