Error in lrm() function with factors: subscript out of bounds about rms

[harrelfe]

Produce a minimal self-contained working example and I'll debug.

---

Frank E Harrell Jr Professor and Chairman School of Medicine

Department of Biostatistics Vanderbilt University

On Fri, Jul 22, 2016 at 10:11 AM, Lucas Sala [email protected]
wrote:

Dear Prof. Harrel,

I'm trying to fit a logistic regression model using lrm() function in a
dataset with 47 factors and 1 integer (the target variable) and I'm getting
the following error:

str(dataset)
'data.frame': 147166 obs. of 48 variables:
$ X1 : Factor w/ 7 levels "01",..: 3 5 3 4 1 2 7 2 5 6 ...
$ X2 : Factor w/ 2 levels "01",..: 2 2 2 2 1 1 2 1 2 2 ...
$ X3 : Factor w/ 11 levels "11:",..: 2 2 2 8 2 2 2 2 2 2 ...
$ X4 : Factor w/ 8 levels "08: QTDE_DIAS_DIST_M0 > 27",..: 2 2 2 3 2 2 2 2
2 2 ...
.
.
.
$ TARGET : int 1 1 1 1 1 1 1 1 1 1 ...
require(rms)
fit <- lrm(TARGET ~ ., data = dataset, x=TRUE, y=TRUE, se.fit=TRUE)
Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds

Is there any clues on what's going on?

Thanks in advance,
Lucas

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#29, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABGO2oyz3A5_9WxFR6QL8l4MM_OojkBMks5qYN2tgaJpZM4JS2iw
.

[lucasevsala]

Dear Prof. Harrel,
Here is a self-contained working example:

Load rms package

require(rms)

Generate simulated data

dataset <- data.frame(
X1 = factor(c('05: X1 <= 178','01: X1 <= 6', '03: X1 <= 52', '05: X1 <= 178')),
X2 = factor(c('04: X2 <= 75','01: X2 <= 6', '05: X2 > 75', '05: X2 > 75')),
X3 = factor(c('04: X3 <= 552','01: X3 <= 1', '04: X3 <= 552', '06: X3 > 1313')),
TARGET = c(0, 1, 1, 0)
)

Fit logistic regression model

fit <- lrm(TARGET ~ ., data = dataset, x=TRUE, y=TRUE, se.fit=TRUE)

When executed, I get the same error as before:

Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds

Thanks in advance,
Lucas Sala

[harrelfe]

rms does not support models like ~ . because that would imply that all the continuous variables act linearly, which would be unusual.

[kuiperjh]

Dear Prof. Harrel,

Apologies, I am not entirely sure what the situation around this problem is at the moment. Like Lucas, I love to use rms for regression models, and I also have problems using lrm. It simply does not seem to work with factors, regardless whether they are factors in the dataframe or based on as.factor() or catg() in the regression analysis.

A simple example (based on Lucas'):

dataset <- data.frame(
X1 = factor(c('C','P', 'P', 'P')),TARGET = c(0, 1, 1, 0)
)
fit<-lrm(TARGETX1,data=dataset)
Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds
dataset$X2 <- ifelse(dataset$X1=="P",1,0)
fit<-lrm(TARGETX2,data=dataset) #No problem running
fit<-lrm(TARGET~catg(X2),data=dataset)
Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds

Of course, with only two levels in the factor the problem can be analysed by converting the factor to a numeric, but a non-ordered factor with more than two levels does seem to give a problem...

With kind regards, Jan Herman Kuiper

[harrelfe]

I think I addressed this bug earlier and it's fixed now. A new release will be in CRAN in a couple of weeks.

[kuiperjh]

Thank you for the quick response! Yes, I found a few references online to this problem/bug, but all to do with using as.factor() in the formula. Glad it will be resolved, and thank you very much for developing the package in the first place.

[Nuno9]

Thank you for developing the package. I am having a similar problem and I worked around it by using strat(x1) everytime I would use factor(x1). I use it for 2, 3 or 4 categories and it seems to work. Am I correct in assuming it's working as a factor?
Kind regards,
Nuno

[harrelfe]

No, strat is for something else. I don't know of a workaround until the new version comes out. If you have linux I can send you the tarballs to install now.

[Nuno9]

Thank you clarifying Prof. Harrel. I'm using Windows so I'll just wait for the update.
Many Thanks,
Nuno

[kuiperjh]

Dear Prof. Harrel,

Today I ran into very similar problems while using a factor, this time in your function bj(). The problem seems to lie in the column names of the matrix X. Many of your rms functions use a design matrix X, for which you collect the column name attributes
X <- Design(eval.parent(m))
atrx <- attributes(X)
mmcolnames <- atr$mmcolnames
I have a factor called "Strands", with the labels "Four", Six" and "Eight". This gives mmcolnames of "StrandsSix" and "StrandsEight". Later, X is converted into a design matrix
X <- model.matrix(sformula, X)
This changes the column names of factors, in my case "Strands[T.Six]" and "Strands[T.Eight]". You seem to have anticipated this, because you try to retrieve an alternative form of mmcolnames and then check if mmcolnames matches the column names in the model matrix X
alt <- attr(mmcolnames, 'alt')
if(! all(mmcolnames %in% colnames(X)) && length(alt))
mmcolnames <- alt
However, in none of my problems mmcolnames gets any attribute (let alone "alt"), so the call to attr returns a [NULL]. Thus, although "all(mmcolnames %in% colnames(X))" returns a FALSE, no alternative form is used. The next statement is where Lucas and I ran into trouble:
X <- X[, mmcolnames, drop=FALSE]
Because mmcolnames is not updated this statement returns an error because the actual column names in X do not match those in mmcolnames. This only occurs with factors because numerical columns keep their original name after a call to design.matrix. Simply using
X <- X[, -1, drop=FALSE]
solved the problem, but I guess the problem may lie in providing attributes to mmcolnames?

with kind regards, Jan Herman

[harrelfe]

Please post a minimal working example requiring no external data

[kuiperjh]

Thank you for your quick reply, and my apologies. Here it goes - this is the same example as Lucas' above - together with the output I got from the browser. I used lrm() instead of bj() - the underlying issue seems the same.

dataset <- data.frame(
 X1 = factor(c('05: X1 <= 178','01: X1 <= 6', '03: X1 <= 52', '05: X1 <= 178')),
 X2 = factor(c('04: X2 <= 75','01: X2 <= 6', '05: X2 > 75', '05: X2 > 75')),
 X3 = factor(c('04: X3 <= 552','01: X3 <= 1', '04: X3 <= 552', '06: X3 > 1313')),
 TARGET = c(0, 1, 1, 0)
 )
fit <- lrm(TARGET ~ ., data = dataset, x=TRUE, y=TRUE, se.fit=TRUE)

This gives the above error message:

Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds

When I then browse the contents of the variables, the issue becomes clear:

Enter a frame number, or 0 to exit   

1: lrm(TARGET ~ ., data = dataset, x = TRUE, y = TRUE, se.fit = TRUE)

Selection: 1
Called from: top level 
Browse[1]> mmcolnames
[1] "X103: X1 < 52"   "X105: X1 < 178"  "X204: X2 < 75"   "X205: X2 > 75"   "X304: X3 < 552"  "X306: X3 > 1313"
Browse[1]> attributes(mmcolnames)
NULL
Browse[1]> colnames(X)
[1] "(Intercept)"         "X1[T.03: X1 <= 52]"  "X1[T.05: X1 <= 178]" "X2[T.04: X2 <= 75]"  "X2[T.05: X2 > 75]"   "X3[T.04: X3 <= 552]" "X3[T.06: X3 > 1313]"

Therefore, the call X[, mmcolnames, drop = FALSE] will fail, because the column names do not match. This is no problem with numerical covariates, because their column name does not change after the call to model.matrix() in the functions just above the line that gives the error: X <- model.matrix(Terms.ns, X) in lrm() and X <- model.matrix(sformula, X) in bj().

With kind regards, Jan Herman

[harrelfe]

Jan that was very helpful in my finding the bug. I was not escaping <= in levels as I had escaped >=. The fix is now committed and will be in the next release to CRAN. Thank you.

[kuiperjh]

Dear Frank, My pleasure, I am really glad I could do something helpful - after all I use your programs a lot! Best wishes, Jan Herman

…

On 25 May 2017 at 23:16, Frank Harrell ***@***.***> wrote: Jan that was very helpful in my finding the bug. I was not escaping <= in levels as I had escaped >=. The fix is now committed and will be in the next release to CRAN. Thank you. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#29 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AV2vmUeqt6glQvqVAUf2tUuyZUmegFacks5r9f3TgaJpZM4JS2iw> .

-- Jan Herman Kuiper, PhD Senior Lecturer in Biomechanics, Keele University, UK Postal address: RJAH Orthopaedic Hospital, Oswestry, SY10 7AG, United Kingdom Phone: [phone] Fax:[phone]

[jmmax]

Hello, when will the next release to cran be? I'm using version 5.1.1 and getting this error. Could you possibly send me the tarballs? Thanks!

[harrelfe]

I expect updates on CRAN around 2017-09-22. Current tarballs are found here:
http://data.vanderbilt.edu/fh/attach/Hmisc_4.0-4.tar.gz
http://data.vanderbilt.edu/fh/attach/rms_5.1-2.tar.gz

Best to update Hmisc before updating rms

[jmmax]

Hi Frank, Thanks a lot. Although, I am still getting the error with the new version: packageVersion("rms") [1] '5.1.2' d <- data.frame( + X1 = factor(c('05: X1 <= 178','01: X1 <= 6', '03: X1 <= 52', '05: X1 <= 178')), + X2 = factor(c('04: X2 <= 75','01: X2 <= 6', '05: X2 > 75', '05: X2 > 75')), + X3 = factor(c('04: X3 <= 552','01: X3 <= 1', '04: X3 <= 552', '06: X3 > 1313')), + TARGET = c(0, 1, 1, 0) + )

lrm(TARGET ~ ., data = d)

Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds

…

________________________________ From: Frank Harrell <[email protected]> Sent: 11 September 2017 16:13 To: harrelfe/rms Cc: Jessye-MM; Comment Subject: Re: [harrelfe/rms] Error in lrm() function with factors: subscript out of bounds (#29) I expect updates on CRAN around 2017-09-22. Current tarballs are found here: http://data.vanderbilt.edu/fh/attach/Hmisc_4.0-4.tar.gz http://data.vanderbilt.edu/fh/attach/rms_5.1-2.tar.gz Best to update Hmisc before updating rms — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#29 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AQZ3skjmY9_9fdEM-wsA-3hmIXXzfVHCks5shU4RgaJpZM4JS2iw>.

[harrelfe]

rms does not support ~ .. Try repeating this with specifics on the right hand side of the model formula.

[jmmax]

packageVersion("rms") [1] '5.1.2' lrm(TARGET ~ X1 + X2 + X3, data = d) Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds

…

________________________________ From: Frank Harrell <[email protected]> Sent: 11 September 2017 17:07:25 To: harrelfe/rms Cc: Jessye-MM; Comment Subject: Re: [harrelfe/rms] Error in lrm() function with factors: subscript out of bounds (#29) rms does not support ~ .. Try repeating this with specifics on the right hand side of the model formula. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#29 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AQZ3stLCEn83PAIhvD2qsN8vABm7sNQoks5shVq9gaJpZM4JS2iw>.

[harrelfe]

Thanks. I'll debug further.

[harrelfe]

I got farther than you with the new tarballs, even with the ~. notation. Then I ran into a problem just due to your small sample size. I changed the test program to the following and everything worked fine:

d <- expand.grid(
X1 = factor(c('05: X1 <= 178','01: X1 <= 6', '03: X1 <= 52', '05: X1 <= 178')),
X2 = factor(c('04: X2 <= 75','01: X2 <= 6', '05: X2 > 75', '05: X2 > 75')),
X3 = factor(c('04: X3 <= 552','01: X3 <= 1', '04: X3 <= 552', '06: X3 > 1313')),
rep = 1 : 100)
set.seed(1)
d$TARGET <- sample(0 : 1, nrow(d), replace=TRUE)

lrm(TARGET ~ ., data = d)

[mbbrigitte]

I installed the new version that you posted as well (Himsc_4.0-4 and rms_5.1-2, running on R 3.4.0) and still get the same error with the code that you just posted above:
Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds

[alialvi92]

Hello, I am still getting the error "Error in X[, mmcolnames, drop = FALSE] : subscript out of bounds" in rms package. I was NOT getting this error previously and have only recently started getting it.
My model has factor variables with 2 and more than 2 values. If i remove the variable with > 2 values, the model works. I have provided a reproducible example below.

> Data <- data.frame(
  X = sample(1:700),
  Y = sample(c("yes", "no"),700, replace = TRUE),
  Z = sample (c("Back pain", "Leg Pain", "Back pain = Leg pain"),700, replace = TRUE)
)
> fit<- lrm(Y~X+Z, data= Data)

Some more information:

> packageVersion("rms")
[1] ‘5.1.2’

> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS  10.13.3

[harrelfe]

I had a bug when there was an = sign inside a value label. Fix committed and will be in next release to CRAN.

[babasaraki]

Hi Professor Harrelfe,
I run rms package in R to plot nomogram in order to predict birth weight from TC and HB but unfortunately it keeps saying Unable to fit model using “lrm.fit” and I am sure what went wrong. Here is the full steps of the code that I following based on the rms documentation.

`n <- 1203
set.seed(17)
d <- data.frame(age = rnorm(n, 28, 7),
hand.breath = rnorm(n, 3.44, 0.34),
thigh.circumference = rnorm(n, 15, 2),
sex = factor(sample(c('female','male'), n,TRUE)))

d <- upData(d,
L = .4*(sex=='male') + .045*(age-28) +
(log(hand.breath - 2)-5.2)(-2(sex=='female') + 2*(sex=='male')),
y = ifelse(runif(n) < plogis(L), 1, 0))

ddist <- datadist(d); options(datadist='ddist')

f <- lrm(y ~ lsp(age,28) + sex * rcs(hand.breath, 4) + thigh.circumference,
data=d)
nom <- nomogram(f, fun=function(x)1/(1+exp(-x)), # or fun=plogis
fun.at=c(.001,.01,.05,seq(.1,.9,by=.1),.95,.99,.999),
funlabel="Birth Weight")

plot(nom, xfrac=.45)
print(nom)
nom <- nomogram(f, age=seq(10,90,by=10))
plot(nom, xfrac=.45)
g <- lrm(y ~ sex + rcs(age, 3) * rcs(hand.breath, 3), data=d)
nom <- nomogram(g, interact=list(age=c(5,10,15,20)),
conf.int=c(.7,.9,.95))
plot(nom, col.conf=c(1,.5,.2), naxes=7)`

I am stock at this code f <- lrm(y ~ lsp(age,28) + sex * rcs(hand.breath, 4) + thigh.circumference, data=d)
Please kindly help me out to complete and get the nomogram plotted.

Thank you so much.