Coder Social home page Coder Social logo

cregg's Introduction

title author output
Simple Conjoint Analyses and Visualization
Thomas J. Leeper
md_document
variant
markdown_github

cregg is a package for analyzing and visualizing the results of conjoint ("cj") factorial experiments using methods described by Hainmueller, Hopkins, and Yamamoto (2014). It provides functionality that is useful for analyzing and otherwise examining conjoint experimental data through a main function - cj() - that simply wraps around a number of analytic tools:

  • Estimation of average marginal component effects (AMCEs) for fully randomized conjoint designs (as well as designs involving an unlimited number of two-way constraints between features) and munging of AMCE estimates into tidy data frames, via amce()
  • Calculation of marginal means (MMs) for conjoint designs and munging them into tidy data frames via mm()
  • Tabulation of display frequencies of feature levels via cj_table() and cj_freqs() and cross-tabulation of feature restrictions using cj_props()
  • Diagnostics to assess preference heterogeneity, including an omnibus statstical test (cj_anova()) and tidying of differences in MMs (mm_diffs()) and AMCEs (amce_diffs()) across subgroups

In addition, the package provides a number of tools that are likely useful to conjoint analysts:

  • ggplot2-based visualizations of AMCEs and MMs, via plot() methods for all of the above
  • Tidying of raw "wide"-format conjoint survey datasets into "long" or "tidy" datasets using cj_tidy()
  • Diagnostics to choose feature reference categories, via amce_by_reference()

To demonstrate package functionality, the package includes three example datasets:

  • taxes, a full randomized choice task conjoint experiment conducted by Ballard-Rosa et al. (2016)
  • immigration, a partial factorial conjoint experiment with several restrictions between features conducted by Hainmueller, Hopkins, and Yamamoto (2014)
  • conjoint_wide, a simulated "wide"-format conjoint dataset that is used to demonstrate functionality of cj_tidy()

The design of cregg follows a few key princples:

  • Following tidy data principles throughout, so that all of the main functions produce consistently structured, metadata-rich data frames. Thus the response from any function is a tidy data frame that can easily be stacked with others (e.g., for computing AMCEs for subsets of respondents) and then producing ggplot2 visualizations.
  • A formula-based interface that meshes well with the underlying survey-based effect estimation API.
  • A consistent interface for both unconstrained and two-way constrained designs that relies only on formula notation without any package-specific "design" specification. Conjoint designs involving two-way constraints between features are easily supported using simple formula notation: Y ~ A + B + C implies an unconstrained design, while Y ~ A * B + C implies a constraint between levels of features A and B. cregg figures out the constraints automatically without needing to further specify them explicitly.

cregg also provides some sugar:

  • Using "label" attributes on variables to provide pretty printing, with options to relabel features or plots on the fly. The cj_df() function (and data frame class "cj_df") is designed to preserve these attributes during subsetting.
  • Using factor base levels rather than trying to set baseline levels atomically
  • A convenient API (via the cj(..., by = ~ group) idiom) for repeated, subgroup operations without the need for lapply() or for loops
  • All functions have arguments in data-formula order, making it simple to pipe into them via the magrittr pipe (%>%).

A detailed website showcasing package functionality is available at: https://thomasleeper.com/cregg/. Contributions and feedback are welcome on GitHub.

The package, whose primary point of contact is cj(), takes its name from the surname of a famous White House Press Secretary.

Basic Code Examples

The package includes several example conjoint datasets, which is used here and and in examples:

library("cregg")
data("immigration")
data("taxes")

The package provides straightforward calculation and visualization of descriptive marginal means (MMs). These represent the mean outcome across all appearances of a particular conjoint feature level, averaging across all other features. In forced choice conjoint designs, MMs by definition average 0.5 with values above 0.5 indicating features that increase profile favorability and values below 0.5 indicating features that decrease profile favorability. For continuous outcomes, MMs can take any value in the full range of the outcome. Calculation of MMs entail no modelling assumptions are simply descriptive quantities of interest:

# descriptive plotting
f1 <- ChosenImmigrant ~ Gender + Education + LanguageSkills + CountryOfOrigin + Job + JobExperience + JobPlans + ReasonForApplication + 
    PriorEntry
plot(mm(immigration, f1, id = ~CaseID), vline = 0.5)

plot of chunk mmplot

cregg functions uses attr(data$feature, "label") to provide pretty printing of feature labels, so that variable names can be arbitrary. These can be overwritten using the feature_labels argument to override these settings. Feature levels are always deduced from the levels() of righthand-side variables in the model specification. All variables should be factors with levels in desired display order. Similarly, the plotted order of features is given by the order of terms in the RHS formula unless overridden by the order of variable names given in feature_order.

A more common analytic approach for conjoints is to estimate average marginal component effects (AMCEs) using some form of regression analysis. cregg uses glm() and svyglm() to perform estimation and margins to generate average marginal effect estimates. Designs can be specified with any interactions between conjoint features but only AMCEs are returned. (No functionality is provided at the moment for explict estimation of feature interaction effects.) Just like for mm(), the output of cj() (or its alias, amce()) is a tidy data frame:

# estimation
amces <- cj(taxes, chose_plan ~ taxrate1 + taxrate2 + taxrate3 + taxrate4 + taxrate5 + taxrate6 + taxrev, id = ~ID)
head(amces[c("feature", "level", "estimate", "std.error")], 20L)
                          feature         level      estimate   std.error
1           Tax rate for <$10,000      <10k: 0%  0.0000000000          NA
2           Tax rate for <$10,000      <10k: 5% -0.0139987267 0.008367718
3           Tax rate for <$10,000     <10k: 15% -0.0897702241 0.009883554
4           Tax rate for <$10,000     <10k: 25% -0.2215066470 0.012497932
5    Tax rate for $10,000-$35,000    10-35k: 5%  0.0000000000          NA
6    Tax rate for $10,000-$35,000   10-35k: 15% -0.0161677383 0.010015769
7    Tax rate for $10,000-$35,000   10-35k: 25% -0.0849079259 0.015824370
8    Tax rate for $10,000-$35,000   10-35k: 35% -0.1868125806 0.021074682
9    Tax rate for $25,000-$85,000    35-85k: 5%  0.0000000000          NA
10   Tax rate for $25,000-$85,000   35-85k: 15%  0.0005356495 0.008242105
11   Tax rate for $25,000-$85,000   35-85k: 25% -0.0533364485 0.009713809
12   Tax rate for $25,000-$85,000   35-85k: 35% -0.1083416179 0.011917151
13  Tax rate for $85,000-$175,000   85-175k: 5%  0.0000000000          NA
14  Tax rate for $85,000-$175,000  85-175k: 15%  0.0194226595 0.007719126
15  Tax rate for $85,000-$175,000  85-175k: 25%  0.0108897506 0.008078966
16  Tax rate for $85,000-$175,000  85-175k: 35% -0.0015463277 0.008431674
17 Tax rate for $175,000-$375,000  175-375k: 5%  0.0000000000          NA
18 Tax rate for $175,000-$375,000 175-375k: 15%  0.0384042184 0.008581007
19 Tax rate for $175,000-$375,000 175-375k: 25%  0.0504838117 0.008867028
20 Tax rate for $175,000-$375,000 175-375k: 35%  0.0716090284 0.009162901

This makes it very easy to modify, combine, print, etc. the resulting output. It also makes it easy to visualize using ggplot2. A convenience visualization function is provided:

# plotting of AMCEs
plot(amces)

plot of chunk plot_amce

To provide simple subgroup analyses, the cj() function provides a by argument to iterate over subsets of data and calculate AMCEs or MMs on each subgroup. For example, we may want to ensure that there are no substantial variations in preferences within-respondents across multiple conjoint decision tasks:

immigration$contest_no <- factor(immigration$contest_no)
mm_by <- cj(immigration, ChosenImmigrant ~ Gender + Education + LanguageSkills, id = ~CaseID, estimate = "mm", by = ~contest_no)
plot(mm_by, group = "contest_no", vline = 0.5)

plot of chunk mm_by

A more formal test of these differences is provided by a nested model comparison test:

cj_anova(immigration, ChosenImmigrant ~ Gender + Education + LanguageSkills, by = ~contest_no)
Analysis of Deviance Table

Model 1: ChosenImmigrant ~ Gender + Education + LanguageSkills
Model 2: ChosenImmigrant ~ Gender + Education + LanguageSkills + contest_no + 
    Gender:contest_no + Education:contest_no + LanguageSkills:contest_no
  Resid. Df Resid. Dev Df Deviance      F Pr(>F)
1     13949     3353.0                          
2     13905     3343.9 44    9.088 0.8589 0.7334

which provides a test of whether any of the interactions between the by variable and feature levels differ from zero.

Again, a detailed website showcasing package functionality is available at: https://thomasleeper.com/cregg/ and the content thereof is installed as a vignette. The package documentation provides further examples.

Installation

CRAN Downloads Travis Build Status codecov.io

This package can be installed directly from CRAN. To install the latest development version you can pull from GitHub:

if (!require("remotes")) {
    install.packages("remotes")
}
remotes::install_github("leeper/cregg")

cregg's People

Contributors

emitrokhina avatar leeper avatar mbarnfield avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

cregg's Issues

Subgroup analyses

Rather than having to loop over respondent characteristics or profile constraints, it would be nice to have a function (or feature of existing functions) to do the analysis on a set of specified subgroups.

Error in model.frame.default(weights, data = list(OUTCOME = c(1, 1, 1, : object is not a matrix

Cannot get past this error code and not sure what the issue is. Tried lots of things to match some of the formatting of the 'immigration' data set, but cant get anything to work. I stacked 4 data.frames exported from read.qualtrics where I added a group variable based off race/ethnic group.

Happy to send the data directly, but don't to post here.

Put your code here:

## load package
library("cregg")

model <- selected ~ Convenience + Crime + Housing.Cost + Neighborhood.Composition +
  Neighborhood.Quality + School.Quality + Type.of.Place

out <- cj(df, model, id = ~ Response.ID, estimate = "mm", by ~ group)


## session info for your system
sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: OS X El Capitan 10.11.6

Add examples for partitioning constrained designs

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

When designs are constrained, AMCEs for the constrained features are only defined over subsets of the other features against which they are constrained. It would be nice to have an example (e.g., using the immigration dataset) that shows how to clearly partition based on the constraints.

amce_diffs() needs to account for clustering

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Why is it possible to get MMs but not AMCEs (same data set), "Error in svydesign.default(ids = id, weights = ~1, data = data) : Design has only one primary sampling unit"

I wonder if this is a possible bug, or just an error message that is not 100% user friendly.

Basically the MM is all fine, and the AMCE is not produced at all.

Do you need the reproducible example?

Put your code here:

## load package
library("cregg")

> all_mm <- mm(
+   datacj,
+   choice0 ~ location + reimburse + purchase + price + provider,
+   id = "userid",
+   weights = NULL,
+   feature_order = NULL,
+   feature_labels = my_labels,
+   level_order = c("descending"),
+ )
Warning messages:
1: In logLik.svyglm(x) : svyglm not fitted by maximum likelihood.
2: In logLik.svyglm(x) : svyglm not fitted by maximum likelihood.
3: In logLik.svyglm(x) : svyglm not fitted by maximum likelihood.
4: In logLik.svyglm(x) : svyglm not fitted by maximum likelihood.
5: In logLik.svyglm(x) : svyglm not fitted by maximum likelihood.


> amce(
+   datacj,
+   choice0 ~ location + reimburse + purchase + price + provider,
+   id = "userid",
+   weights = NULL,
+   feature_order = NULL,
+   feature_labels = my_labels,
+   level_order = c("descending")
+ )
Error in svydesign.default(ids = id, weights = ~1, data = data) : 
  Design has only one primary sampling unit
> traceback
function (x = NULL, max.lines = getOption("traceback.max.lines", 
    getOption("deparse.max.lines", -1L))) 
{
    n <- length(x <- .traceback(x, max.lines = max.lines))
    if (n == 0L) 
        cat(gettext("No traceback available"), "\n")
    else {
        for (i in 1L:n) {
            xi <- x[[i]]
            label <- paste0(n - i + 1L, ": ")
            m <- length(xi)
            srcloc <- if (!is.null(srcref <- attr(xi, "srcref"))) {
                srcfile <- attr(srcref, "srcfile")
                paste0(" at ", basename(srcfile$filename), "#", 
                  srcref[1L])
            }
            if (isTRUE(attr(xi, "truncated"))) {
                xi <- c(xi, " ...")
                m <- length(xi)
            }
            if (!is.null(srcloc)) {
                xi[m] <- paste0(xi[m], srcloc)
            }
            if (m > 1) 
                label <- c(label, rep(substr("          ", 1L, 
                  nchar(label, type = "w")), m - 1L))
            cat(paste0(label, xi), sep = "\n")
        }
    }
    invisible(x)
}
<bytecode: 0x12a605e78>
<environment: namespace:base>


## session info for your system

> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Big Sur 11.6

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.5.1       Hmisc_4.7-1         Formula_1.2-4       lattice_0.20-44     labelled_2.10.0     tikzDevice_0.12.3.1 gridExtra_2.3      
 [8] readstata13_0.10.0  miceadds_3.15-21    mice_3.15.0         haven_2.4.3         dplyr_1.0.7         cregg_0.4.0         cjoint_2.1.0       
[15] survey_4.1-1        survival_3.2-11     Matrix_1.5-1        ggplot2_3.3.5       lmtest_0.9-38       zoo_1.8-9           sandwich_3.0-1     
[22] tidyr_1.1.4         pacman_0.5.1       

loaded via a namespace (and not attached):
 [1] splines_4.1.1       shiny_1.7.1         assertthat_0.2.1    latticeExtra_0.6-30 ggstance_0.3.5      pillar_1.6.3        backports_1.2.1    
 [8] glue_1.4.2          digest_0.6.28       RColorBrewer_1.1-2  promises_1.2.0.1    checkmate_2.1.0     colorspace_2.0-2    cowplot_1.1.1      
[15] htmltools_0.5.2     httpuv_1.6.3        plyr_1.8.6          pkgconfig_2.0.3     broom_0.7.9         purrr_0.3.4         xtable_1.8-4       
[22] scales_1.1.1        jpeg_0.1-9          later_1.3.0         tibble_3.1.5        htmlTable_2.4.1     generics_0.1.0      farver_2.1.0       
[29] ellipsis_0.3.2      withr_2.4.2         nnet_7.3-16         cli_3.0.1           magrittr_2.0.1      crayon_1.4.1        mime_0.12          
[36] deldir_1.0-6        fansi_0.5.0         foreign_0.8-81      data.table_1.14.2   tools_4.1.1         hms_1.1.1           mitools_2.4        
[43] lifecycle_1.0.1     stringr_1.4.0       interp_1.1-3        munsell_0.5.0       cluster_2.1.2       compiler_4.1.1      rlang_0.4.11       
[50] rstudioapi_0.13     htmlwidgets_1.5.4   filehash_2.4-3      base64enc_0.1-3     labeling_0.4.2      gtable_0.3.0        DBI_1.1.1          
[57] R6_2.5.1            knitr_1.36          fastmap_1.1.0       utf8_1.2.2          stringi_1.7.5       Rcpp_1.0.7          vctrs_0.3.8        
[64] rpart_4.1-15        png_0.1-7           xfun_0.26           tidyselect_1.1.1  

cj_tidy does not work with tibbles

I was using the package, and I had an issue with cj_tidy. It took me some time but I found that the problem was with the reshaping function in line 27 of the code. My suggestion is to have a test to see if the dataset is a data.frame or not. Once I find the problem, it worked great!

AMEs and AMIEs

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

It could be cool to implement tidy AME (of features not just AMCEs of levels) and AMIE (interactions again between features) estimation via FindIt (see https://amstat.tandfonline.com/doi/full/10.1080/01621459.2018.1476246#.XXJKGi-ZMWo and https://CRAN.R-project.org/package=FindIt). I am going to fork the repo and attempt to write these functions so they're consistent with the rest of cregg, but I understand this might be beyond the intended scope of the package and therefore not merged into the master. Also understand that these aren't yet widely used or recognised estimands. Might take me a while.

AMCEs in constrained designs

In fully randomized, unconstrained designs, the AMCE = AME of each factor level. In constrained designs, the AMCE is the weighted average of AMEs for each possible combination (in their examples, weights are intended display probabilities).

Really, this statistic doesn't make any sense but it is what Hainmueller et al. propose. It's probably better to just present multiple AMEs for each feature across the levels of other features against which it is constrained. But, that's also difficult.

Add argument to control level order within features

Add an argument level_order (or similar) that regulates whether feature levels are presented in final output in ascending or descending order within each feature. That will be reflected graphically later on with the base level either at the top or bottom of each feature.

1 - 2*alpha confidence intervals returned by mm_diffs()

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

It seems that there is an issue with calculation of confidence intervals in mm_diffs. Instead of returning a 1 - alpha confidence interval, a 1 - 2*alpha confidence interval is currently returned.
In lines 71-72 of mm_diffs(), the confidence intervals are calculated so:

        # CIs
        mm_split[[i]][["lower"]] <- mm_split[[i]][["estimate"]] - (stats::qnorm(1-alpha) * mm_split[[i]][["std.error"]])
        mm_split[[i]][["upper"]] <- mm_split[[i]][["estimate"]] + (stats::qnorm(1-alpha) * mm_split[[i]][["std.error"]])

But the function documentation specifies that alpha is a "A numeric value indicating the significance level at which to calculate confidence intervals for the MMs (by default 0.95, meaning 95-percent CIs are returned)." The default of alpha = 0.05 gives a 90-percent confidence interval as of right now. I'm sure the solution is obvious to you; changing alpha to alpha/2 in both cases above would fix the issue. Or adding + (alpha/2) within qnorm(), which seems to be what is done in mm() and amce_diffs()

This is mainly a problem for subsequent plotting, I think.

Thank you for this package, it's awesome!

Put your code here:

## load package
library("cregg")

data("immigration")
alpha <- .05
# Differences in MMs by Gender feature
temp <- mm_diffs(immigration, 
                 ChosenImmigrant ~ LanguageSkills + Education,
                 ~ Gender, id = ~ CaseID,
                 alpha = alpha)

#confidence interval from mm_diffs()
temp[, c("lower", "upper")]

#95% confidence interval calculated "by hand"
data.frame(lower = temp$estimate - qnorm(1 - alpha/2) * temp$std.error,
           upper = temp$estimate + qnorm(1 - alpha/2) * temp$std.error)


## session info for your system
sessionInfo()

Same factor levels in different attributes

Hi Thomas,
I encountered a small problem with cj(): in case you have two levels across any attributes that have the same name, cj() will not work. While there are tweaks around this (of course), it is not very convenient. In my example, we simply have a candidate experiment in which the attributes are priorities and the attribute levels are whether the candidates have or not have these priorities. Hence, the attribute levels are "Is a priority" vs. "Is not a priority" and this is repeated for several attributes.
Could potentially be fixed.
Thanks so much for providing this great package!
Chris

`facet_wrap` produces a NA panel

My issue is about package functionality: when using facet_wrap I get a panel for NA values. However, the gg object (which the function plot() plots does not contain any NAs. I can see you run into the same issue. From your website, and using the same coded provided (see below), a NA panel is produced.

## load package
library("cregg")

## plot with facetting
plot(stacked) + ggplot2::facet_wrap(~ contest_no, nrow = 1L)


## session info for your system
R version 4.3.2 (2023-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Sonoma 14.2.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Europe/Helsinki
tzcode source: internal

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] gtable_0.3.4     ggplot2_3.4.3    cregg_0.4.0      vtable_1.4.6     kableExtra_1.3.4
[6] dplyr_1.1.3      pacman_0.5.1    

loaded via a namespace (and not attached):
 [1] sandwich_3.0-2    utf8_1.2.3        generics_0.1.3    xml2_1.3.5        stringi_1.7.12   
 [6] lattice_0.21-9    digest_0.6.33     magrittr_2.0.3    evaluate_0.22     fastmap_1.1.1    
[11] plyr_1.8.9        Matrix_1.6-1.1    DBI_1.1.3         survival_3.5-7    httr_1.4.7       
[16] rvest_1.0.3       fansi_1.0.5       viridisLite_0.4.2 scales_1.2.1      cli_3.6.1        
[21] crayon_1.5.2      mitools_2.4       rlang_1.1.1       cowplot_1.1.1     munsell_0.5.0    
[26] splines_4.3.2     withr_2.5.1       tools_4.3.2       colorspace_2.1-0  webshot_0.5.5    
[31] vctrs_0.6.3       R6_2.5.1          zoo_1.8-12        lifecycle_1.0.3   stringr_1.5.0    
[36] ggstance_0.3.6    pkgconfig_2.0.3   pillar_1.9.0      Rcpp_1.0.11       glue_1.6.2       
[41] systemfonts_1.0.4 lmtest_0.9-40     xfun_0.40         tibble_3.2.1      tidyselect_1.2.0 
[46] rstudioapi_0.15.0 knitr_1.44        farver_2.1.1      htmltools_0.5.6.1 survey_4.2-1     
[51] labeling_0.4.3    rmarkdown_2.25    svglite_2.1.2     compiler_4.3.2   

Add support for svyciprop() CIs for MMs

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Currently, CIs are from svymean() but might be better from svyciprop() when the outcome is binary. We can make this optional but it doesn't seem to work with svyby() so may require refactoring.

Variable importance metric?

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Is there a way to get an idea of "variable importance" using the cregg package? For example, one might look at the plotted marginal means and ask me, "Well, which variable had the biggest effect?" I was thinking maybe looking at the variance of the marginal means? By way of an example, in the code below, I added a noise variable to immigration. I would want noise to perform very poorly on this variable importance metric.

Another ludicrous example might be testing profiles of political candidates where we say their party and whether or not they like macaroni and cheese. We'd want some metric that says "Political party matters X times more than whether or not they like macaroni and cheese." Is that possible to derive from the output of cj()? I wasn't sure if there is some canonical way of approaching this.

library(cregg)
library(tidyverse)
data("immigration")

set.seed(1839)
immigration$noise <- factor(sample(letters[1:5], nrow(immigration), TRUE))

f <- ChosenImmigrant ~ Gender + Education +
  LanguageSkills + CountryOfOrigin + Job + JobExperience +
  JobPlans + ReasonForApplication + PriorEntry + noise

res <- cj(immigration, f, id = ~ CaseID)

# maybe?
res %>% 
  group_by(feature) %>% 
  summarise(est_var = var(estimate)) %>% 
  arrange(desc(est_var))

Axis in mm-diff plots

This is a question about package functionality.

I successfully reproduced the immigration example of conditional marginal means. However, also in your documentation, I do not like the x-axis dimensions of the figure.
Does the cregg package allow to change the x-axis range per plot? I would like the ranges to have the same distances, but not the ranges. In this case: 0.3 to 0.7 for the High and Low Ethnocentrism ones and -0.2 to 0.2 for the difference plot.
Options like range free don't seem to work.

Any help on this package is highly appreciated.

Put your code here:

## load package
library("cregg")
data("immigration")
## code goes here
# calculate conditional MMs
mms <- cj(na.omit(immigration), ChosenImmigrant ~ ReasonForApplication + LanguageSkills, id = ~CaseID, estimate = "mm", by = ~ethnosplit)
diff_mms <- cj(na.omit(immigration), ChosenImmigrant ~ ReasonForApplication + LanguageSkills, id = ~CaseID, estimate = "mm_diff", 
    by = ~ethnosplit)
plot(rbind(mms, diff_mms)) + ggplot2::facet_wrap(~BY, ncol = 3L)

## session info for your system
sessionInfo()
> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS 13.3

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] stargazer_5.2.3     data.table_1.14.8   gridExtra_2.3       xtable_1.8-4        Hmisc_4.7-1         Formula_1.2-4       survival_3.2-11    
 [8] lattice_0.20-44     cregg_0.4.0         tikzDevice_0.12.3.1 forcats_0.5.1       stringr_1.5.0       dplyr_1.1.1         purrr_1.0.1        
[15] readr_2.1.4         tidyr_1.3.0         tibble_3.2.1        ggplot2_3.3.5       tidyverse_1.3.1     pacman_0.5.1       

loaded via a namespace (and not attached):
 [1] fs_1.6.1            lubridate_1.7.10    RColorBrewer_1.1-2  httr_1.4.2          tools_4.1.1         backports_1.2.1     utf8_1.2.3         
 [8] R6_2.5.1            rpart_4.1-15        DBI_1.1.1           colorspace_2.0-2    nnet_7.3-16         withr_2.5.0         tidyselect_1.2.0   
[15] compiler_4.1.1      cli_3.6.1           rvest_1.0.1         htmlTable_2.4.1     xml2_1.3.3          sandwich_3.0-1      labeling_0.4.2     
[22] scales_1.2.1        checkmate_2.1.0     lmtest_0.9-38       digest_0.6.28       foreign_0.8-81      base64enc_0.1-3     jpeg_0.1-9         
[29] pkgconfig_2.0.3     htmltools_0.5.5     dbplyr_2.1.1        fastmap_1.1.0       htmlwidgets_1.6.2   rlang_1.1.0         readxl_1.4.1       
[36] rstudioapi_0.13     farver_2.1.0        generics_0.1.3      zoo_1.8-9           jsonlite_1.7.2      magrittr_2.0.3      interp_1.1-3       
[43] Matrix_1.5-1        Rcpp_1.0.7          munsell_0.5.0       fansi_1.0.4         lifecycle_1.0.3     stringi_1.7.12      plyr_1.8.6         
[50] ggstance_0.3.5      grid_4.1.1          crayon_1.5.2        deldir_1.0-6        haven_2.5.2         splines_4.1.1       hms_1.1.3          
[57] knitr_1.42          pillar_1.9.0        reprex_2.0.1        glue_1.6.2          mitools_2.4         latticeExtra_0.6-30 modelr_0.1.8       
[64] png_0.1-7           vctrs_0.6.1         tzdb_0.3.0          cellranger_1.1.0    gtable_0.3.0        assertthat_0.2.1    xfun_0.38          
[71] broom_1.0.4         survey_4.1-1        filehash_2.4-3      cluster_2.1.2 

Add example of combining features

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

It would be nice to demonstrate that if you want higher-order marginal means, you need to combine features together. So if you have two features with levels (A,B), (C,D), you need to create a new factor variable with levels (AC, AD, BC, BD). This is easily done with interaction().

Error with weights and marginal means

Thanks for a great package.

I'm hitting an error when using weights. The traceback info is below. The MWE runs without a weights argument. Any suggestions would be welcome.

8: getElement(x, "call")
7: getCall.default(object)
6: getCall(object)
5: update.default(weights, 0 ~ .)
4: update(weights, 0 ~ .)
3: all.vars(update(weights, 0 ~ .))
2: mm(data = data, formula = formula, id = id, weights = weights, 
       feature_order = feature_order, feature_labels = feature_labels, 
       level_order = level_order, ...)
1: cj(data = test, formula = dv ~ IV1 + IV2, weights = "wts", estimate = "mm")
## load package
library("cregg")
  test.formula <- dv ~ IV1 + IV2
  test <- data.frame(dv = sample(c(0, 1), 1000, replace = TRUE), 
                        ID = rep(1:500, 2), 
                        IV1 = sample(letters[1:3], 1000, replace = TRUE), 
                        IV2 = sample(letters[4:6], 1000, replace = TRUE),
                        wts = runif(1000, 0, 2))
  mmtest <- cj(test, test.formula, id = ~ID, estimate = "mm", 
                  weights = "wts")




## session info for your system
R version 3.5.3 (2019-03-11)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] cregg_0.3.0

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2       survey_3.35-1    rstudioapi_0.10  magrittr_1.5     splines_3.5.3    tidyselect_0.2.5
 [7] munsell_0.5.0    colorspace_1.4-1 lattice_0.20-38  R6_2.4.0         rlang_0.4.0      plyr_1.8.4      
[13] dplyr_0.8.0.1    tools_3.5.3      grid_3.5.3       ggstance_0.3.1   gtable_0.3.0     survival_2.43-3 
[19] yaml_2.2.0       lazyeval_0.2.2   assertthat_0.2.1 lmtest_0.9-36    tibble_2.1.1     crayon_1.3.4    
[25] Matrix_1.2-15    purrr_0.3.2      ggplot2_3.1.0    glue_1.3.1       sandwich_2.5-0   compiler_3.5.3  
[31] pillar_1.3.1     scales_1.0.0     pkgconfig_2.0.2  zoo_1.8-5       

Change point shapes in plot of AMCEs or MMs

Please specify whether your issue is about:

  • [x ] a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Hi Thomas: As mentioned, I'm having trouble manually changing the shapes of points when plotting marginal means or AMCEs. Note that using scale_colour_manual() works but scale_shape_manual() is ignored.

library("cregg")

data("immigration")

mm_by <- cj(immigration, ChosenImmigrant ~ Gender + Education + LanguageSkills, id = ~CaseID, estimate = "mm", by = ~contest_no)

plot(mm_by, group = "contest_no", vline = 0.5) + scale_shape_manual(values=c(1, 2, 3, 4, 5)) + scale_colour_manual(values=c("red", "blue", "green", "purple", "black"))

Add variance tests

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

The test suite currently only tests point estimate accuracy but it should also test variances.

Feature header style

This is not a bug report - but rather a clarification question. Any advice as to how to remove parentheses around feature headers in plots? Thanks a lot for your work on this!

Support ACIE

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

We could do this. It's low priority for me but would gladly accept help.

Types of features should be validated.

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

If you are reporting (1) a bug or (2) a question about code, please supply:

Right now package doesn't validate if features are factors. It leads to rather errors like this

## load package
library("cregg")

df <- tibble::tibble(
  choice = c(0, 1), f1 = c("a", "b"), f2 = c("d", "e"), id = c("1", "2")
)

mm(df, choice ~ f1 + f2, id = ~id)

## Error in fix.by(by.y, y) : 'by' must specify a uniquely valid column
## In addition: Warning messages:
## 1: Setting row names on a tibble is deprecated. 
## 2: Setting row names on a tibble is deprecated. 
## Called from: fix.by(by.y, y)

That's because make_term_levels_df blindly assumes that features are factors and ends up with a list of NULLs

lapply(df[c("f1", "f2")], levels)
## $f1
## NULL
## 
## $f2
## NULL

The problem affects at least amce, cj and mm.

A quick fix would be to simply test if that's the case:

if(!all(sapply(data[feature_names], is.factor))) stop("All features should be factors")

Alternatively one could attempt to fix the problem by explicitly casting to factors (maybe with a warning).

Additional diagnostics

Possibilities:

  • Sparsity or combinations of feature levels
  • Carryover assumption test (subset analysis by task round)
  • Balance testing (Hainmueller et al. 2014, 25)
  • Left/right preference test
  • Reference category sensitivity (a72d8c0)

Color change

Please specify whether your issue is about:

  • a question about package functionality

See below for these things. I just copied and pasted the example for MM's from the package README.

I was curious if you could change the colors to gray scale, for example, instead of rainbow colors for journal publication? Couldn't figure out how to do this. Thanks!

Example here:

## load package
library("cregg")

data("immigration")

# taken from README
f1 <- ChosenImmigrant ~ Gender + Education + LanguageSkills + CountryOfOrigin + Job + JobExperience + JobPlans + ReasonForApplication + 
    PriorEntry
plot(mm(immigration, f1, id = ~CaseID), vline = 0.5) # change colors here?

## session info for your system
sessionInfo()

# OUTPUT:
### R version 3.5.2 (2018-12-20)
### Platform: x86_64-apple-darwin15.6.0 (64-bit)
### Running under: macOS Mojave 10.14.3

Formally test for difference in effect sizes between attribute levels

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Dear Thomas,

I have a question about how to formally test whether estimates for two attribute levels within one dimension are significantly different from each other. E.g., in the example below, is the estimated effect for a person from Iraq significantly different from the effect size for a person coming from China?

Thank you!

## load package
library("cregg")

## code goes here
data("immigration")
# estimating AMCEs with constraints
x <- amce(immigration, ChosenImmigrant ~ Gender + ReasonForApplication * CountryOfOrigin,
     id = ~CaseID)
plot(x)

## session info for your system
sessionInfo()

Subgroup analysis on a factor with levels identical to feature levels produces wrong estimates

Moved from @m-jankowski at #13:

This is also a problem when using mm_diffs().

In my case, I wanted to conduct a subgroup analysis conditional on the gender of the respondents (labeled as "Male" or "Female"). The conjoint experiment, however, also contained levels with these labels ("Male" and "Female"). Particularly problematic is that mm_diffs() did not throw an error message, but returned wrong estimates without any warnings.

Here is an artificial example using the immigration data:

# Data
data("immigration")

# Create subgroups
immigration$ethnosplit <- cut(immigration$ethnocentrism, 2)

# Rename subgroup levels
immigration$subgroup <- as.factor(ifelse(as.numeric(immigration$ethnosplit) == 1, 
                                          "Female", 
                                          "Male"))

# Estimate correct MMs by subgroup
mm_correct <- cj(na.omit(immigration),
                 ChosenImmigrant ~ Gender + Education + LanguageSkills,
                 estimate = "mm",
                 id = ~ CaseID, 
                 by = ~ ethnosplit)

plot(mm_correct,
     group = "ethnosplit",
     vline = 0.5)

image

# Differences between subgroups

mmdiff_correct <- mm_diffs(na.omit(immigration), 
                   ChosenImmigrant ~ Gender + Education + LanguageSkills,
                   id = ~ CaseID, 
                   by = ~ ethnosplit)

plot(mmdiff_correct)

image

# Using subgroups with identical level names returns wrong estimates

mmdiff_problem <- mm_diffs(na.omit(immigration),
                   ChosenImmigrant ~ Gender + Education + LanguageSkills,
                   id = ~ CaseID, 
                   by = ~ subgroup)

plot(mmdiff_problem)

image

Weird issue with constraint in 'taxes' data

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Something weird going on here:

> amce(taxes, chose_plan ~ taxrate2 * taxrev)
Waiting for profiling to be done...
Error in coefs_tmp[!aliased, ] <- coef_summary : 
  number of items to replace is not a multiple of replacement length
> traceback()
2: get_coef_summary(mod = mod, data = data, id = id, alpha = alpha) at amce.R#135
1: amce(taxes, chose_plan ~ taxrate2 * taxrev)
> cj(taxes, chose_plan ~ taxrate2 * taxrev, estimate = "amce")
Waiting for profiling to be done...
Error in coefs_tmp[!aliased, ] <- coef_summary : 
  number of items to replace is not a multiple of replacement length
> traceback()
3: get_coef_summary(mod = mod, data = data, id = id, alpha = alpha) at amce.R#135
2: amce(data = data, formula = formula, id = id, weights = weights, 
       feature_order = feature_order, feature_labels = feature_labels, 
       level_order = level_order, ...) at cj.R#89
1: cj(taxes, chose_plan ~ taxrate2 * taxrev, estimate = "amce")

Seems to only occur with this feature combination, but not without constraint:

> amce(taxes, chose_plan ~ taxrate2 + taxrev)
Waiting for profiling to be done...
     outcome statistic                      feature       level    estimate   std.error         z            p        lower       upper
1 chose_plan      amce Tax rate for $10,000-$35,000  10-35k: 5%  0.00000000          NA        NA           NA           NA          NA
2 chose_plan      amce Tax rate for $10,000-$35,000 10-35k: 15%  0.02907463 0.008612538  3.375849 7.358823e-04  0.012194365  0.04595489
3 chose_plan      amce Tax rate for $10,000-$35,000 10-35k: 25%  0.01679525 0.010284125  1.633124 1.024430e-01 -0.003361265  0.03695176
4 chose_plan      amce Tax rate for $10,000-$35,000 10-35k: 35% -0.04091056 0.011919199 -3.432324 5.984314e-04 -0.064271757 -0.01754936
5 chose_plan      amce                  Tax revenue        <75%  0.00000000          NA        NA           NA           NA          NA
6 chose_plan      amce                  Tax revenue      75-95% -0.04588052 0.008989190 -5.103965 3.326091e-07 -0.063499005 -0.02826203
7 chose_plan      amce                  Tax revenue     95-105% -0.05998096 0.011307260 -5.304641 1.128948e-07 -0.082142777 -0.03781913
8 chose_plan      amce                  Tax revenue    105-125% -0.07391775 0.011029219 -6.701994 2.055952e-11 -0.095534623 -0.05230088
9 chose_plan      amce                  Tax revenue       >125% -0.12287030 0.012610874 -9.743202 1.972386e-22 -0.147587158 -0.09815344

Doesn't happen with unconstrained features:

> cj(taxes, chose_plan ~ taxrate1 * taxrev, estimate = "amce")
Waiting for profiling to be done...
     outcome statistic               feature     level     estimate   std.error           z            p        lower        upper
1 chose_plan      amce Tax rate for <$10,000  <10k: 0%  0.000000000          NA          NA           NA           NA           NA
2 chose_plan      amce Tax rate for <$10,000  <10k: 5%  0.006844784 0.008657404   0.7906278 4.291612e-01 -0.007395378  0.021084946
3 chose_plan      amce Tax rate for <$10,000 <10k: 15% -0.050383031 0.008578737  -5.8730128 4.279453e-09 -0.064493798 -0.036272265
4 chose_plan      amce Tax rate for <$10,000 <10k: 25% -0.167008614 0.009084111 -18.3846949 1.742172e-75 -0.181950648 -0.152066581
5 chose_plan      amce           Tax revenue      <75%  0.000000000          NA          NA           NA           NA           NA
6 chose_plan      amce           Tax revenue    75-95% -0.002781705 0.009200863  -0.3023309 7.623998e-01 -0.017915778  0.012352367
7 chose_plan      amce           Tax revenue   95-105% -0.011051298 0.010595316  -1.0430362 2.969315e-01 -0.028479042  0.006376446
8 chose_plan      amce           Tax revenue  105-125% -0.034789897 0.009172065  -3.7930278 1.488215e-04 -0.049876602 -0.019703193
9 chose_plan      amce           Tax revenue     >125% -0.072428072 0.010124390  -7.1538211 8.439503e-13 -0.089081211 -0.055774933

Doesn't occur with immigration:

> cj(immigration, ChosenImmigrant ~ Education * Job, id = ~ CaseID)
           outcome statistic                feature               level     estimate  std.error          z            p        lower      upper
1  ChosenImmigrant      amce Educational Attainment           No Formal  0.000000000         NA         NA           NA           NA         NA
2  ChosenImmigrant      amce Educational Attainment           4th Grade  0.032242795 0.01564257  2.0612208 3.928198e-02  0.006513054 0.05797254
3  ChosenImmigrant      amce Educational Attainment           8th Grade  0.055971117 0.01568247  3.5690243 3.583132e-04  0.030175749 0.08176649
4  ChosenImmigrant      amce Educational Attainment         High School  0.117940436 0.01569985  7.5122007 5.814154e-14  0.092116478 0.14376439
5  ChosenImmigrant      amce Educational Attainment    Two-Year College  0.164346505 0.02408703  6.8230296 8.914028e-12  0.124726870 0.20396614
6  ChosenImmigrant      amce Educational Attainment      College Degree  0.191558995 0.02414755  7.9328533 2.141674e-15  0.151839805 0.23127818
7  ChosenImmigrant      amce Educational Attainment     Graduate Degree  0.178682719 0.01774300 10.0706030 7.451979e-24  0.149498080 0.20786736
8  ChosenImmigrant      amce                    Job             Janitor  0.000000000         NA         NA           NA           NA         NA
9  ChosenImmigrant      amce                    Job              Waiter -0.004555776 0.01742414 -0.2614635 7.937351e-01 -0.033215933 0.02410438
10 ChosenImmigrant      amce                    Job Child Care Provider  0.013684131 0.01745927  0.7837744 4.331725e-01 -0.015033819 0.04240208
11 ChosenImmigrant      amce                    Job            Gardener  0.015863618 0.01760330  0.9011731 3.674963e-01 -0.013091228 0.04481846
12 ChosenImmigrant      amce                    Job   Financial Analyst  0.065638780 0.03093098  2.1221049 3.382893e-02  0.014761848 0.11651571
13 ChosenImmigrant      amce                    Job Construction Worker  0.033689581 0.01758459  1.9158585 5.538311e-02  0.004765508 0.06261365
14 ChosenImmigrant      amce                    Job             Teacher  0.074107007 0.01752927  4.2276146 2.361819e-05  0.045273920 0.10294009
15 ChosenImmigrant      amce                    Job Computer Programmer  0.078108991 0.03011503  2.5936881 9.495257e-03  0.028574178 0.12764380
16 ChosenImmigrant      amce                    Job               Nurse  0.089129556 0.01748420  5.0977211 3.437667e-07  0.060370613 0.11788850
17 ChosenImmigrant      amce                    Job  Research Scientist  0.135076380 0.03014113  4.4814640 7.413274e-06  0.085498637 0.18465412
18 ChosenImmigrant      amce                    Job              Doctor  0.166308291 0.03004119  5.5360091 3.094415e-08  0.116894933 0.21572165

What's going on here?

Please make CREGG Compatibility with MICE

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • [YES] a suggested code or documentation change, improvement to the code, or feature request

I'd like to be able to run cregg on multiple datasets in mice's mira object. OR, I would appreciate help in combining multiple cregg results datasets into a mira object. Ultimately, I'd like to use the pool function on multiple cregg supplied results.

cj(..., by = ) operations produce faulty results when by factor has blank levels

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Reported by Farsan Ghassim via email:

## load package
library("cregg")

## code goes here
iris$foo <- factor(c("a", "b", ""))
cregg::cj(iris, Sepal.Length ~ Species, estimate = "mm", by = ~ foo)
##   BY      outcome statistic feature      level estimate  std.error        z p    lower    upper foo
## 1    Sepal.Length        mm Species versicolor 6.023529 0.13406704 44.92923 0 5.760763 6.286296   a
## 2    Sepal.Length        mm Species     setosa 4.950000 0.09657727 51.25430 0 4.760712 5.139288   a
## 3    Sepal.Length        mm Species  virginica 6.570588 0.15190449 43.25473 0 6.272861 6.868316   a
## 4  a Sepal.Length        mm Species  virginica 6.756250 0.14368309 47.02189 0 6.474636 7.037864   b
## 5  a Sepal.Length        mm Species     setosa 5.052941 0.09286290 54.41291 0 4.870933 5.234949   b
## 6  a Sepal.Length        mm Species versicolor 5.770588 0.12793126 45.10694 0 5.519848 6.021329   b

## session info for your system
sessionInfo()

Add option to print estimates on figures

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Currently one would have to add a geom_text() with some messy sprintf() to get numeric values added. This could be an option or the default.

Reshaping functionality

Please specify whether your issue is about:

  • a possible bug
  • a question about package functionality
  • a suggested code or documentation change, improvement to the code, or feature request

Add a function to reshape a "wide" conjoint dataset into a "long" conjoint dataset, with some flexibility over how that works.

mm() variance clarity

Is it sufficiently clear that mm() returns domain estimates rather than SEs based on subsetting the data?

x <- structure(list(level = structure(c(1L, 1L, 1L, 1L, 2L, 2L, 2L,  2L), .Label = c("John", "Kate"), class = "factor"), outcome = c(0L,  0L, 1L, 1L, 0L, 0L, 1L, 1L), weight = c(1L, 1L, 1L, 1L, 0L, 0L,  0L, 0L)), row.names = 1:8, class = "data.frame")

# what people might be expecting
with(subset(x, level == "John"), sqrt(sum((outcome - mean(outcome))^2)/3/4))
svymean(~outcome, svydesign(ids = ~1, weights = ~ 1, data = subset(x, level == "John")))

# what is actually returned (all are equivalent)
## mm()
mm(x, outcome ~ level)

## unweighted data, subset to John
svymean(~outcome, subset(svydesign(ids = ~1, weights = ~ 1, data = x), level == "John"))

## weighted data (Kate weight == 0), subset to John
svymean(~outcome, subset(svydesign(ids = ~1, weights = ~ weight, data = x), level == "John"))

## weighted data (Kate weight == 0), full data frame
svymean(~outcome, svydesign(ids = ~1, weights = ~ weight, data = x))

[ ] Document this better, pointing to vignette: https://cran.r-project.org/web/packages/survey/vignettes/domain.pdf
[ ] Add option to not calculate variances as if subsets are random samples of population?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.