Update constructor to be able to handle dense matrix input.

The web app hangs when running 'Selection Vs Background' differential expression on a file that does not support the latest differential expression. It does not display the warning that DE is not supported and it stop with the progress bar at dowloading.

The cause of this error is an async call to getCellOrder() in order to populate the second (background selection. The async call makes it impossible to trap the 'STATIC_FILE_FIELD_MISSING' exception raised by the file reader when the field is not found.

The ideal solution is to add a function to the datacontroller to indicate if it currently supports DE or not, before calling differential expression.

This applies to the current devel-branch.
When using Chrome Version 61.0.3163.100 on Mac OS with a static file some parts of the heatmap fail to show. It was reproducible for showing multiple genes, but it also sometimes happens for the aspect-Heatmap when first loading the interface.

It seems to have to do with the Javascript Error:

RemoteFileReader.js:33 GET http://pklab.med.harvard.edu/nikolas/pagoda2/staticDemo/10xPBMC.bin net::ERR_CACHE_OPERATION_NOT_SUPPORTED

Currently, if batch is not NULL, when running

myPagoda2WebObject <-                                                                                                                                                                                   
    make.p2.app(                                                                                                                                                                                        
        r = myPagoda2Object,                                                                                                                                                                            
        dendrogramCellGroups = annot,                                                                                                                                                                   
        additionalMetadata = list(),                                                                                                                                                                    
        geneSets = geneSets                                                                                                                                                                             
    );  

you will get an error:

Error in list(data = , palette = rainbow(n = length(levels(r$batch))),  (from #3) :
  argument 1 is empty 

This is because of the code:

if ( "batch" %in% names([email protected]) ) {
        if ( !is.null([email protected]$batch)  ) {
            metadata$batch <- list(
                data = ,
                palette = rainbow(n = length(levels(r$batch))),
                displayname = 'Batch'
            )
        }
    }

To fix, change to something like:

if ( "batch" %in% names([email protected]) ) {
        if ( !is.null([email protected]$batch)  ) {
            metadata$batch <- list(
                data = r$batch,
                palette = rainbow(n = length(levels(r$batch))),
                displayname = 'Batch'
            )
        }
    }

PAGODA2 uses slotNames unique to dgCMatrix class objects ex. p. Coercion into sparse matrices can also produce dgTMatrix objects it seems, which when passed to PAGODA causes errors. It would be good to check and reclass as needed within Pagoda2$new()

Hi
I am interested in correcting multiple batch factors using pagoda2 method. It seems that currently only one batch factor can be used as input.
The batch effect correction method from the code

  if(!is.null(batch)) {

      cat("batch ... ")
      # dataset-wide gene average
      gene.av <- (Matrix::colSums(counts)+length(levels(batch)))/(sum(depth)+length(levels(batch)))
      # pooled counts, df for all genes
      tc <- colSumByFac(counts,as.integer(batch))[-1,,drop=F]
      tc <- t(log(tc+1)- log(as.numeric(tapply(depth,batch,sum))+1))
      bc <- exp(tc-log(gene.av))
      # adjust every non-0 entry
      count.gene <- rep(1:counts@Dim[2],diff(counts@p))
      counts@x <<- counts@x/bc[cbind(count.gene,as.integer(batch)[counts@i+1])]
  }

looks like that we are calculating a scale factor as CPM_{g,batch} / CPM_{g}, then we use the raw counts of each gene in each cell to divide this scale factor as rescaled counts. Is that right ?

I wonder that what is intuitive explanation for this scale factor ? What if I want to rescaled the counts again by another factor such as cell cycling phase ?

Thanks

GO Set generation for web app is slow and duplicates alot of the functionality in go set generation for the overdispersion analysis

I am running this walkthrough and got the adjusted variance as shown in that webpage after executing this line:

r$adjustVariance(plot=T,gam.k=10)

However, based on your R code here, the axis label should be xlab='log[ magnitude ]',ylab='log[ variance ]' rather than xlab='log10[ magnitude ]',ylab='log10[ variance ]'. The plot code (here and here) seems to be wrong.

I am not sure whether my understanding is correct. Can you verify this?

Variance normalization, KNN, cluster detection, and tSNE embedding seem to spuriously fail if I have more than 30,000 cells due to insufficient memory issues. Such large datasets with >30,000 cells will likely be more common in the near future. Other fields like Cytof have already developed solutions for these problems such as by doing density based subsampling. I would recommend we implement something similar.

I'm getting the following error at the very end of the pagoda2 installation. What else should I provide to enable resolving of the issue?
Thanks

installing to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error: package or namespace load failed for ‘pagoda2’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so':
  dlopen(/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so, 6): Symbol not found: __ZN19RuntimeErrorWrapperC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiPKc
  Referenced from: /Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so
  Expected in: flat namespace
 in /Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so
Error: loading failed
Execution halted

Hi Nikolas,

I'm currently generating my pagoda 2 web app, and when trying to run p2.toweb.hdea I receive an error:

Error in namedNames(l) : could not find function "namedNames"

From the pipeline helpers I see that this function is within hierDiffToGenesets, which is used in p2.toweb.hdea.

Is there a package I should be loading for this function?

Thank you!

I tried different cluster methods (infomap, multilevel, walktrap) on ~2000 cells, and always got over 15 clusters. By eyes, there should be many clusters. How can I lower the cluster numbers? Any parameters I can adjust? Thanks.

Currently pagoda2 uses cor method for pearson correlation. It's quite inefficient. There are 2 packages, which provide better implementation coop and HiClimR. On my tests they both perform ~5 times faster (HiClimR is a bit better). Also, neither of them has additional dependencies.

Hi,

I went through most of the installation process for mac (including the change of used compiler) but unfortunately I could not finalise the installation due to the following error :

In file included from /Users/Pierre/nmslib-1.6//similarity_search/lshkit/include/lshkit.h:197:0,
from ../inst/include/pagoda2.h:17,
from RcppExports.cpp:4:
/Users/Pierre/nmslib-1.6//similarity_search/lshkit/include/lshkit/multiprobelsh-fitdata.h:55:10: fatal error: gsl/gsl_multifit.h: No such file or directory
#include <gsl/gsl_multifit.h>
^~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [RcppExports.o] Error 1
ERROR: compilation failed for package ‘pagoda2’

• removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2’
  Installation failed: Command failed (1)

I have checked that the nmslib directory was correctly selected and that other needed packages are correctly installed... Do you have any idea from where it might come from ?

Thank you in advance for your help !

Best

Pierre

Hi,

First I could not installed cairo in R,

install.packages("Cairo")
trying URL 'http://cran.rstudio.com/src/contrib/Cairo_1.5-9.tar.gz'
Content type 'application/x-gzip' length 86982 bytes (84 KB)
==================================================
downloaded 84 KB

• installing source package ‘Cairo’ ...
  ** package ‘Cairo’ successfully unpacked and MD5 sums checked
  checking for gcc... x86_64-conda_cos6-linux-gnu-cc
  checking whether the C compiler works... yes
  checking for C compiler default output file name... a.out
  checking for suffix of executables...
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether we are using the GNU C compiler... yes
  checking whether x86_64-conda_cos6-linux-gnu-cc accepts -g... yes
  checking for x86_64-conda_cos6-linux-gnu-cc option to accept ISO C89... none needed
  checking how to run the C preprocessor... x86_64-conda_cos6-linux-gnu-cc -E
  checking for grep that handles long lines and -e... /usr/bin/grep
  checking for egrep... /usr/bin/grep -E
  checking for ANSI C header files... yes
  checking for sys/wait.h that is POSIX.1 compatible... yes
  checking for sys/types.h... yes
  checking for sys/stat.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for memory.h... yes
  checking for strings.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for unistd.h... yes
  checking for string.h... (cached) yes
  checking sys/time.h usability... yes
  checking sys/time.h presence... yes
  checking for sys/time.h... yes
  checking for unistd.h... (cached) yes
  checking for an ANSI C-conforming const... yes
  checking for pkg-config... /usr/bin/pkg-config
  checking whether pkg-config knows about cairo... no
  configure: CAIRO_LIBS is unset, attempting to guess it.
  configure: CAIRO_CFLAGS=
  checking if R was compiled with the RConn patch... no
  checking cairo.h usability... no
  checking cairo.h presence... no
  checking for cairo.h... no
  configure: error: Cannot find cairo.h! Please install cairo (http://www.cairographics.org/) and/or set CAIRO_CFLAGS/LIBS correspondingly.
  ERROR: configuration failed for package ‘Cairo’
• removing ‘/home/kawai/anaconda3/lib/R/library/Cairo’
  Warning in install.packages :
  installation of package ‘Cairo’ had non-zero exit status

The downloaded source packages are in
‘/tmp/Rtmpmgtj6z/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done

# so I installed with conda instead:
conda install -c r r-cairo
(Then I have cairo in R)

And then it gave me a fatal error when I install pagoda2

install_github("hms-dbmi/pagoda2")
Downloading GitHub repo hms-dbmi/pagoda2@master
from URL https://api.github.com/repos/hms-dbmi/pagoda2/zipball/master
Installing pagoda2
'/home/kawai/anaconda3/lib/R/bin/R' --no-site-file --no-environ --no-save
--no-restore --quiet CMD INSTALL
'/tmp/Rtmpmgtj6z/devtools30e2532c50dc/hms-dbmi-pagoda2-b991a6d'
--library='/home/kawai/anaconda3/lib/R/library' --install-tests

• installing source package ‘pagoda2’ ...
  ** libs
  x86_64-conda_cos6-linux-gnu-c++ -std=gnu++11 -I/home/kawai/anaconda3/lib/R/include -DNDEBUG -I"/home/kawai/anaconda3/lib/R/library/Rcpp/include" -I"/home/kawai/anaconda3/lib/R/library/RcppArmadillo/include" -I"/home/kawai/anaconda3/lib/R/library/RcppProgress/include" -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/home/kawai/anaconda3/include -I"../inst/include" -I"./n2/include" -I"./include" -fopenmp -fpic -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/home/kawai/anaconda3/include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix -c RcppExports.cpp -o RcppExports.o
  In file included from RcppExports.cpp:4:0:
  ../inst/include/pagoda2.h:14:10: fatal error: boost/progress.hpp: No such file or directory
  #include <boost/progress.hpp>
  ^~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  make: *** [RcppExports.o] Error 1
  ERROR: compilation failed for package ‘pagoda2’
• removing ‘/home/kawai/anaconda3/lib/R/library/pagoda2’
  Installation failed: Command failed (1)

It will be highly appreciated if you could inform me how I can solve this.

Thank you!

When I try to run basicP2proc() on my sparse matrix from 10X data, I get an error message:

Error in irlba(x, nv = nPcs, nu = 0, center = cm, right_only = FALSE, :
max(nu, nv) must be strictly less than min(nrow(A), ncol(A))

Any thoughts on resolving this issue? Thanks!!

Hello,

I'm encountering the following error when attempting to install pagoda2:

**clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG  -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppProgress/include" -I/usr/local/include  -O3 -march=native -std=c++11 -pthread -fPIC -fopenmp -I"./include" -fPIC  -Wall -g -O2 -c base.cc -o base.o
clang: error: unsupported option '-fopenmp'
make[1]: *** [base.o] Error 1
make: *** [sublibraries] Error 1
ERROR: compilation failed for package ‘pagoda2’
removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2’
Installation failed: Command failed (1)**

I've made sure that clang4, gfortran, etc. are installed. I also updated my ~/.R/Makevars:

CC=/usr/local/clang4/bin/clang
CXX=/usr/local/clang4/bin/clang++
CXX11=/usr/local/clang4/bin/clang++
CXX14=/usr/local/clang4/bin/clang++
CXX17=/usr/local/clang4/bin/clang++
CXX1X=/usr/local/clang4/bin/clang++
LDFLAGS=-L/usr/local/clang4/lib

For what it's worth, it appears that the installer uses the -fopenmp argument without generating an error in the previous line of output:

> /usr/local/clang4/bin/clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG  -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppProgress/include" -I/usr/local/include  -I"../inst/include" -I"./n2/include" -I"./include" -fopenmp  -fPIC  -Wall -g -O2 -c rank.cpp -o rank.o

Any help would be greatly appreciated!
-Jack

Hi,

I following the tutorial, but I used the 10x pbmc data [(http://pklab.med.harvard.edu/velocyto/notebooks/R/SCG71.nb.html)]:
Here is the code:
`
library(velocyto.R)
library(Matrix)
dat <- readRDS(url("http://pklab.med.harvard.edu/velocyto/mouseBM/cell.counts.matrices.rds"))
emat <- dat$exon
library(pagoda2)
r <- Pagoda2$new(emat,modelType='plain',trim=10,log.scale=T)
2303 cells, 14708 genes; normalizing ... using plain model winsorizing ... log scale ... done.

r$adjustVariance(plot = T, gam.k = 10)
calculating variance fit ... using gam Error in uniquecombs(matrix(x, n, object$dim)) :
object 'C_RuniqueCombs' not found`

I updated the Matrix package and all other packages. It did not fix the issue.

Any thoughts?

Here is my session info:
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] mgcv_1.8-23 nlme_3.1-131 velocyto.R_0.5
[4] pagoda2_0.0.0.9002 Matrix_1.2-12 BiocInstaller_1.28.0
[7] devtools_1.13.4

loaded via a namespace (and not attached):
[1] Rcpp_0.12.15 compiler_3.4.3 pillar_1.1.0
[4] git2r_0.21.0 base64enc_0.1-3 tools_3.4.3
[7] digest_0.6.15 bit_1.1-12 RSQLite_2.0
[10] memoise_1.1.0 tibble_1.4.2 lattice_0.20-35
[13] pkgconfig_2.0.1 rlang_0.1.6 igraph_1.1.2
[16] DBI_0.7 curl_3.1 parallel_3.4.3
[19] cluster_2.0.6 org.Mm.eg.db_3.5.0 withr_2.1.1
[22] httr_1.3.1 knitr_1.19 S4Vectors_0.16.0
[25] IRanges_2.12.0 triebeard_0.3.0 dendsort_0.3.3
[28] stats4_3.4.3 bit64_0.9-7 grid_3.4.3
[31] Biobase_2.38.0 R6_2.2.2 AnnotationDbi_1.40.0
[34] tcltk_3.4.3 Rook_1.1-1 irlba_2.3.2
[37] org.Hs.eg.db_3.5.0 magrittr_1.5 GO.db_3.5.0
[40] blob_1.1.0 urltools_1.7.0 pcaMethods_1.70.0
[43] BiocGenerics_0.24.0 MASS_7.3-48 RMTstat_0.3
[46] brew_1.0-6 rjson_0.2.15

Hi,

I went through the installation process (on a Mac) successfully until I got an error at the very end:

install_github("hms-dbmi/pagoda2") gives the following error:

In file included from RcppExports.cpp:4:
./../inst/include/pagoda2.h:14:10: fatal error: 'space.h' file not found
#include "space.h"
^~~~~~~~~
1 error generated.
make: *** [RcppExports.o] Error 1
ERROR: compilation failed for package ‘pagoda2’

This may be a bug? I would very much appreciate your help.
Thanks so much,
Katja

I double-clicked on an aspect to view the GO terms associated with it.

The text in the "Gene sets in Aspect" window became garbled, and it stayed that way even as I clicked on other things:

Notably, the first 3 terms become un-clickable. This includes 004058, 0005770, 0002685. The 4th term is clickable.

Unfortunately, I can't give you detailed instructions to reproduce this error. If I find a way to reproduce it reliably, I'll update this issue.

Hi,

I installed pagoda2 via docker container, and tried to learn pagoda2 by running the command lines in PAGODA2 8k PBMC walkthrough. After loading the pagoda2 package, I tried to run the following command:
cd <- read.10x.matrices(list(PBMC8K='pbmc8k/raw_gene_bc_matrices/GRCh38'))[[1]]

However, I got an error saying that:
could not find function "read.10x.matrices"

Could you please tell me what I am missing?

Also, is there a manual for pagoda2 that documents available functions in pagoda2?

Thanks!

serializeToStaticFast() fails with uninformative error when makeGeneKnnGraph() has not been run. The correct behaviour is to give a warning and build the file regardless as this component is optional.

I used the following code on data with 4 clusters (1, 2, 3 and 4):

p2 <- basicP2proc(countMatrix, n.cores = 20)
p2$getKnnClusters(method = igraph::infomap.community, type = 'PCA' ,name = 'infomap', test.stability=T, plot=T)

After that, names of some clusters become string like "2 + 3", meaning, if I understood correctly, that these clusters were merged. Afterwards, I run webapp with the following code:

ext.res <- extendedP2proc(p2, n.cores = 20, organism = 'mm')

p2 <- ext.res$p2
go.env <- ext.res$go.env
rm(ext.res)

metadata.listfactors <- list(
    infomap = p2$clusters$PCA$infomap,
    multilevel = p2$clusters$PCA$multilevel,
    walktrap = p2$clusters$PCA$walktrap
);
metadata.forweb <- factorListToMetadata(metadata.listfactors)

p2.webobject <- webP2proc(p2, additionalMetadata = metadata.forweb, title = 'P2App', go.env = go.env)
p2.webobject$mainDendrogram$cluster.sizes
p2.webobject$serializeToStaticFast('serialisedApp.bin');

show.app(p2.webobject, name="P2App", browse = T)

JavaScript part iterates over integer indexes and fails to understand string cluster name.

Probably a basic question, but I am new to installing these sorts of programs into R.

I followed the commands to install NMSLIB and the largeVis folders. Does it matter where on my computer I install these (i.e. do the directory names matter?)

When I run these commands in R, I get an error:
install.packages('devtools')
library(devtools)
install('~/Library/tmp/largeVisCustom/largeVis')

ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0'
ld: warning: directory not found for option '-L/usr/local/gfortran/lib'
ld: library not found for -lgfortran
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [largeVis.so] Error 1
ERROR: compilation failed for package ‘largeVis’

• removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/largeVis’
  Error: Command failed (1)

Any ideas on how to fix?

Thanks!

Core functionality is undocumented

I would like to use custom annotation with BioMart to 10x genomics data on PAGODA2 walkthrough.
Tutorial uses GO number with below commands however I want to see with GO annotation on browser.

I just did

library(biomaRt)
library(GO.db)
ensembl <- useMart("ENSEMBL_MART_ENSEMBL", dataset = "hsapiens_gene_ensembl", host="www.ensembl.org")
go <- getBM(attributes = c("hgnc_symbol", "go_id"), filters = "hgnc_symbol", values = rownames(counts), mart = ensembl)
go$term <- Term(go$go_id)
s = split(go$hgnc_symbol, paste(go$go_id,go$term))
go.env <- list2env(s)
r$testPathwayOverdispersion(go.env,verbose=T,correlation.distance.threshold=0.95,recalculate.pca=F,top.aspects=15)

However after app command, below error appeared.

app <- p2.make.pagoda1.app(r,go.env,inner.clustering=TRUE,embeddingType='tSNE',clusterType='community',min.group.size=1)
Error in mget(df$name, GO.db::GOTERM, ifnotfound = NA) :
second argument must be an environment

If you can help me, this is really appreciated.

I encountered the following error when running tSNE on a 50k cell dataset:

Error: C stack usage 97164344788 is too close to the limit
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

This is really a multicore tSNE bug, but it should be trapped correctly in pagoda and reported appropriately to the user.

when I install pagoda2 in Mac OS (version 10.13.4), R reports an error when compiling:

devtools::install_github('hms-dbmi/pagoda2', build_vignettes = FALSE)

/usr/local/clang4/bin/clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/clang4/lib -o pagoda2.so RcppExports.o binaryExport.o checkfunctions.o edgeweights.o gradients.o largeVis.o misc2.o n2knn.o pagoda1_liftover.o rank.o -L/usr/lib/ -L. -lpthread -lboost_filesystem -lboost_system -lstdc++ -ln2 -lgsl -lgslcblas -lm -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm -fopenmp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation

ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0'
ld: warning: directory not found for option '-L/usr/local/gfortran/lib'
ld: library not found for -lgsl
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [pagoda2.so] Error 1
ERROR: compilation failed for package ‘pagoda2’

• removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2’
  Installation failed: Command failed (1)

Cell selections generated from R with the writePagoda2SelectionFile() won't load on the web interface. The reason for this is that the web interface won't load a file that ends in newline. The correct behaviour is to expect a file ending in newline but tolerate one that doesn't

Dear developer,
I posted installation problem and got the response. I appreciate for that, but I still met the installation problem, the log information is shown bellow. There is a warning during installation: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0'. I am using Mac OS 10.12.6 system, and I cannot install gfortran-4.8.2. However, if I install the latest gfortran, I can only get: /usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/. Could you help to resolve this version problem? or you have some suggestions. Thanks very much!!

install_github("hms-dbmi/pagoda2")
Downloading GitHub repo hms-dbmi/pagoda2@master
from URL https://api.github.com/repos/hms-dbmi/pagoda2/zipball/master
Installing pagoda2
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ
--no-save --no-restore --quiet CMD INSTALL
'/private/var/folders/8k/xcdm75p95tz3nz3r9lykjtvh0000gn/T/RtmppNyY8z/devtools21aa34b5ee7e/hms-dbmi-pagoda2-470f1bd'
--library='/Library/Frameworks/R.framework/Versions/3.4/Resources/library'
--install-tests

• installing source package ‘pagoda2’ ...
  ** libs
  /usr/local/clang4/bin/clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I/usr/local/include -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include" -I"../inst/include" -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/lshkit/include" -fopenmp -fPIC -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
  In file included from RcppExports.cpp:4:
  In file included from ./../inst/include/pagoda2.h:15:
  In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
  /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:53:12: warning: 'WARN' macro redefined [-Wmacro-redefined]

define WARN(exp) (FILE_LINE_LINK "WARNING: " exp)

       ^

/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:60:9: note: previous definition is here
#define WARN WARNING(NULL)
^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:66:9: warning: 'ISNAN' macro redefined [-Wmacro-redefined]
#define ISNAN std::isnan
^
/Library/Frameworks/R.framework/Resources/include/R_ext/Arith.h:74:11: note: previous definition is here

define ISNAN(x) R_isnancpp(x)

      ^

2 warnings generated.
/usr/local/clang4/bin/clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I/usr/local/include -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include" -I"../inst/include" -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/lshkit/include" -fopenmp -fPIC -Wall -g -O2 -c binaryExport.cpp -o binaryExport.o
In file included from binaryExport.cpp:7:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:53:12: warning: 'WARN' macro redefined [-Wmacro-redefined]

define WARN(exp) (FILE_LINE_LINK "WARNING: " exp)

       ^

/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:60:9: note: previous definition is here
#define WARN WARNING(NULL)
^
In file included from binaryExport.cpp:7:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:66:9: warning: 'ISNAN' macro redefined [-Wmacro-redefined]
#define ISNAN std::isnan
^
/Library/Frameworks/R.framework/Resources/include/R_ext/Arith.h:74:11: note: previous definition is here

define ISNAN(x) R_isnancpp(x)

      ^

2 warnings generated.
/usr/local/clang4/bin/clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I/usr/local/include -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include" -I"../inst/include" -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/lshkit/include" -fopenmp -fPIC -Wall -g -O2 -c misc2.cpp -o misc2.o
In file included from misc2.cpp:7:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:53:12: warning: 'WARN' macro redefined [-Wmacro-redefined]

define WARN(exp) (FILE_LINE_LINK "WARNING: " exp)

       ^

/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:60:9: note: previous definition is here
#define WARN WARNING(NULL)
^
In file included from misc2.cpp:7:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:66:9: warning: 'ISNAN' macro redefined [-Wmacro-redefined]
#define ISNAN std::isnan
^
/Library/Frameworks/R.framework/Resources/include/R_ext/Arith.h:74:11: note: previous definition is here

define ISNAN(x) R_isnancpp(x)

      ^

2 warnings generated.
/usr/local/clang4/bin/clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I/usr/local/include -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include" -I"../inst/include" -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/lshkit/include" -fopenmp -fPIC -Wall -g -O2 -c nsmlib.cpp -o nsmlib.o
In file included from nsmlib.cpp:3:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:53:12: warning: 'WARN' macro redefined [-Wmacro-redefined]

define WARN(exp) (FILE_LINE_LINK "WARNING: " exp)

       ^

/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:60:9: note: previous definition is here
#define WARN WARNING(NULL)
^
In file included from nsmlib.cpp:3:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:66:9: warning: 'ISNAN' macro redefined [-Wmacro-redefined]
#define ISNAN std::isnan
^
/Library/Frameworks/R.framework/Resources/include/R_ext/Arith.h:74:11: note: previous definition is here

define ISNAN(x) R_isnancpp(x)

      ^

nsmlib.cpp:102:7: warning: unused variable 'nanswers' [-Wunused-variable]
int nanswers=nqueries*k;
^
nsmlib.cpp:86:14: warning: variable 'index' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]
} else if (indexType == INDEX_TYPE_LP) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
nsmlib.cpp:90:3: note: uninitialized use occurs here
index->CreateIndex(IndexParams);
^~~~~
nsmlib.cpp:86:10: note: remove the 'if' if its condition is always true
} else if (indexType == INDEX_TYPE_LP) {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nsmlib.cpp:81:22: note: initialize the variable 'index' to silence this warning
Index *index;
^
= nullptr
4 warnings generated.
/usr/local/clang4/bin/clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I/usr/local/include -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include" -I"../inst/include" -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/lshkit/include" -fopenmp -fPIC -Wall -g -O2 -c pagoda1_liftover.cpp -o pagoda1_liftover.o
In file included from pagoda1_liftover.cpp:1:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:53:12: warning: 'WARN' macro redefined [-Wmacro-redefined]

define WARN(exp) (FILE_LINE_LINK "WARNING: " exp)

       ^

/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:60:9: note: previous definition is here
#define WARN WARNING(NULL)
^
In file included from pagoda1_liftover.cpp:1:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:66:9: warning: 'ISNAN' macro redefined [-Wmacro-redefined]
#define ISNAN std::isnan
^
/Library/Frameworks/R.framework/Resources/include/R_ext/Arith.h:74:11: note: previous definition is here

define ISNAN(x) R_isnancpp(x)

      ^

2 warnings generated.
/usr/local/clang4/bin/clang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppArmadillo/include" -I/usr/local/include -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include" -I"../inst/include" -I"/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/lshkit/include" -fopenmp -fPIC -Wall -g -O2 -c rank.cpp -o rank.o
In file included from rank.cpp:1:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:53:12: warning: 'WARN' macro redefined [-Wmacro-redefined]

define WARN(exp) (FILE_LINE_LINK "WARNING: " exp)

       ^

/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:60:9: note: previous definition is here
#define WARN WARNING(NULL)
^
In file included from rank.cpp:1:
In file included from ../inst/include/pagoda2.h:15:
In file included from /Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/space.h:32:
/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/include/utils.h:66:9: warning: 'ISNAN' macro redefined [-Wmacro-redefined]
#define ISNAN std::isnan
^
/Library/Frameworks/R.framework/Resources/include/R_ext/Arith.h:74:11: note: previous definition is here

define ISNAN(x) R_isnancpp(x)

      ^

2 warnings generated.
/usr/local/clang4/bin/clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/clang4/lib -o pagoda2.so RcppExports.o binaryExport.o misc2.o nsmlib.o pagoda1_liftover.o rank.o -L/usr/lib/ -L/Users/zhengrongbin/Documents/nmslib-1.6/similarity_search/release -lNonMetricSpaceLib -lgsl -lpthread -lboost_filesystem -lboost_system -lstdc++ -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm -fopenmp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0'
installing to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error: package or namespace load failed for ‘pagoda2’ in dyn.load(file, DLLpath = DLLpath, ...):
cannot load the shared object ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so’：:
dlopen(/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so, 6): Library not loaded: libboost_filesystem.dylib
Referenced from: /Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so
Reason: image not found
Error：loading failed
stop execution
ERROR: loading failed

• removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2’
  Installation failed: Command failed (1)

Hi,

I've finally try to run Pagoda2. I like it and I find the web app really helpful!
Especially this new interactive version looks promising.
However, I remember in pagoda 1, it was nice to see GOs related to each aspects, while now I can only
see these numbers, which is not clear to me what they are..
In the example bin file you prepared looks like in the attached screenshot:

Also I'm not sure about the meaning of white parts in the heatmaps with aspects.

Any helps will be really appreciated!

Thanks a lot,

Elisabetta

Hi,

Congratulations to this great package! I am very impressed about how fast it is.

For the differential expression analysis using "getDifferentialGenes": What are the methods for this (did you use scde?), and could you explain me what the columns of the result output mean: "Z", "M", "highest", and "fe"?

Thank you very much,
Katja

nmslib is a very 'heavy' dependency (often requiring administrative privileges to install). Such difficulty in installation may impede adoption of pagoda2. I recommend using RANN (https://cran.r-project.org/web/packages/RANN/RANN.pdf) as a fallback for those unable or unwilling to install nmslib (just like RtSNE.multicore is a sort of fallback/alternative for those who don't want to or unable to install the custom largeViz).

An example of RANN:

require(RANN)
## where mat would be a matrix of PCs by cells
knn <- RANN::nn2(t(mat), k=k)[[1]]

## convert to adjacency matrix
adj <- matrix(0, ncol(mat), ncol(mat))
rownames(adj) <- colnames(adj) <- colnames(mat)
invisible(lapply(seq_len(ncol(mat)), function(i) {
        adj[i,colnames(mat)[knn[i,]]] <<- 1
}))

You can then convert to a graph to be able to use the igraph clustering methods as done currently:

## convert to graph for clustering
require(igraph)
g <- igraph::graph.adjacency(adj, mode="undirected")
km <- igraph::cluster_walktrap(g)
## community membership
com <- km$membership
names(com) <- km$names

For the plotEmbedding function, if the selected embedding is tSNE, then panel.first = grid() should not be included in the final plot(emb, col = adjustcolor(cols, alpha = alpha)... line.

Regex selection of cell on the web interface doesn't work

You might consider sorting the dendrograms in the display. By default, the tree branches returned by hclust are not sorted by similarity.

After sorting with dendsort::dendsort(), the most similar branches are shown on the left and the most distant branches are shown on the right.

Example

Here's an example of the default unsorted tree:

mat_cluster_cols <- hclust(dist(t(mat)))

plot(mat_cluster_cols, main = "Unsorted Dendrogram", xlab = "", sub = "")

And the sorted one:

# install.packages("dendsort")
library(dendsort)

sort_hclust <- function(...) as.hclust(dendsort(as.dendrogram(...)))

mat_cluster_cols <- hclust(dist(t(mat)))

mat_cluster_cols <- sort_hclust(mat_cluster_cols)

plot(mat_cluster_cols, main = "Sorted Dendrogram", xlab = "", sub = "")

I have a matrix of data that I have already cleaned as part of my earlier analysis (filtered genes/cells, normalised data to logTPMs and then performed batch normalisation using Limma). I'm attempting to follow the vignette as I'm interested in using PAGODA2 to perform pathway level analysis, but I'm already stuck: I'm unclear from the documentation about creating a pagoda2 object as to whether or not it is possible to create an object from data that has already been normalised (and is in log scale), and if so, how this can be indicated in the parameters? Or does PAGODA2 only work on matrices of raw counts?

Advice much appreciated!

Best wishes,

Billy

Hi, I have this problem at the final step of installation. Do you know any solutions? I list the final part of the log information. Thanks a lot.
installing to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error: package or namespace load failed for ‘pagoda2’ in dyn.load(file, DLLpath = DLLpath, ...):
cannot load the shared object ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so’：:
dlopen(/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so, 6): Library not loaded: libboost_filesystem.dylib
Referenced from: /Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2/libs/pagoda2.so
Reason: image not found

Does this the problem on the version of the dependance?

Hi,

I really would like to use pagoda2 but I have a problem during the installation process.
I followed all the steps described in the github installation page but still something is not working.
My computer is a Mac OS El Capitan v.10.11.6 and after I run the last command install_github("hms-dbmi/pagoda2") on R I got this error:
clang: error: unsupported option '-fopenmp'
make[1]: *** [base.o] Error 1
make: *** [sublibraries] Error 1
ERROR: compilation failed for package 'pagoda2'

• removing '/Library/Frameworks/R.framework/Versions/3.4/Resources/library/pagoda2'

Can you help me to solve this problem?

Thank you in advance.
Bests,

Elisabetta

Running through the walkthrough I encountered warnings with p2.make.pagoda1.app when generating the app (which then caused running the app to fail)

Warning message:
  6 function calls resulted in an error

I traced the issue back to line 62 of p1app4p2.r:

        hcc <- fastcluster::hclust(dd, method = "ward.D")

I did not have this package installed (although it is in the "Suggests" list of DESCRIPTION). Installing the package resolved this problem.

I notice elsewhere this is handled thusly:

  if(is.element("fastcluster", installed.packages()[, 1])) {
    y <- fastcluster::hclust(cc, method = cluster.method)
  } else {
    y <- stats::hclust(cc, method = cluster.method)
  }

Should this strategy be used for p2.make.pagoda1.app as well?

Importing a selections file on the web application fails if a last empty line exists

Rcpp code uses std::count, replace with Rcpp::Rout

Breast cancer data from Azizi et. al (azizi.breast.rds from conos tests), sample BC5

plotEmbedding() crashes if the groups parameter specified does not contain any cells in the app, it should just print everything in the NA color

If you use p2.metadata.from.factor with a factor with 2 levels it results in a object with the same color for both classes:

fact <- as.factor(c(1,2)); names(fact) <- c("A","B"); test <- p2.metadata.from.factor(fact,displayname="test"); test
-->
$palette
"#FF0000FF" "#FF0000FF"