Improve npm install time / space about create-react-app

[gaearon]

@kentcdodds Thanks so much for the tip.
This should be better after #41.

[wtgtybhertgeghgtwtg]

A big problem here is deduping sub-dependencies. At time of writing, react-scripts can use

• five different versions of readable-stream
• three different versions of each async, minimatch, minimist, faye-websocket, wordwrap, source-map, acorn, and estraverse
• two different versions of each debug, ms, uuid, punycode, qs, assert-plus, commander, supports-color, string-width, is-fullwidth-code-point, is-glob, is-extglob, mime, sockjs-client, url-parse, babel-plugin-transform-es2015-destructuring, balanced-match, glob, inherits, uglify-js, yargs, window-size, cliui, camelcase, json5, once, js-yaml, esprima, path-exists, strip-bom, fsevents, colors, iconv-lite, isarray, babel-runtime, jsesc, babel-core, callsites, utila, domelementtype, domutils, doctrine, regexpu-core, autoprefixer, and babel-plugin-transform-es2015-parameters

There are a handful of other things that take up some time (e.g. packages that depend on stuff for a CLI that isn't used), but deduping is probably a bigger issue.

[wtgtybhertgeghgtwtg]

Another thing that would help would be getting some of the dependencies to exclude tests/examples/config files from their packages. Some of them have a lot of tests (for example, sha.js has over 700 KB of tests), but it's mostly just because they add up.

[wtgtybhertgeghgtwtg]

I have a list of things that could be done to slim down dependencies, but it has hundreds of items. Should I make the issue and only list the larger ones?

[vjeux]

Sorted by size

7.1 M  fsevents
7.1 M  caniuse-db
6.6 M  core-js
5.2 M  fbjs
4.5 M  lodash
3.1 M  es5-ext
3.0 M  react
2.2 M  eslint
1.5 M  sha.js
1.4 M  crypto-browserify
1.3 M  webpack
988 K  pako
936 K  eslint-plugin-jsx-a11y
932 K  babel-runtime
888 K  escope
784 K  source-map
760 K  postcss
688 K  clean-css
676 K  bluebird
604 K  json5
576 K  webpack-dev-server
568 K  uglify-js
568 K  renderkid
496 K  htmlparser2
476 K  acorn
460 K  immutable
456 K  svgo
452 K  source-map-support
444 K  nan
420 K  rx-lite
420 K  csso
412 K  serve-index
400 K  eslint-plugin-import
392 K  iconv-lite
384 K  sockjs-client
376 K  dom-converter
368 K  js-yaml
360 K  eslint-plugin-react
352 K  shelljs
332 K  autoprefixer
328 K  jsx-ast-utils
280 K  webpack-core
276 K  optimist
272 K  babylon
268 K  stream-cache
252 K  xregexp
252 K  resolve
244 K  table
236 K  babel-traverse
236 K  babel-plugin-transform-regenerator
232 K  esprima
228 K  express
220 K  sockjs
216 K  is-my-json-valid
216 K  buffer
212 K  ua-parser-js
204 K  source-list-map
200 K  browserify-zlib
192 K  promise
192 K  node-libs-browser
188 K  watchpack
188 K  coa
180 K  argparse
172 K  stream-browserify
172 K  mime-db
168 K  readable-stream
168 K  babel-types
160 K  doctrine
156 K  mkdirp
156 K  es6-set
156 K  enhanced-resolve
144 K  postcss-selector-parser
144 K  babel-generator
144 K  babel-core
136 K  q
124 K  yargs
124 K  url
120 K  qs
116 K  node-fetch
116 K  es6-map
116 K  domhandler
112 K  tv4
112 K  send
108 K  websocket-driver
108 K  inquirer
108 K  html-minifier
104 K  postcss-merge-longhand
104 K  events
104 K  cliui
100 K  he
 96 K  pretty-error
 96 K  es6-iterator
 96 K  entities
 92 K  readdirp
 92 K  ipaddr.js
 92 K  espree
 92 K  chokidar
 88 K  webpack-dev-middleware
 88 K  relateurl
 88 K  original
 88 K  minimist
 88 K  loader-utils
 88 K  event-emitter
 88 K  es6-symbol
 88 K  cli-width
 88 K  async
 84 K  websocket-extensions
 84 K  uuid
 84 K  http-browserify
 84 K  eventsource
 84 K  es6-weak-map
 80 K  util
 80 K  sprintf-js
 80 K  optionator
 80 K  js-base64
 80 K  faye-websocket
 80 K  eslint-config-airbnb-base
 80 K  cssnano
 76 K  sax
 76 K  mime
 76 K  html-webpack-plugin
 76 K  colors
 72 K  toposort
 72 K  querystring
 72 K  lodash._baseiteratee
 72 K  json3
 72 K  http-proxy
 72 K  eslint-config-airbnb
 72 K  convert-source-map
 68 K  uniqid
 68 K  text-table
 68 K  glob
 68 K  esrecurse
 68 K  css-loader
 64 K  whet.extend
 64 K  micromatch
 64 K  json-stable-stringify
 64 K  esutils
 64 K  domutils
 60 K  wordwrap
 60 K  reduce-function-call
 60 K  reduce-css-calc
 60 K  css-select
 60 K  cross-spawn
 56 K  utila
 56 K  prelude-ls
 56 K  memory-fs
 56 K  loose-envify
 56 K  domain-browser
 56 K  depd
 56 K  debug
 56 K  d
 56 K  commander
 56 K  big.js
 56 K  assert
 52 K  typedarray
 52 K  timers-browserify
 52 K  regjsparser
 52 K  postcss-ordered-values
 52 K  asap
 48 K  vm-browserify
 48 K  react-dom
 48 K  querystring-es3
 48 K  no-case
 48 K  levn
 48 K  isomorphic-fetch
 48 K  globals
 48 K  estraverse
 48 K  color-convert
 48 K  clone
 44 K  yallist
 44 K  through
 44 K  regenerate
 44 K  negotiator
 44 K  minimatch
 44 K  macaddress
 44 K  errno
 44 K  connect-history-api-fallback
 44 K  colormin
 44 K  browserslist
 44 K  babel-plugin-transform-es2015-classes
 40 K  run-async
 40 K  preserve
 40 K  postcss-value-parser
 40 K  postcss-modules-local-by-default
 40 K  postcss-discard-overridden
 40 K  opn
 40 K  jsesc
 40 K  isexe
 40 K  graceful-fs
 40 K  deep-is
 40 K  date-now
 40 K  cssesc
 40 K  console-browserify
 40 K  clap
 40 K  babel-plugin-transform-es2015-parameters
 40 K  babel-plugin-transform-es2015-block-scoping
 40 K  babel-loader
 40 K  acorn-jsx
 36 K  xtend
 36 K  type-check
 36 K  tapable
 36 K  style-loader
 36 K  regenerator-runtime
 36 K  progress
 36 K  postcss-modules-values
 36 K  postcss-minify-font-values
 36 K  postcss-loader
 36 K  pbkdf2-compat
 36 K  jsonify
 36 K  has-own
 36 K  emojis-list
 36 K  core-util-is
 36 K  chalk
 36 K  babel-plugin-transform-es2015-modules-commonjs
 36 K  Base64
 32 K  ripemd160
 32 K  regexpu-core
 32 K  punycode
 32 K  prr
 32 K  lodash._stringtopath
 32 K  isarray
 32 K  find-up
 32 K  file-entry-cache
 32 K  encoding
 32 K  defined
 32 K  css-selector-tokenizer
 32 K  content-disposition
 32 K  compression
 32 K  color-string
 32 K  base64-js
 32 K  babel-plugin-transform-es2015-destructuring
 32 K  babel-helpers
 32 K  async-each
 32 K  amdefine
 28 K  which
 28 K  whatwg-fetch
 28 K  url-parse
 28 K  uglify-to-browserify
 28 K  serve-static
 28 K  rimraf
 28 K  requires-port
 28 K  querystringify
 28 K  process
 28 K  private
 28 K  postcss-svgo
 28 K  postcss-reduce-initial
 28 K  postcss-reduce-idents
 28 K  postcss-minify-selectors
 28 K  postcss-merge-rules
 28 K  postcss-discard-comments
 28 K  nth-check
 28 K  lru-cache
 28 K  lodash._baseeach
 28 K  is-property
 28 K  is-buffer
 28 K  imurmurhash
 28 K  ieee754
 28 K  icss-replace-symbols
 28 K  glob-parent
 28 K  generate-object-property
 28 K  generate-function
 28 K  fs.realpath
 28 K  flat-cache
 28 K  fill-range
 28 K  fastparse
 28 K  dom-serializer
 28 K  cookie
 28 K  concat-map
 28 K  batch
 28 K  babel-register
 28 K  accepts
 24 K  util-deprecate
 24 K  uniq
 24 K  type-is
 24 K  string_decoder
 24 K  statuses
 24 K  regjsgen
 24 K  pseudomap
 24 K  proxy-addr
 24 K  process-nextick-args
 24 K  postcss-zindex
 24 K  postcss-reduce-transforms
 24 K  postcss-discard-unused
 24 K  postcss-convert-values
 24 K  pluralize
 24 K  path-to-regexp
 24 K  os-browserify
 24 K  on-finished
 24 K  num2fraction
 24 K  ms
 24 K  media-typer
 24 K  lodash.rest
 24 K  js-tokens
 24 K  inherits
 24 K  indexof
 24 K  indexes-of
 24 K  http-errors
 24 K  graceful-readlink
 24 K  etag
 24 K  eslint-loader
 24 K  content-type
 24 K  color
 24 K  builtin-modules
 24 K  braces
 24 K  babel-helper-replace-supers
 20 K  vary
 20 K  utils-merge
 20 K  user-home
 20 K  upper-case-first
 20 K  upper-case
 20 K  unpipe
 20 K  uniqs
 20 K  tryit
 20 K  swap-case
 20 K  supports-color
 20 K  strip-json-comments
 20 K  repeating
 20 K  regex-cache
 20 K  range-parser
 20 K  randomatic
 20 K  postcss-unique-selectors
 20 K  postcss-normalize-url
 20 K  postcss-normalize-charset
 20 K  postcss-minify-params
 20 K  postcss-minify-gradients
 20 K  postcss-message-helpers
 20 K  postcss-merge-idents
 20 K  postcss-filter-plugins
 20 K  postcss-discard-empty
 20 K  postcss-discard-duplicates
 20 K  postcss-colormin
 20 K  postcss-calc
 20 K  pinkie
 20 K  path-browserify
 20 K  parseurl
 20 K  on-headers
 20 K  mute-stream
 20 K  mime-types
 20 K  methods
 20 K  merge-descriptors
 20 K  lower-case-first
 20 K  lower-case
 20 K  lodash.words
 20 K  lodash.tostring
 20 K  lodash.endswith
 20 K  lodash.deburr
 20 K  lodash._basetostring
 20 K  kind-of
 20 K  jsonpointer
 20 K  is-upper-case
 20 K  is-lower-case
 20 K  invariant
 20 K  interpret
 20 K  ignore
 20 K  glob-base
 20 K  fresh
 20 K  forwarded
 20 K  flatten
 20 K  finalhandler
 20 K  fast-levenshtein
 20 K  expand-range
 20 K  eventemitter3
 20 K  eslint-import-resolver-node
 20 K  encodeurl
 20 K  detect-indent
 20 K  css-what
 20 K  cookie-signature
 20 K  compressible
 20 K  color-name
 20 K  center-align
 20 K  bytes
 20 K  balanced-match
 20 K  babel-template
 20 K  babel-plugin-transform-es2015-template-literals
 20 K  babel-plugin-transform-es2015-spread
 20 K  babel-plugin-transform-es2015-for-of
 20 K  babel-plugin-transform-es2015-computed-properties
 20 K  babel-messages
 20 K  babel-helper-function-name
 20 K  babel-helper-define-map
 20 K  babel-helper-builder-react-jsx
 20 K  alphanum-sort
 20 K  align-text
 16 K  xml-char-classes
 16 K  write
 16 K  wrappy
 16 K  window-size
 16 K  url-loader
 16 K  tty-browserify
 16 K  to-fast-properties
 16 K  title-case
 16 K  strip-ansi
 16 K  string-width
 16 K  strict-uri-encode
 16 K  sort-keys
 16 K  snake-case
 16 K  slice-ansi
 16 K  shebang-regex
 16 K  setprototypeof
 16 K  sentence-case
 16 K  right-align
 16 K  restore-cursor
 16 K  resolve-from
 16 K  repeat-string
 16 K  repeat-element
 16 K  read-json-sync
 16 K  query-string
 16 K  postcss-modules-scope
 16 K  pkg-up
 16 K  pkg-dir
 16 K  pinkie-promise
 16 K  pify
 16 K  path-is-inside
 16 K  path-is-absolute
 16 K  path-exists
 16 K  path-case
 16 K  pascal-case
 16 K  parse-glob
 16 K  param-case
 16 K  os-tmpdir
 16 K  os-homedir
 16 K  onetime
 16 K  once
 16 K  object.omit
 16 K  object-assign
 16 K  number-is-nan
 16 K  normalize-url
 16 K  normalize-range
 16 K  normalize-path
 16 K  longest
 16 K  lodash.findindex
 16 K  lodash.find
 16 K  lodash.cond
 16 K  lodash.camelcase
 16 K  lodash._root
 16 K  lodash._createcompounder
 16 K  lodash._basefindindex
 16 K  lodash._basefind
 16 K  lazy-cache
 16 K  isobject
 16 K  is-svg
 16 K  is-resolvable
 16 K  is-primitive
 16 K  is-posix-bracket
 16 K  is-plain-obj
 16 K  is-number
 16 K  is-glob
 16 K  is-fullwidth-code-point
 16 K  is-finite
 16 K  is-extglob
 16 K  is-extendable
 16 K  is-equal-shallow
 16 K  is-dotfile
 16 K  is-binary-path
 16 K  is-absolute-url
 16 K  inflight
 16 K  https-browserify
 16 K  home-or-tmp
 16 K  header-case
 16 K  has-flag
 16 K  has-ansi
 16 K  globby
 16 K  for-own
 16 K  for-in
 16 K  figures
 16 K  extglob
 16 K  expand-brackets
 16 K  escape-string-regexp
 16 K  escape-html
 16 K  ee-first
 16 K  dot-case
 16 K  domelementtype
 16 K  destroy
 16 K  del
 16 K  decamelize
 16 K  contains-path
 16 K  constants-browserify
 16 K  constant-case
 16 K  concat-stream
 16 K  code-point-at
 16 K  cli-cursor
 16 K  change-case
 16 K  camelcase
 16 K  camel-case
 16 K  brace-expansion
 16 K  binary-extensions
 16 K  babel-preset-react
 16 K  babel-preset-es2016
 16 K  babel-preset-es2015
 16 K  babel-plugin-transform-strict-mode
 16 K  babel-plugin-transform-react-jsx-source
 16 K  babel-plugin-transform-react-jsx-self
 16 K  babel-plugin-transform-react-jsx
 16 K  babel-plugin-transform-react-inline-elements
 16 K  babel-plugin-transform-react-display-name
 16 K  babel-plugin-transform-react-constant-elements
 16 K  babel-plugin-transform-object-rest-spread
 16 K  babel-plugin-transform-flow-strip-types
 16 K  babel-plugin-transform-exponentiation-operator
 16 K  babel-plugin-transform-es2015-unicode-regex
 16 K  babel-plugin-transform-es2015-typeof-symbol
 16 K  babel-plugin-transform-es2015-sticky-regex
 16 K  babel-plugin-transform-es2015-shorthand-properties
 16 K  babel-plugin-transform-es2015-object-super
 16 K  babel-plugin-transform-es2015-literals
 16 K  babel-plugin-transform-es2015-function-name
 16 K  babel-plugin-transform-es2015-duplicate-keys
 16 K  babel-plugin-transform-es2015-block-scoped-functions
 16 K  babel-plugin-transform-es2015-arrow-functions
 16 K  babel-plugin-syntax-object-rest-spread
 16 K  babel-plugin-syntax-jsx
 16 K  babel-plugin-syntax-flow
 16 K  babel-plugin-syntax-exponentiation-operator
 16 K  babel-plugin-syntax-async-functions
 16 K  babel-plugin-check-es2015-constants
 16 K  babel-helper-regex
 16 K  babel-helper-optimise-call-expression
 16 K  babel-helper-hoist-variables
 16 K  babel-helper-get-function-arity
 16 K  babel-helper-explode-assignable-expression
 16 K  babel-helper-call-delegate
 16 K  babel-helper-builder-binary-assignment-operator-visitor
 16 K  babel-code-frame
 16 K  arrify
 16 K  array-unique
 16 K  array-uniq
 16 K  array-union
 16 K  array-flatten
 16 K  arr-flatten
 16 K  arr-diff
 16 K  anymatch
 16 K  ansi-styles
 16 K  ansi-regex
 16 K  ansi-escapes
 12 K  slash
 12 K  set-immediate-shim
 12 K  require-uncached
 12 K  readline2
 12 K  prepend-http
 12 K  postcss-modules-extract-imports
 12 K  ncname
 12 K  json-loader
 12 K  is-stream
 12 K  is-path-inside
 12 K  is-path-in-cwd
 12 K  is-path-cwd
 12 K  html-comment-regex
 12 K  get-stdin
 12 K  filename-regex
 12 K  file-loader
 12 K  exit-hook
 12 K  damerau-levenshtein
 12 K  css-color-names
 12 K  callsites
 12 K  caller-path
 12 K  boolbase

[eanplatter]

Perhaps by removing npm progress? Perhaps by making an .npmrc file with progress=false. I believe there are still big performance gains doing that.

[vjeux]

If we remove npm progress, we should implement some sort of spinner. Otherwise, the user would be waiting for 2 entire minutes with no feedback on what is happening :(

[eanplatter]

I agree, we could make a spinner that looks a lot better too, using something like Ora

[eanplatter]

I am doing a time test now, I'll post the difference here in a couple minutes.

[eanplatter]

Hrmm, looks like it isn't really an issue anymore with npm's performance. I got 1.29 without progress and 1.25 with progress 😖

[gaearon]

Yea, there was a bug in npm related to that but it was fixed.

[lacker]

Maybe it would be simpler to make something that wasn't specific to this project, that just sped up npm in general. Like if you could specify one cached blob for particular architectures, and then npm installed it, or at least it was a standard mechanism.

[kentcdodds]

Perhaps the project could use bundledDependencies?

[gaearon]

I don’t think we can do much beyond what bundledDependencies provides us.
Let’s stick with it for now and see how it goes.

I’ve read about known issues with npm 2.x + shrinkwrap + bundledDependencies. npm doesn’t plan to fix it. Shrinkwrap isn’t very common in beginner projects so I think we’ll punt on this. If people want to use shrinkwrap and have issues with our package, they can either upgrade to npm 3 (even on older node) or eject (and then all dependencies will be theirs).

[gaearon]

Hopefully it should get better with Yarn now that 0.8.0 will use it out of the box when it's installed.

[wtgtybhertgeghgtwtg]

I am super looking forward to that, but all those versions are still going to be installed. I think the --flat flag might help, but I haven't tried it.

[gaearon]

Why are they getting installed? Is this because some packages depend on incompatible versions? Can we help by sending PRs to the packages using the more outdated versions?

[wtgtybhertgeghgtwtg]

Why are they getting installed? Is this because some packages depend on incompatible versions?

Yes.

Can we help by sending PRs to the packages using the more outdated versions?

I have been trying to do so. The dependency graph shows where multiple versions are used, but it is kinda discouraging to look at.
I'm sorry, that software doesn't take into account deduplication as I had thought it did. The numbers I had give are inaccurate.

[gaearon]

@wtgtybhertgeghgtwtg Would you like to create a new issue aimed at slimming our dependencies down? Nobody is working on it but it should be doable given there's somebody who can lead the effort.

[wtgtybhertgeghgtwtg]

I would love to. I'll try to get that started today.

[gaearon]

Yea that's a good start.