elastic / stack-docs

Elastic Stack Documentation

stack-docs

This repository is the home for the cross-stack (Elastic Stack) documentation, such as the Installation and Configuration Guide and the Machine Learning Guide.

Directories

Within this repo, the /docs/en/ directory is structured as follows:

Directory	Description
getting-started	Contains the source files for the Getting Started Guide.
gke-on-prem	Contains the source files for Elastic Stack and Google Cloud's Anthos
glossary	Contains the source files for the Elastic Glossary.
install-upgrade	Contains the source files for the Elastic Installation and Upgrade Guide.
stack/ml	Contains the source files for the Machine Learning Guide.

Backporting

In general, we backport documentation changes only to live stack versions.

To automatically create backport PRs, add the appropriate backport labels (such as backport-8.1).

If no backport PRs are required or you want to backport manually, add the backport-skip label. We recommend using the backport tool to easily open backport PRs. If you need help, ping @mlr-docs and we'd be happy to help.

Build

To build the docs:

1. Identify which book you want to build and whether it has any dependencies on other repositories. You can see the required repositories for each book in either the conf.yaml or the doc_build_aliases.sh file.

2. Check out the elastic/docs repository and any other necessary repositories.

3. Run the appropriate build_docs script, as listed in the doc_build_aliases.sh file.

License

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License.

stack-docs's Issues

Document using Metricbeat for collecting and shipping monitoring metrics

Related to elastic/beats#7035

• Update overview (benefits of new method) in https://www.elastic.co/guide/en/elastic-stack-overview/master/xpack-monitoring.html

• Update architectural diagrams (include Metricbeat) in https://www.elastic.co/guide/en/elastic-stack-overview/master/how-monitoring-works.html See https://github.com/elastic/Design/issues/1090 and #125

• Add new MetricBeat installation, configuration steps for Kibana data collection in https://www.elastic.co/guide/en/elastic-stack-overview/master/monitoring-production.html

• Add new MetricBeat installation, configuration steps for all other products' data collection in https://www.elastic.co/guide/en/elastic-stack-overview/master/monitoring-production.html, as that functionality is enabled.

• Update the monitoring pages in the Kibana Reference to align with these changes (e.g. https://www.elastic.co/guide/en/kibana/master/monitoring-xpack-kibana.html) See elastic/kibana#23736

• Update the monitoring pages in the Elasticsearch Reference to align with these changes (e.g. https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-monitoring.html). See elastic/elasticsearch#34339

• Update the monitoring pages in the other product references to align with these changes (e.g. https://www.elastic.co/guide/en/logstash/current/configuring-logstash.html, https://www.elastic.co/guide/en/beats/filebeat/master/monitoring.html), as that functionality is enabled.

6.0 Upgrade Project

This issue encapsulates the work we know we need to do for the 6.0 upgrade project. It's a superset of #14, so I'll close that issue. We will undoubtedly discover things we need to do along the way (the Rumsfeldian "unknown unknowns".) Teams and tasks are broken down by product. Each product also includes its X-Pack features. There is also a "stack" team that covers cross-product considerations. Elasticsearch and Kibana are a bit heavy on things we already know we need to do, so any extra help there would be welcomed. We could also use some more eyes on Logstash, as @jsvd is currently a one-man army!

Elasticsearch

Team

• @clintongormley
• @imotov
• @tvernum

Todos

• Verify upgrade procedure
• Verify breaking changes
• Document rolling upgrades from 5.6 to 6.0
• Document implications of Security requiring TLS between nodes as of 6.0
  • i.e. If you don't have TLS between nodes, you can't do a rolling upgrade from 5.6 to 6.0; you have to do a full-cluster restart
• Document manual reindexing of internal indices (e.g. .watcher, .security)
  • This is for X-Pack users who don't want to use the upgrade assistant tool in Kibana. Probably rare, but a valid use case.
• Verify es-hadoop docs
  • Upgrade docs
  • Breaking changes docs

Logstash

Team

• @jsvd
• TBD

Todos

• Verify (write?) upgrade procedure
  • Looks like this hasn't been updated for 6.0 yet
• Verify (write?) breaking changes
  • Looks like these haven't been updated for 6.0 yet
• Any considerations re: removal of multiple types that need to be documented?
  • Have plugins that we maintain accounted for this?
  • Should we provide some guidance for maintainers of community plugins?

Kibana

Team

• @pickypg
• @LeeDr
• @archanid

Todos

• Verify (write?) upgrade procedure
  • Looks like this hasn't been updated for 6.0 yet
• Verify breaking changes
• Document upgrade assistant (@archanid) #18
• Document manual reindexing of .kibana for users without X-Pack (@tylersmalley) #19

Beats

Team

• @tsg
• @dedemorton

Todos

• Verify (write?) upgrade procedure
  • Looks like these haven't been updated for 6.0 yet
• Verify (write?) breaking changes
  • Looks like these haven't been updated for 6.0 yet
• Any considerations re: removal of multiple types that need to be documented?

Cloud

Team

• @stejacks
• @stu-elastic
• @beiske
• @igor-kupczynski

Todos

• Specifics TBD
  • Is there anything Cloud-specific re: upgrading from 5.x to 6.0? Anything that needs to be changed here, other than 5.6->6.0 not requiring a full-cluster restart?
  • Any special considerations for ECE?
  • @nrichers is working on ECE-specific upgrade documentation, so perhaps it will ultimately just be a matter of linking to it from these docs (https://www.elastic.co/guide/en/cloud/current/upgrading.html; WIP 6.0 doc PR)

Stack

Team

• @inqueue
• @cwurm
• @msimos
• Steven G.

Todos

• Verify stack upgrade order from 5.0 still holds
• Update full-stack upgrade doc
• Test procedure including X-Pack
• Test procedure excluding X-Pack

Misc.

• Mine customer feedback sources (casesearch, enhancements, Discuss, etc.) for any common suggestions/complaints/praise re: the 2.x->5.0 procedure.
  • The idea here is to learn from whatever (if any) mistakes we made last time and get us thinking upfront about potential customer pain points.
  • @michaelbaamonde and @zuketo to lead.
• Document migrating to and living in a post-types world.
  • Reindexing internal indices to their new, typeless mappings is covered under individual products, but there's a broader question of guiding users re: their own data models. We should produce some artifact that discusses some approaches, possibly using our internal experiences with things like Monitoring as a jumping-off point.
  • See https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
• Do we need a migration guide for scripting now that we've removed support for all languages but Painless?
• Ensure instructions also work for our Docker images
  • Docker is A Very Special Snowflake and is consistently surprising, so assume nothing in this scorched hellscape of containers and orchestrators that we, the wailing and the doomed, have inherited

Tactical doc updates for our products in Docker / Kubernetes

In order to improve the getting started experience for our users that deploy with Docker or Kubernetes we need some updates / changes to the docs. Target date for this is 30 November 2018.

Docker config files and commands in GitHub:

• Filebeat
• Metricbeat
• Heartbeat
• Packetbeat
• Auditbeat
• Elasticsearch
• Kibana

Updates to these docs for Docker will be tracked here:

• Filebeat
• Metricbeat
• Heartbeat
• Packetbeat
• Auditbeat
  - [ ] Elasticsearch No change needed
  - [ ] Kibana No change needed

Updates to these docs for Kubernetes will be tracked here:

• Filebeat
• Metricbeat
  - [ ] Heartbeat : not supported yet
  - [ ] Packetbeat : not supported yet
  - [ ] Auditbeat : not supported yet

Other docs that will be updated through the Helm project (later, not part of this issue):
- [ ] Elasticsearch
- [ ] Kibana
- [ ] APM Server

PRs in:
-[x] Beats
-[ ] Elasticsearch No change needed
-[ ] Kibana No change needed

cc: @liztauscher @kate-smith

@tylerjl @exekias Can you volunteer someone to help review? Similar to the blogs and demos I have been running past you, our goal here is to publish good advice in the docs, and we don't want to embarrass ourselves by publishing something that works but is deprecated.

[DOCS] Add security warning in Installation and Upgrade Guide

Information related to https://www.elastic.co/blog/elastic-stack-6-3-0-and-6-3-1-may-disable-security-for-trial-licenses
... should be added to the upgrade information (e.g. as an "Important" section near the top of this page: https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html)

Upgrade Path Builder

This is still very much a WIP, but time is getting very short so opening this issue to get more eyeballs on it.

Looking for any and all feedback, but the most important thing is making sure that the various paths are fleshed out appropriately. That goes beyond making sure there aren't any paths that throw errors, the actual content & recommendations have to be accurate. And, of course, it needs to look decent -- @cjcenizal gets the cred for the styling, any bad line breaks, weirdo spacing, etc. are on me, along with the content issues.

You can try out a live preview at: https://upgrade-adventure.firebaseapp.com/find_path.html

This is built using Twine--not a great longterm solution, but there wasn't enough time to roll our own. It uses the Sugarcube 2 story format. You can load the attached find_path.txt file into the online version of Twine to see a visualization of the paths. Right now, that looks like this:

I've also attached the "proofing" version that contains just the content source. (If you import find_path.txt into Twine, you can also view the proofing version in the app.)

find_path.txt
find_path_source.txt

Remove glossary from all but master branch

The https://github.com/elastic/docs/blob/master/conf.yaml indicates that we're only generating the Glossary from a single branch: master.

Therefore the glossary folder should be removed from all other branches. Otherwise folks might make changes in branches that aren't used or backport unnecessarily.

non-super user can't create cross cluster index pattern

Original comment by @LeeDr:

Kibana version: 5.5.0

Elasticsearch version: 5.5.0

Server OS version: Ubuntu

Browser version: Chrome

Browser OS version: Ubuntu

Original install method (e.g. download page, yum, from source, etc.): tar.gz

Description of the problem including expected versus actual behavior:
I think I'm giving a user roles with permissions that should allow them to create a cross cluster index pattern, but it fails.

Steps to reproduce:

1. set up 2 es nodes so that one is a "local" admin cluster that Kibana uses (9200), and the other is a remote "data" cluster (9210).
2. Add makelogs data to both clusters
3. create a makelogs_reader role, that has index patterns makelogs-*, local:makelogs-*, data:makelogs-*, *:makelogs-* and privs read, view_index_metadata, read_cross_cluster
4. create a kibana_css role that is just like the kibana_user role except add the read_cross_cluster priv on the .kibana* index. (I didn't think this step should be necessary since .kibana isn't cross cluster, but I tried it when things didn't work with kibana_user role)
5. Create a makelogs_reader user with makelogs_reader and kibana_css roles
6. log in as that user and try to create any index pattern containing :.
   It fails right after you type the : like data:

Up to the point where I type data I can see Kibana checking if that index exists and getting a 404 as expected. But as soon as I type the : I get the red toast error banner and the console shows this;

Errors in browser console (if relevant):

getFieldsForWildcard(data:)
VM10731:1 GET https://localhost:5601/api/index_patterns/_fields_for_wildcard?pattern=data…5B%22_source%22%2C%22_id%22%2C%22_type%22%2C%22_index%22%2C%22_score%22%5D 500 (Internal Server Error)
(anonymous) @ VM10731:1
(anonymous) @ commons.bundle.js?v=15347:37
sendReq @ commons.bundle.js?v=15347:37
serverRequest @ commons.bundle.js?v=15347:37
processQueue @ commons.bundle.js?v=15347:38
(anonymous) @ commons.bundle.js?v=15347:38
$eval @ commons.bundle.js?v=15347:39
$digest @ commons.bundle.js?v=15347:39
$apply @ commons.bundle.js?v=15347:39
(anonymous) @ commons.bundle.js?v=15347:39
completeOutstandingRequest @ commons.bundle.js?v=15347:36
(anonymous) @ commons.bundle.js?v=15347:36
commons.bundle.js?v=15347:38 Error: An internal server error occurred
    at kibana.bundle.js?v=15347:228
    at processQueue (commons.bundle.js?v=15347:38)
    at commons.bundle.js?v=15347:38
    at Scope.$eval (commons.bundle.js?v=15347:39)
    at Scope.$digest (commons.bundle.js?v=15347:39)
    at Scope.$apply (commons.bundle.js?v=15347:39)
    at done (commons.bundle.js?v=15347:37)
    at completeRequest (commons.bundle.js?v=15347:37)
    at XMLHttpRequest.xhr.onload (commons.bundle.js?v=15347:37)
(anonymous) @ commons.bundle.js?v=15347:38
(anonymous) @ commons.bundle.js?v=15347:37
processQueue @ commons.bundle.js?v=15347:38
(anonymous) @ commons.bundle.js?v=15347:38
$eval @ commons.bundle.js?v=15347:39
$digest @ commons.bundle.js?v=15347:39
$apply @ commons.bundle.js?v=15347:39
done @ commons.bundle.js?v=15347:37
completeRequest @ commons.bundle.js?v=15347:37
xhr.onload @ commons.bundle.js?v=15347:37

I know the cross cluster config is OK and data:makelogs-* works fine for the elastic super user.

Provide logs and/or server output (if relevant):

The 2 roles I created:

  "makelogs_reader": {
    "cluster": [],
    "indices": [
      {
        "names": [
          "makelogs-*",
          "data:makelogs-*",
          "*:makelogs-*",
          "local:makelogs-*"
        ],
        "privileges": [
          "read",
          "view_index_metadata",
          "read_cross_cluster"
        ],
        "field_security": {
          "grant": [
            "*"
          ]
        }
      }
    ],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  },
  "kibana_ccs": {
    "cluster": [],
    "indices": [
      {
        "names": [
          ".kibana*"
        ],
        "privileges": [
          "manage",
          "create",
          "index",
          "delete",
          "read_cross_cluster"
        ],
        "field_security": {
          "grant": [
            "*"
          ]
        }
      }
    ],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }

Standardize our configuration parameter docs to indicate ECE/Elasticsearch Service vs. on-prem support

From @jakommo:

I’ve seen many issues where ECE customers tried to apply on-prem configs, because this was the first search result that came up on google. I was wondering if it would make sense to have both (ECE and on-prem) instructions on the same page, or at least add a note to on-prem docs with a link to the ECE config for this feature.

We probably don't want to merge these docs as they are today, but we should track the issue of how we can improve the documentation of configuration parameters to indicate what they apply to, regardless of your entry point into the docs.

Among the issues that end up confusing customers today:

• We whitelist parameters for Elasticsearch Service and blacklist parameters for ECE. These are documented only in our cloud docs, but not, say, in our ES or Kibana docs. If you start in our cloud docs, you'll be mostly fine, although we don't document the blacklisted parameters exhaustively today. These docs to link to the ES or Kibana docs.

• People googling for content can easily end up on the ES or Kibana docs looking at a configuration parameter, with no indication that the parameter is not supported for our cloud products. They try it and

One proposal might be to standardize how we document configuration parameters and add a field that indicates whether or not the parameter is supported on-prem, in ECE, or on the Elasticsearch Service.

Source: https://docs.google.com/document/d/1iUZGa1K-wM5Fu4_Ho3Wep5MQyG-d47XJWve2oGcRyak/edit#bookmark=id.2f62wskyv2rk

Clarify compatibility in upgrading stack document

When doing upgrade testing from 5.6.14 to 6.6.0 using this doc:
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html

When ES/Kibana are upgraded to 6.6.0 and Logstash is still on 5.6.14. I was confused by the below statement, when I went to check monitoring for Logstash and it did not appear.

Beats and Logstash 5.6 are compatible with Elasticsearch 6.6.0 to give you flexibility in scheduling the upgrade

@ycombinator pointed me to this matrix to verify monitoring 5.6 is not compatible with 6.6.0

https://www.elastic.co/support/matrix#matrix_compatibility

Maybe we can clarify a bit in the upgrading elastic stack document too about that.

Confusion between elasticsearch and kibana SSL settings

I wasn't sure which repo to use for this issue, but I think this is probably the best fit, even if the fix ends up elsewhere

I've run into 2 situations recently where someone tried to us ES stylexpack.* settings to configure SSL in Kibana.
One is on discuss, the other was on private communications.

I'm not sure exactly where the issue lies, whether the docs for setting up TLS is Elasticsearch need to have a link to the Kibana docs, or whether the Kibana docs are hard to find, but 2 cases in as many weeks (that I've seen) implies that we should do something to clarify the setup process.

Update Logs + Infrastructure docs to specify shipper compatibility and index customization

If you look at the forums, as well as support cases, shipper compatibility with the new Infrastructure and Logs UI is one of the most frequent questions around getting started.
https://discuss.elastic.co/c/infrastructure
https://discuss.elastic.co/c/logs

Looking at our documentation, we are missing sufficient detail:
https://www.elastic.co/guide/en/infrastructure/guide/current/install-infrastructure-monitoring.html

At a minimum, I think we should add information around:
1. The fact that shippers of versions 6.5 and above are require
2. That users can customize in kibana.yml which index(es) this Logs and Infrastructure UIs looks at.

cc @alvarolobato @makwarth

Create a Getting Started with the Elastic Stack guide

We already have something like this in the Beats Platform Reference. With a bit of massaging, we could easily create something for the stack.

7.0 stack upgrade docs feedback

Context

As @bleskes coordinates testers for the 7.0 stack upgrade process, we'll gather feedback on the docs and record it here.

• The page we're using to guide us: https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html
• Notes for testers: https://docs.google.com/spreadsheets/d/1LtmjfCgtohS_H6fMk3fSKh3tmEqyjioZbFk2giFDeQA/edit#gid=590440521

Update upgrade docs for 6.0.0

Let's use this issue to organize the effort to update the upgrade docs for 6.0.0.

The first step is for everyone to take a look at the existing upgrade docs and identify what's missing or needs to be updated:

https://www.elastic.co/guide/en/elastic-stack/current/index.html

Last time around, we decided to point to the docs for each component for detailed instructions. Now is the time to decide if we want to continue with that approach.

• Update stack docs
  • Restructure content - break it up into smaller sections.
  • How to do a rolling upgrade from 5.x to 6.x
• Elasticsearch & X-Pack install and upgrades
• Kibana & X-Pack install and upgrades
• Logstash & X-Pack install and upgrades
• Beats install and upgrades

Document manual reindexing of .kibana for users without X-Pack

Parent issue: #15

[DOCS] Update machine learning tutorial with refresh rate

Related to elastic/kibana#21328

The machine learning getting started tutorial should be updated to reflect the new refresh rate options in the Job Management, Anomaly Explorer, and Single Metric Viewer pages.

Some of the screenshots in the Kibana Reference might need to be updated too.

Document Firewell exception requirements for the stack

This request came to us from Shawn Clink ([email protected]):

It would be helpful if the firewall exception requirements for all the stack were listed in the setup docs (like the kibana configuration) located here:

https://www.elastic.co/guide/en/kibana/current/settings.html

Make solution docs easier to discover by adding links

This request comes from @tbragin:

Elastic Stack documentation: https://www.elastic.co/guide/en/elastic-stack/current/index.html
Currently these pages are at the top of all the other content. None of the books there lead to solution-oriented content.

Recommendation: Add pointers from Installing and Getting Started guides for Elastic Stack to next steps of setting up solutions.

Add clould-centric docs about getting started wtih the stack

Right now, the stack getting started docs show how to install and run the OSS components of the stack. For users who are interested in cloud, though, the setup is much simpler. They can eliminate the first two parts of the setup (installing ES and Kibana) and simply set up a cloud account. Then when they configure Beats, they set the cloud.id and cloud.auth config options, and are quickly able to see results (with minimal effort).

The stack getting started guide does mention that there is a hosted Elastic Service available on cloud, but I'm wondering if a full tutorial that highlights the simplicity of cloud might be worthwhile. The new tutorial would be very similar to the full stack (OSS) tutorial, except:

• The sections about installing ES and Kibana would be replaced with some simple steps about activating a trial account.
• The info about setting up Beats would be shared across the two tutorials, with some of the config steps conditionally coded to include cloud-specific config info.
• The Logstash section might go away completely (TBD).

Having a tutorial at the same level as the stack getting started (with a similar layout) raises the visibility of cloud and shows how easy it is to get started quickly. This is also a quick win because most of the content can be shared/reused with the OSS version of the tutorial.

@nrichers @lcawl Opening this as an item for discussion. If you think this is a good idea, we talk about ownership. Probably makes the most sense for Lisa (or me, if she's too busy) to own the overall tutorial and for Nik to advise on the contents of the cloud part. WDYT?

Improve linking between Infrastructure Monitoring docs and Kibana docs

The topic about getting started with the Infrastructure UI needs to have a pointer to the Kibana documentation about UI settings:

https://www.elastic.co/guide/en/kibana/master/infrastructure-ui-settings-kb.html

In general, make sure there are sufficient pointers between the topics.

Upgrade Guide - Incorrect upgrade path

to reproduce:

• First off, we need some information about your stack.
  • 5.0-5.5
  • no Cloud
• Do you use X-Pack?
  • yes
• Which X-Pack features do you use?
  • Security
• Transport Layer Security (TLS) is required as of 6.0
  • no
• Do you have any indices that were created in 2.x?
  • no
• Upgrading from 5.x to 6.3
  • Upgrade to 5.6 first

Which returns:

Here you go!
These are the steps you need to take to upgrade to 6.3.
Upgrade to 5.6
Back up your data.
Use the Elasticsearch Migration plugin to check for upgrade issues.
Address any 5.x breaking changes that affect your applications:
Elasticsearch breaking changes
Upgrade one node at a time to Elasticsearch 5.6.
Install Elasticsearch 5.6.
Install X-Pack.
Restart the node.

Prepare to upgrade to 6.3
Back up your data.
Address any 6.0 breaking changes that affect your applications:
Elasticsearch breaking changes
If you're using the default changeme password, change your password.
Check the Elasticsearch deprecation log.
Upgrade to 6.3
Upgrade Elasticsearch to 6.3.
You can perform a rolling upgrade to Elasticsearch 6.3.
Remove the X-Pack plugin before restarting Elasticsearch.
Run bin/elasticsearch-plugin remove x-pack.

However, 6.x requires TLS
The path is therefore missing to enable TLS and the full cluster restart that comes with it.

Instead of the above, the upgrade to 6.x should follow the following path (taken from selecting "jump straing to 6.3" in the last question). In fact, going to 5.6 does not bring any benefits in this case and we should consider removing that question when TLS is not enabled. 5.6 is only required if you plan to roll to 6.x, which is not possible in this case:

Perform a Full Cluster Restart Upgrade to 6.3
Stop sending data to your cluster.
Shut down your cluster and install Elasticsearch 6.3 on all nodes.
Remove the X-Pack plugin before restarting Elasticsearch.
Run bin/elasticsearch-plugin remove x-pack.
Enable Transport Layer Security (TLS) to secure cluster communications.
TLS is required as of 6.0 to prevent unidentified nodes from joining a cluster and encrypt inter-node communications.
Restart your Elasticsearch cluster.
Create a temporary super user and upgrade the internal .security index.
Delete the temporary super user.

[ML] Inconsistent categorization_filters content for ML jobs in Kibana

Original comment by @lcawl:

On 5.4 and 5.5 builds from s3, there is disparity with respect to the categorization_filters content.

For example, when I run the following API in the Dev Tools tab in Kibana:

PUT _xpack/ml/anomaly_detectors/it_ops_logs
{
  "description": "IT Ops application logs",
  "analysis_config": {
    "categorization_field_name": "message",
    "bucket_span":"30m",
    "detectors": [
      {
      "function": "count",
      "by_field_name": "mlcategory",
      "detector_description": "Unusual message counts"
    }],
    "categorization_filters":"\\[statement:.*\\]"
  },
  "analysis_limits": {
      "categorization_examples_limit": 5
  },
  "data_description": {
    "time_field": "time",
    "time_format": "epoch_ms"
  }
}

It returns the following:

...
"analysis_config": {
    "bucket_span": "30m",
    "categorization_field_name": "message",
    "categorization_filters": [
      """\[statement:.*\]"""
    ]
...

Ditto when I run GET _xpack/ml/anomaly_detectors/it_ops_logs from the Dev Tools tab.

If I run it from the command line, however, I get the following:

curl -u elastic:changeme -XGET 'localhost:9200/_xpack/ml/anomaly_detectors/it_ops_logs'
{"count":1,"jobs":[{"job_id":"it_ops_logs","job_type":"anomaly_detector","job_version":"5.5.0","description":"IT Ops application logs","create_time":1497295377972,"analysis_config":{"bucket_span":"30m","categorization_field_name":"message","categorization_filters":["\\[statement:.*\\]"],"detectors":[{"detector_description":"Unusual message counts","function":"count","by_field_name":"mlcategory","detector_rules":[],"detector_index":0}],"influencers":[]},"analysis_limits":{"categorization_examples_limit":5},"data_description":{"time_field":"time","time_format":"epoch_ms"},"model_snapshot_retention_days":1,"results_index_name":"shared"}]}Lisas-MBP:~ lcawley$ 

Note that the categorization_filters content matches what I specified in the PUT command in this case.

The "categorization_filters":"""[statement:.*]""" does not actually seem to be invalid (I can successfully create a job in the Dev Tools with that syntax, it's just a bit confusing that it's returning a different syntax in Dev Tools vs command line.

For interest's sake, to get this to work in the advanced job wizard I must use the following syntax:

.... the categorization_filters property then has the desired format in the Edit JSON tab:

Update Kibana installation instructions for Debian and RPM

See #198
The Getting started guide contains Kibana installation instructions here:
https://www.elastic.co/guide/en/elastic-stack-get-started/master/get-started-elastic-stack.html#install-kibana

The "deb or rpm" instructions use tar.gz files. They ought to be updated to match the specific instructions in https://www.elastic.co/guide/en/kibana/master/deb.html#install-deb and https://www.elastic.co/guide/en/kibana/master/rpm.html#install-rpm

Ideally the instructions can be re-used so they stay in synch

role_mapping.yml breaks with upgrade to ES >= 6.3 (and removal of X-pack)

We just spent a significant amount of time figuring out why Active Directory / LDAP stops working after upgrading from ES 5.6 to ES 6.4.

The reason is:

• in ES 6.2 (with x-pack), the default path for role_mapping.yml is ES_PATH_CONF/x-pack/role_mapping.yml (see https://www.elastic.co/guide/en/elasticsearch/reference/6.2/security-settings.html)
• in ES 6.3 (no x-pack), the default path for role_mapping.yml is ES_PATH_CONF/role_mapping.yml (see https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html)

We did not find any explicit documentation of this change, even though it did actually break functionality.
In particular, also the Interactive Upgrade Guide should highlight these and similar required changes.

Differentiate machine learning job types

In 7.1, we need to be able to differentiate between "anomaly detector jobs" and "data frame analytics jobs" in the documentation.

NOTE: Page names should remain unchanged to ease switching between versions. (For example, after 7.1 is released we don't want someone who selects "show current version" from the 7.0 "Put Job" page to be told the page doesn't exist.)

• Create attributes in https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc that we can use in case the exact job types change: elastic/docs#688
• Update the existing content in the Stack Overview: https://www.elastic.co/guide/en/elastic-stack-overview/master/xpack-ml.html, https://www.elastic.co/guide/en/elastic-stack-overview/master/ml-troubleshooting.html, https://www.elastic.co/guide/en/elastic-stack-overview/master/ml-limitations.html. See elastic/elasticsearch#44888, #444
• Update the existing content in the Elasticsearch Reference: https://www.elastic.co/guide/en/elasticsearch/reference/master/ml-settings.html, https://www.elastic.co/guide/en/elasticsearch/reference/master/ml-apis.html, https://www.elastic.co/guide/en/elasticsearch/reference/master/api-definitions.html
  See elastic/elasticsearch#44839, elastic/elasticsearch#44866, elastic/elasticsearch#44883, elastic/elasticsearch#44986, ...
• Update the existing content in the Kibana Guide: https://www.elastic.co/guide/en/kibana/master/xpack-ml.html
  See elastic/kibana#41965
• Update the existing terminology in the Glossary: https://www.elastic.co/guide/en/elastic-stack-glossary/current/index.html See #238
• Update the existing content in the Java REST client reference:
  https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/_machine_learning_apis.html: See elastic/elasticsearch#44839, elastic/elasticsearch#44866, elastic/elasticsearch#44883, elastic/elasticsearch#44986 ...

Create solution-based tutorials

We need more cross-stack documentation that addresses specific use cases. We have a lot of info on the website in blogs that describe specific use cases, but that content gets stale over time. We need to pull some of that content into the core docs (or create new tutorials) so the information stays more current.

Installing the Elastic stack is broken on 6.0.0-beta2

The docs are broken, showing this:

The content looks right to me:

[[install-order-elastic-stack]]
=== Installation Order

Install the Elastic Stack products you want to use in the following order:

. Elasticsearch ({esref}/install-elasticsearch.html[install instructions])
. {xpack} for Elasticsearch ({esref}/install-elasticsearch.html[install instructions])
. Kibana ({esref}/install-elasticsearch.html[install])
. {xpack} for Kibana ({esref}/install-elasticsearch.html[install instructions])
. Logstash ({esref}/install-elasticsearch.html[install])
. {xpack} for Logstash ({esref}/install-elasticsearch.html[install instructions])
. Beats ({esref}/install-elasticsearch.html[install instructions])
. Elasticsearch Hadoop ({esref}/install-elasticsearch.html[install instructions])

Installing in this order ensures that the components each product depends
on are in place.

Therefore I am passing this on to an expert.

SQL JDBC Plugin Example Typo

Example at the bottom of this page, shows text as:

jdbc:es://http://server:3456/timezone=UTC&page.size=250

Correct text is:

jdbc:es://http://server:3456?timezone=UTC&page.size=250

Expand the intro docs

It'd be fantastic if we had an intro to the Elastic Stack and it's various use cases under the Overview page. I'm thinking of 3 high level areas we can start with;

• IT Analytics - System logs and metrics, security
• Business Analytics - App logs and metrics
• Search - Traditionally text only

Using generic use cases for each, we can then break the stack down and say "for X, install A", "don't install logstash for search" etc, etc.

RFC: Test environments for 6.0 upgrade

Goals and Challenges

One of the 6.0 upgrade team's major deliverables is a set of test environments that internal testers (who, crucially, are not on the upgrade team) can use to vet our documentation. We'd like that test infrastructure to reflect reality as much as possible. Given, however, the sheer configurability of the Elastic Stack, and the number of operating systems and JVMs we support, it's impossible to be exhaustive without incurring a combinatorial explosion of different configurations. So, we should identify the variables that are most likely to affect the upgrade process, and make sure that our test environments account for them. For example, whether a user is using shard filtering for cold indices probably won't affect the steps they take to upgrade the stack, but whether they're using X-Pack Security certainly will.

Variables

Here are some variables that come to mind:

• The ingest products being used
  • Should we assume Beats -> Logstash -> ES?
  • Should we test other architectures, such as Beats -> ES?
• The version being upgraded from
  • 5.6 -> 6.0 implies a rolling restart of Elasticsearch. There are, however, certainly going to be users who opt to do a full cluster restart from a pre-5.6 version to 6.0.
• If X-Pack is enabled:
  • Is Security enabled?
    • If so, is TLS between nodes enabled?
  • Is Kibana delpoyed?
    • If not, user's will need to use the X-Pack upgrade API directly.
  • Note: I think it's safe to simply include the rest of X-Pack, as upgrading the various internal indices is purely an additive step; if you're not running Alerting or Monitoring, you have less to worry about, and less for us to test.
• Host OS
  • We should probably test on at least one Debian-derived Linux distro and one RHEL-derived Linux distro.
  • What about Windows?

Accounting for these alone results in a large number of scenarios. We'd have, for at least two or three operating systems (RHEL-derived, Debian-derived, Windows), and potentially two different ingest architectures:

• An environment without X-Pack (OSS only)
• An environment with X-Pack Security, but without TLS between nodes
• An environment with X-Pack Security, including TLS between nodes

Like I was saying: combinatorial explosion.

Proposed environments

So, we'll need to pare this down in a sane manner. I think it's reasonable to build something like the following for our (to-be formed) group of testers:

Running 5.6:

1. A complete stack (Beats, LS, ES, Kibana) including X-Pack with TLS between nodes. (see #17 (comment))
2. Same as above, but without TLS between nodes.
3. An OSS-only, complete stack.

Running 5.5 (or some other 5.x version below 5.6):

1. A complete stack (Beats, LS, ES, Kibana), including X-Pack with TLS between nodes.
2. Same as above, but without TLS between nodes.
3. An OSS-only, complete stack. (#17 (comment))

This is clearly not exhaustive, but it's a starting point that covers the more complex scenarios well. I could see arguments for including more, however.

Notes and questions

• Which beats should we run? To what extent does their common libbeat foundation guarantee like behavior with respect to upgrade logic?
• In order to cut down on the number of ingestion architectures we test, perhaps it would make sense to configure beats to output data to both Logstash and Elasticsearch. Alternatively, this could be a good reason to use multiple beats. We could have filebeat write to Logstash, and metricbeat directly to ES, for example.
• I think each environment should have a three-node ES cluster. A single-node cluster is too trivial to upgrade, and a two-node cluster is basically a bad practice. Three seems like a good minimum.
• My leaning is that it's probably overkill to spin up one of each of the proposed environments for each operating system we intend to test. Could we get away with simply using a RHEL-derived distro for half of the proposed environments, and a Debian-derived distro for the other half?
• I've gone back and forth on whether we should build an OSS-only environment, and I'm curious to hear everyone's thoughts. The argument against it is that upgrading an environment with X-Pack is basically a superset of upgrading one without X-Pack. The argument for it is that the lack of access to the Upgrade Assistant UI could make the process of upgrading a bit trickier. I think that's a critical enough distinction to warrant it. Additionally, it's the simplest scenario, and encountering problems there could be a canary in the coal mine.
• Should we build a Windows-specific environment?

Document remote monitoring cluster version requirement for upgrades?

We have a note here on version dependency, i.e. they can't do an export from a 6.x cluster to a 5.x monitoring cluster during upgrades: https://www.elastic.co/guide/en/x-pack/current/monitoring-production.html#monitoring-production

The problem is that this note is in the "setup" section of the monitoring guide and users are unlikely going to revisit this page as part of the upgrade. It will be helpful if we can call this out (or cross link) as part of the stack upgrade documentation.

[DOCS] Create getting started information for container deployments

We would like to make it easier for customers to deploy the Elastic Stack on Docker and Kubernetes. To that end, we'd like to create tutorials similar to https://www.elastic.co/guide/en/elastic-stack-overview/master/get-started-elastic-stack.html for those environments.

Document upgrade assistant

Parent issue #15

Document upgrade order for tribe and CCS

This is one area that is not covered today in our docs today.

Even though tribe is deprecated, it is still in the product on 6.x. There will be questions around upgrading tribe node implementations for those who are not ready to switch to CCS yet. Afaik, they need to upgrade the tribe to 6.0 first before the downstream clusters because if the tribe remains on 5.x, it will not be able to join any downstream clusters that have indices created on 6.0. For example, can they upgrade tribe to 6.0 first and then do rolling restarts of downstream 5.6 clusters to 6.0? This is something we will have to sync up with dev on our recommendations. This will probably depend on the outcome of #17, but I do want to make sure that the results are documented :)

Note that some customers may resist switching to CCS right away because, eg.

• CCS will require that they make changes to their application (not an issue if they use Kibana, but if they don't, then app changes will be needed to use the new CCS syntax).
• Some customers are more comfortable waiting for a new GA-ed feature to bake in before deploying to production.
• Or that they need to upgrade to 6.0 to take advantage of some other new features, and prefer to tackle the tribe node piece later

So if we decide that we will not be testing/supporting tribe for rolling upgrades because the tribe node is deprecated, we will just have to document it to set the right expectations upfront.

Even for CCS, there will be questions on upgrade ordering (upgrade CCS first? Downstream clusters?).

Add APM Server to interactive upgrade guide

Add APM Server to the list of selectable stack components and show it in the list of steps if it is selected.

Clarify machine learning upgrade steps

Per elastic/elasticsearch#37942 (comment)

We should probably expand the section in the upgrade docs to match current reality more closely....

index.asciidoc product listing got added to the end of the upgrading-stack.asciidoc

Seems like when the docs got built, it picked up the product listing from the stack-docs/index.asciidoc and tagged it on to the end of the stack-docs/upgrading-stack.asciidoc page

Improve language around backups before an upgrade

In the "Preparing for an upgrade section" of this page, we mention "Back up your data. You cannot roll back to an earlier version unless you have a backup of your data. For information about creating snapshots, see Snapshot and Restore."

Due to the incremental nature of snapshots, we may want to mention that a new repo must be created after upgrading indexes in 5.6 (and before the upgrade is performed). Otherwise, the user will need to perform a snapshot restore on a version of ES < 6.0.

One possibility for changing the language:

"Back up your data. You cannot roll back to an earlier version unless you have a backup of your data. Note: snapshots are incremental, snapshot repos created in earlier versions of Elasticsearch may not be restorable in newer versions of Elasticsearch. For this reason, it is important to upgrade all indexes in 5.6, create a new snapshot repo, perform a snapshot, and then upgrade Elasticsearch. For information about creating snapshots, see Snapshot and Restore."

CC @eskibars

Is the description「Elasticsearch 8.0.0-alpha1」correct?

Is the description「Elasticsearch 8.0.0-alpha1」correct?
master branch So is there no problem?

https://www.elastic.co/guide/en/elastic-stack/master/upgrading-elastic-stack.html

monitoring cluster page enhancement.

https://www.elastic.co/guide/en/elastic-stack-overview/current/monitoring-production.html

will be better if you mention adding it to elasticsearch.yml

xpack.monitoring.collection.enabled: true

With current instruction found on that page, it won't work as xpack.monitoring.collection.enabled: true is not set.

Add docs for integrating with Google "GKE On-Prem"

Release notes should point to release blogs

Right now, our release notes contain a laundry list of changes. They don't do a very good job of highlighting the major changes that were added for each release.

As a quick fix, the release notes for each product should point to the blog posts that contain the release highlights.

We should do this for all 6.x versions, and possibly 5.6.

The links need to be added to the release notes in the appropriate repo. Use this meta issue to track completion of the work:

• Elasticsearch
• Kibana
• Logstash
• Beats

Upgrade guide from 5.0-5.5 mentions Upgrade Assistant

Following the interactive upgrade guide https://www.elastic.co/products/upgrade_guide

Repro:

-What version of Elasticsearch are you currently running?
5.0-5.5
-Are you running in Elastic Cloud?
No
-Which products are you using with Elasticsearch?
Kibana
-Do you use X-Pack?
Yes
... next steps irrelevant
-Jump straight to 6.0

The resulting steps now mention using the Upgrade Assistant which is a Kibana 5.6 feature. Nowhere in the process is deploying a Kibana 5.6 node, or upgrading Kibana to 5.6 is mentioned.

In this case I believe the xpack migration API should be referenced instead of the upgrade assistant, with steps to upgrade the .kibana and .security indices after installing Elasticsearch 6.0+ (which means starting Elasticsearch 6 with the old .kibana and .security indices? That's how we seem to mention in the docs - haven't tested this path myself yet)

[DOCS] Supporting Upgrades from 6.x to 7.x

This issue is meant to give a high level overview of all the work needed to update the documentation for users upgrading from 6.x to 7.x.

Installation and Upgrade Guide (https://www.elastic.co/guide/en/elastic-stack/7.0/index.html):

• Update upgrade steps. See #242
• Verify list of "notable" breaking changes.
• Address feedback in #226
• Add a topic that discusses the benefits of upgrading. See #287
• Provide information about how to fix a “broken” upgrade if you upgrade from a pre-6.7 version and do not use the Migration Assistant (or should this be in the Elasticsearch Reference?)

Elasticsearch Reference:

• Update upgrade steps in https://www.elastic.co/guide/en/elasticsearch/reference/7.0/setup-upgrade.html. See elastic/elasticsearch#39944
• Update machine learning considerations. See elastic/elasticsearch#40006
• Deprecate migration APIs. See elastic/elasticsearch#40075

Kibana Guide:

• Verify upgrade steps in https://www.elastic.co/guide/en/kibana/7.0/upgrade.html). See elastic/kibana#33543 and elastic/kibana#34121
• Verify Upgrade Assistant screenshots and details (https://www.elastic.co/guide/en/kibana/6.7/upgrade-assistant.html)

Logstash Reference:

• Verify upgrade steps in https://www.elastic.co/guide/en/logstash/7.0/upgrading-logstash.html See elastic/logstash#10621
• Update Breaking Changes docs https://www.elastic.co/guide/en/logstash/7.0/breaking-changes.html See elastic/logstash#10632

Beats Platform Reference:

• Verify upgrade steps in https://www.elastic.co/guide/en/beats/libbeat/7.0/upgrading.html See elastic/beats#11378
• Add release highlights: elastic/beats#11621

Cloud:

• Author ESS- and ECE- specific upgrade docs. See https://github.com/elastic/cloud/issues/27673

APM?

• 7.0 Breaking changes, Release highlights, Migration assistant, ECS changes elastic/apm-server#1940

Clarify File-based grants are not visible with API calls

The following is true of all versions: The APIs (user, role, role mapping) only work with objects that were defined via the API.

Objects (users, roles, mappings) defined in files need to be managed via the files (or the users CLI tool which uses the files).

API exploration calls (and consecutively, Kibana) will not report roles and mappings defined in files.

Example, where role production_user is defined in roles.yml and user "test" is a file realm user created with the users CLI tool.

curl -X GET -u test:password http://localhost:9200/_xpack/security/_authenticate
{"username":"test","roles":["production_user"],"full_name":null,"email":null,"metadata":{},"enabled":true}[root@localhost x-pack]# 

curl -X GET -u test:password http://localhost:9200/_xpack/security/role/production_user
{}

curl -X GET -u elastic:password http://localhost:9200/_xpack/security/user/test
{}

Some places to update docs to reflect this behavior:

https://www.elastic.co/guide/en/elastic-stack-overview/6.6/defining-roles.html#roles-management-file

https://www.elastic.co/guide/en/elastic-stack-overview/6.6/mapping-roles.html#mapping-roles-file

{ref} attribute not defined?

Looking at the current published version of the terminology list in the glossary (https://www.elastic.co/guide/en/elastic-stack-glossary/current/terms.html), there are multiple instances of incomplete links being rendered:

For example:

This value can be overridden by specifying a routing value at index time, or a /mapping-routing-field.html[routing field] in the mapping.

After examining the source, it seems that the {ref} attribute is not defined, which is breaking the declaration of the links.

Improve getting started with stack content

A couple of areas where we can improve the getting started with the stack content:

• Make it clear that Logstash isn't always required. In fact, it's often not. I'm not sure this is obvious with the way the tutorial is currently set up.
• Show how to start Logstash with config reload enabled. It's easy to do and means that users won't have to stop and restart Logstash to pick up the config file changes.

Elasticsearch Documentation: "_doc" or "doc"

A user contacted us through the "Contact Us" form and wrote:

In the documentation it says to use “_doc” for a query in the following page:

https://www.elastic.co/guide/en/elasticsearch/reference/6.3/docs-update.html

I’ve been fiddling around with a 6.3 installation, and in my experiences only doc works. The documentation changed in 6.1 to 6.2 to specify _doc.
<<<

[DOCS] Expand details about Data Visualizer

Original comment by @lcawl:

Per LINK REDACTED

"... if we want to fully document the functionality in the Data Visualizer, items that are not covered are:

• Use of the query bar to narrow the data being visualized
• Use of the filter boxes in the Metric and Fields section to filter the fields being displayed
• Use of the field types dropdown in the Fields section to view one field type at a time e.g. view only IP type fields
• Further discussion on the metric distribution chart - shows where the values in the data are clustered
• boolean field example
• IP field example
• geo_point field example
• text field example (up to 10 example values of this field are displayed)
• checkboxes for showing empty fields i.e. those which don't appear in any documents
• Caution when altering the 'sample' size field as it will lead to increased query run times and the load on the cluster
• If the fields match recognized configurations, links to the specialized wizards will appear on the right ..."

Some of this might be appropriate for help inside the UI, but it should be considered for inclusion in the Kibana User Guide > Machine Learning nonetheless.

Create documentation about deploying the Elastic Stack

Would be nice to provide more information about deploying the stack in a production environment. Right now, we have most of that information buried in the guides for each product, which makes a high level understanding of deployment options difficult.