jaegertracing / jaeger-kubernetes Goto Github PK
View Code? Open in Web Editor NEWSupport for deploying Jaeger into Kubernetes
Home Page: https://jaegertracing.io/
License: Apache License 2.0
jaeger-kubernetes's People
Contributors
Stargazers
Watchers
Forkers
pavolloffay objectiser jpkrohling kameshsampath mikeln jukylin dogild liqin323 xiaosefengyun sjanulonoks jorwalk tteat nov1n adrianosantos geoand maximmold manpreetsingh6 dicarlo2 yanyanzeng franklinharry b0go shaunokeefe frankgreco eundoosong zdicesare aergonus zeroingsu heidsoft pikbot etsangsplk saikiranchalla1 newbare ray891 bobhenkel dgurindapalli mayur-patil huhenry ajbouh iaskebba bangau1 luizportela santhoshkumarvs karanmagdani-zz andy2046cynthia vorstella mtritschler paradyme-management vhung92 thomasfrandsen michaelwan ldelgadop44 janstenpickle jufab caohm bradleyfalk jefferyq longdajiangjun zhengweisk fanchuanbing allanperes kmfischer3 muchl jamiejoin cloud-land tapincugur julisman bharathkumar1093 esukram allanpedroni lemonqc caoy83 sampson2016 billsking vipinvkmenon ming-ddtechcg kirinse ajaykhetan jgehrcke ch-erikye dean-coakley weijen autonomic-ai vikrambe vinta-iaas-labs subnetmarco yooneo rajputvishwas denys1yermakov prashanthga laashub-soa muqp sabinthomas humayun-rashid lancger prune998 umapathireddy nonfu jiangxuelei siddhuoo7 luogreenjaeger-kubernetes's Issues
need a docker image for jaeger-collector 0.6.0
Hi there
i saw that both jaeger-collector and jaeger-query have released 0.6.0, which supports a small but critical feature -- supports username and password input when connecting to cassandra.
It seems that, the docker image only have 0.5.0.
I am wondering if it is possible to have image 0.6.0 published.
maybe we can contribute back by making the docker image for you guys ?
Best
Dan
How can jaeger-client talk to agent running as Kubernetes service?
Its more of a question than an issue. I am new to Kubernetes and trying to understand how does this deployment work? given the fact that "jaeger-client" expects agent to be running on localhost?
So if I have a web service running as a pod, how will jaeger client in that service will connect to jaeger agent running as part of jaeger-all-in-one service?
Stand-alone Agent sidecar by NodePort
I want to expose the agent port, so that other external applications can access and write jaegertracing log, can?
Enable tests for production template
Currently tests for production template are disabled because kubernets-client used in arquillian-cube does not support StatefulSet.
PR to k8s-client with StatefulSet has been merged fabric8io/kubernetes-client#814. However, we need this change to land in arq-cube arquillian/arquillian-cube#755.
Error while connecting to external elasticsearch instance
I'm getting "msg":"Failed to init storage builder","error":"health check timeout: no Elasticsearch node available message when I'm trying to connect to external elasticsearch instance. Database exists.
# curl http://elasticsearch.default.svc.cluster.local:9200 {
"name" : "elasticsearch-data-1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "vCApM7DoRoC6D7q6U8WI7Q",
"version" : {
"number" : "5.5.2",
"build_hash" : "b2f0c09",
"build_date" : "2017-08-14T12:33:14.154Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}
deployment.yml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jaeger-query
labels:
app: jaeger
jaeger-infra: query-deployment
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: jaeger
jaeger-infra: query-pod
spec:
containers:
- image: jaegertracing/jaeger-query:latest
name: jaeger-query
ports:
- containerPort: 16686
protocol: TCP
command:
- "/go/bin/query-linux"
- "--span-storage.type=elasticsearch"
- "--es.server-urls=http://elasticsearch.default.svc.cluster.local:9200"
- "--es.num-shards=3"
- "--query.static-files=/go/jaeger-ui/"
readinessProbe:
httpGet:
path: "/"
port: 16686
restartPolicy: Always
Full error output is:
{"level":"fatal","ts":1508329287.1442423,"caller":"query/main.go:92","msg":"Failed to init storage builder","error":"health check timeout: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available\ngithub.com/uber/jaeger/vendor/github.com/olivere/elastic.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/olivere/elastic/client.go:84\ngithub.com/uber/jaeger/pkg/es/config.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/es/config/config.go:102\ngithub.com/uber/jaeger/cmd/builder.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/builder/doc.go:20\nmain.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:139\nruntime.main\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:172\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086\nhealth check timeout\ngithub.com/uber/jaeger/vendor/github.com/olivere/elastic.(*Client).startupHealthcheck\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/olivere/elastic/client.go:1067\ngithub.com/uber/jaeger/vendor/github.com/olivere/elastic.NewClient\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/olivere/elastic/client.go:240\ngithub.com/uber/jaeger/pkg/es/config.(*Configuration).NewClient\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/es/config/config.go:50\ngithub.com/uber/jaeger/cmd/query/app/builder.(*StorageBuilder).newESBuilder\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/app/builder/elastic_search.go:24\ngithub.com/uber/jaeger/cmd/query/app/builder.NewStorageBuilder\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/app/builder/storage.go:70\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:90\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:135\nruntime.main\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:183\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:235\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:92\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:135"}
Documenting sender configuration
When trying out the template with a simple example service, I needed to use the following config to get the service reporting spans to the server:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: example-app
spec:
replicas: 3
template:
metadata:
labels:
app: example-app
spec:
containers:
- name: example-app
image: <YOUR IMAGE/VERSION>
ports:
- name: web
containerPort: 8080
env:
- name: JAEGER_SERVICE_NAME
value: <YOUR SERVICE NAME>
- name: JAEGER_AGENT_HOST
value: jaeger-all-in-one-agent
- name: JAEGER_SAMPLER_TYPE
value: const
- name: JAEGER_SAMPLER_PARAM
value: "1"This example uses the Jaeger TracerResolver, but the tracer could be directly configured using the same information - the main issue was setting the UDP sender's host name to jaeger-all-in-one-agent.
Wonder where best to record this info - should we add it to the README?
Collector/Query pods crashloopbackoff due to storage/keyspace issues
My company is currently in the process of implementing Jaeger solutions for our dev environment, our implementation of the cassandra backed helm chart is exhibiting an issue where our collector and query pods are in consistent CrashLoopBackoff state due to keyspace does not exist and `no connections were made when creating the session errors.
The cluster was created Friday 06/29/18 on GKE managed clusters. Based off pod ages it appears the minion hosting Cassandra was migrated 7/1/18, all the other jaeger related pods came back online as expected however Query and Collector went into CrashLoopBackoff with the following errors reported. The cassandra schema job completed successfully and we were able to run traces through the system prior to it entering this state.
Query pod logs:
{"level":"info","ts":1530557025.932619,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":16687,"status":"unavailable"}
2018/07/02 18:43:45 error: failed to connect to 10.88.233.5:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2018/07/02 18:43:46 error: failed to connect to 10.88.233.5:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1530557026.146703,"caller":"query/main.go:100","msg":"Failed to init storage factory","error":"no connections were made when creating the session","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/main.go:100\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:698\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:783\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:736\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/main.go:165\nruntime.main\n\t/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/proc.go:185"}Collector pod logs:
{"level":"info","ts":1530556753.660484,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
2018/07/02 18:39:13 error: failed to connect to 10.88.233.5:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2018/07/02 18:39:13 error: failed to connect to 10.88.233.5:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1530556753.8979948,"caller":"collector/main.go:95","msg":"Failed to init storage factory","error":"no connections were made when creating the session","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:95\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:698\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:783\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:736\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:178\nruntime.main\n\t/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/proc.go:185"}Cassandra pod logs:
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V
CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
INFO [main] 2018-07-02 00:23:56,208 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
INFO [main] 2018-07-02 00:23:56,573 Config.java:495 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.88.233.5; broadcast_rpc_address=10.88.233.5; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; client_encryption_options=<REDACTED>; cluster_name=jaeger; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@3e3047e6; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_materialized_views=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.88.233.5; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=tjb-jgr-cassandra-v6-1-cassandra-0.tjb-jgr-cassandra-v6-1-cassandra.default.svc.cluster.local,}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@37e547da; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO [main] 2018-07-02 00:23:56,574 DatabaseDescriptor.java:367 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2018-07-02 00:23:56,574 DatabaseDescriptor.java:425 - Global memtable on-heap threshold is enabled at 249MB
INFO [main] 2018-07-02 00:23:56,575 DatabaseDescriptor.java:429 - Global memtable off-heap threshold is enabled at 249MB
INFO [main] 2018-07-02 00:23:56,712 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO [main] 2018-07-02 00:23:56,713 DatabaseDescriptor.java:729 - Back-pressure is disabled with strategy org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}.
INFO [main] 2018-07-02 00:23:56,851 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
INFO [main] 2018-07-02 00:23:56,945 JMXServerUtils.java:246 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
INFO [main] 2018-07-02 00:23:56,953 CassandraDaemon.java:473 - Hostname: tjb-jgr-cassandra-v6-1-cassandra-0.tjb-jgr-cassandra-v6-1-cassandra.default.svc.cluster.local
INFO [main] 2018-07-02 00:23:56,956 CassandraDaemon.java:480 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_171
INFO [main] 2018-07-02 00:23:56,959 CassandraDaemon.java:481 - Heap size: 998.438MiB/998.438MiB
INFO [main] 2018-07-02 00:23:56,960 CassandraDaemon.java:486 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4411776(4308K) committed = 4456448(4352K) max = 251658240(245760K)
INFO [main] 2018-07-02 00:23:56,974 CassandraDaemon.java:486 - Metaspace Non-heap memory: init = 0(0K) used = 18583264(18147K) committed = 19136512(18688K) max = -1(-1K)
INFO [main] 2018-07-02 00:23:56,974 CassandraDaemon.java:486 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2241304(2188K) committed = 2359296(2304K) max = 1073741824(1048576K)
INFO [main] 2018-07-02 00:23:56,975 CassandraDaemon.java:486 - Par Eden Space Heap memory: init = 214827008(209792K) used = 111719168(109100K) committed = 214827008(209792K) max = 214827008(209792K)
INFO [main] 2018-07-02 00:23:56,975 CassandraDaemon.java:486 - Par Survivor Space Heap memory: init = 26804224(26176K) used = 0(0K) committed = 26804224(26176K) max = 26804224(26176K)
INFO [main] 2018-07-02 00:23:56,975 CassandraDaemon.java:486 - CMS Old Gen Heap memory: init = 805306368(786432K) used = 0(0K) committed = 805306368(786432K) max = 805306368(786432K)
INFO [main] 2018-07-02 00:23:56,976 CassandraDaemon.java:488 - Classpath: /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.9.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.13.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.13.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.2.2.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jstackjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.4.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/usr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.11.2.jar:/usr/share/cassandra/apache-cassandra-thrift-3.11.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar
INFO [main] 2018-07-02 00:23:56,979 CassandraDaemon.java:490 - JVM Arguments: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms1024M, -Xmx1024M, -Xmn256M, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]
WARN [main] 2018-07-02 00:23:57,044 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO [main] 2018-07-02 00:23:57,045 StartupChecks.java:140 - jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
WARN [main] 2018-07-02 00:23:57,046 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
INFO [main] 2018-07-02 00:23:57,048 SigarLibrary.java:44 - Initializing SIGAR library
INFO [main] 2018-07-02 00:23:57,060 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN [main] 2018-07-02 00:23:57,063 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
WARN [main] 2018-07-02 00:23:57,073 StartupChecks.java:332 - Directory /var/lib/cassandra/data doesn't exist
WARN [main] 2018-07-02 00:23:57,077 StartupChecks.java:332 - Directory /var/lib/cassandra/commitlog doesn't exist
WARN [main] 2018-07-02 00:23:57,078 StartupChecks.java:332 - Directory /var/lib/cassandra/saved_caches doesn't exist
WARN [main] 2018-07-02 00:23:57,079 StartupChecks.java:332 - Directory /var/lib/cassandra/hints doesn't exist
INFO [main] 2018-07-02 00:23:57,138 QueryProcessor.java:116 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO [main] 2018-07-02 00:23:57,677 ColumnFamilyStore.java:411 - Initializing system.IndexInfo
INFO [main] 2018-07-02 00:23:58,987 ColumnFamilyStore.java:411 - Initializing system.batches
INFO [main] 2018-07-02 00:23:59,001 ColumnFamilyStore.java:411 - Initializing system.paxos
INFO [main] 2018-07-02 00:23:59,028 ColumnFamilyStore.java:411 - Initializing system.local
INFO [main] 2018-07-02 00:23:59,034 ColumnFamilyStore.java:411 - Initializing system.peers
INFO [main] 2018-07-02 00:23:59,040 ColumnFamilyStore.java:411 - Initializing system.peer_events
INFO [main] 2018-07-02 00:23:59,045 ColumnFamilyStore.java:411 - Initializing system.range_xfers
INFO [main] 2018-07-02 00:23:59,051 ColumnFamilyStore.java:411 - Initializing system.compaction_history
INFO [main] 2018-07-02 00:23:59,056 ColumnFamilyStore.java:411 - Initializing system.sstable_activity
INFO [main] 2018-07-02 00:23:59,063 ColumnFamilyStore.java:411 - Initializing system.size_estimates
INFO [main] 2018-07-02 00:23:59,069 ColumnFamilyStore.java:411 - Initializing system.available_ranges
INFO [main] 2018-07-02 00:23:59,075 ColumnFamilyStore.java:411 - Initializing system.transferred_ranges
INFO [main] 2018-07-02 00:23:59,081 ColumnFamilyStore.java:411 - Initializing system.views_builds_in_progress
INFO [main] 2018-07-02 00:23:59,087 ColumnFamilyStore.java:411 - Initializing system.built_views
INFO [main] 2018-07-02 00:23:59,092 ColumnFamilyStore.java:411 - Initializing system.hints
INFO [main] 2018-07-02 00:23:59,099 ColumnFamilyStore.java:411 - Initializing system.batchlog
INFO [main] 2018-07-02 00:23:59,105 ColumnFamilyStore.java:411 - Initializing system.prepared_statements
INFO [main] 2018-07-02 00:23:59,111 ColumnFamilyStore.java:411 - Initializing system.schema_keyspaces
INFO [main] 2018-07-02 00:23:59,116 ColumnFamilyStore.java:411 - Initializing system.schema_columnfamilies
INFO [main] 2018-07-02 00:23:59,122 ColumnFamilyStore.java:411 - Initializing system.schema_columns
INFO [main] 2018-07-02 00:23:59,132 ColumnFamilyStore.java:411 - Initializing system.schema_triggers
INFO [main] 2018-07-02 00:23:59,141 ColumnFamilyStore.java:411 - Initializing system.schema_usertypes
INFO [main] 2018-07-02 00:23:59,151 ColumnFamilyStore.java:411 - Initializing system.schema_functions
INFO [main] 2018-07-02 00:23:59,161 ColumnFamilyStore.java:411 - Initializing system.schema_aggregates
INFO [main] 2018-07-02 00:23:59,163 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO [main] 2018-07-02 00:23:59,301 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO [main] 2018-07-02 00:23:59,335 ColumnFamilyStore.java:411 - Initializing system_schema.keyspaces
INFO [main] 2018-07-02 00:23:59,342 ColumnFamilyStore.java:411 - Initializing system_schema.tables
INFO [main] 2018-07-02 00:23:59,349 ColumnFamilyStore.java:411 - Initializing system_schema.columns
INFO [main] 2018-07-02 00:23:59,356 ColumnFamilyStore.java:411 - Initializing system_schema.triggers
INFO [main] 2018-07-02 00:23:59,363 ColumnFamilyStore.java:411 - Initializing system_schema.dropped_columns
INFO [main] 2018-07-02 00:23:59,369 ColumnFamilyStore.java:411 - Initializing system_schema.views
INFO [main] 2018-07-02 00:23:59,377 ColumnFamilyStore.java:411 - Initializing system_schema.types
INFO [main] 2018-07-02 00:23:59,429 ColumnFamilyStore.java:411 - Initializing system_schema.functions
INFO [main] 2018-07-02 00:23:59,435 ColumnFamilyStore.java:411 - Initializing system_schema.aggregates
INFO [main] 2018-07-02 00:23:59,442 ColumnFamilyStore.java:411 - Initializing system_schema.indexes
INFO [main] 2018-07-02 00:23:59,444 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO [MemtableFlushWriter:1] 2018-07-02 00:24:00,086 CacheService.java:112 - Initializing key cache with capacity of 49 MBs.
INFO [MemtableFlushWriter:1] 2018-07-02 00:24:00,131 CacheService.java:134 - Initializing row cache with capacity of 0 MBs
INFO [MemtableFlushWriter:1] 2018-07-02 00:24:00,133 CacheService.java:163 - Initializing counter cache with capacity of 24 MBs
INFO [MemtableFlushWriter:1] 2018-07-02 00:24:00,134 CacheService.java:174 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO [CompactionExecutor:2] 2018-07-02 00:24:00,459 BufferPool.java:230 - Global buffer pool is enabled, when pool is exhausted (max is 249.000MiB) it will allocate on heap
INFO [main] 2018-07-02 00:24:00,570 StorageService.java:600 - Populating token metadata from system tables
INFO [main] 2018-07-02 00:24:00,677 StorageService.java:607 - Token metadata:
INFO [pool-3-thread-1] 2018-07-02 00:24:00,765 AutoSavingCache.java:173 - Completed loading (4 ms; 8 keys) KeyCache cache
INFO [main] 2018-07-02 00:24:00,827 CommitLog.java:152 - No commitlog files found; skipping replay
INFO [main] 2018-07-02 00:24:00,828 StorageService.java:600 - Populating token metadata from system tables
INFO [main] 2018-07-02 00:24:00,847 StorageService.java:607 - Token metadata:
INFO [main] 2018-07-02 00:24:01,042 QueryProcessor.java:163 - Preloaded 0 prepared statements
INFO [main] 2018-07-02 00:24:01,043 StorageService.java:618 - Cassandra version: 3.11.2
INFO [main] 2018-07-02 00:24:01,044 StorageService.java:619 - Thrift API version: 20.1.0
INFO [main] 2018-07-02 00:24:01,044 StorageService.java:620 - CQL supported versions: 3.4.4 (default: 3.4.4)
INFO [main] 2018-07-02 00:24:01,045 StorageService.java:622 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
INFO [main] 2018-07-02 00:24:01,135 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 49 MB and a resize interval of 60 minutes
INFO [main] 2018-07-02 00:24:01,145 MessagingService.java:753 - Starting Messaging Service on /10.88.233.5:7000 (eth0)
WARN [main] 2018-07-02 00:24:01,154 SystemKeyspace.java:1089 - No host ID found, created ec0a1667-7cf4-4527-81c4-82826eb2cb57 (Note: This should happen exactly once per node).
INFO [main] 2018-07-02 00:24:01,176 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED
INFO [ScheduledTasks:1] 2018-07-02 00:24:01,852 TokenMetadata.java:498 - Updating topology for all endpoints that have changed
WARN [main] 2018-07-02 00:24:32,240 Gossiper.java:1425 - Unable to gossip with any seeds but continuing since node is in its own seed list
INFO [main] 2018-07-02 00:24:32,335 StorageService.java:707 - Loading persisted ring state
INFO [main] 2018-07-02 00:24:32,336 StorageService.java:825 - Starting up server gossip
INFO [main] 2018-07-02 00:24:32,422 StorageService.java:886 - This node will not auto bootstrap because it is configured to be a seed node.
INFO [main] 2018-07-02 00:24:32,455 BootStrapper.java:228 - Generated random tokens. tokens are [-8598687621909694459, 2732304674628695086, -8860938353720218287, 3603121280014290014, -3274839593087906376, 2540826393528779592, 7141523298092347247, 5744873283902064866, -7921900519888437381, -1146463088658294905, -2999363497196282396, -2509460247053675936, -7274126478848838301, -5385980385914145695, -6104459401405295966, 6294445431741383814, 1078682267355037729, -5200096417112327145, -8913844052103848577, -6356331455035816923, 3251725814565572167, -1062718915627337136, 5545551232001845400, 1978695947911724014, 2361963748491582556, 7426946334475283750, 7422936577861223419, 6069968566359303156, 5669726325736028576, 5132135004995014301, 4124600807624542534, 1537450561155796272, 5277762343774889040, -983983987811862429, -7818523433678184293, -73668304523309456, -7257757848279164084, -5474078356652921352, 6311456682753612663, -654459396026615325, 5929971473513933658, -5217623993511960468, -5030925211957581340, 1228459630447193964, 2337688762763043393, -8482424441451239225, 8675276978224022891, -1318663781791636595, 1549881071847697622, 3735103928859555416, -6095625247075891193, 3713223273642648611, 1896543874232548925, -4449172050220549698, 2400107068246423605, -8813540133190584405, -882893964790777086, 3682214358731872277, -3798723842540595595, 2620810713072276324, -7214280725441651099, 3628378292546740494, 1594306956278608396, -2177805640746426495, -3768481370523578713, -6745793347561775568, 1086104651517358432, 5461056951350717034, 4606348639422674099, 4074506251894573710, -1459319788231470199, -5975973699600900059, 7757652435294530645, 2909691532615133877, 6870610390547832049, -1609644791176405822, -8379532566252402143, 2274801834156045382, 2129016329598736012, -6412557530528876451, -8651959309161087027, -2659395783193598071, -4160287556494896696, 3625494648636353474, 7397400596515149361, -6719937702269294798, -7014209276554578571, -7829151726480408478, -2381514338488666631, 9075709079654999426, -2778735504756399314, 4025089221088405616, -5717785889529630022, -8416725898748321072, 3952554299354715476, -7998202923141724238, -2207316021011988258, 3431890366216180499, -7219796072602005644, 349858152339773396, 8841281322393138286, 8603821531494313018, -366279638548686465, 4115297281877901743, -3150624570265420913, -652531314244463177, -6289942773733940217, 3767356552566091660, -8859572097856862544, -6218309270723478790, -3616694505960240760, -8062673431176448650, 3753024802079289848, -7101159810307419648, 5576257543571359127, 7763161217865159775, 6697717178824549770, 4240996597697744049, 8463501737338486671, 5307899266743309521, 7208788331510222486, 8178067228650587053, -1084873020099187423, 7790899799641712317, 805419693715848148, 4009789801887706546, 8486941818562642569, -4963926790863722115, -3613053096250010387, 4194578307505886451, -850644247544742793, 5837324935034890588, -184302913095183617, 7396656578956559375, -6493473036446463915, 7883620420721120354, 3737351893721971358, 1435074390668031607, -4645965456288214865, -5633252849672630296, 4356617146688878481, -7563773075165020808, -8516922155735766260, 6241138012769610520, 1181484967975311031, 336845208926491965, -4602254685735012775, -5150261006070385172, -496725161381954053, -2020498316479235748, -406013006284347877, 8056670129610300377, 8874222966151677894, 1356343850148676864, 7265716360661489304, -2762391465502602968, -765136435890521056, -6873265108376059847, -9889580490323966, 3152763081117363495, 768103020618635948, 1816558363428037037, 7934483731989406950, -7161414495688725632, 2577302725028764762, 4620049087810043567, 7184774257380084860, 8192393541062106456, 897599977287852510, 5762138658750069079, -3038329508404979407, 5627659242710494607, -2658334413752010112, -5957399573499042279, 8401966165356590767, -7164504126636102015, 7967686051435415142, 1467960940513357549, -2870061654887031419, -6599430867292309819, 3227523364807078733, -3197836052681875938, 8514359278007927363, 1476672070836018552, 5911640681502797523, 6184323446464446874, 4838834359642898927, -2304764370970336810, 1126417663279943242, 9093475620453874312, -7562893333745941516, -3569157858219359323, 9167487741004124771, -2597358713114786589, -9105696804995655395, -7989617891707543557, 2117494638467453302, -1892031643124978528, 9014422604467346342, 290255090746440929, -1943118428583499908, 8777731963810488391, -247445843110597030, -8271992775286376315, 3176080438523396513, 8102563939361032871, 1395373565843091261, -147775919615967173, -3363612067636846579, -2363608114989635143, 4273229217999207175, -6813219357366538617, 1042950920147530265, -2121560467921862965, -4706779168881952056, 4901126794150215453, -8391094857140220910, -4529585814444892304, 3672335618017708916, 5914220975675763033, -4157561411348775217, -56369168916630513, 6889634048929369898, -8252525254234240554, 8064657547312260056, 7910323944156404736, 2045357292614179647, -2793911123117424948, 3471818282376737912, -764538689129647653, 4433758549117562280, -4251853295404369260, -358585045533933268, -4391418224258099637, -4856299549488344634, -3116272244100162840, -7606608110001435580, 396308056732067231, -2847194527534495270, 4267719102328488573, -1623659299228293651, -8356897408464072752, 8553514187999435385, -5973790122480952389, -1831467611004040479, -6444911650432728764, 8691751154983327289, -6152474320379918272, -8420002825873485631, 3319638135011409030, 1916073268641761287, -8110259650500729621, -2925331046029816700, -134334794948397375, 1966050039354742983, 306652076743437859]
INFO [main] 2018-07-02 00:24:32,462 MigrationManager.java:331 - Create new Keyspace: KeyspaceMetadata{name=system_traces, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}, tables=[org.apache.cassandra.config.CFMetaData@5a6809a6[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,flags=[COMPOUND],params=TableParams{comment=tracing sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [client command coordinator duration request started_at parameters]],partitionKeyColumns=[session_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[client, command, session_id, coordinator, request, started_at, duration, parameters],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@710975f9[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,flags=[COMPOUND],params=TableParams{comment=tracing events, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [activity source source_elapsed thread]],partitionKeyColumns=[session_id],clusteringColumns=[event_id],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[activity, event_id, session_id, source, thread, source_elapsed],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO [MigrationStage:1] 2018-07-02 00:24:32,846 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO [MigrationStage:1] 2018-07-02 00:24:32,850 ColumnFamilyStore.java:411 - Initializing system_traces.events
INFO [MigrationStage:1] 2018-07-02 00:24:32,855 ColumnFamilyStore.java:411 - Initializing system_traces.sessions
INFO [main] 2018-07-02 00:24:32,875 MigrationManager.java:331 - Create new Keyspace: KeyspaceMetadata{name=system_distributed, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[org.apache.cassandra.config.CFMetaData@57015db3[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [coordinator exception_message exception_stacktrace finished_at parent_id range_begin range_end started_at status participants]],partitionKeyColumns=[keyspace_name, columnfamily_name],clusteringColumns=[id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[status, id, coordinator, finished_at, participants, exception_stacktrace, parent_id, range_end, range_begin, exception_message, keyspace_name, started_at, columnfamily_name],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@28a848aa[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [exception_message exception_stacktrace finished_at keyspace_name started_at columnfamily_names options requested_ranges successful_ranges]],partitionKeyColumns=[parent_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,columnMetadata=[requested_ranges, exception_message, keyspace_name, successful_ranges, started_at, finished_at, options, exception_stacktrace, parent_id, columnfamily_names],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@12349b05[cfId=5582b59f-8e4e-35e1-b913-3acada51eb04,ksName=system_distributed,cfName=view_build_status,flags=[COMPOUND],params=TableParams{comment=Materialized View build status, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[] | [status]],partitionKeyColumns=[keyspace_name, view_name],clusteringColumns=[host_id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[view_name, status, keyspace_name, host_id],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO [MigrationStage:1] 2018-07-02 00:24:33,016 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
INFO [MigrationStage:1] 2018-07-02 00:24:33,019 ColumnFamilyStore.java:411 - Initializing system_distributed.parent_repair_history
INFO [MigrationStage:1] 2018-07-02 00:24:33,024 ColumnFamilyStore.java:411 - Initializing system_distributed.repair_history
INFO [MigrationStage:1] 2018-07-02 00:24:33,028 ColumnFamilyStore.java:411 - Initializing system_distributed.view_build_status
INFO [main] 2018-07-02 00:24:33,041 StorageService.java:1449 - JOINING: Finish joining ring
INFO [main] 2018-07-02 00:24:33,179 MigrationManager.java:331 - Create new Keyspace: KeyspaceMetadata{name=system_auth, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[org.apache.cassandra.config.CFMetaData@1e317848[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,flags=[COMPOUND],params=TableParams{comment=role definitions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [can_login is_superuser salted_hash member_of]],partitionKeyColumns=[role],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[salted_hash, member_of, role, can_login, is_superuser],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@666b1d14[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,flags=[COMPOUND],params=TableParams{comment=role memberships lookup table, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[role],clusteringColumns=[member],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, member],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@60b8239b[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,flags=[COMPOUND],params=TableParams{comment=permissions granted to db roles, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | [permissions]],partitionKeyColumns=[role],clusteringColumns=[resource],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, resource, permissions],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@21989c9f[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,flags=[COMPOUND],params=TableParams{comment=index of db roles with permissions granted on a resource, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@65ad3eab, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[resource],clusteringColumns=[role],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[resource, role],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO [MigrationStage:1] 2018-07-02 00:24:33,309 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
INFO [MigrationStage:1] 2018-07-02 00:24:33,312 ColumnFamilyStore.java:411 - Initializing system_auth.resource_role_permissons_index
INFO [MigrationStage:1] 2018-07-02 00:24:33,321 ColumnFamilyStore.java:411 - Initializing system_auth.role_members
INFO [MigrationStage:1] 2018-07-02 00:24:33,326 ColumnFamilyStore.java:411 - Initializing system_auth.role_permissions
INFO [MigrationStage:1] 2018-07-02 00:24:33,332 ColumnFamilyStore.java:411 - Initializing system_auth.roles
INFO [main] 2018-07-02 00:24:33,346 Gossiper.java:1670 - Waiting for gossip to settle...
INFO [main] 2018-07-02 00:24:41,347 Gossiper.java:1701 - No gossip backlog; proceeding
INFO [main] 2018-07-02 00:24:41,659 NativeTransportService.java:70 - Netty using native Epoll event loop
INFO [main] 2018-07-02 00:24:41,705 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO [main] 2018-07-02 00:24:41,705 Server.java:156 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO [main] 2018-07-02 00:24:41,741 CassandraDaemon.java:529 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO [OptionalTasks:1] 2018-07-02 00:24:43,492 CassandraRoleManager.java:356 - Created default superuser role 'cassandra'
INFO [IndexSummaryManager:1] 2018-07-02 01:24:01,139 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [Service Thread] 2018-07-02 01:41:03,375 GCInspector.java:284 - ConcurrentMarkSweep GC in 279ms. CMS Old Gen: 170698008 -> 155521608; Par Eden Space: 96182408 -> 96645488;
INFO [IndexSummaryManager:1] 2018-07-02 02:24:01,152 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 03:24:01,160 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [CompactionExecutor:39] 2018-07-02 04:24:00,141 AutoSavingCache.java:394 - Saved KeyCache (16 items) in 9 ms
INFO [IndexSummaryManager:1] 2018-07-02 04:24:01,165 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 05:24:01,170 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 06:24:01,175 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 07:24:01,225 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [CompactionExecutor:86] 2018-07-02 08:24:00,138 AutoSavingCache.java:394 - Saved KeyCache (17 items) in 6 ms
INFO [IndexSummaryManager:1] 2018-07-02 08:24:01,229 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 09:24:01,326 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 10:24:01,331 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 11:24:01,334 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [CompactionExecutor:125] 2018-07-02 12:24:00,138 AutoSavingCache.java:394 - Saved KeyCache (18 items) in 6 ms
INFO [IndexSummaryManager:1] 2018-07-02 12:24:01,337 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 13:24:01,341 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 14:24:01,344 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 15:24:01,347 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [CompactionExecutor:186] 2018-07-02 16:24:00,138 AutoSavingCache.java:394 - Saved KeyCache (21 items) in 5 ms
INFO [IndexSummaryManager:1] 2018-07-02 16:24:01,351 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 17:24:01,356 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 18:24:01,359 IndexSummaryRedistribution.java:76 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2018-07-02 19:24:01,362 IndexSummaryRedistribution.java:76 - Redistributing index summaries
Job or CronJob for spark-dependencies
Create a job or cronJob for https://github.com/jaegertracing/spark-dependencies.
Naming to support other backends
The Elasticsearch backend is ready, I assume we'll eventually want to have templates for it as well. How should this affect the naming of directories and files in this repo?
Traces randomly appearing/disappearing from web UI query
Problem - what in Jaeger blocks you from solving the requirement?
Traces are randomly appearing and disappearing in the web UI if I try to perform searches for all operations.
Any open questions to address
I took the production deployment and made some modifications to suit my needs with a Cassandra cluster as the persistent storage. However, for some reason, the traces that are coming from my applications are randomly and intermittently appearing and disappearing in the web UI. I can't seem to identify what the issue is. Below are the YAML configs I currently have running. They are running within a 'jaeger' namespace, with Nginx ingress for the query service. The agent pods are accessed via NodePort currently as we work on migrating our application to the k8s cluster and convert them into microservices.
Collector:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
creationTimestamp: 2018-06-13T06:12:38Z
generation: 3
labels:
app: jaeger
jaeger-infra: collector-deployment
name: jaeger-collector
namespace: jaeger
resourceVersion: "6435888"
selfLink: /apis/apps/v1/namespaces/jaeger/deployments/jaeger-collector
uid: c5dd174c-6ed0-11e8-8721-0050568f492d
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: jaeger
jaeger-infra: collector-pod
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
app: jaeger
jaeger-infra: collector-pod
spec:
containers:
- command:
- /go/bin/collector-linux
- --config-file=/conf/collector.yaml
env:
- name: SPAN_STORAGE_TYPE
valueFrom:
configMapKeyRef:
key: span-storage-type
name: jaeger-configuration
image: jaegertracing/jaeger-collector:1.5.0
imagePullPolicy: IfNotPresent
name: jaeger-collector
ports:
- containerPort: 14267
protocol: TCP
- containerPort: 14268
protocol: TCP
- containerPort: 9411
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /conf
name: jaeger-configuration-volume
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
items:
- key: collector
path: collector.yaml
name: jaeger-configuration
name: jaeger-configuration-volume
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2018-06-13T06:12:38Z
lastUpdateTime: 2018-06-27T20:09:16Z
message: ReplicaSet "jaeger-collector-5c9757d497" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: 2018-06-27T20:10:04Z
lastUpdateTime: 2018-06-27T20:10:04Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 3
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Query:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
creationTimestamp: 2018-06-13T06:12:38Z
generation: 2
labels:
app: jaeger
jaeger-infra: query-deployment
name: jaeger-query
namespace: jaeger
resourceVersion: "6435897"
selfLink: /apis/apps/v1/namespaces/jaeger/deployments/jaeger-query
uid: c5f584e0-6ed0-11e8-8721-0050568f492d
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: jaeger
jaeger-infra: query-pod
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
app: jaeger
jaeger-infra: query-pod
spec:
containers:
- command:
- /go/bin/query-linux
- --config-file=/conf/query.yaml
env:
- name: SPAN_STORAGE_TYPE
valueFrom:
configMapKeyRef:
key: span-storage-type
name: jaeger-configuration
image: jaegertracing/jaeger-query:1.5.0
imagePullPolicy: IfNotPresent
name: jaeger-query
ports:
- containerPort: 16686
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 16686
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /conf
name: jaeger-configuration-volume
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
items:
- key: query
path: query.yaml
name: jaeger-configuration
name: jaeger-configuration-volume
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2018-06-13T06:12:38Z
lastUpdateTime: 2018-06-27T20:09:41Z
message: ReplicaSet "jaeger-query-bfc859864" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: 2018-06-27T20:10:04Z
lastUpdateTime: 2018-06-27T20:10:04Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 2
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Agent:
apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "3"
creationTimestamp: 2018-06-13T06:12:38Z
generation: 3
labels:
app: jaeger
jaeger-infra: agent-daemonset
name: jaeger-agent
namespace: jaeger
resourceVersion: "6435952"
selfLink: /apis/apps/v1/namespaces/jaeger/daemonsets/jaeger-agent
uid: c607e875-6ed0-11e8-8721-0050568f492d
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: jaeger
jaeger-infra: agent-instance
template:
metadata:
creationTimestamp: null
labels:
app: jaeger
jaeger-infra: agent-instance
spec:
containers:
- command:
- /go/bin/agent-linux
- --config-file=/conf/agent.yaml
image: jaegertracing/jaeger-agent:1.5.0
imagePullPolicy: IfNotPresent
name: agent-instance
ports:
- containerPort: 5775
hostPort: 5775
protocol: UDP
- containerPort: 6831
hostPort: 6831
protocol: UDP
- containerPort: 6832
hostPort: 6832
protocol: UDP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /conf
name: jaeger-configuration-volume
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
items:
- key: agent
path: agent.yaml
name: jaeger-configuration
name: jaeger-configuration-volume
updateStrategy:
type: OnDelete
status:
currentNumberScheduled: 6
desiredNumberScheduled: 6
numberAvailable: 6
numberMisscheduled: 0
numberReady: 6
observedGeneration: 3
updatedNumberScheduled: 6
Cassandra:
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: 2018-06-20T22:02:16Z
generation: 3
labels:
app: jaeger
name: cassandra
namespace: jaeger
resourceVersion: "6265495"
selfLink: /apis/apps/v1/namespaces/jaeger/statefulsets/cassandra
uid: 987b5a14-74d5-11e8-8721-0050568f492d
spec:
podManagementPolicy: OrderedReady
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: cassandra
serviceName: cassandra
template:
metadata:
creationTimestamp: null
labels:
app: cassandra
jaeger-infra: cassandra-replica
spec:
containers:
- command:
- /docker-entrypoint.sh
- -R
env:
- name: MAX_HEAP_SIZE
value: 1024M
- name: HEAP_NEWSIZE
value: 256M
- name: CASSANDRA_LISTEN_ADDRESS
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: CASSANDRA_CLUSTER_NAME
value: jaeger
- name: CASSANDRA_DC
value: dc1
- name: CASSANDRA_RACK
value: rack1
- name: CASSANDRA_ENDPOINT_SNITCH
value: GossipingPropertyFileSnitch
- name: CASSANDRA_SEEDS
value: cassandra-0.cassandra
image: cassandra:3.11
imagePullPolicy: Always
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- nodetool drain
name: cassandra
ports:
- containerPort: 7000
name: intra-node
protocol: TCP
- containerPort: 7001
name: tls-intra-node
protocol: TCP
- containerPort: 7199
name: jmx
protocol: TCP
- containerPort: 9042
name: cql
protocol: TCP
- containerPort: 9160
name: thrift
protocol: TCP
resources: {}
securityContext:
capabilities:
add:
- IPC_LOCK
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /cassandra_data
name: cassandra-data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 1800
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: cassandra-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: fast
status:
phase: Pending
status:
collisionCount: 0
currentReplicas: 3
currentRevision: cassandra-64c697f78
observedGeneration: 3
readyReplicas: 3
replicas: 3
updateRevision: cassandra-64c697f78
Services:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-06-13T06:12:38Z
labels:
app: jaeger
jaeger-infra: collector-service
name: jaeger-collector
namespace: jaeger
resourceVersion: "3682379"
selfLink: /api/v1/namespaces/jaeger/services/jaeger-collector
uid: c5e60b13-6ed0-11e8-8721-0050568f492d
spec:
clusterIP: 10.107.96.143
ports:
- name: jaeger-collector-tchannel
port: 14267
protocol: TCP
targetPort: 14267
- name: jaeger-collector-http
port: 14268
protocol: TCP
targetPort: 14268
- name: jaeger-collector-zipkin
port: 9411
protocol: TCP
targetPort: 9411
selector:
jaeger-infra: collector-pod
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-06-13T05:42:18Z
labels:
app: jaeger
jaeger-infra: agent-service
name: jaeger-agent
namespace: jaeger
resourceVersion: "5481075"
selfLink: /api/v1/namespaces/jaeger/services/jaeger-agent
uid: 88c5ad8f-6ecc-11e8-8721-0050568f492d
spec:
clusterIP: 10.107.104.15
externalTrafficPolicy: Cluster
ports:
- name: jaeger-agent-udp
nodePort: 30831
port: 6831
protocol: UDP
targetPort: 6831
selector:
jaeger-infra: agent-instance
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
type: NodePort
status:
loadBalancer: {}
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-06-11T22:16:44Z
labels:
app: jaeger
jaeger-infra: query-service
name: jaeger-query
namespace: jaeger
resourceVersion: "3436616"
selfLink: /api/v1/namespaces/jaeger/services/jaeger-query
uid: 1feaf2b1-6dc5-11e8-8721-0050568f492d
spec:
clusterIP: 10.101.239.12
ports:
- name: jaeger-query
port: 80
protocol: TCP
targetPort: 16686
selector:
jaeger-infra: query-pod
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
[Cassandra] Docker image
hi,
i would like to deply jaegertracing on a Kubernetes cluster using production YAML files, but i see that Cassandra image use is jpkroehling/cassandra and not the official image.
It is possible to use the official image ?
Production config
Any plans to add production yamls similar to: https://github.com/jaegertracing/jaeger-openshift ? Currently all-in-one isn't really suitable for it.
DaemonSet for all-in-one
During the review of #75, it has been suggested that the all-in-one template could also be deployed as a DaemonSet. As this is "all-in-one", it would require the single pod to run as DaemonSet.
This issue is to gather thoughts and suggestions.
READ.md needs to be update in #production-setup section.
After commit f930d0a (Split backing storage from the main templates.) change, README.md seems to be outdated.
In READ.md, #production-setup section needs to be updated as following.
Production setup
This template deploys the Collector, Query Service (with UI) and Cassandra storage (StatefulSet) as separate individually scalable services.
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production/cassandra.ymlkubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/jaeger-production-template.ymlOr you can deploy the Collector, Query Service (with UI) and ElasticSearch storage (StatefulSet) as separate individually scalable services.
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production-elasticsearch/elasticsearch.ymlkubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/jaeger-production-template.ymlNote that it's OK to have the Query and Collector pods to be in an error state for the first minute or so. This is
because these components attempt to connect to Cassandra right away and hard fail if they can't after N attempts.
Once everything is ready, kubectl get service jaeger-query tells you where to find Jaeger URL.
Agent ports
https://www.jaegertracing.io/docs/deployment/#agent
The yamls are missing the new http port
- containerPort: 5778
protocol: TCP
Error reading service_names from storage: Cannot achieve consistency level LOCAL_ONE
Hi!
I just created a Cassandra cluster with an operator (https://github.com/instaclustr/cassandra-operator) and the Cassandra schema job succeeded but when going to the Jaeger Query UI I get the following error:
{ "data": null, "total": 0, "limit": 0, "offset": 0, "errors": [ { "code": 500, "msg": "Error reading service_names from storage: Cannot achieve consistency level LOCAL_ONE" } ] }
--
I also get this error in the Jaeger Collector logs.
I saw this error in the following issue that seems related:
jaegertracing/jaeger#513
But when adding the specified env variables to the CustomResourceDefiniton of the Cassandra cluster they don't seem to be propagated to the cassandra pod but just to it's sidecar (since they still are in alpha it may be an issue on their side).
I tried to add them manually to the statefulset and restart the cassandra pods but it doesn't work. It looks like the operator override these changes quickly (need more time to confirm).
While I'm trying to figure out how to set these variables without troubles with the operator, does anyone already see this error and resolved it by another way? Since the error is not that much explicit I'm wondering if it comes from here.
Thanks
k8s deployment issues
Requirement - what kind of business use case are you trying to solve?
deploy jeager
Problem - what in Jaeger blocks you from solving the requirement?
I tried both the all-in-one as well as the production yaml.
To verify basic health of the service was trying to open the UI but looks like the container is not listening on the port 16686.
I can see a log like:
{"level":"info","ts":1532641841.5165153,"caller":"standalone/main.go:304","msg":"Starting jaeger-query HTTP server","port":0}
does the all-in-one expect some configuration?
This is actually the full log of the pod:
{"level":"info","ts":1532641841.3107834,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
{"level":"info","ts":1532641841.3112895,"caller":"memory/factory.go:55","msg":"Memory storage configuration","configuration":{"MaxTraces":0}}
{"level":"info","ts":1532641841.3155985,"caller":"tchannel/builder.go:94","msg":"Enabling service discovery","service":"jaeger-collector"}
{"level":"info","ts":1532641841.3156722,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"127.0.0.1:14267"}
{"level":"info","ts":1532641841.3163402,"caller":"standalone/main.go:187","msg":"Starting agent"}
{"level":"info","ts":1532641841.3170893,"caller":"standalone/main.go:227","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1532641841.3171778,"caller":"standalone/main.go:237","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1532641841.3713417,"caller":"standalone/main.go:257","msg":"Listening for Zipkin HTTP traffic","zipkin.http-port":9411}
{"level":"info","ts":1532641841.51646,"caller":"standalone/main.go:298","msg":"Registering metrics handler with jaeger-query HTTP server","route":"/metrics"}
{"level":"info","ts":1532641841.5165153,"caller":"standalone/main.go:304","msg":"Starting jaeger-query HTTP server","port":0}
{"level":"info","ts":1532641841.516577,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1532641842.3158925,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1}
{"level":"info","ts":1532641842.3159378,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"127.0.0.1:14267"}
{"level":"info","ts":1532641842.3163357,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}
Also this one does not look too good:
{"level":"info","ts":1532641842.3158925,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1}
do you have any suggestion on how to debug this?
Reorganize templates
The current organization of the templates has become confusing (production vs. production-elasticsearch, for instance).
A recommendation is the following:
.
โโโ development (currently, it's the all-in-one)
โ โโโ development.yaml
โโโ production
โโโ cassandra.yaml
โโโ elasticsearch.yaml
โโโ jaeger.yaml
Of course, the tests have to be adjusted for this.
Authorized access to UI
Hello,
I haven't found any option related to authorization. Is it possible to protect access to UI with login/passwd?
Depending on application it is possible that some sensitive data can be dumped into spans and it is strictly desirable to protect such data.
Jaeger POD going up and down
As part of deploying OpenTracing Prometheus Examples, I installed the Jaeger OpenTracing tracing system:
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml
Once the pods are all started, Jaeger is going up and down:
zji@~/opentracing-prometheus-example/simple> kubectl get pod
NAME READY STATUS RESTARTS AGE
accountmgr-59fd8dc6c8-gzthk 1/1 Running 16 1d
accountmgr-59fd8dc6c8-mrjlx 1/1 Running 25 1d
accountmgr-59fd8dc6c8-vmrr6 1/1 Running 26 1d
belligerent-monkey-grafana-85d5d4988f-gj5r6 1/1 Running 0 23h
jaeger-deployment-84c69bc88d-drlqh 0/1 Running 0 16m
my-grafana-grafana-86db98495c-6qxj2 1/1 Running 0 23h
ordermgr-7998c8794c-gsxbh 1/1 Running 18 1d
ordermgr-7998c8794c-kt2rw 1/1 Running 23 1d
ordermgr-7998c8794c-nb8sk 1/1 Running 23 1d
prometheus-865564b8f5-6726h 1/1 Running 1 1d
zji@~/opentracing-prometheus-example/simple> kubectl describe pod/jaeger-deployment-84c69bc88d-drlqh
Name: jaeger-deployment-84c69bc88d-drlqh
Namespace: default
Node: minikube/192.168.99.100
Start Time: Thu, 08 Mar 2018 16:49:23 -0800
Labels: app=jaeger
jaeger-infra=jaeger-pod
pod-template-hash=4072567448
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"jaeger-deployment-84c69bc88d","uid":"4cf32ede-2261-11e8-8d6d-080...
Status: Running
IP: 172.17.0.5
Controlled By: ReplicaSet/jaeger-deployment-84c69bc88d
Containers:
jaeger:
Container ID: docker://630157a0610d2170b31f98d925f3df3f6aac36c5c5afb551d60d67e0eaf70409
Image: jaegertracing/all-in-one
Image ID: docker-pullable://jaegertracing/all-in-one@sha256:9668ff6d5c49c385af2da7f96fad31ceb0d746861a52cf5febe3d369b57f5c01
Ports: 5775/UDP, 6831/UDP, 6832/UDP, 16686/TCP, 9411/TCP
State: Running
Started: Thu, 08 Mar 2018 16:49:37 -0800
Ready: False
Restart Count: 0
Readiness: http-get http://:16686/ delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
COLLECTOR_ZIPKIN_HTTP_PORT: 9411
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-k49dr (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-k49dr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-k49dr
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19m default-scheduler Successfully assigned jaeger-deployment-84c69bc88d-drlqh to minikube
Normal SuccessfulMountVolume 19m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-k49dr"
Normal Pulling 19m kubelet, minikube pulling image "jaegertracing/all-in-one"
Normal Pulled 19m kubelet, minikube Successfully pulled image "jaegertracing/all-in-one"
Normal Created 19m kubelet, minikube Created container
Normal Started 19m kubelet, minikube Started container
Warning Unhealthy 18m (x5 over 18m) kubelet, minikube Readiness probe failed: Get http://172.17.0.5:16686/: dial tcp 172.17.0.5:16686: getsockopt: connection refused
Warning Unhealthy 13m (x16 over 15m) kubelet, minikube Readiness probe failed: Get http://172.17.0.5:16686/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Normal SuccessfulMountVolume 12m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-k49dr"
Normal SuccessfulMountVolume 7m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-k49dr"
thanks
Investigate and fix Elasticsearch test failures
See #43 : although the PR had nothing to do with ES, the tests were not passing. The tests are probably failing intermittently and need to be fixed.
Agent sidecar resources defined in README.
Not really sure how much resources a sidecar needs. Is 128Mi too much?
resources:
limits:
memory: 128Mi
requests:
memory: 128MiCassandra Replication factor
The production template currently uses MODE=test which translates to
replication="{'class': 'SimpleStrategy', 'replication_factor': '1'}".
We deploy c* StatefulSet with 3 nodes, does it mean that one row is stored only on one of these 3 c* nodes? So if one node goes down do we loose data? What should we use? Something like
replication="{'class': 'NetworkTopologyStrategy', '$datacenter': '${replication_factor}' }" where replication_factor >= 2. Or what is the ideal number? Should it be equal to the number of c* nodes? @yurishkuro @jsanda (I'm new to c*)
I have tried to setup the production cluster locally but I get some errors:
ccm create test -v 3.9 -n 3 -s
MODE=prod DATACENTER=local sh plugin/storage/cassandra/schema/create.sh | ccm node1 cqlsh
go run cmd/query/main.go --query.static-files=jaeger-ui-build/build/ -cassandra.keyspace=jaeger_v1_local-> Go to UI and get some traces
Error: {"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":500,"msg":"Cannot achieve consistency level ONE"}]} or:
go run cmd/query/main.go --query.static-files=jaeger-ui-build/build/ -cassandra.keyspace=jaeger_v1_local ๎ฒ 3:23
{"level":"info","ts":1500038642.715631,"caller":"query/main.go:76","msg":"Starting jaeger-query HTTP server","port":16686}
{"level":"error","ts":1500038655.0899656,"caller":"spanstore/reader.go:376","msg":"Failed to exec query","error":"Cannot achieve consistency level ONE","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:176\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).executeQuery\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:376\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).queryByService\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:362\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).findTraceIDs\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:280\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).FindTraces\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:236\ngithub.com/uber/jaeger/cmd/query/app.(*APIHandler).search\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/cmd/query/app/handler.go:215\ngithub.com/uber/jaeger/cmd/query/app.(*APIHandler).(github.com/uber/jaeger/cmd/query/app.search)-fm\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/cmd/query/app/handler.go:124\nnet/http.HandlerFunc.ServeHTTP\n\t/home/ploffay/bin/go/src/net/http/server.go:1726\ngithub.com/uber/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74\nnet/http.HandlerFunc.ServeHTTP\n\t/home/ploffay/bin/go/src/net/http/server.go:1726\nnet/http.(Handler).ServeHTTP-fm\n\t/home/ploffay/bin/go/src/net/http/h2_bundle.go:4084\nnet/http.HandlerFunc.ServeHTTP\n\t/home/ploffay/bin/go/src/net/http/server.go:1726\ngithub.com/uber/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/vendor/github.com/gorilla/mux/mux.go:114\ngithub.com/uber/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/home/ploffay/projects/golang/src/github.com/uber/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/uber/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:53\nnet/http.serverHandler.ServeHTTP\n\t/home/ploffay/bin/go/src/net/http/server.go:2202\nnet/http.(*conn).serve\n\t/home/ploffay/bin/go/src/net/http/server.go:1579"}
I think @jpkrohling got the same error here https://github.com/uber/jaeger/pull/157#discussion_r116272696
How do you edit the configmap to enable 6831 UDP?
I am trying to configure the Jaeger Collector to accept messages on 6831 UDP but there aren't any documentation on this. Can someone please provide some instructions?
Could not find aggregation of distinct_services
how can i do ~~~~
deploy jaeger-production-template-with-elasticsearch.yml
k8s version 1.6.4
jaeger-query
curl 10.254.100.41/api/services
{"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":500,"msg":"Could not find aggregation of distinct_services"}]}
jaeger-collector logs
{"level":"info","ts":1508414827.1502573,"caller":"healthcheck/handler.go:46","msg":"Health Check server started","http-port":14269} {"level":"info","ts":1508414828.176905,"caller":"collector/main.go:113","msg":"Starting Jaeger Collector HTTP server","http-port":14268} {"level":"info","ts":1508414828.176981,"caller":"healthcheck/handler.go:88","msg":"Health Check state change","http-status":204} {"level":"info","ts":1508414828.1772332,"caller":"collector/main.go:154","msg":"Listening for Zipkin HTTP traffic","zipkin.http-port":9411}
jaeger-query logs
{"level":"info","ts":1508414826.0206602,"caller":"healthcheck/handler.go:46","msg":"Health Check server started","http-port":16687} {"level":"info","ts":1508414828.0497098,"caller":"healthcheck/handler.go:88","msg":"Health Check state change","http-status":204} {"level":"info","ts":1508414828.049764,"caller":"query/main.go:92","msg":"Starting jaeger-query HTTP server","port":16686}
jaeger-agent logs
{"level":"info","ts":1508415421.4560003,"caller":"tchannel/bulider.go:89","msg":"Enabling service discovery","service":"jaeger-collector"} {"level":"info","ts":1508415421.4561126,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"jaeger-collector.default.svc:14267"} {"level":"info","ts":1508415421.4566708,"caller":"agent/main.go:53","msg":"Starting agent"} {"level":"info","ts":1508415422.4564009,"caller":"peerlistmgr/peer_list_mgr.go:159","msg":"Not enough connected peers","connected":0,"required":1} {"level":"info","ts":1508415422.4564948,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector.default.svc:14267"} {"level":"info","ts":1508415422.4603477,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"0.0.0.0:14267"}
cassandra nodes refuse to come up. Seed provider couldn't lookup host cassandra-0.cassandra.jaeger.svc.cluster.local
when i try to use the production template, I always get this:
~ โฏโฏโฏ kubej get po
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 4m
cassandra-1 0/1 CrashLoopBackOff 5 4m
cassandra-2 0/1 CrashLoopBackOff 5 3m
cassandra-0 logs
INFO [main] 2017-09-13 14:06:29,146 StorageService.java:857 - This node will not auto bootstrap because it is configured to be a seed node.
and starts up fine.
the other two log:
INFO [main] 2017-09-13 14:09:32,907 DatabaseDescriptor.java:710 - Back-pressure is disabled with strategy org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}.
INFO [main] 2017-09-13 14:09:33,134 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
WARN [main] 2017-09-13 14:09:33,155 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-0.cassandra.jaeger.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR [main] 2017-09-13 14:09:33,158 CassandraDaemon.java:706 - Exception encountered during startup: The seed provider lists no seeds.
The seed provider lists no seeds.
note that the statefulset defines:
- name: CASSANDRA_SEEDS
value: cassandra-0.cassandra.jaeger.svc.cluster.local
i've tried some others like cassandra-0 with same result.
I'll try some more.
i'm pretty sure at one point i was able to exec into one of the other containers before it shut down, and a ping to cassandra-0.cassandra.jaeger.svc.cluster.local worked fine. so i'm puzzled
Make integration test reusable
I was looking at BaseETest.java in light of the docker-compose question https://github.com/uber/jaeger/issues/286.
It seems the test is a fairly generic end-to-end check of "is my Jaeger installation wired and working correctly" (@black-adder - didn't we have a similar one in the main repo?)
My question is - why not implement that test in the main jaeger repo, in Go, and build it as a Docker image, with some parameters controlled via env variables? Then if issue 286 results in a docker-compose file in the repo, it can be tested by adding the e2e as the last step, and k8s and openshift templates can also be tested with the same docker image.
sampling.strategies doesn't work in all in one image version 1.6
Hi, I'm using all-in-one docker image version 1.6 and i want to discard some certain services by setting sampling strategies. Here was my configration and i found that jaeger still trace istio-mixer and istio-policy services.
---
kind: ConfigMap
apiVersion: v1
metadata:
name: istio-tracing
labels:
jaeger-infra: istio-tracing
data:
config.json: |-
{
"service_strategies": [
{
"service": "istio-mixer",
"type": "const",
"param": 0
},
{
"service": "istio-policy",
"type": "const",
"param": 0
},
{
"service": "istio-telemetry",
"type": "const",
"param": 0
}
]
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-tracing
namespace: istio-system
labels:
app: istio-tracing
chart: tracing-0.1.0
release: istio
heritage: Tiller
spec:
replicas: 1
template:
metadata:
labels:
app: jaeger
annotations:
sidecar.istio.io/inject: "false"
scheduler.alpha.kubernetes.io/critical-pod: ""
spec:
volumes:
- name: config-volume
configMap:
name: istio-tracing
containers:
- name: jaeger
image: "docker.io/jaegertracing/all-in-one:1.6"
imagePullPolicy: IfNotPresent
args:
- '--sampling.strategies-file=/etc/conf/config.json'
ports:
- containerPort: 9411
- containerPort: 16686
- containerPort: 5775
protocol: UDP
- containerPort: 6831
protocol: UDP
- containerPort: 6832
protocol: UDP
volumeMounts:
- name: config-volume
mountPath: /etc/conf
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: COLLECTOR_ZIPKIN_HTTP_PORT
value: "9411"
- name: MEMORY_MAX_TRACES
value: "50000"
livenessProbe:
initialDelaySeconds: 360
httpGet:
path: /
port: 16686
readinessProbe:
initialDelaySeconds: 360
httpGet:
path: /
port: 16686
resources:
requests:
cpu: 10m
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 2
preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- weight: 2
preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- ppc64le
- weight: 2
preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- s390x
Split configmap templates to separate files
Config map templates contains storage definitions. If people are using external storages - we even encourage to do so then templates cannot be used without commenting out storage definitions.
Rename service/deployment etc to remove 'all-in-one'
The Service and Deployment metadata names are currently 'jaeger-all-in-one' and 'jaeger-all-in-one-agent'.
This means that when being used by an application, the application's configuration would need to use the 'all-in-one' to reference the target (e.g. agent). So when switching to a 'production' version of the jaeger deployment, the applications would need to be changed.
Wondering if it would be better to use a consistent set of names, across the all-in-one and production templates, so that the applications/configurations are not impacted by a change - i.e. it is simply an administration issue whether the all-in-one or production templates have been deployed.
how to add my application service to jaeger
i have installed jaeger which is mentioned in git
core@logan-1-coreos-378 ~ $ kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml -n jaeger
deployment.extensions "jaeger-deployment" created
service "jaeger-query" created
service "jaeger-collector" created
service "jaeger-agent" created
service "zipkin" created
core@logan-1-coreos-378 ~ $ kubectl get svc -n jaeger
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP,5778/TCP 12s
jaeger-collector ClusterIP 10.3.3.216 <none> 14267/TCP,14268/TCP,9411/TCP 12s
jaeger-query LoadBalancer 10.3.3.255 <pending> 80:7359/TCP 12s
zipkin ClusterIP None <none> 9411/TCP 12s
able to see one pod is running
core@logan-1-coreos-378 ~ $ kubectl get pods --all-namespaces | grep jaeger
jaeger jaeger-deployment-5bd9f6d7ff-qj7wk 1/1 Running 0 5m
but my request is not being captured in traces
i did portforwarding 10.3.3.255 . but services shows only 1 "jaeger-query"
Questions:
- Is there any other config to add my application services to jaeger?
few logs says that as follows:
orders-signup-uwsgi-deployment-66fbb77bd-8ckbr jaeger-agent {"level":"info","ts":1540558760.5167942,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector.jaeger-infra.svc:14267"}
orders-signup-uwsgi-deployment-66fbb77bd-8ckbr jaeger-agent {"level":"error","ts":1540558760.5441663,"caller":"peerlistmgr/peer_list_mgr.go:171","msg":"Unable to connect","host:port":"jaeger-collector.jaeger-infra.svc:14267","connCheckTimeout":0.25,"error":"dial tcp: lookup jaeger-collector.jaeger-infra.svc on 10.3.0.10:53: no such host","stacktrace":"github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).ensureConnections\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:171\ngithub.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).maintainConnections\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:101"}
Deployment of Jaeger Query 1.7 failing on AWS EKS
I am working on upgrading our Jaeger deployment on AWS EKS to Jaeger 1.7, and am running into a crashloopbackoff state with Jaeger Query. All other components install correctly.
We are using an Elastic Search back end via the AWS ES service. When the pod attempts to start, it fails, and returns a "Could not create static assets handler","error":"Cannot load index.html" error via kubectl logs. Our prior install runs without issue using Jaeger Query 1.5.
Logs and pod description from Kubectl are attached.
1.7query_pod.log
1.7query_crash.log
Elasticsearch in production always Back-off restarting failed container
elasticsearch version:
docker.elastic.co/elasticsearch/elasticsearch:5.6.0
k8s cluster version
1.10
describe
# kubectl describe pods -n jaeger elasticsearch-0
Name: elasticsearch-0
Namespace: jaeger
Node: node-1/192.168.205.128
Start Time: Sat, 28 Apr 2018 16:44:35 +0800
Labels: app=jaeger-elasticsearch
controller-revision-hash=elasticsearch-8684f69799
jaeger-infra=elasticsearch-replica
statefulset.kubernetes.io/pod-name=elasticsearch-0
Annotations: <none>
Status: Running
IP: 192.168.3.197
Controlled By: StatefulSet/elasticsearch
Containers:
elasticsearch:
Container ID: docker://941824d0c9186862372c793d41d578a5e34c0972c877771d00629dc375593530
Image: docker.elastic.co/elasticsearch/elasticsearch:5.6.0
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:f95e7d4256197a9bb866b166d9ad37963dc7c5764d6ae6400e551f4987a659d7
Port: <none>
Host Port: <none>
Command:
bin/elasticsearch
Args:
-Ehttp.host=0.0.0.0
-Etransport.host=127.0.0.1
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Sat, 28 Apr 2018 16:50:57 +0800
Finished: Sat, 28 Apr 2018 16:50:57 +0800
Ready: False
Restart Count: 6
Readiness: exec [curl --fail --silent --output /dev/null --user elastic:changeme localhost:9200] delay=5s timeout=4s period=5s #success=1 #failure=3
Environment: <none>
Mounts:
/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8l8qt (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-8l8qt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8l8qt
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m default-scheduler Successfully assigned elasticsearch-0 to node-1
Normal SuccessfulMountVolume 7m kubelet, node-1 MountVolume.SetUp succeeded for volume "data"
Normal SuccessfulMountVolume 7m kubelet, node-1 MountVolume.SetUp succeeded for volume "default-token-8l8qt"
Normal Pulling 6m (x4 over 7m) kubelet, node-1 pulling image "docker.elastic.co/elasticsearch/elasticsearch:5.6.0"
Normal Pulled 6m (x4 over 7m) kubelet, node-1 Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:5.6.0"
Normal Created 6m (x4 over 7m) kubelet, node-1 Created container
Normal Started 6m (x4 over 7m) kubelet, node-1 Started container
Warning BackOff 2m (x22 over 7m) kubelet, node-1 Back-off restarting failed containerlog
# kubectl logs -n jaeger elasticsearch-0
# nothing shown.Automate tests
CI currently does not test anything. It only compiles and tests are done manually as part of the review.
Travis and CircleCi does not support running virtualization environments, therefore it's not possible to start minikube.
Some links:
Intermittent ES failure with NPE
As can be scene here.
[ERROR] Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 49.396 s <<< FAILURE! - in io.jaegertracing.kubernetes.ElasticearchETest
[ERROR] testReportSpanToCollector(io.jaegertracing.kubernetes.ElasticearchETest) Time elapsed: 1.212 s <<< ERROR!
java.lang.RuntimeException: Could not lookup value for field private java.net.URL io.jaegertracing.kubernetes.deployment.BaseETest.collectorUrl
Caused by: java.lang.NullPointerException
[ERROR] testUiResponds(io.jaegertracing.kubernetes.ElasticearchETest) Time elapsed: 0.317 s <<< ERROR!
java.lang.RuntimeException: Could not lookup value for field private java.net.URL io.jaegertracing.kubernetes.deployment.BaseETest.collectorUrl
Caused by: java.lang.NullPointerException
[ERROR] hitDependencyScreen(io.jaegertracing.kubernetes.ElasticearchETest) Time elapsed: 0.326 s <<< ERROR!
java.lang.RuntimeException: Could not lookup value for field private java.net.URL io.jaegertracing.kubernetes.deployment.BaseETest.collectorUrl
Caused by: java.lang.NullPointerException
[ERROR] testDependencyLinks(io.jaegertracing.kubernetes.ElasticearchETest) Time elapsed: 0.249 s <<< ERROR!
java.lang.RuntimeException: Could not lookup value for field private java.net.URL io.jaegertracing.kubernetes.deployment.BaseETest.collectorUrl
Caused by: java.lang.NullPointerException
[ERROR] testReportZipkinSpanToCollector(io.jaegertracing.kubernetes.ElasticearchETest) Time elapsed: 0.173 s <<< ERROR!
java.lang.RuntimeException: Could not lookup value for field private java.net.URL io.jaegertracing.kubernetes.deployment.BaseETest.collectorUrl
Caused by: java.lang.NullPointerException
Deleting namespace: itest-f450b2be...
Namespace: itest-f450b2be, successfully deleted
Enable dependency test for production template
Currently dependency test for production template is disabled, because the endpoint returns 404, see jaegertracing/jaeger-openshift#19 and jaegertracing/jaeger-openshift#11
How to use configmap for sampling strategies
See jaegertracing/jaeger#970
cc @Yisaer
Use DaemonSet by default instead of sidecar
Hello,
Just curious to know the reasons & advantages of using Jaeger agent as a side car instead Daemon set.
Many Thanks,
Teja
Can't run jaeger-query behind ingress
Hi,
I am trying to setup jaeger accordingly to https://github.com/jaegertracing/jaeger-kubernetes and using Ingress resource for accessing Jaeger UI. But I see some issues with UI, in particular - incorrect urls for .js|.css file requests. I tried to set query.prefix option in config for this purpose but it is ignored.
Config:
apiVersion: v1
data:
agent: |
collector:
host-port: "jaeger-collector:14267"
collector: |
es:
server-urls: http://elasticsearch.logging:9200
collector:
zipkin:
http-port: 9411
query: |
es:
server-urls: http://elasticsearch.logging:9200
query:
static-files: /go/jaeger-ui/
prefix: /tracing/jaeger
span-storage-type: elasticsearch
kind: ConfigMap
metadata:
labels:
app: jaeger
jaeger-infra: configuration
name: jaeger-configuration
namespace: tracing
The ingress resource looks like:
apiVersion: v1
kind: List
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/add-base-url: "true"
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
name: jaeger
namespace: tracing
resourceVersion: "542749"
spec:
rules:
- host: etsys-sm-107.vms
http:
paths:
- backend:
serviceName: jaeger-query
servicePort: jaeger-query
path: /tracing/jaeger
tls:
- hosts:
- etsys-sm-107.vms
secretName: tracing-certs
But the urls look like:
https://etsys-sm-107.vms/static/css/main.2a0fcb74.css
https://etsys-sm-107.vms/static/js/main.b18b9217.js
I am using image: jaegertracing/jaeger-query:1.2
Please advise.
Jaeger should use healthcheck for readinessProbe
Query and standalone currently have readinessProbe using Query port(16686).
Wouldn't it be better using healthcheck port for all components(Collector, Query, Standalone)?
They have their own port for healthcheck.
(Note that Standalone healthcheck is in PR)
Another question,
Per my understanding, there are 2 probes in kubernetes. livenessProbe and readinessProbe.
Why readinessProbe is used?
Production setup with Elasticsearch points to the wrong file on README.md
On Elasticsearch setup kubectl create -f points to
https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production-with-elasticsearch/jaeger-production-template-with-elasticsearch.yml
It should be
https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production-elasticsearch/jaeger-production-template-with-elasticsearch.yml
Pinned Image Versions
The deployment versions are lagging behind the released versions three months now. Is there a reason they're pinned to 1.2
Use SPAN_STORAGE env var instead of deprecated command line option
Mention Helm support in README
Reschedule `jaeger-cassandra-schema-job` upon failure
I'm using the production YAML file to deploy Jager :
root@kmaster01:~# kubectl --namespace=opentracing get pod
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 5m
cassandra-1 1/1 Running 1 5m
cassandra-2 1/1 Running 3 5m
jaeger-collector-1578691591-gm9dn 0/1 CrashLoopBackOff 3 6m
jaeger-query-1666845303-113hb 0/1 CrashLoopBackOff 2 6m
root@kmaster01:~# kubectl --namespace=opentracing logs jaeger-collector-1578691591-gm9dn
2017/09/06 14:42:56 error: failed to connect to 10.244.48.54:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to 10.244.80.25:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to 10.244.128.36:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to 10.244.48.54:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to 10.244.80.25:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:57 error: failed to connect to 10.244.128.36:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1504708977.0914874,"caller":"collector/main.go:79","msg":"Unable to set up builder","error":"no connections were made when creating the session","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:208\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:79\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:122"}
root@kmaster01:~# kubectl --namespace=opentracing logs jaeger-query-1666845303-113hb
2017/09/06 14:43:12 error: failed to connect to 10.244.128.36:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to 10.244.80.25:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to 10.244.48.54:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to 10.244.128.36:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to 10.244.80.25:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to 10.244.48.54:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1504708993.0407972,"caller":"query/main.go:71","msg":"Failed to init storage builder","error":"no connections were made when creating the session","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:208\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:71\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:100"}
Any idea why i've got these errors ? Thanks.
Production template isolated to namespace only
Hello!
How can I setup Jaeger to only see traces within a single namespace versus querying the entire thing in kubernetes?
The best case would be if there was a single jaeger + elastic search setup that handles all namespaces, but I can deploy a JaegerUI that only filters a single namespace?
Use case is to deploy jaeger multiple times to have dedicated instances to see services in namespace A and B.
Ingress Configuration
While trying to expose jaeger to a custom base-path, I was running into odd errors. Stuff like can't load resources, everything redirecting to the main landing page with no results, and http parsing errors.
It turns out for some other services, we had the annotation ingress.kubernetes.io/rewrite-target set to / which threw off all the requests.
So if anyone else searches/runs into this problem without understanding ingresses, the default ingress works.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: jaeger-ingress
spec:
rules:
- http:
paths:
- path: /jaeger
backend:
serviceName: jaeger-query
servicePort: 80
You don't want a rewrite-target
metadata:
annotations:
ingress.kubernetes.io/rewrite-target: /
name: main-ingress
it'll output errors like
Uncaught SyntaxError: Unexpected token < jaeger:1
Uncaught SyntaxError: Unexpected token < main.4f0c2202.js:1
Resource interpreted as Stylesheet but transferred with MIME type text/html: "http://....com/jaeger/static/css/main.160e898e.css". jaeger:10
As far as I can tell it's because the k8s ingress doesn't play nice with angular
Exception (org.apache.cassandra.io.FSReadError) encountered during startup: java.io.EOFException
Hello folks,
I know this is not really a jaeger question per se, but i have a problem sometimes with cassandra when i deploy the production template from this repo, maybe someone knows an answer.
basically typically the whole jaeger setup (incl cassandra) runs fine for a while, and then suddenly the cassandra container starts crashlooping with this error:
Exception (org.apache.cassandra.io.FSReadError) encountered during startup: java.io.EOFException
FSReadError in /var/lib/cassandra/hints/718a84bd-5f5c-4ca2-b4c6-93774ef9876c-1506428454916-1.hints
at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:235)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65)
at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:97)
at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:88)
at org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:66)
at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.java:127)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:627)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803)
at org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:344)
at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:231)
... 21 more
it seems to be preceeded by:
INFO [pool-3-thread-1] 2017-09-26 14:48:41,871 AutoSavingCache.java:262 - Harmless error reading saved cache /var/lib/cassandra/saved_caches/KeyCache-e.db
java.io.UTFDataFormatException: malformed input around byte 1
at java.io.DataInputStream.readUTF(DataInputStream.java:656) ~[na:1.8.0_131]
at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_131]
at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:215) ~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:164) [apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:160) [apache-cassandra-3.11.0.jar:3.11.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
INFO [pool-3-thread-1] 2017-09-26 14:48:41,871 AutoSavingCache.java:173 - Completed loading (19 ms; 275 keys) KeyCache cache
disk has not run full and i also don't see any io errors in dmesg
Create CronJob for elastic search index deletion script
ES doesn't have a built in TTL so it might be useful to have a configurable cronjob that allows users to set their own TTL.
https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/es/es_indices_clean.sh
Use the CronJob that comes out of #36 for guidance
failed to "StartContainer" for "cassandra" with CrashLoopBackOff
Hi, I deploy Jager via
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production/jaeger-production-template.yml
and get errors :
I use k8s in Mac.
Why I've got these errors? Thinks!
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.