Comments (15)
brightredchilli
commented on July 19, 2024
1
Understood, we will monitor this and report back the next time it happens
from ruby-client.
NicoZelaya
commented on July 19, 2024
1
Closing the issue for now but if you're still having issues, please feel free to reopen it.
Thanks in advance!
Nico.
from ruby-client.
chillaq
commented on July 19, 2024
Hi, thanks for the details, is the redis instance shared with other apps? does it have any eviction policy? the fact that split names are empty and are resetted when something changes seems to indicate something like that.
Also, is it possible to turn on debug or info logging for Synchronizer at least until the issue is recreated, then post the log here?
Thanks
Bilal
from ruby-client.
brightredchilli
commented on July 19, 2024
Hi @chillaq, redis instance is not shared, i can't see its configs but am told that there is 'maxmemory_policy:volatile-lru' set. I've turned on the logging, will report back here. Thanks.
from ruby-client.
brightredchilli
commented on July 19, 2024
Actually, sorry, correction, it's actually allkeys-lru
maxmemory:402653184
maxmemory_human:384.00M
maxmemory_policy:allkeys-lru
from ruby-client.
brightredchilli
commented on July 19, 2024
So I have a bunch of logs, but im not sure what you might be looking for:
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[66317] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[242] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:49 GMT] Etag:["-866568843--gzip"] Last-Modified:[Fri, 24 Jan 2020 16:06:08 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[HIT, HIT] X-Cache-Hits:[1, 1] X-Served-By:[cache-dca17783-DCA, cache-sea4479-SEA] X-Timer:[S1602185209.298931,VS0,VE1]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:74: >>>> Fetched segment: islands_team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:74: >>>> Fetched segment: Member_Web_Team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:32: Adding to segment mx_support_squad
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:32: Adding to segment us_adyen
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:32: Adding to segment islands_team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:32: Adding to segment Member_Web_Team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:46: Removing from segment mx_support_squad
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:46: Removing from segment us_adyen
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:46: Removing from segment islands_team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 segment.go:46: Removing from segment Member_Web_Team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:80: [GET] https://sdk.split.io/api/segmentChanges/staging_support_android_testing?since=1579183257665
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:109: REQUEST TIME TOOK: 7 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[7613] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[107] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:49 GMT] Etag:["1190873507--gzip"] Last-Modified:[Thu, 16 Jan 2020 14:00:57 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[HIT, HIT] X-Cache-Hits:[1, 2] X-Served-By:[cache-dca17776-DCA, cache-sea4462-SEA] X-Timer:[S1602185210.801588,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:80: [GET] https://sdk.split.io/api/segmentChanges/mx_support_squad?since=1576088069626
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:74: >>>> Fetched segment: staging_support_android_testing
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:77: Segments returned by the server are empty
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:80: [GET] https://sdk.split.io/api/segmentChanges/us_adyen?since=1565182786497
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:80: [GET] https://sdk.split.io/api/segmentChanges/islands_team?since=1579881968011
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:80: [GET] https://sdk.split.io/api/segmentChanges/Member_Web_Team?since=1583970530099
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:109: REQUEST TIME TOOK: 41 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[7613] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[89] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:49 GMT] Etag:["1655596557--gzip"] Last-Modified:[Fri, 24 Jan 2020 16:06:08 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[HIT, HIT] X-Cache-Hits:[1, 6] X-Served-By:[cache-dca17747-DCA, cache-sea4452-SEA] X-Timer:[S1602185210.843329,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:109: REQUEST TIME TOOK: 42 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[37825] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[93] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:49 GMT] Etag:["-1135706123--gzip"] Last-Modified:[Wed, 11 Dec 2019 18:14:29 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 6] X-Served-By:[cache-dca17738-DCA, cache-sea4452-SEA] X-Timer:[S1602185210.843487,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:109: REQUEST TIME TOOK: 41 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:109: REQUEST TIME TOOK: 41 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:74: >>>> Fetched segment: mx_support_squad
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:77: Segments returned by the server are empty
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[7613] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[85] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:49 GMT] Etag:["348709715--gzip"] Last-Modified:[Wed, 07 Aug 2019 12:59:46 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[HIT, HIT] X-Cache-Hits:[1, 6] X-Served-By:[cache-dca17771-DCA, cache-sea4452-SEA] X-Timer:[S1602185210.843469,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:74: >>>> Fetched segment: islands_team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:77: Segments returned by the server are empty
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[7733] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[92] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:49 GMT] Etag:["-1615162288--gzip"] Last-Modified:[Wed, 11 Mar 2020 23:48:50 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[HIT, HIT] X-Cache-Hits:[1, 6] X-Served-By:[cache-dca17753-DCA, cache-sea4452-SEA] X-Timer:[S1602185210.843486,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:74: >>>> Fetched segment: us_adyen
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:77: Segments returned by the server are empty
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:74: >>>> Fetched segment: Member_Web_Team
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:49 fetchsegments.go:77: Segments returned by the server are empty
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:53 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602184596925
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:53 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:53 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:53 client.go:109: REQUEST TIME TOOK: 9 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:53 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[610] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[63] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:53 GMT] Etag:["-697333022--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:16:36 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 1] X-Served-By:[cache-dca17748-DCA, cache-sea4459-SEA] X-Timer:[S1602185214.572702,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:53 fetchsplits.go:96: Saved splits: 0 Removed splits: 0 of Total items: 0
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:58 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602184596925
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:58 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:58 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:58 client.go:109: REQUEST TIME TOOK: 43 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:58 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[615] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[63] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:26:58 GMT] Etag:["-697333022--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:16:36 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 4] X-Served-By:[cache-dca17748-DCA, cache-sea4439-SEA] X-Timer:[S1602185219.617963,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:26:58 fetchsplits.go:96: Saved splits: 0 Removed splits: 0 of Total items: 0
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:03 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602184596925
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:03 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:03 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:03 client.go:109: REQUEST TIME TOOK: 38 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:03 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[620] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[63] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:03 GMT] Etag:["-697333022--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:16:36 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 6] X-Served-By:[cache-dca17748-DCA, cache-sea4430-SEA] X-Timer:[S1602185224.659235,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:03 fetchsplits.go:96: Saved splits: 0 Removed splits: 0 of Total items: 0
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 counter.go:79: Posting go proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 metrics.go:36: Posting Counter Metrics for SplitSyncProducerMode-2.6.1 10.1.20.47
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:148: [POST] https://events.split.io/api/metrics/counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:154: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:168: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkmachineip:[10.1.20.47] Splitsdkmachinename:[ip-10-1-20-47] Splitsdkversion:[SplitSyncProducerMode-2.6.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 counter.go:79: Posting go proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 metrics.go:36: Posting Counter Metrics for SplitSyncProducerMode-2.6.1 10.1.20.47
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:148: [POST] https://events.split.io/api/metrics/counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:154: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:168: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkmachineip:[10.1.20.47] Splitsdkmachinename:[ip-10-1-20-47] Splitsdkversion:[SplitSyncProducerMode-2.6.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 metrics.go:16: Posting Metrics for SplitSyncProducerMode-2.6.1 10.1.20.47
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:148: [POST] https://events.split.io/api/metrics/times
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:154: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:168: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkmachineip:[10.1.20.47] Splitsdkmachinename:[ip-10-1-20-47] Splitsdkversion:[SplitSyncProducerMode-2.6.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:179: REQUEST TIME TOOK: 64 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:183: [RESPONSE_HEADERS] map[Content-Length:[0] Date:[Thu, 08 Oct 2020 19:27:04 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl0sn0vq] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:179: REQUEST TIME TOOK: 62 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:183: [RESPONSE_HEADERS] map[Content-Length:[0] Date:[Thu, 08 Oct 2020 19:27:04 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl0spe7q] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:179: REQUEST TIME TOOK: 63 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:04 client.go:183: [RESPONSE_HEADERS] map[Content-Length:[0] Date:[Thu, 08 Oct 2020 19:27:04 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl0ubuef] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:08 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602184596925
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:08 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:08 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:08 client.go:109: REQUEST TIME TOOK: 42 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:08 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[625] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[63] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:08 GMT] Etag:["-697333022--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:16:36 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 2] X-Served-By:[cache-dca17748-DCA, cache-sea4453-SEA] X-Timer:[S1602185229.703115,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:08 fetchsplits.go:96: Saved splits: 0 Removed splits: 0 of Total items: 0
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 counter.go:79: Posting go proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 metrics.go:36: Posting Counter Metrics for SplitSyncProducerMode-2.6.1 10.1.20.47
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:148: [POST] https://events.split.io/api/metrics/counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:154: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:168: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkmachineip:[10.1.20.47] Splitsdkmachinename:[ip-10-1-20-47] Splitsdkversion:[SplitSyncProducerMode-2.6.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 counter.go:79: Posting go proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 metrics.go:36: Posting Counter Metrics for SplitSyncProducerMode-2.6.1 10.1.20.47
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:148: [POST] https://events.split.io/api/metrics/counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:154: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:168: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkmachineip:[10.1.20.47] Splitsdkmachinename:[ip-10-1-20-47] Splitsdkversion:[SplitSyncProducerMode-2.6.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 metrics.go:16: Posting Metrics for SplitSyncProducerMode-2.6.1 10.1.20.47
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:148: [POST] https://events.split.io/api/metrics/times
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:154: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:168: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkmachineip:[10.1.20.47] Splitsdkmachinename:[ip-10-1-20-47] Splitsdkversion:[SplitSyncProducerMode-2.6.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:179: REQUEST TIME TOOK: 63 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:183: [RESPONSE_HEADERS] map[Content-Length:[0] Date:[Thu, 08 Oct 2020 19:27:09 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl488sdn] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:179: REQUEST TIME TOOK: 63 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:183: [RESPONSE_HEADERS] map[Content-Length:[0] Date:[Thu, 08 Oct 2020 19:27:09 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl489hbz] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:179: REQUEST TIME TOOK: 64 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:183: [RESPONSE_HEADERS] map[Content-Length:[0] Date:[Thu, 08 Oct 2020 19:27:09 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl489fku] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 metrics.go:16: Posting Metrics for SplitSyncProducerMode-2.6.1 10.1.20.47
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:148: [POST] https://events.split.io/api/metrics/times
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:154: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:168: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkmachineip:[10.1.20.47] Splitsdkmachinename:[ip-10-1-20-47] Splitsdkversion:[SplitSyncProducerMode-2.6.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:179: REQUEST TIME TOOK: 62 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:09 client.go:183: [RESPONSE_HEADERS] map[Content-Length:[0] Date:[Thu, 08 Oct 2020 19:27:09 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl4a1ls8] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:13 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602184596925
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:13 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:13 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:13 client.go:109: REQUEST TIME TOOK: 36 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:13 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[630] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[63] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:13 GMT] Etag:["-697333022--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:16:36 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 7] X-Served-By:[cache-dca17748-DCA, cache-sea4430-SEA] X-Timer:[S1602185234.741751,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:13 fetchsplits.go:96: Saved splits: 0 Removed splits: 0 of Total items: 0
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:80: [GET] https://events.split.io/api/version
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:109: REQUEST TIME TOOK: 63 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:113: [RESPONSE_HEADERS] map[Cache-Control:[no-cache, no-transform] Content-Length:[87] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:14 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Request-Id:[1inbl7zqvwl] X-Xss-Protection:[1; mode=block]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:80: [GET] https://sdk.split.io/api/version
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:109: REQUEST TIME TOOK: 43 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:14 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[1447] Cache-Control:[no-cache, no-transform] Content-Encoding:[gzip] Content-Length:[98] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:14 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 2] X-Served-By:[cache-dca17731-DCA, cache-sea4481-SEA] X-Timer:[S1602185234.173160,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:18 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602184596925
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:18 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:18 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:18 client.go:109: REQUEST TIME TOOK: 35 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:18 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[635] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[63] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:18 GMT] Etag:["-697333022--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:16:36 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, HIT] X-Cache-Hits:[0, 3] X-Served-By:[cache-dca17748-DCA, cache-sea4434-SEA] X-Timer:[S1602185239.777789,VS0,VE0]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:18 fetchsplits.go:96: Saved splits: 0 Removed splits: 0 of Total items: 0
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:23 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602184596925
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:23 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:23 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:23 client.go:109: REQUEST TIME TOOK: 109 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:23 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[0] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[609] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:23 GMT] Etag:["-1993565555--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:27:20 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, MISS] X-Cache-Hits:[0, 0] X-Served-By:[cache-dca17769-DCA, cache-sea4483-SEA] X-Timer:[S1602185244.818976,VS0,VE69]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:23 fetchsplits.go:96: Saved splits: 1 Removed splits: 0 of Total items: 1
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:131: Posting LOCAL proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:131: Posting LOCAL proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:79: Posting go proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:131: Posting LOCAL proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:131: Posting LOCAL proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:131: Posting LOCAL proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:131: Posting LOCAL proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:79: Posting go proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 counter.go:79: Posting go proxy counters
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:25 latency.go:85: Posting go proxy latencies
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:28 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602185240905
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:28 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:28 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:29 client.go:109: REQUEST TIME TOOK: 111 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:29 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[0] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[607] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:29 GMT] Etag:["2106851981--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:27:27 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, MISS] X-Cache-Hits:[0, 0] X-Served-By:[cache-dca17773-DCA, cache-sea4431-SEA] X-Timer:[S1602185249.936143,VS0,VE69]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:29 fetchsplits.go:96: Saved splits: 1 Removed splits: 0 of Total items: 1
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:34 client.go:80: [GET] https://sdk.split.io/api/splitChanges?since=1602185247094
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:34 client.go:88: Authorization [ApiKey]: ahmt...q742
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:34 client.go:98: [REQUEST_HEADERS] [REQUEST_HEADERS]map[Accept-Encoding:[gzip] Authorization:[Bearer ahmt...q742] Content-Type:[application/json] Splitsdkversion:[go-0.0.1] User-Agent:[SplitIO-GO-AGENT/0.1]][END_REQUEST_HEADERS] [END_REQUEST_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:34 client.go:109: REQUEST TIME TOOK: 129 millis
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:34 client.go:113: [RESPONSE_HEADERS] map[Accept-Ranges:[bytes] Age:[0] Cache-Control:[no-transform, max-age=60, s-maxage=60] Content-Encoding:[gzip] Content-Length:[63] Content-Type:[application/json] Date:[Thu, 08 Oct 2020 19:27:34 GMT] Etag:["-266994420--gzip"] Last-Modified:[Thu, 08 Oct 2020 19:27:27 GMT] Strict-Transport-Security:[max-age=15770000; includeSubDomains] Vary:[Accept-Encoding, Cookie] Via:[1.1 varnish 1.1 varnish] X-Cache:[MISS, MISS] X-Cache-Hits:[0, 0] X-Served-By:[cache-dca17770-DCA, cache-sea4458-SEA] X-Timer:[S1602185254.064699,VS0,VE74]] [END_RESPONSE_HEADERS]
SPLITIO-AGENT | DEBUG: 2020/10/08 19:27:34 fetchsplits.go:96: Saved splits: 0 Removed splits: 0 of Total items: 0
from ruby-client.
chillaq
commented on July 19, 2024
Hi, thanks for the log, were you able to repro the issue during the log time?
Also, could you send the Synchronizer configurations (either the config file or command line parameters)
Thanks
Bilal
from ruby-client.
brightredchilli
commented on July 19, 2024
We don't use any custom configurations, just the split synchronizer image as is, and these env vars:
- SPLIT_SYNC_API_KEY
- SPLIT_SYNC_LOG_DEBUG
- SPLIT_SYNC_LOG_STDOUT
- SPLIT_SYNC_LOG_VERBOSE
- SPLIT_SYNC_REDIS_HOST
- SPLIT_SYNC_REDIS_PASS
- SPLIT_SYNC_REDIS_TLS
from ruby-client.
chillaq
commented on July 19, 2024
Hi, thanks for the parameters, I want to confirm if the issue occurred during the debug log file time.
Thanks
Bilal
from ruby-client.
brightredchilli
commented on July 19, 2024
yes, it does.
from ruby-client.
brightredchilli
commented on July 19, 2024
Hi @chillaq, do you have any idea how we might further narrow down this issue?
from ruby-client.
chillaq
commented on July 19, 2024
Nothing yet, pinging the team now.
from ruby-client.
NicoZelaya
commented on July 19, 2024
Hi @brightredchilli !
My name is Nico and I'm part of the SDK team here at Split. We wanted to follow up on this issue.
So far we took a good look at our code for Ruby SDK and the Synchronizer and didn't find anything odd. We ran some tests for ~2 days that never hit this problem.
For now, we think the issue is related with something external that is either expiring the Split keys that were not written too recently, or just clearing part of your Redis data.
We suspect this is the case because, if the fact of updating a Split in the UI makes it visible from the SDK, the connection seems to be fine (otherwise it won't see it) and it indicates it's able to retrieve Split information.
It would also confirm the Synchronizer is connected to Redis and able to write to it, everytime we receive an update from the cloud we just override the Split data for it's given Redis key so the update was processed accordingly.
Furthermore, if you restart the Synchronizer and it fixes the issue it is because there's no record of the Split changeNumber that it can access in Redis, at the very least, for it's given configuration. I see you don't define a prefix so it should just pick it up if there.
Is it safe to assume you're not changing the SPLIT_SYNC_API_KEY value and it remained constant? If the SPLIT_SYNC_API_KEY is changed but there's information assigned on the configured prefix for a different one, we'll clear the data and resync it all to avoid mixing Splits between Split environments.
I would suggest as a first test, increasing the memory limits in your policy and test it, to see if either it fails after a longer time or doesn't fail at all. I am assuming no other process on your end could clear the Redis data.
Something that would be really beneficial for any further debugging would be to get the output of a keys SPLITIO* command on Redis after a successful restart, and then the output of the same command when it can't find Splits anymore.
If after the failure you update a Split and it starts being returned by the Ruby SDK, issue the same command again so we can confirm which other keys are present apart from the one containing this Split.
Hope this helps you narrow down the issue, let us know how we can be of further assistance.
Thanks in advance,
Nico.
from ruby-client.
brightredchilli
commented on July 19, 2024
Hi @NicoZelaya, thanks for the response. We are bumping up our redis tier, it was previously at 2.3gb and realized that it had been hitting some OOM limits. What is the recommended memory size for the redis instance that split connects to?
from ruby-client.
sanzmauro
commented on July 19, 2024
Hi @brightredchilli !
We don't really feel comfortable recommending an instance size until we figure out the cause of the keys eviction. But for reference, we use a t2.micro instance (1gb ram) shared across many different testing accounts (each with a different prefix), with more than 1.5k splits and counting. Please keep in mind that the usage is not super-realisting since the amount of impressions & events generated is low compared to a real application.
Before changing your redis instance we'd like to try to figure out the root cause of the split-related keys being evicted. In order to get more insights on this, we need to now the state of the redis instance/db the sync/sdks are using when the problem starts happening.
We suspect that impressions and/or events might be queuing up in redis faster than they're flushed to the split servers
It would be of great help if you could send us the output of keys * after you start seeing this failures, prior to restarting the synchronizer or flushing the redis instance.
Also, the split-synchronizer exposes a couple of admin endpoints that might prove helpful in this situation:
- /admin/events/queueSize
- /admin/impressions/queueSize
The above return a JSON response indicating the number of impressions/events that are still on redis pending to be sent. If you could monitor these endpoints say every minute for some time while the app is running, we might be able to see that the number is constantly increasing over time, hence taking up more memory from redis. (admin port is 3010 by default, you will need to either expose in the container or ssh into it, and execute curl locally)
We think making your instance bigger will most likely help, but if there's too big of an offset between the rate at which you generate impressions (or events), and the rate at which you evict them, it's likely that we might be just delaying the issue. The synchronizer can be fine tuned via config options to solve the issue if this is the case.
Another possibility would be you having a large number of really big segments, that could put more pressure on redis to evict other keys, whenever they're queried or updated. If we're dealing with the latter, then increasing the redis instance capacity will most likely be necessary. We can help you validate this if you reach out via [email protected] with your organization information.
from ruby-client.
Related Issues (20)
- JRuby support HOT 4
- [redacted] please ignore
- [redacted]
- cache_ttl in consumer mode HOT 1
- splits_repository.splits is significantly more costly than splits_repository.split_names HOT 2
- `fix_latencies` is running every restart by each app server, affecting Redis performance HOT 4
- Redis.exists? deprecation warning from redis v4.2.2 HOT 3
- block_until_ready doesn't timeout if the api key is invalid HOT 9
- Quickstart: SDK Setup code does not work HOT 4
- 7.2.3 performance issues HOT 5
- High CPU usage on wrong split key HOT 4
- Update Rake version to support development on Ruby 3 HOT 2
- Update CI to target non-EOL Ruby HOT 2
- Split logger errors HOT 2
- SDK not sync configuration changes in Rails console HOT 3
- Faraday 2.0+ compatibility HOT 5
- Splitio: #IOError: stream closed in another thread HOT 5
- Stream Event Parsing Silently Failing and Not Falling Back to Polling HOT 3
- Error during a parsing event HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ruby-client.