Comments (15)
@zuiderkwast Did we secretly do this?
I must have pocket-clicked close...
from valkey.
It would be great if anyone wants to add such tests, because we're a bit overloaded at this point. It doesn't have to be in TCL (maybe Python instead?) and it doesn't even have to be in this repo, if we're testing different forks. We can make a separate interop testing repo. It is about
- cluster bus (nodes of different versions in the same cluster)
- replication (primary/replica of different versions)
- RDB and AOF files (read and write by different versions)
- Anything else?
from valkey.
@valkey-io/core-team We chatted a bit in the core meeting and want to have a vote if we want to pull in the compatibility test listed above and run it as part of the CI.
We still need a separate effort to do interversion operability tests (clusterbus and replication).
from valkey.
@valkey-io/core-team WDYT ?
from valkey.
i am willing to spend time. if anyone guide me some starting point.
from valkey.
We can make a separate interop testing repo. It is about
I think it should be this repo, I don't think we want to start introducing tests elsewhere for now.
from valkey.
I'm fine either way, but what id'd like to see is some prototyping to get something running. Use any language, any client lib, but be ready to discard it later.
from valkey.
There are potential benefits of having a completely separate repo for this. It would need to check out various versions of valkey anyway (so it can't just run out of the checked out repo). It could also test valkey against various forks and versions. We can have test a cluster with mixed nodes of KeyDB, Dragonfly, Redict, Redis, Valkey, various versions. We can run redis testsuites on the valkey binary. Etc.
from valkey.
@madolson @zuiderkwast @pragnesh @mattsta Hi everyone! Is anyone already working on this?
We currently have a Python test script that provides the same functionality as the TCL tests, but written in Python. It has been working well for the past two years and has significantly improved the efficiency of writing and debugging tests compared to TCL.
Here is a relatively old version: https://pypi.org/project/pybbt/
If you're interested, I'm willing to spend 1-2 weeks adapting it for Valkey and writing a few test examples.
from valkey.
@suxb201 how is the test setup/teardown done? How long does a similar tcl test takes in python?
from valkey.
- To create an instance, use a clear statement.
- Instances made inside an
@subcase
are safely destroyed when the@subcase
ends. - Instances created within an
@case
will be destroyed when the@case
concludes. - The
@subcase
is concurrent, which makes testing efficient. - There is only one
@case
per file, and the tests are composed of multiple files. - Logs, AOF and RDB files will be arranged in a temporary directory, making it easier to debug.
Here's a example:
from testsuite import *
# Master can replicate command longer than client-query-buffer-limit on replica
@subcase()
def replication_query_buffer_limit():
t0 = Tair() # create a process
t1 = Tair() # create another process
t1.wait_slaveof(t0) # call 'slaveof' and then wait for the replication connection to be set up
t0.do("config", "set", "client-query-buffer-limit", 2000000)
t1.do("config", "set", "client-query-buffer-limit", 1048576) # 1024*1024 = 1mb
value = "x" * 1100000
ASSERT_TRUE(t0.do("set", "k", value)) # 2000000 > 1100000 > 1048576
t0.wait_consistent()
ASSERT_EQ(t1.do("get", "k"), value.encode())
ASSERT_EQ(t1.digest(), t0.digest())
@case(tags=["flag0", "flag1", "flag2"])
def main():
replication_query_buffer_limit()
if __name__ == '__main__':
main()
from valkey.
It seems to me that porting the tests to a different language is a different topic. I think it can be discussed here: #94.
Personally, I think it would be beneficial to create a separate interoperability-testing repo where scripts and CI jobs can
- check out multiple versions of Valkey, Redis, clients and other tools
- test mixed clusters, e.g. test #52
- test replication between different versions
- create dumps (rdb and aof) and load them it in other versions, newer and older
from valkey.
Reference: https://github.com/tair-opensource/resp-compatibility. Alibaba has some compatibility testing already we should investigate.
from valkey.
I am the owner of resp-compatibility. It was originally designed to detect the Redis version compatible with the Redis-Like system from the interface, but it is also very suitable for version compatibility testing of Redis/Valkey itself. From the discussion in the issue, it can currently do
tests | YES or NO |
---|---|
api | YES |
cluster bus (nodes of different versions in the same cluster) | NO(But can improve) |
replication (primary/replica of different versions) | NO(But can improve) |
RDB and AOF files | YES |
We can:
- Run on daily-ci.
- Run this test when a new version is released.
- Run every time a PR is submitted (about 2 minutes).
Please share any thoughts you have, thank you.
from valkey.
@zuiderkwast Did we secretly do this?
from valkey.
Related Issues (20)
- [NEW] Enhanced Function API: off-thread JSON/MsgPack decoding of args and encoding of response HOT 1
- [Daily CI] Based line for Daily CI status (All test cases can passed)
- [BUG] `CLIENT CAPA redirect` may result in bouncing redirects during failover HOT 10
- [BUG]In big cluster Redis (Valkey) replica node is not able to sync since 6.2.11 HOT 2
- [NEW] Introduce server level network ingress / egress metrics for cluster bus. HOT 1
- [BUG] Replica can run cross-slot MULTI/EXEC, upon module hooks on primary HOT 2
- [Test failure] IO error with TLS
- [BUG] The result of the PING command does not match the documentation when using RESP3 HOT 1
- [BUG] A large negative number(e.g. -LONG_MAX) in HRANDFIELD/ZRANDMEMBER can easily cause Valkey to crash. HOT 1
- [NEW] Migrate from Redis 7.4.0 to Valkey 7.2.5 HOT 9
- [NEW] Introduce slot-level memory metrics
- [Test failure] Test issues with empty shard migrations
- [BUG] "CLIENT CAPA redirect" has no effect in scripts HOT 1
- [BUG] Cluster: Inconsistent MOVED/READONLY errors for read-only client
- [NEW] Introduce a new client capability for throttling
- [NEW] Add support for setting the group on a unix domain socket HOT 1
- Feature Request: Add Error Status Field for Diskless Syncs HOT 4
- [NEW] HTTP support for Valkey HOT 4
- [NEW] Support proxy protocol HOT 2
- Migrate TLS.md into README.md HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from valkey.