datastax / dsbench-labs

DSBench - A Database Testing Power Tool

DSBench is obsolete and no longer actively developed or maintained; leverage NoSQLBench instead

Refer to the following resources to find additional information about NoSQLBench, An Open Source, Pluggable, NoSQL Benchmarking Suite.

• Introduction blog: https://www.datastax.com/blog/2020/03/nosqlbench
• Github repo: https://github.com/nosqlbench/nosqlbench
• NoSQLBench website: http://docs.nosqlbench.io/#/
• Comparing stress tools for Apache Cassandra™

DSBench

DSBench allows you to run realistic workloads against your CQL systems without having to build your own testing harness. It builds on a set of core testing concepts and machinery to give you a performance testing power suit. Specifically, it allows you to control and customize your client operations so that they are representative of a real workload that you would run with your application. If you don't want to control for this level of detail, you can use one of the built-in workloads with a single command.

This repository is where we manage issues, feature requests, and releases for DSBench.

DSBench is released as a Linux compatible binary. Future releases may include support for Mac or Windows.

Supported Systems

In this initial release, DSBench supports CQL based systems and console output. In dsbench terms, this means:

• The CQL activity type
  • The initial release of the CQL activity type uses the DataStax driver version 1.9.0
• The stdout activity type. (this is a way to see operations in diagnostic mode)

Activity types are how DSBench gets its support for different protocols or client drivers. The initial release of DSBench includes only two, but additional types will be supported in future releases.

Basic Examples give a preview of some of the things you can do with DSBench and how it works. The real docs are included in the tool itself.

Getting Started

On Linux systems:

1. Download dsbench from DSBench Releases
2. Make it executable: chmod +x dsbench
3. Start dsbench in docserver mode: dsbench docserver http://0.0.0.0:12345/
4. Browse to the system you are running it on and you will be presented with the Guidebook.

On other systems:

1. Download dsbench.jar from DSBench Releases
2. Install a Java 12 or newer JDK.
3. Start dsbench in docserver mode: java -jar dsbench.jar docserver http://0.0.0.0:12345/
4. Browse to the system you are running it on and you will be presented with the Guidebook.

Running DSBench on docker

The Guidebook provides the most essential place to get access to the docs.

Release Plans are how we plan to prioritize and scope releases going forward.

Support

Community Support

DSBench is released as a DataStax Labs tech preview.

It is supported by a community of active users at DataStax DSBench Community.

Bug Fixes

If you think you have found a bug, you may file a bug report. DSBench is actively used within DataStax, and verified bugs will get attention as resources permit.

Feature Requests

If you would like to see something in DSBench that is not there yet, please submit a feature request.

Sponsors

DSBench development is sponsored by DataStax.

DataStax offers a suite of distributed data management products and cloud services powered by Apache Cassandra. We simplify achieving zero downtime for enterprise apps that are active everywhere–DataStax delivers the ultimate hybrid and multi-cloud database. For more information, visit www.DataStax.com and follow us on @DataStax

dsbench-labs's Issues

Activity Parameter terminology in the Docs

Clarify core parameters type specific activity parameters.

Use GraalVM instead of Nashorn

Since Nashorn is being deprecated from the JVM, we will be switching over to the GraalVM approach.

Tagged Metrics

The current integration with the docker metrics feature uses graphite and a graphite forwarder in order to load metrics into prometheus. This makes differentiation and distinction of different metrics sources more difficult because of name/path overloading issues.

A better approach would be to provide metrics tags for each metrics instance, sending these over a transport that supports metrics tagging explicitly. Along with this change, the docker dashboards should be updated in order to provide access to the richer and cleaner functionality offered by tagging.

Guidebook - error page link seems to be wrong

Copy link on one system yielded this instead of the correct link:

http://10.101.32.78:12345/index.html#/docs/https://github.com/datastax/dsbench-labs/issues/https://github.com/datastax/dsbench-labs/issues/https://github.com/datastax/dsbench-labs/issues/https://github.com/datastax/dsbench-labs/issues/https://github.com/datastax/dsbench-labs/issues/new?labels=APPUSE,UX,documentation

guidebook - Allow UI preferences to persist across pages

The guidebook menu works fairly well, but the "keep open" setting isn't not integrated yet with localStorage and Vuex.

Function reference should be included in docs

The function reference available on virtdata does not include the downstream functions that are included in dsbench.

initial cql-iot.yaml has DSE-specific options

Two versions of this file may need to be maintained so that you can have the best version of the workload for your cluster type. For now, a cql-iot-oss version will be added.

Community site staging

We need a community space created on the community.datastax.com site so that we can support new users, answer questions, and be responsive in general to those wanting to learn how to use dsbench.

If possible, the site should be nominative, like http://community.datastax.com/dsbench, rather than a simple numbered space.

This site should not be active by default. It needs to be created and linked in as part of the initial dsbench release.

• Initial discussion with community site admins about naming and release timing
• staging of site in disabled form (not possible)
• internal testing, if possible of staged site (not possible)
• cross-links as support resource form other docs once the URL is set

Guidebook doesn't bind to outside address

Users who are accessing the guidebook from a different system than they run dsbench on will want an ability to share the guidebook on an external socket. \

A --public option should be provided to bind the guidebook to the 0.0.0.0 address, and a --listen-on <addr> option should be provided to let them bind to any local socket.

Upgrade Java Driver

The newest Java driver supports all server types. This should be used so that users don't have to deal with swapping driver versions or setting additional parameters. Further, testing with the current driver means that users will be loading their systems more like their applications would, to the degree that the driver behavior may be different or improved.

Guidebook - pin menu state should persist

Guidebook - The initial page doesn't load a global index.md

Category paths can now load a proxy index.md content pane, but there is no simple content for the main entry, the index.md at the root of the doc content. This needs to be added.

Simplify Logging System

Logback and SLF4J haven't actually helped keep sanity in the logging integration. Log4J version 2 has matured enough to meet all requirements, including programmatic configuration. Using it across all the modules will simplify configuration and work better with most other run-time dependencies. If needed, SLF4j can be used in front of log4j2 in the future, but for now the intent is to completely collapse the API to just Log4j2

Internal notice and review before public release

The internal dsbench community needs to have fair notice and review time to help sanity check the dsbench artifact before it is initially released.

• internal notice to stake holders and community advocates
• sign-off of stake holders and community advocates, no red flags
• internal sanity checks of core functionality
• internal feedback on docs and next steps for improvements

push initial release to DataStax Labs dockerhub

depends on #14

UI: Scenario Builder

guidebook - page caching

Presently, for any page loaded to a client, all of the markdown files have to be loaded and parsed. This is very inefficient and makes the user experience worse with larger sets of documentation. The markdown services already provide a cache version which can be used to avoid reloading for hot-load scenarios.

This enhancement is to read and cache the markdown on the client, and, when a reload is requested, to re-read the cache version from the service and avoid doing a full reload and parse when possible.

UI: Result Visualization

Teaser Video for dsbench first time user, getting started in 1 minute

Initial guidebook content

The docs we have already collected for the new user experience in the guidebook need to be adapted and edited for the guidebook app.

• curation of existing content
• editing and reformatting
• sanity check all front matter and ordering
• initial sanity check of doc-guided first steps walkthrough

Open new space at comunity.datastax.com/labs/__ for dsbench

Guidebook - Add support for TLS (https)

In the initial release, only http://... is supported for a user-provided binding address.

no arguments on the command line causes an error

If you don't give it any arguments, it throws an exception. Easy fix: Adding it to the issues.
I removed an outdated contextual help file, but forgot to remove the auto-hook that displays it from scripts/auto/default.js

The fix is to remove this file from the artifact and retest.

Guidebook - Additional Docs to add

• How to share workloads on the community site.
• How to find built-in workloads.
• A description of the built-in workloads.
• Explanation of ecosystem, including internal tooling and incubation
• Links to other places where dev-level contributors can help upstream

Guidebook - docserver app assumes localhost for services

This is a blocker for release

docker-metrics and docserver artifact version conflicts

There is a runtime error that occurs because of conflicting dependency version in some of the upstream libraries, namely between netty/jetty as used by java-docker and netty/jetty as used by the docserver. Further, it is causing impact to a partner team who is wanting to use the current version of [ebdse/dsbench] for their own testing. (this partner already has permission to use ebdse)

Notify Erick Ramirez for Community cross-post

Send an announcement to the users list

Support Windows and Mac with binary artifacts

Guidebook - highlight selected category

When a category is selected and its summary content is what is displayed in the main content area, it should remain lit so users know what the focus is for that content.

DocMenu Ideas

• Indent or recolor sub-menu items to show they're submenu items (nested inside groups)
• Fix UX on index.md

DataStax Labs docker support

Sebastian has already added a docker file.

• dockerfile preparation

Continuous Integration for releases

After the initial release, major releases need to be automated.

• automation for any version updates for the DataStax labs downloads
• automation for docker publishing
• automation for adding to dsbench-labs

Guidebook -

Since the guidebook is an app which uses doc sources as services from DSBench, and the user can configure the service URL to connect to the guidebook, these service endpoints (port, specifically) need to be auto configured together. For now, service port 12345 is required to make it work well.

• Allow the host VM to pass URIs to Nuxt environment
  • Provide template based configuration in static views
  • Inject basic service URIs into hosted apps

guidebook - topic ordering

The topics within the menu need to follow an order that makes sense for users. This issues is for adding support in the guidebook to honor a weight field in the metadata.

• add support for ordering entries within a topic during rendering
• add support for extracting the weight from the frontmatter
• sanity testing

Configurable color theme for docs view

This is low priority: It might be a good idea to look at color choices from IntelliJ, or Visual Studio Code, or the JavaDocs webpage, to find inspirations on a set of colors for different UI elements in the docs view. But given that color choices more depend on personal taste, it might be better to make the color theme configurable.

Naming Sanity: rename all docserver references to guidebook

It's better to consolidate the naming of this before rather than after the release.

Initial repo docs for new users

The README, and other docs which explain the release conventions and general project status need to be completed on this dsbench-labs repo.

Scope of docs should include at the minimum:

• Description of DSBench functionality - why people should use it.
• Initial getting started docs that point users to the guidebook and examples.
• Showcase of DataStax sponsorship and links to products etc, as appropriate.
  • Explanation of support available.
• Links to supporting resources, like the community site.
• A description of release plans, how, when, ...
  • A brief roadmap, including big things users will be looking forward to
  • scope balancing conventions: UX, TECH, other

Docs - Workload Construction Guide

Add a docs guide that tells users how to

1. select or construct bindings
2. verify and troubleshoot bindings
3. configure statements
4. map bindings and statements together

Grafana - enable share with datastax functionality

The docker metrics feature can be leveraged to allow a user to share their results with us.

guidebook - self-standing app

The current guidebook service integration assumes a shared service URI that can resolve resources as path-based resources. This means that separate focused iteration on the app is not possible, which increases iteration and testing time dramatically.

This change is to allow the docs app to work completely independently inside its own development space, accessing the doc services via CORS. CORS is already supported on the docs, so this is a minimal change to pathing and service URIs in the guidebook app.

The result will be a significant increase in developer agility. It will also allow embedded UIs to be developed easily as with any modern web framework, using the guidebook as a reference point.

• carve guidebook into seperate app for tooling and initial development
• complete initial phase of guidebook refinement separately
• embed guidebook back into main project and re-test

Improve docker landing page

Using information from this guide:
https://docs.docker.com/docker-hub/publish/publish/