Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(scylladb): add scylladb provider #2919

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
672e6a7
feat(scylladb): add base packages
danielhe4rt Nov 21, 2024
41ed131
feat(scylladb): container implementation with file config and shard-a…
danielhe4rt Nov 21, 2024
0003747
feat(scylladb): container base test implementation with files and sha…
danielhe4rt Nov 21, 2024
6a472cb
feat(scylladb): add makefile test runner command
danielhe4rt Nov 21, 2024
264d462
feat(scylladb): add alternator integration
danielhe4rt Nov 22, 2024
680c279
feat(scylladb): add WithCustomCommand and code cleanup
danielhe4rt Nov 22, 2024
a995a1c
feat(scylladb): implementation cleanup with test coverage
danielhe4rt Nov 22, 2024
8f33187
docs(scylladb): add scylladb base documentation module
danielhe4rt Nov 22, 2024
7559c61
docs: add links to the version it was introduced
mdelapenya Nov 26, 2024
75de222
chore: add scaffolding files
mdelapenya Nov 26, 2024
37ca8fc
chore: rename to Container
mdelapenya Nov 26, 2024
7f7a476
chore: run mod tidy
mdelapenya Nov 26, 2024
e780e4f
fix: wrong state after code generation
mdelapenya Nov 26, 2024
95ca13f
chore: concat strings
mdelapenya Nov 26, 2024
01a4fef
chore: use int
mdelapenya Nov 26, 2024
4388885
chore: calculate variable once
mdelapenya Nov 26, 2024
c0f97ce
chore: simplify types
mdelapenya Nov 26, 2024
82e5352
chore(scylladb): WithConfig standardising
danielhe4rt Dec 18, 2024
cf070be
Merge branch 'testcontainers:main' into main
danielhe4rt Jan 8, 2025
90caba2
Merge branch 'testcontainers:main' into main
danielhe4rt Jan 10, 2025
aabf4d9
chore(scylladb): WithConfig enhancements
danielhe4rt Jan 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ jobs:
matrix:
go-version: [1.22.x, 1.x]
platform: [ubuntu-latest]
module: [artemis, azurite, cassandra, chroma, clickhouse, cockroachdb, compose, consul, couchbase, databend, dolt, dynamodb, elasticsearch, etcd, gcloud, grafana-lgtm, inbucket, influxdb, k3s, k6, kafka, localstack, mariadb, meilisearch, milvus, minio, mockserver, mongodb, mssql, mysql, nats, neo4j, ollama, openfga, openldap, opensearch, postgres, pulsar, qdrant, rabbitmq, redis, redpanda, registry, surrealdb, valkey, vault, vearch, weaviate, yugabytedb]
module: [artemis, azurite, cassandra, chroma, clickhouse, cockroachdb, compose, consul, couchbase, databend, dolt, dynamodb, elasticsearch, etcd, gcloud, grafana-lgtm, inbucket, influxdb, k3s, k6, kafka, localstack, mariadb, meilisearch, milvus, minio, mockserver, mongodb, mssql, mysql, nats, neo4j, ollama, openfga, openldap, opensearch, postgres, pulsar, qdrant, rabbitmq, redis, redpanda, registry, scylladb, surrealdb, valkey, vault, vearch, weaviate, yugabytedb]
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: ${{ matrix.go-version }}
Expand Down
4 changes: 4 additions & 0 deletions .vscode/.testcontainers-go.code-workspace
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,10 @@
"name": "module / registry",
"path": "../modules/registry"
},
{
"name": "module / scylladb",
"path": "../modules/scylladb"
},
{
"name": "module / surrealdb",
"path": "../modules/surrealdb"
Expand Down
131 changes: 131 additions & 0 deletions docs/modules/scylladb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# ScyllaDB

Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

## Introduction

The Testcontainers module for ScyllaDB, a NoSQL database fully compatible with Apache Cassandra and DynamoDB, allows you
to create a ScyllaDB container for testing purposes.

## Adding this module to your project dependencies

Please run the following command to add the ScyllaDB module to your Go dependencies:

```shell
go get github.com/testcontainers/testcontainers-go/modules/scylladb
```

## Usage example

<!--codeinclude-->
[Creating a ScyllaDB container](../../modules/scylladb/examples_test.go) inside_block:ExampleRun
<!--/codeinclude-->

## Module Reference

### Run function

- Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

The ScyllaDB module exposes one entrypoint function to create the ScyllaDB container, and this function receives three parameters:

```golang
func Run(ctx context.Context, img string, opts ...testcontainers.ContainerCustomizer) (*ScyllaDBContainer, error)
```

- `context.Context`, the Go context.
- `string`, the Docker image to use.
- `testcontainers.ContainerCustomizer`, a variadic argument for passing options.

!!! info
By default, we add the `--developer-mode=1` flag to the ScyllaDB container to disable the various checks Scylla
performs.
Also In scenarios in which static partitioning is not desired - like mostly-idle cluster without hard latency
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: typo

Suggested change
Also In scenarios in which static partitioning is not desired - like mostly-idle cluster without hard latency
Also in scenarios in which static partitioning is not desired - like mostly-idle cluster without hard latency

requirements, the --overprovisioned command-line option is recommended. This enables certain optimizations for ScyllaDB
to run efficiently in an overprovisioned environment. You can change it by using the `WithCustomCommand` function.

### Container Options

When starting the ScyllaDB container, you can pass options in a variadic way to configure it.

#### Image

If you need to set a different ScyllaDB Docker image, you can set a valid Docker image as the second argument in the
`Run` function. Eg:

```golang
scylladb.Run(context.Background(), "scylladb/scylla:6.2.1")
// OR
scylladb.Run(context.Background(), "scylladb/scylla:5.6")
```

{% include "../features/common_functional_options.md" %}

#### With Database Configuration File (scylla.yaml)

- Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

In the case you have a custom config file for ScyllaDB, it's possible to copy that file into the container before it's
started, using the `WithConfigFile(cfgPath string)` function.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: let's use the WithConfig(io.Reader) method here, as suggested in the review comments. It's also applied to the existing tests, so we should change the method call in there.


<!--codeinclude-->
[With Config YAML](../../modules/scylladb/examples_test.go) inside_block:runScyllaDBContainerWithConfigFile
<!--/codeinclude-->
!!!warning
You should provide a valid ScyllaDB configuration file when using the function, otherwise the container will fail to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: see above

Suggested change
You should provide a valid ScyllaDB configuration file when using the function, otherwise the container will fail to
You should provide a valid ScyllaDB configuration file as an `io.Reader` when using the function, otherwise the container will fail to

start. The configuration file should be a valid YAML file and follows
the [ScyllaDB configuration file](https://github.com/scylladb/scylladb/blob/master/conf/scylla.yaml).

#### With Shard Awareness

- Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

If you want to test ScyllaDB with shard awareness, you can use the `WithShardAwareness` function. This function will
configure the ScyllaDB container to use the `19042` port and ask the container to wait until the port is ready.

<!--codeinclude-->
[With Shard Awareness](../../modules/scylladb/examples_test.go) inside_block:runScyllaDBContainerWithShardAwareness
<!--/codeinclude-->

#### With Alternator (DynamoDB Compatible API)

- Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

If you want to test ScyllaDB with the Alternator API, you can use the `WithAlternator` function. This function will
configure the ScyllaDB container to use the port any port you want and ask the container to wait until the port is
ready.
By default, you can choose the port `8000`.

<!--codeinclude-->
[With Alternator API](../../modules/scylladb/examples_test.go) inside_block:runScyllaDBContainerWithAlternator
<!--/codeinclude-->

#### With Custom Commands

- Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

If you need to pass any flag to the ScyllaDB container, you can use the `WithCustomCommand` function. This also rewrites
predefined commands like `--developer-mode=1`. You can check
the [ScyllaDB Docker Best Practices](https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/tips/best-practices-scylla-on-docker.html) for more information.

<!--codeinclude-->
[With Custom Commands](../../modules/scylladb/examples_test.go) inside_block:runScyllaDBContainerWithCustomCommands
<!--/codeinclude-->

### Container Methods

The ScyllaDB container exposes the following methods:

#### ConnectionHost

- Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

This method returns the host and port of the ScyllaDB container, depending on the feature you want.
If you just want to test it with a single node and a single core, you can use port `9042`. However, if you're planning
to
more than one core, you should use the **shard-awareness** port `19042`. If you're planning to use the **Alternator**?
API, you should use the port you select in the `WithAlternator` function.

<!--codeinclude-->
[Get connection host](../../modules/scylladb/examples_test.go) inside_block:BaseConnectionHost
<!--/codeinclude-->
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ nav:
- modules/redis.md
- modules/redpanda.md
- modules/registry.md
- modules/scylladb.md
- modules/surrealdb.md
- modules/valkey.md
- modules/vault.md
Expand Down
5 changes: 5 additions & 0 deletions modules/scylladb/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
include ../../commons-test.mk

.PHONY: test
test:
$(MAKE) test-scylladb
226 changes: 226 additions & 0 deletions modules/scylladb/examples_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
package scylladb_test

import (
"context"
"fmt"
"log"
"path/filepath"

"github.com/gocql/gocql"

"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/modules/scylladb"
)

func ExampleRun_withCustomCommands() {
ctx := context.Background()

// runScyllaDBContainerWithCustomCommands {
scyllaContainer, err := scylladb.Run(ctx,
"scylladb/scylla:6.2",
scylladb.WithCustomCommand("--memory", "1G"),
scylladb.WithCustomCommand("--smp", "2"),
)
// }
defer func() {
if err := testcontainers.TerminateContainer(scyllaContainer); err != nil {
log.Printf("failed to terminate container: %s", err)
}
}()
if err != nil {
log.Printf("failed to start container: %s", err)
return
}

state, err := scyllaContainer.State(ctx)
if err != nil {
log.Printf("failed to get container state: %s", err)
return
}

fmt.Println(state.Running)

connectionHost, err := scyllaContainer.ConnectionHost(ctx, 9042) // Non ShardAwareness port
if err != nil {
log.Printf("failed to get connection host: %s", err)
return
}

runGoCQLExampleTest(connectionHost)

// Output:
// true
}

func ExampleRun_withAlternator() {
ctx := context.Background()

// runScyllaDBContainerWithAlternator {
scyllaContainer, err := scylladb.Run(ctx,
"scylladb/scylla:6.2",
scylladb.WithAlternator(8000), // Choose which port to use on Alternator
)
// }
defer func() {
if err := testcontainers.TerminateContainer(scyllaContainer); err != nil {
log.Printf("failed to terminate container: %s", err)
}
}()
if err != nil {
log.Printf("failed to start container: %s", err)
return
}

state, err := scyllaContainer.State(ctx)
if err != nil {
log.Printf("failed to get container state: %s", err)
return
}

fmt.Println(state.Running)

connectionHost, err := scyllaContainer.ConnectionHost(ctx, 8080) // Alternator port
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: in L61 we are setting the alternator port as 8000. Is this correct?

Suggested change
connectionHost, err := scyllaContainer.ConnectionHost(ctx, 8080) // Alternator port
connectionHost, err := scyllaContainer.ConnectionHost(ctx, 8000) // Alternator port

if err != nil {
log.Printf("failed to get connection host: %s", err)
return
}

runGoCQLExampleTest(connectionHost)

// Output:
// true
}

func ExampleRun_withShardAwareness() {
ctx := context.Background()

// runScyllaDBContainerWithShardAwareness {
scyllaContainer, err := scylladb.Run(ctx,
"scylladb/scylla:6.2",
scylladb.WithShardAwareness(),
)
// }
defer func() {
if err := testcontainers.TerminateContainer(scyllaContainer); err != nil {
log.Printf("failed to terminate container: %s", err)
}
}()
if err != nil {
log.Printf("failed to start container: %s", err)
return
}

state, err := scyllaContainer.State(ctx)
if err != nil {
log.Printf("failed to get container state: %s", err)
return
}

fmt.Println(state.Running)

connectionHost, err := scyllaContainer.ConnectionHost(ctx, 19042) // ShardAwareness port
if err != nil {
log.Printf("failed to get connection host: %s", err)
return
}

runGoCQLExampleTest(connectionHost)

// Output:
// true
}

func ExampleRun_withConfigFile() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: as above, we probably want to use the WithConfig(io.Reader) here

ctx := context.Background()

// runScyllaDBContainerWithConfigFile {
scyllaContainer, err := scylladb.Run(ctx,
"scylladb/scylla:6.2",
scylladb.WithConfigFile(filepath.Join("testdata", "scylla.yaml")),
)
// }
defer func() {
if err := testcontainers.TerminateContainer(scyllaContainer); err != nil {
log.Printf("failed to terminate container: %s", err)
}
}()
if err != nil {
log.Printf("failed to start container: %s", err)
return
}
// }

state, err := scyllaContainer.State(ctx)
if err != nil {
log.Printf("failed to get container state: %s", err)
return
}

fmt.Println(state.Running)

connectionHost, err := scyllaContainer.ConnectionHost(ctx, 9042) // Non ShardAwareness port
if err != nil {
log.Printf("failed to get connection host: %s", err)
return
}

runGoCQLExampleTest(connectionHost)

// Output:
// true
}

func ExampleRun() {
// runBaseScyllaDBContainer {
ctx := context.Background()

scyllaContainer, err := scylladb.Run(ctx,
"scylladb/scylla:6.2",
scylladb.WithShardAwareness(),
)
defer func() {
if err := testcontainers.TerminateContainer(scyllaContainer); err != nil {
log.Printf("failed to terminate container: %s", err)
}
}()
if err != nil {
log.Printf("failed to start container: %s", err)
return
}
// }

state, err := scyllaContainer.State(ctx)
if err != nil {
log.Printf("failed to get container state: %s", err)
return
}

fmt.Println(state.Running)

// BaseConnectionHost {
connectionHost, err := scyllaContainer.ConnectionHost(ctx, 9042)
// }
if err != nil {
log.Printf("failed to get connection host: %s", err)
return
}
runGoCQLExampleTest(connectionHost)

// Output:
// true
}

func runGoCQLExampleTest(connectionHost string) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: let's make this function to return the error, and fail the examples calling it if non nil.

cluster := gocql.NewCluster(connectionHost)
session, err := cluster.CreateSession()
if err != nil {
log.Printf("failed to create session: %s", err)
}
defer session.Close()

var driver string
err = session.Query("SELECT driver_name FROM system.clients").Scan(&driver)
if err != nil {
log.Printf("failed to query: %s", err)
}
}
Loading
Loading