Skip to content

Redpanda

Since v0.20.0

Introduction

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM! This Testcontainers module provides three APIs:

  • Kafka API
  • Schema Registry API
  • Redpanda Admin API

Adding this module to your project dependencies

Please run the following command to add the Redpanda module to your Go dependencies:

go get github.com/testcontainers/testcontainers-go/modules/redpanda

Usage example

ctx := context.Background()

redpandaContainer, err := redpanda.Run(ctx,
    "docker.redpanda.com/redpandadata/redpanda:v23.3.3",
    redpanda.WithEnableSASL(),
    redpanda.WithEnableKafkaAuthorization(),
    redpanda.WithEnableWasmTransform(),
    redpanda.WithBootstrapConfig("data_transforms_per_core_memory_reservation", 33554432),
    redpanda.WithBootstrapConfig("data_transforms_per_function_memory_limit", 16777216),
    redpanda.WithNewServiceAccount("superuser-1", "test"),
    redpanda.WithNewServiceAccount("superuser-2", "test"),
    redpanda.WithNewServiceAccount("no-superuser", "test"),
    redpanda.WithSuperusers("superuser-1", "superuser-2"),
    redpanda.WithEnableSchemaRegistryHTTPBasicAuth(),
)
defer func() {
    if err := testcontainers.TerminateContainer(redpandaContainer); err != nil {
        log.Printf("failed to terminate container: %s", err)
    }
}()
if err != nil {
    log.Printf("failed to start container: %s", err)
    return
}

Module Reference

Run function

Info

The RunContainer(ctx, opts...) function is deprecated and will be removed in the next major release of Testcontainers for Go.

The Redpanda module exposes one entrypoint function to create the Redpanda container, and this function receives three parameters:

func Run(ctx context.Context, img string, opts ...testcontainers.ContainerCustomizer) (*RedpandaContainer, error)
  • context.Context, the Go context.
  • string, the Docker image to use.
  • testcontainers.ContainerCustomizer, a variadic argument for passing options.

Image

Use the second argument in the Run function to set a valid Docker image. In example: Run(context.Background(), "docker.redpanda.com/redpandadata/redpanda:v23.1.7").

Container Options

When starting the Redpanda container, you can pass options in a variadic way to configure it.

TLS Encryption

If you need to enable TLS use WithTLS with a valid PEM encoded certificate and key.

Additional Listener

There are scenarios where additional listeners are needed, for example if you want to consume/from another container in the same network

You can use the WithListener option to add a listener to the Redpanda container.

ctr, err := redpanda.Run(ctx,
    "redpandadata/redpanda:v23.2.18",
    network.WithNetwork([]string{"redpanda-host"}, rpNetwork),
    redpanda.WithListener("redpanda:29092"), redpanda.WithAutoCreateTopics(),
)

Container defined in the same network

kcat, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
    ContainerRequest: testcontainers.ContainerRequest{
        Image: "confluentinc/cp-kcat:7.4.1",
        Networks: []string{
            rpNetwork.Name,
        },
        Entrypoint: []string{
            "sh",
        },
        Cmd: []string{
            "-c",
            "tail -f /dev/null",
        },
    },
    Started: true,
})

Produce messages using the new registered listener

_, _, err = kcat.Exec(ctx, []string{"kcat", "-b", "redpanda:29092", "-t", "msgs", "-P", "-l", "/tmp/msgs.txt"})

Adding Service Accounts

It's possible to add service accounts to the Redpanda container using the WithNewServiceAccount option, setting the service account name and its password. E.g. WithNewServiceAccount("service-account", "password").

Adding Super Users

When a super user is needed, you can use the WithSuperusers option, passing a variadic list of super users. E.g. WithSuperusers("superuser-1", "superuser-2").

Enabling SASL

The WithEnableSASL() option enables SASL scram sha 256 authentication. By default, no authentication (plaintext) is used. When setting an authentication method, make sure to add users as well and authorize them using the WithSuperusers() option.

WithEnableKafkaAuthorization

The WithEnableKafkaAuthorization enables authorization for connections on the Kafka API.

WithEnableWasmTransform

The WithEnableWasmTransform enables wasm transform.

Warning

Should not be used with RP versions before 23.3

WithEnableSchemaRegistryHTTPBasicAuth

The WithEnableSchemaRegistryHTTPBasicAuth enables HTTP basic authentication for the Schema Registry.

WithAutoCreateTopics

The WithAutoCreateTopics option enables the auto-creation of topics.

WithTLS

The WithTLS option enables TLS encryption. It requires a valid PEM encoded certificate and key, passed as byte slices. E.g. WithTLS([]byte(cert), []byte(key)).

WithBootstrapConfig

WithBootstrapConfig adds an arbitrary config key-value pair to the Redpanda container. Per the name, this config will be interpolated into the generated bootstrap config file, which is particularly useful for configs requiring a restart when otherwise applied to a running Redpanda instance. E.g. WithBootstrapConfig("config_key", config_value), where config_value is of type any.

WithAdminAPIAuthentication

Enables Admin API Authentication by setting admin_api_require_auth cluster configuration property to true. It also configures a bootstrap superuser account via RP_BOOTSTRAP_USER environment variable.

The following options are exposed by the testcontainers package.

Basic Options

Lifecycle Options

Files & Mounts Options

Build Options

Logging Options

Image Options

Networking Options

Advanced Options

Experimental Options

Container Methods

The Redpanda container exposes the following methods:

KafkaSeedBroker

KafkaSeedBroker returns the seed broker that should be used for connecting to the Kafka API with your Kafka client. It'll be returned in the format: "host:port" - for example: "localhost:55687".

seedBroker, err := ctr.KafkaSeedBroker(ctx)

SchemaRegistryAddress

SchemaRegistryAddress returns the address to the schema registry API. This is an HTTP-based API and thus the returned format will be: http://host:port.

schemaRegistryURL, err := ctr.SchemaRegistryAddress(ctx)

AdminAPIAddress

AdminAPIAddress returns the address to the Redpanda Admin API. This is an HTTP-based API and thus the returned format will be: http://host:port.

adminAPIURL, err := ctr.AdminAPIAddress(ctx)