Skip to content
This repository was archived by the owner on Jan 26, 2023. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,5 @@ attache-control
!attache-control/
!attache-check/

# Mac
*.DS_Store
Comment thread
beautifulentropy marked this conversation as resolved.
61 changes: 44 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,15 @@ Nomad and Consul.
- Create a new cluster when no cluster is present
- Add new primary node and perform a shard slot rebalance
- Add new replica node to the primary node with the least replicas
- Full support for Redis mTLS and ACL Auth
- Full support for Consul mTLS and ACL Tokens

#### To Do
- Drain and failover and FORGET an existing primary node
- Remove and FORGET an existing replica node
- Redis ACL
- Redis Password
- Redis mTLS
- Redis Static ClusterHub Port selection (from Nomad Job Specification)
- [x] Redis ACL
- [x] Redis Password
- [x] Redis mTLS
- [ ] Drain, failover, and FORGET an existing primary node
- [ ] Remove and FORGET an existing replica node

### `attache-check`
A sidecar that servers an HTTP API that allows Consul to track the health of
Expand All @@ -23,13 +24,24 @@ once they've joined a cluster.

#### Usage
```shell
$ attache-check -help
Usage of ./attache-check:
-check-serv-addr string
address this utility should listen on
address this utility should listen on (e.g. 127.0.0.1:8080)
-redis-auth-password-file string
redis-server password file path, (required)
-redis-auth-username string
redis-server username, (required)
-redis-node-addr string
redis-server listening address
redis-server listening address, (required)
-redis-tls-ca-cert string
Redis client CA certificate file, (required)
-redis-tls-cert-file string
Redis client certificate file, (required)
-redis-tls-key-file string
Redis client key file, (required)
-shutdown-grace duration
duration to wait before shutting down (e.g. '1s') (default 5s)
duration to wait before shutting down (e.g. '1s') (default 5s)
```

### `attache-control`
Expand All @@ -48,7 +60,7 @@ Usage of ./attache-control:
-attempt-limit int
Number of times to attempt joining or creating a cluster before exiting (default 20)
-await-service-name string
Consul Service for any newly created Redis Cluster Nodes
Consul Service for newly created Redis Cluster Nodes, (required)
-consul-acl-token string
Consul client ACL token
-consul-addr string
Expand All @@ -60,27 +72,42 @@ Usage of ./attache-control:
-consul-tls-cert string
Consul client certificate file
-consul-tls-enable
Enable TLS for the Consul client
Enable mTLS for the Consul client
-consul-tls-key string
Consul client key file
-dest-service-name string
Consul Service for any existing Redis Cluster Nodes
Consul Service for healthy Redis Cluster Nodes, (required)
-lock-kv-path string
KV path used by Consul to aquire a distributed lock for operations (default "service/attache/leader")
Consul KV path used as a distributed lock for operations (default "service/attache/leader")
-log-level string
Set the log level (default "info")
-redis-auth-password-file string
Redis password file path, (required)
-redis-auth-username string
Redis username, (required)
-redis-node-addr string
redis-server listening address
redis-server listening address, (required)
-redis-primary-count int
Total number of expected Redis shard primary nodes
Total number of expected Redis shard primary nodes, (required)
-redis-replica-count int
Total number of expected Redis shard replica nodes
-redis-tls-ca-cert string
Redis client CA certificate file, (required)
-redis-tls-cert-file string
Redis client certificate file, (required)
-redis-tls-key-file string
Redis client key file, (required)
```

### Running the Example Nomad Job
Note: these steps assume that you have the Nomad and Consul binaries installed
Note: these steps assume that you have the `nomad` and `consul` binaries installed
on your machine and that they exist in your `PATH`.

Build the attache-control and attache-check binaries:
```shell
$ go build -o attache-check ./cmd/attache-check/main.go && go build -o attache-control ./cmd/attache-control/main.go ./cmd/attache-control/config.go
```

Start the Consul server in `dev` mode:
```shell
$ consul agent -dev -datacenter dev-general -log-level ERROR
Expand All @@ -93,7 +120,7 @@ $ sudo nomad agent -dev -bind 0.0.0.0 -log-level ERROR -dc dev-general

Start a Nomad job deployment:
```shell
nomad job run -verbose ./example/redis-cluster.hcl
$ nomad job run -verbose -var-file=./example/vars-file.hcl ./example/job-specification.hcl
```

Open the Nomad UI: http://localhost:4646/ui
Expand Down
25 changes: 19 additions & 6 deletions cmd/attache-check/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@ import (

"github.com/gorilla/mux"
redis "github.com/letsencrypt/attache/src/redis/client"
"github.com/letsencrypt/attache/src/redis/config"
logger "github.com/sirupsen/logrus"
)

// CheckHandler is a wrapper around an inner redis.Client.
// CheckHandler is a wraps an inner redis client with some methods for handling
// health check requests.
type CheckHandler struct {
redis.Client
}
Expand All @@ -36,21 +38,32 @@ func (h *CheckHandler) StateOk(w http.ResponseWriter, r *http.Request) {
func main() {
checkServAddr := flag.String("check-serv-addr", "", "address this utility should listen on (e.g. 127.0.0.1:8080)")
shutdownGrace := flag.Duration("shutdown-grace", time.Second*5, "duration to wait before shutting down (e.g. '1s')")
redisNodeAddr := flag.String("redis-node-addr", "", "redis-server listening address")

logger.Infof("starting %s", os.Args[0])
var redisOpts config.RedisOpts
flag.StringVar(&redisOpts.NodeAddr, "redis-node-addr", "", "redis-server listening address, (required)")
flag.StringVar(&redisOpts.Username, "redis-auth-username", "", "redis-server username, (required)")
flag.StringVar(&redisOpts.PasswordFile, "redis-auth-password-file", "", "redis-server password file path, (required)")
flag.StringVar(&redisOpts.CACertFile, "redis-tls-ca-cert", "", "Redis client CA certificate file, (required)")
flag.StringVar(&redisOpts.CertFile, "redis-tls-cert-file", "", "Redis client certificate file, (required)")
flag.StringVar(&redisOpts.KeyFile, "redis-tls-key-file", "", "Redis client key file, (required)")
flag.Parse()

if *checkServAddr == "" {
logger.Fatal("Missing required opt 'check-serv-addr'")
}

if *redisNodeAddr == "" {
logger.Fatal("Missing required opt 'redis-node-addr'")
err := redisOpts.Validate()
if err != nil {
logger.Fatal(err)
}

logger.Infof("starting %s", os.Args[0])

router := mux.NewRouter()
redisClient := redis.New(*redisNodeAddr, "")
redisClient, err := redis.New(redisOpts)
if err != nil {
logger.Fatalf("redis: %s", err)
}
handler := CheckHandler{*redisClient}
router.HandleFunc("/clusterinfo/state/ok", handler.StateOk)

Expand Down
140 changes: 41 additions & 99 deletions cmd/attache-control/config.go
Original file line number Diff line number Diff line change
@@ -1,141 +1,83 @@
package main

import (
"errors"
"flag"
"fmt"
"net/http"
"time"

consul "github.com/hashicorp/consul/api"
logger "github.com/sirupsen/logrus"
c "github.com/letsencrypt/attache/src/consul/config"
r "github.com/letsencrypt/attache/src/redis/config"
)

// CLIOpts is exported for use with flag.Parse().
type CLIOpts struct {
// Exported for use with flag.Parse()
RedisNodeAddr string
// Exported for use with flag.Parse()
RedisOpts r.RedisOpts
RedisPrimaryCount int
// Exported for use with flag.Parse()
RedisReplicaCount int
// Exported for use with flag.Parse()
LockPath string
// Exported for use with flag.Parse()
AttemptInterval time.Duration
// Exported for use with flag.Parse()
AttemptLimit int
// Exported for use with flag.Parse()
AwaitServiceName string
// Exported for use with flag.Parse()
DestServiceName string
// Exported for use with flag.Parse()
LogLevel string
// Exported for use with flag.Parse()
ConsulOpts ConsulOpts
LockPath string
AttemptInterval time.Duration
AttemptLimit int
AwaitServiceName string
DestServiceName string
LogLevel string
ConsulOpts c.ConsulOpts
}

type ConsulOpts struct {
// Exported for use with flag.Parse()
DC string
// Exported for use with flag.Parse()
Address string
// Exported for use with flag.Parse()
ACLToken string
// Exported for use with flag.Parse()
TLSEnable bool
// Exported for use with flag.Parse()
TLSCACert string
// Exported for use with flag.Parse()
TLSCert string
// Exported for use with flag.Parse()
TLSKey string
}
func (c CLIOpts) Validate() error {
if c.RedisPrimaryCount == 0 {
return errors.New("missing required opt: 'redis-primary-count'")
Comment thread
beautifulentropy marked this conversation as resolved.
}

if c.DestServiceName == "" {
return errors.New("missing required opt: 'dest-service-name'")
}

func (c *ConsulOpts) MakeConsulClient() (*consul.Client, error) {
config := consul.DefaultConfig()
config.Datacenter = c.DC
config.Address = c.Address
config.Token = c.ACLToken
if c.TLSEnable {
config.Scheme = "https"
tlsConfig := consul.TLSConfig{
Address: c.Address,
CAFile: c.TLSCACert,
CertFile: c.TLSCert,
KeyFile: c.TLSKey,
}
tlsClientConf, err := consul.SetupTLSConfig(&tlsConfig)
if err != nil {
return nil, fmt.Errorf("error creating TLS client config for consul: %w", err)
}
config.HttpClient.Transport = &http.Transport{
TLSClientConfig: tlsClientConf,
}
if c.AwaitServiceName == "" {
return errors.New("missing required opt: 'await-service-name'")
}

client, err := consul.NewClient(config)
err := c.ConsulOpts.Validate()
if err != nil {
return nil, err
return err
}

return client, nil
err = c.RedisOpts.Validate()
if err != nil {
return err
}
return nil
}

func ParseFlags() CLIOpts {
var conf CLIOpts

// CLI
flag.StringVar(&conf.RedisNodeAddr, "redis-node-addr", "", "redis-server listening address")
flag.IntVar(&conf.RedisPrimaryCount, "redis-primary-count", 0, "Total number of expected Redis shard primary nodes")
flag.IntVar(&conf.RedisPrimaryCount, "redis-primary-count", 0, "Total number of expected Redis shard primary nodes, (required)")
flag.IntVar(&conf.RedisReplicaCount, "redis-replica-count", 0, "Total number of expected Redis shard replica nodes")
flag.StringVar(&conf.LockPath, "lock-kv-path", "service/attache/leader", "Consul KV path used as a distributed lock for operations")
flag.DurationVar(&conf.AttemptInterval, "attempt-interval", 3*time.Second, "Duration to wait between attempts to join or create a cluster")
flag.IntVar(&conf.AttemptLimit, "attempt-limit", 20, "Number of times to attempt joining or creating a cluster before exiting")
flag.StringVar(&conf.AwaitServiceName, "await-service-name", "", "Consul Service for any newly created Redis Cluster Nodes")
flag.StringVar(&conf.AwaitServiceName, "await-service-name", "", "Consul Service for newly created Redis Cluster Nodes, (required)")
flag.StringVar(&conf.DestServiceName, "dest-service-name", "", "Consul Service for healthy Redis Cluster Nodes, (required)")
flag.StringVar(&conf.LogLevel, "log-level", "info", "Set the log level")

// Redis
flag.StringVar(&conf.RedisOpts.NodeAddr, "redis-node-addr", "", "redis-server listening address, (required)")
flag.StringVar(&conf.RedisOpts.Username, "redis-auth-username", "", "Redis username, (required)")
flag.StringVar(&conf.RedisOpts.PasswordFile, "redis-auth-password-file", "", "Redis password file path, (required)")
flag.StringVar(&conf.RedisOpts.CACertFile, "redis-tls-ca-cert", "", "Redis client CA certificate file, (required)")
flag.StringVar(&conf.RedisOpts.CertFile, "redis-tls-cert-file", "", "Redis client certificate file, (required)")
flag.StringVar(&conf.RedisOpts.KeyFile, "redis-tls-key-file", "", "Redis client key file, (required)")

// Consul
flag.StringVar(&conf.DestServiceName, "dest-service-name", "", "Consul Service for any existing Redis Cluster Nodes")
flag.StringVar(&conf.ConsulOpts.DC, "consul-dc", "dev-general", "Consul client datacenter")
flag.StringVar(&conf.ConsulOpts.Address, "consul-addr", "127.0.0.1:8500", "Consul client address")
flag.StringVar(&conf.ConsulOpts.ACLToken, "consul-acl-token", "", "Consul client ACL token")
flag.BoolVar(&conf.ConsulOpts.TLSEnable, "consul-tls-enable", false, "Enable TLS for the Consul client")
flag.BoolVar(&conf.ConsulOpts.EnableTLS, "consul-tls-enable", false, "Enable mTLS for the Consul client")
flag.StringVar(&conf.ConsulOpts.TLSCACert, "consul-tls-ca-cert", "", "Consul client CA certificate file")
flag.StringVar(&conf.ConsulOpts.TLSCert, "consul-tls-cert", "", "Consul client certificate file")
flag.StringVar(&conf.ConsulOpts.TLSKey, "consul-tls-key", "", "Consul client key file")

flag.Parse()

return conf
}

func ValidateConfig(conf CLIOpts) {
if conf.ConsulOpts.TLSEnable {
if conf.ConsulOpts.TLSCACert == "" {
logger.Fatal("Missing required opt: 'consul-tls-ca-cert")
}

if conf.ConsulOpts.TLSCert == "" {
logger.Fatal("Missing required opt: 'consul-tls-cert")
}

if conf.ConsulOpts.TLSKey == "" {
logger.Fatal("Missing required opt: 'consul-tls-key")
}
}

if conf.RedisNodeAddr == "" {
logger.Fatal("Missing required opt: 'redis-node-addr'")
}

if conf.RedisPrimaryCount == 0 {
logger.Fatal("Missing required opt: 'redis-primary-count'")
}

if conf.DestServiceName == "" {
logger.Fatal("Missing required opt: 'dest-service-name'")
}

if conf.AwaitServiceName == "" {
logger.Fatal("Missing required opt: 'await-service-name'")
}
}
Loading