Skip to content
This repository was archived by the owner on Jan 26, 2023. It is now read-only.

Update allocations no longer require that healthy nodes be destroyed#16

Merged
beautifulentropy merged 8 commits intomainfrom
improved-update-allocation
Dec 10, 2021
Merged

Update allocations no longer require that healthy nodes be destroyed#16
beautifulentropy merged 8 commits intomainfrom
improved-update-allocation

Conversation

@beautifulentropy
Copy link
Copy Markdown
Member

@beautifulentropy beautifulentropy commented Dec 10, 2021

Currently, adding an additional node to an existing Redis Cluster would change
the flags being passed to the attache-control sidecar task for every existing
allocation. The Nomad scheduler would (correctly) trigger a destructive update
(e.g. reallocate, stop, migrate, and restart) of each existing Redis Cluster
node even though they were already healthy. This is because the Nomad scheduler
can only update an allocation in-place when there are no attributes (environment
variables, file templates, etc.) relevant to any of that job's tasks being
updated.

This PR updates attache-control to fetch these counts from a Consul KV path
instead.

To ensure consistency between, the scaling configuration stored in the Consul
and the total number of nodes in the Nomad job specification, I've added a
Terraform file that sets both of them. For more information, see the updated
README.

Lastly, we're approaching the point where more folks may begin to touch this
code so I've also taken the time to comment up my exported structs, fields, and
methods.

jsha
jsha previously approved these changes Dec 10, 2021
Copy link
Copy Markdown

@jsha jsha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. A bit curious - why is the new file example/redis-cluster.tf part of this PR?

@beautifulentropy beautifulentropy force-pushed the improved-update-allocation branch 3 times, most recently from bffab2c to bf82fcb Compare December 10, 2021 04:58
@beautifulentropy
Copy link
Copy Markdown
Member Author

beautifulentropy commented Dec 10, 2021

Looks good to me. A bit curious - why is the new file example/redis-cluster.tf part of this PR?

I've updated the commit message to make this a little clearer:

To ensure consistency between, the scaling configuration stored in the Consul
and the total number of nodes in the Nomad job specification, I've added a
Terraform file that sets both of them. For more information, see the updated
README.

@beautifulentropy beautifulentropy force-pushed the improved-update-allocation branch 2 times, most recently from ae5e6c1 to c4ae18d Compare December 10, 2021 05:14
@beautifulentropy beautifulentropy force-pushed the improved-update-allocation branch from c4ae18d to dc83e0f Compare December 10, 2021 05:15
@beautifulentropy beautifulentropy marked this pull request as ready for review December 10, 2021 05:15
Comment thread cmd/attache-control/config.go Outdated
Comment thread cmd/attache-control/config.go
Comment thread src/consul/config/config.go Outdated
Comment thread src/consul/service/service.go Outdated
Comment thread cmd/attache-control/main.go Outdated
Comment thread src/consul/scaling/scaling.go Outdated
aarongable
aarongable previously approved these changes Dec 10, 2021
Copy link
Copy Markdown

@aarongable aarongable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with just a couple nits.

Comment thread src/consul/client/client.go Outdated
Comment thread src/consul/client/client.go Outdated
Comment thread src/consul/client/client.go
Co-authored-by: Aaron Gable <aaron@letsencrypt.org>
Co-authored-by: Aaron Gable <aaron@letsencrypt.org>
@beautifulentropy beautifulentropy merged commit 9a9d44b into main Dec 10, 2021
@beautifulentropy beautifulentropy deleted the improved-update-allocation branch December 10, 2021 21:38
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants