Elevating IaC Workflows with Spacelift Stacks and Dependencies 🛠️

Register for the July 23 demo →

Product

How to Set Up a Spacelift Worker Pool on EC2

How to Set Up a Spacelift Worker Pool on EC2

Initiating the configuration of a worker pool within Spacelift might initially appear to be a complex task. However, it is remarkably straightforward and accessible. 

This post is designed to serve as a comprehensive guide that streamlines and expedites the setup process. It aims to provide clear, step-by-step instructions, ensuring that you can approach this task with confidence.

What we will cover:

  1. Prerequisites
  2. Generating the Worker pool
  3. Running the worker automation module

Prerequisites

Before creating the worker pool we need to:

  • Generate a worker private key
  • Generate a Spacelift API key
  • Set up a cloud Integration to AWS in your Spacelift account
  • Get Spacelift’s mothership IPs
  • Set up a network

1. Generate a worker private key

To generate a worker private key, simply go to your terminal and run the following command:

openssl req -new -newkey rsa:4096 -nodes -keyout spacelift.key -out spacelift.csr

This command will generate two files: spacelift.key and spacelift.csr. Remember to store them securely, as they will be essential when you are creating the worker pool and initiating workers within it.

2. Generate a Spacelift API key

In your account, click in the bottom left corner on your username and select Organization settings.

Generate a Spacelift API key

Then, under access, select API keys and click on Create API key:

API key

Add the required details for the API key, such as the name, space, and role, and then click on Create. Ensure you save the generated file in a safe place, as you will need it to use it when we run an automation.

save the generated file in a safe place

The API secret will be in this file and we will need it later:

api secret file

3. Create an AWS Cloud Integration

The process for creating an AWS Cloud Integration is easy to follow and can be found here. 

If you don’t want to use dynamic credentials, you can pass static ones as environment variables when we create the stack that does the automation.

4. Get Spacelift’s mothership IPs

We will need the Spacelift’s mothership IPs to whitelist connections to them in the security groups our automation will use.

There is an automation available that you can easily run to get the IPs:

For that, go to your Spacelift account and create a stack, add a name, and select a space for it, in the VCS screen select Raw Git and provide the following URL: https://github.com/saturnhead/mothership_ips/ 

Get Spacelift’s mothership IPS

Select either OpenTofu or Terraform and click on Create & Continue. In the Define behavior tab, check the Administrative option:

Define behavior

Now skip to the summary, confirm, and trigger a run. After the plan finishes successfully, you will see the IPs. 

5. Network setup

We advise you to use at least two public subnets and two security groups. The subnets should have a route to an internet gateway with 0.0.0.0/0 destination.

The security groups should permit the following traffic:

  • Ingress: 443 for all Spacelift Mothership IPs
  • Egress: 443 for 0.0.0.0/0

Generating the Worker pool

In your Spacelift account, go to Worker Pools and click on Create Worker Pool.

Create Worker Pool

Add a name to your worker pool, upload the spacelift.csr file, and select a space in which you want to use your worker pool. You have the option to add a description and label. After you click on Create, a worker pool config will download automatically.

worker pool config

Create a repository that uses the worker pool module

The module that we will use can be found here. Based on this module, we’ve created a repository that can be leveraged for automating the worker creation. You can fork this repository and use it as it is, or you can customize it to your liking.

Create the worker pool stack

Go to your Spacelift account, select Stacks, and then click on Create Stack.

Create Stack

Add a name, select a space for your stack, and click on Next. In the next screen, select your VCS provider, the forked repository, and the branch. If you have made changes and modified the path to the OpenTofu/Terraform scripts, ensure you set up the project root accordingly.

Next, select the vendor (OpenTofu or Terraform will both work), and click on Create & continue:

select the vendor

After this, you can go directly to the “Attach cloud screen” and attach the cloud integration you have created:

Attach cloud screen

Click on Attach, Skip to Summary, and Confirm. You will be redirected to the newly created stack.

Before running the stack, head to the environment and fill in the following env vars:

  • TF_VAR_spacelift_api_key_endpoint – this will be your Spacelift account (e.g https://saturnhead.app.spacelift.io)
  • TF_VAR_spacelift_api_key_id – the API key id, can be easily taken from your Organization Settings – API keys and by clicking on the copy button (ensure you click on it, as they Key ID is longer that what is shown in the app for security reasons)
api keys

TF_VAR_spacelift_api_key_secret – the secret of the API key id. This can be taken from the file that downloaded automatically when you created the API key:

api secret file
  • TF_VAR_worker_pool_private_key – the private key from the worker pool base encoded, which you can get from running: cat spacelift.key | base64 -b 0 | pbcopy
  • TF_VAR_worker_pool_id –  the id of the worker pool that you can copy from the worker pool page
TF_VAR_worker_pool_id
  • TF_VAR_worker_pool_config – the content of the config file that was automatically downloaded when we created the worker pool. The file should be in this format – worker-pool-wp_id.config
  • TF_VAR_worker_pool_security_groups – the security groups used by the worker pool (these security groups should have traffic whitelisted from the Spacelift mothership). Provide at least two.
  • TF_VAR_worker_pool_subnets – the subnets used by the worker pool; provide at least two.

After you’ve populated all of these env vars, you are ready to run the code:

run the code

Go to tracked runs and click on Trigger. The plan will show 15 resources to add:

Trigger

Next, confirm the run and wait for it to apply. In the end, you should see that everything was created successfully:

confirm the run

Now, if you go back to your Worker Pools, you should see the worker pool has one worker:

the worker pool has one worker

Based on demand, the worker pool will increase/decrease the number of workers and if you haven’t changed the OpenTofu code, you will have a minimum of one and a maximum of five.

Key points

In this post, we’ve seen how to set up private workers in Spacelift using EC2. The overall process is simple, but you need to be careful to pass the correct environment variables in the stack that runs the automation.

If you want to learn more about Spacelift, you can book a demo with one of our engineers, and if you don’t have an account, you can create one here.

The Most Flexible CI/CD Automation Tool

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Start free trial

The Practitioner’s Guide to Scaling Infrastructure as Code

Transform your IaC management to scale

securely, efficiently, and productively

into the future.

ebook global banner
Share your data and download the guide