Elevating IaC Workflows with Spacelift Stacks and Dependencies 🛠️

Register for the July 23 demo →

General

Pulumi State Management – How It Works & Where to Store It

Manage Pulumi State

Pulumi is an open-source Infrastructure as Code (IaC) tool that makes it possible for you to define infrastructure components in the programming language of your choice – Javascript, Python, Typescript, Go, C#, and Java. This removes the time required to learn another DSL, and you can start developing infrastructure right away.

Apart from this, Pulumi Cloud offers multiple features like modular configuration, secrets management, and automation, which help streamline infrastructure management. 

In this post, we will see how Pulumi state works, the types of state backends, and do a hands-on migration of the state from the local backend to Pulumi Cloud.

What is a state in Pulumi?

One of the crucial aspects of IaC is the metadata management. Metadata is the information about the desired configuration and what has already been provisioned in the real world. This metadata is stored in the form of “state”, which typically are JSON files. All the operations like creating, updating, deleting, or fetching the information about currently provisioned infrastructure resources depend on the state. 

For example, if we provision an EC2 instance using Pulumi, its information is stored in the state file. If we execute the Pulumi code again, Pulumi detects the existence of an EC2 instance and thus decides not to create another EC2 instance.

Pulumi tries to abstract away the complexity of state management to let the developers focus on the code. However, there are times when the understanding of the state management, especially its migration to other backends, matters.

What is the Pulumi state backend?

A group of infrastructure resources provisioned by a single Pulumi project is known as a stack. All the metadata about the resources/components managed by this stack is stored in the state file. The location where these state files are stored and managed is called the state backend. 

The state backends can be broadly classified as service and self-managed. The self-managed backends are further classified into local and 3rd party state backends.

The diagram below shows the three types of backends.

pulumi state diagram

When developers write the code using Pulumi IaC, they execute it using Pulumi CLI. They may choose to self-manage the state locally (1) or on 3rd party platforms (2) like AWS S3, Google Cloud Storage, or Azure Blob. The most efficient state backend for Pulumi is provided by the Pulumi Cloud (3).

The choice and configuration of the state backend is not a daily activity in a Pulumi project. It is a one-off but important activity.

To configure the state backend in Pulumi, we will do the following.

  1. Configure a Pulumi project with a local state backend
  2. Write the code to create an EC2 instance
  3. Analyze the directory structure and state files
  4. Migrate the backend to Pulumi Cloud
  5. Delete/destroy the EC2 instance

The hands-on walkthrough below will introduce various state backend management commands on the way. We will use Typescript as the programming language of choice.

Step 1: Configure a Pulumi project with a local state backend

Before you start writing the code to create an EC2 instance on AWS using Pulumi, it is important to choose a state backend. Since we have decided to use a local state backend, the very first Pulumi CLI command to run is to log in to the same. 

Create a project directory, and within that directory, create another subdirectory named “state” where all our state management files would reside. You can choose any other name for the subdirectory. 

Run the command below in the project root directory.

pulumi login file://./state

Let’s see the output:

Logged in to Sumeets-Laptop.local as ldt (file://./state)

The file structure after running this command looks like below:

pulumi state metadata

The files meta.yaml and meta.yaml.attrs created in the .pulumi subdirectory are not the state files. The state files will be created once we run either pulumi up or pulumi preview command.

It is also possible to log in to the local backend using the simpler version of the command, as shown below. However, --local is just a syntactic sugar. In the backend Pulumi still uses file:///path/to/state, which specifies the location of the file stored on the local system. In the example above, we have used a relative path which helps us create the state backend directory within our project directory.

pulumi login --local

By running the pulumi login command, we have successfully configured the local backend. 

When you use –local flag, the state files are managed in the default directory: /Users/ldt/.pulumi/stacks/plmsttbcknd.

Step 2: Write Typescript code to create an EC2 instance

As the first step towards creating an EC2 instance using Pulumi and Typescript, you need to initialize the stack. EC2 instance will be part of this stack. To initialize the new stack, run pulumi new command and follow the instructions. This command works well when run in an empty repository.

However, since we have a non-empty repository owing to the existence of state file location, we will have to use the --force flag as shown below. 

Pulumi provides multiple starter template options to begin coding our IaC. Since we already know the programming language we would use (Typescript) and the target cloud provider (AWS), we can simply include the template information in this command as well.

pulumi new aws-typescript --force

The output:

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name (plmsttbcknd):  
project description (A minimal AWS TypeScript Pulumi program):  
Created project 'plmsttbcknd'

stack name (dev): plmsttbcknd 
Created stack 'plmsttbcknd'
Enter your passphrase to protect config/secrets:  
Re-enter your passphrase to confirm:  

aws:region: The AWS region to deploy into (us-east-1): eu-central-1 
Saved config

Installing dependencies...


added 450 packages, and audited 451 packages in 45s

59 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

In the follow-up inputs, Pulumi provides a way to set the project name, description, stack name, passphrase for config/secrets, and AWS region. 

Feel free to set the values you feel are right. Pulumi CLI will soon install all the required packages and create a bunch of files in the project root directory, as seen in the screenshot below.

Add the code to create an EC2 instance in the index.ts file from the above list:

import * as aws from '@pulumi/aws';

const server = new aws.ec2.Instance("myEC2Instance", {
   ami: "ami-04dfd853d88e818e8",
   instanceType: "t2.micro",
   tags: {
       Name: "myEC2Instance",
   },
});

export const publicIp = server.publicIp;
export const publicDns = server.publicDns;

To make sure the code works well and to understand the changes Pulumi CLI would perform, run the preview command as shown below.

pulumi preview

The output:

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):  
Enter your passphrase to unlock config/secrets
Previewing update (plmsttbcknd):
Downloading plugin: 20.65 MiB / 20.65 MiB [=========================] 100.00% 3s
                                                                                [resource plugin docker-4.5.1] installing
     Type                 Name                     Plan       
 +   pulumi:pulumi:Stack  plmsttbcknd-plmsttbcknd  create     
 +   └─ aws:ec2:Instance  myEC2Instance            create     

Outputs:
    publicDns: output<string>
    publicIp : output<string>

Resources:
    + 2 to create

If everything was correct, it rightly indicates that it would create a stack, then create an EC2 instance within that stack, and output a couple of properties related to the EC2 instance.

Step 3: Analyze the directory files

The operations performed till now have caused some changes/additions to the directory structure of the project’s root directory as well as the state subdirectory, as seen below.

Let us explain these one by one.

In the project root directory:

  1. node_modules – Since we are using Typescript, all the modules used by this project will be downloaded in the node_modules directory. In our example, we are using @pulumi/aws module. The source code for the same is made available in node_modules.
  2. .gitignore – To exclude files to be pushed to the remote Git repository
  3. index.ts – This is where we develop the IaC for the EC2 instance. (Entry-point)
  4. package-lock.json – Used to ensure consistency and dependency locking for any Typescript project.
  5. package.json – Typescript project manifest file.
  6. Pulumi.plmsttbcknd.yaml – Stack specific Pulumi configuration. (eg. environment variables).
  7. Pulumi.yaml – Pulumi project metadata like name, description, runtime, entry point, etc.
  8. tsconfig.json – Configuration file for Typescript compiler

In the state/.pulumi subdirectory:

  1. locks/ – It is used for locking the state during Pulumi execution to avoid race conditions and potential corruption due to multiple writes.
  2. stacks/ – Directory to store state files and their attributes in JSON format. The state file is created based on the name given to the stack. If you run the pulumi new command again, you can create a new stack with a different name. A corresponding state file will also be created in this location.

The locks/ subdirectory may seem empty when no operations are running. When we execute commands like pulumi preview, up, delete, etc., a file with a unique name is created, which puts a lock on the state file so that a single execution process can edit the file. This is a very important functionality, especially when working in a team setup.

To see this in action, run the pulumi up command as shown below.

pulumi up
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):  
Enter your passphrase to unlock config/secrets
Previewing update (plmsttbcknd):
     Type                 Name                     Plan       
 +   pulumi:pulumi:Stack  plmsttbcknd-plmsttbcknd  create     
 +   └─ aws:ec2:Instance  myEC2Instance            create     

Outputs:
    publicDns: output<string>
    publicIp : output<string>

Resources:
    + 2 to create

Do you want to perform this update?  [Use arrows to move, type to filter]
  yes
> no
  details

The CLI asks for the confirmation but do not confirm anything yet. 

Observe the locks/ directory. A JSON file with a unique identifier is created. The contents of the file describe the process ID, username, and other identifying attributes, as seen below.

{
  "Pid":3795,
  "Username":"ldt",
  "hostname":"Sumeets-Laptop.local",
  "timestamp":"2024-03-04T21:59:35.075363+01:00"
}

Select yes (or no), and hit enter. The lock file disappears again. Assuming you selected yes, confirm if the EC2 instance with the desired configuration was created in the AWS management console.

pulumi ec2

After the first “create” operation, a couple of more files are created to back up the state files, as seen in the screenshot below.

Migrate the state file to Pulumi Cloud

Step 4: Migrate the state file to Pulumi Cloud

As a prerequisite for this step, you need to have an account on Pulumi Cloud.

We are currently managing the state backend locally, which is not a great idea when working in a team. For corruption-free state file management, the remote state backends play a crucial role. Thus, it makes sense to migrate the state to a state backend service like Pulumi Cloud.

Performing the migration in Pulumi involves a few steps, to be performed using Pulumi CLI:

  1. Login into the current state backend (we have already logged into the local state)
  2. Select the stack
  3. Export the stack and save it in the target file.
  4. Log out of the current state backend
  5. Log in to the new state backend (Pulumi Cloud in this example)
  6. Initialize the new stack
  7. Import the stack saved in a file from the old state backend

We need not log in to the current state backend as we have already logged in. If, for some reason, you have logged out, please refer to Step 1. Select the stack by running the following CLI command. Use the stack name to do the same.

pulumi stack select plmsttbcknd

Export the stack and save it in a file using the export command below. Here, we have stored the state in a .json file. You can also choose to save this file in a different directory on your local system.

pulumi stack export --show-secrets --file plmsttbcknd.exported.json

We can see the exported JSON file in the directory below.

Pulumi state

Log out of the current state backend.

pulumi logout

Log in to the new state backend. In this example, since we are logging into Pulumi Cloud, we need not specify anything since this is the default state backend.

pulumi login

Output:

Manage your Pulumi stacks by logging in.
Run `pulumi --help` for alternative login options.
Enter your access token from https://app.pulumi.com/account/tokens
    or hit <ENTER> to log in using your browser                   : 


Hit enter to authorize. Once the browser based authentication is successful, the Pulumi CLI displays the success message below.

We've launched your web browser to complete the login process.

Waiting for login to complete...


  Welcome to Pulumi!

  Pulumi helps you create, deploy, and manage infrastructure on any cloud using
  your favorite language. You can get started today with Pulumi at:

      https://www.pulumi.com/docs/get-started/

  Tip: Resources you create with Pulumi are given unique names (a randomly
  generated suffix) by default. To learn more about auto-naming or customizing resource
  names see https://www.pulumi.com/docs/intro/concepts/resources/#autonaming.


Logged in to pulumi.com as sumeetninawe (https://app.pulumi.com/sumeetninawe)

Initialize the new stack in the newly logged-in state backend. While initializing this new stack, it is important to use the same name, or else the import won’t work.

pulumi stack init plmsttbcknd

The output:

Created stack 'plmsttbcknd'

Finally, import the stack by running the import command below. Select the file we created previously to export the state backup.

pulumi stack import --file plmsttbcknd.exported.json

The output:

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):  
Enter your passphrase to unlock config/secrets
Import complete.

Login to the Pulumi Cloud and see if the stack was imported successfully.

pulumi cloud state

Navigate to the resources section of this stack and see if you are able to find the EC2 instance, as seen below.

pulumi cloud backend stacks

If this is the case, then you have successfully migrated the state backend from the local machine to Pulumi Cloud.

Step 5: Destroy the resources

From here on, you can leverage various functionalities offered by Pulumi Cloud, which is also a default state backend. If you want to delete the EC2 instance from this stack, you can do it either by using Pulumi CLI on a local machine while being logged into the Pulumi Cloud, or you can also select the “Delete” action from the UI.

The command to delete the resources:

pulumi destroy

And the confirmation:

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):  
Enter your passphrase to unlock config/secrets
Previewing destroy (plmsttbcknd)

View in Browser (Ctrl+O): https://app.pulumi.com/sumeetninawe/plmsttbcknd/plmsttbcknd/previews/a25f8ce2-80be-4ffe-930a-ad5af8efb23e

     Type                 Name                     Plan       
 -   pulumi:pulumi:Stack  plmsttbcknd-plmsttbcknd  delete     
 -   └─ aws:ec2:Instance  myEC2Instance            delete     

Outputs:
  - publicDns: "ec2-3-123-253-73.eu-central-1.compute.amazonaws.com"
  - publicIp : "3.123.253.73"

Resources:
    - 2 to delete

Do you want to perform this destroy?  [Use arrows to move, type to filter]
> yes
  no
  details

Confirmation of deletion:

Do you want to perform this destroy? yes
Destroying (plmsttbcknd)

View in Browser (Ctrl+O): https://app.pulumi.com/sumeetninawe/plmsttbcknd/plmsttbcknd/updates/2

     Type                 Name                     Status            
 -   pulumi:pulumi:Stack  plmsttbcknd-plmsttbcknd  deleted (1s)      
 -   └─ aws:ec2:Instance  myEC2Instance            deleted (32s)     

Outputs:
  - publicDns: "ec2-3-123-253-73.eu-central-1.compute.amazonaws.com"
  - publicIp : "3.123.253.73"

Resources:
    - 2 deleted

Duration: 36s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained. 
If you want to remove the stack completely, run `pulumi stack rm plmsttbcknd`.

Introducing Spacelift – a powerful alternative to both Pulumi and Terraform Cloud

Spacelift is the most flexible management platform for Infrastructure as Code. Spacelift provides excellent capabilities to automate running your Pulumi stacks. That means you don’t have to configure your own CI pipelines. Especially if you have several stacks and want to connect them together. And it does not stop there.

Spacelift supports both Pulumi and Terraform, and apart from these tools, you also get OpenTofu, Kubernetes, CloudFormation, Ansible, and Terragrunt, making sure that all of your workflows can be achieved easily.

Spacelift’s pricing model is based on concurrency, ensuring that you won’t need to have a calculator with you at all times to try and predict costs – pricing is very predictable.

With Spacelift you: 

If you need an IaC management solution that handles workflows for Pulumi and other tools as well, create a free account with Spacelift today, or book a demo with one of our engineers.

Key points

In this blog post, we performed hands-on exercises on the Pulumi state backend, which provided the understanding of setting up a local backend, creating and managing infrastructure resources, and migrating to the Pulumi Cloud backend. 

We also dived deep into the directory structure and examined the generated state files, providing insights into how Pulumi manages the infrastructure state. We also presented different backend options and the migration process, which enables you to make informed decisions about how to store and manage your Pulumi state effectively.

The Most Flexible CI/CD Automation Tool

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Start free trial

The Practitioner’s Guide to Scaling Infrastructure as Code

Transform your IaC management to scale

securely, efficiently, and productively

into the future.

ebook global banner
Share your data and download the guide