Creating an SQS Queue Using Terraform: A Step-by-Step Guide

In this article, we will use Terraform to create and manage an SQS Queue while exploring SQS’s Properties.

Image by author

Amazon Simple Queue Service (SQS) is a fully managed message queuing service provided by AWS. With SQS, messages can be stored and retrieved from queues ensuring reliable delivery.

It enables reliable and scalable communication between various components of distributed systems, allowing decoupling of applications and asynchronous processing of messages.

Some alternatives to SQS include RabbitMQ and Apache Kafka.

If you’re interested in learning more about RabbitMQ, you can check out my previous articles on this topic.

Terraform is an infrastructure as code (Iac) tool that automates infrastructure provisioning. It is the most popular and offers support for multiple clouds.

In this article, we will,

• Use Terraform to create and manage an SQS Queue.
• Explore the SQS Properties

Setting up the Environment

1. Install AWS CLI and configure

If you don’t have AWS CLI already, install AWS CLI to your system based on your OS by following the instructions here.

Install or update the latest version of the AWS CLI

The next step is to configure your AWS profile to use in the CLI. Follow the instructions from the official documentation based on your machine type.

Install Terraform | Terraform | HashiCorp Developer

2. Install Terraform

Install Terraform in your system, if you don’t have it already. Follow the instructions based on your platform on the official site given below.

Install Terraform | Terraform | HashiCorp Developer

3. Initialise Project

Create an empty project directory and create a file called main.tf within it.

Firstly we need to add information about the provider (Here, we will be using AWS Provider). We will also configure AWS credentials, so we could provision resources in the AWS cloud.

You can configure the AWS credentials in the provider section. Here, I am using the path to AWS config and credential files in my system to authenticate. Also, I am mentioning the region and profile that I will be using by default. (You can also use Access and Secret keys instead of config files)

Now that we have set the provider set, we have to initialise Terraform to use it. (This command will download/update the providers).

terraform init

Create an SQS queue

You can create two types of queues with SQS:

• Standard Queue
• FIFO Queue.

The main difference is that FIFO Queues maintain the order of messages sent to them. This article focuses on the standard queue.

We can create an SQS queue in Terraform using the resource aws_sqs_queue.

Note: You can create a new file for SQS configuration(eg sqs.tf) or continue in the same file(main.tf).
Separating resources into different files makes it easier to understand and maintain the infrastructure.

SQS Queue

All arguments for creating an SQS queue are optional, meaning there are no required arguments for creating a queue. If you don’t provide a value for an argument, it will automatically use a default value.

Code explanation:

For the resource aws_sqs_queue,

delay_seconds (default — 0 seconds)

Possible values: 0 to 900 (15 minutes)
Any message that is sent to the queue remains invisible to consumers for the duration of this delay period.

visibility_timeout_seconds (default — 30 seconds)

Possible values: 0 to 43200 (12 hours)
It determines the duration during which a message remains invisible to other consumers after it has been retrieved by a consumer. This allows the consumer enough time to process the message before it becomes available for other consumers to retrieve. Read more here.

max_message_size (default — 262144 bytes [256 KiB])

Possible Values: 1024 bytes (1 KiB) up to 262144 bytes (256 KiB)
The maximum size of the message that can be sent to the SQS queue. If a message exceeds this size, it will be rejected.

message_retention_seconds (default — 345600 seconds[4days])

Possible Values: 60 (1 minute) to 1209600 (14 days)
This argument sets the duration, for which messages are retained in the queue. After this duration, any messages that haven’t been processed or deleted will be automatically removed from the queue.

receive_wait_time_seconds (default — 0 seconds)

Possible Values: 0 to 20 (seconds)
This argument is useful to provide a time period that a request could wait for a message to become available in the queue. If no messages are available within this time, the request will return an empty response.

sqs_managed_sse_enabled (boolean)

This argument enables Server-Side Encryption (SSE) managed by SQS for messages stored in the queue. When enabled, SQS automatically encrypts the messages at rest to enhance data security.

To note:

We can set Server Side Encryption(SSE) for our queue in two ways:

1. Using Amazon SQS Key (SSE-SQS)
   — The argument sqs_managed_sse_enabled is set to true in this case.
2. Using an AWS KMS key (SSE-KMS)
   — The argument kms_master_key_id can be used to provide a KMS key to use for encryption.

Reference: Documentation for the arguments of SQS.

Attaching a Policy to our SQS Queue

Let’s create a policy which grants permission to any AWS entity to send and receive messages from our SQS queue.

By using a policy, we can control access to the queue and allow only authorized entities to perform any operations on it.

Policy and attaching the policy to SQS

In this aws_iam_policy_document resource,

sid → (statement id) A unique id for this policy statement.

Effect → Allow or block

Principals → Who can access? (In this case, any AWS resource is allowed)

Actions → What specific actions are allowed or blocked? (For example, sending messages to the queue)

Resources → Which resource do these permissions apply to? (Here, it refers to our SQS queue)

After defining the policy, we are attaching it to our queue using aws_sqs_queue_policy.

Terraform apply

We have the Queue ready to be deployed!

Let’s look at the terraform commands that we would be needing every time we deploy any changes.

Try the above commands in that order.

terraform plan gives you details about the resources that will be deployed, and if it runs successfully and verified the resources, we can proceed to deploy the resources by using terraform apply command.

Let’s test our queue now!

After the changes are deployed,

• Go to AWS Console and sign in.
• Go to Simple Queue Service(SQS) and go to Queues
• You can view the queue we created and its properties.

• Click on the name of the queue to view it in detail.
• Let's send a message to the queue, Click on Send and receive messages button.

• Enter a message in the Message Body, and enter a delay value (in seconds) for Delivery Delay, then proceed to click Send Message button to finally send the message.

• Once the message is sent, scroll down to the Receive messages section and click on the Poll for messages button to receive the message.

Under the Messages, we can see that there is a message received from the queue.

If you click on the ID of the message, we can view the message and its details.

Yay!! Our queue is working! Now, you can connect this queue to other services to leverage the benefits of SQS.

You can make changes to the code and deploy the updated infrastructure by trying the above commands in the same order.

The terraform state files save the current state of the infrastructure.

Backup

The terraform state files (terraform.tfstate and terraform.tfstate.backup) can be backed up in storage — using which we can provision the resources again anytime if something goes wrong.

Note: It is recommended to make changes in the terraform code once deployed and not directly in the AWS console. Terraform manages and tracks the state of your infrastructure. If you modify resources in AWS manually, it can lead to inconsistencies between the desired state in your Terraform code and the actual state of the resources. This can cause conflicts and make it difficult to manage and update your infrastructure accurately.

Terraform destroy

If you want to destroy all the resources created using Terraform in this project, you can use terraform destroy command.
This could be particularly helpful when you are learning Terraform, you can delete all the resources easily at the end, to avoid costs.

If you have any doubts, or any suggestions to improve this article, feel free to add them in the comments.

Github repo:

GitHub - SharmilaS22/medium-sqs-terraform: Code for the medium article on creating a standard sqs…

Happy learning!