System testing: Localstack + Terraform

Image for post
Image for post

I am a big believer in the importance of testing. It is cool when you can write a batch of tests before implementing the feature and then see how the number of green tests goes up. And TDD is great. Lately, I changed my main activities. And start spending most of the time working with DevOps and Microservices on AWS. The first time it feels like a fun work to do. But soon I understood that I don’t like it since most of the time it is a routine and a big number of technologies make you sick and tired. More often I found myself testing services by hands since you can’t be sure that after deploy service will work with other services correctly.

After some time I decided that it can’t last this way any longer and I need to find the way to start testing these services. The main problem is to testing how service work on AWS infrastructure. While researching this question I found a technology — Localstack. Which allows you to use AWS services locally and it was definitely what I needed.

With Localstack in place, I decided to write testing pipeline. The first step is to run Localstack. You specify AWS services need for the tests and region if not default one needed.

export SERVICES=s3,lambda
localstack start

Then you run Terraform commands inside the directory with .tf files.

terraform init
terraform apply -lock=false -auto-approve

Now you have your infrastructure running on localstack. And you can run tests. For example:

npm test

Then we want to destroy our infrastructure, since we don’t want to have the dynamoDB or S3 with old data next time tests will be launched.

terraform destroy -lock=false -auto-approve

After this, you can destroy processes with Localstack services. For example, if we used in our tests AWS Lambda and DynamoDB — we will run these commands.

# fuser -n tcp -k $DYNAMODB_PORT
# fuser -n tcp -k $LAMBDA_PORT

That’s it! This way you can run tests using Terraform and Localstack. However, I would like to take a closer look at Terraform part.

We have AWS Lambda that works with DynamoDB. And we have Terraform modules. If you the only one who works with terraforms and responsible for DevOps it is not such bad practice to keep all terraforms code and state in one place. This way you can take a specific module from the repository. And use appropriate modules for testing.

Image for post
Image for post
structure of the repo
provider "aws" {
skip_credentials_validation = true
skip_metadata_api_check = true
s3_force_path_style = true
access_key = "mock_access_key"
secret_key = "mock_secret_key"
endpoints {
dynamodb = "http://localhost:4569"
lambda = "http://localhost:4574"
module "dynamodb" {
source = "git::ssh://<YOUR_REPO>//modules/dynamodb"
env = "test"
module "lambda" {
source = "git::ssh://<YOUR_REPO>//modules/lambda"
env = "test"

If you have small loosely coupled microservice and terraforms for it — such an approach for testing can be not so bad. But if you have a large service with a bunch of dependencies, I am not sure that this approach will have any sense. Since there are caveats out there, for example, when I write AWS Lambda that triggered by DynamoDB streams I found that Localstack has an endpoint for streams, but Terraform lacking such one. It was a problem, and I was forced to make additional changes in my infrastructure repository so that I can turn off streams. The second hit was IAM. And it is quite a big problem since I already have IAM stuff in the module. Because of this, I was forced to create IAM stuff for part of the test environment on real AWS.

Reach the next level of focus and productivity with

Image for post
Image for post

Written by

Software engineer, creator of More at

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store