placeholder
thoughts and learnings in software engineering by Rotem Tamir

Serverless Applications, continuous delivery with AWS Lambda and API Gateway — Part 2, Deploying Lambda

We built a serverless microservice to manage our customers’ pipeline configuration in [ironSource atom](https://ironsrc.com/atom) We built a serverless microservice to manage our customers’ pipeline configuration in ironSource atom

Recap

In the previous part of this series, we began the tale of the serverless microservice we created in ironSource atom to help us manage the configuration of our customers’ big-data pipelines. We discussed the huge benefits we get from working with AWS Lambda and API Gateway in terms of availability and general peace of mind in terms of our production environment. But we also reviewed some pain points in the development process. These pain points set us on a course to to build an automated flow of deploying a serverless micro-service.

In the first part of this trilogy (you are now reading “The Empire Strikes Back”) we discussed setting up proper unit-tests for our Lambda functions in Node.js, so we can have a good time developing on our own machine.

In this part, we will describe some tools we adopted and practices we’ve developed for actually interacting with AWS Lambda, since working directly with Amazon’s UI can be cumbersome and error-prone.

Introducing Apex

First of all, several tools exist in the market today to help streamline the process of working with Lambda, ranging from small utility libraries to full blown solutions like “Serverless” (now in beta). When evaluating different solutions we chose Apex. Its minimal approach was appealing to us, it was well documented and seemed to have a decent sized community of developers (currently over 3.5k stars on GitHub).

Apex is simple to use and well-documented, so I won’t go into details here on how to set up or get started with it. Instead, I’d rather just go over how we use it in our team. Apex relies on some pretty straightforward JSON files to describe configuration and on a pre-defined structure for your project. In this post, we’ll be working on a hypothetical microservice which allows others to search for repositories in GitHub. This service will expose a single endpoint: “GET /repos” and receive input in the query string like “/repos?q=rotemtam”. Working with Apex, our sample project structure would look something like this:

├── Dockerfile
├── infra
│   └── webpack.config.js
├── package.json
└── src
    ├── functions
    │   └── getRepositories
    │       ├── function.json
    │       ├── src
    │       │   └── index.js
    │       └── tests
    │           └── getRepsitories.spec.js
    ├── lib
    │   ├── lambda-co-runner.js
    │   └── mock-lambda-context.js
    ├── model
    │   └── Repository.js
    └── project.json

Apex sample project structure (view the source here)

  • In the “src/” directory of our project we have a project.json file describing to Apex general things about the project
  • We have a functions directory with a separate directory for each Lambda function.
  • Inside each function’s directory, we have a function.json file describing configuration for that specific function.
  • We’ll get into the Dockerfile and the “infra/” directory in a bit.

When we run “apex deploy” Apex will create new versions of our functions in our AWS account and update the configuration.

Sharing dependencies between functions

TJ Holowaychuk (one of Apex’s creators) recently published an article on Medium with some “Do’s and Don’ts of AWS Lambda”, one of which is very relevant to our topic: “Don’t substitute FaaS with writing good libraries”, which basically means that your actual Lambda functions should be very small. You should take advantage of the package/module system in your language to encapsulate whatever logic you need in an ordinary module and use it. A general rule of thumb would be: Handle input and high-level flow-control in the Lambda function, leave all the rest to your libraries. This way you can easily re-use them in other contexts.

Remember our hypothetical microservice for searching GitHub? Well, assume it contained a Lambda function which would take a string q as input and use that to search GitHub for repositories with that string in their name. There is no reason to include the call to the GitHub API in our Lambda function. Instead we could do something like this:

'use strict';

const Repository = require('../../../model/Repository')
  , LambdaRunner = require('../../../lib/lambda-co-runner');

function *main(e) {
  let query = e.queryParams.q
    , res = yield Repository.search(query);
  return {items: res.items};
}

module.exports = {
  default: LambdaRunner(main)
}

Which would rely on a “Repository” module like this:

'use strict';

const request = require('axios');

function *search(query) {
  let options = {
    params: {
        q: query,
        sort: 'stars',
        order: 'desc'
    },
    headers: {
        'User-Agent': 'rotemtam'
    },
    responseType: 'json'
  };
  let res = yield request.get('https://api.github.com/search/repositories', options);
  return {items: res.data.items};
}

module.exports = {
  search: search
};

By default, Apex will take whatever is in the function directory, put it in a zip file and upload to Lambda. So basically, if you want to include any modules you wrote or 3rd party stuff from NPM, you need to include it there. That works fine for a project with only one function, but what if you have several functions that share modules? Would you have a Node.js package.json file for each function and manage the dependencies per function? Copy and paste your custom modules into each folder?

Using Webpack and Apex hooks to create deployable artifacts

Fortunately, Apex lets you customize your build process by specifying life-cycle hooks, namely custom “build” and “clean” commands. We take advantage of this to make use of Webpack to create our deployable artifacts. At build time, webpack will intelligently create a single Javascript file which apex will zip up and deploy. Here’s an example Apex project.json file taken from our example repo:

{
  "name": "SampleProject",
  "description": "A demo project of serverless ci",
  "role": "arn:aws:iam::<aws account number>:role/<lambda role name>",
  "runtime":"nodejs4.3",
  "memory": 128,
  "timeout": 10,
  "handler": "lib.default",
  "hooks": {
    "build": "../../../node_modules/.bin/webpack --config ../../../infra/webpack.config.js",
    "clean": "rm -fr lib"
  }
}

The only caveat with this method is that the official AWS Javascript SDK doesn’t play nice with Webpack. Check out this discussion to learn more about it. Luckily for us, it turns out that the SDK comes pre-bundled in the Lambda context (super thanks to Victor Delgado for pointing this out), so we can basically tell webpack to ignore it in our webpack config:

module.exports = {
  entry: './src/index.js',
  target: 'node',
  output: {
    path: './lib',
    filename: 'index.js',
    libraryTarget: 'commonjs2'
  },
  externals: {
    'aws-sdk': 'aws-sdk'
  },
  module: {
    loaders: [
      {
        test: /\.json$/,
        loader: 'json-loader'
      }
    ]
  }
}

Hooking things up to a CI/CD service

The next part of our process is to hook this up to a CI/CD service. We currently use Jenkins and Codefresh for our builds, but this could easily be done with any other service. Let’s list our requirements for a build job for our microservice:

  • Install all required packages, install Apex.
  • Run unit tests, exit if they fail.
  • Run “apex deploy” to push our Lambda functions to production.
  • (Build a Swagger file describing our API Gateway) — we’ll get to this in the next post
  • (Redeploy our API Gateway) — coming soon!

Building inside a Docker container

No truly hip deployment process would be complete without using Docker somewhere, right? Seriously though, there are many advantages to using Docker in your build process.

People usually consider Docker containers for running production tasks. Docker containers really are a great way to have complete control over the environment which your code runs in. With Docker you can have an exact replica of your production environment. But the same benefit is true for doing any other type of task, really. By running CI/CD tasks in Docker, you have complete control over the way things happen and you can use whatever tools you like. I usually find that it’s just easier (and faster) for me to run my builds inside Docker containers. Here’s a Dockerfile which I’d use for a task like we described above:

FROM node:4.2.3

# Install Apex
WORKDIR /tmp
RUN wget https://raw.githubusercontent.com/apex/apex/master/install.sh && bash /tmp/install.sh

# Install dependencies
COPY package.json /srv/package.json
WORKDIR /srv
RUN npm i && mkdir /srv/src && mkdir /srv/infra

# Copy code into container
COPY src /srv/src
COPY infra /srv/infra

# Run
WORKDIR /srv/src
CMD ["apex", "deploy"]

Our final build script

Our build script might look like this:

#!/bin/bash -e

PATH_TO_AWS_CREDENTIALS=~/.aws/credentials
AWS_REGION=us-east-1

# build the image
docker build . -t apex-lambda-deployer

# run unit tests
docker run apex-lambda-deployer npm test

# deploy
docker run -e AWS_REGION=$AWS_REGION -v $PATH_TO_AWS_CREDENTIALS:/root/.aws/credentials apex-lambda-deployer

Lets break it down:

  • Depending on where your build job runs, you might need to deal with setting AWS environment variables. For example, if you run from an EC2 instance that has a role which allows it to interact with Lambda you won’t need it.
  • Apex needs to know what region it’s deploying to, which we set using the $AWS_REGION environment variable
  • We first run “npm test” inside our container, if all unit tests pass we continue.
  • If all is good, we continue to run “apex build” inside our container.

We take this build script and plug it into whatever system we’re using to build our code, and BAM! We have an automated CI/CD flow for working with AWS Lambda!

Wrapping Up

  • In the previous post of this series, we discussed how to setup unit-tests for Lambda functions.
  • In this post, we’ve outlined how to create an automated workflow for deploying Lambda functions to production environments.
  • In the third and last part of this trilogy (Return of the Jedi), we will review how we set up API Gateway to make our service available as a RESTful Web API and how we automate the grunt work of continuously deploying it with a cool tool we wrote.

See you soon!

By Rotem Tamir on August 30, 2016.

Canonical link

Exported from Medium on June 16, 2019.