Developing Golang AWS Lambda Functions with Gorm & Amazon RDS for PostgreSQL

Last time, I shared my first AWS Lambda deployment. In this tutorial, I am going to demonstrate more complicated stuff. Our scenario is, creating and getting the user from the database whose architecture consists of completely AWS components.

Our toolbox is like below:

• Golang
• Gorm which is ORM for Golang. Our architecture is ORM based so if you want to use a different database, you can easily adapt our approach.
• AWS Lambda to deploy our serverless function
• AWS API Gateway to route requests
• AWS IAM for identity and access management
• AWS RDS for PostgreSQL, Amazon serves a free tier for Relational Database online. I prefer PostgreSQL which is opensource and fast.

— — — — — — — — — — — — — — — — — — — — — — — — — —

Advanced Topics:

Developing Complete Authorization Modules with AWS Lambda & Go & PostgreSQL & Event-Driven SQS…

Update: 14th Feb 2020

Project structure refactored with go modules

https://medium.com/@yunskilic/managing-dependencies-with-go-modules-4a6111d641cc

— — — — — — — — — — — — — — — — — — — — — — — — — — —

You can find source files at Github: https://github.com/yunuskilicdev/gormAws

Let’s start with Go & Gorm

The folder structure of the source code

Connector.go is responsible to open database connection only

Our model is located at model directory

We have 2 different serverless functions. One of them is responsible for creating a user, other’s responsibility is getting users

.env is used for getting environmental variables which I will describe below.

User.go:

package model

import "github.com/jinzhu/gorm"

type User struct {
   gorm.Model
   Email string `json:"email"`
   Name  string `json:"name"`
}

Our model is derived from gorm.Model which provides id, created_at, updated_at, deleted_at columns automatically. In order to focus on architectural concepts, our model is very simple which consists of two fields(name and email).

Connector.go

package db

import (
   "fmt"
   "github.com/jinzhu/gorm"
   _ "github.com/jinzhu/gorm/dialects/postgres"
   "github.com/joho/godotenv"
   "os"
)

type PostgresConnector struct {
}

func (p *PostgresConnector) GetConnection() (db *gorm.DB, err error) {
   e := godotenv.Load()
   if e != nil {
      fmt.Print(e)
   }
   username := os.Getenv("db_user")
   password := os.Getenv("db_pass")
   dbName := os.Getenv("db_name")
   dbHost := os.Getenv("db_host")
   dbURI := fmt.Sprintf("host=%s user=%s dbname=%s sslmode=disable password=%s", dbHost, username, dbName, password)
   return gorm.Open("postgres", dbURI)
}

Our connector class is only responsibility is the opening connection. While deploying your code, you can give environmental variables. The best approach is reading the database connection parameters from these variables.

os.Getenv("db_user")

is the way of reading variable at go and AWS. But at the development phase, you can use “github.com/joho/godotenv” to read local variables. So you have to create .env file into your source code directory. So I also created .env file for development phase purpose. An example definition of the variable at .env file like below:

db_host=test1
db_user=test2
db_name=test3
db_pass=test4

But we need a PostgreSQL server to fill these properties. I preferred AWS RDS.

AWS RDS: Set up, operate, and scale a relational database in the cloud with just a few clicks.

Also AWS RDS servers a free tier which makes it is a good choice for PoC developments. There are different types of RDMS. But I am a PostgreSQL fun so I choose PostgreSQL.

AWS RDS for PostgreSQL creation steps:

• At AWS RDS dashboard, create database
• Choose PostgreSQL
• Check Only enable options eligible for RDS Free Usage Tier & Next
• Fill settings fields & Next
• If you like to use publicly choose Public accessibility (I choose)
• Fill fields and create it
• After a while creation operation will be done. You can see your new database at databases menu. Click its name. Then you can get the endpoint URL from the database instance which is db_host. You can fill the values of db_user, db_name, db_pass whose values are filled while creating the database by you.
• Our database is ready now :)

Create User function like below:

package main

import (
   "context"
   "github.com/aws/aws-lambda-go/lambda"
   "gormAws/db"
   "gormAws/model"
)

type CreateUserRequest struct {
   Email string `json:"email"`
   Name  string `json:"name"`
}

func HandleRequest(ctx context.Context, request CreateUserRequest) (model.User, error) {
   postgresConnector := db.PostgresConnector{}
   db2, err := postgresConnector.GetConnection()
   defer db2.Close()
   if err != nil {
      return model.User{}, err
   }
   db2.AutoMigrate(&model.User{})
   account := &model.User{}
   account.Email = request.Email
   account.Name = request.Name
   db2.Create(account)
   return *account, nil
}
func main() {
   lambda.Start(HandleRequest)
}

As you know “github.com/aws/aws-lambda-go/lambda” is used for AWS Lambda. The function opens a connection from AWS RDS for PostgreSQL. It reads input JSON and parses it into CreateUserRequest type.

Gorm Auto Migrate: Automatically migrate your schema, to keep your schema update to date.

Then create a user object and fill its fields. Then you can create the object at DB. Finally returns created object.

Let’s deploy our function to AWS Lambda.

Last time, I wrote an article about how to deploy golang functions to the AWS Lambda for the first time. If you have not any experience with AWS, you can check out with this link:

How To Add Golang AWS Lambda For Who Don’t Know Anything About AWS

After deployment finish, let's test our function.

I called my function goCreateUser. At Lambda dashboard click on your function.

• Click test to create test case

• Save test case.
• When you click the test button again, your function will be tested with these inputs.

If you click test again, oooopsss there will be an error probably.

Problem:

Lambda function could not access AWS RDS instance

Solutions:

1-) You can open RDS to the whole internet.

2-) Lambda and RDS should be in the same VPC and security group.

I suggest the second solution. Let’s apply it.

• At Lambda dashboard, click your function(in my case: goCreateUser)
• Select default vpc, choose some subnets, select rds-launch-wizard-X

for the security group. Remember, you should use these values for the RDS configurations.

Important note: In order to save this configuration, you have to add extra permission which is called AWSLambdaVPCAccessExecutionRole. You can add this role at IAM dashboard to the role which is related to your Lambda function.

My role’s appearance:

Lambda configuration:

• Configure RDS:

• VPC, Subnets should be the same. Also, you can make the security group the same.
• Click on rds-launch-wizard-x
• Select inbound at the bottom pane & Click edit

• Top two rows make accessible from the internet. For basic development phases requirements, you can open RDS to the internet. But you should close later.
• Also, you make accessible for the security group. For example, rds-launch-wizard-x has an id, in my case start with sg******* , paste here to allow that group.

Now our RDS is ready for Lambda requests. When you click test again at lambda function’s page. Result is:

Create a user function is ready. Let’s move to the get user function.

package main

import (
   "context"
   "github.com/aws/aws-lambda-go/lambda"
   "gormAws/db"
   "gormAws/model"
)

type GetUserRequest struct {
   Email string `json:"email"`
   Name  string `json:"name"`
}

func HandleRequest(ctx context.Context, request GetUserRequest) ([]model.User, error) {
   postgresConnector := db.PostgresConnector{}
   db2, err := postgresConnector.GetConnection()
   defer db2.Close()
   if err != nil {
      return []model.User{}, err
   }
   db2.AutoMigrate(&model.User{})
   account := &model.User{}
   if request.Email != "" {
      account.Email = request.Email
   }
   if request.Name != "" {
      account.Name = request.Name
   }
   var users []model.User
   db2.Where(account).Find(&users)
   return users, nil
}
func main() {
   lambda.Start(HandleRequest)
}

Again our logic is very simple. A user can be searched with a name and/or email.

You can add where clause and select expression easily with gorm.

db2.Where(account).Find(&users)

Let’s test it. Deploy source code to AWS. I called goGetUser. After deployment

click goGetUser and then click to the test to create test cases.

Result:

[
  {
    "ID": 2,
    "CreatedAt": "2019-03-05T10:10:55.404383Z",
    "UpdatedAt": "2019-03-05T10:10:55.404383Z",
    "DeletedAt": null,
    "email": "test",
    "name": "test"
  },
  {
    "ID": 3,
    "CreatedAt": "2019-03-05T10:32:20.630973Z",
    "UpdatedAt": "2019-03-05T10:32:20.630973Z",
    "DeletedAt": null,
    "email": "test",
    "name": "test"
  },
  {
    "ID": 4,
    "CreatedAt": "2019-03-05T10:32:28.735692Z",
    "UpdatedAt": "2019-03-05T10:32:28.735692Z",
    "DeletedAt": null,
    "email": "test11",
    "name": "test"
  },
  {
    "ID": 6,
    "CreatedAt": "2019-03-05T12:24:17.143682Z",
    "UpdatedAt": "2019-03-05T12:24:17.143682Z",
    "DeletedAt": null,
    "email": "test11",
    "name": "test"
  }
]

Our functions are ready. So let’s assign URL and HTTP method to them, then open the whole internet. We need AWS API Gateway to handle these.

• Click Services, write API Gateway and click on it.
• Then click Create API

• Click gormAws & Then resource & Action & Create Method
• An empty dropdown list will appear. Choose HTTP Method.
• For create user method choose post
• Double click on post.

• Enter your create user function to the lambda function textbox then save.(goCreateUser)
• Select the same things for get user function. But choose the GET HTTP method.

But there is an important point that you have to think about. Our functions are waiting for JSON as input. But GET method does not have a request body. So you have to deal with this.

• Click GET
• You will see 4 different rectangles Method Request, Integration Request, Integration Response, Method Response.
• Click method request to create query params. Add email and name as query params.

• Finally, you should parse these query params to JSON.
• Choose Integration Request this time, at the bottom select Mapping Templates.
• Add mapping templates, write application/json and click on it.

• Our API is ready to deploy.
• Click the actions button & Deploy API.
• Choose stage and deploy.
• AWS will give you an Invoke URL: https://xxxxxx.execute-api.eu-central-1.amazonaws.com/dev
• Use your endpoint in high spirits:)