Nikolay Matvienko personal blog
Contacts
#serverless #architecture #codestructure #lambda

Serverless projects architecture and code structure

Popular serverless architecture problems using Lambda:
  • Are you just getting started with serverless and don't know how to organize your code?
  • You already have experience, but what are called functions turns into projects?
  • Since the number of functions grows, maintainability becomes a nightmare?
      And if you are not experiencing these problems, then most likely the number of your functions is not large yet.

      I tell you:
      • how to reuse code in a distributed serverless architecture
      • how to avoid vendor lock-in problem
      • how to quickly create new microservices on Lambda
      in order to achieve high development speed.
      The first thing you need to do when designing a distributed serverless architecture with Lambda / Cloud Functions is to take care of the architecture of the code until your functions are in the tens.
      Nikolay Matvienko
      Serverless expert
      @matvi3nko

      Introduction to the problem

      The Cloud Function base concept

      In its basic concept, the idea of cloud functionality is pretty simple. It accepts the message, processes it, and sends the result back or to the next service.

      Services that integrate with AWS Lambda

      The number of services that integrate with AWS Lambda is growing due to Lambda versatility. Today this service solves a large number of different tasks and is truly universal. But due to:
      – number of functions used
      – number of varieties of the types of received event messages.
      With AWS Lambda you can solve completely different tasks: from DevOps to microservices and ETL pipelines, because it integrates with a lot of services:
      • S3
      • App Sync
      • Sagemaker
      • API Gateway
      • Cloud WatchKinesis Streams
      • DynamoDB and DynamoDB Streams
      • RDS and Aurora (you can call it even directly in SQL)
      • Step Functions
      • SNS, SQS, SES
      • and many others ...
      The number of your functions is growing rapidly.
      But most of the cloud functions/projects use the same Cloud Services in their combinations and your code is duplicated.
      For example:
      1. API Gateway <–> Lambda <–> DynamoDB
      2. SQS <–> Lambda <–> S3
      3. SNS <–> Lambda <–> Kinesis
      4. Cloud Watch <–> Lambda <–> Step Functions
      And as soon as you have 3+ serverless projects you face these questions
      1. How to abstract from cloud provider SDK and avoid vendor lock-in problem in cloud computing?
      2. What is the proper size and structure of the cloud function project in serverless?
      3. How to share code between different projects and reuse common logic
      In order to answer these questions, let's look inside the serverless project.
      Regardless of project type, is it just a one file (script) or well-structured code (microservice), the logic can be divided into three steps according to the basic concept:
      1. Event parsing before processing, since every event type has a completely different interface / contract.
      2. Event processing. The complexity is different, it can also use other services.
      3. Message preparation before sending according to Service interface / contract

      General internal structure of serverless application

      The diagram below shows the main blocks of logic for two scenarios of using the service: microservice or node in ETL/pipeline architecture.

      Event parsing before processing

      It is a kind of pipeline for processing events. Very similar to middleware. But it is called most often only at the stage "Before" event processing and not at "After", since after processing the event can be sent to another service.
      Contains:
      • monitoring/customer metrics agent
      • context extractor and parser
      • event body decoder
      • batch reader (for parallel event processing)
      • event validator
      • error handler
      • application initializer (create GraphQL server)
      • DB connections set up
      • caching mechanism
      • checking for event duplicate (for exactly one processing)
      • ... and many others

      Event processing

      It is good practice to split the application into abstract layers and separate the business logic from the infrastructure logic. Thus, the logic of working with the infrastructure can be easily separated, replaced and, most importantly, reused.
      Contains:
      • Application layer – handler function (or for example GraphQL server)
      • Business logic layer – your Service classes
      • Domain and Persistence logic layers – your domain models, DB entities and DB repositories classes
      • Infrastructure layer – adapters to Cloud Services and infrastructure clients libraries.

        Event sending

        To send a message, we need to write the logic for preparing a message for the required format. For this, adapters are written. They also allow not to be tied to real AWS (GCP , Azure) services and make it possible to replace them. It solves vendor lock problem.

        Using an client adapters will also save you from the need to duplicate SDK using logic and allow you to simply reuse it.

        New code for N+1 project

        When starting to create a new project, the developer should focus only on business logic, and not deal with setting up and assembling the project in parts.

        Solution

        Your personal serverless framework and project template

        A set of adapters and factories for their initialization is already a good shared library. Add there Lambda handler factory that bootstraps project and you have your framework!

        Code example

        Simple SQS Adapter in Shared Library

        Move this code into separate projects/repositories in order to use it as package.json dependencies of your Lambda projects.
        Below is an example of an adapter for a SQS service client in TypeScript for Node.js.

        This adapter encapsulates the logic for:
        • sending
        • receiving
        • deleting messages
        • and implements compression and encoding of the message.

        Possible expansion scenarios:
        • handle message groups
        • handle retries
        • handle duplications
        • etc 
        /**
 * Sqs adapter. This is an adapter for AWS SQS Client.
 */
export class SqsAdapter {
  constructor(
    private readonly logger: Logger,
    private readonly sqs: SQS,
    private readonly autoCompressionService: AutoCompressionService,
    private readonly sqsUrl: string
  ) {}

  public async receiveMessage(options: { limit?: number } = {}): Promise<SQS.Message[]> {
    try {
      const data = await this.sqs.receiveMessage({
        QueueUrl: this.sqsUrl,
        MaxNumberOfMessages: options.limit
      }).promise();

      const messages = await Promise.all(
        data.Messages.map(async message => Object.assign({}, message, {
          Body: await this.resolveMessageBody(message.Body)
        }))
      );

      this.logger.log(SqsAdapter.name, this.receiveMessage.name, '', data);

      return messages;
    } catch (err) {
      this.logger.error(SqsAdapter.name, this.receiveMessage.name, err);
      throw err;
    }
  }

  public async sendMessage(body: Object|string, compressionType: CompressionType = CompressionType.None): Promise<SQS.SendMessageResult> {
    try {
      const data = await this.sqs.sendMessage({
        QueueUrl: this.sqsUrl,
        MessageBody: await this.autoCompressionService.compress(body, compressionType)
      }).promise();
      this.logger.log(SqsAdapter.name, this.sendMessage.name, '', data);
      return data;
    } catch (err) {
      this.logger.error(SqsAdapter.name, this.sendMessage.name, err);
      throw err;
    }
  }

  public async sendMessageBatch(items: Object[]|string[], compressionType: CompressionType = CompressionType.None): Promise<SQS.SendMessageBatchResult> {
    const entries = await Promise.all(
      items.map(async (entry, i) => ({
        Id: i.toString(),
        MessageBody: await this.autoCompressionService.compress(entry, compressionType)
      }))
    );

    try {
      const data = await this.sqs.sendMessageBatch({
        QueueUrl: this.sqsUrl,
        Entries: entries
      }).promise();
      this.logger.log(SqsAdapter.name, this.sendMessageBatch.name, '', data);
      return data;
    } catch (err) {
      this.logger.error(SqsAdapter.name, this.sendMessageBatch.name, err);
      throw err;
    }
   }

  public async deleteMessage(receiptHandle: string): Promise<object> {
    try {
      const data = await this.sqs.deleteMessage({
        QueueUrl: this.sqsUrl,
        ReceiptHandle: receiptHandle
      }).promise();
      this.logger.log(SqsAdapter.name, this.deleteMessage.name, '', data);
      return data;
    } catch (err) {
      this.logger.error(SqsAdapter.name, this.deleteMessage.name, err);
      throw err;
    }
  }

  public async deleteMessageBatch(receiptHandles: string[]): Promise<SQS.DeleteMessageBatchResult> {
    try {
      const data = await this.sqs.deleteMessageBatch({
        QueueUrl: this.sqsUrl,
        Entries: receiptHandles.map((receiptHandle, i) => ({
          Id: i.toString(),
          ReceiptHandle: receiptHandle
        }))
      }).promise();
      this.logger.log(SqsAdapter.name, this.deleteMessageBatch.name, '', data);
      return data;
    } catch (err) {
      this.logger.error(SqsAdapter.name, this.deleteMessageBatch.name, err);
      throw err;
    }
  }

  public async resolveMessageBody(body: string): Promise<string> {
    return await this.autoCompressionService.decompress(body);
  }
}

        SQS Adapter Factory in Shared Library

        Encapsulate logic of adapter creation in a factory in order to avoid this code in projects. 
        export function createSqsAdapter (awsConfig: AwsConfig, logger: Logger, host: string, name: string): SqsAdapter {
    const sqsClient = new AWS.SQS({ ...awsConfig, host});
    const autoCompressionService = createAutoCompressionService(logger);
    return new SqsAdapter(logger, sqsClient, autoCompressionService, `${host}${name}`);
};

        And finally code in your Lambda project

           const someQueue: SqsAdapter = createSqsAdapter(awsConfig, logger, host, queueName);

   const results = await someQueue.sendMessageBatch([{ name: 'a' }, { name: 'b' }]);
        This is all the code above that the developer needs to write in the AWS Lambda project in order to pass messages in batch. It's literally 2 lines of code!

        Case 1: Microservice

        API microservice (REST/GraphQL) that stores/reads data from DynamoDB and sends notifications further along the chain.
        Thus, every time I create a new microservice, I write only business logic and and domain logic for its database (orange blocks on diagram). All the rest of the code (blue blocks):
        • Event/context parsing
        • GraphQL server initialization and it's caching
        • DB ORM initialization and connections creation
        • Validators
        • Custom metrics logic
        I get from the shared library(ies) and microservice template projects via npm install.

        Case 2: Async Job processing

        Continuation of the chain above. SNS sends message to SQS and then to Lambda that processes it stores in S3

        Summary

        Regardless of the set of cloud services used, writing the logic for working with them comes down only to writing adapters in Shared Labrary(ies) and importing them into Lambda projects.
        So keep focusing on business, not infrastructure!
        Main rules:
        1. Keep your business logic separate from infrastructure logic
        2. Abstract from the infrastructure, because it changes frequently in cloud
        3. And move this logic into separate projects – build you own Shared Library with custom adapters
        4. Keep AWS SDK in Shared Library and not in microservices
        5. Reuse the logic of creating a serverless project as a template and move it into Template Project – your custom framework.
        Ask me a question
        Feel free to write me. I really love to communicate.
        matvi3nko@gmail.com
        @matvi3nko
        Facebook
        Twitter
        Instagram