We want to deliver our products as temporary acceptance systems to be tested by our developers, testers, and pre-sales engineers internally before their release. How do we have setup such system using CodeBuild, CloudFormation and ECS ?
Our company, GBANDSMITH edits multiple software solutions (the 360Suite portfolio) to help our customers use efficiently SAP BusinessObjects (BOBJ in this article). But as our testing processing are improving, we are increasingly struggling with the installation and deployment of multiple products on test environments. It takes time and generates frustration for developers and pre-sales engineers.
We now have dedicated resources for testing itself, and we don’t want them to struggle with the installation/provisioning phases but to focus on testing.
The GB&SMITH Labs team has been asked to find a solution on this issue!
We have worked to deliver a global solution that answers the following needs:
From the beginning, we wanted to work on this project only with resources available on our main cloud environment, Amazon Web Services. The goal was to prove that the team is able to deliver a real cloud solution, and to trial-and-error on some new tools available on AWS. So the global design of the solution is intended to use instances only if needed.
We are using the following products of Amazon Web Service:
Let’s dig a bit on our choice. Choosing the right tools is not obvious, so let me take a few minutes to explain why we have created such a stack.
ECS is one of the solutions in AWS if you want to deploy docker containers. We think that it is easier to deal with docker containers and images to deploy our products. It is easier and faster than working with instance images:
maven clean install )docker build . )docker tag ...and then docker push ...)We are dealing here with couple of Mb (around 500 Mb for a docker image) instead of multiple Gb for a instance image. There is no easy way to generate a instance image containing our product from a CI solution like CodeBuild (the best I know in this area is the very good Packer tool from HashiCorp).
In an ideal scenario, I would avoid an EC2 instance. It relies on an EC2 image (Amazon Machine Image), and it is bulky to modify. But here we want to deliver a BI solution, and officially, this solution is not compatible with Docker containers. I know some users are trying to dockerize BOBJ (here for example), but we haven’t succeed in this task.
RDS is the de-facto standard solution to deploy a database in AWS. Both our products relies on a database, so we need to ship one. We chose RDS.
Actually it is not the only solution, we can also imagine adding a database container in our ECS task, but this solution complexifies the connectivity between the BOBJ instance and the database.
CloudFormation is the Infrastructure as Code solution in AWS. There are also other tools available like Terraform from HashiCorp (again!).
We chose to use CloudFormation in this project because it allows us to create a full stack and also to tear-down the stack very easily from the AWS API when the user doesn’t need the solution anymore (it is a bit more difficult to teardown a terraform setup, not a single AWS API call !)
Our company mainly uses Jenkins as CI solution. Starting on CodeBuild is not obvious ! The goal of the project was to use a completly AWS solution without any external product.
CodeBuild is quite a mature product now, which allows to setup a project with conditional builds, and to consume multiple git repositories. It also allows to push docker images to ECR, and to trigger the CloudFormation stack creation. Here is an exemple of the buildspec provided in the project:
version: 0.2cache: paths: - '/root/.m2/**/*'phases: install: runtime-versions: java: openjdk8 pre_build: commands: - echo Logging in to Amazon ECR... - $(aws ecr get-login --no-include-email --region $AWS_REGION) build: commands: - | if test "$BUILD_EYES" = "1"; then echo "Build 360eyes docker image and push it to ECR"; cd $CODEBUILD_SRC_DIR/360ace/eyes; mvn clean install -Pdocker; docker tag 360eyes/360eyes.rest.service:latest xxx.dkr.ecr.$AWS_REGION.amazonaws.com/360ace/eyes:latest; docker push xxx.dkr.ecr.$AWS_REGION.amazonaws.com/360ace/eyes:latest; fi - echo Launching CloudFormation template - cd $CODEBUILD_SRC_DIR/360ace - aws cloudformation create-stack --template-body file://360ace.yaml --stack-name ace-$(date +%d%H%M)
This is just does the following:
A good point is that it allows to checkout a specific branch (like a developer working branch) or a specific tag (a release) of our product repositories.
Here we see that we have some temporary resources created when a solution is up, and some more stable resources that continue to live even when no solution is running:
The temporary resources are directly managed by the CloudFormation stack: when the stack is created it fires up all the linked resources. When the stack is stopped, then the linked resources are deleted.
The stable resources are designed to live all the time. But instead of creating them directly in the AWS Console, we prefer to use Terraform (again again !) to manage that. It allows us to have a clean setup that we can reproduce on multiple AWS accounts. We are also able to review the infrastructure changes within a git repository and to take full advantage of Infrastructure as Code.
As a result now our employees can start a temporary solution of our solutions just by triggering a CI job:
And all is presented in a quick and dirty React application (the home container):
OK, the solution is here and available but there is still a lot of room for improvements.
Because we are only on AWS, we imagined some additional improvements to our system: it is possible to setup a chatbot that is able to create and tear down a system just by speaking to it. It communicates only with the AWS API (CodeBuild and CloudFormation):
Currently we are relying on a RDS snapshot to pop-up some existing data in the RDS database. It is easy to setup but it is dramatically long to fire, around 20 minutes…We are thinking on how we can speed up this (maybe by using a different system).
We can also imagine adding an option to select a specific BOBJ version to start (BI4.1, BI4.2, …), or select the RDS backend type (MySQL, PostgresSQL, Oracle…). All of that is possible and we are waiting for our internal customers to get some usage feedback !
Cloud is hard. We have just scratched the surface of the issues and the options offered by a cloud platform like AWS. We still have lots of methods, skills, and tools to learn. But we are very happy to be able to deliver a working tool to our co-workers, waiting for the next project !
See you in a new article with all the info on this new MCQ season…
E-commerce's rapid expansion has changed how businesses function and how consumers shop, emphasizing the need…
In modern web development, scalability has become a key concern. As applications grow in complexity…
We meet again as usual for the start of the start of autumn season QCM…
The WeLoveDevs MCQs winter season came to an end on February 29th, and Manuvai Rehua…
There are several announcements today. The first is that it's the end of Winter and…