What the heck is DevOps and why does it matters?
Say hello to DORA (Google Cloud’s DevOps Research and Assessment)
Spanish version here!
In this article/glossary of terms, I’m going to introduce, for those who are not yet familiar with, the term DevOps. And it happens that during our journey towards the world of data, sometimes we hear terms and tools that we have never seen before. This makes us feel our impostor syndrome to the fullest. The idea is to know a little bit about the DevOps world, to decide if we make the transition to a DevOps career— very lucrative and in demand by the way — or if we continue to enter the data world.
We will cover the following topics:
- What is DevOps?
- What are the benefits of DevOps?
- 7 Essential DevOps Practices
- What about DevOps Tooling?
- Google Cloud’s DevOps Research and Assessment (DORA)
- Are you an Elite DevOps performer?
Are you comfortable? Well let’s get started!
What is DevOps?
DevOps is a paradigm shift that bridges the gap between the siloed development and operations teams. The word “DevOps” is a mashup of “development” and “operations”. DevOps describes approaches to speeding up the processes by which an idea (like a new software feature, a request for enhancement, or a bug fix) goes from development to deployment in a production environment where it can provide value to the user.
What are the benefits of DevOps?
DevOps offers a productive environment that nurtures mutual collaboration, fruitful communication, and seamless integration across teams in an organization.
Let’s see some benefits of implementing DevOps in an organization:
Increase the speed of your deployments:
You can increase the speed and frequency of your releases, adapt short release cycles, and reduce time to market with the help of DevOps. When you release the product and its features quickly, you can quickly perform quick bug fixes, respond to customer needs faster, and get a competitive edge in the market with improved quality products.
Improve the stability of your software
Practices like CI/CD and real-time monitoring come in handy to maintain system stability.
Build security in from the start
It helps you track security issues, detect them, and resolve them before they could cause any harm.
7 Essential DevOps Practices
1. Agile Software Development
Agile is one of the software development approaches that focus on team collaboration, higher adaptability to changing demands, user feedback, and shorter release cycles.
2. Embrace Failure
To fully embrace cloud computing through DevOps, a brand must change their attitude towards failure. By accepting failure, companies foster a “climate for learning” which will positively impact organizational culture.
3. Version Control
Version control involves managing codes in multiple versions with revisions and modification history, so it becomes easy to review the codes and recover them.
It’s an essential DevOps practice that helps your development teams to collaborate more effectively, store codes safely, divide tasks between your team members, merge changes, handle conflicts easily, and roll back code changes if needed.
4. Continuous Integration (CI) & Continuous Delivery (CD)
Continuous Integration (CI) is a DevOps best practice for integrating code changes from multiple contributors into a single repository. CI helps developers to integrate code changes regularly and evaluate the code’s correctness with automated tools.
A source code version management system is used to achieve continuous integration. Continuous integration aims to detect and fix errors earlier in the development cycle, increase overall code quality, and speed up deployment.
The frequency will increase naturally as teams enhance the overall process by working together. Bugs discovered early in the development cycle help teams prevent larger problems after the code is pushed to production. This reduces technical debt over time.
Continuous delivery is a DevOps technique for getting code from continuous integration to production.
On top of automated testing, CD offers an automatic release procedure, which you can easily deploy with just a few clicks.
5. Infrastructure as Code (IaC)
IaC defines system topologies and resources to allow your team to manage the resources during the coding phase reliably in a controlled manner. You can store and version those definitions in a version control system, so your team can review and revert quickly. It also helps reduce human errors in complex production environments.
6. Microservices
A distributed application with a microservices architecture combines numerous services, each executing a specific feature or function. Each service can be deployed independently, perform its rations, and communicate with other services via a web interface in the microservices architectural pattern. Different languages or frameworks can be used to write the different services in a microservices architecture, and they can be deployed individually or as a group.
It’s easier to avoid a single point of failure in a microservices design by ensuring that any issue with one service doesn’t affect another. Microservices are also easy to expand because the DevOps team can simply add a new microservice to extend the functionality without affecting the rest of the application.
7. Observability
Monitoring has become significantly more complex as the industry has evolved away from monolithic, on-premise systems and applications to cloud-native, microservice-based solutions. As a result, observability is becoming increasingly popular. The three pillars of observability are sometimes referred to as logs, traces, and metrics.
The majority of system components and applications produce logs, which contain time-series data on the system’s or application’s operation. Traces are used to track the logic flow within the application. CPU/RAM reservation or utilization, disk storage, network connectivity, and other metrics are among the metrics available.
What about DevOps Tooling?
While adopting a specific tool or technology is not the same as adopting DevOps, when the DevOps culture is present and the processes are defined, people can implement and streamline DevOps practices if they choose the proper tools. Let’s review some of them.
Containers
Containers make moving applications between development, testing, and production environments easier. Using containers lets developers package and isolate their apps with everything they need to run, including application files, runtime environments, dependent libraries, and configurations.
At its core, DevOps relies on automating routine operational tasks and standardizing environments across an app’s lifecycle. Containers can offer standardized environments.
Docker
Docker is an open-source platform that enables developers to build, deploy, run, update and manage containers.
Kubernetes
Modern software is increasingly run as fleets of containers, sometimes called microservices. But you need a platform to manage them. That’s where Kubernetes comes in.
Kubernetes, is an open-source software that automatically manages, scales, and maintains multi-container workloads in desired states.
Also referred to as K8s, Kubernetes was developed originally by Google, and released in 2014, but now the Cloud Native Computing Foundation takes care of its maintenance.
Kubernetes is becoming the de-facto standard for running microservices infrastructures in the cloud. Most of the bigger cloud providers offer managed Kubernetes clusters:
- Kubernetes on Google Cloud: Google Kubernetes Engine (GKE for short) is the industry’s first fully managed Kubernetes service that implements full Kubernetes API, 4-way autoscaling, release channels, and multi-cluster support.
- Kubernetes on AWS: Amazon Elastic Kubernetes Service (EKS), a certified conformant, managed Kubernetes service that manages clusters of Amazon Elastic Compute Cloud (EC2) compute instances and runs containers on those instances.
- Kubernetes on Azure: Microsoft Azure Kubernetes Service (AKS) deploys and scales containers on a managed Kubernetes.
Helm
Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Some popular Helm Charts are mysql, trino, presto, clickhouse, mariadb, postgresql, pgadmin, ibm db2, apache ignite, sql server, zeppelin among others.
Git
Git is a free and open-source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.
GitHub
GitHub is an internet hosting service for software development and version control using Git. It provides the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous integration, and wikis for every project.
GitLab
GitLab is a web-based Git repository that provides free open and private repositories, issue-following capabilities, and wikis.
Jenkins
Jenkins is used to build and test your product continuously, so developers can continuously integrate changes into the build. Jenkins is the most popular open source CI/CD tool on the market today.
Terraform
Terraform is an infrastructure as code tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share. Terraform can manage low-level components like compute, storage, and networking resources, as well as high-level components like DNS entries and SaaS features.
ELK Stack
ELK Stack is a powerful combination of three open-source, popular tools: Elasticsearch, Logstash, & Kibana.
- Elasticsearch is a RESTful, open source, distributed analytics and search tool built on Apache License. It supports various languages, offers high performance, includes schema-free JSON files, etc. that make this tool a great alternative for search and log analytics.
- Logstash is also an open-source tool for data ingestion that you can use to gather data from different sources, modify it, and send them to wherever you want. It supports 200+ plugins and pre-built filters.
- Kibana is an open-source data exploration and visualization tool to review logs and events. It has effortless, pre-built, interactive filters and aggregations, and comes with geospatial support for data visualization.
There are many, many tools but these are enough. What about the cloud? Well, according to DORA:
Cloud Infrastructure helps… but only if you do it right.
Google Cloud’s DevOps Research and Assessment (DORA)
Google Cloud’s DevOps Research and Assessment (DORA) team has conducted a seven-year research program. This research has validated several technical, process, measurement, and cultural capabilities that drive higher software delivery and organizational performance. To allow teams to benchmark themselves against the industry, DORA used cluster analysis to form meaningful performance categories (such as low, medium, high, or elite performers).
The DORA team has identified four key metrics that indicate the performance of a software development team:
- Deployment Frequency — How often an organization successfully releases to production.
- Lead Time for Changes — The amount of time it takes a commit to get into production.
- Change Failure Rate — The percentage of deployments causing a failure in production.
- Time to Restore Service — How long it takes an organization to recover from a failure in production.
At a high level, Deployment Frequency and Lead Time for Changes measure velocity, while Change Failure Rate and Time to Restore Service measure stability. And by measuring these values, and continuously iterating to improve on them, a team can achieve significantly better business outcomes.
This is what they found:
Comparing the elite group against the low performers, DORA found that elite performers have…
- 973x more frequent code deployments
- 6570x faster lead time from commit to deploy
- 3x lower change failure rate (changes are 1/3 less likely to fail)
- 6570x faster time to recover from incidents
Are you an Elite DevOps performer?
You can measure your team’s software delivery performance in less than a minute by going to this link.
Thanks for reading! How about another reading?
Do you want more?
Hit 50 times that 👏 button and something wonderful will happen.
- Follow David Regalado for more nerdy talks!
- Follow Dataengineering Latam (Data Engineering Latam) for more content related to Data Engineering, Data Science and Data Management.
