DevOps + MLOps
Communications & Media

From Inconsistent to Integrated: How Automating CI/CD Boosts Quality & Security

Trility collaborated with the client to identify inconsistencies and unreliability in this client’s product lifecycle. By implementing DevOps practices, security automation, and automated CI/CD pipelines, they improved three KPIs: increased frequency of releases, reduced change failure rates, and improved mean time to resolution (MTTR).

Problem Statement

The client faced a critical business problem where product development was not meeting the customer needs in a timely fashion due to inconsistent lifecycle practices across development, delivery, implementation, and support. This led to a reputation for unreliable products, hindering timely market delivery and customer adoption of system upgrades, ultimately impacting revenue and growth.

Technically, the client struggled with a highly fragmented and ad-hoc development environment. They had over 600 developers using more than 70 coding languages across eight engineering value streams with limited security scanning, leading to critical vulnerabilities. 

The existing CI/CD process lacked standardization, relying on chained makefiles and one-off containers rather than true pipeline features. This resulted in inconsistent versioning, build processes, and packaging across different value streams, making it extremely challenging to manage and deploy software effectively. Many of the software dependencies were deprecated and unsupported, creating significant security and maintenance risks.

Solution Approach

Trility initially addressed the immediate need for vulnerability detection and a more secure software development process by implementing a comprehensive security scanning suite for static software composition, and container analysis. Trility also built the necessary infrastructure to host and integrate these tools into the existing pipelines. Then, Trility migrated the client's disparate CI/CD processes to GitLab, establishing a standardized, modular pipeline-as-a-service approach:

  • Implemented standardized, adaptive pipelines (Python, Maven, Helm) with a Project Feature Detector, reducing manual effort and inconsistencies. 

  • All GitLab configurations (projects, directories, permissions) were managed via Terraform (Infrastructure as Code). 

  • Adopted Test-Driven Development for pipeline components and introduced Renovate for automated dependency management, including vulnerability updates, which also integrated disparate software sources into containerized network functions. 

Outcomes

The client gained a simplified ecosystem resulting in consistent, higher-quality products. Each product line has a solid foundation to ensure confidence in quality, reliability, and security upon commercial launch.

Reliability & Security: Dependency management reduces vulnerabilities and keeps software up-to-date, minimizing cyberattack exposure. Automatic scanning on merge requests identifies security gaps, establishes portfolio-wide monthly reporting on all vulnerabilities, and tracks them by value stream to track against KPI goals.

CI/CD Pipeline-as-a-Service: Eliminated inconsistent pipelines, reducing deployment steps from 52 to 8, moving full network deployment from 8 days to 50 minutes.

Reduced Time-to-Market: Simplified, automated delivery enables daily or weekly internal releases and monthly external ones, an improvement over previous quarterly cadences.

Cultural Shifts: Standardization freed engineers for core development, fostering a culture where DevOps serves developers, leading to better tools and efficient workflows. DevOps now owns the platform and promotes standardized requests.

Project Attributes

  • Reduced COA
  • Reduced COO
  • Reduced Risk
  • Reduced Technical Debt
  • Accelerate Delivery
  • Increased Uptime
  • Increased Automation
  • Increased Scalability
  • Reusable Patterns
  • Increased Capabilities
  • Increased Security
  • Coaching
  • Documentation
  • Paired Programming
  • Videos
  • Learning Sessions

Technologies Used

  • Terraform
  • Jenkins
  • Kubernetes
  • Coverity
  • Black Duck
  • SonarQube
  • Aqua Security tfsec
  • Python
  • Maven
  • Helm
  • GitLab
  • Renovate