Vulnerabilities in CI / CD pipelines are a found food for cybercriminals. This keeps your continuous integration and delivery pipeline secure. […]
Cyberattacks that exploit vulnerabilities in continuous integration/Continuous Delivery (CI/CD) pipelines and developer tools underscore the need to increase the security of development infrastructures. In particular, the codecov supply chain attack should be a warning to all companies never to store secret information in CI/CD environment variables-no matter how secure the environment may seem.
The Codecov attackers managed to sneak in a bash uploader, which is used by thousands of developers to grab credentials, keys, and API tokens from customer environments. The intruders went undetected for over two months and, according to media reports, were able to compromise hundreds of restricted customer networks. Attacks on automation tools such as Jenkins, GitHub Actions or cloud-native container environments have also prompted companies to research and deploy effective protections for these tools. The following best practices will help you secure your CI / CD pipelines.
The codecov supply chain attack was so successful because the environment variables exfiltrated by the attackers contained hard-coded, secret information, including passwords, tokens, and keys. These credentials also gave attackers access to companies ‘ private GitHub repositories. This resulted in a further exfiltration of confidential.
Several Codecov customers, including HashiCorp, Twilio, Rapid7 and Monday.com fully understood the impact of the supply chain attack. However, the most significant incident in this context took place at the Japanese e-commerce giant Mercari: more than 27,000 records providing information on the finances of customers, merchants, business partners, employees, contractors and various entities were published by external actors after the codecov attack.
In particular, the question of why personal data such as financial information of customers was stored in private GitHub repositories was raised. Similar concerns have been raised regarding HashiCorp’s private GPG key stored in the CI/CD environment. This is the secret key that HashiCorp uses to sign and verify released software releases. An attacker could have misused this key to sign a malicious one.
Organizations need to rethink what information can be stored in CI/CD tools, environment variables, and private GitHub repositories. When an application requests a credential or token, it is recommended that you set up credentials for an account or resource with the lowest privileges. Thus, the damage is limited even if the information is compromised.
CI / CD automation tools such as GitHub Actions allow developers to set up scheduled tasks for their code repositories, such as automatically checking and processing incoming pull requests. But what if the user making the pull request for an open source project has malicious intentions?
In April 2021, GitHub Actions was abused by attackers who made automated pull requests to hundreds of repositories. Their goal: to mine cryptocurrency via the infrastructure of GitHub. This large-scale attack occurred after the “bug” was reported on GitHub Actions back in early February. Such pull requests can be misused to mine cryptocurrency or execute malicious code via those of GitHub. If project managers act negligently and merge these pull requests, they introduce malicious code into their repository and the broader software supply chain. In May, GitLab reported similar cryptomining attacks on its platform.
Since CI / CD tools such as GitHub Actions and GitLab are intended to facilitate or automate important tasks, gatekeeping becomes a challenge: What was once intended as a function quickly becomes a function after abuse by cybercriminals . GitHub recently announced new features to prevent cryptomining attackers from abusing its Actions platform: “First-time contributor pull requests require manual approval from a repository worker with write privileges before running any Actions workflows. When an initial contributor opens a pull request, they’ll see a message that a maintainer needs to approve their Actions workflow before it runs,“ explains GitHub product manager Chris Patterson in a blog post.
Leading CI / CD solutions and DevOps platforms can follow the example of GitHub and integrate security checks to prevent misuse of their infrastructure.
There is nothing like adhering to standard best practices. This includes, for example, configuring your production containers correctly, hardening them against common attack vectors, and securing your pipeline configuration.
Especially simple misconfigurations are sometimes particularly difficult for human employees to recognize. The next question is whether your Docker-based environments have security vulnerabilities. It is therefore helpful to regularly perform a security audit of your containers and scan the images and files for security problems. It is advisable to invest in reliable cloud-native container security solutions that can automate much of these tasks. After all, countless vulnerabilities are reported every year – it is almost impossible to keep track (manually). More and more companies are also using Kubernetes frameworks and Docker containers to deploy applications. Container security solutions with built-in Web Application firewalls can detect and block suspicious network traffic at an early stage, which can prevent major compromises even if the attackers are able to penetrate a container.
Using tools to automatically detect quality issues, vulnerabilities, and other bugs in program code such as memory leaks or race conditions is an effective strategy to secure your CI/CD pipeline from the start. It is important to know that it is not always cyber attacks that need to be prevented. Seemingly harmless bugs can also have a big impact, as the failure of the global content delivery network Fastly has shown.
Solutions such as GitHub’s code scanner seamlessly integrate with existing code workflows and provide basic protections at no cost. Ultimately, the goal of any organization should be to help its developers do the best job possible while preventing bugs or vulnerabilities in the applications as much as possible.
This process must run with minimal friction between development and security teams. Real-time notifications that alert developers to possible errors during programming can not only secure the CI/CD workflow, but also save time for all parties involved.
In March 2021, attackers used a cryptomining botnet called “z0Miner” to mine the Monero cryptocurrency on vulnerable Jenkins and ElasticSearch servers. In order to initiate their criminal activities, the attackers used remote code execution in the servers connected to the Internet and thus tried to infect and take over the automation infrastructure.
As it became known last year, Jenkins servers could also be used by attackers to launch a DDoS attack. This resulted from a UDP amplification reflection DDoS vulnerability. Timely patching is essential to protect automation tools and pipelines against such serious vulnerabilities and to ensure the security of your CI/CD infrastructure.
Bringing in the latest updates and patches is good advice, but how do you know that the update has not been tampered with? The advice, which has enriched the mantra of security experts for decades, to simply relentlessly” update to the latest version ” was questioned at the latest after the supply chain attack on SolarWinds. In this supply chain attack, malicious updates of Orion software enabled the attackers to “leak”malicious code to over 18,000 customers.
In the case of the Codecov supply chain attack, a simple integrity check led to the discovery of the two-month-old compromise. A customer noticed a discrepancy between the checksum (hash) of the bash uploader hosted on the server and the legitimate checksum listed in Codecov’s GitHub repository.
So it’s generally a bad idea to blindly apply product updates. Instead, as part of a defense-in-depth approach, you should check the integrity of all updates, patches, and downloads to eliminate the risk of a supply chain attack.
This post is based on an article from our US sister publication CSO Online.
* Ax Sharma is a security and technology expert and writes for the US sister publication CSO Online.