Search overlay panel for performing site-wide searches

Boost Performance & Scale with Postgres Advanced. Join Pilot Now!

Security Incidents

Plans to Re-enable the GitHub Integration

[Update: May 25, 2022 – GitHub integration is now re-enabled. You can connect to GitHub immediately or wait for the enhanced integration as described below. To re-establish your GitHub connection now, please follow these instructions.]

We know you are waiting for us to re-enable our integration with GitHub, and we’ve committed to you that we would only do so following a security review. We are happy to report that the review has now …

How I Broke `git push heroku main`

Incidents are inevitable. Any platform, large or small will have them. While resiliency work will definitely be an important factor in reducing the number of incidents, hoping to remove all of them (and therefore reach 100% uptime) is not an achievable goal.

We should, however, learn as much as we can from incidents, so we can avoid repeating them.

In this post, we will look at one of those incidents, #2105, see how it happened (spoiler: I messed up), and what we’re doing to avoid it from happening again (spoiler: I’m not fired).

Bug Bounties and Black Swans: How Heroku Expects the Unexpectable

There’s obviously more to security than humans, technology, and vendors with all of their implementations and expertise. At Heroku we believe that security is a byproduct of excellence in engineering.

All too often, software is written solely with the happy path in mind, and security assurances of that software has its own dangerous assumptions. A mature security program should challenge assumptions of security controls, move to testing continuously, and prepare for the unexpectable.

This means …

Rolling the Heroku Redis Fleet

Over the past few weeks, Heroku proactively updated our entire Redis fleet with a version of Redis not vulnerable to CVE-2018-11218. This was an embargoed vulnerability, so we did this work without notifying our customers about the underlying cause. As always, our goal was to update all Heroku Redis instances well before the embargo expired.

As a Data Infrastructure Engineer at Heroku, I wanted to share how we manage large fleet operations such as …

Rails Asset Pipeline Directory Traversal Vulnerability (CVE-2018-3760)

All previously released versions of Sprockets, the software that powers the Rails asset pipeline, contain a directory traversal vulnerability. This vulnerability has been assigned CVE-2018-3760.

How do I know if I'm affected?

Rails applications are vulnerable if they have this setting enabled in their application:

# config/environments/production.rb config.assets.compile = true # setting to true makes your app vulnerable

Note: The default value of this setting that ships with Rails in production.rb is …

Ruby CVE-2017-17405: Vulnerability in Ruby’s FTP Implementation

At Heroku we consistently monitor vulnerability feeds for new issues. Once a new vulnerability drops, we jump into action to triage and determine how our platform and customers may be affected. Part of this process involves evaluating possible attack scenarios not included in the original vulnerability report. We also spend time looking for “adjacent” and similar bugs in other products. The following Ruby vulnerability was identified during this process.

Vulnerability Triage

A vulnerability, CVE-2017-8817, …

How We Found and Fixed a Filesystem Corruption Bug

As part of our commitment to security and support, we periodically upgrade the stack image, so that we can install updated package versions, address security vulnerabilities, and add new packages to the stack. Recently we had an incident during which some applications running on the Cedar-14 stack image experienced higher than normal rates of segmentation faults and other “hard” crashes for about five hours. Our engineers tracked down the cause of the error to …

Pulling the Thread on Kafka’s Compacted Topics

At Heroku, we're always working towards improving operational stability with the services we offer. As we recently launched Apache Kafka on Heroku, we've been increasingly focused on hardening Apache Kafka, as well as our automation around it. This particular improvement in stability concerns Kafka's compacted topics, which we haven't talked about before. Compacted topics are a powerful and important feature of Kafka, and as of 0.9, provide the capabilities supporting a number of important …

Dawn of the Dead Ends: Fixing a Memory Leak in Apache Kafka

At Heroku, we're always working towards increased operational stability with the services we offer. As we recently launched the beta of Apache Kafka on Heroku, we've been running a number of clusters on behalf of our beta customers.

Over the course of the beta, we have thoroughly exercised Kafka through a wide range of cases, which is an important part of bringing a fast-moving open-source project to market as a managed service. This breadth …

Retrospectives

Retrospectives are a valuable tool for software engineering teams. Heroku consistently uses retrospectives to review operational incidents, root cause problems, and generate remediation tasks to improve our systems. Increasingly we use retrospectives for another purpose: to improve teamwork and interactions on projects. Here we intentionally avoid technical discussions and focus on the emotional and human aspects of work, with the goal of creating positive insights into how to improve as a team.

Subscribe to the full-text RSS feed for Security Incidents.