Security Incidents
- News
- Last Updated: May 19, 2022
- Shyam Subramanyan
[Update: May 25, 2022 – GitHub integration is now re-enabled. You can connect to GitHub immediately or wait for the enhanced integration as described below. To re-establish your GitHub connection now, please follow these instructions.]
We know you are waiting for us to re-enable our integration with GitHub, and we’ve committed to you that we would only do so following a security review. We are happy to report that the review has now been completed.
One of the areas of focus was a review of the scope of tokens we request from GitHub and store on your behalf. …
- Engineering
- Last Updated: October 01, 2020
- Damien Mathieu
Incidents are inevitable. Any platform, large or small will have them. While resiliency work will definitely be an important factor in reducing the number of incidents, hoping to remove all of them (and therefore reach 100% uptime) is not an achievable goal.
We should, however, learn as much as we can from incidents, so we can avoid repeating them.
In this post, we will look at one of those incidents, #2105, see how it happened (spoiler: I messed up), and what we’re doing to avoid it from happening again (spoiler: I’m not fired).
Our Git server …
- Engineering
- Last Updated: April 30, 2024
- Wade
There’s obviously more to security than humans, technology, and vendors with all of their implementations and expertise. At Heroku we believe that security is a byproduct of excellence in engineering.
All too often, software is written solely with the happy path in mind, and security assurances of that software has its own dangerous assumptions. A mature security program should challenge assumptions of security controls, move to testing continuously, and prepare for the unexpectable.
This means asking hard questions about the bigger picture. Think bigger than the development lifecycle, backing away from the fixations of confirming effective corrections and remediations. This …
- Engineering
- Last Updated: June 27, 2018
- Camille Baldock
Over the past few weeks, Heroku proactively updated our entire Redis fleet with a version of Redis not vulnerable to CVE-2018-11218. This was an embargoed vulnerability, so we did this work without notifying our customers about the underlying cause. As always, our goal was to update all Heroku Redis instances well before the embargo expired.
As a Data Infrastructure Engineer at Heroku, I wanted to share how we manage large fleet operations such as this one. The most important aspect of our job is keeping customers safe from security vulnerabilities, while also minimizing disruption and downtime. Those two objectives …
- Engineering
- Last Updated: June 19, 2018
- Richard Schneeman
All previously released versions of Sprockets, the software that powers the Rails asset pipeline, contain a directory traversal vulnerability. This vulnerability has been assigned CVE-2018-3760.
How do I know if I'm affected?
Rails applications are vulnerable if they have this setting enabled in their application:
# config/environments/production.rb
config.assets.compile = true # setting to true makes your app vulnerable
Note: The default value of this setting that ships with Rails in production.rb is false. By default, Rails apps running in production mode are not vulnerable to this exploit.
To remediate this vulnerability, applications …
- Engineering
- Last Updated: June 03, 2024
- Etienne Stalmans
At Heroku we consistently monitor vulnerability feeds for new issues. Once a new vulnerability drops, we jump into action to triage and determine how our platform and customers may be affected. Part of this process involves evaluating possible attack scenarios not included in the original vulnerability report. We also spend time looking for “adjacent” and similar bugs in other products. The following Ruby vulnerability was identified during this process.
A vulnerability, CVE-2017-8817, was identified in libcurl. The FTP function contained an out of bounds read when processing wildcards. As soon as the vulnerability was made public, we …
- Engineering
- Last Updated: February 15, 2017
- Owen Jacobson
As part of our commitment to security and support, we periodically upgrade the stack image, so that we can install updated package versions, address security vulnerabilities, and add new packages to the stack. Recently we had an incident during which some applications running on the Cedar-14 stack image experienced higher than normal rates of segmentation faults and other “hard” crashes for about five hours. Our engineers tracked down the cause of the error to corrupted dyno filesystems caused by a failed stack upgrade. The sequence of events leading up to this failure, and the technical details of the failure, …
- Engineering
- Last Updated: January 11, 2017
- Tom Crayford
At Heroku, we're always working towards improving operational stability with the services we offer. As we recently launched Apache Kafka on Heroku, we've been increasingly focused on hardening Apache Kafka, as well as our automation around it. This particular improvement in stability concerns Kafka's compacted topics, which we haven't talked about before. Compacted topics are a powerful and important feature of Kafka, and as of 0.9, provide the capabilities supporting a number of important features.
The bug we had been seeing is that an internal thread that's used by Kafka to implement compacted topics (which we'll …
- News
- Last Updated: June 03, 2024
- Tom Crayford
At Heroku, we're always working towards increased operational stability with the services we offer. As we recently launched the beta of Apache Kafka on Heroku, we've been running a number of clusters on behalf of our beta customers.
Over the course of the beta, we have thoroughly exercised Kafka through a wide range of cases, which is an important part of bringing a fast-moving open-source project to market as a managed service. This breadth of exposure led us to the discovery of a memory leak in Kafka, having a bit of an adventure debugging it, and then contributing a …
- Engineering
- Last Updated: August 14, 2014
- Noah Zoschke
Retrospectives are a valuable tool for software engineering teams. Heroku consistently uses retrospectives to review operational incidents, root cause problems, and generate remediation tasks to improve our systems. Increasingly we use retrospectives for another purpose: to improve teamwork and interactions on projects. Here we intentionally avoid technical discussions and focus on the emotional and human aspects of work, with the goal of creating positive insights into how to improve as a team.
The most common times people conduct retrospectives are after some bad incident, or at the conclusion of a big project. These are worthwhile times, but …
Subscribe to the full-text RSS feed for Security Incidents.