Search overlay panel for performing site-wide searches

Boost Performance & Scale with Postgres Advanced. Join Pilot Now!

kafka

Building an Event-Driven Architecture with Managed Data Services

Modern applications have an unceasing buzz of user activity and data flows. Users send a flurry of one-click reactions to social media posts. Wearable tech and other IoT sensors work nonstop to transmit event data from their environments. Meanwhile, customers on e-commerce sites perform shopping cart actions or product searches which can bring immediate impact to operations. Today’s software organizations need the ability to process and respond to this rich stream of real-time data.

That’s why they adopt an event-driven architecture (EDA) for their applications.

Long gone are the days of monolithic applications with components tightly coupled…

Streaming Data Connectors Beta: Capture Postgres Changes in Apache Kafka on Heroku

Today we are announcing a beta release of our new streaming data connector between Heroku Postgres and Apache Kafka on Heroku . Heroku runs millions of Postgres services and tens of thousands of Apache Kafka services, and we increasingly see developers choosing to start with Apache Kafka as the foundation of their data architecture. But for those who are Postgres-first, it is challenging to adopt without a full app rewrite. Developers want a seamless integration between the two services, and we are delivering it today, at no additional charge, for Heroku Private Spaces and Shield Spaces customers.

Moving beyond Postgres and…

Why Frequent Maintenances Are Essential for Secure Heroku Data Services

There are many reasons to choose Heroku Data services, but keeping the services you use secure and up-to-date rank near the top. This foundation of trust is the most important commitment we make to our customers, and frequent and timely maintenances are one way we deliver on this promise.

We do everything we can to minimize downtime, which is typically between 10 – 60 seconds per maintenance. There are ways for you to minimize disruption too (see the tips and tricks below). The rest of the post explains how we think about Heroku Data maintenances, how we perform…

How Heroku Operates its Multi-Tenant Apache Kafka Services

This blog post is adapted from a talk given by Ali Hamidi at  Data Council SF ’19  titled “ Operating Multi-Tenant Kafka Services for Developers on Heroku .”

https://www.youtube.com/embed/-AtHKoTNR1k

Thousands of developers use  Heroku’s Apache Kafka service  to process millions of transactions on our platform—and many of them do so through our multi-tenant Kafka service. Operating Kafka clusters at this scale requires careful planning to ensure capacity and uptime across a wide range of customer use cases. With significant automation and test suites, we’re able to do this without a massive operations team.

In this post,…

Kafka Streams on Heroku

Designing scalable, fault tolerant, and maintainable stream processing systems is not trivial. The Kafka Streams Java library paired with an Apache Kafka cluster simplifies the amount and complexity of the code you have to write for your stream processing system.

Unlike other stream processing systems, Kafka Streams frees you from having to worry about building and maintaining separate infrastructural dependencies alongside your Kafka clusters. However, you still need to worry about provisioning, orchestrating, and monitoring infrastructure for your Kafka Streams applications.

Heroku makes it easy for you to deploy, run, and scale your Kafka Streams applications by…

Kafka Everywhere: New Plans and Pricing for Apache Kafka on Heroku

Event-driven architectures are on the rise, in response to fast-moving data and constellations of inter-connected systems. In order to support this trend, last year we released Apache Kafka on Heroku – a gracefully integrated, fully managed, and carefully optimized element of Heroku's platform that is the culmination of years of experience of running many hundreds of Kafka clusters in production and contributing code to the Kafka ecosystem.

Today, we are excited to announce additional plans and pricing in our Kafka offering in order to make Apache Kafka more accessible, and to better support development, testing, and low volume…

Apache Kafka on Heroku is Now Generally Available

Many of the compelling and engaging application experiences we enjoy every day are powered by event-based systems; requesting a ride and watching its progress, communicating with a friend or large group in real time, or connecting our increasingly intelligent devices to our phones and each other. Behind the scenes, similar architectures let developers connect separate services into single systems, or process huge data streams to generate real-time insights. Together, these event-driven architectures and systems are quickly becoming a powerful complement to the relational database and app server models that have been at the core of Internet applications for…

Powering the Heroku API With Streams & Apache Kafka

We recently launched Apache Kafka on Heroku into beta. Just like we do with Heroku Postgres, our internal engineering teams have been using our Kafka service to power a number of our internal systems.

The Big Idea

The Heroku platform comprises a large number of independent services. Traditionally we’ve used HTTP calls to communicate between these services. While this approach is simple to implement and easy to reason about, it has a number of drawbacks. Synchronous calls mean that the top-level request time will be gated by the slowest backend component. Also, internal API calls create tight point-to-point…

Apache Kafka 0.10 Evaluating Performance in Distributed Systems

At Heroku, we’re always striving to provide the best operational experience with the services we offer. As we’ve recently launched Heroku Kafka, we were excited to help out with testing of the latest release of Apache Kafka, version 0.10, which landed earlier this week. While testing Kafka 0.10, we uncovered what seemed like a 33% throughput drop relative to the prior release. As others have noted , “it’s slow” is the hardest problem you’ll ever debug, and debugging this turned out to be very tricky indeed. We had to dig deep into Kafka’s configuration and operation to uncover what was…

Subscribe to the full-text RSS feed for kafka.