Rate limiting design

Rate limiting design is the process of designing a rate limiting system for a network or service. This involves identifying the desired rate limit and determining how to implement it in a way that meets the needs of the system. 

This may involve choosing a specific algorithm for rate limiting, such as token bucket or leaky bucket, and setting appropriate parameters for that algorithm. It may also involve integrating the rate limiting system with other traffic management systems and monitoring tools to ensure that it is effective.

Designing a rate limiter is an important step in protecting against malicious or excessive traffic that can overwhelm a system or service. There are several considerations and techniques that can be used to effectively implement rate limiting.

Rate limit considerations

There are several considerations that are important to know when designing a rate limiter:

Determine the appropriate rate limit 

The rate limit should be set based on the capacity and performance characteristics of the service or system being protected. It should be high enough to allow legitimate traffic to pass through, but low enough to protect against malicious or excessive traffic.

Implement burst protection 

Burst protection allows a limited number of requests to be made at a higher rate, followed by a period of lower rate limiting. This can be useful for services that expect occasional bursts of traffic.

Use a distributed rate limiter 

A distributed rate limiter can be used to enforce rate limiting across multiple instances of a service or system. This can be particularly useful in a distributed or cloud-based environment.

Monitor and adjust the rate limit 

It is important to regularly monitor the rate limit to ensure that it is effective in protecting the service or system. The rate limit may need to be adjusted based on changes in traffic patterns or other factors.

Other factors 

In addition to the rate of requests, other factors such as the size of requests and the number of unique users or IP addresses may also need to be considered when designing a rate limiter.

Rate limiting algorithms and techniques

There are several algorithms and techniques that can be used for rate limiting:

• Fixed rate limiting: This approach sets a fixed limit on the number of requests that can be made to a service in a given time period. For example, a service may allow 100 requests per minute.
• Token bucket: This algorithm allows a certain number of requests to be made within a given time period, and then refills a “bucket” of tokens at a fixed rate. When the bucket is empty, further requests are blocked until more tokens become available.
• Leaky bucket: This algorithm is similar to the token bucket approach, but rather than blocking requests when the bucket is empty, it allows them to be made at a reduced rate.
• Fixed window: In this technique, the rate limiting is based on the number of requests made within a fixed time window, such as the past minute or hour.
• Sliding window: This technique uses a moving window to track the number of requests made over a given time period. The window can be adjusted to be larger or smaller depending on the desired level of rate limiting.
• Counter-based: This technique uses a counter to track the number of requests made within a given time period. When the counter reaches the maximum allowed number of requests, further requests are blocked until the next time period begins.
• Weighted token bucket: This technique allows different types of requests to have different weights, so that more resource-intensive requests can be rate limited differently than simpler requests.
• Fixed token ratio: This technique allows a certain number of requests to be made for every token that is consumed. The rate of requests is therefore directly proportional to the rate at which tokens are consumed.
• Fixed token count: This technique allows a fixed number of tokens to be consumed per request, regardless of the type or complexity of the request.

There are several best practices that can be followed when designing a rate limiter:

1. Identify the needs of the system: Before designing a rate limiter, it is important to understand the requirements of the system and the goals of the rate limiting. This will help to ensure that the rate limiter is designed in a way that meets the needs of the system.
2. Choose an appropriate algorithm: There are several different algorithms that can be used for rate limiting. It is important to choose an algorithm that is appropriate for the needs of the system and that can be implemented effectively.
3. Set appropriate limits: The rate limit should be set at a level that is appropriate for the needs of the system. This may involve setting different limits for different types of traffic, or for different times of day.
4. Monitor and adjust the rate limit as needed: The rate limit should be monitored to ensure that it is effective and that it is not causing problems for the system. If necessary, the rate limit can be adjusted to ensure that it is providing the desired level of protection.
5. Use other traffic management techniques in conjunction with rate limiting: Rate limiting should be used in conjunction with other traffic management techniques, such as traffic prioritization, to ensure that important traffic is able to get through even when the network is busy. This can help to ensure that the system remains available and responsive even under heavy load.

Gloo Gateway exposes Envoy’s rate-limit API, which allows users to provide their own implementation of an Envoy gRPC rate-limit service. Gloo Gateway provides an enhanced version of Lyft’s rate limit service that supports the full Envoy rate limit server API (with some additional enhancements, e.g. rule priority), as well as a simplified API built on top of this service. 

Gloo Gateway uses this rate-limit service to enforce rate-limits. The rate-limit service can work in tandem with the Gloo Gateway external auth service to define separate rate-limit policies for authorized & unauthorized users. The Gloo Gateway rate-limit service is enabled and configured by default, no configuration is needed to point Gloo Gateway toward the rate-limit service.

Get started with Gloo Mesh / Gloo Gateway today!