Feat: configurable rate limit on sources so Airweave does not take up all requests of users (enterprise feature)

[marc-rutzou]

Summary by cubic

Adds cross-pod source rate limiting with a Redis-backed sliding window to protect external API quotas. Limits are configurable per org and source via API and a new “Rate Limiting” settings tab.

• New Features

  • Redis-based distributed rate limiter enforced by a new AirweaveHttpClient wrapper (applies to sync and federated search). Exceeding limits returns HTTP 429 with Retry-After.
  • Source rate_limit_level added (“org” or “connection”); applied across sources (most set to org, Notion set to connection). Per-org limits stored in source_rate_limits.
  • API: GET/PUT/DELETE /source-rate-limits for listing and managing limits. Feature flag: SOURCE_RATE_LIMITING. Admin flag changes invalidate org cache.
  • Frontend: Organization Settings gains a “Rate Limiting” tab (behind feature flag) to view/edit source limits.

• Migration

  • Run Alembic migrations to add source.rate_limit_level and the source_rate_limits table (with created_by_email and modified_by_email tracking).
  • Enable the SOURCE_RATE_LIMITING feature flag, then configure limits in the new UI or via the API.

^{Written for commit 44e9cdd. Summary will update automatically on new commits.}

[cubic-dev-ai]

9 issues found across 29 files

Prompt for AI agents (all 9 issues)

Understand the root cause of the following 9 issues and fix them.


<file name="backend/airweave/schemas/source_rate_limit.py">

<violation number="1" location="backend/airweave/schemas/source_rate_limit.py:39">
Rule violated: **Check for Cursor Rules Drift**

The new per-organization/source rate limiting (introduced via `SourceRateLimit` schemas) replaces the static plan-based Redis limiter described in `.cursor/rules/api-layer.mdc` §6. Update the Cursor rule to cover the configurable source/org limits so AI guidance stays accurate.</violation>

<violation number="2" location="backend/airweave/schemas/source_rate_limit.py:45">
Redefining `limit` on `SourceRateLimit` without `Field(...)` overrides the inherited validator, so non-positive limits will now be accepted. Please restore the Field definition (or remove the override) to keep the gt&gt;0 constraint.</violation>

<violation number="3" location="backend/airweave/schemas/source_rate_limit.py:46">
Overriding `window_seconds` here removes the inherited Field metadata, dropping the gt&gt;0 constraint and default value. Keep the Field definition so validation still rejects invalid windows.</violation>
</file>

<file name="backend/airweave/api/v1/endpoints/admin.py">

<violation number="1" location="backend/airweave/api/v1/endpoints/admin.py:703">
context_cache.invalidate_organization does not exist on ContextCacheService, so this await will raise at runtime. Please remove or replace it with a valid cache invalidation call.</violation>
</file>

<file name="backend/airweave/platform/http_client/airweave_client.py">

<violation number="1" location="backend/airweave/platform/http_client/airweave_client.py:77">
self._logger is optional but debug() is called unconditionally; this raises AttributeError whenever the wrapper is built without a logger. Please guard the logging calls when the logger is absent.</violation>
</file>

<file name="backend/airweave/core/source_rate_limit_helpers.py">

<violation number="1" location="backend/airweave/core/source_rate_limit_helpers.py:57">
ApiContext requires request_id and logger; instantiating it here without those mandatory fields will raise a ValidationError whenever ctx is omitted, so the helper breaks before creating or updating limits.</violation>
</file>

<file name="backend/airweave/core/source_rate_limiter_service.py">

<violation number="1" location="backend/airweave/core/source_rate_limiter_service.py:300">
The rate-limit check and increment are not atomic: concurrent requests can both see the same pre-increment count (&lt; limit) and each perform this zadd, allowing the limit to be exceeded. Combine the prune/count/add/expire into a single atomic pipeline or Lua script.</violation>
</file>

<file name="backend/airweave/platform/db_sync.py">

<violation number="1" location="backend/airweave/platform/db_sync.py:459">
Rule violated: **Check for Cursor Rules Drift**

SourceCreate now requires the `_rate_limit_level` attribute to keep distributed rate limiting in sync, but `source-connector-implementation.mdc` still lacks any guidance to set `_rate_limit_level`. Cursor-generated connectors will miss this property and skip rate-limit enforcement, so please update the Cursor rule alongside this change.</violation>
</file>

<file name="backend/airweave/api/v1/endpoints/source_rate_limits.py">

<violation number="1" location="backend/airweave/api/v1/endpoints/source_rate_limits.py:20">
Rule violated: **Check for Fern Documentation Drift**

📝 Documentation drift: this new `/source-rate-limits` API adds configurable per-org/source limits and the Pipedream proxy override, while `fern/docs/pages/rate-limits.mdx` still documents fixed plan-based limits. Please update that manual doc (and related examples) to match the new behavior.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

No issues found across 1 file