fix(search): adds query token limit validation

[orhanrauf]

Implements validation to limit the number of tokens in search queries.

This prevents excessively long queries, improving performance and preventing potential issues with the underlying search infrastructure. The limit is set to 2048 tokens. Adds token validation to both the new SearchRequest and the legacy LegacySearchRequest schemas.

---

Summary by cubic

Add a 2048-token limit to search queries to prevent overly long requests and improve performance and stability. Applies to both SearchRequest and LegacySearchRequest.

• Bug Fixes
  • Enforce token counting using tiktoken (cl100k_base).
  • Reject empty queries in SearchRequest.

^{Written for commit a5deaf3. Summary will update automatically on new commits.}

[cubic-dev-ai]

1 issue found across 2 files

Prompt for AI agents (all 1 issues)

Understand the root cause of the following 1 issues and fix them.


<file name="backend/airweave/schemas/search_legacy.py">

<violation number="1" location="backend/airweave/schemas/search_legacy.py:47">
Rule violated: **Check for Cursor Rules Drift**

The new 2048-token validator for `LegacySearchRequest.query` is not reflected in `.cursor/rules/search-module.mdc`. Please update the Cursor rule (LegacySearchRequest section) to document the token limit/tokenization so Cursor stays in sync with the code.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}