FT.AGGREGATE - RESP3 parser doesn't handle null correctly

[mhyllander]

Bug Report

When the result of an ftAggregate operation is parsed, the parser doesn't handle null values correctly. Nulls appear to be filtered out or skipped. This causes the parser to be out of sync with the data stream, so that values are parsed as keys, and keys as values.

Current Behavior

I have an index on JSON. Some fields in the JSON can be null. Here is example raw output:

127.0.0.1:6379>  ft.aggregate idx:rews '@partnerId:(9001) @customerId:(15)' GROUPBY 5 @siteCode @siteCity @siteCountry @siteGeo @siteAsn REDUCE COUNT 0 AS __count
1# attributes => (empty array)
2# format => STRING
3# results =>
   1) 1# extra_attributes =>
         1# "siteCode" => "Remote2"
         2# "siteCity" => "Huddinge"
         3# "siteCountry" => "SE"
         4# "siteGeo" => "59\\.24\\,17\\.98"
         5# "siteAsn" => "5501"
         6# "__count" => "1"
      2# values => (empty array)
   2) 1# extra_attributes =>
         1# "siteCode" => (nil)
         2# "siteCity" => "Stockholm"
         3# "siteCountry" => "SE"
         4# "siteGeo" => "59\\.33\\,18\\.07"
         5# "siteAsn" => (nil)
         6# "__count" => "1"
      2# values => (empty array)
   3) 1# extra_attributes =>
         1# "siteCode" => (nil)
         2# "siteCity" => "H\xc3\xa4gersten"
         3# "siteCountry" => "SE"
         4# "siteGeo" => "59\\.31\\,17\\.98"
         5# "siteAsn" => (nil)
         6# "__count" => "1"
      2# values => (empty array)
   4) 1# extra_attributes =>
         1# "siteCode" => "Remote1"
         2# "siteCity" => "Stockholm"
         3# "siteCountry" => "SE"
         4# "siteGeo" => "59\\.34\\,18\\.01"
         5# "siteAsn" => "5500"
         6# "__count" => "1"
      2# values => (empty array)
4# total_results => (integer) 4
5# warning => (empty array)

Note that some values are (nil). When I call ftAggregate using Lettuce, I get the following search results:

search result fields: HashMap(siteGeo -> 59\.24\,17\.98, siteCity -> Huddinge, siteAsn -> 5501, siteCountry -> SE, siteCode -> Remote2, __count -> 1)
search result fields: HashMap(Stockholm -> siteCountry, 59\.33\,18\.07 -> siteAsn, SE -> siteGeo, siteCode -> siteCity, __count -> 1)
search result fields: HashMap(Hägersten -> siteCountry, SE -> siteGeo, 59\.31\,17\.98 -> siteAsn, siteCode -> siteCity, __count -> 1)
search result fields: HashMap(siteGeo -> 59\.34\,18\.01, siteCity -> Stockholm, siteAsn -> 5500, siteCountry -> SE, siteCode -> Remote1, __count -> 1)

Take the second item as an example, where siteCode and siteAsn are null. This results in

siteCode -> siteCity
Stockholm -> siteCountry
SE -> siteGeo
59\.33\,18\.07 -> siteAsn
__count -> 1

The null value after siteCode causes the following keys and values to be swapped, until the null value after siteAsn swaps them back again so that the final __count field is correct.

Environment

• Lettuce version(s): 6.8.1.RELEASE
• LettuceMod version: 4.5.0
• Redis version: 8.2.3

Possible Solution

Additional context

We recently upgraded from LettuceMod 4.1.2 to 4.5.0, and with that to Lettuce 6.8.1.RELEASE. As a result of the upgrade, we began using the new JSON and Search APIs in Lettuce instead of LettuceMod.

The only reason we still use LettuceMod is because it supports ftInfo, which is missing in Lettuce. We use a LettuceMod client to call ftInfo; for everything else, we use a Lettuce client.

I found a similar issue for LettuceMod redis/lettucemod#59, and I'm pretty sure this was for RESP2. The LettuceMod clients don't seem to work well with RESP3, but we used RESP2 successfully. Now that we switched to a Lettuce client, we also switched to RESP3, and it looks like it suffers from the same problem that was fixed for RESP2 in LettuceMod.