Skip to content

bug: SQLite - Precision loss converting float results from complex expressions #11743

@discreteds

Description

@discreteds

What happened?

The SQLite backend loses floating-point precision when executing complex arithmetic expressions containing a division operator!
The generated SQL is correct and returns precise float values, but Ibis's .execute() method rounds them to integers, causing silent data corruption.

Simpler arithmetic expressions work fine.
Other backends convert results correctly.

Short reproduction:

import ibis
import pandas as df

df = pd.DataFrame({'a': [20], 'b': [3], 'c': [10]})
con = ibis.sqlite.connect()
t = con.create_table('test', df)

# Complex expression with division
expr = (t.a + t.b) * t.c - t.a / t.b
result = t.select(expr.name('result')).execute()

print(result['result'][0])
# Actual:   224.0  (WRONG - rounded!)
# Expected: 223.333...  (what SQL actually returns)

Longer reproduction with comparisons:

import ibis
import pandas as pd

# Test data
df = pd.DataFrame({
    'a': [10, 20, 30],
    'b': [2, 3, 4],
    'c': [5, 10, 15]
})

# Create Ibis table
con = ibis.sqlite.connect()
t = con.create_table('test', df)

# Complex expression: (a + b) * c - a / b
expr = (t.a + t.b) * t.c - t.a  / t.b

# Test 1: Ibis execute (WRONG)
result_ibis = t.select(expr.name('result')).execute()
print("Ibis execute():", result_ibis['result'].tolist())

# Test 2: Raw SQL on same connection (CORRECT)
sql = ibis.to_sql(t.select(expr.name('result')), dialect='sqlite')
cursor = con.con.cursor()
cursor.execute(sql)
result_raw = cursor.fetchall()
print("Ibis cursor.fetchall():", [row[0] for row in result_raw])

# Expected values (Python)
expected = [
    (10+2)*5 - 10/2,    # 55.0
    (20+3)*10 - 20/3,   # 223.333...
    (30+4)*15 - 30/4    # 502.5
]
print("Expected:", expected)

#Ibis execute(): [55.0, 224.0, 503.0]
#Ibis cursor.fetchall(): [55.0, 223.33333333333334, 502.5]
#Expected: [55.0, 223.33333333333334, 502.5]

More complex comparisons across backends:

test_precision_loss_all_backends.py

Impact

It is a weird edge case, but certainly real for many use cases.

Cause

I (and Claude) haven't been able to pinpoint the issue, but have narrowed it down to:

  1. SQL Generation is Perfect ✓
    - Ibis correctly generates CAST(a AS REAL) / b
    - The SQL is syntactically correct
  2. SQLite Returns Correct Values ✓
    - When executing the SQL directly: [(55.0,), (223.333...,), (502.5,)]
    - SQLite is working perfectly
  3. The Bug is in Ibis's Execution Pipeline ✗
    - The cursor passed to _fetch_from_cursor() contains: [(55,), (224,), (503,)] as integers
    - Somewhere between raw_sql() and _fetch_from_cursor(), float values become integers

What version of ibis are you using?

11.0.0

What backend(s) are you using, if any?

Error on sqlite
Working on polars, duckdb

Relevant log output

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIncorrect behavior inside of ibis

    Type

    No type

    Projects

    Status

    backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions