-
Notifications
You must be signed in to change notification settings - Fork 682
Open
Labels
bugIncorrect behavior inside of ibisIncorrect behavior inside of ibis
Description
What happened?
The SQLite backend loses floating-point precision when executing complex arithmetic expressions containing a division operator!
The generated SQL is correct and returns precise float values, but Ibis's .execute() method rounds them to integers, causing silent data corruption.
Simpler arithmetic expressions work fine.
Other backends convert results correctly.
Short reproduction:
import ibis
import pandas as df
df = pd.DataFrame({'a': [20], 'b': [3], 'c': [10]})
con = ibis.sqlite.connect()
t = con.create_table('test', df)
# Complex expression with division
expr = (t.a + t.b) * t.c - t.a / t.b
result = t.select(expr.name('result')).execute()
print(result['result'][0])
# Actual: 224.0 (WRONG - rounded!)
# Expected: 223.333... (what SQL actually returns)Longer reproduction with comparisons:
import ibis
import pandas as pd
# Test data
df = pd.DataFrame({
'a': [10, 20, 30],
'b': [2, 3, 4],
'c': [5, 10, 15]
})
# Create Ibis table
con = ibis.sqlite.connect()
t = con.create_table('test', df)
# Complex expression: (a + b) * c - a / b
expr = (t.a + t.b) * t.c - t.a / t.b
# Test 1: Ibis execute (WRONG)
result_ibis = t.select(expr.name('result')).execute()
print("Ibis execute():", result_ibis['result'].tolist())
# Test 2: Raw SQL on same connection (CORRECT)
sql = ibis.to_sql(t.select(expr.name('result')), dialect='sqlite')
cursor = con.con.cursor()
cursor.execute(sql)
result_raw = cursor.fetchall()
print("Ibis cursor.fetchall():", [row[0] for row in result_raw])
# Expected values (Python)
expected = [
(10+2)*5 - 10/2, # 55.0
(20+3)*10 - 20/3, # 223.333...
(30+4)*15 - 30/4 # 502.5
]
print("Expected:", expected)
#Ibis execute(): [55.0, 224.0, 503.0]
#Ibis cursor.fetchall(): [55.0, 223.33333333333334, 502.5]
#Expected: [55.0, 223.33333333333334, 502.5]More complex comparisons across backends:
test_precision_loss_all_backends.py
Impact
It is a weird edge case, but certainly real for many use cases.
Cause
I (and Claude) haven't been able to pinpoint the issue, but have narrowed it down to:
- SQL Generation is Perfect ✓
- Ibis correctly generates CAST(a AS REAL) / b
- The SQL is syntactically correct - SQLite Returns Correct Values ✓
- When executing the SQL directly: [(55.0,), (223.333...,), (502.5,)]
- SQLite is working perfectly - The Bug is in Ibis's Execution Pipeline ✗
- The cursor passed to _fetch_from_cursor() contains: [(55,), (224,), (503,)] as integers
- Somewhere between raw_sql() and _fetch_from_cursor(), float values become integers
What version of ibis are you using?
11.0.0
What backend(s) are you using, if any?
Error on sqlite
Working on polars, duckdb
Relevant log output
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
bugIncorrect behavior inside of ibisIncorrect behavior inside of ibis
Type
Projects
Status
backlog