How does KDB decimal tolerance affect the `in` operator?

Viewed 75

Let's say I have a range:

a: (0.005*til 13) - 0.03
-0.03 -0.025 -0.02 -0.015 -0.01 -0.005 0 0.005 0.01 0.015 0.02 0.025 0.03

When I do -0.01 in a I get 0b, which I wasn't expecting. When I do -0.015 in a I get 1b.

Even more curiously, when I index directly:

-0.01 = a 4

I get 1b.

What's happening here?

2 Answers

The in keyword does not use comparison tolerance (wiggle room) when checking for list membership (see https://code.kx.com/q/basics/precision/#:~:text=distinct%20except%20group%20in%20inter%20union%20xgroup).

You can see that the same problem doesn't arise when a is defined as so:

q)a:-0.03 -0.025 -0.02 -0.015 -0.01 -0.005 0 0.005 0.01 0.015 0.02 0.025 0.03
q)-0.01 in a
1b

The list a generated in the way you provide gives an element that isn't precisely -0.01 and this is why in returns 0b:

q)a: (0.005*til 13) - 0.03
q)-0.01 - a 4
-1.734723e-018

This page details what does and doesn't use comparison tolerance: https://code.kx.com/q/basics/precision/#use

In general you should never need to check for "float equals" or "float in", it almost always ends in trouble! If you must, use some closeness measure.

Related