How do I select one or more random rows from a table using SQLAlchemy?
How do I select one or more random rows from a table using SQLAlchemy?
This is very much a database-specific issue.
I know that PostgreSQL, SQLite, MySQL, and Oracle have the ability to order by a random function, so you can use this in SQLAlchemy:
from sqlalchemy.sql.expression import func, select
select.order_by(func.random()) # for PostgreSQL, SQLite
select.order_by(func.rand()) # for MySQL
select.order_by('dbms_random.value') # For Oracle
Next, you need to limit the query by the number of records you need (for example using .limit()).
Bear in mind that at least in PostgreSQL, selecting random record has severe perfomance issues; here is good article about it.
If you are using the orm and the table is not big (or you have its amount of rows cached) and you want it to be database independent the really simple approach is.
import random
rand = random.randrange(0, session.query(Table).count())
row = session.query(Table)[rand]
This is cheating slightly but thats why you use an orm.
Some SQL DBMS, namely Microsoft SQL Server, DB2, and PostgreSQL have implemented the SQL:2003 TABLESAMPLE clause. Support was added to SQLAlchemy in version 1.1. It allows returning a sample of a table using different sampling methods – the standard requires SYSTEM and BERNOULLI, which return a desired approximate percentage of a table.
In SQLAlchemy FromClause.tablesample() and tablesample() are used to produce a TableSample construct:
# Approx. 1%, using SYSTEM method
sample1 = mytable.tablesample(1)
# Approx. 1%, using BERNOULLI method
sample2 = mytable.tablesample(func.bernoulli(1))
There's a slight gotcha when used with mapped classes: the produced TableSample object must be aliased in order to be used to query model objects:
sample = aliased(MyModel, tablesample(MyModel, 1))
res = session.query(sample).all()
Since many of the answers contain performance benchmarks, I'll include some simple tests here as well. Using a simple table in PostgreSQL with about a million rows and a single integer column, select (approx.) 1% sample:
In [24]: %%timeit
...: foo.select().\
...: order_by(func.random()).\
...: limit(select([func.round(func.count() * 0.01)]).
...: select_from(foo).
...: as_scalar()).\
...: execute().\
...: fetchall()
...:
307 ms ± 5.72 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [25]: %timeit foo.tablesample(1).select().execute().fetchall()
6.36 ms ± 188 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [26]: %timeit foo.tablesample(func.bernoulli(1)).select().execute().fetchall()
19.8 ms ± 381 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Before rushing to use SYSTEM sampling method one should know that it samples pages, not individual tuples, so it might not be suitable for small tables, for example, and may not produce as random results, if the table is clustered.
If using a dialect that does not allow passing the sample percentage / number of rows and seed as parameters, and a driver that does not inline values, then either pass the values as literal SQL text if they are static, or inline them using a custom SQLA compiler extension:
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql import TableSample
@compiles(TableSample)
def visit_tablesample(tablesample, self, asfrom=False, **kw):
""" Compile `TableSample` with values inlined.
"""
kw_literal_binds = {**kw, "literal_binds": True}
text = "%s TABLESAMPLE %s" % (
self.visit_alias(tablesample, asfrom=True, **kw),
tablesample._get_method()._compiler_dispatch(self, **kw_literal_binds),
)
if tablesample.seed is not None:
text += " REPEATABLE (%s)" % (
tablesample.seed._compiler_dispatch(self, **kw_literal_binds)
)
return text
from sqlalchemy import table, literal, text
# Static percentage
print(table("tbl").tablesample(text("5 PERCENT")))
# Compiler inlined values
print(table("tbl").tablesample(5, seed=literal(42)))
This is my function to select random row(s) of a table:
from sqlalchemy.sql.expression import func
def random_find_rows(sample_num):
if not sample_num:
return []
session = DBSession()
return session.query(Table).order_by(func.random()).limit(sample_num).all()
Use this simplest method this example on choosing a random question from the database:-
#first import the random module
import random
#then choose what ever Model you want inside random.choise() method
get_questions = random.choice(Question.query.all())
Theres a couple of ways through SQL, depending on which data base is being used.
(I think SQLAlchemy can use all these anyways)
mysql:
SELECT colum FROM table
ORDER BY RAND()
LIMIT 1
PostgreSQL:
SELECT column FROM table
ORDER BY RANDOM()
LIMIT 1
MSSQL:
SELECT TOP 1 column FROM table
ORDER BY NEWID()
IBM DB2:
SELECT column, RAND() as IDX
FROM table
ORDER BY IDX FETCH FIRST 1 ROWS ONLY
Oracle:
SELECT column FROM
(SELECT column FROM table
ORDER BY dbms_random.value)
WHERE rownum = 1
However I don't know of any standard way