Use an Oracle clob in a predicate created from a String > 4k

Viewed 1142

I'm trying to create a clob from a string of > 4000 chars (supplied in the file_data bind variable) to be used in a Oracle SELECT predicate below:

myQuery=
select *
from dcr_mols
WHERE flexmatch(ctab,:file_data,'MATCH=ALL')=1;

If I add TO_CLOB() round file_data it fails the infamous Oracle 4k limit for a varchar (it's fine for <4k strings). The error (in SQL Developer) is:

ORA-01460: unimplemented or unreasonable conversion requested
01460. 00000 -  "unimplemented or unreasonable conversion requested"

FYI The flexmatch function is used for searching molecules , and is a described here : http://help.accelrysonline.com/ulm/onelab/1.0/content/ulm_pdfs/direct/developers/direct_2016_developersguide.pdf

The function itself is a bit complicated but the essence is the 2nd parameter has to be a clob. So my question is how do I convert a Java String bind_variable of over 4000 chars to a clob in sql (or Java).

I tried the method below (which works when inserting clobs) in Java (Spring boot 2) using:

MapSqlParameterSource parameters = new MapSqlParameterSource();
parameters.addValue("file_data", fileDataStr,Types.CLOB);
jdbcNamedParameterTemplate.query(myQuery,parameters,…

This method should work but it fails with a converluted flexmatch error which FYI is:

SQL state [99999]; error code [29902]; ORA-29902: error in executing ODCIIndexStart() routine\nORA-20100: 
MDL-0203: Unable to read from CLOB (csfrm=1, csid=873): 
ORA-22922: nonexistent LOB value\nMDL-0021: Unable to copy LOB to string\nMDL-1051: Molstructure search query is not a valid molecule\nMDL-0976: 
Molecule index search initialization failed\nORA-06512: at \"C$MDLICHEM80.MDL_MXIXMDL\", line 329\nORA-06512: at \"C$MDLICHEM80.MDL_MXIXMDL\", line 309\n; nested exception is java.sql.SQLException: 
ORA-29902: error in executing ODCIIndexStart() routine\nORA-20100: MDL-0203: Unable to read from CLOB (csfrm=1, csid=873): 
ORA-22922: nonexistent LOB value\nMDL-0021: Unable to copy LOB to string\nMDL-1051: Molstructure search query is not a valid molecule\nMDL-0976: 
Molecule index search initialization failed\nORA-06512: at \"C$MDLICHEM80.MDL_MXIXMDL\", line 329\nORA-06512: at \"C$MDLICHEM80.MDL_MXIXMDL\", line 309\n"

Note i'm using SpringBoot 2 but I can't get any method using an OracleConnection (obtained from my Spring NamedParametersJdbcTemplate object) to work (even on clobs <4k) , so I suspect i've done something stupid. I've tried:

 @Autowired
 NamedParameterJdbcTemplate  jdbcNamedParameterTemplate;
OracleConnection conn =  this.jdbcNamedParameterTemplate.getJdbcTemplate().getDataSource().getConnection().unwrap(OracleConnection.class);
Clob myClob =  conn.createClob();
myClob.setString( 1, fileDataStr);
MapSqlParameterSource parameters = new MapSqlParameterSource();
parameters.addValue("file_data", myClob,Types.CLOB);

application.properties :

spring.datasource.url=jdbc:oracle:thin:@//${ORA_HOST}:${ORA_PORT}/${ORA_SID}
spring.datasource.username=${ORA_USER}
spring.datasource.password=${ORA_PASS}

Note it works fine if I go old school and use a non spring connection plus a PreparedStatement,which has a setClob() method:

OracleDataSource ods = new OracleDataSource();
String url ="jdbc:oracle:thin:@//" + ORA_HOST +":"+ORA_PORT +"/"+ORA_SID;
ods.setURL(url);
ods.setUser(user);
ods.setPassword(passwd);
Connection conn = ods.getConnection();
Clob myClob=conn.createClob();
PreparedStatement ps = conn.prepareStatement("select dcr_number from dcr_mols WHERE flexmatch(ctab,?,'MATCH=ALL')=1");
myClob.setString(1,myMol);
ps.setClob(1,myClob);
ResultSet rs =ps.executeQuery();

But I would prefer a Spring 2 solution in Java or Sql . Any help, suggestions appreciated.

5 Answers

Stream it. You can't just paste a huge value into a SQL statement.

You'll need to:

  • Insert an empty BLOB in the INSERT statement (using EMPTY_BLOB()? ...don't quite remember).
  • Get the output stream for the empty blob.
  • Then get an input stream from the file. Please don't load the entire file in memory.
  • Then transfer blocks from the input stream into the output stream using buffering. A 16 KB buffer should do.
  • Close both streams.

This is the standard way of dealing with massive data in Oracle. Plenty of examples out there.

Retrieving massive data (BLOB and CLOB types) works the same way. Just use InputStreams in that case.

Reading the documentation of the "BIOVIA Direct" API, there's an interesting example on page 27, excerpt shown below:

select ...
from ...
where flexmatch(
ctab,
(select ctab from nostruct_table),
'all'
)=1

It uses an already loaded CLOB. Therefore I guess a decent-enough solution would be to load the CLOB into a table of yours (or pre-load them all in advance), and then just use them.

Step #1 - Load your CLOB into a table:

create table mol_file (
  id number(12) primary key not null,
  content clob
);

insert into mol_file (id, content) values (:id, :content);

And use your Java code to perform the CLOB insertion (probably using streams) as shown in another answer (plenty of examples in the Internet). For example, insert your mol data content with ID = 123.

Step #2 - Run your query, using the already loaded mol file:

select *
from dcr_mols
WHERE flexmatch(
        ctab,
        (select content from mol_file where id = :id),
        'MATCH=ALL'
      ) = 1;

You can set the :id parameter to 123 to use the file you loaded before (or any other one).

You can call your old fashion function like this. create a Class to handle the ResultSet

 class MyPreparedStatementCallback implements PreparedStatementCallback {
    public Object doInPreparedStatement(PreparedStatement preparedStatement)
            throws SQLException, DataAccessException {
        ResultSet rs = preparedStatement.executeQuery();
        List result = new LinkedList();
        rs.close();
        return result;
    }
}

and call your query using the JdbcTemplate in your method

 jdbcTemplate.execute(new PreparedStatementCreator() {
        @Override
        public PreparedStatement createPreparedStatement(Connection connection)

                throws SQLException, DataAccessException {

            PreparedStatement ps = connection.prepareStatement("select dcr_number from dcr_mols WHERE flexmatch(ctab,?,'MATCH=ALL')=1");
            Clob myClob =  connection.createClob();
            myClob.setString( 1, fileDataStr);
            MapSqlParameterSource parameters = new MapSqlParameterSource();
            parameters.addValue("file_data", myClob, Types.CLOB);
            ps.setClob(1,myClob);
            return ps;

        };
    }, new MyPreparedStatementCallback());

I had to revert to using a PreparedStatement, but i've improved the normal implementation a bit by getting the connection from Spring and using apache commons BeanListHandler to map the ResultSet to an object List

import org.apache.commons.dbutils.ResultSetHandler;
import org.apache.commons.dbutils.handlers.BeanListHandler;

@Autowired
DataSource dataSource;

List<MyDao> myMethod(String fileData){
    String myQuery="select * from dcr_mols WHERE flexmatch(ctab,?,'MATCH=ALL')=1";

try {
    Connection conn = this.dataSource.getConnection()   // Get connection from spring

    Clob myClob =  conn.createClob();   // Open a dB clob 
    myClob.setString( 1, fileData);     // add data to clob
    PreparedStatement ps = conn.prepareStatement(myQuery);
    ps.setClob(1,myClob);              // Add a clob into the PreparedStatement
    ResultSet rs =ps.executeQuery();   // Execute the prepared statement

    //ResultSetHandler<List<MyDao>> handler = new BeanListHandler<MyDao>(MyDao.class);   // Define the ResultSet handler
    ResultSetHandler<List<MyDao>> handler = new BeanListHandler<MyDao>(MyDao.class, new BasicRowProcessor(new GenerousBeanProcessor()));  // This is better than the above handler , because GenerousBeanProcessor removes the requirement for the column names to exactly match the java variables

    List<MyDao> myDaoList = handler.handle(rs);   // Map ResultSet to List of MyDao objects
    }catch (Exception e) {
        e.printStackTrace();
    }

return myDaoList;
}

You can declare fileDataStr as CLOB using con which is the connection

java.sql.Clob fileDataStr = oracle.sql.CLOB.createTemporary
(con, false, oracle.sql.CLOB.DURATION_SESSION);

and then use it like below

 parameters.addValue("file_data", fileDataStr,Types.CLOB);

Also if you are using SID instead of Service name in your connection string try changing your property file like below

spring.datasource.url=jdbc:oracle:thin:@//${ORA_HOST}:${ORA_PORT}:${ORA_SID}
Related