Oracle Table-valued Functions returns erroneous decimals in Data Factory

Viewed 135

I am working on a cloud datawarehouse using Azure Data Factory v2. Quite a few of my data sources are on-prem Oracle 12g databases. Extracting tables 1-1 is not a problem. However, from time to time I need to extract data generated by parametrized computations on the fly in my Copy Activities.

Since I cannot use PL/SQL stored procedures as sources in ADF, I instead use table-valued functions in the source database and query them in the copy activity.

In the majority of the cases, this works fine. However, when my table-valued function returns a decimal type column, ADF sometimes returns erroneous values. That is: executing the TVF on the source db and previeweing/copying through ADF yields different results.

I have done some experiments if the absolute value or the sign of the decimal number matters, but I cannot find any pattern in which decimals are returned correctly and which are not.

Here are a few examples of the erroneously mapped numbers:

Value in Oracle db Value in ADF
-658388.5681 188344991.6319
-205668.1648 58835420.6352
10255676.84 188213627.97348
  1. Has any of you experienced similar problems?
  2. Do you know if this is a bug in ADF (which is not integrating well to PL/SQL in the first place)?

First hypothesis

At first I thought the issue was related to NLS, casting or something similar. I tested this hypothesis by creating a table on the Oracle db side, persisted the output form the TVF there and then extracted from the table in ADF. Using this method, the decimals were returned correctly in ADF. Thus the hypothesis does not hold.

Second hypothesis

It might have to do with user accesses. However the linked service used in ADF uses the same db credentials as the ones used to log in to the database to execute the TVF there.

Observation

The error seems to happen more often when a lot of aggregate functions are involved in the tvf's logic

Minimum reproducible example

Oracle db:

CREATE OR REPLACE TYPE test_col AS OBJECT
(
dec_col NUMBER(20,5)
)
/

CREATE OR REPLACE TYPE test_tbl AS TABLE OF test_col;

create or replace function test_fct(param date) return test_tbl
AS
ret_tbl test_tbl;
begin
select
    test_col(
    <"some complex logic which return a decimal">
    )
 bulk collect into ret_tbl
 from <"some complex joins and group by's">;
 
 return ret_tbl;
end test_fct;

select dec_col from table(test_fct(sysdate));

ADF: Dataset:

{
    "name": "test_dataset",
    "properties": {
        "linkedServiceName": {
            "referenceName": "some_name",
            "type": "LinkedServiceReference"
        },
        "folder": {
            "name": "some_name"
        },
        "annotations": [],
        "type": "OracleTable",
        "structure": [
            {
                "name": "dec_col",
                "type": "Decimal"
            }
        ]
    }
}

Pipeline:

{
    "name": "pipeline1",
    "properties": {
        "activities": [
            {
                "name": "Copy data1",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "OracleSource",
                        "oracleReaderQuery": "select * from table(test_fct(sysdate))",
                        "partitionOption": "None",
                        "queryTimeout": "02:00:00"
                    },
                    "enableStaging": false
                },
                "inputs": [
                    {
                        "referenceName": "test_dataset",
                        "type": "DatasetReference"
                    }
                ]
            }
        ],
        "annotations": []
    }
}
0 Answers
Related