Polybase to connect local CSV file

user_3321517 0
Viewed 3133

I'm unable to access a local CSV file from SQL Server 2019 Polybase. This is a simple 3-columned text file. I have also created a local system DSN (from ODBC32 UI).

I got the sample code from here. However, the driver in the link (cdata) is not free. Any assistance in solving this issue will be greatly appreciated. 

create master key encryption by password = 'Polybase2CSV';

create database scoped credential csv_creds
with identity = 'username', secret = 'password';

create external data source csv_source
with ( 
  location = 'odbc://localhost',
  connection_options = 'DSN=CustomerDSN', -- this is the DSN name 
  -- PUSHDOWN = ON | OFF,
  credential = csv_creds
);


CREATE EXTERNAL TABLE Customer
(
    CUSTOMERID int,
    CUSTOMERNAME varchar(250),
    DEPARTMENT varchar(250)
) WITH (
    LOCATION='customer.txt',
    DATA_SOURCE=csv_source
);
polybase sql-server
2 Answers

This requires a few steps to make it work successfully. As prerequisites, you'll need to make sure SQL Server 2019 has been updated to CU4 (KB4548597) to fix a few known issues. For a free solution, you will need to install the 64-bit version Microsoft Access Database Engine 2016 Redistributable. This will install the 64-bit version of the ODBC drivers.

With these two things in place, you can now create the external data source. I recommend disabling PUSHDOWN. I've seen it cause some problems with this particular driver.

If you want to directly connect to the CSV file that contains a header row, you can create the external data source by simply specifying the Access Text Driver and the folder that will contain the files:

CREATE EXTERNAL DATA SOURCE MyODBC
WITH 
( 
    LOCATION = 'odbc://localhost',
    CONNECTION_OPTIONS = 'Driver=Microsoft Access Text Driver (*.txt, *.csv);Dbq=F:\data\files\',
    PUSHDOWN = OFF
);

To use the data source, you need to create an external table definition that reflects the file format. The LOCATION parameter will be the name of the file to load. You can wrap the file name and driver name in braces to avoid issues with special characters. It's important to make sure the column names you define for this table match the names in the header row. Because you're using CU4, if a data type doesn't match the driver's expectations, you'll get an error indicating which data types were expected.

CREATE EXTERNAL TABLE dbo.CsvData
(
    Name nvarchar(128),
    Count int,
    Description nvarchar(255)
)
WITH
(
    LOCATION='[filename.csv]',
    DATA_SOURCE = [MyODBC]
)

If you want to define the column names, data types, etc., in the ODBC Data Sources (64-bit ) UI, choose the Microsoft Access Text Driver. You can then select the folder, file types, and definition of the text file format. Make sure to use the 64-bit data sources. Once you're done defining the format details, you'll see a schema.ini file is created in the folder which contains those details.

For the external data source, you'll specify the name of the DSN:

CREATE EXTERNAL DATA SOURCE MyODBC
WITH 
( 
    LOCATION = 'odbc://localhost',
    CONNECTION_OPTIONS = 'DSN=LocalCSV',
    PUSHDOWN = OFF
);

The EXTERNAL TABLE is created the same way as before, with the column names and data types matching the definition you declared in the DSN.

user_109479930

To create a data source directly you'll need to buy that driver. That is option 1, but since it out the window. You have two more options. Import that data directly to SQL Server or if you really want to use PolyBase. Load that Data into Staging SQL Tables then create External Tables referencing that staging table.

My assumptions: CSV data isn't stale. structure/schema will remain constant. Create a staging table. Use 

Import-DbaCSV -Path "D:\CustomerTest\Customer.csv"`
              -SqlInstance ServerName`
              -Database DBName`
              -Table "Customer"

Then recycle your code to either connect to PolyBase or Use the data directly. 

CREATE DATABASE SCOPED CREDENTIAL csv_creds
WITH IDENTITY = 'username', SECRET = 'password';

CREATE EXTERNAL DATA SOURCE csv_source
    WITH ( LOCATION = 'sqlserver://SERVERNAME:PORTNUMBER',
    PUSHDOWN = ON,
    CREDENTIAL = csv_creds);

You can then periodically run the PS function to load data into the table as needed.

user_70622490
Related