Collation and datatype incompatibility on strings

user_5825963 0
Viewed 324

I am very confused on the behavior of the system on the case where both collation and datatype differences are involved.

As a minimal example, I am inputting the same Unicode value to the single column of two different tables. In one table the column is varchar and of a certain collation, and on the other it's nvarchar and of another collation. Code and results:

create table cn(code nvarchar(max) collate Latin1_General_CI_AS)
create table cv(code varchar(max) collate SQL_Latin1_General_CP1253_CI_AI)

insert cn select N'3VT18021δ'
insert cv select N'3VT18021δ'

select * from cn
select * from cv

--1. 
select * from cn inner join cv on cn.code=cv.code 
-- Cannot resolve the collation conflict between "SQL_Latin1_General_CP1253_CI_AI" and "Latin1_General_CI_AS" in the equal to operation.

--2. 
select * from cn inner join cv on cn.code=cv.code collate SQL_Latin1_General_CP1253_CI_AI   
-- returns one row

--3. 
select * from cn inner join cv on cn.code =cv.code collate Latin1_General_CI_AS 
-- returns 0 rows

--4. 
select * from cn inner join cv on cn.code collate SQL_Latin1_General_CP1253_CI_AI =cv.code   
-- returns one row

--5. 
select * from cn inner join cv on cn.code collate Latin1_General_CI_AS =cv.code 
-- returns one row

My notes:

Case 1: collation difference, I understand

Cases 2 and 5: return (correctly) one row. Why does collating a field to its own collation do any good?

Cases 3 and 4: Why converting one's collation to the other works one time, but not the other?

Of course, all these get further complication from the datatype difference.

sqldatatypes collation tsql sql-server
2 Answers

Cases 2 and 5: return (correctly) one row. Why does collating a field to its own collation do any good?

When you explicitly use COLLATE on a value in a clause both sides of the expression are explicitly converted to that collation, thus there is no conflict.

Cases 3 and 4: Why converting one's collation to the other works one time, but not the other?

One of your columns is a varchar, so when it's changed from one collation to the other, its value changes. This is, specifically, when you COLLATE the value in your table cv to the collation Latin1_General_CI_AS. As 'δ' isn't a character available in the collation for a varchar, it changes to a 'd' and '3VT18021d' does not equal N'3VT18021δ'. You can see this with the below:

SELECT code COLLATE Latin1_General_CI_AS
FROM cv;

You would need to explicitly convert the value to a nvarchar first:

select *
from cn
     inner join cv on cn.code = CONVERT(nvarchar(MAX),cv.code) collate Latin1_General_CI_AS;
--Returns one row now

Edit: To explain why Query 3 does not return data, and Query 5 does, this is because of the positioning of the COLLATEs and when the implicit conversion happens.

cn.code =cv.code collate Latin1_General_CI_AS --3
cn.code collate Latin1_General_CI_AS =cv.code --5

For Query 3, the COLLATE expression is on cv.code, which is the varchar. As a result the value has it's collation changed first and the character 'δ' is lost. Then it is implicitly converted to an nvarchar, due to data type precedence.

For Query 5, however, the COLLATE is on cn.code the nvarchar. As a result when the value's collation is changed no characters are lost. As cv.code doesn't have an explicit COLLATE, it is instead first converted to an nvarchar (due to data type precendence) and then collated; causing no loss of characters.

user_20299830

A collation is a part of the dataype. Internal representation of chars may differs if you use different collation and many constraints does not have the same behaviour when using different collations (PRIMARY KEY, UNIQUE, CHECK...).

The mixing of different collation in operators (=, LIKE, +) and in some function (CONCAT...) results systematically as an error until you impose a specific collation for this operation. So there is a COLLATE key word acting as an operator to disambiguate which collation can be used.

SQL Server distinguishes two kind of collations.

  1. Technical collations with a name beginning with SQL_
  2. semantical collation for functionnalities purpose, with a name that begin with a language name

Technical collations must be used only to recover imported data that have a specific encoding... As an example, you can have collations that are the strict equivalent of IBM EBCDIC, but it will be a stupid idea to keep this collations for SQL Server tables manipulations !

Semantical collations are widely use to facilitate application functionnalities... Do you want a CI or CS (case behaviour), AI or AS (diacritical behaviour), WS (wide behaviour like 2 = ²), etc...

Using this queries :

select CAST(code AS VARBINARY(max)) from cn;
select CAST(code AS VARBINARY(max)) from cv;

You will find that the last caracter does not have the same code. It is why the results is no rows when using the Latin1_General_CI_AS collation...

You will see that "B403" char of the NVARCHAR(max) dataype which is encoded on 2 bytes cannot be translated into the PAGE CODE CP1253 on a 1 byte per char...

In fact B4 byte in VARCHAR with SQL_Latin1_General_CP1253_CI_AI is "ä" not "δ"

In other words trying to put 1 byte in 2 bytes is easy... Just some zero to add. But, conversely, trying to put 2 bytes in one is possible only if the byte on the right is zeroed...

user_126598720
Related