Explanation of code that constructs correlation matrix

Viewed 111

I am referring to this answer.

The code to construct a correlation matrix given a table of columns is

u cor/:\:u:flip t where t is a table.

Reading right to left, I understand up till u:flip t. May I please ask for an explanation on what the rest of the code does?

Thanks

2 Answers

If you substitute for a something which gives more visual output, such as join with two vectors, it should be easier to see what the derived function cor/:\: is doing

q)"123","abc"       // simple join
"123abc"

q)"123",/:"abc"     // join left arg to each item of right arg
"123a"
"123b"
"123c"

q)"123",/:\:"abc"   // join each item of left arg to each item of right
"1a" "1b" "1c"
"2a" "2b" "2c"
"3a" "3b" "3c"

Back to a simple example of cor

q)show t:([]a:3?1.0;b:3?1.0;c:3?1.0)
a         b          c
-------------------------------
0.7935513 0.6377554  0.3573039
0.2037285 0.03845637 0.02547383
0.7757617 0.8972357  0.688089

q)u cor/:\:u:flip t
 | a         b         c
-| -----------------------------
a| 1         0.9474878 0.8529413
b| 0.9474878 1         0.975085
c| 0.8529413 0.975085  1

q)show data:value flip t;           // extract the data for clarity
0.7935513 0.2037285  0.7757617
0.6377554 0.03845637 0.8972357
0.3573039 0.02547383 0.688089

q)cor[data 0;]each data             // first row cor each row
1 0.9474878 0.8529413
q)cor[data 1;]each data             // second row cor each row
0.9474878 1 0.975085
q)cor[data 2;]each data             // last row cor each row
0.8529413 0.975085 1

q){cor[x]each data}each data        // all at once
1         0.9474878 0.8529413
0.9474878 1         0.975085
0.8529413 0.975085  1

q)data cor/:\:data                  // derived function much nicer
1         0.9474878 0.8529413
0.9474878 1         0.975085
0.8529413 0.975085  1

If you are looking into correlation matrices then it might be a good idea to have a look into what they are, this might give some context to the inputs/outputs/code.

https://www.displayr.com/what-is-a-correlation-matrix/?msclkid=f68768aeab8e11ecbca30d34e2ba880f

In this case, we are finding the correlation between some matrix/table u:flip t and itself.

The rest of the query is comprised of function cor and two kdb+ iterators each right /: and each left \:.
https://code.kx.com/q/ref/cor/?msclkid=748e645bab8d11ecab715a544f547398 https://code.kx.com/q/wp/iterators/?msclkid=d2172906ab8c11ecac2b46902bbe505d

Each right will apply each item of the right-hand argument to each of the left-hand argument

q)1 ,/: 10 20 30
1 10
1 20
1 30

While each left will apply each item of the left-hand argument to each of the right-hand arguments

q)1 2 3 ,\: 10
1 10
2 10
3 10

If we use both simultaneously as illustrated below where we join , each element of the left-hand list \: with each element of the right-hand list /:

q)1 2 3,/:\:10 20 30
1 10  1 20  1 30
2 10  2 20  2 30
3 10  3 20  3 30

Thenu cor/:\:u:flip t can be understood to be taking each element of u and finding its correlation with every element within u, achieved through the use of cor/:\:.

Related