Recovering transformation matrix from metric_learning LMNN algorithm

Viewed 163

I am using the LMNN module from scikit-learn metric_learning (http://contrib.scikit-learn.org/metric-learn/index.html), and I am attempting to recover the linear transformation matrix (L.T) from the learned Mahalanobis (M) matrix.

The reason I am trying to recover this linear transformation is that I am fitting my dataset using cloud compute, but am testing it on a local machine. This means I can not save or recover the LMNN model after fitting on cloud compute, but I can save the learned M matrix and use a decomposition to find the learned linear transformation. I can then apply that learned linear transformation to my test sets on a local machine.

The problem is that I can't seem to reconcile the results from the LMNN module's built in transformation with the learned linear transformation from the decomposed M matrix. Here's an example:

import numpy as np
from metric_learn import LMNN
from sklearn.datasets import load_iris
iris_data = load_iris()
X = iris_data['data']
Y = iris_data['target']

lmnn = LMNN(k=5, learn_rate=1e-6)
X_transformed = lmnn.fit_transform(X, Y)
M_matrix = lmnn.get_mahalanobis_matrix()
array([[ 2.47937397,  0.36313715, -0.41243858, -0.78715282],
       [ 0.36313715,  1.69818843, -0.90042673, -0.0740197 ],
       [-0.41243858, -0.90042673,  2.37024271,  2.18292864],
       [-0.78715282, -0.0740197 ,  2.18292864,  2.9531315 ]])

# cholesky decomp of M_matrix
eigvalues, eigcolvectors = np.linalg.eig(M_matrix)
eigvalues_diag = np.diag(eigvalues)
eigvalues_diag_sqrt = np.sqrt(eigvalues_diag)
L = eigcolvectors.dot(eigvalues_diag_sqrt.dot(np.linalg.inv(eigcolvectors)))
L_transpose = np.transpose(L)
L_transpose.dot(L) # check to confirm that matches M_matrix
array([[ 2.47937397,  0.36313715, -0.41243858, -0.78715282],
       [ 0.36313715,  1.69818843, -0.90042673, -0.0740197 ],
       [-0.41243858, -0.90042673,  2.37024271,  2.18292864],
       [-0.78715282, -0.0740197 ,  2.18292864,  2.9531315 ]])

# test fit_transform() vs. transform() using LMNN functions
lmnn.transform(X[0:4, :])
array([[8.2487    , 4.41337015, 0.14988465, 0.52629361],
       [7.87314906, 3.77220291, 0.36015873, 0.525688  ],
       [7.59410008, 4.03369392, 0.17339877, 0.51350962],
       [7.41676205, 3.82012155, 0.47312948, 0.68515535]])

X_transformed[0:4, :]
array([[8.2487    , 4.41337015, 0.14988465, 0.52629361],
       [7.87314906, 3.77220291, 0.36015873, 0.525688  ],
       [7.59410008, 4.03369392, 0.17339877, 0.51350962],
       [7.41676205, 3.82012155, 0.47312948, 0.68515535]])

# test manual transform of X[0:4, :]
X[0:4, :].dot(L_transpose)
array([[8.22608756, 4.45271327, 0.24690081, 0.51206068],
       [7.85071271, 3.81054846, 0.45442718, 0.51144826],
       [7.57310259, 4.06981377, 0.26240745, 0.50067674],
       [7.39356544, 3.85511015, 0.55776916, 0.67615584]])

As seen above, the first four rows of the original dataset X[0:4, :] when transformed by the LMNN module (using either fit_transform(X, Y) or transform(X[0:4, :]) give different results from the manual transformation.

class MetricTransformer(six.with_metaclass(ABCMeta)):

  @abstractmethod
  def transform(self, X):
    """Applies the metric transformation.
    Parameters
    ----------
    X : (n x d) matrix
      Data to transform.
    Returns
    -------
    transformed : (n x d) matrix
      Input data transformed to the metric space by :math:`XL^{\\top}`

class MahalanobisMixin(six.with_metaclass(ABCMeta, BaseMetricLearner,
                                          MetricTransformer)):
  r"""Mahalanobis metric learning algorithms.
  Algorithm that learns a Mahalanobis (pseudo) distance :math:`d_M(x, x')`,
  defined between two column vectors :math:`x` and :math:`x'` by: :math:`d_M(x,
  x') = \sqrt{(x-x')^T M (x-x')}`, where :math:`M` is a learned symmetric
  positive semi-definite (PSD) matrix. The metric between points can then be
  expressed as the euclidean distance between points embedded in a new space
  through a linear transformation. Indeed, the above matrix can be decomposed
  into the product of two transpose matrices (through SVD or Cholesky
  decomposition): :math:`d_M(x, x')^2 = (x-x')^T M (x-x') = (x-x')^T L^T L
  (x-x') = (L x - L x')^T (L x- L x')`
  • What am I missing here?

Thanks!

2 Answers

Per contributors from the team who built the module (http://contrib.scikit-learn.org/metric-learn/index.html), this discrepancy is due to floating point precision errors. The LMNN module first computes the linear transformation L.T then computes the M matrix by computing L.T.dot(L). So any attempt to recover the original transformation loses precision both during the computation of M and then the refactoring M.

metric-learn contributor here, @BeginnersMindTruly you're right, for LMNN we indeed learn the L matrix directly during training, from which we compute M at the end, so computing back the transformation L from M may lead to numerical differences.

As for your particular use case of accessing directly the learned matrix L, you should be able to do that using the components_ attribute of your metric learner, at the end of training.

Related