How to fine-tune the number of cluster in k-means clustering and incremental way of building the model using BigQuery ML

user_9811614 0
Viewed 701

Used K-means clustering Model for detecting anomaly using BigQuery ML.

Datasets information

date Date
trade_id INT
trade_name STRING
agent_id INT
agent_name String
total_item INT

Mapping - One trade has multiple agent based on date.
Model Trained with below information by sum(total_iteam)

trade_id 
trade_name
agent_id
agent_name 

Number of cluster: 4

Need to find the anomaly for each trades and agent based on date. Model is trained with given set of data and distance_from_closest_centroid is calculated. for each trade and agent based on date distance is called. Rightest distance is consider as a anomaly. Using this information

Questions 1. How to find the number of cluster need to use for model(eg: Elbow method used for selecting minimal cluster number selection).

Questions 2. How to build the model in case when trade data added on daily basis. Its is possible to build the incremental way of building the model on daily basis.

google-bigquery machine-learning
2 Answers

As the question was updated, I will sum up our discussion as an answer to further contribute to the community.

  1. Regarding your first question "How to find the number of cluster need to use for model(eg: Elbow method used for selecting minimal cluster number selection).".

According to the documentation, if you omit the num_clusters option, BigQuery ML will choose a reasonable default based on the total number of rows in the training data. However, if you want to select the most optimal number, you can perform hyperarameter tunning, which is the process of selecting one (or a set) of optimal hyperparameter for a learning algorithm, in your case K-means within BigQuery ML. In order to determine the ideal number of clusters you would run CREATE MODEL query for different values of num_clusters. Then , finding the error measure and select the point which it is at the minimum value. You can select the error measure in the training tab Evaluation, it will show the Davies–Bouldin index and Mean squared distance.

  1. Your second question was "How to build the model in case when trade data added on daily basis. Its is possible to build the incremental way of building the model on daily basis."

K-means is an unsupervised leaning algorithm. So you will train your model with your current data. Then store it in a data set. This model is already trained and can certainly be used with new data, using the ML.PREDICT. So it will use the model to predict which clusters the new data belong to.

As a bonus information, I would like to share this link for the documentation which explains how K-means in BigQuery ML can be used to detect data anomaly.

UPDATE:

Regarding your question about retraining the model:

Question: "I want to rebuild the model because new trade information has to be updated in my existing model. In this case is this possible to append the model with only two month of data or should we need to rebuild the entire model?"

Answer: You would have to retrain the whole model if new relevant data arrives. There is not the possibility to append the model with only two months of new data. Although, I must mention that you can and should use warm_start to retrain your already existing model, here.

user_125713870

As per @Alexandre Moraes omiting the num_clusters using K-means, BigQuery ML will choose a reasonable amount based in the number of rows in the training data. In addition, you can also use hyperparameter tuning to determine a optimal number of clusters. Thus, you would have to run the CREATE MODEL query for different values of num_clusters, find the error measure and pick the point which the error is minimum, link. –

user_98116140
Related