how to update vocabulary after training Word2vec model using deeplearning4java

Viewed 17

I used deeplearning4j library to do vectorization task using word2vec. I need to vectorize new word after training the model using a specific corpus. so how to add this new word and update the training to get a new weight vector for the new word? my code as folows:

Test word2Vec = new Test();
   word2Vec.train();
    
    //test the generated trained file
    Word2Vec word2VecModel = WordVectorSerializer.readWord2VecModel(new File(word2Vec.modelFilePath));
   
    
    double cosi=word2VecModel.similarity("httpdbpediaorgresourcethe_terminator", "httpdbpediaorgresourceterminator_salvation");
    System.out.println(cosi);

  }
  
  public  void train() throws IOException {
        SentenceIterator sentenceIterator = new FileSentenceIterator(new File(inputFilePath));

        TokenizerFactory tokenizerFactory = new DefaultTokenizerFactory();
        tokenizerFactory.setTokenPreProcessor(new CommonPreprocessor());

        Word2Vec vec = new Word2Vec.Builder()
                .layerSize(100)
                .windowSize(5)
                .epochs(5) //3-50 https://arxiv.org/pdf/1301.3781.pdf
                .elementsLearningAlgorithm(new SkipGram<VocabWord>())
                .iterate(sentenceIterator)
                .tokenizerFactory(tokenizerFactory)
                .build();
        vec.fit();
      
        WordVectorSerializer.writeWordVectors(vec, modelFilePath);
      
    }
0 Answers
Related