Word2Vec + LSTM on API Sequence

I would at least split the trace lines in three:

• Module (make a dictionary and an embedding)
• Function (make a dictionary and an embedding)
• Parameters (make a dictionary and an embedding - see details later)

Since this is a very specific application, I believe it would be best to keep the embeddings trainable (the whole point of the embeddings is to create meaningful vectors, and the meanings depend a lot on the model that is going to use them. Question: how did you create the word2vec model? From what data does it learn?).

This model would have more inputs. All of them as integers from zero to max dictionary index. Consider using mask_zero=True and padding all files to maxFileLines.

moduleInput = Input(maxFileLines,) 
functionInput = Input(maxFileLines,)    

For the parameters, I'd probably make a subsequence as if the list of parameters were a sentence. (Again, mask_zero=True, and pad up to maxNumberOfParameters)

parametersInput = Input(maxFileLines, maxNumberOfParameters)

Function and module embeddings:

moduleEmb = Embedding(.....mask_zero=True,)(moduleInput)    
functionEmb = Embedding(.....mask_zero=True)(functionInput)

Now, for the parameters, I though of creating a sequence of sequences (maybe this is too much). For that, I first transfer the lines dimension to the batch dimension and work with only length = maxNumberOfParameters:

paramEmb = Lambda(lambda x: K.reshape(x,(-1,maxNumberOfParameters)))(parametersInput)
paramEmb = Embedding(....,mask_zero=True)(paramEmb)
paramEmb = Lambda(lambda x: K.reshape(x,(-1,maxFileLines,embeddingSize)))(paramEmb)

Now we concatenate all of them in the last dimension and we're ready to get into the LSTMs:

joinedEmbeddings = Concatenate()([moduleEmb,functoinEmb,paramEmb])
out = LSTM(...)(joinedEmbeddings)
out = ......

model = Model([moduleInput,functionInput,parametersInput], out)

How to prepare the inputs

With this model, you need three separate inputs. One for the module, one for the function and one for the parameters.

These inputs will contain only indices (no vectors). And they don't need a previous word2vec model. Embeddings are word2vec transformers.

So, get the file lines and split. First we split by commas, then we split the API calls by spaces:

import numpy as np

#read the file
loadedFile = open(fileName,'r')
allLines = [l.strip() for l in loadedFile.readlines()] 
loadedFile.close()

#split by commas
splitLines = []
for l in allLines[1:]: #use 1 here only if you have headers in the file
    splitLines.append (l.split(','))
splitLines = np.array(splitLines)

#get the split values and separate ids, targets and calls
ids = splitLines[:,0]
targets = splitLines[:,1]
calls = splitLines[:,2]

#split the calls by space, adding dummy parameters (spaces) to the max length
splitCalls = []
for c in calls:
    splitC = c.strip().split(' ')

    #pad the parameters (space for dummy params)
    for i in range(len(splitC),maxParams+2):
        splitC.append(' ') 

    splitCalls.append(splitC)

splitCalls = np.array(splitCalls)

modules = splitCalls[:,0]
functions = splitCalls[:,1]
parameters = splitCalls[:,2:] #notice the parameters have an extra dimension

Now lets make the indices:

modIndices, modCounts = np.unique(modules,return_counts=True)
funcIndices, funcCounts = np.unique(functions,return_counts=True)

#for de parameters, let's flatten the array first (because we have 2 dimensions)
flatParams = parameters.reshape((parameters.shape[0]*parameters.shape[1],))
paramIndices, paramCounts = np.unique(flatParams,return_counts=True)

These will create a list of unique words and get their counts. Here you can customize which words you're going to group in "another word" class. (Maybe based on the counts, if the count is too little, make it an "another word").

Let's then make the dictionaries:

def createDic(uniqueWords):
    dic = {}
    for i,word in enumerate(uniqueWords):
         dic[word] = i + 1 # +1 because we want to reserve the zeros for padding     
    return dic

Just take care with the parameters, because we used a dummy space there:

moduleDic = createDic(modIndices)
funcDic = createDic(funcIndices)
paramDic = createDic(paramIndices[1:]) #make sure the space got the first position here    
paramDic[' '] = 0

Well, now we just replace the original values:

moduleData = [moduleDic[word] for word in modules]
funcData = [funcDic[word] for word in functions]
paramData = [[paramDic[word] for word in paramLine] for paramLine in parameters]

Pad them:

for i in range(len(moduleData),maxFileLines):
    moduleData.append(0)
    funcData.append(0)
    paramData.append([0] * maxParams)

Do this for every file, and store in a list of files:

moduleTrainData = []  
functionTrainData = []
paramTrainData = []
for each file do the above and:
    moduleTrainData.append(moduleData)
    functionTrainData.append(funcData)
    paramTrainData.append(paramData)

moduleTrainData = np.asarray(moduleTrainData)
functionTrainData = np.asarray(functionTrainData)
paramTrainData = np.asarray(paramTrainData)

That's all for the inputs.

model.fit([moduleTrainData,functionTrainData,paramTrainData],outputLabels,...)