Get word for (position, word) in dataframe column if position is in another column

user_5945369 0
Viewed 39

I have this form of dataframe :

ind_position                     preprocessed_sent                                       original_sent
[1]                              [In, 4, dogs, injected, intravenously, i.v, with...]    In 4 dogs injected itravenously i.v. with...
[2, 3, 6, 7]                     [The, kinetics, and, cellular, characteristics...]      The kinetics and cellular characteristics...

I want to get only the words in preprocessed_sent whose position is in the ind_position. I'm doing it already with df.iterrows, but I want to avoid iterating through every dataframe row and instead do the same with one line of code. Is that possible? Here's how I'm doing it so far with df.iterrows:

inds_list = []
nlp = spacy.load('en_core_web_sm')
cols = ["ind_position"]
for _, row in df.iterrows():   
    doc = nlp(row["original_sent"])
    tokens = [token.text for token in doc if not token.pos_ == "PUNCT"]
    marked_ind = [word for position, word in enumerate(tokens) for col in cols if col  == "ind_position" if
                position in row[col]]
    inds_list.append(marked_ind)
df["inds_list"] = inds_list

Here's where I came so far with the optimization :

docs = list(nlp.pipe(df['original_sent']))

def tokenize_pipe(doc):
    token_list = [token.text for token in doc if not token.pos_ == "PUNCT"]
    return token_list

def preprocess_pipe(texts):
    preproc_pipe = []
    for doc in nlp.pipe(texts):
        preproc_pipe.append(tokenize_pipe(doc))
    return preproc_pipe

df['preprocessed_sent'] = preprocess_pipe(df['original_sent'])

marked_ind = [word for position, word in df['preprocessed_sent'] if position==df["ind_position"]]

print(marked_ind)



 >>>> marked_ind = [word for position, word in df['preprocessed_sent'] if    position == df["indication_pos"]]
 >>>> ValueError: too many values to unpack (expected 2)

How can I get marked_ind with one line of code and without this error?

pandas nlp python
1 Answers

If I understand your question correctly, you want to index the list in "preprocessed_sent" using the indexes in the list at "ind_position". You could apply this function on each row:

df["selection"] = df.apply(lambda row: [row.preprocessed_sent[i] for i in row.ind_position], axis=1]

Then, "selection" will be the list of elements from "preprocessed_sent" indexed by "ind_position".

user_32810970
Related