Only print the content of the first file within the folder even though I want to print all the files

Viewed 50

I have this function that returns all the files in the folder after deleting the stop words from them, but the problem is that when I print the result of this function, only the content of the first file is printed, and I want to print all the files after deleting the stop words from them.

How can I solve the problem?

def remove_stop_word_from_files():
    stop_words_list = get_stop_words()
    dir_path = 'C:/Users/Super/Desktop/IR/homework/Lab4/corpus/corpus/'
    save_dir = "C:/Users/Super/Desktop/IR/homework/Files_Without_SW/"

    for document in os.listdir(dir_path):
        with open(dir_path + document, "r") as reader:
            save_file = open(save_dir + document, 'w')
            text = reader.read()
            text_tokens = word_tokenize(text)
            tokens_without_sw = [word.replace(',', '').replace('.', '') for word in 
                     text_tokens if (word not in stop_words_list)]
            save_file.writelines(["%s " % item.replace(',', '').replace('.', '') for item in 
            tokens_without_sw])
            
    return tokens_without_sw
    
print(remove_stop_word_from_files())
3 Answers

You return statement is within the loop. You need to reduce its indent by one level. This function returns after doing its first iteration.

In addition, you are clobbering it after each iteration, rather than appending a running tally.

def remove_stop_word_from_files():
    stop_words_list = get_stop_words()
    dir_path = 'C:/Users/Super/Desktop/IR/homework/Lab4/corpus/corpus/'
    save_dir = "C:/Users/Super/Desktop/IR/homework/Files_Without_SW/"

    all_tokens_without_sw = []
    for document in os.listdir(dir_path):
        with open(dir_path + document, "r") as reader, \
             open(save_dir + document, 'w') as save_file::
            text = reader.read()
            text_tokens = word_tokenize(text)
            tokens_without_sw =[word.replace(',', '').replace('.', '')
                                for word in text_tokens
                                if (word not in stop_words_list)])
            save_file.writelines(["%s " % item.replace(',', '').replace('.', '')
                                 for item in tokens_without_sw])
            all_tokens_without_sw.extend(tokens_without_sw)
            
    return all_tokens_without_sw
    
print(remove_stop_word_from_files())

The problem is you used return tokens_without_sw inside a loop. When you use a return statement inside the loop, it will break the loop. Instead try to use yield to continue return the result as for that the result is going to be a list. So, your code should be like:

def remove_stop_word_from_files():
    stop_words_list = get_stop_words()
    dir_path = 'C:/Users/Super/Desktop/IR/homework/Lab4/corpus/corpus/'
    save_dir = "C:/Users/Super/Desktop/IR/homework/Files_Without_SW/"

    for document in os.listdir(dir_path):
        with open(dir_path + document, "r") as reader:
            save_file = open(save_dir + document, 'w')
            text = reader.read()
            text_tokens = word_tokenize(text)
            tokens_without_sw = [word.replace(',', '').replace('.', '') for word in 
                     text_tokens if (word not in stop_words_list)]
            save_file.writelines(["%s " % item.replace(',', '').replace('.', '') for item in 
            tokens_without_sw])
            
        yield tokens_without_sw
    
print(list(remove_stop_word_from_files()))

The line

return tokens_without_sw

will cause the function to end on the first iteration of the for loop. Instead of returning tokens_without_sw in the for loop, you could create another variable like all_tokens_without_sw where you would append tokens_without_sw at the end of the for loop. Then after the for loop, you could return all_tokens_without_sw.

def remove_stop_word_from_files():
    stop_words_list = get_stop_words()
    dir_path = 'C:/Users/Super/Desktop/IR/homework/Lab4/corpus/corpus/'
    save_dir = "C:/Users/Super/Desktop/IR/homework/Files_Without_SW/"
    
    all_tokens_without_sw = []

    for document in os.listdir(dir_path):
        with open(dir_path + document, "r") as reader:
            save_file = open(save_dir + document, 'w')
            text = reader.read()
            text_tokens = word_tokenize(text)
            tokens_without_sw = [word.replace(',', '').replace('.', '') for word in 
                     text_tokens if (word not in stop_words_list)]
            save_file.writelines(["%s " % item.replace(',', '').replace('.', '') for item in 
            tokens_without_sw])
            
        all_tokens_without_sw = all_tokens_without_sw + tokens_without_sw

    return all_tokens_without_sw
    
print(remove_stop_word_from_files())
Related