I am working on a program that crawls Internet articles using the web crawling method.
The program is started by entering the start and end pages of the website.

This program works in the following order.

1. web-crawling of articles information(title, sort, time, contents)
2. Remove special characters
3. Only nouns are extracted.

The problem maybe occurs lies in extracting nouns in the process of cleaning the content of the article. It works until the stage before noun extraction.

The error message is as follows
ValueError: Length of passed values is 4, index implies 5
To solve this problem, I coded using a method of adding DataFrame append. But it doesn't solve the problem.

Use konlypy method(Korean morpheme analyzer)

import urllib.request
import urllib.parse
from bs4 import BeautifulSoup
import pandas as pd
import requests
import re
from konlpy.tag import Okt
from pandas import Series

i = input('Start page? : ')
k = input('End page? : ')

startpage = int(i)
lastpage = int(k)
count = int(i)

# Definition of text cleaning function
def text_cleaning(text):
    hangul = re.compile('[^ㄱ-ㅣ가-힣]+')
    result = hangul.sub(' ', text)
    return result

# Definition of nouns extraction function
def get_nouns(x):
    nouns_tagger = Okt()
    nouns = nouns_tagger.nouns(x)
    nouns = [noun for noun in nouns if len(noun)>1]
    nouns = [noun for noun in nouns if noun not in stopwords]
    return nouns

# dataframe formation
columns = ['Title', 'Sort', 'Datetime', 'Article']
news_info = pd.DataFrame(columns=columns)
idx = 0

Web-site page loop

while startpage<lastpage + 1:
  url = f'http://www.koscaj.com/news/articleList.html?page={startpage}&total=72698&box_idxno=&sc_section_code=S1N2&view_type=sm'
  html = urllib.request.urlopen(url).read()
  soup = BeautifulSoup(html, 'html.parser')
  links = soup.find_all(class_='list-titles')

  print(f'-----{count}page result-----')
# Articles loop in the web-site page
  for link in links:
    news_url = "http://www.koscaj.com"+link.find('a')['href']
    news_link = urllib.request.urlopen(news_url).read()
    soup2 = BeautifulSoup(news_link, 'html.parser')

    # an article's title
    title = soup2.find('div', {'class':'article-head-title'})

    if title:
        title = soup2.find('div', {'class':'article-head-title'}).text
    else:
        title = ''
           
    # an article's sort
    sorts = soup2.find('nav', {'class':'article-head-nav auto-marbtm-10'})
    try:
        sorts2 = sorts.find_all('a')
        sort = sorts2[2].text
    except:
        sort =''
    
    # an article's time
    date = soup2.find('div',{'class':'info-text'})
    try:
        datetime = date.find('i', {'class':'fa fa-clock-o fa-fw'}).parent.text.strip()
        datetime = datetime.replace("승인", "")
    except:
        datetime = ''

    # an article's content
    article = soup2.find('div', {'id':'article-view-content-div'})
    if article:
        article = soup2.find('div', {'id':'article-view-content-div'}).text
        article = article.replace("\n", "")
        article = article.replace("\r", "")
        article = article.replace("\t", "")
        article = article.replace("[전문건설신문] koscaj@kosca.or.kr", "")
        article = article.replace("저작권자 © 대한전문건설신문 무단전재 및 재배포 금지", "")
        article = article.replace("전문건설신문", "")
        article = article.replace("다른기사 보기", "")

    else:
        article = ''

    # Remove special characters
    news_info['Title'] = news_info['Title'].apply(lambda x: text_cleaning(x))
    news_info['Sort'] = news_info['Sort'].apply(lambda x: text_cleaning(x))
    news_info['Article'] = news_info['Article'].apply(lambda x: text_cleaning(x))

So far, the program works without any problems. But if you see the program error message, it is indicated that the operation is not working because the input value and index are different.

Text data cleaning for extraction nouns

    # Dataframe for storing after crawling individual articles
    row = [title, sort, datetime, article]
    series = pd.Series(row, index=news_info.columns)
    news_info = news_info.append(series, ignore_index=True)
    
    
    
    # Load Korean stopword dictionary file    
    path = "C:/Users/이바울/Desktop/이바울/코딩파일/stopwords-ko.txt"
    with open(path, encoding = 'utf-8') as f:
        stopwords = f.readlines()
    
    stopwords = [x.strip() for x in stopwords]

    news_info['Nouns'] = news_info['Article'].apply(lambda x: get_nouns(x))    


  startpage += 1
  count += 1

news_info.to_excel(f'processing{lastpage-int(1)}-{startpage-int(1)}.xlsx')

print('Complete')

After setting the existing 4 columns in the Pandas DataFrame, the append was used to add the column extracted as a noun as the 5th column. I know this method adds a column regardless of the index name. And if you look at the image link at the bottom, as a result, the first article is crawled and shows the results. From the next article, it does not work and an error occurs.

enter image description here(Program error result)
enter link description here(Korean stopwords dictionary)