[Selenium] Selenium 이용해서 네이버 블로그 크롤링

Selenium 패키지는 chromdriver를 제어하여 원하는 정보를 얻기 위해 사용하는 패키지다.

selenium 패키지 설치 방법 및 자세한 사항은 아래 사이트를 참조하길 바란다. 

 

https://wikidocs.net/137914

 

네이버 블로그에서 정보를 검색했을 때, 나오는 리스트를 자동으로 스크롤하여 크롤링하는 코드

 

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import pandas as pd
import re
from openpyxl.cell.cell import ILLEGAL_CHARACTERS_RE

# set up the webdriver
driver = webdriver.Chrome()

#open the web page
url ='크롤링하기 원하는 사이트'
driver.get(url)

# wait for the page to load
time.sleep(3)

# scroll down to load more posts
body = driver.find_element(By.TAG_NAME, 'body')
for i in range(1000):
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(1)

    # get the posts
    posts = driver.find_elements(By.CSS_SELECTOR, 'ul.lst_total > li')

    # create lists to store the data
    titles = []
    descriptions = []
    urls = []

    # loop through each post and extract the title, description, and URL
    for post in posts:
        try:
            title = post.find_element(By.CSS_SELECTOR, '.api_txt_lines.total_tit').text
            description = post.find_element(By.CSS_SELECTOR, '.api_txt_lines.dsc_txt').text
            url = post.find_element(By.CSS_SELECTOR, '.api_txt_lines.total_tit').get_attribute('href')
            titles.append(title)
            descriptions.append(description)
            urls.append(url)

        except:
            title = ILLEGAL_CHARACTERS_RE.sub(r'',title)
            description = ILLEGAL_CHARACTERS_RE.sub(r'',description)
            url = post.find_element(By.CSS_SELECTOR, '.api_txt_lines.total_tit').get_attribute('href')
            titles.append(title)
            descriptions.append(description)
            urls.append(url)


    # create a dataframe and export to excel
    df = pd.DataFrame({'Title': titles, 'Description': descriptions, 'URL': urls})
    df.to_excel('제목.xlsx', index=False)

# close the webdriver
driver.quit()

 

' ' 부분만 해당 사이트의 console 창을 확인하여 변경하거나 자신이 원하는 이름으로 변경하면 된다.

 

 

 

[참조 사이트]

https://mokeya.tistory.com/398