Get final resultant text of HTML

Question

I have the following string in:

html = ' hello'

Is there any Python library that can convert this into the following string?

'a. hello'

EDIT: this needs to work for arbirary HTML/CSS (so using ol tag's type attribute, li tag's value attr, CSS's content, counters, and probably hundreds of other HTML/CSS patterns for lists and other things).

EDIT 2: I tried Lynx, that actually would have worked if not for the fact that Lynx apparently can't handle list-style-type and supposedly other common CSS things.

user_1889685 · Answer

You can do that by using a headless browser like Selenium Webdriver because we need to use Window.getComputedStyle() to see which ol li items have lower-alpha value for list-style-type. There is no way to get the text of list items arithmetic/alpha indexes.

We can generate these numbers based on CSS and HTML parameters. HTML lists can get very complex because they may have more than 26 items, which the letters will have to be aa., ab. etc. There are also start and reversed ol attributes. The start defines from where the order starts, for example for

c

reversed

c.

b.

a.

Instructions to install Selenium with Chrome Webdriver

Use pip to install Selenium:
```
pip3 install selenium
```
Download Chrome Webdriver here and add to to your system PATH. Be careful to select the version of your local Chrome installation. Latest Chrome is 80.0.3987.

Python script to scrape ordered list items and generate the `lower-alpha` counters

In this script I use a live URL, but you can check at the bottom of the script I explain how you can use 'data:text/html;' to pass the webdriver some custom HTML like you have.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import string


def get_content(link):
    driver.get(link)

    # Get all page ordered lists
    for ol in WebDriverWait(driver,5).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "ol"))):
        # Get all items from the current ordered list
        list_items = ol.find_elements_by_css_selector("li")
        list_items_count = len(list_items)

        # Get the list start attribute, will return 1 if not present
        ol_start = int(ol.get_attribute("start"))

        # Get the list reversed attribute, will return None if not present
        ol_reversed = ol.get_attribute("reversed")

        # Print information about the ordered list
        print("OL with %s items starting at %s, reversed: %s" % (
            len(list_items), 
            ol_start, 
            "yes" if ol_reversed else "no"))

        # Counter for the letters.
        # If the list is reversed begin count from the last item to the first,
        # else count from first (start) to last
        li_letter = list_items_count if ol_reversed else ol_start

        # Keep count how many list items found with lower-alpha list-style-type
        list_items_found = 0

        for li in list_items:
            # Execute javascript getComputedStyle to get the list item computed style
            list_style_type = driver.execute_script("return window.getComputedStyle(arguments[0])['list-style-type']", li)

            # If the list item computed style 'list-style-type' has 'lower-alpha' value
            if list_style_type == "lower-alpha":
                # Print generated alpha counter and the item text
                print("%s. %s" % (get_alpha_num(li_letter), li.text))

                # If the list is reversed, decrease letter by 1, else increase it
                li_letter += -1 if ol_reversed else 1

                # Keep counting how many items found with 'lower-alpha'
                list_items_found += 1

        # If no items with 'lower-alpha' found do something
        if list_items_found == 0:
            print("No list items found with 'lower-alpha' list style type")
        print()

# Function to convert numbers to letters 1 => a, 26 => aa
def get_alpha_num(num):
    letters = string.ascii_lowercase
    letters_count = len(letters)
    result = ''
    cnum = num - 1

    while(cnum // letters_count > 0):
        cnum //= letters_count
        result += list(letters)[cnum - 1]

    result += list(letters)[((num - 1) % letters_count)]
    return result

if __name__ == '__main__':
    URL = 'https://zikro.gr/dbg/html/lists.html'

    # If you want to parse HTML code from a string
    # then you can use a 'data:text/html;' URL with the HTML contents like this:
    # 
    # html_content = ' hello
there'
    # URL = "data:text/html;charset=utf-8,{html_content}".format(html_content=html_content)
    # 
    # Will result to this:
    #  OL with 2 items starting at 1, reversed: no
    #  a. hello
    #  b. there

    chrome_options = Options()

    # Make headless
    # chrome_options.add_argument("--headless")

    with webdriver.Chrome(options=chrome_options) as driver:
       get_content(URL)

Result

If you try to parse a page with multiple lists like this one:

#ol-a-css li {
  list-style-type: lower-alpha;
}


  Unordered list item 1
  Unordered list item 2
  Unordered list item 3
  Unordered list item 4



  Simple ordered list item 1
  Simple ordered list item 2
  Simple ordered list item 3
  Simple ordered list item 4



  Lower alpha, ordered list item 1
  Lower alpha, ordered list item 2
  Lower alpha, ordered list item 3
  Lower alpha, ordered list item 4



  Lower alpha start=3, ordered list item 1
  Lower alpha start=3, ordered list item 2
  Lower alpha start=3, ordered list item 3
  Lower alpha start=3, ordered list item 4



  Lower alpha reversed, ordered list item 1
  Lower alpha reversed, ordered list item 2
  Lower alpha reversed, ordered list item 3
  Lower alpha reversed, ordered list item 4



  Lower alpha CSS, ordered list item 1
  Lower alpha CSS, ordered list item 2
  Lower alpha CSS, ordered list item 3
  Lower alpha CSS, ordered list item 4
  Lower alpha CSS, ordered list item 5
  Lower alpha CSS, ordered list item 6
  Lower alpha CSS, ordered list item 7
  Lower alpha CSS, ordered list item 8
  Lower alpha CSS, ordered list item 9
  Lower alpha CSS, ordered list item 10
  Lower alpha CSS, ordered list item 11
  Lower alpha CSS, ordered list item 12
  Lower alpha CSS, ordered list item 13
  Lower alpha CSS, ordered list item 14
  Lower alpha CSS, ordered list item 15
  Lower alpha CSS, ordered list item 16
  Lower alpha CSS, ordered list item 17
  Lower alpha CSS, ordered list item 18
  Lower alpha CSS, ordered list item 19
  Lower alpha CSS, ordered list item 20
  Lower alpha CSS, ordered list item 21
  Lower alpha CSS, ordered list item 22
  Lower alpha CSS, ordered list item 23
  Lower alpha CSS, ordered list item 24
  Lower alpha CSS, ordered list item 25
  Lower alpha CSS, ordered list item 26
  Lower alpha CSS, ordered list item 27
  Lower alpha CSS, ordered list item 28
  Lower alpha CSS, ordered list item 29
  Lower alpha CSS, ordered list item 30
  Lower alpha CSS, ordered list item 31

You'll get a result like this:

OL with 4 items starting at 1, reversed: no
No list items found with 'lower-alpha' list style type

OL with 4 items starting at 1, reversed: no
a. Lower alpha, ordered list item 1
b. Lower alpha, ordered list item 2
c. Lower alpha, ordered list item 3
d. Lower alpha, ordered list item 4

OL with 4 items starting at 3, reversed: no
c. Lower alpha start=3, ordered list item 1
d. Lower alpha start=3, ordered list item 2
e. Lower alpha start=3, ordered list item 3
f. Lower alpha start=3, ordered list item 4

OL with 4 items starting at 1, reversed: yes
d. Lower alpha reversed, ordered list item 1
c. Lower alpha reversed, ordered list item 2
b. Lower alpha reversed, ordered list item 3
a. Lower alpha reversed, ordered list item 4

OL with 31 items starting at 1, reversed: no
a. Lower alpha CSS, ordered list item 1
b. Lower alpha CSS, ordered list item 2
c. Lower alpha CSS, ordered list item 3
d. Lower alpha CSS, ordered list item 4
e. Lower alpha CSS, ordered list item 5
f. Lower alpha CSS, ordered list item 6
g. Lower alpha CSS, ordered list item 7
h. Lower alpha CSS, ordered list item 8
i. Lower alpha CSS, ordered list item 9
j. Lower alpha CSS, ordered list item 10
k. Lower alpha CSS, ordered list item 11
l. Lower alpha CSS, ordered list item 12
m. Lower alpha CSS, ordered list item 13
n. Lower alpha CSS, ordered list item 14
o. Lower alpha CSS, ordered list item 15
p. Lower alpha CSS, ordered list item 16
q. Lower alpha CSS, ordered list item 17
r. Lower alpha CSS, ordered list item 18
s. Lower alpha CSS, ordered list item 19
t. Lower alpha CSS, ordered list item 20
u. Lower alpha CSS, ordered list item 21
v. Lower alpha CSS, ordered list item 22
w. Lower alpha CSS, ordered list item 23
x. Lower alpha CSS, ordered list item 24
y. Lower alpha CSS, ordered list item 25
z. Lower alpha CSS, ordered list item 26
aa. Lower alpha CSS, ordered list item 27
ab. Lower alpha CSS, ordered list item 28
ac. Lower alpha CSS, ordered list item 29
ad. Lower alpha CSS, ordered list item 30
ae. Lower alpha CSS, ordered list item 31

user_7706444 · Answer

There is not a library for this that I know of, however you can still use beautifulsoup and the string built-ins. This is kind of an edge-case, as there are only so many different list-style-types.

The strings you are parsing are structured, and if these special list style types exist it tells you what kind they will be in the style tag.

Using BeautifulSoup and the string built-ins

from bs4 import BeautifulSoup
import string


html = """
 hello
hi
Hey
"""

soup = BeautifulSoup(html, "lxml")

# Some obscure hidden tag
hidden_tag = "list-style-type:"

# The style that was listed if there are any
style = None
# First parse the style tags
for s in soup.find_all("style"):
    # If the style tag is for a list, and the hidden tag type is there
    if "li" in s.text and hidden_tag in s.text:
        text = s.text
        # Grab whatever is from the end of the hidden tag, to the first instance of ";" 
        # and strip white space
        style = text[text.find(hidden_tag) + len(hidden_tag):text.find(";")].strip()
# Create a list of the different style types you could encounter.
# For this, I just used lower/upper-alpha but you could use ascii character codes to represent many other styles

style_sets = {
    "lower-alpha": list(string.ascii_lowercase),
    "upper-alpha": list(string.ascii_uppercase),
    # Etc. for whatever possible styles you might encounter
}

# Iterate through the soup as you normally would using bs4.
for ol in soup.find_all("ol"):
    ind = 0
    for li in ol.find_all("li"):
        # If the style type is found we print it using that style type
        if style is not None:
            print("{}. {}".format(style_sets[style][ind], li.text))
            ind += 1
        else:
            print(li.text)

This gives you final strings of:

a. hello
b. hi
c. Hey
a. hello2
b. hi2
c. Hey2

Get final resultant text of HTML

2 Answers

Instructions to install Selenium with Chrome Webdriver

Python script to scrape ordered list items and generate the lower-alpha counters

Result

Python script to scrape ordered list items and generate the `lower-alpha` counters