Get values from same class name values in colly web scraping

user_13547531 0
Viewed 862

i am working on small web scraping application using go language and colly web scraping framework which is built in Go

here is the html code of website

<div clas="cc">  
    <div class="list">
        <span class="countrybg" style="background-image: url(countryimage);"></span>
        <span class="continet">Asia</span>
        <span class="country">india</span>
    </div>
    <div class="list">
        <span class="countrybg" style="background-image: url(countryimage);"></span>
        <span class="continet">Africa</span>
        <span class="country">Brazil</span>
    </div>
</div>

now i want to fetch all the three span elements one by one and append to array

i tried with this code but it does not work but it return as AsiaAfrica
but i want the values separately and want to fetch the image url of countrybg class

c := make([]string, 10) 
element.ForEach(".list span", func(_ int, elem *colly.HTMLElement) {
            result := element.ChildText("span:nth-child(2)")
            c = append(c, result)
})

the example output should be like

countrybg = ['image1url' ,'image2url']
continet = ['Asia' ,'Africa']
country = ['india' ,'Brazil']

can any one help to get this

go-colly go
1 Answers

I ran a local server on port 8081 and tried getting the values you are looking for. There are many ways to do what you need, this is just one:

package main

import (
    "fmt"
    "regexp"

    "github.com/gocolly/colly"
)

func main() {
    c := colly.NewCollector()

    countrybgs := []string{}
    continents := []string{}
    countries := []string{}

    r := regexp.MustCompile(`background-image: url\((.*)\);`)

    /*
        <div clas="cc">
            <div class="list">
                <span class="countrybg" style="background-image: url(image1url);"></span>
                <span class="continet">Asia</span>
                <span class="country">india</span>
            </div>
            <div class="list">
                <span class="countrybg" style="background-image: url(image2url);"></span>
                <span class="continet">Africa</span>
                <span class="country">Brazil</span>
            </div>
        </div>
    */

    c.OnHTML("span", func(e *colly.HTMLElement) {
        switch class := e.Attr("class"); class {
        case "countrybg":
            countrybgs = append(countrybgs, r.FindStringSubmatch(e.Attr("style"))[1])
        case "continet":
            continents = append(continents, e.Text)
        case "country":
            countries = append(countries, e.Text)
        }
    })

    c.Visit("http://localhost:8081")

    fmt.Println(countrybgs)
    fmt.Println(continents)
    fmt.Println(countries)
}

the output:

> go run .
[image1url image2url]
[Asia Africa]
[india Brazil]
user_41905650
Related