Limited Output When Using Gazpacho and Pandas to Scrape

Question

I'm trying to scrape a web page for the table of countries and their areas.
My code compiles and runs but only outputs the top two rows, when I want them all.
I thought the problem may lie with .head(), so I played around with it passing numbers and leaving it out all together, but I can't get it to print more than two.

Any help would be appreciated!

from gazpacho import get, Soup
import pandas as pd

url = "https://www.cia.gov/library/publications/the-world-factbook/rankorder/2147rank.html"
response = get(url)
soup = Soup(response)
df0 = pd.read_html(str(soup.find('table')))[0]

print(df0[['Rank', 'Country', '(SQ KM)']].head())

user_7752296 · Answer

First off, no need to use Pandas' .read_html() AND BeautifulSoup/requests AND gazpacho. Pandas actually uses beautifulsoup under the hood and uses requests as well.

Secondly, I don't have an issue with it not printing out more than 2 rows. Where are you running this? Is it possible you have a setting/preference that only outputs x amount of lines?

import pandas as pd

url = "https://www.cia.gov/library/publications/the-world-factbook/rankorder/2147rank.html"
df0 = pd.read_html(url)[0]

print(df0[['Rank', 'Country', '(SQ KM)']])

Output:

print(df0[['Rank', 'Country', '(SQ KM)']].to_string())
     Rank                                        Country   (SQ KM)
0       1                                         Russia  17098242
1       2                                     Antarctica  14000000
2       3                                         Canada   9984670
3       4                                  United States   9833517
4       5                                          China   9596960
5       6                                         Brazil   8515770
6       7                                      Australia   7741220
7       8                                          India   3287263
8       9                                      Argentina   2780400
9      10                                     Kazakhstan   2724900
10     11                                        Algeria   2381741
11     12              Congo, Democratic Republic of the   2344858
12     13                                      Greenland   2166086
13     14                                   Saudi Arabia   2149690
14     15                                         Mexico   1964375
15     16                                      Indonesia   1904569
16     17                                          Sudan   1861484
17     18                                          Libya   1759540
18     19                                           Iran   1648195
19     20                                       Mongolia   1564116
20     21                                           Peru   1285216
21     22                                           Chad   1284000
22     23                                          Niger   1267000
23     24                                         Angola   1246700
24     25                                           Mali   1240192
25     26                                   South Africa   1219090
26     27                                       Colombia   1138910
27     28                                       Ethiopia   1104300
28     29                                        Bolivia   1098581
29     30                                     Mauritania   1030700
30     31                                          Egypt   1001450
31     32                                       Tanzania    947300
32     33                                        Nigeria    923768
33     34                                      Venezuela    912050
34     35                                        Namibia    824292
35     36                                     Mozambique    799380
36     37                                       Pakistan    796095
37     38                                         Turkey    783562
38     39                                          Chile    756102
39     40                                         Zambia    752618
40     41                                          Burma    676578
41     42                                    Afghanistan    652230
42     43                                    South Sudan    644329
43     44                                         France    643801
44     45                                        Somalia    637657
45     46                       Central African Republic    622984
46     47                                        Ukraine    603550
47     48                                     Madagascar    587041
48     49                                       Botswana    581730
49     50                                          Kenya    580367
50     51                                          Yemen    527968
51     52                                       Thailand    513120
52     53                                          Spain    505370
53     54                                   Turkmenistan    488100
54     55                                       Cameroon    475440
55     56                               Papua New Guinea    462840
56     57                                         Sweden    450295
57     58                                     Uzbekistan    447400
58     59                                        Morocco    446550
59     60                                           Iraq    438317
60     61                                       Paraguay    406752
61     62                                       Zimbabwe    390757
62     63                                          Japan    377915
63     64                                        Germany    357022
64     65                         Congo, Republic of the    342000
65     66                                        Finland    338145
66     67                                        Vietnam    331210
67     68                                       Malaysia    329847
68     69                                         Norway    323802
69     70                                  Cote d'Ivoire    322463
70     71                                         Poland    312685
71     72                                           Oman    309500
72     73                                          Italy    301340
73     74                                    Philippines    300000
74     75                                        Ecuador    283561
75     76                                   Burkina Faso    274200
76     77                                    New Zealand    268838
77     78                                          Gabon    267667
78     79                                 Western Sahara    266000
79     80                                         Guinea    245857
80     81                                 United Kingdom    243610
81     82                                         Uganda    241038
82     83                                          Ghana    238533
83     84                                        Romania    238391
84     85                                           Laos    236800
85     86                                         Guyana    214969
86     87                                        Belarus    207600
87     88                                     Kyrgyzstan    199951
88     89                                        Senegal    196722
89     90                                          Syria    185180
90     91                                       Cambodia    181035
91     92                                        Uruguay    176215
92     93                                       Suriname    163820
93     94                                        Tunisia    163610
94     95                                     Bangladesh    148460
95     96                                          Nepal    147181
96     97                                     Tajikistan    144100
97     98                                         Greece    131957
98     99                                      Nicaragua    130370
99    100                                   Korea, North    120538
100   101                                         Malawi    118484
101   102                                        Eritrea    117600
102   103                                          Benin    112622
103   104                                       Honduras    112090
104   105                                        Liberia    111369
105   106                                       Bulgaria    110879
106   107                                           Cuba    110860
107   108                                      Guatemala    108889
108   109                                        Iceland    103000
109   110                                   Korea, South     99720
110   111                                        Hungary     93028
111   112                                       Portugal     92090
112   113                                         Jordan     89342
113   114                                     Azerbaijan     86600
114   115                                        Austria     83871
115   116                           United Arab Emirates     83600
116   117                                        Czechia     78867
117   118                                         Serbia     77474
118   119                                         Panama     75420
119   120                                   Sierra Leone     71740
120   121                                        Ireland     70273
121   122                                        Georgia     69700
122   123                                      Sri Lanka     65610
123   124                                      Lithuania     65300
124   125                                         Latvia     64589
125   126                                       Svalbard     62045
126   127                                           Togo     56785
127   128                                        Croatia     56594
128   129                         Bosnia and Herzegovina     51197
129   130                                     Costa Rica     51100
130   131                                       Slovakia     49035
131   132                             Dominican Republic     48670
132   133                                        Estonia     45228
133   134                                        Denmark     43094
134   135                                    Netherlands     41543
135   136                                    Switzerland     41277
136   137                                         Bhutan     38394
137   138                                  Guinea-Bissau     36125
138   139                                         Taiwan     35980
139   140                                        Moldova     33851
140   141                                        Belgium     30528
141   142                                        Lesotho     30355
142   143                                        Armenia     29743
143   144                                Solomon Islands     28896
144   145                                        Albania     28748
145   146                              Equatorial Guinea     28051
146   147                                        Burundi     27830
147   148                                          Haiti     27750
148   149                                         Rwanda     26338
149   150                                      Macedonia     25713
150   151                                       Djibouti     23200
151   152                                         Belize     22966
152   153                                    El Salvador     21041
153   154                                         Israel     20770
154   155                                       Slovenia     20273
155   156                                  New Caledonia     18575
156   157                                           Fiji     18274
157   158                                         Kuwait     17818
158   159                                      Swaziland     17364
159   160                                    Timor-Leste     14874
160   161                                   Bahamas, The     13880
161   162                                     Montenegro     13812
162   163                                        Vanuatu     12189
163   164              Falkland Islands (Islas Malvinas)     12173
164   165                                          Qatar     11586
165   166                                    Gambia, The     11300
166   167                                        Jamaica     10991
167   168                                         Kosovo     10887
168   169                                        Lebanon     10400
169   170                                         Cyprus      9251
170   171                                    Puerto Rico      9104
171   172                                      West Bank      5860
172   173                                         Brunei      5765
173   174                            Trinidad and Tobago      5128
174   175                               French Polynesia      4167
175   176                                     Cabo Verde      4033
176   177       South Georgia and South Sandwich Islands      3903
177   178                                          Samoa      2831
178   179                                     Luxembourg      2586
179   180                                        Comoros      2235
180   181                                      Mauritius      2040
181   182                                 Virgin Islands      1910
182   183                                  Faroe Islands      1393
183   184                                      Hong Kong      1108
184   185                          Sao Tome and Principe       964
185   186                       Turks and Caicos Islands       948
186   187                                       Kiribati       811
187   188                                        Bahrain       760
188   189                                       Dominica       751
189   190                                          Tonga       747
190   191                Micronesia, Federated States of       702
191   192                                      Singapore       697
192   193                                    Saint Lucia       616
193   194                                    Isle of Man       572
194   195                                           Guam       544
195   196                                        Andorra       468
196   197                       Northern Mariana Islands       464
197   198                                          Palau       459
198   199                                     Seychelles       455
199   200                                        Curacao       444
200   201                            Antigua and Barbuda       443
201   202                                       Barbados       430
202   203              Heard Island and McDonald Islands       412
203   204  Saint Helena, Ascension, and Tristan da Cunha       394
204   205               Saint Vincent and the Grenadines       389
205   206                                      Jan Mayen       377
206   207                                     Gaza Strip       360
207   208                                        Grenada       344
208   209                                          Malta       316
209   210                                       Maldives       298
210   211                                 Cayman Islands       264
211   212                          Saint Kitts and Nevis       261
212   213                                           Niue       260
213   214                      Saint Pierre and Miquelon       242
214   215                                   Cook Islands       236
215   216                                 American Samoa       199
216   217                               Marshall Islands       181
217   218                                          Aruba       180
218   219                                  Liechtenstein       160
219   220                         British Virgin Islands       151
220   221                              Wallis and Futuna       142
221   222                               Christmas Island       135
222   223                                       Dhekelia       131
223   224                                       Akrotiri       123
224   225                                         Jersey       116
225   226                                     Montserrat       102
226   227                                       Anguilla        91
227   228                                       Guernsey        78
228   229                                     San Marino        61
229   230                 British Indian Ocean Territory        60
230   231            French Southern and Antarctic Lands        55
231   232                                   Saint Martin        54
232   233                                        Bermuda        54
233   234                                  Bouvet Island        49
234   235                               Pitcairn Islands        47
235   236                                 Norfolk Island        36
236   237                                   Sint Maarten        34
237   238                                          Macau        28
238   239                                         Tuvalu        26
239   240                               Saint Barthelemy        25
240   241  United States Pacific Island Wildlife Refuges        22
241   242                                          Nauru        21
242   243                        Cocos (Keeling) Islands        14
243   244                                        Tokelau        12
244   245                                Paracel Islands         8
245   246                                      Gibraltar         7
246   247                                    Wake Island         7
247   248                              Clipperton Island         6
248   249                                 Navassa Island         5
249   250                                Spratly Islands         5
250   251                    Ashmore and Cartier Islands         5
251   252                              Coral Sea Islands         3
252   253                                         Monaco         2
253   254                        Holy See (Vatican City)         0

user_8506548 · Answer

You can also use lxml for this

import requests
import lxml.html

url = 'https://www.cia.gov/library/publications/the-world-factbook/rankorder/2147rank.html'
response = requests.get(url, timeout=5)
tree = lxml.html.fromstring(response.text)
# Extract the table
table = tree.get_element_by_id('rankOrder')
data = table.xpath('//tr/td//text()')
# Separate the columns
rank = data[0::4]
country = data[1::4]
sq_km = data[2::4]
date_of_info = data[3::4]

If you need a data frame the rest is just formatting

# If you want a data frame
import pandas
df = pandas.DataFrame(dict(country=country, sq_km=sq_km, date_of_info=date_of_info))
df
                         country       sq_km date_of_info
0                         Russia  17,098,242           \r
1                     Antarctica  14,000,000           \r
2                         Canada   9,984,670           \r
3                  United States   9,833,517           \r
4                          China   9,596,960           \r
..                           ...         ...          ...
249              Spratly Islands           5           \r
250  Ashmore and Cartier Islands           5           \r
251            Coral Sea Islands           3           \r
252                       Monaco           2           \r
253      Holy See (Vatican City)           0           \r

[254 rows x 3 columns]

Limited Output When Using Gazpacho and Pandas to Scrape

2 Answers