Getting “no table found” error when web scraping with pandas for OTC Markets screener website

Question

I want to extract various statistics from this website(https://www.otcmarkets.com/research/stock-screener). Unfortunately, pandas do not recognize the tables presented. Here is my code:

import requests

import pandas as pd

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36'}

def Get_table(screen):

    tables = pd.read_html(screen)

    tables.columns = tables.iloc[0]
    
    return tables

screen = requests.get('https://www.otcmarkets.com/research/stock-screener', headers = header).text

table = Get_table(screen)

ValueError: No tables found

user_10035985 · Answer

The page loads the data from external source (URL). You can use this example how to load the data from API and create a dataframe:

import json
import pandas as pd

url = "https://www.otcmarkets.com/research/stock-screener/api"
data = json.loads(requests.get(url).json())
df = pd.json_normalize(data["stocks"])

Prints:

    securityId               reportDate symbol               securityName              market  marketId     securityType country         state forexCountry  caveatEmptor  industryId                           industry  volume  volumeChange  dividendYield  dividendPayer  morningStarRating  penny     price  shortInterest  shortInterestPercent  shortInterestRatio   pct1Day   pct5Day  pct4Weeks  pct13Weeks  pct52Weeks isBank  perfQxComp4Weeks  perfQxComp13Weeks  perfQxComp52Weeks  perfQxBillion4Weeks  perfQxBillion13Weeks  perfQxBillion52Weeks  perfQxBanks4Weeks  perfQxBanks13Weeks  perfQxBanks52Weeks  perfQxIntl4Weeks  perfQxIntl13Weeks  perfQxIntl52Weeks  perfQxUs4Weeks  perfQxUs13Weeks  perfQxUs52Weeks  perfQb4Weeks  perfQb13Weeks  perfQb52Weeks  perfSp4Weeks  perfSp13Weeks  perfSp52Weeks  perfQxDiv4Weeks  perfQxDiv13Weeks  perfQxDiv52Weeks  perfQxCan4Weeks  perfQxCan13Weeks  perfQxCan52Weeks
0       117230  Aug 3, 2021 12:00:00 AM   MHGU  MERITAGE HOSPTLTY GRP INC  OTCQX U.S. Premier         1     Common Stock     USA      Michigan          USA         False        5812                      Eating places     216      0.625623       1.122500           True                3.0   True   21.3800          171.0                100.00            0.000025 -0.003263 -0.003263  -0.049778   -0.021510    0.388312      N         -1.934711          -0.346011           1.087740            -1.748822             -0.327350              1.101413         -11.044760           -0.532955            0.755842         -1.887983          -0.284616           1.105790        2.474310         0.094201         0.557874      1.046712       0.172299       1.418802     -6.221993      -0.463678       1.170963        -1.515941         -0.269834          1.011993         0.034056          0.013836          0.026113
1       130262  Aug 3, 2021 12:00:00 AM  MHGUP    MERITAGE HOSPTLTY PFD B  OTCQX U.S. Premier         1  Preferred Stock     USA      Michigan          USA         False        5812                      Eating places       0      2.984908       2.100000           True                NaN   True   38.0000            NaN                   NaN            0.000000       NaN       NaN        NaN         NaN         NaN      N               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
2        32227  Aug 3, 2021 12:00:00 AM   TYCB  TAYLOR(CLVN B)BKG BRLN MD  OTCQX U.S. Premier         1     Common Stock     USA      Maryland          USA         False        6712             Bank holding companies       1      0.867442       3.300000           True                3.0   True   35.1000            NaN                   NaN            0.000000       NaN       NaN        NaN         NaN         NaN      Y               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
3        31499  Aug 3, 2021 12:00:00 AM   STBI        STURGIS BANCORP INC  OTCQX U.S. Premier         1     Common Stock     USA      Michigan          USA         False        6035       Federal savings institutions    1000      1.142256       3.355200           True                3.0   True   19.0750            NaN                   NaN            0.000000 -0.008576 -0.002614   0.003947   -0.046250   -0.046250      Y          0.153422          -0.743970          -0.129556             0.138681             -0.703847             -0.131184           0.875847           -1.145925           -0.090025          0.149717          -0.611962          -0.131705       -0.196212         0.202544        -0.066446     -0.083004       0.370465      -0.168987      0.493403      -0.996969      -0.139468         0.120214         -0.580178         -0.120534        -0.002701          0.029750         -0.003110
4        27295  Aug 3, 2021 12:00:00 AM   PSBP           PSB HOLDING CORP  OTCQX U.S. Premier         1     Common Stock     USA      Maryland          USA         False        6022             State commercial banks       0      5.595744       0.645856          False                3.0   True   27.8700           19.0                -96.31            0.000012       NaN       NaN        NaN         NaN         NaN      Y               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
5        24830  Aug 3, 2021 12:00:00 AM   OCBI       ORANGE CTY BNCRP INC  OTCQX U.S. Premier         1     Common Stock     USA      New York          USA         False        6712             Bank holding companies       5      0.109266       2.352900           True                3.0   True   34.0000          200.0                100.00            0.000045       NaN       NaN        NaN         NaN         NaN      Y               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
6        20776  Aug 3, 2021 12:00:00 AM   MNBP           MARS BANCORP INC  OTCQX U.S. Premier         1     Common Stock     USA  Pennsylvania          USA         False        6021          National commercial banks     139      1.306208       3.084700           True                3.0   True   20.7475            NaN                   NaN            0.000000  0.004722  0.037375  -0.090022   -0.949396   -0.943158      Y         -3.498879         -15.271837          -2.641976            -3.162702            -14.448208             -2.675186         -19.974187          -23.522956           -1.835841         -3.414373         -12.562040          -2.685816        4.474732         4.157719        -1.355001      1.892953       7.604722      -3.446082    -11.252328     -20.465275      -2.844113        -2.741543        -11.909604         -2.457995         0.061590          0.610687         -0.063426
7        83455  Aug 3, 2021 12:00:00 AM   MNAT        MARQUETTE NATL CORP  OTCQX U.S. Premier         1     Common Stock     USA      Illinois          USA         False        6022             State commercial banks       0      0.978290       2.934800           True                3.0   True   36.8000           49.0                 68.97            0.000011       NaN       NaN        NaN         NaN         NaN      Y               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
8        18948  Aug 3, 2021 12:00:00 AM   KISB           KISH BANCORP INC  OTCQX U.S. Premier         1     Common Stock     USA  Pennsylvania          USA         False        6712             Bank holding companies     173      0.440998       3.411800           True                3.0   True   34.0000            2.0                  0.00            0.000001  0.008005  0.011905   0.035954    0.054264    0.387755      Y          1.397410           0.872875           1.086181             1.263146              0.825800              1.099834           7.977452            1.344475            0.754759          1.363660           0.717994           1.104205       -1.787155        -0.237638         0.557074     -0.756023      -0.434654       1.416768      4.494046       1.169710       1.169285         1.094940          0.680704          1.010542        -0.024598         -0.034904          0.026076
9       266615  Aug 3, 2021 12:00:00 AM   KCLI   KANSAS CITY LIFE INS NEW  OTCQX U.S. Premier         1     Common Stock     USA      Missouri          USA         False        6311                     Life insurance     106      0.210918       2.494200           True                3.0   True   43.3000          719.0                  0.00            0.000074  0.000000  0.000000  -0.029148   -0.048352    0.503472      N         -1.132893          -0.777777           1.410328            -1.024044             -0.735830              1.428056          -6.467393           -1.197997            0.980000         -1.105531          -0.639770           1.433731        1.448862         0.211748         0.723321      0.612915       0.387300       1.839572     -3.643364      -1.042273       1.518232        -0.887678         -0.606542          1.312116         0.019942          0.031102          0.033858
10       12485  Aug 3, 2021 12:00:00 AM   FBAK            FIRST NB ALASKA  OTCQX U.S. Premier         1     Common Stock     USA        Alaska          USA         False        6021          National commercial banks     360      1.697465       5.493600           True                3.0   True  233.0000           62.0                 37.78            0.000020 -0.004274  0.040179   0.000000   -0.046879    0.308989      Y               NaN          -0.754085           0.865540                  NaN             -0.713417              0.876420                NaN           -1.161505            0.601442               NaN          -0.620282           0.879903             NaN         0.205298         0.443913           NaN       0.375502       1.128974           NaN      -1.010525       0.931763              NaN         -0.588067          0.805266              NaN          0.030154          0.020779
11       11749  Aug 3, 2021 12:00:00 AM   FETM      FENTURA FINANCIAL INC  OTCQX U.S. Premier         1     Common Stock     USA      Michigan          USA         False        6022             State commercial banks   19144      0.671837       1.230000           True                3.0   True   26.0000          185.0                -86.89            0.000040 -0.009524  0.006581   0.000000    0.037924    0.477273      Y               NaN           0.610042           1.336938                  NaN              0.577141              1.353744                NaN            0.939637            0.929004               NaN           0.501797           1.359123             NaN        -0.166082         0.685681           NaN      -0.303775       1.743845           NaN       0.817497       1.439227              NaN          0.475736          1.243837              NaN         -0.024394          0.032096
12       10994  Aug 3, 2021 12:00:00 AM   ENBP      ENB FINANCIAL CORP PA  OTCQX U.S. Premier         1     Common Stock     USA  Pennsylvania          USA         False        6021          National commercial banks      20      0.517902       2.912200           True                3.0   True   23.3500            NaN                   NaN            0.000000       NaN       NaN        NaN         NaN         NaN      Y               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
13        8482  Aug 3, 2021 12:00:00 AM   CNIG   CORNING NATURAL GAS HLDG  OTCQX U.S. Premier         1     Common Stock     USA      New York          USA         False        4923  Gas transmission and distribution     260      0.173180       2.510000           True                3.0   True   24.3000            1.0                  0.00            0.000000  0.019723  0.022727   0.023158    0.031847    0.494465      N          0.900077           0.512288           1.385097             0.813596              0.484660              1.402508           5.138305            0.789068            0.962468          0.878338           0.421389           1.408081       -1.151112        -0.139469         0.710380     -0.486957      -0.255097       1.806662      2.894630       0.686500       1.491071         0.705254          0.399503          1.288642        -0.015844         -0.020485          0.033252
14        6722  Aug 3, 2021 12:00:00 AM   CBAF       CITBA FINANCIAL CORP  OTCQX U.S. Premier         1     Common Stock     USA       Indiana          USA         False        6712             Bank holding companies       0      0.317975       2.142900           True                3.0   True   28.0000            3.0                  0.00            0.000002       NaN       NaN        NaN         NaN         NaN      Y               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
15        5489  Aug 3, 2021 12:00:00 AM   CPTP   CAPITAL PROPERTIES INC A  OTCQX U.S. Premier         1     Common Stock     USA  Rhode Island          USA         False        6519      Lessors of Real Property, NEC       0      1.535617       2.002900           True                3.0   True   13.9800            NaN                   NaN            0.000000       NaN       NaN        NaN         NaN         NaN      N               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
16      107523  Aug 3, 2021 12:00:00 AM   BHWB      BLACKHAWK BANCORP INC  OTCQX U.S. Premier         1     Common Stock     USA     Wisconsin          USA         False        6022             State commercial banks     200      3.166295       1.239400           True                3.0   True   35.5000          116.0                 24.73            0.000041  0.000000  0.014286   0.021583    0.109375    0.783920      Y          0.838855           1.759389           2.195918             0.758257              1.664503              2.223521           4.788806            2.709957            1.525887          0.818595           1.447207           2.232357       -1.072816        -0.478989         1.126229     -0.453835      -0.876100       2.864263      2.697743       2.357698       2.363928         0.657284          1.372043          2.043000        -0.014766         -0.070354          0.052718
17        1888  Aug 3, 2021 12:00:00 AM   CFNB   CALIFORNIA FIRST LEASING  OTCQX U.S. Premier         1     Common Stock     USA    California          USA         False        6172                    Finance Lessors       0      0.095971       2.918900           True                3.0   True   18.5000         1002.0                  0.00            0.000097       NaN       NaN        NaN         NaN         NaN      Y               NaN                NaN                NaN                  NaN                   NaN                   NaN                NaN                 NaN                 NaN               NaN                NaN                NaN             NaN              NaN              NaN           NaN            NaN            NaN           NaN            NaN            NaN              NaN               NaN               NaN              NaN               NaN               NaN
18       53651  Aug 3, 2021 12:00:00 AM   BNCC                BNCCORP INC  OTCQX U.S. Premier         1     Common Stock     USA  North Dakota          USA         False        6021          National commercial banks     623      0.444949       0.000000           True                3.0   True   39.2500          103.0                  0.00            0.000029  0.006410  0.037265   0.012903    0.014212    0.365217      Y          0.501509           0.228610           1.023048             0.453324              0.216281              1.035908           2.862985            0.352124            0.710890          0.489397           0.188046           1.040024       -0.641382        -0.062239         0.524695     -0.271325      -0.113838       1.334421      1.612844       0.306353       1.101322         0.392957          0.178280          0.951806        -0.008828         -0.009142          0.024560
19        7590  Aug 3, 2021 12:00:00 AM   CNAF   COMML NATL FINCL CORP PA  OTCQX U.S. Premier         1     Common Stock     USA  Pennsylvania          USA         False        6022             State commercial banks    3100      0.671460       9.951200           True                3.0   True   20.5000          378.0                -70.14            0.000132  0.000000  0.014851   0.006382    0.072737    0.138889      Y          0.248046           1.170032           0.389056             0.224214              1.106931              0.393947           1.416032            1.802181            0.270345          0.242055           0.962425           0.395512       -0.317228        -0.318538         0.199537     -0.134197      -0.582626       0.507468      0.797712       1.567921       0.418823         0.194356          0.912439          0.361963        -0.004366         -0.046787          0.009340

Getting “no table found” error when web scraping with pandas for OTC Markets screener website

1 Answers