Beautiful Soup is adding whitespaces between every character

Question

I'm running Beautiful Soup on some HTML documents. All I want is to extract the text without any tags.

from bs4 import BeautifulSoup

def aex_extract_text(html):
    soup = BeautifulSoup(html, 'lxml')
    text = soup.get_text()
    return text

My input is somewhat pathological in the number of redundant tags that it's got. But it is automatically generated HTML, so there shouldn't be any missing angles or anything of that nature. It's some Drupal system that creates the HTML. On almost all documents it runs fine, but on about 1/50 I'm getting the first few lines good and then whitespace between every character, and the tags aren't being removed.

Anyone know what causes this sort of breakdown? I can't see anything obvious. The fact that it's breaking out in the middle of a paragraph is confusing. I don't need perfect preservation of the text, so if I need to pull out a few characters I don't mind.

Example input:

[...]


    
  
    The 2^nd  Respondent was declared the duly 
elected Member of Parliament for the  Chilanga Constituency.
    
  
    The result was  unsuccessfully challenged, in the High Court, to have it declared  null and void on a number of allegations 
not relevant to this appeal  as it will be seen later in this judgment.
    
  
    It is also not in  dispute that the learned trial judge informed the parties of his  intention to call some witnesses 
after the defence had closed its  case. Five witnesses, in total, were called by the court and they  were designated as court witnesses.
    
  
    This appeal is  premised on one ground of appeal 
which is attacking the learned trial  judge’s jurisdiction to call witnesses on his own motion.

[...]

Example output:

[...]

The 2nd Respondent was declared the duly elected Member of Parliament for the Chilanga Constituency. The result was unsuccessfully challenged, in the High Court, to have it declared null and void on a number of allegations not relevant to this appeal as it will be seen later in this judgment. It is also not in dispute that the learned trial judge informed the parties of his intention to call some witnesses a f t e r t h e d e f e n c e h a d c l o s e d i t s c a s e . F i v e w i t n e s s e s , i n t o t a l , w e r e c a l l e d b y t h e c o u r t a n d t h e y w e r e d e s i g n a t e d a s c o u r t w i t n e s s e s . / F O N T > / F O N T > / P > P L A N G = e n - U S C L A S S = w e s t e r n A L I G N = J U S T I F Y S T Y L E = m a r g i n - b o t t o m : 0 c m ; l i n e - h e i g h t : 1 5 0 % > B R > / P > P L A N G = e n - U S C L A S S = w e s t e r n A L I G N = J U S T I F Y S T Y L E = m a r g i n - b o t t o m :

[...]

UPDATE:

here's the full input up to where it goes wrong:

J 26 (706) IN THE SUPREME COURT OF ZAMBIA SCZ APPEAL NO.182 OF 2007 HOLDEN AT LUSAKA SCZ Judgment No. 31 OF 2008 (CIVIL JURISDICTION) IN THE MATTER OF: THE ELECTORAL ACT NO.12 OF 2006 SECTION 93 OF THE LAWS OF ZAMBIA IN THE MATTER OF: PARLIAMENTARY ELECTION FOR CHILANGA CONSTITUENCY HELD ON THE 28^TH SEPTEMBER, 2006 IN THE MATTER OF: AN ELECTION PETITION BETWEEN: PRISCILLA MWENYA KAMANGA - APPELLANT AND THE ATTORNEY-GENERAL - 1^ST RESPONDENT HON. NG’ANDU PETER MAGANDE - 2^ND RESPONDENT CORAM : Sakala, CJ.,Chirwa, Mumba, Chitengi and Mushabati; JJS. On 25^TH June, 2006 and 19^th August, 2008 For the Appellants : Mr.B.C. Mutale, SC. and Mr L. Kalaluka of Ellis and Company For the 1^st Respondent: Mr M. Mukwasa – State Advocate For the 2^nd Respondent: Mr S.C. Malama, SC. of Jaques and Partners ______________________________________________________________________________ JUDGMENT Mushabati, JS., delivered the judgment of the Court. (707) Cases referred to: Double Mwale Vs The People [1984] Z.R. 76 Simwanza Vs The People [1985] Z.R.15 Jones Vs National Coal Board [1957] 2. ALL E.R. 155 Re Enock and Zaretzky, Bock and Co. [1910] 1 K.B.327 Porter Vs Magill [2002] 1 ALL E.R. 465 Fallow Vs Calvert [1960] 2.Q.B. 201 Water Welles Ltd Vs Jackson [1984] Z.R. 98 Zambia Telecommunications Ltd Vs Celtel (Z) Ltd SCZ No. 90 of 2006 (unreported) Mazoka and others Vs Mwanawasa and others [2005] Z.R. 138 Thomas Mumba and others Vs The People SCZ Appeal No. 92-95 (unreported) Lewanika and others Vs Chiluba [1998] Z.R. 79 Mabenga Vs Wina [2003] Z.R. 110 Legislation referred to: Criminal Procedure Code, Cap. 87 – S.149 Electoral Act, No.12 of 2006 – SS.102(3) and 103(1)(a) and (b) Other works referred to: Halsburys Laws of England 4^th Edition Vol. 17 Page 195 Paragraph 281 This is an appeal against the High Court judgment of 30^th July, 2007 dismissing the appellant’s petition against the election of the 2^nd Respondent as a Member of Parliament for the Chilanga Parliamentary Constituency during the Presidential and Parliamentary General Elections held on 28^th September, 2006, seeking to nullify the 2^nd Respondent’s election on a number of malpractices as pleaded in the petition. (708) We wish to state that in the court below, the petition was prosecuted by two Petitioners but the 2^nd Petitioner, Capt. Cosmas Moono, is not a party to this appeal. However, for clarity’s sake we shall simply refer to the appellant as the Petitioner and the 1^st and 2^nd Respondents as the 1^st and 2^nd Respondents respectively, the titles they held in the court below.

The undisputed facts of this case are that both the Petitioner and the 2^nd Respondent contested the Parliamentary General Elections held on 28^th September, 2006 in the Chilanga Constituency. The Petitioner stood on the Patriotic Front (PF) ticket; while the 2^nd Respondent stood on the ticket of the Movement for Multiparty Democracy (MMD).

UPDATE:

I've voted to close my own question. The answer (which works) is here: BeautifulSoup return unexpected extra spaces

user_860099 · Answer

Look - the input document is invalid HTML - because it has three close tags at beginning:

may be this is the reason. May be also parse and change to

Beautiful Soup is adding whitespaces between every character

1 Answers