Visualizing tweet lengths in python

Question

Lets do necessary imports:

import seaborn as sns
import pandas as pd 
import matplotlib.pyplot as plt

Then lets generate list of thousand random lengthed (0 to 101) strings to use as sample tweets.

from random import choice, randrange
from string import ascii_lowercase, digits

chars = ascii_lowercase + digits
lst = [''.join(choice(chars) for _ in range(randrange(0, 101, 1))) for _ in range(1000)]

Lets create countplot to visualize what is the frequency of tweet lengths:

tweet_lengths = list(map(len, lst))
sns.countplot(tweet_lengths)
plt.title('Tweet lengths')
plt.xlabel('Tweet lengths')
plt.ylabel('# of tweets')

This outputted:

So this is not following normal distribution, obviously because its randomly generated. What was surprising that on actual tweets with some pre processing (removing URLs, mentions etc), it wasnt normal distribution. (a side question: why this could be? Shouldnt it be normal? or random sized urls and my pre-processing breaks the normal distribution?)

Actually, initially I tried to visualize something like this:

df_tweets_lengths = pd.DataFrame(tweet_lengths)

fig, ax = plt.subplots()
# df_1000.text.str.lenvalue_counts().plot(ax=ax, kind='bar')
df_tweets_lengths.plot(ax=ax, kind='bar')

This outputted:

But, I was not able to interpret what this above image is trying to say? is it simply plotting lengths of each tweet one after other for all thousand tweets? If yes, then why there are those bands (four spaces with no blue bars)?

user_12061523 · Answer

You're running into two issues, one conceptual and one programmatic.

randrange generates uniform distribution, not a normal distribution. This means you should expect the output bars to have similar heights.
countplot is the categorical version of histplot, meaning it will plot counts for all values, i.e. if you have a dataset with [1, 1, 1, 2, 2, 3] it will plot three bars, {1:3, 2:2, 3:1}.

The reason you're not actually seeing the uniform distribution in your sample is that your range of x-values is large (101 values), but your sample is small (1000). If you crank up the number of samples to 1,000,000 you see the uniform shape using countplot.

import seaborn as sns
import pandas as pd 
import matplotlib.pyplot as plt
from string import ascii_lowercase, digits
from random import choice, randrange

chars = ascii_lowercase + digits
lst = [''.join(choice(chars) for _ in range(randrange(0, 101, 1))) for _ in range(1000000)]

tweet_lengths = list(map(len, lst))
sns.countplot(tweet_lengths)
plt.title('Tweet lengths')
plt.xlabel('Tweet lengths')
plt.ylabel('# of tweets')

Visualizing tweet lengths in python

1 Answers