I am trying to block Youtube connection with this command but it return an error msg: getsockopt failed strangely: Operation not permitted.
iptables -I OUTPUT -m string --string "youtube.com" --algo bm --to 65535 -j DROP
Any wrong with the command?
From what iptables -m string -h at the bottom tells me, this iptables extension just matches strings within packets.
After the initial DNS query the string 'youtube' is very likely not present in your traffic (or people will just use 'youtu.be' or something), rendering that approach at least partially moot.
This command however will give you a list of all networks being advertised by youtube, which is essentially what the content of your blocklist should be:
whois -h whois.radb.net -- '-i origin AS15169' | grep ^route
Set up some nightly cron to get these and repopulate your iptables ruleset accordingly.