I have a df full of urls and image urls within the parent url. My webscraping script downloads all of the image urls. As you can see image urls 2-8 are the same picture but different dimensions/sizes. I am looking to keep one of the images so 6 versions of the photo are not downloaded. It doesnt matter which size is kept as long as there is only one of them.
This is also a part of a larger dataset and performed weekly. There are about 1500 image urls per week so the structure of the code has to be generic/standardized. I was thinking about doing the max length of the urls grouped by the article url, and matching starts with and ends with phrases but I feel that that isnt the best way and can lead to missing some pictures on sites with shorter urls.
structure(list(image_url = c("https://cloudfront-eu-central-1.images.arcpublishing.com/rtl/HDS4PPYVRD7SEHYOS6N6TFAROU.jpg",
"https://ais-akamai.rtl.de/masters/1769218/1024x0/lewis-hamilton-trennen-nur-noch-acht-punkte-von-max-verstappen.jpg",
"https://ais-akamai.rtl.de/masters/1769218/290x0/lewis-hamilton-trennen-nur-noch-acht-punkte-von-max-verstappen.jpg",
"https://ais-akamai.rtl.de/masters/1769218/345x0/lewis-hamilton-trennen-nur-noch-acht-punkte-von-max-verstappen.jpg",
"https://ais-akamai.rtl.de/masters/1769218/395x0/lewis-hamilton-trennen-nur-noch-acht-punkte-von-max-verstappen.jpg",
"https://ais-akamai.rtl.de/masters/1769218/728x0/lewis-hamilton-trennen-nur-noch-acht-punkte-von-max-verstappen.jpg",
"https://ais-akamai.rtl.de/masters/1769218/399x0/lewis-hamilton-trennen-nur-noch-acht-punkte-von-max-verstappen.jpg",
"https://ais-akamai.rtl.de/masters/1769218/527x0/lewis-hamilton-trennen-nur-noch-acht-punkte-von-max-verstappen.jpg"
), URL = c("https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html",
"https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html",
"https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html",
"https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html",
"https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html",
"https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html",
"https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html",
"https://www.rtl.de/cms/heisser-wuestenendspurt-live-bei-rtl-faellt-am-sonntag-die-formel-1-entscheidung-4875243.html"
)), row.names = 1:8, class = "data.frame")