How to use the YouTube API to extract video titles and next page token?

Question

I'm using the YouTube API in order to retrieve all titles in a playlist.

I managed to retrieve the JSON text from the API and to find the correct elements this post:

Dim wc As New Net.WebClient
Dim incstr As IO.Stream = Nothing
wc.Encoding = Encoding.UTF8
incstr = wc.OpenRead("https://www.googleapis.com/youtube/v3/playlistItems?part=contentDetails%2Csnippet&maxResults=50&playlistId=PL4_Dx88dpu7cEY_cBjTZFFM1tVKF5Plsx&key=yourkey")

Using rd As New IO.StreamReader(incstr)
    RichTextBox1.Text = rd.ReadToEnd
End Using
Dim strBuf As String = (richtextbox1.text)

Dim myjobj As New Json.Linq.JObject
myjobj = Json.Linq.JObject.Parse(strBuf)

dim titleText = JObject.Parse(json)("title")("runs")(0)("text")

But, unfortunately, this code is generating on the last line an error like: System.ArgumentException: 'Can not convert Object to String.'

Also, the API gets back just max 50 elements from the playlist, so as @Jimi suggested:

If you have more that 50 elements in the playList, then you get back a nextPageToken entry in the main class object. You can use this token to query the next page, by setting the pageToken=[The Provided Token String] query tuple.

I do see at the beginning of JSON text:

"nextPageToken": "CGQQAA",
"prevPageToken": "CDIQAQ",

but don't know how to grab them.

So, since the limits is 50 songs at the time, how can I retrieve all the songs at once using next page token?

user_7444103 · Answer

The Google API uses a form of pagination when a PlayList is requested (this also applies to other types of API responses).
The maximum number of entries per response page is 50.

▶ The response format is a JSON object which contains a description of the query response, including the total items requested and the items included in the current response.

▶ If the query includes more items than the maximum per page, the nextPageToken property is set to a string value representing the Page Token, which can be used to query the next page results (you can see how this works in code, in the LoadPlaylistAsync() method of the YouTubePlayList class).
When no other pages are available, nextPageToken is set to null (Nothing).
Inspecting this Token value, we can determine whether the Playlist content is complete.

The API can be called using OAuth 2.0 with Access Key, which requires the Google APIs Client Library, or using a simple HTTTP GET with an URI composed as FormUrlEncodedContent, standard URL query tuples.

Here, I'm using the latter method, since it's quite simplified and works pretty well.
If you have the Google SDK, you can follow the example in the PlaylistItems list documentation page.

I'm using a class structure (model) to deserialize the JSON response: it's quite simpler to handle the content using standard .Net classes, where the JSON properties values have already been converted to strongly typed values in the correct format.
The helper class, YouTubePlayList, contains the class model and the methods needed to deserialize the JSON or serialize the class objects back to a JSON string.

It also performs the HTTP GET request, using a static HttpClient and handles the pagination of the API response, if needed.
The class uses the asynchronous methods that the HttpClient class provides.

▶ The class is initialized passing the access Key required to query the API.
If the Key is not yet available, follow the instructions to create one (is free) in the API Reference documentation page, which will send you to the Developer Console's API Access pane.

To load a PlayList, pass a Playlist Token to the LoadPlaylistAsync() method.
This method returns a PlayList objects that contains all the information related to the API query itself and all the resulting Items, in a List(Of PlayListEntry) objects.
All deserialization and HTTP calls tasks are handled internally.

For example, in a Button.Click handler, initialize the helper class passing the Key, call the LoadPlaylistAsync() method specifying the Token of the Playlist to download, then dispose of the YouTubePlayList class object and just use the returned class object for whatever is needed:

_{I'm using the Playlist Token you provided in a previous question}

Private Async Sub btnLoadPlaylist_Click(sender As Object, e As EventArgs) Handles btnLoadPlaylist.Click
    ' Initialize with the Google API Key
    Dim playListHelper As New YouTubePlayList("AIzaSy---------------------------------")
    ' Load the Playlist passing the playListId
    Dim playList = Await playListHelper.LoadPlaylistAsync("PL4_Dx88dpu7epfH6ybwqJpf9uL2tAl368")
    ' Dispose of the class object, to close and dispose of the HttpClient object
    playListHelper.Dispose()

    For Each item As YouTubePlayList.PlayListEntry In playList.Items
        Dim itemTitle = item.ItemContent.Title
        Dim itemChannel = item.ItemContent.ChannelTitle
        Dim videoAddress = "https://www.youtube.com/watch?v=" & item.ItemContent.ResourceId.VideoId
    Next
End Sub

Sample functionality of the class object at work:

Google PlayList downloader helper class:

Requires Json.Net (Newtonsoft.Json) 12.0.3+

Imports System
Imports System.Collections.Generic
Imports System.Globalization
Imports System.Net.Http
Imports System.Threading.Tasks
Imports Newtonsoft.Json
Imports Newtonsoft.Json.Converters

Public Class YouTubePlayList
    Implements IDisposable

    Private m_Json As String = String.Empty
    Private m_KeyToken As String = String.Empty
    Private m_APIUrl As String = "https://www.googleapis.com/youtube/v3/playlistItems?"
    Private Shared client As HttpClient = Nothing

    Public Sub New(key As String)
        Me.New(key, String.Empty)
    End Sub

    Public Sub New(key As String, json As String)
        m_KeyToken = key
        If Not String.IsNullOrEmpty(json) Then
            m_Json = json
        End If
        client = New HttpClient()
    End Sub

    Public Function Deserialize() As PlayList
        Return Deserialize(m_Json)
    End Function

    Public Function Deserialize(json As String) As PlayList
        If String.IsNullOrEmpty(json) Then Return Nothing
        Return JsonConvert.DeserializeObject(Of PlayList)(json, ConverterSettings.Settings)
    End Function

    Public Function Serialize(root As PlayList) As String
        Return JsonConvert.SerializeObject(root, ConverterSettings.Settings)
    End Function

    Friend NotInheritable Class ConverterSettings
        Public Shared ReadOnly Settings As New JsonSerializerSettings() With {
            .MetadataPropertyHandling = MetadataPropertyHandling.Ignore,
            .DateParseHandling = DateParseHandling.None,
            .Converters = {New IsoDateTimeConverter() With {
                .DateTimeStyles = DateTimeStyles.AssumeUniversal
            }}
        }
    End Class

    Public Async Function LoadPlaylistAsync(playListToken As String) As Task(Of PlayList)
        Dim resultPlayList As PlayList = Nothing
        Dim nextPageToken As String = String.Empty
        Dim listItemsRead As Integer = 0
        Dim resultsPerPage As Integer = 50

        While nextPageToken IsNot Nothing
            Dim queryUrl = Await GetRequestUrl(playListToken, nextPageToken, resultsPerPage)
            Using response = Await client.GetAsync(queryUrl)
                If response.IsSuccessStatusCode Then
                    Dim jsonResponse = Await response.Content.ReadAsStringAsync()
                    Dim playList = Deserialize(jsonResponse)

                    listItemsRead += playList.Items.Count

                    If resultPlayList Is Nothing Then
                        resultPlayList = playList
                    Else
                        resultPlayList.Items.AddRange(playList.Items)
                    End If
                    nextPageToken = playList.NextPageToken
                    resultPlayList.PageInfo.ResultsPerPage = listItemsRead
                End If
            End Using
        End While
        Return resultPlayList
    End Function

    Private Async Function GetRequestUrl(playListToken As String, nextPageToken As String, resultsPerPage As Integer) As Task(Of String)
        Dim content = New FormUrlEncodedContent(New KeyValuePair(Of String, String)() {
            New KeyValuePair(Of String, String)("part", "contentDetails,snippet,id"),
            New KeyValuePair(Of String, String)("maxResults", resultsPerPage.ToString()),
            New KeyValuePair(Of String, String)("pageToken", nextPageToken),
            New KeyValuePair(Of String, String)("playlistId", playListToken),
            New KeyValuePair(Of String, String)("key", m_KeyToken)
        })
        Return m_APIUrl & Await content.ReadAsStringAsync()
    End Function


    Public Class PlayList
        
        Public Property Kind As String
        
        Public Property Etag As String
        
        Public Property NextPageToken As String
        
        Public Property PrevPageToken As String
        
        Public Property Items As List(Of PlayListEntry)
        
        Public Property PageInfo As PageInfo
    End Class

    Public Class PlayListEntry
        
        Public Property Kind As String
        
        Public Property Etag As String
        
        Public Property Id As String
        
        Public Property ItemContent As Content
        
        Public Property ContentDetails As ContentDetails
    End Class

    Public Class ContentDetails
        
        Public Property VideoId As String
        
        Public Property VideoPublishedAt As DateTimeOffset
    End Class

    Public Class Content
        
        Public Property PublishedAt As DateTimeOffset
        
        Public Property ChannelId As String
        
        Public Property Title As String
        
        Public Property Description As String
        
        Public Property Thumbnails As Thumbnails
        
        Public Property ChannelTitle As String
        
        Public Property PlaylistId As String
        
        Public Property Position As Long
        
        Public Property ResourceId As ResourceId
    End Class

    Public Class ResourceId
        
        Public Property Kind As String
        
        Public Property VideoId As String
    End Class

    Public Class Thumbnails
        
        Public Property DefaultImage As ImageDescriptor
        
        Public Property Medium As ImageDescriptor
        
        Public Property High As ImageDescriptor
        
        Public Property Standard As ImageDescriptor
        
        Public Property MaxResolution As ImageDescriptor
    End Class

    Public Class ImageDescriptor
        
        Public Property Url As Uri
        
        Public Property Width As Integer
        
        Public Property Height As Integer
    End Class

    Public Class PageInfo
        
        Public Property TotalResults As Integer
        
        Public Property ResultsPerPage As Integer
    End Class

    Public Sub Dispose() Implements IDisposable.Dispose
        Dispose(True)
        GC.SuppressFinalize(Me)
    End Sub
    Protected Sub Dispose(disposing As Boolean)
        If disposing Then
            client?.CancelPendingRequests()
            client?.Dispose()
            client = Nothing
        End If
    End Sub
End Class

C# version:

using System.Collections.Generic;
using System.Globalization;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;

public class YouTubePlayList : IDisposable
{
    private string m_Json = string.Empty;
    private string m_KeyToken = string.Empty;
    private string m_APIUrl = "https://www.googleapis.com/youtube/v3/playlistItems?";
    private static HttpClient client = null;

    public YouTubePlayList(string key) : this(key, string.Empty) { }
    public YouTubePlayList(string key, string json)
    {
        m_KeyToken = key;
        if (!string.IsNullOrEmpty(json)) {
            m_Json = json;
        }
        client = new HttpClient();
    }

    public PlayList Deserialize() => Deserialize(m_Json);

    public PlayList Deserialize(string json)
    {
        if (string.IsNullOrEmpty(json)) return null;
        return JsonConvert.DeserializeObject(json, ConverterSettings.Settings);
    }

    public string Serialize(PlayList root) =>
        JsonConvert.SerializeObject(root, ConverterSettings.Settings);

    internal sealed class ConverterSettings
    {
        public static readonly JsonSerializerSettings Settings = new JsonSerializerSettings {
            MetadataPropertyHandling = MetadataPropertyHandling.Ignore,
            DateParseHandling = DateParseHandling.None,
            Converters = {
                new IsoDateTimeConverter { DateTimeStyles = DateTimeStyles.AssumeUniversal }
            }
        };
    }

    public async Task LoadPlaylistAsync(string playListToken)
    {
        PlayList resultPlayList = null;
        string nextPageToken = string.Empty;
        int listItemsRead = 0;
        int resultsPerPage = 50;

        while (nextPageToken != null) {
            string queryUrl = await GetRequestUrl(playListToken, nextPageToken, resultsPerPage);
            using (var response = await client.GetAsync(queryUrl)) {
                if (response.IsSuccessStatusCode) {
                    string jsonResponse = await response.Content.ReadAsStringAsync();
                    var playList = Deserialize(jsonResponse);
                    listItemsRead += playList.Items.Count;
                    if (resultPlayList == null) {
                        resultPlayList = playList;
                    }
                    else {
                        resultPlayList.Items.AddRange(playList.Items);
                    }
                    nextPageToken = playList.NextPageToken;
                    resultPlayList.PageInfo.ResultsPerPage = listItemsRead;
                }
            }
        }
        return resultPlayList;
    }
    private async Task GetRequestUrl(string playListToken, string nextPageToken, int resultsPerPage)
    {
        var content = new FormUrlEncodedContent(new KeyValuePair[] {
            new KeyValuePair("part", "contentDetails,snippet,id"),
            new KeyValuePair("maxResults", resultsPerPage.ToString()),
            new KeyValuePair("pageToken", nextPageToken),
            new KeyValuePair("playlistId", playListToken),
            new KeyValuePair("key", m_KeyToken)
        });
        return m_APIUrl + await content.ReadAsStringAsync();
    }

    public class PlayList
    {
        public string Kind { get; set; }
        public string Etag { get; set; }
        public string NextPageToken { get; set; }
        public string PrevPageToken { get; set; }
        public List Items { get; set; }
        public PageInfo PageInfo { get; set; }
    }

    public class PlayListEntry
    {
        public string Kind { get; set; }
        public string Etag { get; set; }
        public string Id { get; set; }
        [JsonProperty("snippet")]
        public Content ItemContent { get; set; }
        public ContentDetails ContentDetails { get; set; }
    }
    public class ContentDetails
    {
        public string VideoId { get; set; }
        public DateTimeOffset VideoPublishedAt { get; set; }
    }

    public class Content
    {
        public DateTimeOffset PublishedAt { get; set; }
        public string ChannelId { get; set; }
        public string Title { get; set; }
        public string Description { get; set; }
        public Thumbnails Thumbnails { get; set; }
        public string ChannelTitle { get; set; }
        public string PlaylistId { get; set; }
        public long Position { get; set; }
        public ResourceId ResourceId { get; set; }
    }

    public class ResourceId
    {
        public string Kind { get; set; }
        public string VideoId { get; set; }
    }

    public class Thumbnails
    {
        [JsonProperty("default")]
        public ImageDescriptor DefaultImage { get; set; }
        public ImageDescriptor Medium { get; set; }
        public ImageDescriptor High { get; set; }
        [JsonProperty("standard", NullValueHandling = NullValueHandling.Ignore)]
        public ImageDescriptor Standard { get; set; }
        [JsonProperty("maxres", NullValueHandling = NullValueHandling.Ignore)]
        public ImageDescriptor MaxResolution { get; set; }
    }

    public class ImageDescriptor
    {
        public Uri Url { get; set; }
        public int Width { get; set; }
        public int Height { get; set; }
    }

    public class PageInfo
    {
        public int TotalResults { get; set; }
        public int ResultsPerPage { get; set; }
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected void Dispose(bool disposing)
    {
        if (disposing) {
            client?.CancelPendingRequests();
            client?.Dispose();
            client = null;
        }
    }
}

How to use the YouTube API to extract video titles and next page token?

1 Answers