Framerate lag spikes from network requests on main thread

user_5495881 0
Viewed 181

I'm currently working on a project to power NPC conversations using OpenAI's GPT3. Each NPC sends a POST request to my server when it's their time to speak in a conversation, which returns an audio file that the NPC plays through its local AudioSource.

Video here: https://www.youtube.com/watch?v=pygM6yDE9hI

One issue that I'm experiencing is jitter and short periods of lag.

enter image description here

I'm a novice when it comes to Unity, but based on the profiler I believe the culprits are the Coroutines handling the network requests.

Below is my code for this functionality:

     // Update is called once per frame
 void Update()
 {
     int index = -1;
     if (conversation != null)
     {
         if (!conversation.processing)
         {
             if (conversation.currentSpeaker.Equals(this.id))
             {
                 Debug.Log(this.id + " getting response");
                 conversation.processing = true;
                 StartCoroutine(getResponse(conversation));
             }
         }
     }
     else if (Datastore.Instance.id2conversation.TryGetValue(id, out index))
     {
         conversation = Datastore.Instance.conversations[index];
     }
 }

 IEnumerator getResponse(Conversation conversation)
 {
     WWWForm form = new WWWForm();
     form.AddField("id", this.id);
     var www = UnityWebRequest.Post("http://" + Datastore.Instance.host + ":3000/generate", form);

     yield return www.SendWebRequest();

     if (interrupted) yield break;

     if (www.isNetworkError)
     {
         Debug.Log(www.error);
     }
     else
     {
         if (www.GetResponseHeaders().Count > 0)
         {
             var jsonData = JSON.Parse(www.downloadHandler.text);

             string stringData = jsonData["audioContent"]["data"].ToString();
             byte[] rawdata = AudioHelpers.ConvertToByteStream(stringData);

             AudioClip clip = AudioHelpers.ConvertToAudioClip(rawdata);

             this.audioSource.clip = clip;
             this.audioSource.Play();

             this.animator.SetBool(this.talkingBoolHash, true);

             Debug.Log("Response Recieved");

             yield return new WaitForSeconds(clip.length);

             if (interrupted) yield break;

             this.conversation.currentSpeaker = jsonData["nextSpeaker"].ToString().Replace("\"", "");
             this.conversation.processing = false;
             this.animator.SetBool(this.talkingBoolHash, false);
         }
     }
 }

How can I improve the performance of this code to remove the periods of decreased framerate?

Is it possible to move this code to another thread via Unity Jobs?

Any help would be greatly appreciated.

Edit: Deep Profiler: Deep Profiler View

Implemented threading for JSON parsing. New code:

// Update is called once per frame
    void Update()
    {
        int index = -1;
        if (conversation != null)
        {
            if (!conversation.processing)
            {
                if (conversation.currentSpeaker.Equals(this.id))
                {
                    Debug.Log(this.id + " getting response");
                    conversation.processing = true;
                    StartCoroutine(getResponse(conversation));
                }
            }
        }
        else if (Datastore.Instance.id2conversation.TryGetValue(id, out index))
        {
            conversation = Datastore.Instance.conversations[index];
        }

    }

    Task<(byte[], string)> ParseAudioData(string rawJson)
    {
        try
        {
            return Task.Run(() => {
                var jsonData = JSON.Parse(rawJson);
                string stringData = jsonData["audioContent"]["data"].ToString();
                byte[] rawdata = AudioHelpers.ConvertToByteStream(stringData);
                string nextSpeaker = jsonData["nextSpeaker"].ToString().Replace("\"", "");
                return Task.FromResult((rawdata, nextSpeaker));
        });
    }
    catch(Exception e)
    {
        Debug.LogException(e);
        throw;
    }
}

    IEnumerator getResponse(Conversation conversation)
    {
        WWWForm form = new WWWForm();
        form.AddField("id", this.id);

        var www = UnityWebRequest.Post("http://" + Datastore.Instance.host + ":3000/generate", form);

        yield return www.SendWebRequest();

        if (interrupted) yield break;

        if (www.isNetworkError)
        {
            Debug.Log(www.error);
        }
        else
        {
            if (www.GetResponseHeaders().Count > 0)
            {
                Task<(byte[], string)> t = ParseAudioData(www.downloadHandler.text);

                yield return t;

                AudioClip clip = AudioHelpers.ConvertToAudioClip(t.Result.Item1);

                this.audioSource.clip = clip;
                this.audioSource.Play();

                this.animator.SetBool(this.talkingBoolHash, true);

                Debug.Log("Response Recieved");

                yield return new WaitForSeconds(clip.length);

                if (interrupted) yield break;

                this.conversation.currentSpeaker = t.Result.Item2;
                this.conversation.processing = false;
                this.animator.SetBool(this.talkingBoolHash, false);
            }
        }
    }

Final Edit: Got it working with help from the answer below!

My final code:

Task<JSONNode> ParseJsonData(string rawJson)
{
    try
    {
        return Task.Run(() =>
        {
            JSONNode jsonData = JSON.Parse(rawJson);
            return jsonData;
        });
    }
    catch (Exception e)
    {
        Debug.LogException(e);
        throw;
    }
}

Task<(byte[], string)> ParseAudioData(JSONNode jsonData)
{
    try
    {
        return Task.Run(() => {
            string stringData = jsonData["audioContent"]["data"].ToString();
            byte[] rawdata = AudioHelpers.ConvertToByteStream(stringData);
            string nextSpeaker = jsonData["nextSpeaker"].ToString().Replace("\"", "");
            return (rawdata, nextSpeaker);
        });
    }
    catch (Exception e)
    {
        Debug.LogException(e);
        throw;
    }
}
IEnumerator PlayDialog(AudioClip clip, string nextSpeaker)
{
    this.audioSource.clip = clip;
    this.audioSource.Play();
    this.animator.SetBool(this.talkingBoolHash, true);

    yield return new WaitForSeconds(clip.length);

    if (interrupted) yield break;

    this.conversation.currentSpeaker = nextSpeaker;
    this.conversation.processing = false;
    this.animator.SetBool(this.talkingBoolHash, false);
}

IEnumerator getResponse(Conversation conversation)
{
    WWWForm form = new WWWForm();
    form.AddField("id", this.id);
    var www = UnityWebRequest.Post("http://" + Datastore.Instance.host + ":3000/generate", form);

    yield return www.SendWebRequest();

    if (interrupted) yield break;

    if (www.isNetworkError)
    {
        Debug.Log(www.error);
    }
    else
    {
        if (www.GetResponseHeaders().Count > 0)
        {
            ParseJsonData(www.downloadHandler.text).ContinueWith((jsonData) => {
                ParseAudioData(jsonData.Result).ContinueWith((t) =>
                {
                    // https://github.com/PimDeWitte/UnityMainThreadDispatcher
                    UnityMainThreadDispatcher.Instance().Enqueue(() =>
                    {
                        AudioClip clip = AudioHelpers.ConvertToAudioClip(t.Result.Item1);
                        StartCoroutine(PlayDialog(clip, t.Result.Item2));
                        Debug.Log("Response Recieved");
                    });
                });
            });
           
        }
    }
}
unity3d task optimization c#
1 Answers

I'd say that probably the lag spikes are coming from the JSON parsing and transforming it into an AudioClip. You can probably do this on a different thread:

Task<byte[]> ParseJsonData (string rawJson)
{
    try
    {
        return Task.Run(() =>
        {
            jsonData = JSON.Parse(rawJson);
            string stringData = jsonData["audioContent"]["data"].ToString();
            return AudioHelpers.ConvertToByteStream(stringData);
        });
    }
    catch (Exception e)
    {
        UnityEngine.Debug.LogException(e);
        throw;
    }
}

And call it like this:

IEnumerator getResponse(Conversation conversation)
{
    WWWForm form = new WWWForm();
    form.AddField("id", this.id);
    var www = UnityWebRequest.Post("http://" + Datastore.Instance.host + ":3000/generate", form);

    yield return www.SendWebRequest();

    if (interrupted) yield break;

    if (www.isNetworkError)
    {
        Debug.Log(www.error);
    }
    else
    {
        if (www.GetResponseHeaders().Count > 0)
        {
            ParseJsonData(rawJson)
                .ContinueWith(ParseAudioData);
            ParseAudioData(www.downloadHandler.text).ContinueWith((rawData) =>
            {
                // https://github.com/PimDeWitte/UnityMainThreadDispatcher
                UnityMainThreadDispatcher.Instance.Enqueue(() =>
                {
                    AudioClip clip = AudioHelpers.ConvertToAudioClip(rawData);
                    StartCoroutine(PlayDialog(clip));
                    Debug.Log("Response Recieved");
                });
        }
    }
}

IEnumerator PlayDialog (AudioClip clip)
{
    this.audioSource.clip = clip;
    this.audioSource.Play();
    this.animator.SetBool(this.talkingBoolHash, true);
    yield return new WaitForSeconds(clip.length);
    if (interrupted) yield break;
    this.conversation.currentSpeaker = jsonData["nextSpeaker"].ToString().Replace("\"", "");
    this.conversation.processing = false;
    this.animator.SetBool(this.talkingBoolHash, false);
}

This will send the operations to a new thread and continue with your code when the operations end. If the lag spikes are coming from parsing the json this should help it. Take care when running code from other threads, since you cannot access most of Unity functionality and it's all async.

Edit: Included some utils for running code on the MainThread, because as others pointed out, this wouldn't work without fiddling with synchronization context.

Also, I'd recommend for you to try to separate your methods with single responsibilities. As of now your coroutine is downloading stuff, parsing it, playing audio and updating the dialog status. This is quite a lot a for a single function.

Your JSON should also be a structured class, it makes little sense to parse it and still access stuff by its string hash.

If this still doesn't solve your issue, you might need to dive a bit deeper into the profiler and check exactly what is causing it spend so much time in the main thread.

user_71750920
Related