Adding Views Between Items in GridView

I have a scrollable GridView where the items have a decent amount of empty space between them. for some of the objects in the grid, I'd like to add arrow images leading from one to the other

For reference to what I mean, think of the World of Warcraft talent trees (http://calculators.iradei.eu/talents/mage). You'll see some of the talents have arrows leading from one item to the next.

The layout I'm dealing with just contains a gridview in a framelayout, nothing else

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout 
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical" 
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <GridView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:numColumns="4"
        android:scrollbars="none"
        android:id="@+id/talent_tree_grid_view">
    </GridView>

</FrameLayout>

I'm trying to add these ImageViews in code, as I don't think there is any way I could add this logic into the adapter for the GridView.

I've tried adding them as children of the parent FrameLayout, based on the x and y coords of the GridView items, and this works, but when scrolling the GridView, the arrows do not also scroll

I've also tried adding them directly as children of the GridView, but it does not support this behavior (it throws java.lang.UnsupportedOperationException)

Are there any ideas as to how I could add these imageviews between the gridview items?