In Flutter, when calling Navigator.pop(context) the ListView.builder flickers the the image that was just displayed, why?

I am using ListView.builder() to construct a list of images. When use taps on an image, I blow the image up and display it in a photo gallery via flutter Navigator:

Here is a quick recording of the issue: https://youtu.be/4C5MtXMSwLk

gallery-page.dart where I build the list of images
Widget build(BuildContext context) {    
    return FutureBuilder(
      future: this.futureGalleryImages,
      builder: (BuildContext context, AsyncSnapshot snapshot) {
        switch (snapshot.connectionState) {
          case ConnectionState.waiting:
            return CircularProgressIndicator();            
          default:
            if (snapshot.hasError)
              return Text('Error: ${snapshot.error}');
            else              
              return _createListView(context, snapshot);              
        }
      },
    );

Once the list view is constructor each ListTile has the open callback for onTap event listener:

void open(BuildContext context, List<GalleryImage> images, final int index){
    Navigator.push(
      context,
      MaterialPageRoute(
        builder: (context) => GalleryPhotoViewWrapper(
          galleryItems: images,
          backgroundDecoration: widget.backgroundDecoration,
          initialIndex: index,
          scrollDirection: verticalGallery ? Axis.vertical : Axis.horizontal
        )
      )
    );
  }

Then in GalleryPhotoViewWrapper I listen for the onVerticalDragUpdate event and simply navigate back to the previous screen, which is the

ListView:
    children: <Widget>[
                GestureDetector(
                    onVerticalDragUpdate: (dragUpdateDetails) {
                      Navigator.pop(context);
                    },

However whenever the Navigator.pop(context) is called inside the onVerticalDragUpdate callback the image flickers when I am taken back to the ListView. I have no idea why. I did notice however that the _createListView callback gets called when you blow up the image and then when you dismiss it. I tried caching the ListView widget in a variable and then just re-displaying that but with no success.