Error: 3 INVALID_ARGUMENT: Cannot convert an array value in an array value in Firestore admin

user_11346397 0
Viewed 1401

I am trying to save multiple tweet objects retrieved using the twitter api in my firstore db, but I am getting this error:

Error: 3 INVALID_ARGUMENT: Cannot convert an array value in an array value

Below is the sample JSON object: all tweet objects are similar to this

TweetObj: {
  "created_at": "Wed Feb 12 15:17:24 +0000 2020",
  "id": 1227612675334660000,
  "id_str": "1227612675334660099",
  "full_text": "Writing online is a superpower.\n\nHere’s a list of people who should start writing online.\n\n1. The Niche Hobbyist\n\nThe Internet rewards people with obscure interests. Writing online will help you find other like-minded hobbyists who you wouldn’t be able to find in real-life.",
  "truncated": false,
  "display_text_range": [
    0,
    274
  ],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": []
  },
  "source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>",
  "in_reply_to_status_id": null,
  "in_reply_to_status_id_str": null,
  "in_reply_to_user_id": null,
  "in_reply_to_user_id_str": null,
  "in_reply_to_screen_name": null,
  "user": {
    "id": 240049622,
    "id_str": "240049622",
    "name": "David Perell",
    "screen_name": "david_perell",
    "location": "New York",
    "description": "\"The Writing Guy\" | I tweet about business, online learning, and Internet writing | My Podcast:  | My writing school: ,
    "url": "",
    "entities": {
      "url": {
        "urls": [
          {
            "url": ,
            "expanded_url": "http://www.perell.com/",
            "display_url": "perell.com",
            "indices": [
              0,
              23
            ]
          }
        ]
      },
      "description": {
        "urls": [
          {
            "url": "",
            "expanded_url": "http://Perell.com/podcast",
            "display_url": "Perell.com/podcast",
            "indices": [
              96,
              119
            ]
          },
          {
            "url": "",
            "expanded_url": "http://writeofpassage.school",
            "display_url": "writeofpassage.school",
            "indices": [
              141,
              164
            ]
          }
        ]
      }
    },
    "protected": false,
    "followers_count": 89464,
    "friends_count": 763,
    "listed_count": 1539,
    "created_at": "Wed Jan 19 01:41:27 +0000 2011",
    "favourites_count": 51075,
    "utc_offset": null,
    "time_zone": null,
    "geo_enabled": true,
    "verified": false,
    "statuses_count": 21700,
    "lang": null,
    "contributors_enabled": false,
    "is_translator": false,
    "is_translation_enabled": false,
    "profile_background_color": "C0DEED",
    "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_tile": false,
    "profile_image_url": "http://pbs.twimg.com/profile_images/913816689892044800/rF7O3Bfz_normal.jpg",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/913816689892044800/rF7O3Bfz_normal.jpg",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/240049622/1543613054",
    "profile_link_color": "129CE6",
    "profile_sidebar_border_color": "C0DEED",
    "profile_sidebar_fill_color": "DDEEF6",
    "profile_text_color": "333333",
    "profile_use_background_image": true,
    "has_extended_profile": true,
    "default_profile": false,
    "default_profile_image": false,
    "following": true,
    "follow_request_sent": false,
    "notifications": true,
    "translator_type": "none"
  },
  "geo": null,
  "coordinates": null,
  "place": {
    "id": "01a9a39529b27f36",
    "url": "https://api.twitter.com/1.1/geo/id/01a9a39529b27f36.json",
    "place_type": "city",
    "name": "Manhattan",
    "full_name": "Manhattan, NY",
    "country_code": "US",
    "country": "United States",
    "contained_within": [],
    "bounding_box": {
      "type": "Polygon",
      "coordinates": [
        [
          [
            -74.026675,
            40.683935
          ],
          [
            -73.910408,
            40.683935
          ],
          [
            -73.910408,
            40.877483
          ],
          [
            -74.026675,
            40.877483
          ]
        ]
      ]
    },
    "attributes": {}
  },
  "contributors": null,
  "is_quote_status": false,
  "retweet_count": 357,
  "favorite_count": 2145,
  "favorited": false,
  "retweeted": false,
  "lang": "en"
}

Above is the exact object that is giving the error In the above snippet, the argument threads is an array containing multiple tweet objects just like the one described earlier.

Help would be very much appreciated.

google-cloud-firestore firebase arrays javascript
4 Answers

You could try to transparently serialize/unserialize your Tweet in order to:

  • convert all arrays to objects before saving in Firestore
  • convert all objects which keys are numeric-ish to arrays after retrieving a record from firestore

Depending on which data is supposed to be indexed and queried, you could also use JSON.stringify() on some parts of the record which structure is not supported by Firestore.

user_59808660

I got the exact same error from Firestore admin, while handling complex JSON objects like yours.

The problem came down to the fact that some fields contain nested arrays. Triple-nested in your case, like for your bounding_box.coordinates field.

I my case, I flattened the JSON object structure (transforming nested arrays into simple vectors) before pushing it to Firestore, but you could also "stringify" the multi-dimensional arrays (using JSON.stringify) and parse it back (JSON.parse) when retrieving these specific fields.

user_137148700

I would like to share another approach, which I have just implemented for a very similar use case to yours.

I believe this different approach will provide a systemic/robust solution to your problem (as it just did for me).

Due to the nature of incoming streaming users data, we are never sure of the actual dimension of these occasional nested arrays throwing errors once we call the Firestore API.

A very robust solution, which I also find elegant, is to use popular "flattening" packages from the npm Community. I implemented flat and it works very well on streaming JSON objects containing random-dimensions nested arrays located at random depth within the JSON graphs.

Trying to handle these nested arrays with pseudo-manual serialisation techniques (as mentioned in other solutions) is difficult to test for production code and can be failure-prone.

"Flattening" the incoming JSON object is a systemic way to make sure the Firebase API calls will go smooth, while not affecting data integrity.

After installing the flat package from npm (or any other similar flattening packages from the Community), you can just follow the 3 steps below:

  1. call the flatten method on the JSON object of interest (no magic here: flatten just creates new JSON keys, one-level deep only, with smart key numbering and corresponding values for different depths in the tree - fairly straightforward but effective technique)
  2. store the resulting flatten object in Firestore
  3. read the Firestore entry when needed and call the unflatten method to get the original JSON object back for further processing.

In a nutshell, what flatten does:

Flattens the object - it'll return an object one level deep, regardless of how nested the original object was.

Source: Github flat page

Note that I tested this approach in both Node.js and frontend - it works nicely and avoid going through cumbersome serialization tricks.

user_137148700

In My case, I added the list of list.

BEFORE

var array = []

'items': [array]

AFTER

var array = []

'items': array
user_88223370
Related