Exports from Google Cloud Datastore set app id on key properties to b~<app_id>. This makes imports not useable

Thanks for reporting. This is a bug in the emulator, we are working on a fix. The string "b~" is the full app id in the cloud datastore. You should be able to get consistent data by appending this "b~" in the import request:

curl -X POST localhost:<emulator_port>/v1/projects/b~<app-id>:import -H 'Content-Type: application/json' -d '{"input_url":"<local_folder>/<file>.overall_export_metadata"}'

I did the following workaround to fix the key properties after importing the datastore locally. Note that this does not solve the Google export bug. We will need to wait for Google to do that but meanwhile the following worked for me. I created a script which updates all key properties and I ran that script in the local Interactive Console. Let Address be the data object we want to update and Person be the data object the key property references to. Let's assume that dm is your data model class and person is the referenced property. Then you can do something like this in Python:

from google.appengine.ext import ndb

addresses = dm.Address.query().fetch(None)

for addr in addresses:
    if addr.person:
        addr.person = ndb.Key('Person', addr.person.id())

addrKeys = ndb.put_multi(addresses)

print 'finished processing {}'.format(len(addrKeys))

For large datasets you need to do this in batches by querying your dataset using pagination and a cursor. i.e.

DATA_PAGE = 10000

counter = memcache.get('counter')
if not counter:
    counter = 0

adr_cursor = memcache.get('adr_cursor')
if adr_cursor:
    print('got cursor from mem')
    addresses, next_cursor, more = dm.Address.query().fetch_page(DATA_PAGE, start_cursor = adr_cursor)
else:
    print('no cursor avail')
    addresses, next_cursor, more = dm.Address.query().fetch_page(DATA_PAGE)

# Iterate over the results
for addr in addresses:
    if addr.person:
        addr.person = ndb.Key('Person', addr.person.id())
        counter += 1

ndb.put_multi(addresses)

memcache.set(key="counter", value=counter)

if more and next_cursor:
    memcache.set(key="adr_cursor", value=next_cursor)
    print 'processed {} more'.format(counter)
else:
    print 'finished processing {}'.format(counter) 

You need to press execute in the interactive console until all records are processed. I modified and executed this script for all my affected datasets and properties.

Posting my quick-and-ugly code for cleaning up nested KeyProperties for running once after importing the datastore.

The way I've been handling this is to:

1. Import with no prefix on the app id, ex curl -X POST localhost:<port>/v1/projects/<app-id>:import
2. Run with no prefix on the app id, ex -A <app-id>
3. Run a handler once to change KeyProperties to the correct dev app id, ex "dev~" + app_identity.get_application_id().

This code abstracts some of the work, but you still have to identify which of your entities and properties need correcting. This project is still on python 2.7 using webapp2, so it will need some tweaking for newer projects. Also, I highly recommend adding some logging as you change things, because this of course takes a long time for large datastores.

### For changing all of the KeyProperty references inside of an imported
### NDB datastore from production to dev server.
class FixLocalImportHandler(BaseHandler):
  devAppID = "dev~" + app_identity.get_application_id()

  @staticmethod
  def fixAKeyProperty(keyProp):
    return ndb.Key(flat = keyProp.flat())

  @staticmethod
  def fixAModel(modelClass, orderField, attrString, lambdaItem):    
    chunkSize = 500
    chunkOffset = 0
    hadAny = True

    while hadAny:
      hadAny = False

      for item in modelClass.query().order(orderField).fetch(offset=chunkOffset, limit=chunkSize):
        hadAny = True
        needsPut = False

        oldAttr = getattr(item, attrString)

        # attribute could be a repeated propery, in which case we get a list
        # if there's any item with the bad app id, then needs fixing
        if isinstance(oldAttr, list):
          if len(filter(lambda item: item.app() != FixLocalImportHandler.devAppID, oldAttr)) > 0:
            newAttrList = [ndb.Key(flat=oldAttrLi.flat()) for oldAttrLi in oldAttr]
            setattr(item, attrString, newAttrList)
            needsPut = True
        else: # just a single item prop
          if oldAttr.app() != FixLocalImportHandler.devAppID:
            newAttr = ndb.Key(flat=oldAttr.flat())
            setattr(item, attrString, newAttr)
            needsPut = True

        # hook for additional work that needs to be done for this item (like update a search index)
        if(lambdaItem is not None):
          lambdaItem(item)

        if needsPut:
          item.put()

      # WATCH INDENT HERE for outer forloop. lol
      chunkOffset = chunkOffset + chunkSize

  def get(self):
    # This code is only for correcting app ids on local test environments
    if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
      self.display_message("DO NOT ACCIDENTALLY CORRECT LOCAL IMPORT")
      return

    self.response.headers['Content-Type'] = 'text/plain'

    # Call for each entity and KeyProperty field that needs fixing.
    FixLocalImportHandler.fixAModel(AModelClass, AModelClass.sortByField, "nameOfFieldToFix", None)
    FixLocalImportHandler.fixAModel(AModelClass, AModelClass.sortByField, "anotherNameOfFieldToFix", None)
    FixLocalImportHandler.fixAModel(AModelClass2, AModelClass2.sortByField, "nameOfFieldToFix", None)
    FixLocalImportHandler.fixAModel(AModelClass2, AModelClass2.sortByField, "anotherNameOfFieldToFix", None)