AWS amplify/dynamo/appsync - how to sync data locally

user_574519 0
Viewed 131

All I want to do is essentially take the exact dynamdb tables with their data that exist in a remote instance ( e.g. amplify staging environment/api) and import those locally.

I looked at datasync but that seemed to be FE only. I want to take the exact data from staging and sync that data to my local amplify instance - is this even possible? I can't find any information that is helping right now.

Very used to using mongo/postgres etc. and literally being able to take a DB dump and just import that...I may be missing something here?

aws-appsync amplify amazon-dynamodb aws-amplify
3 Answers

I'm not too well versed on local Amplify instances, but for DynamoDB there is a product which you can use locally called DynamoDB Local.

Full list of download instructions for DynamoDB Local are available here: Downloading And Running DynamoDB Local

If you have docker installed in your local machine, you can easily download and start the DynamoDB Local service with a few commands:

Download

docker pull amazon/dynamodb-local

Run

docker run --name dynamodb -p 8000:8000 -d amazon/dynamodb-local -jar DB.jar

This will allow you to avail of 90% of DynamoDB features locally. However, migrating data from DynamoDB Web Service to DynamoDB local is not something that is provided out of the box. For that, you would need to create a small script which you run locally, read data from your existing table and write to your local instance.

An example of reading from one table and writing to a second can be found in the docs here: Copy Amazon Dynamodb Tables Across Accounts

One change you will have make is manually setting the endpoint_url for DynamoDB Local:

dynamodb_client = boto3.Session(
        aws_access_key_id=args['AWS_ACCESS_KEY_ID'],
        aws_secret_access_key=args['AWS_SECRET_ACCESS_KEY'],
        aws_session_token=args['TEMPORARY_SESSION_TOKEN'],
        endpoint_url='YOUR_DDB_LOCAL_ENDPOINT_URL'
    ).client('dynamodb')
user_79096760

You have to build a custom script that reads from the online DynamoDB and then populate the Local DynamoDB. I found the docker image be just perfect to have an instance, make sure to give the jar file name to prevent the image to be ephemeral and have persistence of data.

Sort of macro instructions:

Download Docker Desktop (if you want)

Start docker desktop and in a terminal ask for Dynamo DB official image:

https://hub.docker.com/r/amazon/dynamodb-local/

docker pull amazon/dynamodb-local

And then run the docker container:

docker run --name dynamodb -p 8000:8000 -d amazon/dynamodb-local -jar DB.jar

Now you can start a python script that get the data from the online DB and copy in the local dynamoDB, as in official docs:

https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-a-custom-implementation.html

Working out the connections to the local container (localhost:8000) you shall be able to copy all the data.

user_82795210

How about using dynamodump

Download the data from AWS to your local machine:

python dynamodump.py -m backup -r REGION_NAME -s TABLE_NAME

Then import to Local DynamoDB:

dynamodump -m restore -r local -s SOURCE_TABLE_NAME -d LOCAL_TABLE_NAME --host localhost --port 8000
user_13848390
Related