org_data_transfer

Building an export:

1. turn on extract database:

   assume-role -e p -- aws rds start-db-instance --db-instance-identifier mpdx-api-extract-prod

2. reset transfer database

    run-locally -e s \
      --docker-args "\
        -e DATADOG_TRACE=false\
        -e DB_ENV_POSTGRESQL_DB=mpdx_extract\
        -e DB_ENV_POSTGRESQL_USER=mpdx_extract_admin\
        -e DB_ENV_POSTGRESQL_PASS=REDACTED\
        -e DB_PORT_5432_TCP_ADDR=mpdx-api-extract-prod.ctzggtk79wff.us-east-1.rds.amazonaws.com" \
      -p false bash

3. start data transfer:

   • prod_console

   • Organization::Restorer.restore("05df083a-6836-4148-a15a-ae6d72d00749")

4. wait for sidekiq batch to finish

5. Dump transfer database to compressed file:

   pg_dump -Fc \
      -h mpdx-api-extract-prod.ctzggtk79wff.us-east-1.rds.amazonaws.com \
      -f ./transfer-test1.dmp \
      -U mpdx_extract_admin \
      -d mpdx_extract

6. Transfer all s3 files to transfer bucket:

   Organization::Restorer.new("05df083a-6836-4148-a15a-ae6d72d00749").restore_attachments(progress_bar: true)

7. Sync all files from transfer bucket to a local directory:

   assume-role -e p -- aws s3 sync s3://mpdx-transmit-production docker_data/s3

8. Zip dir:

   tar -cf - docker_data/s3 | xz -1 -c - -> docker_data/s3.tar.xz

9. turn off extract database:

   assume-role -e p -- aws rds stop-db-instance --db-instance-identifier mpdx-api-extract-prod

Run MPDX locally with export

1. pg restore:

   • git checkout public-dockerfile

   • Start the api database server: docker-compose up -d db

   • Create database: createdb -h localhost -p 54322 -U postgres mpdx_extract

   • Restore data to database:

     pg_restore -c -d mpdx_extract -h localhost -p 54322 -U postgres --no-privileges --no-owner ./transfer-test.dmp

2. build mpdx_api docker image:

   • bin/public_build.sh (requires SIDEKIQ_CREDS to be set on host env)

3. run mpdx api:

   • docker-compose up

4. run mpdx_web:

   • cd mpdx_web

   • npm install

   • API_URL=http://localhost:50000/api/v2 OAUTH_URL=http://auth.lvh.me:50000/auth/user npm run start

5. open http://localhost:8081