CartoDB location sync

Purpose

PRP relies on real geo data for many data models and business logic in place. UNICEF has been utilizing CartoDB service as a single source of truth when it come to Location and geographic polygon data. Similarly done in PMP, PRP also has an identical copy of CartoDB sync Celery task with Django admin interface on Location table.

Requirements

CartoDB sync process is a Celery task, therefore it requires a Celery worker running in the background.

Local environment setup steps

Docker-compose environment spins up Celery workers for you. Make sure your docker-compose is up: fab up If your docker-compose is running, that means Celery worker should be also active.

Docker cloud environment steps

Make sure Docker cloud has stackfile definitions for celery containers from local docker-compose.yml: beater-prp, celerycam-prp, worker-prp, flower-prp, prp-redis

beater-prp:
  command: python manage.py celery beat --loglevel=debug
  deployment_strategy: high_availability
  environment:
    - CELERY_VISIBILITY_TIMEOUT=18000
    - DJANGO_ALLOWED_HOST=
    - DJANGO_DEBUG=true
    - DOMAIN_NAME=
    - EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
    - EMAIL_HOST_PASSWORD=
    - EMAIL_HOST_USER=apikey
    - ENV=dev
    - EXCLUDE_BASIC_AUTH=1
    - PMP_API_PASSWORD=
    - PMP_API_USER=
    - POSTGRES_DB=postgres
    - POSTGRES_HOST=prp-db
    - POSTGRES_PASSWORD=
    - POSTGRES_USER=postgres
    - 'REDIS_URL=redis://prp-redis:6379/'
    - SECRET_KEY=
    - AWS_S3_ACCESS_KEY_ID=ID
    - AWS_S3_SECRET_ACCESS_KEY=SECRET
    - AWS_STORAGE_BUCKET_NAME=bucket
    - AWS_S3_REGION_NAME=eu-central-1
    - PRP_FRONTEND_HOST=localhost:8082
    - AZURE_B2C_CLIENT_ID=ID
    - AZURE_B2C_CLIENT_SECRET=SECRET
    - AZURE_B2C_POLICY_NAME=POLICY
  image: 'unicef/etools-prp:develop'
  restart: always
  tags:
    - prp
    - staging
    - unicef
  target_num_containers: 1
celerycam-prp:
  autoredeploy: true
  command: python manage.py celerycam
  deployment_strategy: high_availability
  environment:
    - CELERY_VISIBILITY_TIMEOUT=18000
    - DJANGO_ALLOWED_HOST=
    - DJANGO_DEBUG=true
    - DOMAIN_NAME=
    - EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
    - EMAIL_HOST_PASSWORD=
    - EMAIL_HOST_USER=apikey
    - ENV=dev
    - EXCLUDE_BASIC_AUTH=1
    - PMP_API_PASSWORD=
    - PMP_API_USER=
    - POSTGRES_DB=postgres
    - POSTGRES_HOST=prp-db
    - POSTGRES_PASSWORD=
    - POSTGRES_USER=postgres
    - 'REDIS_URL=redis://prp-redis:6379/'
    - SECRET_KEY=
    - AWS_S3_ACCESS_KEY_ID=ID
    - AWS_S3_SECRET_ACCESS_KEY=SECRET
    - AWS_STORAGE_BUCKET_NAME=bucket
    - AWS_S3_REGION_NAME=eu-central-1
    - PRP_FRONTEND_HOST=localhost:8082
    - AZURE_B2C_CLIENT_ID=ID
    - AZURE_B2C_CLIENT_SECRET=SECRET
    - AZURE_B2C_POLICY_NAME=POLICY
  ldsfskdljfsdf98483u4530495iofjweiosjfsdjfLKJSLDFJ09
  image: 'unicef/etools-prp:develop'
  restart: always
  tags:
    - prp
    - staging
    - unicef
  target_num_containers: 1
flower-prp:
  autoredeploy: true
  command: 'flower --address=0.0.0.0 --port=8082 --broker=redis://prp-redis:6379/0 -l DEBUG --auto_refresh=False --debug=True --autoreload=False --url_prefix=flower'
  environment:
    - CELERY_VISIBILITY_TIMEOUT=18000
    - DJANGO_ALLOWED_HOST=
    - DJANGO_DEBUG=true
    - DOMAIN_NAME=
    - EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
    - EMAIL_HOST_PASSWORD=
    - EMAIL_HOST_USER=apikey
    - ENV=dev
    - EXCLUDE_BASIC_AUTH=1
    - "EXTRA_SETTINGS=reqrep ^([^\\ :]*)\\ /flower/(.*)     \\1\\ /\\2"
    - PMP_API_PASSWORD=
    - PMP_API_USER=
    - POSTGRES_DB=postgres
    - POSTGRES_HOST=prp-db
    - POSTGRES_PASSWORD=
    - POSTGRES_USER=postgres
    - 'REDIS_URL=redis://prp-redis:6379/0'
    - SECRET_KEY=
    - AWS_S3_ACCESS_KEY_ID=ID
    - AWS_S3_SECRET_ACCESS_KEY=SECRET
    - AWS_STORAGE_BUCKET_NAME=bucket
    - AWS_S3_REGION_NAME=eu-central-1
    - PRP_FRONTEND_HOST=localhost:8082
    - AZURE_B2C_CLIENT_ID=ID
    - AZURE_B2C_CLIENT_SECRET=SECRET
    - AZURE_B2C_POLICY_NAME=POLICY
    - VIRTUAL_HOST_WEIGHT=1
  expose:
    - '8082 
  image: 'unicef/etools-prp:develop'
  restart: always
  tags:
    - prp
    - staging
    - unicef
  target_num_containers: 1
prp-redis:
  autoredeploy: true
  image: 'tivix/etools-prp-redis:develop'
  restart: on-failure
  tags:
    - prp
    - staging
    - unicef
  target_num_containers: 1
worker-prp:
  autoredeploy: true
  command: python manage.py celery worker -E --loglevel=info
  deployment_strategy: every_node
  environment:
    - CELERY_VISIBILITY_TIMEOUT=18000
    - C_FORCE_ROOT=1
    - DJANGO_ALLOWED_HOST=
    - DJANGO_DEBUG=true
    - DOMAIN_NAME=
    - EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
    - EMAIL_HOST_PASSWORD=
    - EMAIL_HOST_USER=apikey
    - ENV=dev
    - EXCLUDE_BASIC_AUTH=1
    - PMP_API_PASSWORD=
    - PMP_API_USER=
    - POSTGRES_DB=postgres
    - POSTGRES_HOST=prp-db
    - POSTGRES_PASSWORD=
    - POSTGRES_USER=postgres
    - 'REDIS_URL=redis://prp-redis:6379/0'
    - SECRET_KEY=
    - AWS_S3_ACCESS_KEY_ID=ID
    - AWS_S3_SECRET_ACCESS_KEY=SECRET
    - AWS_STORAGE_BUCKET_NAME=bucket
    - AWS_S3_REGION_NAME=eu-central-1
    - PRP_FRONTEND_HOST=localhost:8082
    - AZURE_B2C_CLIENT_ID=ID
    - AZURE_B2C_CLIENT_SECRET=SECRET
    - AZURE_B2C_POLICY_NAME=POLICY
  image: 'unicef/etools-prp:develop'
  restart: always
  tags:
    - prp
    - staging
    - unicef

How to sync locations

• Open up a browser and go `http://localhost:8082 api/admin/core/cartodbtable/` and log in.

• In order to get real data, you'll need to go to etools.carto.com for getting dataset names. Consult UNICEF CartoDB expert to get dataset import metadata.

• Click "ADD CARTO DB TABLE" button in Django admin and fill the following information:

  • Domain: etools

  • API KEY: *****

  • Table name: (Use above dataset name)

  • Display name: Use country name and location type for human readable format

  • Location type: (Create a new one with right admin level)

    • Use this convention for name field: <ISO3 code in uppercase>-Admin Level <LEVEL_NUMBER>

    • Ex) AFG-Admin Level 1

  • Name Column

  • Pcode Column

  • Parent Code Column

  • Parent: (Choose the parent carto db table object if admin level is not 0)

  • Country

• Repeat Step 4 for each country until every single admin level carto db table is created.

• After creating a carto db table, select 1 carto db table at a time (with admin level descending) and do "Import sites" Django admin action.

• After that, the new locations should be created if import is successful.