OCHA Integration - Research Summary

This is a detailed summary of initial research on how to integrate the systems. See child pages for documentation of actual implementation.

OCHA API's information

Documentation

Detailed documentation is available here: api.hpc.tools/docs/v1/

This documentation is intended for users of the HPC Service public API, which provides data in JSON or XML formats.

For the API v2 endpoints (currently only data for Chad 2018 is available).

It includes the changes:

Old OPS API does not provide any documentation.

Authorization

Access to API is public, however there is limit to 1 request per second. Logged users does not have this limitation. API is using HTTP Basic Authentication.

GET /v1/public/fts/flow?year=2016 HTTP/1.1
Host: api.hpc.tools
Content-Type: application/json
Authorization: Basic base64(clientABC:passwordXYZ)

External ID

Synced models should be updated with additional fields, to document their origin. This is achieved using a dedicated base model class.

Column

Type

external_id

integer

external_source

choice - HPC or OPS

Exception should be made for some simple tables (such as Country, etc.)

All further mappings includes those fields by default.

Response Plan API

List of available Response Plans

Endpoint get a list of current Response Plans in given country:

https://api.hpc.tools/v1/public/plan/country/<COUNTRY_ISO>

Example: https://api.hpc.tools/v1/public/plan/country/UKR

Field mapping

PRP Response Plan

HPC Plan

title

name

workspace

emergencies[0] if present else use locations[0]

start

startDate

end

endDate

plan_type

categories -> name

Sample JSON outputs

OCHA provided sample JSON outputs for some plans:

https://www.dropbox.com/sh/ogk4lrg4qyieyj7/AADlWxjBLyZHF5aszeCUrlwpa?dl=0

Response Plan Details

To get details of response plan, another endpoint is required:

https://api.hpc.tools/v1/public/rpm/plan/id/<plan_id>

Example: https://api.hpc.tools/v1/public/rpm/plan/id/514

Same endpoints returns different values depends on content parameter:

content: required (one of basic, entities, measurements)Specify the detail level for the content of the API response:

  • basic : only basic content for a plan.

  • entities : plan structure with entities and attachments.

  • measurements : plan structure with monitoringPeriods, entities, attachments and measurements.

In summary - to get plan structure with monitoringPeriods, entities, attachments and measurements, endpoint looks like: https://api.hpc.tools/v1/public/rpm/plan/id/514?format=json&content=measurements

Within the API output, there is a parentID within the Clusters Objectives or Cluster Activities linking them back to the Cluster (the parent).

If the Cluster Objectives or Activities support a particular Strategic or Cluster Objective then the parameter to look for is the value.support.

Clusters

There is a global cluster list, but then plans/responses can have their own cluster names sometimes as well.

However same Cluster name contains different ID's for different plans, example:

https://api.hpc.tools/v1/public/rpm/plan/id/641?format=json&content=entities

{
    "id": 4042,
    "name": "Education",
    "customReference": "L5411",
    "value": {
        "description": null,
        "icon": null,
        "categories": []
    },
    "planId": 641,
    "entityPrototype": {
        "id": 2213,
        "refCode": "CL",
        "type": "GVE",
        "value": {
            "name": {
                "en": {
                    "singular": "Cluster",
                    "plural": "Clusters"
                }
            },
            "possibleChildren": [
                {
                    "refCode": "CO",
                    "cardinality": "0-N",
                    "id": 2215
                },
                {
                    "refCode": "CA",
                    "cardinality": "0-N",
                    "id": 2216
                }
            ]
        }
    },
    "attachments": [],
    "clusterNumber": "L5411"
}

https://api.hpc.tools/v1/public/rpm/plan/id/475?format=json&content=entities

{
    "id": 1956,
    "name": "Education",
    "customReference": "1956",
    "value": {
        "description": null,
        "icon": null,
        "categories": []
    },
    "planId": 475,
    "entityPrototype": {
        "id": 2021,
        "refCode": "CL",
        "type": "GVE",
        "value": {
            "name": {
                "en": {
                    "singular": "Cluster",
                    "plural": "Clusters"
                }
            },
            "possibleChildren": [
                {
                    "refCode": "CO",
                    "cardinality": "0-N",
                    "id": 2023
                },
                {
                    "refCode": "CA",
                    "cardinality": "0-N",
                    "id": 2024
                }
            ]
        }
    },
    "attachments": [],
    "clusterNumber": "1956"
},

Endpoint for access global Clusters list:

https://api.hpc.tools/v1/public/global-cluster

Field mapping

https://api.hpc.tools/v1/public/rpm/plan/id/{id}?content=entities

PRP Cluster model fields

RPM Plan API fields

type

governingEntities -> name(1)

response_plan

id

(1) where value->name->en->singular is Cluster

Cluster activity

To retrieve cluster activities it has to be used content=entities parameter:

https://api.hpc.tools/v1/public/rpm/plan/id/<plan_id>?content=entities

Field mapping

PRP Cluster Activity model fields

RPM Plan API fields

title

planEntities -> value -> description (1)

cluster_objective

support (2)

locations

attachments (3)

reportables

attachments (4)

(1) where planEntities -> value -> name -> en -> singular = "Cluster Activity"

(2) There's either relation to Strategic Objective (SO) or Cluster Objective (CO) (In PRP both will be treated as Cluster Objective), activity can link to multiple objectives - due to schema limitations only the first objective can be currently linked

(3) iterate over all Indicators in: attachments -> value -> metrics -> values -> disaggregated -> locationsand "type": "indicator"

(4) indicators are stored in attachments with "type": "indicator"

Cluster objectives

To retrieve cluster objectives it has to be used content=entities parameter:

https://api.hpc.tools/v1/public/rpm/plan/id/<plan_id>?content=entities

In some case the 3 layer structure of strategic objectives > cluster objectives > cluster activities is not always followed. Cluster activity can link to strategic objective. Strategic objectives are multi-cluster.

Field mapping

PRP Cluster Objective model fields

RPM Plan API fields

title

planEntities -> value -> description (1)

cluster

parent -> parentId (2)

locations

attachments (3)

reportables

attachments (4)

(1) where planEntities -> value -> name -> en -> singular = "Cluster Objective"

(2) parentId points to Cluster ID - might be empty, need to resolve in PRP

(3) iterate over all Indicators in: attachments -> value -> metrics -> values -> disaggregated -> locationsand "type": "indicator"

(4) indicators are stored in attachments with "type": "indicator"

Indicators and Disaggregation data

To retrieve disaggregation data it has to be used content=measurements parameter:

https://api.hpc.tools/v1/public/rpm/plan/id/<plan_id>?content=measurements

Field mapping

PRP Reportable model fields

RPM Plan API fields

target

attachments -> value -> metrics -> values -> totals -> "type": "target"

baseline

attachments -> value -> metrics -> values -> totals -> "type": "baseline"

in_need

attachments -> value -> metrics -> values -> totals -> "type": "inNeed"

is_cluster_indicator

True

locations

attachments -> value -> metrics -> values -> disaggregated -> locations

disaggregations

disaggregations <-> attachments -> value -> metrics -> values -> disaggregated -> categories

inNeed is sent for entire plan / response. Also sent per indicator in old platform. Depends on country we might get different results.

Disaggregations are defined as mix of ids, for example: "ids": [1, 4], where 1 is Child, 4 is a Male. There is possibility to get them into PRP Disaggregation and DisaggregationValue model. Disaggregation ID are unique per system.

TODO: Disaggregations matrix needs to be analyzed.

Locations

Response plan and Disaggregation data have assigned locations.

Response Plan location field mapping

PRP Location

RPM API Plan Location

title

name

gateway -> admin_level

adminLevel

However disaggregation admin_level is provided only in OPS. By default in RMP, we should take value 0.

Disaggregation data location field mapping

PRP Location

RPM API Plan Location

title

name

gateway -> admin_level

0

Projects API

Projects are available in old OPS and new Project Module (new OPS).

Overview

For the new Projects Module, the project data is available here:

https://api.hpc.tools/v1/public/project/plan/637

However, there is a new v2 path that provides more project details than the v1 path.

The v2 path doesn’t currently have a listing like this, and so you would need the individual project id before getting more details e.g.

https://api.hpc.tools/v2/public/project/52716

Old OPS API provide only three endpoints (https://ops.unocha.org/api):

  • by Project ID

  • by Appeal ID (short details)

  • by Appeal ID (full details)

Response is available in XML and JSON (depends on request suffix .xml or .json).

The old OPS is accessible under this URL, and does not require any authentication:

https://ops.unocha.org/api/v1/project/appeal/[id].xml

Where the [id] is the appeal id. Note that this id is different from the plan id in the HPC database. The response from this endpoint is published (finalized) project data for the appeal.

List of available Projects

New Project Module

To get list of available projects for given plan:

https://api.hpc.tools/v1/public/project/plan/637

We are able to get the Organisation Id https://api.hpc.tools/v1/public/organization. Business Analyst for UNICEF will add the Organisation ID into the OCHA External Partner field in PRP Django Admin

Filtering is not available, so iteration over all set is required (organizations -> organization -> name or abbreviation should be same like choosed PRP organization / partner).

Old OPS

To find the appeal id’s for any particular country you are interested in, use this example:

https://ftsarchive.unocha.org/api/v1/appeal/year/2017.xml

This will give you a list of all available appeals for the year 2017, and then you can search within the result set.

Now having proper appeal ID we can list all projects:

https://ops.unocha.org/api/v1/projectshort/appeal/951.xml

To get detailed results (with assigned organizations):

https://ops.unocha.org/api/v1/project/appeal/951.xml

Similar to New Project Module, filtering is not available. In case of limit projects to choosed partner only, we have iterate over each project and compare organisations -> organisation -> name or abbreviation.

Project details

During import selected Project, system synchronize all related information like Partner Activities, Objectives, Locations, Indicators etc.

New Project Module

To get project details:

https://api.hpc.tools/v2/public/project/52716

Fields mapping

PRP Partner Project model

New Project Module response

title

name

description

objective

additional_information

objective

start_date

startDate

end_date

endDate

status

published

total_budget

data -> incoming -> fundingTotal FTS (1)

funding_source

NEED TO BE MOVED AS MODEL (2)

clusters

globalClusters -> name

locations

locations

partner

organizations

reportables

Only for Chad 2018, Libya

(1) To get total budget we have to use FTS endpoint:

https://api.hpc.tools/v1/public/fts/flow?projectId=47561

where projectId is id field from project details endpoint, for example:

https://api.hpc.tools/v2/public/project/47561

(2) Since funding_source might have more then one funding relation must be changed to 0 -> N.

class FundingSource(models.model):
    partner_project = models.ForeignKey(PartnerProject,
        related_name="funding_sources")
    source_name = models.CharField(max_length=255)
    source_id = models.IntegerField()
    source_type = models.CharField(max_length=255)
    original_amount = models.DecimalField(decimal_places=2, max_digits=12)
    original_currency = models.CharField(
        choices=CURRENCIES,
        default=CURRENCIES.usd,
        max_length=16,
    )
    exchange_rate = models.DecimalField(decimal_places=3, max_digits=8,
        default=1)

FTS field mapping

PRP FundingSource model fields

FTS API response

partner_project

Partner Project object

source_name

data -> flows -> sourceObjects -> name

source_id

data -> flows -> sourceObjects -> id

source_type

",".join(data -> flows -> sourceObjects -> organizationTypes)

original_amount

data -> flows -> originalAmount

original_currency

data -> flows -> originalCurrency

exchange_rate

data -> flows -> exchangeRate

Additional fields to capture

PRP Partner Project model new fields

New Project Module response

code

code

priorization_classification

projectPriority (1)

Only v1 holds this information except Chad and Libya (that works only on v2)

Old OPS

To get project details we have to find interesting project in response:

https://ops.unocha.org/api/v1/project/appeal/2897.json

I do not see possibility of filtering data.

Fields mapping

PRP Partner Project model

OPS response

title

title

description

description -> ProjectDescription -> Value (where "type": "objectives")

additional_information

n/a

start_date

startdate

end_date

endDate

status

published

total_budget

data -> incoming -> fundingTotal FTS (1)

funding_source

NEED TO BE MOVED AS MODEL (2)

clusters

cluster

locations

projectEGF -> egf_location

partner

organisations

reportables

n/a

(1) and (2) same like in New Project Module - data comes from FTS

Clusters

New Project Module

There is globalClusters section where provided ID is global for all projects in system.

PRP Cluster model fields

PRP Cluster model

RPM API fields

type

globalClusters -> name (1)

response_plan

plans -> id (matched by external_id)

Old OPS

PRP Cluster model

OPS API fields

type

cluster -> value

response_plan

appeal -> id (matched by external_id)

Partner

New Project Module and Old OPS

Partner / Organization is stored in organizations section.

We need to assign OCHA partners with partners synchronized with PMP system. Contact with Rob Avram from Unicef for details. PRP already synchronize Partners with PMP.

Locations

New Project Module

PRP Location

RPM API Location

title

locations -> name

gateway -> admin_level

locations -> adminLevel

latitude

locations -> latitude

longitude

locations -> longitude

p_code

locations -> pcode

Old OPS

PRP Location

OPS API Location

title

locations -> value

gateway -> admin_level

locations -> level

gateway -> name

locations -< name

Indicators and Disaggregation data

New Project Module

To get the full set of information for a project based on its id, you will need to connect to several different endpoints.

1. First contact the v1 endpoint to get the list of projects for Chad 2018, which will give you the project id’s.

2. https://api.hpc.tools/v1/public/project/plan/637

3. Next get the plan’s specific set of fields from this endpoint. These are the customized fields relevant to this plan only. Each condition field has a “name” and “id” value. You will need this later on to link a project’s set of answers to these fields.

https://api.hpc.tools/v2/public/plan/637/procedure

4. From the first endpoint, loop for each project to get the “id” and the “currentPublishedVersionId” values for each object in the “data” array.

5. Then use the project id here to get the overview information of the project:

https://api.hpc.tools/v2/public/project/52765

6. Use the project currentPublishedVersionId to get the cluster indicators it is linked to. This also includes the project’s values for the disaggregation models set up for each cluster indicator. https://api.hpc.tools/v2/public/project-version/104969/attachments

7. Use the project currentPublishedVersionId to get the budget breakdown for each organization in the project. Data[0] is for the whole project, data[n] is for each organization in the project. https://api.hpc.tools/v2/public/project-version/104969/segments

8. Use the project currentPublishedVersionId to get the values for each of the plan’s custom fields. https://api.hpc.tools/v2/public/project-version/104969/fields

Old OPS

Old OPS countries does not have endpoint for Indicators (yet).

PreviousCartoDB location syncNextResponse Plan Import

Last updated