{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "import itertools\n",
    "import os\n",
    "\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# Entity Service: Multiparty linkage demo\n",
    "This notebook is a demonstration of the multiparty linkage capability that has been implemented in the Entity Service.\n",
    "\n",
    "We show how five parties may upload their hashed data to the Entity Service to obtain a multiparty linkage result. This result identifies each entity across all datasets in which they are included."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Check the status of the Entity Service\n",
    "Ensure that it is running and that we have the correct version. Multiparty support was introduced in version 1.11.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'project_count': 10, 'rate': 20496894, 'status': 'ok'}\n",
      "{'anonlink': '0.11.2', 'entityservice': 'v1.11.0', 'python': '3.6.8'}\n"
     ]
    }
   ],
   "source": [
    "SERVER = os.getenv(\"SERVER\", \"https://testing.es.data61.xyz\")\n",
    "PREFIX = f\"{SERVER}/api/v1\"\n",
    "print(requests.get(f\"{PREFIX}/status\").json())\n",
    "print(requests.get(f\"{PREFIX}/version\").json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Create a new project\n",
    "We create a new multiparty project for five parties by specifying the number of parties and the output type (currently only the `group` output type supports multiparty linkage). Retain the `project_id`, so we can find the project later. Also retain the `result_token`, so we can retrieve the results (careful: anyone with this token has access to the results). Finally, the `update_tokens` identify the five data data providers and permit them to upload CLKs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "project_id: 8eeb1050f5add8f78ff4a0da04219fead48f22220fb0f15e\n",
      "\n",
      "result_token: c8f22b577aac9432871eeea02cbe504d399a9776add1de9f\n",
      "\n",
      "update_tokens: ['6bf0f1c84c17116eb9f93cf8a4cfcb13d49d288a1f376dd8', '4b9265070849af1f0546f2adaeaa85a7d0e60b10f9b4afbc', '3ff03cadd750ce1b40cc4ec2b99db0132f62d8687328eeb9', 'c1b562ece6bbef6cd1a0541301bb1f82bd697bce04736296', '8cfdebbe12c65ae2ff20fd0c0ad5de4feb06c9a9dd1209c8']\n"
     ]
    }
   ],
   "source": [
    "project_info = requests.post(\n",
    "    f\"{PREFIX}/projects\",\n",
    "    json={\n",
    "        \"schema\": {},\n",
    "        \"result_type\": \"groups\",\n",
    "        \"number_parties\": 5,\n",
    "        \"name\": \"example project\"\n",
    "    }\n",
    ").json()\n",
    "project_id = project_info[\"project_id\"]\n",
    "result_token = project_info[\"result_token\"]\n",
    "update_tokens = project_info[\"update_tokens\"]\n",
    "\n",
    "print(\"project_id:\", project_id)\n",
    "print()\n",
    "print(\"result_token:\", result_token)\n",
    "print()\n",
    "print(\"update_tokens:\", update_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Upload the hashed data\n",
    "This is where each party uploads their CLKs into the service. Here, we do the work of all five data providers inside this for loop. In a deployment scenario, each data provider would be uploading their own CLKs using their own update token.\n",
    "\n",
    "These CLKs are already hashed using [clkhash](https://github.com/data61/clkhash), so for each data provider, we just need to upload their corresponding hash file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data provider 1: {\n",
      "  \"message\": \"Updated\",\n",
      "  \"receipt_token\": \"c7d9ba71260863f13af55e12603f8694c29e935262b15687\"\n",
      "}\n",
      "\n",
      "Data provider 2: {\n",
      "  \"message\": \"Updated\",\n",
      "  \"receipt_token\": \"70e4ed1b403c4e628183f82548a9297f8417ca3de94648bf\"\n",
      "}\n",
      "\n",
      "Data provider 3: {\n",
      "  \"message\": \"Updated\",\n",
      "  \"receipt_token\": \"b56fe568b93dc4522444e503078e16c18573adecbc086b6a\"\n",
      "}\n",
      "\n",
      "Data provider 4: {\n",
      "  \"message\": \"Updated\",\n",
      "  \"receipt_token\": \"7e3c80e554cfde23847d9aa2cff1323aa8f411e4033c0562\"\n",
      "}\n",
      "\n",
      "Data provider 5: {\n",
      "  \"message\": \"Updated\",\n",
      "  \"receipt_token\": \"8bde91367ee52b5c6804d5ce2d2d3350ce3c3766b8625bbc\"\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, token in enumerate(update_tokens, start=1):\n",
    "    with open(f\"data/clks-{i}.json\") as f:\n",
    "        r = requests.post(\n",
    "            f\"{PREFIX}/projects/{project_id}/clks\",\n",
    "            data=f,\n",
    "            headers={\n",
    "                \"Authorization\": token,\n",
    "                \"content-type\": \"application/json\"\n",
    "            }\n",
    "        )\n",
    "    print(f\"Data provider {i}: {r.text}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Begin a run\n",
    "The data providers have uploaded their CLKs, so we may begin the computation. This computation may be repeated multiple times, each time with different parameters. Each such repetition is called a run. The most important parameter to vary between runs is the similarity threshold. Two records whose similarity is above this threshold will be considered to describe the same entity.\n",
    "\n",
    "Here, we perform one run. We (somewhat arbitrarily) choose the threshold to be 0.8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "r = requests.post(\n",
    "    f\"{PREFIX}/projects/{project_id}/runs\",\n",
    "    headers={\n",
    "        \"Authorization\": result_token\n",
    "    },\n",
    "    json={\n",
    "        \"threshold\": 0.8\n",
    "    }\n",
    ")\n",
    "run_id = r.json()[\"run_id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Check the status\n",
    "Let's see whether the run has finished ('state' is 'completed')!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'current_stage': {'description': 'waiting for CLKs',\n",
       "  'number': 1,\n",
       "  'progress': {'absolute': 5,\n",
       "   'description': 'number of parties already contributed',\n",
       "   'relative': 1.0}},\n",
       " 'stages': 3,\n",
       " 'state': 'queued',\n",
       " 'time_added': '2019-06-23T11:17:27.646642+00:00',\n",
       " 'time_started': None}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(\n",
    "    f\"{PREFIX}/projects/{project_id}/runs/{run_id}/status\",\n",
    "    headers={\n",
    "        \"Authorization\": result_token\n",
    "    }\n",
    ")\n",
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now after some delay (depending on the size) we can fetch the results. Waiting for completion can be achieved by directly polling the REST API using `requests`, however for simplicity we will just use the `watch_run_status` function provided in `clkhash.rest_client`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State: completed\n",
      "Stage (3/3): compute output\n"
     ]
    }
   ],
   "source": [
    "import clkhash.rest_client\n",
    "from IPython.display import clear_output\n",
    "\n",
    "for update in clkhash.rest_client.watch_run_status(SERVER, project_id, run_id, result_token, timeout=30):\n",
    "    clear_output(wait=True)\n",
    "    print(clkhash.rest_client.format_run_status(update))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Retrieve the results\n",
    "We retrieve the results of the linkage. As we selected earlier, the result is a list of groups of records. Every record in such a group belongs to the same entity and consists of two values, the party id and the row index.\n",
    "\n",
    "The last 20 groups look like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[[0, 3127], [3, 3145], [2, 3152], [1, 3143]],\n",
       " [[2, 1653], [3, 1655], [1, 1632], [0, 1673], [4, 1682]],\n",
       " [[0, 2726], [1, 2737], [3, 2735]],\n",
       " [[1, 837], [3, 864]],\n",
       " [[0, 1667], [4, 1676], [1, 1624], [3, 1646]],\n",
       " [[1, 1884], [2, 1911], [4, 1926], [0, 1916]],\n",
       " [[0, 192], [2, 198]],\n",
       " [[3, 328], [4, 330], [0, 350], [2, 351], [1, 345]],\n",
       " [[2, 3173], [4, 3176], [3, 3163], [0, 3145], [1, 3161]],\n",
       " [[1, 347], [4, 332], [2, 353], [0, 352]],\n",
       " [[1, 736], [3, 761], [2, 768], [0, 751], [4, 754]],\n",
       " [[1, 342], [2, 349]],\n",
       " [[3, 899], [2, 913]],\n",
       " [[1, 465], [3, 477]],\n",
       " [[0, 285], [1, 293]],\n",
       " [[0, 785], [3, 794]],\n",
       " [[3, 2394], [4, 2395], [0, 2395]],\n",
       " [[1, 1260], [2, 1311], [3, 1281], [4, 1326]],\n",
       " [[0, 656], [2, 663]],\n",
       " [[1, 2468], [2, 2479]]]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(\n",
    "    f\"{PREFIX}/projects/{project_id}/runs/{run_id}/result\",\n",
    "    headers={\n",
    "        \"Authorization\": result_token\n",
    "    }\n",
    ")\n",
    "groups = r.json()\n",
    "groups['groups'][-20:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To sanity check, we print their records' corresponding PII:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 ['samual', 'mason', '05-12-1917', 'male', 'pertb', '405808.756', '07 2284 3649']\n",
      "3 ['samuAl', 'mason', '05-12-1917', 'male', 'peryh', '4058o8.756', '07 2274 3549']\n",
      "2 ['samie', 'mazon', '05-12-1917', 'male', '', '405898.756', '07 2275 3649']\n",
      "1 ['zamusl', 'mason', '05-12-2917', 'male', '', '405898.756', '07 2274 2649']\n",
      "\n",
      "2 ['thomas', 'burfrod', '08-04-1999', '', 'pertj', '182174.209', '02 3881 9666']\n",
      "3 ['thomas', 'burfrod', '09-04-1999', 'male', '', '182174.209', '02 3881 9666']\n",
      "1 ['thomas', 'burford', '08-04-19o9', 'mal4', '', '182175.109', '02 3881 9666']\n",
      "0 ['thomas', 'burford', '08-04-1999', 'male', 'perth', '182174.109', '02 3881 9666']\n",
      "4 ['thomas', 'burf0rd', '08-04-q999', 'mske', 'perrh', '182174.109', '02 3881 9666']\n",
      "\n",
      "0 ['kaitlin', 'bondza', '03-08-1961', 'male', 'sydney', '41168.999', '02 4632 1380']\n",
      "1 ['kaitlin', 'bondja', '03-08-1961', 'malr', 'sydmey', '41168.999', '02 4632 1370']\n",
      "3 [\"k'latlin\", 'bonklza', '03-08-1961', 'male', 'sydaney', '', '02 4632 1380']\n",
      "\n",
      "1 ['chr8stian', 'jolly', '22-08-2009', 'male', '', '178371.991', '04 5868 7703']\n",
      "3 ['chr8stian', 'jolly', '22-09-2099', 'malr', 'melbokurne', '178271.991', '04 5868 7703']\n",
      "\n",
      "0 ['oaklrigh', 'ngvyen', '24-07-1907', 'mslr', 'sydney', '63175.398', '04 9019 6235']\n",
      "4 ['oakleith', 'ngvyen', '24-97-1907', 'male', 'sydiney', '63175.498', '04 9019 6235']\n",
      "1 ['oajleigh', 'ngryen', '24-07-1007', 'male', 'sydney', '63175.498', '04 9919 6235']\n",
      "3 ['oakleigh', 'nguyrn', '34-07-1907', 'male', 'sbdeney', '63175.r98', '04 9019 6235']\n",
      "\n",
      "1 ['georgia', 'nguyen', '06-11-1930', 'male', 'perth', '247847.799', '08 6560 4063']\n",
      "2 ['georia', 'nfuyen', '06-11-1930', 'male', 'perrh', '247847.799', '08 6560 4963']\n",
      "4 ['geortia', 'nguyea', '06-11-1930', 'male', 'pertb', '247847.798', '08 6560 4063']\n",
      "0 ['egorgia', 'nguyqn', '06-11-1930', 'male', 'peryh', '247847.799', '08 6460 4963']\n",
      "\n",
      "0 ['connor', 'mcneill', '05-09-1902', 'male', 'sydney', '108473.824', '02 6419 9472']\n",
      "2 ['connro', 'mcnell', '05-09-1902', 'male', 'sydnye', '108474.824', '02 6419 9472']\n",
      "\n",
      "3 ['alessandria', 'sherriff', '25-91-1951', 'male', 'melb0urne', '5224r.762', '03 3077 2019']\n",
      "4 ['alessandria', 'sherriff', '25-01-1951', 'male', 'melbourne', '52245.762', '03 3077 1019']\n",
      "0 ['alessandria', \"sherr'lff\", '25-01-1951', 'malr', 'melbourne', '', '03 3977 1019']\n",
      "2 ['alessandria', 'shernff', '25-01-1051', 'mzlr', 'melbourne', '52245.663', '03 3077 1019']\n",
      "1 ['alessandrya', 'sherrif', '25-01-1961', 'male', 'jkelbouurne', '52245.762', '03 3077 1019']\n",
      "\n",
      "2 ['harriyon', 'micyelmor', '21-04-1971', 'male', 'pert1>', '291889.942', '04 5633 5749']\n",
      "4 ['harri5on', 'micyelkore', '21-04-1971', '', 'pertb', '291880.942', '04 5633 5749']\n",
      "3 ['hariso17', 'micelmore', '21-04-1971', 'male', 'pertb', '291880.042', '04 5633 5749']\n",
      "0 ['harrison', 'michelmore', '21-04-1981', 'malw', 'preth', '291880.942', '04 5643 5749']\n",
      "1 ['harris0n', 'michelmoer', '21-04-1971', '', '', '291880.942', '04 5633 5749']\n",
      "\n",
      "1 ['alannah', 'gully', '15-04-1903', 'make', 'meobourne', '134518.814', '04 5104 4572']\n",
      "4 ['alana', 'gully', '15-04-1903', 'male', 'melbourne', '134518.814', '04 5104 4582']\n",
      "2 ['alama', 'gulli', '15-04-1903', 'mald', 'melbourne', '134518.814', '04 5104 5582']\n",
      "0 ['alsna', 'gullv', '15-04-1903', 'male', '', '134518.814', '04 5103 4582']\n",
      "\n",
      "1 ['sraah', 'bates-brownsword', '26-11-1905', 'malr', '', '59685.979', '03 8545 5584']\n",
      "3 ['sarah', 'bates-brownswort', '26-11-1905', 'male', '', '59686.879', '03 8545 6584']\n",
      "2 ['sara0>', 'bates-browjsword', '26-11-1905', 'male', '', '59685.879', '']\n",
      "0 ['saran', 'bates-brownsvvord', '26-11-1905', 'malr', 'sydney', '59685.879', '03 8555 5584']\n",
      "4 ['snrah', 'bates-bro2nsword', '26-11-1005', 'male', 'sydney', '58685.879', '03 8545 5584']\n",
      "\n",
      "1 ['beth', 'lette', '18-01-2000', 'female', 'sydney', '179719.049', '07 1868 6031']\n",
      "2 ['beth', 'lette', '18-02-2000', 'femal4', 'stdq7ey', '179719.049', '07 1868 6931']\n",
      "\n",
      "3 ['tahlia', 'bishlp', '', 'female', 'sydney', '101203.290', '03 886u 1916']\n",
      "2 ['ahlia', 'bishpp', '', 'female', 'syriey', '101204.290', '03 8867 1916']\n",
      "\n",
      "1 ['fzachary', 'mydlalc', '20-95-1916', 'male', 'sydney', '121209.129', '08 3807 4717']\n",
      "3 ['zachary', 'mydlak', '20-05-1016', 'malr', 'sydhey', '121200.129', '08 3807 4627']\n",
      "\n",
      "0 ['jessica', 'white', '04-07-1979', 'male', 'perth', '385632.266', '04 8026 8748']\n",
      "1 ['jezsica', 'whi5e', '05-07-1979', 'male', 'perth', '385632.276', '04 8026 8748']\n",
      "\n",
      "0 ['beriiamin', 'musoluno', '21-0y-1994', 'female', 'sydney', '81857.391', '08 8870 e498']\n",
      "3 ['byenzakin', 'musoljno', '21-07-1995', 'female', 'sydney', '81857.392', '']\n",
      "\n",
      "3 ['ella', 'howie', '26-03-2003', 'male', 'melbourne', '97556.316', '03 3655 1171']\n",
      "4 ['ela', 'howie', '26-03-2003', 'male', 'melboirne', '', '03 3555 1171']\n",
      "0 ['lela', 'howie', '26-03-2903', 'male', 'melbourhe', '', '03 3655 1171']\n",
      "\n",
      "1 ['livia', 'riaj', '13-03-1907', 'malw', 'melbovrne', '73305.107', '07 3846 2530']\n",
      "2 ['livia', 'ryank', '13-03-1907', 'malw', 'melbuorne', '73305.107', '07 3946 2630']\n",
      "3 ['ltvia', 'ryan', '13-03-1907', 'maoe', 'melbourne', '73305.197', '07 3046 2530']\n",
      "4 ['livia', 'ryan', '13-03-1907', 'male', 'melbourne', '73305.107', '07 3946 2530']\n",
      "\n",
      "0 ['coby', 'ibshop', '', 'msle', 'sydney', '211655.118', '02 0833 7777']\n",
      "2 ['coby', 'bishop', '15-08-1948', 'male', 'sydney', '211655.118', '02 9833 7777']\n",
      "\n",
      "1 ['emjkly', 'pareemore', '01-03-2977', 'female', 'rnelbourne', '1644487.925', '03 5761 5483']\n",
      "2 ['emiily', 'parremore', '01-03-1977', 'female', 'melbourne', '1644487.925', '03 5761 5483']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def load_dataset(i):\n",
    "    dataset = []\n",
    "    with open(f\"data/dataset-{i}.csv\") as f:\n",
    "        reader = csv.reader(f)\n",
    "        next(reader)  # ignore header\n",
    "        for row in reader:\n",
    "            dataset.append(row[1:])\n",
    "    return dataset\n",
    "\n",
    "datasets = list(map(load_dataset, range(1, 6)))\n",
    "\n",
    "for group in itertools.islice(groups[\"groups\"][-20:], 20):\n",
    "    for (i, j) in group:\n",
    "        print(i, datasets[i][j])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Despite the high amount of noise in the data, the entity service was able to produce a fairly accurate matching. However, Isabella George and Mia/Talia Galbraith are most likely not an actual match.\n",
    "\n",
    "We may be able to improve on this results by fine-tuning the hashing schema or by changing the threshold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Delete the project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n"
     ]
    }
   ],
   "source": [
    "r = requests.delete(\n",
    "    f\"{PREFIX}/projects/{project_id}\",\n",
    "    headers={\n",
    "        \"Authorization\": result_token\n",
    "    }\n",
    ")\n",
    "print(r.status_code)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}