{ "cells": [ { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "# Entity Service: Multiparty linkage demo\n", "This notebook is a demonstration of the multiparty linkage capability that has been implemented in the Entity Service.\n", "\n", "We show how five parties may upload their hashed data to the Entity Service to obtain a multiparty linkage result. This result identifies each entity across all datasets in which they are included." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "import csv\n", "import itertools\n", "import os\n", "import pandas as pd\n", "\n", "import requests" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each party has a dataset of the following form:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
givennamesurnamedobgendercityincomephone number
id
0tarahilton27-08-1941malecanberra84052.97308 2210 0298
3saJivernre22-12-2972malsperth50104.11802 1090 1906
7sliverpaciorekNaNmalssydney31750.893NaN
9rubygeorge09-05-1939malesydney135099.87507 4698 6255
10eyrinmcampbell29-1q-1983maleperthNaN08 299y 1535
\n", "
" ], "text/plain": [ " givenname surname dob gender city income phone number\n", "id \n", "0 tara hilton 27-08-1941 male canberra 84052.973 08 2210 0298\n", "3 saJi vernre 22-12-2972 mals perth 50104.118 02 1090 1906\n", "7 sliver paciorek NaN mals sydney 31750.893 NaN\n", "9 ruby george 09-05-1939 male sydney 135099.875 07 4698 6255\n", "10 eyrinm campbell 29-1q-1983 male perth NaN 08 299y 1535" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_csv('data/dataset-1.csv', index_col='id').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Comparing the beginning of the first dataset to the second, we can see that the quality of the data is not very good. There are a lot of spelling mistakes and missing information. Let's see how well the entity service does with linking those entities." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
givennamesurnamedobgendercityincomephone number
id
3zaliverner22-12-1972maleperth50104.11802 1090 1906
4samueltremellen21-12-1923malemelbourne159316.09103 3605 9336
5amylodge16-01-1958malecanberra70170.45607 8286 9372
7oIjipacioerk10-02-1959mal3sydney31750.89304 4220 5949
10erinkampgell29-12-1983makeperth331476.59808 2996 1445
\n", "
" ], "text/plain": [ " givenname surname dob gender city income \\\n", "id \n", "3 zali verner 22-12-1972 male perth 50104.118 \n", "4 samuel tremellen 21-12-1923 male melbourne 159316.091 \n", "5 amy lodge 16-01-1958 male canberra 70170.456 \n", "7 oIji pacioerk 10-02-1959 mal3 sydney 31750.893 \n", "10 erin kampgell 29-12-1983 make perth 331476.598 \n", "\n", " phone number \n", "id \n", "3 02 1090 1906 \n", "4 03 3605 9336 \n", "5 07 8286 9372 \n", "7 04 4220 5949 \n", "10 08 2996 1445 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_csv('data/dataset-2.csv', index_col='id').head()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Check the status of the Entity Service\n", "Ensure that it is running and that we have the correct version. Multiparty support was introduced in version 1.11.0." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'project_count': 839, 'rate': 550410, 'status': 'ok'}\n", "{'anonlink': '0.12.5', 'entityservice': 'v1.13.0-beta2', 'python': '3.8.2'}\n" ] } ], "source": [ "SERVER = os.getenv(\"SERVER\", \"https://anonlink.easd.data61.xyz\")\n", "PREFIX = f\"{SERVER}/api/v1\"\n", "print(requests.get(f\"{PREFIX}/status\").json())\n", "print(requests.get(f\"{PREFIX}/version\").json())" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Create a new project\n", "We create a new multiparty project for five parties by specifying the number of parties and the output type (currently only the `group` output type supports multiparty linkage). Retain the `project_id`, so we can find the project later. Also retain the `result_token`, so we can retrieve the results (careful: anyone with this token has access to the results). Finally, the `update_tokens` identify the five data data providers and permit them to upload CLKs." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "project_id: 35697a8223f98ed4112488ae3c87e8134d169a364d35e2e7\n", "\n", "result_token: 075faf5822cfbe3abe4ce47510a7d3190f518768282f83a7\n", "\n", "update_tokens: ['26a30750ba4b7124bc3fd8a36e57bf6211af3fda960c6fb0', '27d17421a4f01c61e4b6ec782486c550da93d350a8d2dbf1', '5c0f98cd55acd48c99bd7f2ddd26af46f6afd31095c7a8a1', 'dcc87296257cb13c9ac3da1e0905c1448a5d51bc9f1fbec3', '9937b6e17abe516e9364cbc88a22593ef78ccdf3d045a907']\n" ] } ], "source": [ "project_info = requests.post(\n", " f\"{PREFIX}/projects\",\n", " json={\n", " \"schema\": {},\n", " \"result_type\": \"groups\",\n", " \"number_parties\": 5,\n", " \"name\": \"example project\"\n", " }\n", ").json()\n", "project_id = project_info[\"project_id\"]\n", "result_token = project_info[\"result_token\"]\n", "update_tokens = project_info[\"update_tokens\"]\n", "\n", "print(\"project_id:\", project_id)\n", "print()\n", "print(\"result_token:\", result_token)\n", "print()\n", "print(\"update_tokens:\", update_tokens)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Upload the hashed data\n", "This is where each party uploads their CLKs into the service. Here, we do the work of all five data providers inside this for loop. In a deployment scenario, each data provider would be uploading their own CLKs using their own update token.\n", "\n", "These CLKs are already hashed using [clkhash](https://github.com/data61/clkhash) (with [this](data/schema.json) linkage schema), so for each data provider, we just need to upload their corresponding hash file." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data provider 1: {\n", " \"message\": \"Updated\",\n", " \"receipt_token\": \"be6ab1dd0833283ec78ce829f7276b53926588d86c503534\"\n", "}\n", "\n", "Data provider 2: {\n", " \"message\": \"Updated\",\n", " \"receipt_token\": \"74a3f479949d5bb2537c5cab01db9d8d08bf0f7aad991c4d\"\n", "}\n", "\n", "Data provider 3: {\n", " \"message\": \"Updated\",\n", " \"receipt_token\": \"5a88765376836d57e37489e9f205e0d5bb8d9abd6d9cfc7a\"\n", "}\n", "\n", "Data provider 4: {\n", " \"message\": \"Updated\",\n", " \"receipt_token\": \"e005523285d21cfec2927d17050faffb1c249a5b8784f2a4\"\n", "}\n", "\n", "Data provider 5: {\n", " \"message\": \"Updated\",\n", " \"receipt_token\": \"e2c10b8f9f5f6ea90978d9cf0f3b25700fbd222658b704bb\"\n", "}\n", "\n" ] } ], "source": [ "for i, token in enumerate(update_tokens, start=1):\n", " with open(f\"data/clks-{i}.json\") as f:\n", " r = requests.post(\n", " f\"{PREFIX}/projects/{project_id}/clks\",\n", " data=f,\n", " headers={\n", " \"Authorization\": token,\n", " \"content-type\": \"application/json\"\n", " }\n", " )\n", " print(f\"Data provider {i}: {r.text}\")" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Begin a run\n", "The data providers have uploaded their CLKs, so we may begin the computation. This computation may be repeated multiple times, each time with different parameters. Each such repetition is called a run. The most important parameter to vary between runs is the similarity threshold. Two records whose similarity is above this threshold will be considered to describe the same entity.\n", "\n", "Here, we perform one run. We (somewhat arbitrarily) choose the threshold to be 0.8." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "r = requests.post(\n", " f\"{PREFIX}/projects/{project_id}/runs\",\n", " headers={\n", " \"Authorization\": result_token\n", " },\n", " json={\n", " \"threshold\": 0.8\n", " }\n", ")\n", "run_id = r.json()[\"run_id\"]" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Check the status\n", "Let's see whether the run has finished ('state' is 'completed')!" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "pycharm": {} }, "outputs": [ { "data": { "text/plain": [ "{'current_stage': {'description': 'waiting for CLKs',\n", " 'number': 1,\n", " 'progress': {'absolute': 5,\n", " 'description': 'number of parties already contributed',\n", " 'relative': 1.0}},\n", " 'stages': 3,\n", " 'state': 'created',\n", " 'time_added': '2020-04-03T01:20:55.141739+00:00',\n", " 'time_started': None}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = requests.get(\n", " f\"{PREFIX}/projects/{project_id}/runs/{run_id}/status\",\n", " headers={\n", " \"Authorization\": result_token\n", " }\n", ")\n", "r.json()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now after some delay (depending on the size) we can fetch the results. Waiting for completion can be achieved by directly polling the REST API using `requests`, however for simplicity we will just use the `watch_run_status` function provided in `anonlinkclient.rest_client`." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "State: completed\n", "Stage (3/3): compute output\n" ] } ], "source": [ "from IPython.display import clear_output\n", "from anonlinkclient.rest_client import RestClient, format_run_status\n", "\n", "rest_client = RestClient(SERVER)\n", "for update in rest_client.watch_run_status(project_id, run_id, result_token, timeout=300):\n", " clear_output(wait=True)\n", " print(format_run_status(update))" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Retrieve the results\n", "We retrieve the results of the linkage. As we selected earlier, the result is a list of groups of records. Every record in such a group belongs to the same entity and consists of two values, the party id and the row index.\n", "\n", "The last 20 groups look like this." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "pycharm": {} }, "outputs": [ { "data": { "text/plain": [ "[[[0, 781], [4, 780]],\n", " [[2, 3173], [4, 3176], [3, 3163], [0, 3145], [1, 3161]],\n", " [[2, 1617], [3, 1620]],\n", " [[0, 444], [1, 423]],\n", " [[4, 391], [1, 409]],\n", " [[1, 347], [4, 332], [2, 353], [0, 352]],\n", " [[1, 3171], [4, 3185], [0, 3153], [2, 3182], [3, 3172]],\n", " [[2, 1891], [4, 1906], [3, 1889]],\n", " [[0, 2139], [4, 2147]],\n", " [[0, 1206], [4, 1205], [2, 1206]],\n", " [[2, 2726], [4, 2710], [3, 2722]],\n", " [[3, 2040], [4, 2059], [2, 2059]],\n", " [[1, 899], [4, 924], [0, 923]],\n", " [[0, 2482], [1, 2494], [4, 2483], [3, 2488], [2, 2509]],\n", " [[3, 741], [4, 736], [2, 749], [1, 722]],\n", " [[1, 1587], [4, 1638]],\n", " [[1, 1157], [4, 1209]],\n", " [[1, 2027], [3, 740]],\n", " [[1, 1260], [2, 1311], [3, 1281], [4, 1326]],\n", " [[1, 1323], [2, 1362], [4, 1384], [0, 1396]]]" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = requests.get(\n", " f\"{PREFIX}/projects/{project_id}/runs/{run_id}/result\",\n", " headers={\n", " \"Authorization\": result_token\n", " }\n", ")\n", "groups = r.json()\n", "groups['groups'][-20:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To sanity check, we print their records' corresponding PII:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "pycharm": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 ['kain', 'mason', '09-07-1932', 'male', 'sydnev', '119435.710', '08 8537 7448']\n", "4 ['kaim', 'iiiazon', '09-07-1932', 'male', 'sydnev', '119445.720', '08 8638 7448']\n", "\n", "2 ['harriyon', 'micyelmor', '21-04-1971', 'male', 'pert1>', '291889.942', '04 5633 5749']\n", "4 ['harri5on', 'micyelkore', '21-04-1971', '', 'pertb', '291880.942', '04 5633 5749']\n", "3 ['hariso17', 'micelmore', '21-04-1971', 'male', 'pertb', '291880.042', '04 5633 5749']\n", "0 ['harrison', 'michelmore', '21-04-1981', 'malw', 'preth', '291880.942', '04 5643 5749']\n", "1 ['harris0n', 'michelmoer', '21-04-1971', '', '', '291880.942', '04 5633 5749']\n", "\n", "2 ['lauren', 'macgowan', '08-01-1960', 'male', '', '43779.493', '03 6533 7075']\n", "3 ['lauren', 'macgowan', '08-01-1950', 'male', 'sydney', '43770.493', '03 6532 7075']\n", "\n", "0 ['joshai', 'browne', '30-10-2904', '', 'melbounfe', '522585.205', '03 7150 7587']\n", "1 ['joshua', 'browne', '30-10-2004', 'female', 'melbourne', '522585.205', '03 7150 7587']\n", "\n", "4 ['feliciti', 'green', '23-02-1909', 'male', '', '183205.299', '08 4794 9870']\n", "1 ['feljcitv', 'greery', '23-02-1998', 'male', '', '183205.299', '08 4794 9970']\n", "\n", "1 ['alannah', 'gully', '15-04-1903', 'make', 'meobourne', '134518.814', '04 5104 4572']\n", "4 ['alana', 'gully', '15-04-1903', 'male', 'melbourne', '134518.814', '04 5104 4582']\n", "2 ['alama', 'gulli', '15-04-1903', 'mald', 'melbourne', '134518.814', '04 5104 5582']\n", "0 ['alsna', 'gullv', '15-04-1903', 'male', '', '134518.814', '04 5103 4582']\n", "\n", "1 ['madison', 'crosswell', '11-06-1990', 'male', 'perth', '151347.559', '03 0936 9125']\n", "4 ['madisori', 'crossw4ll', '11-96-1990', 'male', 'perth', '151347.559', '03 0926 9125']\n", "0 ['madispn', 'crossvvell', '11-06-2990', 'male', 'bperth', '151347.559', '03 0936 9125']\n", "2 ['badisoj', 'cross2ell', '11-06-1990', 'malw', 'eprth', '151347.559', '03 0936 9125']\n", "3 ['mad9son', 'crosswell', '11-06-1990', '', '', '151347.559', '03 0937 9125']\n", "\n", "2 ['harley', 'krin', '29-05-1967', 'maoe', 'melbourne', '120938.846', '08 8095 4760']\n", "4 ['harley', 'green', '29-05-1967', 'male', 'melbourne', '120937.846', '08 8096 4760']\n", "3 ['harley', 'gfeen', '29-04-1967', 'mslr', 'melbourne', '120937.856', '08 8096 4760']\n", "\n", "0 ['nicho1as', 'mak0nw', '06-06-1977', 'male', '', '91255.089', '08 2404 9176']\n", "4 ['nicol', 'maano', '06-06-1977', '', '', '91155.089', '08 2404 9176']\n", "\n", "0 ['james', 'lavender', '08-02-2000', 'male', 'canberra', '88844.369', '02 5862 9827']\n", "4 ['jaiiies', 'lvender', '08-02-2900', 'male', 'canberra', '88844.369', '02 5862 982u']\n", "2 ['jimmy', 'lavendre', '08-02-2000', 'malw', 'canberra', '88844.369', '02 5863 9827']\n", "\n", "2 ['ara', 'hite', '01-05-1994', 'femzle', 'canberra', '29293.820', '03 0641 9597']\n", "4 ['tara', 'white', '01-05-1984', 'female', 'canberra', '29293.820', '03 0641 9597']\n", "3 ['tara', 'white', '01-05-1974', 'femzle', '', '29293.820', '03 0641 0697']\n", "\n", "3 ['spericer', 'pize', '03-04-1983', 'male', 'canberra', '', '03 5691 5970']\n", "4 ['spencer', 'paize', '03-04-1983', 'male', 'canberra', '56328.357', '03 6691 5970']\n", "2 ['spenfer', 'pai2e', '03-04-1893', 'male', 'can1>erra', '56328.357', '03 6691 5970']\n", "\n", "1 ['isbaella', 'darby-cocks', '14-09-1921', 'male', 'pergh', '87456.184', '03 0678 5513']\n", "4 ['isabella', 'darby-cocks', '14-09-1921', 'male', 'perth', '87456.194', '03 0679 5513']\n", "0 ['isabeloa', 'darby-cocks', '14-09-2921', 'make', 'perth', '87456.194', '04 0678 6513']\n", "\n", "0 ['jarrod', 'brone', '09-08-1967', 'mal3', 'perth', '1075t6.775', '08 2829 1110']\n", "1 ['jarrod', 'browne', '09-08-1967', 'male', 'perth', '107556.775', '08 2820 1110']\n", "4 ['jarrod', 'brownb', '09-08-1967', 'mqle', 'pertb', '107556.775', '08 2820 2110']\n", "3 ['jarr0d', 'brown', '09-08-1967', 'male', '', '107546.775', '08 2820 1110']\n", "2 ['jarr0d', 'borwne', '09-08-1067', 'male', 'pertb', '107556.775', '08 2820 1110']\n", "\n", "3 ['marko', 'matthews', '11-04-1992', 'male', 'melbourne', '106467.902', '03 1460 7673']\n", "4 ['marko', 'matthews', '11-0r-1992', 'maoe', 'melhourne', '106467.992', '03 1460 7673']\n", "2 ['marko', 'matthevvs', '11-94-1992', 'mals', 'melbourne', '', '03 1460 7673']\n", "1 ['makro', 'matthews', '11-04-1992', '', 'emlbourne', '106467.903', '03 1460 7673']\n", "\n", "1 ['nkiki', 'spers', '10-02-2007', 'fenale', '', '156639.106', '07 9447 1767']\n", "4 ['nikkui', 'pezes', '10-02-20p7', 'female', '', '156639.106', '07 9447 1767']\n", "\n", "1 ['roby', 'felepa', '25-19-1959', 'male', 'aclonerra', '85843.631', '07 5804 7920']\n", "4 ['robert', 'felepa', '25-10-1959', 'male', 'can1>erra', '85842.631', '07 5804 7929']\n", "\n", "1 ['shai', 'dixon', '24-09-1979', 'female', 'melbourne', '609473.955', '08 4533 9404']\n", "3 ['mia', 'dixon', '24-09-1979', 'female', 'melbourne', '1198037.556', '08 3072 7335']\n", "\n", "1 ['livia', 'riaj', '13-03-1907', 'malw', 'melbovrne', '73305.107', '07 3846 2530']\n", "2 ['livia', 'ryank', '13-03-1907', 'malw', 'melbuorne', '73305.107', '07 3946 2630']\n", "3 ['ltvia', 'ryan', '13-03-1907', 'maoe', 'melbourne', '73305.197', '07 3046 2530']\n", "4 ['livia', 'ryan', '13-03-1907', 'male', 'melbourne', '73305.107', '07 3946 2530']\n", "\n", "1 ['brock', 'budge', '27-09-1960', 'male', 'perth', '209428.166', '02 5106 4056']\n", "2 ['brocck', 'bud9e', '27-09-1960', 'male', 'pertb', '208428.166', '02 5106 4056']\n", "4 ['brock', 'budge', '27-09-1970', 'male', '', '209428.167', '02 5206 4056']\n", "0 ['brock', 'bwudge', '27-09-2860', '', 'perth', '209428.166', '02 5106 3056']\n", "\n" ] } ], "source": [ "def load_dataset(i):\n", " dataset = []\n", " with open(f\"data/dataset-{i}.csv\") as f:\n", " reader = csv.reader(f)\n", " next(reader) # ignore header\n", " for row in reader:\n", " dataset.append(row[1:])\n", " return dataset\n", "\n", "datasets = list(map(load_dataset, range(1, 6)))\n", "\n", "for group in itertools.islice(groups[\"groups\"][-20:], 20):\n", " for (i, j) in group:\n", " print(i, datasets[i][j])\n", " print()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "Despite the high amount of noise in the data, the Anonlink Entity Service was able to produce a fairly accurate matching. However note Mia Galbraith and Talia Galbraith are most likely not an actual match.\n", "\n", "We may be able to improve on this results by fine-tuning the hashing schema or by changing the threshold." ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Delete the project" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "pycharm": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "204\n" ] } ], "source": [ "r = requests.delete(\n", " f\"{PREFIX}/projects/{project_id}\",\n", " headers={\n", " \"Authorization\": result_token\n", " }\n", ")\n", "print(r.status_code)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.0" }, "pycharm": { "stem_cell": { "cell_type": "raw", "metadata": { "collapsed": false }, "source": [] } } }, "nbformat": 4, "nbformat_minor": 4 }