{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "tzwsxS3y2fxR"
},
"source": [
"# Incorporating Human Feedback into Quality Diversity for Diversified Text-to-Image Generation\n",
"\n",
"by [Li Ding](https://liding.info), author of [Quality Diversity through Human Feedback (QDHF)](https://liding.info/qdhf/); edited and reviewed by [Bryon Tjanaka](https://btjanaka.net/)\n",
"\n",
"*This tutorial is part of the series of pyribs tutorials! See [here](https://docs.pyribs.org/en/latest/tutorials.html) for the list of all tutorials and the order in which they should be read.*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T_iiDTVGEwbw"
},
"source": [
"Foundation models such as large language models (LLMs) and text-to-image generation models ([Stable Diffusion](https://arxiv.org/abs/2112.10752), [DALL·E](https://openai.com/dall-e-3), etc.) in effect compress vast archives of human knowledge into powerful and flexible tools, serving as a foundation for down-stream applications. One mechanism to build upon such foundational knowledge is reinforcement learning from human feedback (RLHF), which can make models both easier to use (by aligning them to human instructions), and more competent (by improving their capabilities based on human preferences).\n",
"\n",
"RLHF is a relatively new paradigm, and its deployments often follow the relatively narrow recipe of maximizing a learned reward model of averaged human preferences over model responses. However, there are drawbacks when it is used to optimize for average human preferences, especially in generative tasks that demand diverse and creative model responses. On the other hand, Quality Diversity (QD) algorithms excel at identifying diverse and high-quality solutions but often rely on manually-crafted diversity metrics, as shown in previous examples, such as [DQD](https://docs.pyribs.org/en/latest/tutorials/tom_cruise_dqd.html).\n",
"\n",
"In this work, we introduce [Quality Diversity through Human Feedback (QDHF)](https://liding.info/qdhf/), which aims to broaden the RLHF recipe to include optimizing for interesting diversity among responses, which is of practical importance for many creative applications such as text-to-image generation. Such increased diversity can also improve optimization for complex and open-ended tasks (through improved exploration), personalization (serving individual rather than average human preference), and fairness (to offset algorithmic biases in gender, ethnicity, and more).\n",
"\n",
"
\n",
"\n",
"![Overview of QDHF.](_static/qdhf-method.png)\n",
"\n",
"
\n",
"\n",
"Our main idea is to derive distinct representations of what humans find interestingly different, and use such diversity representations to support optimization. An overview of our method is shown in the diagram above.\n",
"\n",
"In this tutorial, we implement QDHF for the latent space illumination (LSI) experiment. LSI uses QD algorithms to search for a diverse collection of images in a GAN’s latent space, which has been introduced in the previous [DQD tutorial](https://docs.pyribs.org/en/latest/tutorials/tom_cruise_dqd.html). However, instead of generating images of a celebrity, this LSI pipeline uses [Stable Diffusion](https://arxiv.org/abs/2112.10752) to generate diverse images according to a given text prompt, like the one shown below (*prompt: an image of a bear in a national park*).\n",
"\n",
"![QDHF example](_static/qdhf-example.jpg)\n",
"\n",
"Another difference in this tutorial is that, QDHF uses human feedback to derive meaningful diversity metrics, not by manually specifying attributes such as age and hair length. We use a preference model, [DreamSim](https://dreamsim-nights.github.io/), which is trained on a dataset of human judgment of similarity between images, to source feedback in QDHF optimization. Specifically, we implement QDHF with Map-Elites, and run it with the Stable Diffusion+CLIP pipeline.\n",
"\n",
"*This tutorial assumes that you are familiar with Latent Space Illumination (LSI). If you are not yet familiar with it, we recommend reviewing the [DQD tutorial](https://docs.pyribs.org/en/latest/tutorials/tom_cruise_dqd.html) which generates Tom Cruise Images or the [LSI MNIST tutorial](https://docs.pyribs.org/en/latest/tutorials/lsi_mnist.html) which generates diverse MNIST digits.*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LyDVU3JFAXXP"
},
"source": [
"## Setup\n",
"\n",
"Since Stable Diffusion, CLIP, and DreamSim are fairly large models, you will need a GPU to run this tutorial. On Colab, activate the GPU by clicking \"Runtime\" in the toolbar at the top. Then, click \"Change Runtime Type\", and select \"GPU\".\n",
"\n",
"Below, we check what GPU has been provided. The possible GPUs (at the time of writing) are as follows; factory reset the runtime if you do not have a desired GPU.\n",
"\n",
"* V100 = Excellent (*Available only for Colab Pro users*)\n",
"* P100 = Very Good\n",
"* T4 = Good (*preferred*)\n",
"* K80 = Meh\n",
"* P4 = (*Not Recommended*)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"cellView": "form",
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "cr-94yAuzYns",
"outputId": "79f481b6-814d-47de-8e5e-ef921d0080e1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GPU 0: Tesla T4 (UUID: GPU-70c83cee-934f-c99a-60ae-5866505757af)\n"
]
}
],
"source": [
"!nvidia-smi -L"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KN9uICZRAXXV"
},
"source": [
"Now, we install various libraries, including pyribs, CLIP, Stable Diffusion, and DreamSim. This cell will take around 5-10 minutes to run since it involves downloading multiple large files."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oJZEFUAfXX2z",
"tags": []
},
"outputs": [],
"source": [
"%pip install ribs[visualize] torch torchvision git+https://github.com/openai/CLIP tqdm dreamsim diffusers accelerate"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "sdJxWD7KJL98"
},
"outputs": [],
"source": [
"import time\n",
"import numpy as np\n",
"from tqdm import tqdm, trange"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "w0tFClwd_thl"
},
"source": [
"Let's first set up CUDA for PyTorch. We use float16 to save GPU memories for large models."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "fDhs1R66_sRL",
"outputId": "4e6cfbf8-474f-434b-e636-d37e0af6052a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Torch device: cuda\n",
"Torch dtype: torch.float16\n"
]
}
],
"source": [
"import torch\n",
"\n",
"DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"torch.cuda.empty_cache()\n",
"print(\"Torch device:\", DEVICE)\n",
"\n",
"# Use float16 for GPU, float32 for CPU.\n",
"TORCH_DTYPE = torch.float16 if torch.cuda.is_available() else torch.float32\n",
"print(\"Torch dtype:\", TORCH_DTYPE)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gQDY-QKW0Xlg"
},
"source": [
"## Preliminaries: CLIP, Stable Diffusion, and DreamSim\n",
"\n",
"Below we set up CLIP, Stable Diffusion, and DreamSim. The code for these models is adopted from the [DQD tutorial](https://docs.pyribs.org/en/latest/tutorials/tom_cruise_dqd.html), the [DreamSim demo](https://colab.research.google.com/drive/1taEOMzFE9g81D9AwH27Uhy2U82tQGAVI?usp=sharing), and the [Hugging Face Stable Diffusion example](https://huggingface.co/stabilityai/stable-diffusion-2-1). Each cell may take around 3 minutes to run since it involves downloading multiple large files.\n",
"\n",
"*Since this section is focused on the LSI pipeline rather than on pyribs, feel free to skim past and come back to it later.*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sAimcHd1Gh-J"
},
"source": [
"### CLIP for Connecting Text and Images"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "29vaSPaNYEt2",
"outputId": "b814613d-011d-41d8-86f3-95bd2dfc3153"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████| 338M/338M [00:03<00:00, 109MiB/s]\n"
]
}
],
"source": [
"import clip\n",
"\n",
"# This tutorial uses ViT-B/32, you may use other checkpoints depending on your resources and need.\n",
"CLIP_MODEL, CLIP_PREPROCESS = clip.load(\"ViT-B/32\", device=DEVICE)\n",
"CLIP_MODEL.eval()\n",
"for p in CLIP_MODEL.parameters():\n",
" p.requires_grad_(False)\n",
"\n",
"\n",
"def compute_clip_scores(imgs, text, return_clip_features=False):\n",
" \"\"\"Computes CLIP scores for a batch of images and a given text prompt.\"\"\"\n",
" img_tensor = torch.stack([CLIP_PREPROCESS(img) for img in imgs]).to(DEVICE)\n",
" tokenized_text = clip.tokenize([text]).to(DEVICE)\n",
" img_logits, _text_logits = CLIP_MODEL(img_tensor, tokenized_text)\n",
" img_logits = img_logits.detach().cpu().numpy().astype(np.float32)[:, 0]\n",
" img_logits = 1 / img_logits * 100\n",
" # Remap the objective from minimizing [0, 10] to maximizing [0, 100]\n",
" img_logits = (10.0 - img_logits) * 10.0\n",
"\n",
" if return_clip_features:\n",
" clip_features = CLIP_MODEL.encode_image(img_tensor).to(TORCH_DTYPE)\n",
" return img_logits, clip_features\n",
" else:\n",
" return img_logits"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FU0oQar1JL98"
},
"source": [
"### DreamSim for Simulating Human Feedback"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3M_WH715JL98",
"outputId": "d6c30bfd-ab26-4bb7-c62b-8ff52bb2146f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading checkpoint\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 313M/313M [00:19<00:00, 16.5MB/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Unzipping...\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py:143: UserWarning: fan_in_fan_out is set to True but the target module is not a Conv1D. Setting fan_in_fan_out to False.\n",
" warnings.warn(\n"
]
}
],
"source": [
"from dreamsim import dreamsim\n",
"\n",
"DREAMSIM_MODEL, DREAMSIM_PREPROCESS = dreamsim(\n",
" pretrained=True, dreamsim_type=\"open_clip_vitb32\", device=DEVICE\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_zQChflqEwbz"
},
"source": [
"### Stable Diffusion for Generating Images\n",
"By default, we use the [`miniSD-diffusers` checkpoint](https://huggingface.co/lambdalabs/miniSD-diffusers), which generates images at 256x256 for faster inference and less GPU memory usage. Please switch to the `stable-diffusion-2-1-base` checkpoint to generate images with higher resolution and quality."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 743,
"referenced_widgets": [
"b948db64778e43aa94bf2a5a846b3b55",
"7c0a6dcb69fa47c692f0f4a3e43cc9bf",
"ad489828f18243388da6688fc4c07906",
"6fbdfd9bf76e43c69135cf77076513e8",
"7849584ed5364c7eb2790f2ec0fc3a5c",
"5efbd8293117470493cc8febf28a7b35",
"7c86a10243a84196bec278153477aaa3",
"93dd387b048445f586064e02ed6836a0",
"960c4ce00f3149efa35c2277fb2c53cf",
"dcaaee9e77ed49a1902cd450ccbf8647",
"abdee66daa4c4aff9a49de0eba879a80",
"ab4a52f585ea48f48d2f9a49891d40e7",
"f3cae0e30d564b0a94a53d124e4aba8f",
"a56878875a844c30ae508685d8467bf1",
"c7f4f0da52ba4160acb6a0b4e97f87e4",
"06a98c2339a048a097ee41ab7037afce",
"97d90449e6004102b3a8231f5a5d6b45",
"0e61ceb59eae4643960b8bc9175019e8",
"21e010eb370b48a49f2da6b3a6f83cfd",
"be46a6e4e84d46319e41d0f8c86b115e",
"5536fa47db1b400994b4c1541b56c28b",
"35250b317d384c9a977c41f49cc46a55",
"e8f2f34841e0475ea81e58b42b0a0efa",
"fe7d2fa6160f48b9b19f83f03d31cc11",
"b15e169f1ed649e1a0f181e9f06cae6e",
"f47380dfd50e4f4991d8ad6ba8765a39",
"21e65ee88bb54141874d6384ec6a5f24",
"6cbff98510f9470493a114ae943e5433",
"d176a24ab134471b9b54596ba7103e81",
"9dbc29d7961d49268376ad329c9567f4",
"f9f8cb2fc872405689213923d69389ae",
"ecb69d942aba45b5843d91ae1898fd75",
"cab87b814ea340589c4c1d7cb1a73c6c",
"60c87feea4a14513be45869c305d5cf0",
"8e397d1c8186452588648b6f4700fe64",
"6889622737684bd886ab85855f213d27",
"621ad1476f484922b2c8773daf5c5dff",
"b6d4dcfcf807447aa6b5974f15106746",
"664aecd728594a088fcd14d51d042f86",
"5f21d9e8e1c94a10b3d029ba93d2e29b",
"9c44cb1672414dc2b5b04817ce53f5d5",
"5b66227d80714db28e77459bad3d30bf",
"d1cedf1c7baa4dc89b2bc6eeb01c4b86",
"b9d1e22fca1a4cba8b4f17f26bec2c83",
"e83d17232642486bbeff20554154dc9f",
"6692c51c8f074aebabc3a64495e43df0",
"8f59061600da43eb9d7f6657c1488907",
"4f9a0595c776456aaa649d549841345e",
"996c500d88464912a7737ab3f44c2b6c",
"f5ed54f919e04ebe8c37de928c9d54a2",
"57b681bbf1f84a1196d5221ee41f33c1",
"eca245b9820d43229926ba18c173158a",
"8353ec8f473943a08f31809cef8d1f67",
"426d330fea35486fa1a8e2511b10364f",
"96566812f8ff484e937ab0af3cf84fb5",
"ca5303d9d611410ba12b3a9d6e166df7",
"c3564be7eb9c4b57ba1a90e524332f03",
"2c37498eae5e4108b15117064d2dee4b",
"2bb3af22dd0c4fc28d7058e7a15d2f14",
"0f3699812a3c4e55b8e4bf3b0399b405",
"14c992efe9ce419cae57e8647323991a",
"ce0f5d034abe4261b45af798e370cff1",
"e983f26d5c1e4470b472119960726106",
"649b169bb48f49b295da82048477d163",
"ffc0c3fa2c53482ca470bae681982407",
"aa5215660a094c91bafba7469953ca3a",
"01ada1f18f564cf6a9521b32ee2e98ea",
"2920ce76d45f4eba936e35c3d9a0fe67",
"aa6474f724124c6dab5144858a97d9ce",
"d3ffddd22b4c4faf90905fec871b4004",
"344df76ef1ce4b479b3861dabf961a62",
"137d2811d006415caca4cc11f3cf4465",
"a8c0d7f5e46e42dc8a54999558f81906",
"69330c1dc7c948a5ac3a0a1ced914315",
"ee184498dc2e4d549d4782033741dcf0",
"ac8082eb67f84c76a4699b8295c36088",
"71df24c27d9446f08fb86ad184b2a4b3",
"2c0a82bc98ad45d3be0b1d67caadefa1",
"0f5bf02254ab4cf0b5ee71a0b1627b15",
"9e449af4de8e4a62a472f48b123c6343",
"98f2edf6913c41c8870f05f8b4c695c1",
"964b5c1c464645e983f5e66d85be1bd2",
"dec0045f1d6d4671ac81b4599d0b4a65",
"cb6ed4457962453496b025b31af87249",
"b27b412d000b4ce783fd070575a04228",
"13f9aa64762b41548fa85f2c7fc2ecfc",
"c295430c07dd40018a212e4cf35b2d2b",
"20d077b95a024859a11fb7b89938570e",
"005a54fd9a52492e8b39404049d77e6d",
"24a6a110d7f54953b414a663f4d17f47",
"f5572623b0b14071bf02d577df99f9c4",
"67c072a2c82745e3a89abcb616064e4d",
"64f3cebfb8f546779d8cd5a5e8115c74",
"0c7c91398330449f9e576f0165801343",
"99a038c9b0c44579a5a5265e2a5a6193",
"9bef9a26de87438784460f1d89c00fca",
"e297f104e8f24c68a5f0bf6c643b3d5a",
"2ffbee790510429c8347ba3f16426386",
"ad1b7e443f974b50b884b65dcfbc077d",
"e7afe6281e49423ca4e363675410b538",
"77ae39a523db4bcdb027c0bc82637e9d",
"e668b54d8520437a94debd6edea0ada6",
"527c0e8b93d94f5184c84c9a9bcd3494",
"41ca22f3a00f444faa03c1381737f438",
"11b8a7fea0004f39b135c575b1ef7b0e",
"8eda4cb19a7f467c8370ba22f996d5c4",
"e061c04c01bb4e48834341165888b3a8",
"2bc0031f0b8b44268399c3a79470e4c0",
"40e037a25eb64a729e1afcd8568b51de",
"6693276e107e4686af6881f1634dc522",
"edcec18a130d4931b5ed67df7fd01d36",
"07c16355845d4d97922e9ee5081e87fa",
"a9dc3e1e17bc42f3800757ce6803abde",
"00b0ad8c32f6498cb055ff7b2a61512a",
"38063a7b8a6348f58a05260f102e2acd",
"d4bfb68dda4c4d70ad2a9d18fc972187",
"be2e6d851df54a16bde86f970d6faf29",
"63bc7fcab1884c41baef33cbae16a9b1",
"762b0534630d4d739aa4452665236d9e",
"70fbffbaf00542d4bd245f4290f2a852",
"ca6bbc9532b54319bb27b7fbbe5074d2",
"72e9f89e30b6404d82aaa76df37acafd",
"d248678abee14b9b8fd39bfbb226bdba",
"24c02c609aaa44f6a47a5b2a512f84f3",
"77811f8d31be4292a9710a91da400e75",
"7f665f5b5a1f4b0aa2d6d302c8923f70",
"d4c07ca96f8548e7b4ea8209dfa17568",
"fbf837ec590a460492f45c2d4ef7e56a",
"cb8e1d2c32524e6eabb941fdb4698964",
"a9a959dc0399482a9029b1fe658507e9",
"4a817e30dd1a4fac9bd7df56b0b0b5e4",
"fd7174e1bbdb48da8f47412f2a461c8c",
"7bda51d039f44963a7e0a6c73cd5e21e",
"53a7799096d54e9baaaf65408f0a7a4f",
"2eac4fed1841457e95140623eb546b75",
"da8e1f01b5d44b8aa461c2b07b9f715a",
"101c54d9c70448c4988c311504cdf597",
"f270ad1a5b2747d78232985947da721c",
"0816bd249e3c4dc48bb15f4bb33fffe9",
"e7af38b22bf348a58957dd130f4684ff",
"70b33ebc079e4c83b8bee583e75a168a",
"e7fd96799a4941c79e5447b3a46e7fea",
"39808ebab88341f0a83c6e56aedda387",
"09df53a045f8424db21a5ca73e851277",
"8c144f078a3e4664862070b0a6dde625",
"038865d0b53846ebb0ca8021a375252b",
"16b07f6531a448ff9a332791de00bc11",
"3825a9127ecb47789c14e86028c3d7ea",
"ed70d31611d649d2acebb2504d5d00f7",
"056c5709a0c349f0999df698da2051f9",
"89e9eda48a744bf0815f54d2d3da6ea3",
"9e7d37b28d904727a28d4eb05686cb53",
"ab202fcb10c74743ad0e15f5c7e52518",
"bd56528c449f42c6b14da0282f048cc8",
"c337c0459a744599bd42007edea040d6",
"6c5e97aaa0da402dab077b720513a0a0",
"467ba2cd992e4706928539565d406b8b",
"fe4eca7913b747ecadac9c898fa0756f",
"5fb987872561431fb56b66dce8f4b676",
"bfa844bfa74543b7929bec7038411988",
"bf0b4852ebc7459b97c69dad3a0f6325",
"787e0dfafea7453c9b47a758676db5fd",
"d3630d1e03d949018caf7ac3cf4cd525",
"6a2415b2130f4578a23bccf4e2509ee9",
"6a6a241616884f37a8d07c8bc2ab22b5"
]
},
"id": "9aQfZ7HwJL98",
"outputId": "ef623211-3f0d-488b-aa07-6783a7cff226"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
" torch.utils._pytree._register_pytree_node(\n",
"/usr/local/lib/python3.10/dist-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
" torch.utils._pytree._register_pytree_node(\n",
"/usr/local/lib/python3.10/dist-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
" torch.utils._pytree._register_pytree_node(\n",
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
"The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
"To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
"You will be able to reuse this secret in all of your notebooks.\n",
"Please note that authentication is recommended but still optional to access public models or datasets.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b948db64778e43aa94bf2a5a846b3b55",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model_index.json: 0%| | 0.00/541 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"unet/diffusion_pytorch_model.safetensors not found\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ab4a52f585ea48f48d2f9a49891d40e7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Fetching 13 files: 0%| | 0/13 [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e8f2f34841e0475ea81e58b42b0a0efa",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"scheduler/scheduler_config.json: 0%| | 0.00/308 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "60c87feea4a14513be45869c305d5cf0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer/tokenizer_config.json: 0%| | 0.00/806 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e83d17232642486bbeff20554154dc9f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer/merges.txt: 0%| | 0.00/525k [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ca5303d9d611410ba12b3a9d6e166df7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer/vocab.json: 0%| | 0.00/1.06M [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "01ada1f18f564cf6a9521b32ee2e98ea",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"text_encoder/config.json: 0%| | 0.00/612 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2c0a82bc98ad45d3be0b1d67caadefa1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"(…)ature_extractor/preprocessor_config.json: 0%| | 0.00/342 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "005a54fd9a52492e8b39404049d77e6d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer/special_tokens_map.json: 0%| | 0.00/472 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e7afe6281e49423ca4e363675410b538",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"unet/config.json: 0%| | 0.00/776 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "edcec18a130d4931b5ed67df7fd01d36",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"pytorch_model.bin: 0%| | 0.00/492M [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "72e9f89e30b6404d82aaa76df37acafd",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vae/config.json: 0%| | 0.00/547 [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7bda51d039f44963a7e0a6c73cd5e21e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"diffusion_pytorch_model.bin: 0%| | 0.00/3.44G [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "09df53a045f8424db21a5ca73e851277",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"diffusion_pytorch_model.bin: 0%| | 0.00/335M [00:00, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c337c0459a744599bd42007edea040d6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading pipeline components...: 0%| | 0/6 [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from diffusers import StableDiffusionPipeline\n",
"\n",
"IMG_WIDTH = 256\n",
"IMG_HEIGHT = 256\n",
"SD_IN_HEIGHT = 32\n",
"SD_IN_WIDTH = 32\n",
"SD_CHECKPOINT = \"lambdalabs/miniSD-diffusers\"\n",
"\n",
"# IMG_WIDTH = 512\n",
"# IMG_HEIGHT = 512\n",
"# SD_IN_HEIGHT = 64\n",
"# SD_IN_WIDTH = 64\n",
"# SD_CHECKPOINT = \"stabilityai/stable-diffusion-2-1-base\"\n",
"\n",
"BATCH_SIZE = 4\n",
"SD_IN_CHANNELS = 4\n",
"SD_IN_SHAPE = (\n",
" BATCH_SIZE,\n",
" SD_IN_CHANNELS,\n",
" SD_IN_HEIGHT,\n",
" SD_IN_WIDTH,\n",
")\n",
"\n",
"SDPIPE = StableDiffusionPipeline.from_pretrained(\n",
" SD_CHECKPOINT,\n",
" torch_dtype=TORCH_DTYPE,\n",
" safety_checker=None, # For faster inference.\n",
" requires_safety_checker=False,\n",
")\n",
"\n",
"SDPIPE.set_progress_bar_config(disable=True)\n",
"SDPIPE = SDPIPE.to(DEVICE)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HIv5yg0aEwbz"
},
"source": [
"## Aligning Diversity Metrics with Human Preferences\n",
"\n",
"Next, we define a latent projection model to characterize diversity from image features through constrastive learning from human feedback.\n",
"The `DivProj` model projects the image features (e.g., CLIP features with 512-d) onto a low-dimensional latent space (e.g., 2-d), where each dimension represents a numeric diversity metric.\n",
"An intuitive example is that the model may learn \"object size\" as a metric, where low values mean the object is small and high values mean it is large."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "5c7xVDveJL99"
},
"outputs": [],
"source": [
"from torch import nn\n",
"\n",
"\n",
"class DivProj(nn.Module):\n",
" def __init__(self, input_dim, latent_dim=2):\n",
" super().__init__()\n",
" self.proj = nn.Sequential(\n",
" nn.Linear(in_features=input_dim, out_features=latent_dim),\n",
" )\n",
"\n",
" def forward(self, x):\n",
" \"\"\"Get diversity representations.\"\"\"\n",
" x = self.proj(x)\n",
" return x\n",
"\n",
" def calc_dis(self, x1, x2):\n",
" \"\"\"Calculate diversity distance as (squared) L2 distance.\"\"\"\n",
" x1 = self.forward(x1)\n",
" x2 = self.forward(x2)\n",
" return torch.sum(torch.square(x1 - x2), -1)\n",
"\n",
" def triplet_delta_dis(self, ref, x1, x2):\n",
" \"\"\"Calculate delta distance comparing x1 and x2 to ref.\"\"\"\n",
" x1 = self.forward(x1)\n",
" x2 = self.forward(x2)\n",
" ref = self.forward(ref)\n",
" return (torch.sum(torch.square(ref - x1), -1) -\n",
" torch.sum(torch.square(ref - x2), -1))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bV9RVsOAI__y"
},
"source": [
"### Constrastive Learning for Diversity Projection\n",
"\n",
"For contrastive learning, we use a triplet loss to train the diversity projection model. This loss encourages the model to project similar images close to each other in the latent space, and dissimilar images far from each other. The preference labels are given by the DreamSim model, which is pre-trained on the human feedback on image similarity, so we will use it as the source of feedback here.\n",
"\n",
"To avoid redundancy in computation, we first extract DreamSim features from the images, and then calculate the cosine similarity of DreamSim features. This is equivalent to using the DreamSim model directly to calculate the similarity.\n",
"\n",
"The whole `fit_div_proj` process works as follows: The input data is with shape of `(n, 3, image_size, image_size)`, where n is the number of triplets. The second dimension of 3 represents the reference, positive, and negative images. We split the data into traning and validation sets, and then train the diversity projection model with the triplet loss. The output is a diversity projection model that projects image features onto a low-dimensional latent space, where each dimension represents a numeric diversity metric."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "OSLN8j9ZI__y"
},
"outputs": [],
"source": [
"# Triplet loss with margin 0.05.\n",
"# The binary preference labels are scaled to y = 1 or -1 for the loss, where y = 1 means x2 is more similar to ref than x1.\n",
"loss_fn = lambda y, delta_dis: torch.max(\n",
" torch.tensor([0.0]).to(DEVICE), 0.05 - (y * 2 - 1) * delta_dis\n",
").mean()\n",
"\n",
"\n",
"def fit_div_proj(inputs, dreamsim_features, latent_dim, batch_size=32):\n",
" \"\"\"Trains the DivProj model on ground-truth labels.\"\"\"\n",
" t = time.time()\n",
" model = DivProj(input_dim=inputs.shape[-1], latent_dim=latent_dim)\n",
" model.to(DEVICE)\n",
"\n",
" optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n",
"\n",
" n_pref_data = inputs.shape[0]\n",
" ref = inputs[:, 0]\n",
" x1 = inputs[:, 1]\n",
" x2 = inputs[:, 2]\n",
"\n",
" n_train = int(n_pref_data * 0.75)\n",
" n_val = n_pref_data - n_train\n",
"\n",
" # Split data into train and val.\n",
" ref_train = ref[:n_train]\n",
" x1_train = x1[:n_train]\n",
" x2_train = x2[:n_train]\n",
" ref_val = ref[n_train:]\n",
" x1_val = x1[n_train:]\n",
" x2_val = x2[n_train:]\n",
"\n",
" # Split DreamSim features into train and val.\n",
" ref_dreamsim_features = dreamsim_features[:, 0]\n",
" x1_dreamsim_features = dreamsim_features[:, 1]\n",
" x2_dreamsim_features = dreamsim_features[:, 2]\n",
" ref_gt_train = ref_dreamsim_features[:n_train]\n",
" x1_gt_train = x1_dreamsim_features[:n_train]\n",
" x2_gt_train = x2_dreamsim_features[:n_train]\n",
" ref_gt_val = ref_dreamsim_features[n_train:]\n",
" x1_gt_val = x1_dreamsim_features[n_train:]\n",
" x2_gt_val = x2_dreamsim_features[n_train:]\n",
"\n",
" val_acc = []\n",
" n_iters_per_epoch = max((n_train) // batch_size, 1)\n",
" for epoch in range(200):\n",
" for _ in range(n_iters_per_epoch):\n",
" optimizer.zero_grad()\n",
"\n",
" idx = np.random.choice(n_train, batch_size)\n",
" batch_ref = ref_train[idx].float()\n",
" batch1 = x1_train[idx].float()\n",
" batch2 = x2_train[idx].float()\n",
"\n",
" # Get delta distance from model.\n",
" delta_dis = model.triplet_delta_dis(batch_ref, batch1, batch2)\n",
"\n",
" # Get preference labels from DreamSim features.\n",
" gt_dis = torch.nn.functional.cosine_similarity(\n",
" ref_gt_train[idx], x2_gt_train[idx], dim=-1\n",
" ) - torch.nn.functional.cosine_similarity(\n",
" ref_gt_train[idx], x1_gt_train[idx], dim=-1\n",
" )\n",
" gt = (gt_dis > 0).to(TORCH_DTYPE)\n",
"\n",
" loss = loss_fn(gt, delta_dis)\n",
" loss.backward()\n",
" optimizer.step()\n",
"\n",
" # Validate.\n",
" n_correct = 0\n",
" n_total = 0\n",
" with torch.no_grad():\n",
" idx = np.arange(n_val)\n",
" batch_ref = ref_val[idx].float()\n",
" batch1 = x1_val[idx].float()\n",
" batch2 = x2_val[idx].float()\n",
" delta_dis = model.triplet_delta_dis(batch_ref, batch1, batch2)\n",
" pred = delta_dis > 0\n",
" gt_dis = torch.nn.functional.cosine_similarity(\n",
" ref_gt_val[idx], x2_gt_val[idx], dim=-1\n",
" ) - torch.nn.functional.cosine_similarity(\n",
" ref_gt_val[idx], x1_gt_val[idx], dim=-1\n",
" )\n",
" gt = gt_dis > 0\n",
" n_correct += (pred == gt).sum().item()\n",
" n_total += len(idx)\n",
"\n",
" acc = n_correct / n_total\n",
" val_acc.append(acc)\n",
"\n",
" # Early stopping if val_acc does not improve for 10 epochs.\n",
" if epoch > 10 and np.mean(val_acc[-10:]) < np.mean(val_acc[-11:-1]):\n",
" break\n",
"\n",
" print(\n",
" f\"{np.round(time.time()- t, 1)}s ({epoch+1} epochs) | DivProj (n={n_pref_data}) fitted with val acc.: {acc}\"\n",
" )\n",
"\n",
" return model.to(TORCH_DTYPE), acc"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_qwFTWniEwbz"
},
"source": [
"### Helper Functions to Compute Objectives and Measures"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "96S2T01tYElA"
},
"outputs": [],
"source": [
"def compute_diversity_measures(clip_features, diversity_model):\n",
" with torch.no_grad():\n",
" measures = diversity_model(clip_features).detach().cpu().numpy()\n",
" return measures\n",
"\n",
"\n",
"def evaluate_lsi(\n",
" latents,\n",
" prompt,\n",
" return_features=False,\n",
" diversity_model=None,\n",
"):\n",
" \"\"\"Evaluates the objective of LSI for a batch of latents and a given text prompt.\"\"\"\n",
"\n",
" images = SDPIPE(\n",
" prompt,\n",
" num_images_per_prompt=latents.shape[0],\n",
" latents=latents,\n",
" # num_inference_steps=1, # For testing.\n",
" ).images\n",
"\n",
" objs, clip_features = compute_clip_scores(\n",
" images,\n",
" prompt,\n",
" return_clip_features=True,\n",
" )\n",
"\n",
" images = torch.cat([DREAMSIM_PREPROCESS(img) for img in images]).to(DEVICE)\n",
" dreamsim_features = DREAMSIM_MODEL.embed(images)\n",
"\n",
" if diversity_model is not None:\n",
" measures = compute_diversity_measures(clip_features, diversity_model)\n",
" else:\n",
" measures = None\n",
"\n",
" if return_features:\n",
" return objs, measures, clip_features, dreamsim_features\n",
" else:\n",
" return objs, measures"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-zP5FDJzEwb0"
},
"source": [
"## QDHF with pyribs\n",
"\n",
"To use QDHF to generate diverse collections of images, we will use the MAP-Elites algorithm, where the diversity metrics are progressively trained and fine-tuned during optimization."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aggNYej2Ewb0"
},
"source": [
"### Algorithm Setup\n",
"\n",
"To set up the MAP-Elites instance for QDHF, we will set up the following components. First, we create the [`GridArchive`](https://docs.pyribs.org/en/latest/api/ribs.archives.GridArchive.html). The archive is 20x20 and stores Stable Diffusion latent vectors. The dimensions of vectors are (BATCH, 4, H, W) where H=W=32 if generating 256x256 images with `miniSD`, or 64 for 512x512 images with `stable-diffusion-2-1-base`. Next, we create a [`GaussianEmitter`](https://docs.pyribs.org/en/latest/api/ribs.emitters.GaussianEmitter.html) that generates solutions by mutating existing archive solutions with Gaussian noise. Finally, we set up a [`Scheduler`](https://docs.pyribs.org/en/latest/api/ribs.schedulers.Scheduler.html) to combine everything together."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "zGvAiVz7YEWK"
},
"outputs": [],
"source": [
"from ribs.archives import GridArchive\n",
"from ribs.schedulers import Scheduler\n",
"from ribs.emitters import GaussianEmitter\n",
"\n",
"GRID_SIZE = (20, 20)\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"torch.manual_seed(SEED)\n",
"\n",
"def tensor_to_list(tensor):\n",
" sols = tensor.detach().cpu().numpy().astype(np.float32)\n",
" return sols.reshape(sols.shape[0], -1)\n",
"\n",
"\n",
"def list_to_tensor(list_):\n",
" sols = np.array(list_).reshape(\n",
" len(list_), 4, SD_IN_HEIGHT, SD_IN_WIDTH\n",
" ) # Hard-coded for now.\n",
" return torch.tensor(sols, dtype=TORCH_DTYPE, device=DEVICE)\n",
"\n",
"\n",
"def create_scheduler(\n",
" sols,\n",
" objs,\n",
" clip_features,\n",
" diversity_model,\n",
" seed=None,\n",
"):\n",
" measures = compute_diversity_measures(clip_features, diversity_model)\n",
" archive_bounds = np.array(\n",
" [np.quantile(measures, 0.01, axis=0), np.quantile(measures, 0.99, axis=0)]\n",
" ).T\n",
"\n",
" sols = tensor_to_list(sols)\n",
"\n",
" # Set up archive.\n",
" archive = GridArchive(\n",
" solution_dim=len(sols[0]), dims=GRID_SIZE, ranges=archive_bounds, seed=SEED\n",
" )\n",
"\n",
" # Add initial solutions to the archive.\n",
" archive.add(sols, objs, measures)\n",
"\n",
" # Set up the GaussianEmitter.\n",
" emitters = [\n",
" GaussianEmitter(\n",
" archive=archive,\n",
" sigma=0.1,\n",
" initial_solutions=archive.sample_elites(BATCH_SIZE)[\"solution\"],\n",
" batch_size=BATCH_SIZE,\n",
" seed=SEED,\n",
" )\n",
" ]\n",
"\n",
" # Return the archive and scheduler.\n",
" return archive, Scheduler(archive, emitters)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-VZRvHPGEwb0"
},
"source": [
"## LSI with QDHF\n",
"\n",
"Having set up the necessary components above, we now run QDHF to search for images of a given prompt. The default prompt is \"an astronaut riding a horse on mars\".\n",
"\n",
"### Training\n",
"\n",
"The execution loop is similar to the experiments in previous tutorials. The main difference is that when the iteration is in `update_schedule`, we will update the QD archive by fine-tuning the diversity metrics and adding current solutions to the archive again with regard to their updated diversity measures.\n",
"\n",
"The total number of iterations is set to be 200, and it should take ~45min to run on a GPU."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "m4-MisNWJL99",
"outputId": "fc2c6d2c-4b24-401c-fc84-2e462348ca43"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" 0%| | 0/200 [00:00, ?it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initializing archive and diversity projection.\n",
"1.5s (21 epochs) | DivProj (n=1000) fitted with val acc.: 0.764\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 0%| | 1/200 [05:56<19:43:49, 356.93s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 24.97 Coverage: 34.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 1%| | 2/200 [06:03<8:18:25, 151.04s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 25.15 Coverage: 34.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 2%|▏ | 3/200 [06:10<4:39:50, 85.23s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 25.52 Coverage: 35.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 2%|▏ | 4/200 [06:17<2:57:26, 54.32s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.06 Coverage: 36.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 2%|▎ | 5/200 [06:24<2:00:59, 37.23s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.06 Coverage: 36.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 3%|▎ | 6/200 [06:31<1:27:03, 26.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.42 Coverage: 36.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 4%|▎ | 7/200 [06:38<1:05:36, 20.39s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.78 Coverage: 37.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 4%|▍ | 8/200 [06:45<51:34, 16.12s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 27.15 Coverage: 37.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 4%|▍ | 9/200 [06:52<42:07, 13.24s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 27.69 Coverage: 38.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 5%|▌ | 10/200 [06:59<35:42, 11.28s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 27.87 Coverage: 38.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 6%|▌ | 11/200 [07:06<31:16, 9.93s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.23 Coverage: 39.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 6%|▌ | 12/200 [07:13<28:12, 9.00s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.41 Coverage: 39.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 6%|▋ | 13/200 [07:19<26:04, 8.36s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.6 Coverage: 39.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 7%|▋ | 14/200 [07:26<24:34, 7.93s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.96 Coverage: 40.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 8%|▊ | 15/200 [07:33<23:29, 7.62s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.97 Coverage: 40.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 8%|▊ | 16/200 [07:40<22:41, 7.40s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.51 Coverage: 40.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 8%|▊ | 17/200 [07:47<22:04, 7.24s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.69 Coverage: 41.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 9%|▉ | 18/200 [07:54<21:38, 7.14s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.06 Coverage: 41.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 10%|▉ | 19/200 [08:01<21:18, 7.06s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.25 Coverage: 41.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 10%|█ | 20/200 [08:08<21:01, 7.01s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.43 Coverage: 42.0\n",
"Updating archive and diversity projection.\n",
"1.5s (13 epochs) | DivProj (n=2000) fitted with val acc.: 0.752\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 10%|█ | 21/200 [13:05<4:41:01, 94.20s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 23.55 Coverage: 32.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 11%|█ | 22/200 [13:12<3:21:43, 68.00s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 23.73 Coverage: 32.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 12%|█▏ | 23/200 [13:19<2:26:29, 49.66s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 24.09 Coverage: 33.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 12%|█▏ | 24/200 [13:26<1:48:02, 36.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 24.45 Coverage: 33.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 12%|█▎ | 25/200 [13:33<1:21:14, 27.86s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 24.82 Coverage: 34.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 13%|█▎ | 26/200 [13:40<1:02:33, 21.57s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 25.19 Coverage: 34.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 14%|█▎ | 27/200 [13:47<49:30, 17.17s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 25.37 Coverage: 35.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 14%|█▍ | 28/200 [13:53<40:24, 14.10s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 25.55 Coverage: 35.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 14%|█▍ | 29/200 [14:00<34:01, 11.94s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 25.92 Coverage: 35.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 15%|█▌ | 30/200 [14:07<29:33, 10.43s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.48 Coverage: 36.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 16%|█▌ | 31/200 [14:14<26:25, 9.38s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.99 Coverage: 37.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 16%|█▌ | 32/200 [14:21<24:11, 8.64s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 27.34 Coverage: 37.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 16%|█▋ | 33/200 [14:28<22:36, 8.12s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.04 Coverage: 38.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 17%|█▋ | 34/200 [14:35<21:27, 7.75s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.57 Coverage: 39.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 18%|█▊ | 35/200 [14:42<20:38, 7.50s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.95 Coverage: 40.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 18%|█▊ | 36/200 [14:49<20:01, 7.33s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.13 Coverage: 40.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 18%|█▊ | 37/200 [14:56<19:33, 7.20s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.87 Coverage: 41.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 19%|█▉ | 38/200 [15:03<19:13, 7.12s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.23 Coverage: 41.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 20%|█▉ | 39/200 [15:10<18:56, 7.06s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.42 Coverage: 42.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 20%|██ | 40/200 [15:16<18:42, 7.01s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.77 Coverage: 42.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 20%|██ | 41/200 [15:23<18:29, 6.98s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.96 Coverage: 42.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 21%|██ | 42/200 [15:30<18:18, 6.95s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.33 Coverage: 43.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 22%|██▏ | 43/200 [15:37<18:09, 6.94s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.5 Coverage: 43.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 22%|██▏ | 44/200 [15:44<18:02, 6.94s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.68 Coverage: 43.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 22%|██▎ | 45/200 [15:51<17:53, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.21 Coverage: 44.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 23%|██▎ | 46/200 [15:58<17:44, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.74 Coverage: 45.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 24%|██▎ | 47/200 [16:05<17:37, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.75 Coverage: 45.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 24%|██▍ | 48/200 [16:12<17:29, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.29 Coverage: 46.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 24%|██▍ | 49/200 [16:19<17:22, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.81 Coverage: 46.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 25%|██▌ | 50/200 [16:25<17:16, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.55 Coverage: 47.75\n",
"Updating archive and diversity projection.\n",
"2.3s (14 epochs) | DivProj (n=3000) fitted with val acc.: 0.76\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 26%|██▌ | 51/200 [22:04<4:24:18, 106.43s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 25.7 Coverage: 35.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 26%|██▌ | 52/200 [22:11<3:08:52, 76.57s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.05 Coverage: 36.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 26%|██▋ | 53/200 [22:18<2:16:22, 55.67s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.41 Coverage: 36.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 27%|██▋ | 54/200 [22:25<1:39:53, 41.05s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 26.59 Coverage: 36.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 28%|██▊ | 55/200 [22:32<1:14:26, 30.80s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 27.14 Coverage: 37.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 28%|██▊ | 56/200 [22:39<56:43, 23.63s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 27.68 Coverage: 38.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 28%|██▊ | 57/200 [22:46<44:21, 18.61s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 27.86 Coverage: 38.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 29%|██▉ | 58/200 [22:52<35:43, 15.09s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.41 Coverage: 39.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 30%|██▉ | 59/200 [22:59<29:41, 12.63s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.97 Coverage: 40.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 30%|███ | 60/200 [23:06<25:27, 10.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.34 Coverage: 40.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 30%|███ | 61/200 [23:13<22:28, 9.70s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.87 Coverage: 41.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 31%|███ | 62/200 [23:20<20:22, 8.86s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.59 Coverage: 42.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 32%|███▏ | 63/200 [23:27<18:52, 8.26s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.95 Coverage: 42.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 32%|███▏ | 64/200 [23:34<17:47, 7.85s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.31 Coverage: 43.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 32%|███▎ | 65/200 [23:41<17:00, 7.56s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.5 Coverage: 43.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 33%|███▎ | 66/200 [23:48<16:25, 7.36s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.06 Coverage: 44.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 34%|███▎ | 67/200 [23:54<16:00, 7.22s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.6 Coverage: 45.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 34%|███▍ | 68/200 [24:01<15:41, 7.13s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.6 Coverage: 45.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 34%|███▍ | 69/200 [24:08<15:26, 7.07s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.95 Coverage: 45.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 35%|███▌ | 70/200 [24:15<15:13, 7.02s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.95 Coverage: 45.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 36%|███▌ | 71/200 [24:22<15:01, 6.99s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.33 Coverage: 46.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 36%|███▌ | 72/200 [24:29<14:51, 6.97s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.52 Coverage: 46.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 36%|███▋ | 73/200 [24:36<14:42, 6.95s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.88 Coverage: 46.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 37%|███▋ | 74/200 [24:43<14:33, 6.94s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.41 Coverage: 47.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 38%|███▊ | 75/200 [24:50<14:26, 6.94s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.58 Coverage: 47.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 38%|███▊ | 76/200 [24:57<14:18, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.58 Coverage: 47.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 38%|███▊ | 77/200 [25:04<14:10, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.77 Coverage: 48.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 39%|███▉ | 78/200 [25:10<14:02, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.77 Coverage: 48.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 40%|███▉ | 79/200 [25:17<13:55, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.96 Coverage: 48.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 40%|████ | 80/200 [25:24<13:49, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.96 Coverage: 48.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 40%|████ | 81/200 [25:31<13:41, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.96 Coverage: 48.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 41%|████ | 82/200 [25:38<13:35, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.33 Coverage: 48.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 42%|████▏ | 83/200 [25:45<13:29, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.34 Coverage: 48.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 42%|████▏ | 84/200 [25:52<13:22, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.52 Coverage: 49.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 42%|████▎ | 85/200 [25:59<13:14, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.53 Coverage: 49.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 43%|████▎ | 86/200 [26:06<13:07, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.7 Coverage: 49.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 44%|████▎ | 87/200 [26:13<13:01, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 36.06 Coverage: 49.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 44%|████▍ | 88/200 [26:20<12:53, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 36.42 Coverage: 50.24999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 44%|████▍ | 89/200 [26:26<12:46, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 36.78 Coverage: 50.74999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 45%|████▌ | 90/200 [26:33<12:39, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.13 Coverage: 51.24999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 46%|████▌ | 91/200 [26:40<12:32, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.67 Coverage: 52.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 46%|████▌ | 92/200 [26:47<12:25, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.03 Coverage: 52.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 46%|████▋ | 93/200 [26:54<12:18, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.2 Coverage: 52.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 47%|████▋ | 94/200 [27:01<12:12, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.39 Coverage: 53.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 48%|████▊ | 95/200 [27:08<12:05, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.39 Coverage: 53.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 48%|████▊ | 96/200 [27:15<11:58, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.39 Coverage: 53.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 48%|████▊ | 97/200 [27:22<11:51, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.92 Coverage: 53.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 49%|████▉ | 98/200 [27:29<11:44, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.92 Coverage: 53.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 50%|████▉ | 99/200 [27:35<11:36, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.29 Coverage: 54.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 50%|█████ | 100/200 [27:42<11:29, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.46 Coverage: 54.50000000000001\n",
"Updating archive and diversity projection.\n",
"3.0s (12 epochs) | DivProj (n=4000) fitted with val acc.: 0.751\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 50%|█████ | 101/200 [34:09<3:19:07, 120.68s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.94 Coverage: 40.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 51%|█████ | 102/200 [34:15<2:21:21, 86.54s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 28.95 Coverage: 40.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 52%|█████▏ | 103/200 [34:22<1:41:16, 62.65s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.13 Coverage: 40.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 52%|█████▏ | 104/200 [34:29<1:13:29, 45.93s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.13 Coverage: 40.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 52%|█████▎ | 105/200 [34:36<54:11, 34.23s/it] "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.32 Coverage: 40.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 53%|█████▎ | 106/200 [34:43<40:46, 26.03s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 29.68 Coverage: 41.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 54%|█████▎ | 107/200 [34:50<31:26, 20.29s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.22 Coverage: 41.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 54%|█████▍ | 108/200 [34:57<24:56, 16.27s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.41 Coverage: 42.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 55%|█████▍ | 109/200 [35:04<20:24, 13.46s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 30.76 Coverage: 42.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 55%|█████▌ | 110/200 [35:11<17:13, 11.48s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.31 Coverage: 43.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 56%|█████▌ | 111/200 [35:18<14:59, 10.11s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.48 Coverage: 43.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 56%|█████▌ | 112/200 [35:24<13:25, 9.15s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 31.84 Coverage: 44.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 56%|█████▋ | 113/200 [35:31<12:17, 8.48s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.02 Coverage: 44.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 57%|█████▋ | 114/200 [35:38<11:28, 8.00s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.19 Coverage: 44.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 57%|█████▊ | 115/200 [35:45<10:52, 7.67s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.56 Coverage: 45.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 58%|█████▊ | 116/200 [35:52<10:24, 7.44s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 32.93 Coverage: 45.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 58%|█████▊ | 117/200 [35:59<10:04, 7.28s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.29 Coverage: 46.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 59%|█████▉ | 118/200 [36:06<09:48, 7.17s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.47 Coverage: 46.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 60%|█████▉ | 119/200 [36:13<09:35, 7.11s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 33.82 Coverage: 46.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 60%|██████ | 120/200 [36:20<09:24, 7.05s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.37 Coverage: 47.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 60%|██████ | 121/200 [36:27<09:13, 7.01s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.56 Coverage: 47.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 61%|██████ | 122/200 [36:34<09:04, 6.98s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.74 Coverage: 48.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 62%|██████▏ | 123/200 [36:40<08:55, 6.96s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.92 Coverage: 48.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 62%|██████▏ | 124/200 [36:47<08:48, 6.95s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 34.92 Coverage: 48.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 62%|██████▎ | 125/200 [36:54<08:40, 6.94s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.11 Coverage: 48.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 63%|██████▎ | 126/200 [37:01<08:32, 6.93s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.48 Coverage: 49.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 64%|██████▎ | 127/200 [37:08<08:25, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.84 Coverage: 49.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 64%|██████▍ | 128/200 [37:15<08:17, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.85 Coverage: 49.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 64%|██████▍ | 129/200 [37:22<08:18, 7.02s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 35.86 Coverage: 49.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 65%|██████▌ | 130/200 [37:29<08:15, 7.07s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 36.23 Coverage: 50.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 66%|██████▌ | 131/200 [37:37<08:13, 7.15s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 36.42 Coverage: 50.24999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 66%|██████▌ | 132/200 [37:44<08:01, 7.08s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 36.78 Coverage: 50.74999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 66%|██████▋ | 133/200 [37:51<07:51, 7.03s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 36.97 Coverage: 51.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 67%|██████▋ | 134/200 [37:58<07:41, 6.99s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.16 Coverage: 51.24999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 68%|██████▊ | 135/200 [38:04<07:32, 6.96s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.52 Coverage: 51.74999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 68%|██████▊ | 136/200 [38:11<07:25, 6.95s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.71 Coverage: 52.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 68%|██████▊ | 137/200 [38:18<07:17, 6.94s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.72 Coverage: 52.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 69%|██████▉ | 138/200 [38:25<07:09, 6.93s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.9 Coverage: 52.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 70%|██████▉ | 139/200 [38:32<07:02, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.91 Coverage: 52.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 70%|███████ | 140/200 [38:39<06:54, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 37.91 Coverage: 52.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 70%|███████ | 141/200 [38:46<06:47, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.45 Coverage: 53.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 71%|███████ | 142/200 [38:53<06:41, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.45 Coverage: 53.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 72%|███████▏ | 143/200 [39:00<06:34, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.99 Coverage: 53.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 72%|███████▏ | 144/200 [39:07<06:26, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 38.99 Coverage: 53.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 72%|███████▎ | 145/200 [39:14<06:19, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.17 Coverage: 54.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 73%|███████▎ | 146/200 [39:20<06:12, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.19 Coverage: 54.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 74%|███████▎ | 147/200 [39:27<06:05, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.19 Coverage: 54.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 74%|███████▍ | 148/200 [39:34<05:58, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.19 Coverage: 54.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 74%|███████▍ | 149/200 [39:41<05:51, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.37 Coverage: 54.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 75%|███████▌ | 150/200 [39:48<05:45, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.55 Coverage: 54.50000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 76%|███████▌ | 151/200 [39:55<05:38, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.55 Coverage: 54.50000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 76%|███████▌ | 152/200 [40:02<05:31, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 39.74 Coverage: 54.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 76%|███████▋ | 153/200 [40:09<05:24, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.1 Coverage: 55.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 77%|███████▋ | 154/200 [40:16<05:17, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.11 Coverage: 55.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 78%|███████▊ | 155/200 [40:23<05:10, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.29 Coverage: 55.50000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 78%|███████▊ | 156/200 [40:29<05:03, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.3 Coverage: 55.50000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 78%|███████▊ | 157/200 [40:36<04:57, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.3 Coverage: 55.50000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 79%|███████▉ | 158/200 [40:43<04:50, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.49 Coverage: 55.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 80%|███████▉ | 159/200 [40:50<04:43, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.49 Coverage: 55.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 80%|████████ | 160/200 [40:57<04:36, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.5 Coverage: 55.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 80%|████████ | 161/200 [41:04<04:29, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.5 Coverage: 55.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 81%|████████ | 162/200 [41:11<04:23, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.51 Coverage: 55.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 82%|████████▏ | 163/200 [41:18<04:16, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.69 Coverage: 56.00000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 82%|████████▏ | 164/200 [41:25<04:09, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 40.69 Coverage: 56.00000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 82%|████████▎ | 165/200 [41:32<04:01, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.04 Coverage: 56.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 83%|████████▎ | 166/200 [41:39<03:55, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.05 Coverage: 56.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 84%|████████▎ | 167/200 [41:46<03:48, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.05 Coverage: 56.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 84%|████████▍ | 168/200 [41:52<03:40, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.05 Coverage: 56.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 84%|████████▍ | 169/200 [41:59<03:34, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.05 Coverage: 56.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 85%|████████▌ | 170/200 [42:06<03:27, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.24 Coverage: 56.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 86%|████████▌ | 171/200 [42:13<03:20, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.42 Coverage: 56.99999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 86%|████████▌ | 172/200 [42:20<03:13, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.42 Coverage: 56.99999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 86%|████████▋ | 173/200 [42:27<03:06, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.61 Coverage: 57.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 87%|████████▋ | 174/200 [42:34<02:59, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.8 Coverage: 57.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 88%|████████▊ | 175/200 [42:41<02:52, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.8 Coverage: 57.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 88%|████████▊ | 176/200 [42:48<02:45, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.8 Coverage: 57.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 88%|████████▊ | 177/200 [42:55<02:38, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.8 Coverage: 57.49999999999999\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 89%|████████▉ | 178/200 [43:01<02:31, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 41.98 Coverage: 57.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 90%|████████▉ | 179/200 [43:08<02:24, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 42.51 Coverage: 58.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 90%|█████████ | 180/200 [43:15<02:18, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 42.88 Coverage: 59.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 90%|█████████ | 181/200 [43:22<02:11, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.06 Coverage: 59.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 91%|█████████ | 182/200 [43:29<02:04, 6.90s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.23 Coverage: 59.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 92%|█████████▏| 183/200 [43:36<01:57, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.4 Coverage: 59.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 92%|█████████▏| 184/200 [43:43<01:50, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.41 Coverage: 59.75\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 92%|█████████▎| 185/200 [43:50<01:43, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.76 Coverage: 60.25\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 93%|█████████▎| 186/200 [43:57<01:36, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.93 Coverage: 60.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 94%|█████████▎| 187/200 [44:04<01:29, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.93 Coverage: 60.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 94%|█████████▍| 188/200 [44:11<01:22, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.93 Coverage: 60.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 94%|█████████▍| 189/200 [44:17<01:16, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 43.94 Coverage: 60.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 95%|█████████▌| 190/200 [44:24<01:09, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.3 Coverage: 61.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 96%|█████████▌| 191/200 [44:31<01:02, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.3 Coverage: 61.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 96%|█████████▌| 192/200 [44:38<00:55, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.31 Coverage: 61.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 96%|█████████▋| 193/200 [44:45<00:48, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.31 Coverage: 61.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 97%|█████████▋| 194/200 [44:52<00:41, 6.91s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.67 Coverage: 61.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 98%|█████████▊| 195/200 [44:59<00:34, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.67 Coverage: 61.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 98%|█████████▊| 196/200 [45:06<00:27, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.85 Coverage: 61.75000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 98%|█████████▊| 197/200 [45:13<00:20, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.85 Coverage: 61.75000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 99%|█████████▉| 198/200 [45:20<00:13, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 44.85 Coverage: 61.75000000000001\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|█████████▉| 199/200 [45:27<00:06, 6.92s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 45.05 Coverage: 62.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 200/200 [45:34<00:00, 13.67s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"QD score: 45.05 Coverage: 62.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"# Change here for whatever you would like to generate.\n",
"prompt = \"an astronaut riding a horse on mars\"\n",
"\n",
"INIT_POP = 200 # Initial population.\n",
"\n",
"TOTAL_ITRS = 200 # Total number of iterations.\n",
"update_schedule = [1, 21, 51, 101] # Iterations on which to update the archive.\n",
"n_pref_data = 1000 # Number of preferences used in each update.\n",
"\n",
"archive = None\n",
"best = 0.0\n",
"for itr in trange(1, TOTAL_ITRS + 1):\n",
" itr_start = time.time()\n",
"\n",
" # Update archive and scheduler if needed.\n",
" if itr in update_schedule:\n",
" if archive is None:\n",
" tqdm.write(\"Initializing archive and diversity projection.\")\n",
"\n",
" all_sols = []\n",
" all_clip_features = []\n",
" all_dreamsim_features = []\n",
" all_objs = []\n",
"\n",
" # Sample random solutions and get judgment on similarity.\n",
" n_batches = INIT_POP // BATCH_SIZE\n",
" for _ in range(n_batches):\n",
" sols = torch.randn(SD_IN_SHAPE, device=DEVICE, dtype=TORCH_DTYPE)\n",
" objs, _, clip_features, dreamsim_features = evaluate_lsi(\n",
" sols, prompt, return_features=True\n",
" )\n",
" all_sols.append(sols)\n",
" all_clip_features.append(clip_features)\n",
" all_dreamsim_features.append(dreamsim_features)\n",
" all_objs.append(objs)\n",
" all_sols = torch.concat(all_sols, dim=0)\n",
" all_clip_features = torch.concat(all_clip_features, dim=0)\n",
" all_dreamsim_features = torch.concat(all_dreamsim_features, dim=0)\n",
" all_objs = np.concatenate(all_objs, axis=0)\n",
"\n",
" # Initialize the diversity projection model.\n",
" div_proj_data = []\n",
" div_proj_labels = []\n",
" for _ in range(n_pref_data):\n",
" idx = np.random.choice(all_sols.shape[0], 3)\n",
" div_proj_data.append(all_clip_features[idx])\n",
" div_proj_labels.append(all_dreamsim_features[idx])\n",
" div_proj_data = torch.concat(div_proj_data, dim=0)\n",
" div_proj_labels = torch.concat(div_proj_labels, dim=0)\n",
" div_proj_data = div_proj_data.reshape(n_pref_data, 3, -1)\n",
" div_proj_label = div_proj_labels.reshape(n_pref_data, 3, -1)\n",
" diversity_model, div_proj_acc = fit_div_proj(\n",
" div_proj_data,\n",
" div_proj_label,\n",
" latent_dim=2,\n",
" )\n",
"\n",
" else:\n",
" tqdm.write(\"Updating archive and diversity projection.\")\n",
"\n",
" # Get all the current solutions and collect feedback.\n",
" all_sols = list_to_tensor(archive.data(\"solution\"))\n",
" n_batches = np.ceil(len(all_sols) / BATCH_SIZE).astype(int)\n",
" all_clip_features = []\n",
" all_dreamsim_features = []\n",
" all_objs = []\n",
" for i in range(n_batches):\n",
" sols = all_sols[i * BATCH_SIZE : (i + 1) * BATCH_SIZE]\n",
" objs, _, clip_features, dreamsim_features = evaluate_lsi(\n",
" sols, prompt, return_features=True\n",
" )\n",
" all_clip_features.append(clip_features)\n",
" all_dreamsim_features.append(dreamsim_features)\n",
" all_objs.append(objs)\n",
" all_clip_features = torch.concat(\n",
" all_clip_features, dim=0\n",
" ) # n_pref_data * 3, dim\n",
" all_dreamsim_features = torch.concat(all_dreamsim_features, dim=0)\n",
" all_objs = np.concatenate(all_objs, axis=0)\n",
"\n",
" # Update the diversity projection model.\n",
" additional_features = []\n",
" additional_labels = []\n",
" for _ in range(n_pref_data):\n",
" idx = np.random.choice(all_sols.shape[0], 3)\n",
" additional_features.append(all_clip_features[idx])\n",
" additional_labels.append(all_dreamsim_features[idx])\n",
" additional_features = torch.concat(additional_features, dim=0)\n",
" additional_labels = torch.concat(additional_labels, dim=0)\n",
" additional_div_proj_data = additional_features.reshape(n_pref_data, 3, -1)\n",
" additional_div_proj_label = additional_labels.reshape(n_pref_data, 3, -1)\n",
" div_proj_data = torch.concat(\n",
" (div_proj_data, additional_div_proj_data), axis=0\n",
" )\n",
" div_proj_label = torch.concat(\n",
" (div_proj_label, additional_div_proj_label), axis=0\n",
" )\n",
" diversity_model, div_proj_acc = fit_div_proj(\n",
" div_proj_data,\n",
" div_proj_label,\n",
" latent_dim=2,\n",
" )\n",
"\n",
" archive, scheduler = create_scheduler(\n",
" all_sols,\n",
" all_objs,\n",
" all_clip_features,\n",
" diversity_model,\n",
" seed=SEED,\n",
" )\n",
"\n",
" # Primary QD loop.\n",
" sols = scheduler.ask()\n",
" sols = list_to_tensor(sols)\n",
" objs, measures, clip_features, dreamsim_features = evaluate_lsi(\n",
" sols, prompt, return_features=True, diversity_model=diversity_model\n",
" )\n",
" best = max(best, max(objs))\n",
" scheduler.tell(objs, measures)\n",
"\n",
" # This can be used as a flag to save on the final iteration, but note that\n",
" # we do not save results in this tutorial.\n",
" final_itr = itr == TOTAL_ITRS\n",
"\n",
" # Update the summary statistics for the archive.\n",
" qd_score, coverage = archive.stats.norm_qd_score, archive.stats.coverage\n",
"\n",
" tqdm.write(f\"QD score: {np.round(qd_score, 2)} Coverage: {coverage * 100}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pNSvAu17Ewb0"
},
"source": [
"## Visualization\n",
"### Visualizing the Archive\n",
"We use the `grid_archive_heatmap` function to visualize the grid archive as a heatmap."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 423
},
"id": "1fUcpXHjEwb0",
"outputId": "6ac6ffa3-3333-4524-8da3-c433f9763e0a"
},
"outputs": [
{
"data": {
"text/plain": [
"Text(0, 0.5, 'Diversity Metric 2')"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAgcAAAGFCAYAAABpMVB1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABHlElEQVR4nO3deVxU9f4/8NcMMAOigISAcFHAjUwDhCQ0v6VRmC1SXnPhCmLhvSVlkiWmoqlJWRk3o0gT0MrcsrKNruGWRugP0zSNtFRwGdRMRjG2mfP7g5ycgHGWD8wM83r2OI97OXPOZ95zIn3PZ3vLJEmSQERERPQnubUDICIiItvC5ICIiIj0MDkgIiIiPUwOiIiISA+TAyIiItLD5ICIiIj0MDkgIiIiPUwOiIiISA+TAyIiItLD5ICIiIj0MDkgIiKyETt27MD999+PgIAAyGQyfPzxx3qvS5KEzMxMdO3aFW5uboiLi8ORI0f0rrlw4QISExPh4eEBLy8vPPLII7h8+bJJcTA5ICIishHV1dUIDw9HTk5Os68vXrwYr7/+OnJzc1FSUgJ3d3fEx8ejpqZGd01iYiJ+/PFHbN68GZ999hl27NiByZMnmxSHjIWXiIiIbI9MJsNHH32EhIQEAI29BgEBAXj66acxffp0AEBVVRX8/PxQUFCAsWPH4vDhw+jbty/27NmD6OhoAEBhYSFGjBiBkydPIiAgwKj3dm6VT0RERNQO1NTUoK6uzqI2JEmCTCbTO6dUKqFUKk1q59ixY1CpVIiLi9Od8/T0RExMDIqLizF27FgUFxfDy8tLlxgAQFxcHORyOUpKSvDggw8a9V5MDoiIiJpRU1ODkJBAqFQXLGqnY8eOTcb8586di3nz5pnUjkqlAgD4+fnpnffz89O9plKp4Ovrq/e6s7MzvL29ddcYg8kBERFRM+rq6qBSXcDxE+vg4dHBrDbU6isI7v4wKioq4OHhoTtvaq9BW2NyQEREZIBHR1d4dHQz72attrENDw+95MAc/v7+AIDKykp07dpVd76yshIRERG6a86ePat3X0NDAy5cuKC73xhcrUBERGSIVmvZIUhISAj8/f1RVFSkO6dWq1FSUoLY2FgAQGxsLC5evIjS0lLdNVu2bIFWq0VMTIzR78WeAyIiIkMs+UvexPsuX76Mo0eP6n4+duwY9u3bB29vb3Tr1g1PPfUUFi5ciF69eiEkJARz5sxBQECAbkXDjTfeiOHDhyM1NRW5ubmor69HWloaxo4da/RKBYDJARERkc34f//v/2Ho0KG6n9PT0wEAycnJKCgowLPPPovq6mpMnjwZFy9exG233YbCwkK4urrq7nn//feRlpaGO++8E3K5HKNGjcLrr79uUhzc54CIiKgZarUanp6euKD6EB4e7ma2UQ1v/1GoqqqyeM5BW2LPARERkSFayYJhBfv8/s3kgIiIyJA2nHNgK7hagYiIiPSw54CIiMgQB+w5YHJARERkiAMmBxxWICIiIj3sOSAiIjJEsqDnQLLPngMmB0RERAbIJC1kZv4lb+591sbkgIiIyBDOOSAiIiJHx54DIiIiQ7SS+TsdcodEIiKidsgBhxWYHBARERnigMkB5xwQERGRHvYcEBERGSJJ5u9XIHHOARERUfvjgMMKTA6IiIgMccDVCpxzQERERHocrudAq9Xi9OnT6NSpE2QymbXDISIiM0mShEuXLiEgIAByeSt+1+WwQvt3+vRpBAUFWTsMIiISpKKiAv/4xz9a7w1YeKn969SpE4DGXyYPDw8rR2P7/ngmVUg7bi8vF9KOSLWZYj4bACjni/t8dQsmC2lHMWeZkHYAoGHxf4S0o1FrhLQjmnKh7f1+0vWp1WoEBQXp/lxvLTKtFjIzkwNz77M2h0sOrg4leHh4MDkwgotCIaQdNxt81rVKMZ8NAJQCP1+doLgUAmNqcBUTk6bWRpMDG/z9JONxiFg8h0sOiIiITCJJ5u9XwH0OiIiI2iFOSCQiIiI9DpgccJ8DIiIi0sOeAyIiIkMccIdEJgdERESGcFiBiIiIHB17DoiIiAzRShb0HHBYgYiIqP3hPgdERESkh3MO2l5OTg6Cg4Ph6uqKmJgY7N69u8Vr6+vrMX/+fPTo0QOurq4IDw9HYWFhG0ZLRETU/lk1OVi7di3S09Mxd+5c7N27F+Hh4YiPj8fZs2ebvX727Nl4++23sXTpUhw6dAj/+c9/8OCDD+L7779v48iJiMhhSNJfyxlNPTisYLolS5YgNTUVKSkpAIDc3Fx8/vnnyMvLQ0ZGRpPr3333XcyaNQsjRowAADz22GP4+uuv8eqrr+K9995r09hFq31mgrVDaJbb0netHUKrkeqsHUHz5K5W79Brwnl2gZh2hLTSqGFRisDWbE/9vGQh7bjMWymkHYfGYYW2U1dXh9LSUsTFxf0VjFyOuLg4FBcXN3tPbW0tXF1d9c65ublh586drRorERE5sKvJgbmHHbJacnD+/HloNBr4+fnpnffz84NKpWr2nvj4eCxZsgRHjhyBVqvF5s2bsXHjRpw5c6bF96mtrYVardY7iIiIqGW2139pwH//+1/06tULYWFhUCgUSEtLQ0pKCuTylj9GVlYWPD09dUdQUFAbRkxERHbP3PkGlmy7bGVWSw58fHzg5OSEyspKvfOVlZXw9/dv9p4uXbrg448/RnV1NU6cOIGffvoJHTt2RGhoaIvvM3PmTFRVVemOiooKoZ+DiIjaOUlr2WGHrJYcKBQKREVFoaioSHdOq9WiqKgIsbGxBu91dXVFYGAgGhoa8OGHH2LkyJEtXqtUKuHh4aF3EBERGc0Bew6sulohPT0dycnJiI6OxsCBA5GdnY3q6mrd6oWkpCQEBgYiKysLAFBSUoJTp04hIiICp06dwrx586DVavHss89a82MQERG1K1ZNDsaMGYNz584hMzMTKpUKERERKCws1E1SLC8v15tPUFNTg9mzZ+PXX39Fx44dMWLECLz77rvw8vKy0icgIqJ2zwGXMlp9++S0tDSkpaU1+9q2bdv0fr799ttx6NChNoiKiIjoT5YMD3BYgYiIqB1ywKqMdrWUkYiIiFofew6IiIgM4bACERER6bNkvwJOSCQiImp/HLDngHMOiIiISA97DmyE8mXbLI38xxNiSkmLLP1cNztJSDuur9nmM4dcZu0ImqibKeaZy90Ffh+xweckkqZKTHe0TGBpa+fn8oW1ZVccsOeAyQEREZEhDrgJEocViIiISA97DoiIiAzhsAIRERHpYXJAREREejjngIiIiBwdew6IiIgMkaTGw9x77RCTAyIiIkMccM4BhxWIiIgMuZocmHuYQKPRYM6cOQgJCYGbmxt69OiBBQsWQLqmB0KSJGRmZqJr165wc3NDXFwcjhw5IvQjMzkgIiKyES+99BLeeustvPHGGzh8+DBeeuklLF68GEuXLtVds3jxYrz++uvIzc1FSUkJ3N3dER8fj5qaGmFxcFiBiIjIEMmC1QomVnP89ttvMXLkSNx7770AgODgYHzwwQfYvXt3Y3OShOzsbMyePRsjR44EAKxatQp+fn74+OOPMXbsWPPi/Bv2HBARERnShsMKgwYNQlFREX7++WcAwP79+7Fz507cc889AIBjx45BpVIhLi5Od4+npydiYmJQXFws7COz54CIiMgQLSyYkNj4P2q1Wu+0UqmEUqlscnlGRgbUajXCwsLg5OQEjUaDF154AYmJiQAAlUoFAPDz89O7z8/PT/eaCOw5ICIiamVBQUHw9PTUHVlZWc1et27dOrz//vtYvXo19u7di5UrV+KVV17BypUr2zRe9hzYiJppYkojA4BMIawpYaWWRX4+UaWW6+clC2kHALR/iNsFTfmS7ZWSduos5o8KbY245yQT+NWmYeFEMQ0JLCPtfIOTkHakBnFL6YQ9JwDOswuEtdXqBCxlrKiogIeHh+50c70GAPDMM88gIyNDN3egf//+OHHiBLKyspCcnAx/f38AQGVlJbp27aq7r7KyEhEREebF2Az2HBARERkgaSWLDgDw8PDQO1pKDq5cuQK5XP+vZicnJ2j/nBAZEhICf39/FBUV6V5Xq9UoKSlBbGyssM/MngMiIiJD2nCHxPvvvx8vvPACunXrhptuugnff/89lixZgkmTJgEAZDIZnnrqKSxcuBC9evVCSEgI5syZg4CAACQkJJgXYzOYHBAREdmIpUuXYs6cOXj88cdx9uxZBAQE4N///jcyMzN11zz77LOorq7G5MmTcfHiRdx2220oLCyEq6ursDiYHBARERnShtsnd+rUCdnZ2cjOzm7xGplMhvnz52P+/PnmxWQEJgdERESGOGBtBSYHREREhjhgcsDVCkRERKSHPQdERESGsOeg7eXk5CA4OBiurq6IiYnRFZdoSXZ2Nvr06QM3NzcEBQVh2rRpQitRERERXUuSLNjnwNwlkFZm1eRg7dq1SE9Px9y5c7F3716Eh4cjPj4eZ8+ebfb61atXIyMjA3PnzsXhw4exYsUKrF27Fs8991wbR05ERNR+WTU5WLJkCVJTU5GSkoK+ffsiNzcXHTp0QF5eXrPXf/vttxg8eDDGjx+P4OBg3H333Rg3btx1exuIiIjM1oZVGW2F1ZKDuro6lJaW6pWdlMvliIuLa7Hs5KBBg1BaWqpLBn799Vd88cUXGDFiRIvvU1tbC7VarXcQEREZzQGTA6tNSDx//jw0Gk2zZSd/+umnZu8ZP348zp8/j9tuuw2SJKGhoQH/+c9/DA4rZGVl4fnnnxcaOxERORAHnJBoV6sVtm3bhkWLFuHNN99ETEwMjh49iqlTp2LBggWYM2dOs/fMnDkT6enpup/VajWCgoLaKmSjiao0KFrtM2KqKcrdxVWrq50hJianzi5C2gEAJ2dxn09U5TuR1fhEtSV3FdhZKbACorC2BP5FIAkqYCkT+LtpaxUeG2rqLA+EmmW15MDHxwdOTk6orKzUO19ZWakrSfl3c+bMwYQJE/Doo48CaCxleXV/6VmzZjWpZAU0lsVsqfoVERHRdbVh4SVbYbU5BwqFAlFRUXplJ7VaLYqKilosO9lSKUsAdrtchIiIbJukteywR1YdVkhPT0dycjKio6MxcOBAZGdno7q6GikpKQCApKQkBAYGIisrC0BjKcslS5YgMjJSN6wwZ84c3H///bokgYiISCjOOWhbY8aMwblz55CZmQmVSoWIiAgUFhbqJimWl5fr9RTMnj0bMpkMs2fPxqlTp9ClSxdd7WsiIiISQyY5WH+8Wq2Gp6cnqqqq4OHhYe1wbJ6oCYkypcBJUfVifmVFTkhEg+31Hbb7CYnOVt/gtSmRExIFPXNbnZAoE/CvT11Thxte+KDV/jy/+vfFb7PGwcNVYV4brRxja7Gr1QpERERtzZK5A5xzQERE1B5JFsw5sNPOeRvslyMiIiJrYs8BERGRIdo/D3PvtUNMDoiIiAy4Wn7Z3HvtEZMDIiIiQxyw54BzDoiIiEgPew6IiIgMkf48zL3XDjE5ICIiMoBzDqhdqHsuSVhbUoOYdkSWbIZGTDNSjaCGIHgXOlFjlAJLGsts8E8KETvsXdWun7nImOT2+RedxTjngIiIiBydDX4fICIish3cPpmIiIj0cViBiIiIHB17DoiIiAzgsAIRERHpk2D+8ICdLvBgckBERGSAJJlfedlOKzZzzgERERHpY88BERGRAZxzQERERPoccCkjkwMiIiIDHLHngHMOiIiISA97DoiIiAzgagUjnDx5EpcvX25yvr6+Hjt27BASFBERkc3Qyiw77JDRPQdnzpzByJEjUVpaCplMhvHjx+PNN99Ex44dAQAXLlzA0KFDodGIK4Pbmv54NhUuSoXF7bj9910B0QD185KFtAMAdefFpaqKG8T8YossaSyyLVHsdVzRWDJXJzENNYh7UEKfudZOv94ZQ+AzF1n+2Z6eOeccGJCRkQG5XI6SkhIUFhbi0KFDGDp0KH7//XfdNZK99p8QERGRjtE9B19//TU++ugjREdHAwB27dqF0aNHY9iwYSgqKgIAyGS2942OiIjIEpIkgySZ9/ebufdZm9E9B1VVVejcubPuZ6VSiY0bNyI4OBhDhw7F2bNnWyVAIiIia7o6rGDuYY+MTg5CQ0Pxww8/6J1zdnbG+vXrERoaivvuu094cERERNYmSRYkB3Y62m50cnDPPfdg2bJlTc5fTRAiIiLMDiInJwfBwcFwdXVFTEwMdu/e3eK1d9xxB2QyWZPj3nvvNfv9iYiI6C9Gzzl44YUXcOXKleYbcXbGhx9+iFOnTpkcwNq1a5Geno7c3FzExMQgOzsb8fHxKCsrg6+vb5PrN27ciLq6Ot3Pv/32G8LDwzF69GiT35uIiOh6OOfAAGdnZ3h4eBh8vXv37iYHsGTJEqSmpiIlJQV9+/ZFbm4uOnTogLy8vGav9/b2hr+/v+7YvHkzOnTowOSAiIhah1YGyczDXvc5sOr2yXV1dSgtLUVcXJzunFwuR1xcHIqLi41qY8WKFRg7dizc3d2bfb22thZqtVrvICIiopZZNTk4f/48NBoN/Pz89M77+flBpVJd9/7du3fj4MGDePTRR1u8JisrC56enrojKCjI4riJiMhxXN0+2dzDHtl14aUVK1agf//+GDhwYIvXzJw5E1VVVbqjoqKiDSMkIiJ7d3XOgbmHPbJq4SUfHx84OTmhsrJS73xlZSX8/f0N3ltdXY01a9Zg/vz5Bq9TKpVQKpUWx0pERI5JN3/AzHvtkck9B/n5+Vi/fn2T8+vXr8fKlStNakuhUCAqKkq3wyIAaLVaFBUVITY21uC969evR21tLf71r3+Z9J5ERERkmMnJQVZWFnx8fJqc9/X1xaJFi0wOID09HcuXL8fKlStx+PBhPPbYY6iurkZKSgoAICkpCTNnzmxy34oVK5CQkIAbbrjB5PckIiIyliPOOTB5WKG8vBwhISFNznfv3h3l5eUmBzBmzBicO3cOmZmZUKlUiIiIQGFhoW6SYnl5OeRy/RymrKwMO3fuxP/+9z+T3+8quaLxsJQ6JcnyRgAofcX9Bjk3v3DDPIKK8UkNAv8LEVQZTiZwxo3ItkRttyr9IW7fVlGfT6oT93sgg8B9aZ0FfUCBlQadn8sX0k7DohQh7QCwq0qKIjniPgcmJwe+vr744YcfEBwcrHd+//79Zn+LT0tLQ1paWrOvbdu2rcm5Pn36sAIkERG1Ca1WBq2ZcwfMvc/aTE6Xx40bhyeffBJbt26FRqOBRqPBli1bMHXqVIwdO7Y1YiQiIqI2ZHLPwYIFC3D8+HHceeedcHZuvF2r1SIpKcmsOQdERES2zJK5A/bayW1ycqBQKLB27VosWLAA+/fvh5ubG/r372/W1slERES2jnMOTNC7d2/07t1bZCxEREQ2h8lBC9LT07FgwQK4u7sjPT3d4LVLliwREhgREZEjOnXqFGbMmIEvv/wSV65cQc+ePZGfn4/o6GgAgCRJmDt3LpYvX46LFy9i8ODBeOutt9CrVy9hMRiVHHz//feor68HAOzduxcyWfOZUEvniYiI7JVWkkFrZg+Aqff9/vvvGDx4MIYOHYovv/wSXbp0wZEjR9C5c2fdNYsXL8brr7+OlStXIiQkBHPmzEF8fDwOHToEV1dXs+L8O6OSg61bt+r+f3NLC4mIiNqrttw++aWXXkJQUBDy8//a5+LavYUkSUJ2djZmz56NkSNHAgBWrVoFPz8/fPzxx8JWDZq0lLG+vh7Ozs44ePCgkDcnIiKydSJ2SFSr1XpHbW1ts++1adMmREdHY/To0fD19UVkZCSWL1+ue/3YsWNQqVSIi4vTnfP09ERMTAyKi4uFfWaTkgMXFxd069YNGo1GWABERETtXVBQEDw9PXVHVlZWs9f9+uuvuvkDX331FR577DE8+eSTutpFKpUKAHS7CF/l5+ene00Ek1crzJo1C8899xzeffddeHt7CwuEiIjIFmlhwZwDNN5XUVEBDw8P3fmWqgVrtVpER0fr9g2KjIzEwYMHkZubi+TkZLNiMIfJycEbb7yBo0ePIiAgAN27d4e7u/5G/nv37hUWHBERkbWJWMro4eGhlxy0pGvXrujbt6/euRtvvBEffvghAMDf3x8AUFlZia5du+quqaysREREhFkxNsfk5GDkyJFclUBERA5DsmC1gqlJxeDBg1FWVqZ37ueff9ZtNBgSEgJ/f38UFRXpkgG1Wo2SkhI89thjZsXYHJOTg3nz5gl7cyIiIvrLtGnTMGjQICxatAgPP/wwdu/ejWXLlmHZsmUAGrcMeOqpp7Bw4UL06tVLt5QxICAACQkJwuIwOTkIDQ3Fnj17mlRgvHjxIgYMGIBff/1VWHCtSe4mh1xpeZlWj/yVAqIRq3bGBGFtyRSCStkKLNksU4jpuZLqBJb8FVRGGhBXalnuafYGqE0JKtUr6yCwtrXI8sENAn8XBGl4cZKQdkSWSxdZmtyetOUOibfccgs++ugjzJw5E/Pnz0dISAiys7ORmJiou+bZZ59FdXU1Jk+ejIsXL+K2225DYWGhsD0OADOSg+PHjze7WqG2thYnT54UEhQREZGt0P55mHuvqe677z7cd999Lb4uk8kwf/58zJ8/38yors/o5GDTpk26///VV1/B09NT97NGo0FRUZHeRg1ERERkn4xODq6OZchksibLKVxcXBAcHIxXX31VaHBERETWxsJLBmi1jZ0jISEh2LNnD3x8fFotKCIiIluhlUyvkXDtvfbI5DkHx44d0/3/mpoaoRMgiIiIbI0j9hyYPPdUq9ViwYIFCAwMRMeOHXWrE+bMmYMVK1YID5CIiIjalsnJwcKFC1FQUIDFixdDoVDozvfr1w/vvPOO0OCIiIisrXFYwfzDHpmcHKxatQrLli1DYmIinJycdOfDw8Px008/CQ2OiIjI2q4OK5h72COT5xycOnUKPXv2bHJeq9Wivr5eSFBERES2QguZroCSOffaI5N7Dvr27YtvvvmmyfkNGzYgMjJSSFBERERkPSb3HGRmZiI5ORmnTp2CVqvFxo0bUVZWhlWrVuGzzz5rjRiJiIisRpIaD3PvtUcm9xyMHDkSn376Kb7++mu4u7sjMzMThw8fxqeffoq77rqrNWIkIiKyGu2fVRnNPeyRWZVZhgwZgs2bN4uOhYiIyOZIFsw5kBxlzgERERG1b0b3HISGhhp1nb2UbG64qEWDwvIyrTWpYsojK7uIyy5lLgIzVUGLdCWRi31rBJXXdbbNjF7mZoM5uw0u1rbJUsQCS3eLIhP5ey7w90Ar4L9jbW3blNp2xDkHRicHx48fR/fu3TF+/Hj4+voKCyAnJwcvv/wyVCoVwsPDsXTpUgwcOLDF6y9evIhZs2Zh48aNuHDhArp3747s7GyMGDFCWExERERXWTJ3oN3POVi7di3y8vKwZMkS3HPPPZg0aRJGjBgBudz8lHvt2rVIT09Hbm4uYmJikJ2djfj4eJSVlTWbgNTV1eGuu+6Cr68vNmzYgMDAQJw4cQJeXl5mx0BERGSIBJnZcwfa/ZyD0aNH48svv8TRo0cRFRWFadOmISgoCBkZGThy5IhZb75kyRKkpqYiJSUFffv2RW5uLjp06IC8vLxmr8/Ly8OFCxfw8ccfY/DgwQgODsbtt9+O8PBws96fiIiImjL5a39gYCBmzZqFI0eOYPXq1SgpKUFYWBh+//13k9qpq6tDaWkp4uLi/gpGLkdcXByKi4ubvWfTpk2IjY3FlClT4Ofnh379+mHRokXQaDSmfgwiIiKjOGJtBbOWMtbU1GDDhg3Iy8tDSUkJRo8ejQ4dOpjUxvnz56HRaODn56d33s/Pr8UaDb/++iu2bNmCxMREfPHFFzh69Cgef/xx1NfXY+7cuc3eU1tbi9raWt3ParXapDiJiMixcc7BdZSUlGDFihVYt24dQkNDMWnSJHz44Yfo3Llza8WnR6vVwtfXF8uWLYOTkxOioqJw6tQpvPzyyy0mB1lZWXj++efbJD4iImp/HHHOgdHJwU033YSzZ89i/Pjx2L59u8Xj/D4+PnByckJlZaXe+crKSvj7+zd7T9euXeHi4qJXDfLGG2+ESqVCXV2dXgnpq2bOnIn09HTdz2q1GkFBQRbFTkRE1J4ZPefg8OHDqKmpwapVqzB06FB4e3s3exhLoVAgKioKRUVFunNarRZFRUWIjY1t9p7Bgwfj6NGj0Gr/Wtv6888/o2vXrs0mBgCgVCrh4eGhdxARERmLcw4MyM/PF/7m6enpSE5ORnR0NAYOHIjs7GxUV1cjJSUFAJCUlITAwEBkZWUBAB577DG88cYbmDp1Kp544gkcOXIEixYtwpNPPik8NiIiIoDDCgYlJycLf/MxY8bg3LlzyMzMhEqlQkREBAoLC3WTFMvLy/X2UQgKCsJXX32FadOm4eabb0ZgYCCmTp2KGTNmCI+NiIjIUZm1WkGktLQ0pKWlNfvatm3bmpyLjY3Fd99918pRERERNbJkeKDdDysQERE5Ii5lJCIiIj3Sn4e599ojGyz/RkRERNZkcs/B1q1bMXTo0NaIpU25vbwcbja0rLHmaTGlnwFAJrA/SKoXk/fKBJaylbmKaUsSWe1VYPlgUSm7VCfuA4oq+6u9Im6rc7mrwO82on4/RQ4wC4rJFn8PADH//uRt9P1WgvnDCva6WsHkJzt8+HD06NEDCxcuREVFRWvEREREZDO0Fh72yOTk4NSpU0hLS8OGDRsQGhqK+Ph4rFu3DnV1da0RHxERkVVJksyiwx6ZnBz4+Phg2rRp2LdvH0pKStC7d288/vjjCAgIwJNPPon9+/e3RpxERETURiwasBkwYABmzpyJtLQ0XL58GXl5eYiKisKQIUPw448/ioqRiIjIajisYKT6+nps2LABI0aMQPfu3fHVV1/hjTfeQGVlJY4ePYru3btj9OjRomMlIiJqc6ytYIQnnngCH3zwASRJwoQJE7B48WL069dP97q7uzteeeUVBAQECA2UiIjIGlhbwQiHDh3C0qVL8dBDD0GpVDZ7jY+PD7Zu3WpxcERERNT2TB5WmDt3LkaPHt0kMWhoaMCOHTsAAM7Ozrj99tvFREhERGRFjjisYHJyMHToUFy4cKHJ+aqqqnaxORIREdG1rg4rmHvYI5OHFSRJgkzW9MP+9ttvcHd3FxIUERGRrWBVRgMeeughAIBMJsPEiRP1hhU0Gg1++OEHDBo0SHyERERE1KaMTg48PT0BNPYcdOrUCW5ubrrXFAoFbr31VqSmpoqPkIiIyIrYc2BAfn4+ACA4OBjTp0/nEAIRETkELmU0wty5c1sjDocnV4r7BVIsWiWsrdpnBFWLdBHTDABor4jZc0wmsKqfy/yVwtqqz0wW0o7Iz+eckSeknfp5Yj6baM7P5Qtpp37+RCHtAIBM0FdOmaJtKheazI6+UksW9BxI9vMx9RiVHAwYMABFRUXo3LkzIiMjm52QeNXevXuFBUdERERtz6jkYOTIkboJiAkJCa0ZDxERkU2xpEaCvdZWMCo5uHYogcMKRETkSCwpvewwJZsrKipw8uRJ3c+7d+/GU089hWXLlgkNjIiIiKzD5ORg/PjxuroJKpUKcXFx2L17N2bNmoX58+cLD5CIiMiaWLLZCAcPHsTAgQMBAOvWrUP//v3x7bff4v3330dBQYHo+IiIiKzKEWsrmLyUsb6+Xjc58euvv8YDDzwAAAgLC8OZM2fERkdERGRl0p+HuffaI5N7Dm666Sbk5ubim2++webNmzF8+HAAwOnTp3HDDTcID5CIiIjalsnJwUsvvYS3334bd9xxB8aNG4fw8HAAwKZNm3TDDURERO1F4/CAzMzD2tGbx6RhBUmSEBoaivLycjQ0NKBz58661yZPnowOHToID5CIiMiaOKxwHZIkoWfPnlCpVHqJAdBYc8HX11docERERNbmiBMSTUoO5HI5evXqhd9++6214iEiIiIrM3nOwYsvvohnnnkGBw8eFBZETk4OgoOD4erqipiYGOzevbvFawsKCiCTyfQOV1dXYbEQERFdyxH3OTB5KWNSUhKuXLmC8PBwKBQKuLm56b1+4cIFk9pbu3Yt0tPTkZubi5iYGGRnZyM+Ph5lZWUtDlN4eHigrKxM97OhQlBERESWkCTzqyu266qM18rOzhYawJIlS5CamoqUlBQAQG5uLj7//HPk5eUhIyOj2XtkMhn8/f0tet8r01PhrFBY1AYAyAWVIpYEppcXk8SVxXXzE9OOVCvuvxBtjZh2XATOnxVZilhqEFSqt0HcL5Wwz2ejX6OElcnu4CSkHQCQrmjENOQs7suTS2aBsLZEPHOpvm3+5pUggxZm1lYw8z5rMzk5SE4W94dgXV0dSktLMXPmTN05uVyOuLg4FBcXt3jf5cuX0b17d2i1WgwYMACLFi3CTTfd1Oy1tbW1qK2t1f2sVquFxU9ERNRaXnzxRcycORNTp07VfTGvqanB008/jTVr1qC2thbx8fF488034ecn6Jvcn0yecwAAv/zyC2bPno1x48bh7NmzAIAvv/wSP/74o0ntnD9/HhqNpsmH8vPzg0qlavaePn36IC8vD5988gnee+89aLVaDBo0SK8Y1LWysrLg6empO4KCgkyKkYiIHNvVYQVzD3Ps2bMHb7/9Nm6++Wa989OmTcOnn36K9evXY/v27Th9+jQeeughAZ9Sn8nJwfbt29G/f3+UlJRg48aNuHz5MgBg//79bVLOOTY2FklJSYiIiMDtt9+OjRs3okuXLnj77bebvX7mzJmoqqrSHRUVFa0eIxERtR9tPSHx8uXLSExMxPLly/W2DaiqqsKKFSuwZMkSDBs2DFFRUcjPz8e3336L7777zoJP2JTJyUFGRgYWLlyIzZs3Q3HNmP2wYcNMDs7HxwdOTk6orKzUO19ZWWn0nAIXFxdERkbi6NGjzb6uVCrh4eGhdxARERlLxD4HarVa77h2uPvvpkyZgnvvvRdxcXF650tLS1FfX693PiwsDN26dTM4FG8Ok5ODAwcO4MEHH2xy3tfXF+fPnzepLYVCgaioKBQVFenOabVaFBUVITY21qg2NBoNDhw4gK5du5r03kRERG0lKChIb4g7Kyur2evWrFmDvXv3Nvu6SqWCQqGAl5eX3nlDQ/HmMnlCopeXF86cOYOQkBC9899//z0CAwNNDiA9PR3JycmIjo7GwIEDkZ2djerqat3qhaSkJAQGBuoe1Pz583HrrbeiZ8+euHjxIl5++WWcOHECjz76qMnvTUREdD0itk+uqKjQ67m+Wt34WhUVFZg6dSo2b95s9f17TE4Oxo4dixkzZmD9+vWQyWTQarXYtWsXpk+fjqSkJJMDGDNmDM6dO4fMzEyoVCpERESgsLBQN0mxvLwccvlfHRy///47UlNTdVs4R0VF4dtvv0Xfvn1Nfm8iIqLrsWQb5Kv3GTOsXVpairNnz2LAgAG6cxqNBjt27MAbb7yBr776CnV1dbh48aJe74EpQ/HGMjk5WLRoEaZMmYKgoCBoNBr07dsXGo0G48ePx+zZs80KIi0tDWlpac2+tm3bNr2fX3vtNbz22mtmvQ8REZGp2moTpDvvvBMHDhzQO5eSkoKwsDDMmDEDQUFBcHFxQVFREUaNGgUAKCsrQ3l5udFD8cYyOTlQKBRYvnw55syZg4MHD+Ly5cuIjIxEr169hAZGRETkSDp16oR+/frpnXN3d8cNN9ygO//II48gPT0d3t7e8PDwwBNPPIHY2FjceuutQmMxOTnYuXMnbrvtNnTr1g3dunUTGgwREZGtsaRGguhNQV977TXI5XKMGjVKbxMk0UxODoYNG4bAwECMGzcO//rXvzjWT0RE7ZqIOQfm+vvQuqurK3JycpCTk2NZw9dh8lLG06dP4+mnn8b27dvRr18/RERE4OWXX25xh0IiIiKyLyYnBz4+PkhLS8OuXbvwyy+/YPTo0Vi5ciWCg4MxbNiw1oiRiIjIaiQLD3tk8rDCtUJCQpCRkYHw8HDMmTMH27dvFxUXERGRTbDmsIK1mJ0c7Nq1C++//z42bNiAmpoajBw5ssUdn2xRh1eWo4OArZSr/zNBQDQwswRW8xQe4qbAKF9+V1hbojQsnCikHZFlsl3mrRTWVv38iULacZ5dIKQdQFxJY5f54p4TGadhUYq1Q7B7EmRml152mJLNM2fOxJo1a3D69Gncdddd+O9//4uRI0eiQ4cOrREfERERtTGTk4MdO3bgmWeewcMPPwwfH5/WiImIiMhmSDB/eMBORxVMTw527drVGnEQERHZJM45aMGmTZtwzz33wMXFBZs2bTJ47QMPPCAkMCIiIlsgovCSvTEqOUhISIBKpYKvry8SEhJavE4mk0Gj0YiKjYiIiKzAqORAq9U2+/+JiIjaOw4rEBERkR7pz3/MvdcemZQcaLVaFBQUYOPGjTh+/DhkMhlCQkLwz3/+ExMmTIBMZp/rOYmIiFriiD0HRm+9I0kSHnjgATz66KM4deoU+vfvj5tuugknTpzAxIkT8eCDD7ZmnERERNRGjO45KCgowI4dO1BUVIShQ4fqvbZlyxYkJCRg1apVSEpKEh4kERGRtTjiagWjew4++OADPPfcc00SA6CxjHNGRgbef/99ocERERFZ29VhBXMPe2R0cvDDDz9g+PDhLb5+zz33YP/+/UKCIiIishWSZNlhj4xODi5cuAA/P78WX/fz88Pvv/8uJCgiIiKyHqPnHGg0Gjg7t3y5k5MTGhoahATVFv54JhUuCoXF7chdBAQDQO4qph1bVTdb3FyUhotiUnEndyHNABBXtRCAsAqdoqo7AqymaA318wRVwhRYMVQkEb9TLmo1sHiNgGgM0/55mHuvPTI6OZAkCRMnToRSqWz29draWmFBERER2QpHXMpodHKQnHz9LJYrFYiIqN2xZO5Ae08O8vPzWzMOIiIishHcPpmIiMgAzjkgIiIiPZYsSWz3SxmJiIjIMbDngIiIyAAOKxAREZEeSZIgmTk+YO591mYTwwo5OTkIDg6Gq6srYmJisHv3bqPuW7NmDWQyGRISElo3QCIiclisrWAFa9euRXp6OubOnYu9e/ciPDwc8fHxOHv2rMH7jh8/junTp2PIkCFtFCkREZFjsHpysGTJEqSmpiIlJQV9+/ZFbm4uOnTogLy8vBbv0Wg0SExMxPPPP4/Q0NA2jJaIiByNZOFhj6yaHNTV1aG0tBRxcXG6c3K5HHFxcSguLm7xvvnz58PX1xePPPLIdd+jtrYWarVa7yAiIjIWhxXa2Pnz56HRaJpUe/Tz84NKpWr2np07d2LFihVYvny5Ue+RlZUFT09P3REUFGRx3ERE5DiYHNi4S5cuYcKECVi+fDl8fHyMumfmzJmoqqrSHRUVFa0cJRERkX2z6lJGHx8fODk5obKyUu98ZWUl/P39m1z/yy+/4Pjx47j//vt157TaxlWkzs7OKCsrQ48ePfTuUSqVzVaSdHt5Odw8PCz+DFfSJljcBgBIAhfDur76rrC2LqWK+XydlouLCYLKP8ucZULaAQCpTtzXA8WiVULaEVkmWxShZaQzC4S1ZYtstdSyI2qcO2DmUkaxobQZq/YcKBQKREVFoaioSHdOq9WiqKgIsbGxTa4PCwvDgQMHsG/fPt3xwAMPYOjQodi3bx+HDIiISDhHHFaw+iZI6enpSE5ORnR0NAYOHIjs7GxUV1cjJSUFQGMZ6MDAQGRlZcHV1RX9+vXTu9/LywsAmpwnIiISwRFrK1g9ORgzZgzOnTuHzMxMqFQqREREoLCwUDdJsby8HHK5XU2NICIismtWTw4AIC0tDWlpac2+tm3bNoP3FhQUiA+IiIjoTxIkaM2ec2CfXQc2kRwQERHZKg4rEBERkR5HrMrIwXwiIiLSw54DIiIiAxyxZDOTAyIiIgMs2a/AXvc54LACERER6WHPARERkQFaC5YymnuftTE5ICIiMkCCBUsZhUbSdpgcEBERGeCIPQecc0BERER6HLbn4NITqZApFBa3I3cREAwAuUJc+eA/nhBTZhkAoBUTV91z4soH22JJY1ExAUDDwonC2hJFWKnlBnHfooT++1so7t8ftT+SZP7wgJ2uZHTc5ICIiMgYjjiswOSAiIjIAK1kQXJgp10HnHNAREREethzQEREZID05z/m3muPmBwQEREZIMH86or2mRowOSAiIjLIESckcs4BERER6WHPARERkQGSZMGcAztdrcDkgIiIyABHHFZgckBERGSAIyYHnHNARERkI7KysnDLLbegU6dO8PX1RUJCAsrKyvSuqampwZQpU3DDDTegY8eOGDVqFCorK4XGweSAiIjIAEnXd2DeYYrt27djypQp+O6777B582bU19fj7rvvRnV1te6aadOm4dNPP8X69euxfft2nD59Gg899JDQz8xhBSIiIgPaclihsLBQ7+eCggL4+vqitLQU//d//4eqqiqsWLECq1evxrBhwwAA+fn5uPHGG/Hdd9/h1ltvNSvOv2PPARERkQGW9RtYNuegqqoKAODt7Q0AKC0tRX19PeLi4nTXhIWFoVu3biguLrbova7lsD0HLp0AF6WAhszdNutvFFntu2TslTRxZaQtL7T9ZzsCy/T+MVXc53P777tC2hFWZhmATNDXCOf5K8U0RGRn1Gq13s9KpRJKpeG/hLRaLZ566ikMHjwY/fr1AwCoVCooFAp4eXnpXevn5weVSiUsXvYcEBERGaC18B8ACAoKgqenp+7Iysq67vtOmTIFBw8exJo1a1r7IzbhsD0HRERExpBkEiSZed3EVzdPqqiogIeHh+789XoN0tLS8Nlnn2HHjh34xz/+oTvv7++Puro6XLx4Ua/3oLKyEv7+/mbF2Bz2HBAREbUyDw8PvaOl5ECSJKSlpeGjjz7Cli1bEBISovd6VFQUXFxcUFRUpDtXVlaG8vJyxMbGCovXJpKDnJwcBAcHw9XVFTExMdi9e3eL127cuBHR0dHw8vKCu7s7IiIi8O67YsZoiYiI/k6yYDKiqdsuT5kyBe+99x5Wr16NTp06QaVSQaVS4Y8//gAAeHp64pFHHkF6ejq2bt2K0tJSpKSkIDY2VthKBcAGhhXWrl2L9PR05ObmIiYmBtnZ2YiPj0dZWRl8fX2bXO/t7Y1Zs2YhLCwMCoUCn332GVJSUuDr64v4+HgrfAIiImrPtNBCZubsc62J97311lsAgDvuuEPvfH5+PiZOnAgAeO211yCXyzFq1CjU1tYiPj4eb775plnxtcTqycGSJUuQmpqKlJQUAEBubi4+//xz5OXlISMjo8n1f39gU6dOxcqVK7Fz504mB0REJJw5mxlde69J1xtRqMnV1RU5OTnIyckxKyZjWHVYoa6uDqWlpXrrNeVyOeLi4oxarylJEoqKilBWVob/+7//a81QiYiIHIZVew7Onz8PjUYDPz8/vfN+fn746aefWryvqqoKgYGBqK2thZOTE958803cddddzV5bW1uL2tpa3c9/X2tKRERkiFamhczM1QqmDivYCqsPK5ijU6dO2LdvHy5fvoyioiKkp6cjNDS0yZAD0FjE4vnnn2/7IImIqF1oyzkHtsKqyYGPjw+cnJyaVJO63npNuVyOnj17AgAiIiJw+PBhZGVlNZsczJw5E+np6bqf1Wo1goKCxHwAIiJq9xwxObDqnAOFQoGoqCi99ZparRZFRUUmrdfUarV6QwfXUiqVTdaXEhERUcusPqyQnp6O5ORkREdHY+DAgcjOzkZ1dbVu9UJSUhICAwN1W01mZWUhOjoaPXr0QG1tLb744gu8++67uuUfREREIrXlagVbYfXkYMyYMTh37hwyMzOhUqkQERGBwsJC3STF8vJyyOV/dXBUV1fj8ccfx8mTJ+Hm5oawsDC89957GDNmjLU+AhERtWNaaCCDxux77ZFMMmZRZTuiVqvh6emJyikPw0NpeX0/11e5O6O9UqckCWtL6Wt7/xnJXGTC2hJVwVJkpUiXzAJhbZF9uvrneVVVVasMGV9tv7fXaDjJXMxqQyPV4+eL61stxtZiE9snExERke2w+rACERGRLeM+B0RERKSncc6BeR3t9jrngMkBERGRQeavVoCd9hxwzgERERHpYc8BERGRAVpJA3O/Szfea3+YHBARERnATZCIiIhIjwQNJDN7DiQ7nZDIOQdERESkhz0HREREBjTuVcB9DoiIiOhPEiQL5hzY3tbqxuCwAhEREelhzwEREZEBkqSBBPMKmUlcykhERNT+cM6BA3FdtByudlQ+k/7yxxMThLQjSeJG1WROwpqCYpGY8si2iGWWyR41LmU0s+eASxmJiIioPXDYngMiIiJjSJIFOyRKHFYgIiJqdzjngIiIiPQ44moFzjkgIiIiPew5ICIiMsARd0hkckBERGRA44REc4cVOOeAiIioHdJY8P2fcw6IiIioHWDPARERkQGNQwMcViAiIqI/MTkgIiIiPVpoITO7toJ9Jgecc0BERER6HK7nQJIa55yq1WorR0Lm+qOuTkg7dXXicuP6WnFrmRX83SQyytU/x6/+ud5aOKzgAC5dugQACAoKsnIkRC1YstbaERDZlUuXLsHT07PV2rdkC2R73T7Z4ZKDgIAAVFRUoFOnTpDJzMsE25JarUZQUBAqKirg4eFh7XDsDp+f5fgMLcPnZ7mWnqEkSbh06RICAgJa9f0bdznkDontmlwuxz/+8Q9rh2EyDw8P/sFiAT4/y/EZWobPz3LNPcPW7DFwZA6XHBAREZnCknkDnHNARETUDjlicsCljDZOqVRi7ty5UCqV1g7FLvH5WY7P0DJ8fpbjM2x7Mqm114AQERHZIbVaDU9PTygV/4BMZt53aUnSorbuJKqqquxqzgmHFYiIiAxwxGEFJgdEREQGOGJywDkHREREpIc9B0RERAZZ8u2fPQckwIULF5CYmAgPDw94eXnhkUceweXLlw1e/8QTT6BPnz5wc3NDt27d8OSTT6KqqqoNo7aunJwcBAcHw9XVFTExMdi9e7fB69evX4+wsDC4urqif//++OKLL9ooUttlyjNcvnw5hgwZgs6dO6Nz586Ii4u77jNv70z9HbxqzZo1kMlkSEhIaN0A7YCpz/DixYuYMmUKunbtCqVSid69e7faf8uSpLXosEsS2ZThw4dL4eHh0nfffSd98803Us+ePaVx48a1eP2BAwekhx56SNq0aZN09OhRqaioSOrVq5c0atSoNozaetasWSMpFAopLy9P+vHHH6XU1FTJy8tLqqysbPb6Xbt2SU5OTtLixYulQ4cOSbNnz5ZcXFykAwcOtHHktsPUZzh+/HgpJydH+v7776XDhw9LEydOlDw9PaWTJ0+2ceS2wdTnd9WxY8ekwMBAaciQIdLIkSPbJlgbZeozrK2tlaKjo6URI0ZIO3fulI4dOyZt27ZN2rdvn9C4qqqqJACSk5O35OzsY9bh5OQtAZCqqqqExtbamBzYkEOHDkkApD179ujOffnll5JMJpNOnTpldDvr1q2TFAqFVF9f3xph2pSBAwdKU6ZM0f2s0WikgIAAKSsrq9nrH374Yenee+/VOxcTEyP9+9//btU4bZmpz/DvGhoapE6dOkkrV65srRBtmjnPr6GhQRo0aJD0zjvvSMnJyQ6fHJj6DN966y0pNDRUqqura9W4HDk54LCCDSkuLoaXlxeio6N15+Li4iCXy1FSUmJ0O1fX0zo7t+8pJXV1dSgtLUVcXJzunFwuR1xcHIqLi5u9p7i4WO96AIiPj2/x+vbOnGf4d1euXEF9fT28vb1bK0ybZe7zmz9/Pnx9ffHII4+0RZg2zZxnuGnTJsTGxmLKlCnw8/NDv379sGjRImg0rVMBUZIkC4YV7HMrofb9t4edUalU8PX11Tvn7OwMb29vqFQqo9o4f/48FixYgMmTJ7dGiDbl/Pnz0Gg08PPz0zvv5+eHn376qdl7VCpVs9cb+3zbG3Oe4d/NmDEDAQEBTZIuR2DO89u5cydWrFiBffv2tUGEts+cZ/jrr79iy5YtSExMxBdffIGjR4/i8ccfR319PebOndsKUWoAmFvF1z6TA/YctIGMjAzIZDKDh7F/EBuiVqtx7733om/fvpg3b57lgRNdx4svvog1a9bgo48+gqurq7XDsXmXLl3ChAkTsHz5cvj4+Fg7HLul1Wrh6+uLZcuWISoqCmPGjMGsWbOQm5vbKu/niBMS2XPQBp5++mlMnDjR4DWhoaHw9/fH2bNn9c43NDTgwoUL8Pf3N3j/pUuXMHz4cHTq1AkfffQRXFxcLA3b5vn4+MDJyQmVlZV65ysrK1t8Xv7+/iZd396Z8wyveuWVV/Diiy/i66+/xs0339yaYdosU5/fL7/8guPHj+P+++/XndNqG//ycHZ2RllZGXr06NG6QdsYc34Hu3btChcXFzg5OenO3XjjjVCpVKirq4NCoWjVmB0Bew7aQJcuXRAWFmbwUCgUiI2NxcWLF1FaWqq7d8uWLdBqtYiJiWmxfbVajbvvvhsKhQKbNm1ymG9wCoUCUVFRKCoq0p3TarUoKipCbGxss/fExsbqXQ8AmzdvbvH69s6cZwgAixcvxoIFC1BYWKg3R8bRmPr8wsLCcODAAezbt093PPDAAxg6dCj27duHoKCgtgzfJpjzOzh48GAcPXpUl1gBwM8//4yuXbu2UmKgtfCwQ9aeEUn6hg8fLkVGRkolJSXSzp07pV69euktZTx58qTUp08fqaSkRJKkxtm0MTExUv/+/aWjR49KZ86c0R0NDQ3W+hhtZs2aNZJSqZQKCgqkQ4cOSZMnT5a8vLwklUolSZIkTZgwQcrIyNBdv2vXLsnZ2Vl65ZVXpMOHD0tz587lUkYTn+GLL74oKRQKacOGDXq/b5cuXbLWR7AqU5/f33G1gunPsLy8XOrUqZOUlpYmlZWVSZ999pnk6+srLVy4UGhcV1cryOAqyWVuZh0yuNrlagUmBzbmt99+k8aNGyd17NhR8vDwkFJSUvT+0D127JgEQNq6daskSZK0detWCY0zXpocx44ds86HaGNLly6VunXrJikUCmngwIHSd999p3vt9ttvl5KTk/WuX7dundS7d29JoVBIN910k/T555+3ccS2x5Rn2L1792Z/3+bOndv2gdsIU38Hr8XkoJGpz/Dbb7+VYmJiJKVSKYWGhkovvPCC8C9EV5MDQCnJZK5mHYDSLpMDlmwmIiJqxtWSzYASMpl5qxUa/4qtNalkc05ODl5++WWoVCqEh4dj6dKlGDhwoFnvby7OOSAiIjKo7eYcrF27Funp6Zg7dy727t2L8PBwxMfHN5ms3trYc0BERNSMv3oOnCEzc58DCRKABqN7DmJiYnDLLbfgjTfeANA4OTMoKAhPPPEEMjIyzIrBHOw5ICIiMkgy+x9TNkESsWOpKNzngIiI6Los62RXq9V6PyuVSiiVSr1zInYsFYU9B0RERM1QKBR/bsSksejo2LEjgoKC4OnpqTuysrLa/gOZgD0HREREzXB1dcWxY8dQV1dnUTuSJDVZ7fD3XgPAsh1LRWNyQERE1AJXV9c223X22t0iExISAPy1W2RaWlqbxHAVhxWImiGTyfDxxx9bNYZ58+YhIiLCqjG0hjvuuANPPfWUtcMgsknp6elYvnw5Vq5cicOHD+Oxxx5DdXU1UlJS2jYQa+7ARNSWkpOTdbv5OTs7S76+vlJcXJy0YsUKSaPR6F175swZqaamxkqRNrp06ZJ0/vx53c+idtLLz8+XAEhhYWFNXlu3bp0EQOrevbtJbQKQPvroI6Ou/e233yS1Wm1S+9f68MMPpbvuukvy9vaWAEjff/+92W0R2SJDu0W2FfYckEMZPnw4zpw5g+PHj+PLL7/E0KFDMXXqVNx3331oaGjQXefv79/smKAoGo1Gr2hMczp27IgbbrihVd7f3d0dZ8+ebbI8asWKFejWrVurvOfVcVtvb2906tTJ7Haqq6tx22234aWXXhIVGpFNSUtLw4kTJ1BbW4uSkhKDhfdaC5MDcihKpRL+/v4IDAzEgAED8Nxzz+GTTz7Bl19+iYKCAt111w4rDBo0CDNmzNBr59y5c3BxccGOHTsAALW1tZg+fToCAwPh7u6OmJgYbNu2TXd9QUEBvLy8sGnTJvTt2xdKpRLl5eXYtm0bBg4cCHd3d3h5eWHw4ME4ceIEAP1hhXnz5mHlypX45JNPIJPJIJPJsG3bNgwbNqzJWOS5c+egUCiaVJ+8lrOzM8aPH4+8vDzduZMnT2Lbtm0YP358k+s/+eQTDBgwAK6urggNDcXzzz+vS6aCg4MBAA8++CBkMpnu56vxv/POOwgJCdGN2/59WKG2thYzZsxAUFAQlEolevbsiRUrVrQY+4QJE5CZmam3FpyIxGJyQA5v2LBhCA8Px8aNG5t9PTExEWvWrPlzj/RGa9euRUBAAIYMGQKgMdMvLi7GmjVr8MMPP2D06NEYPnw4jhw5orvnypUreOmll/DOO+/gxx9/hLe3NxISEnD77bfjhx9+QHFxMSZPntzsHu7Tp0/Hww8/rOv5OHPmDAYNGoRHH30Uq1evRm1tre7a9957D4GBgRg2bJjBzz1p0iSsW7cOV65cAdCYwAwfPrzJGutvvvkGSUlJmDp1Kg4dOoS3334bBQUFeOGFFwAAe/bsAQDk5+fjzJkzup8B4OjRo/jwww+xceNG7Nu3r9k4kpKS8MEHH+D111/H4cOH8fbbb6Njx44GYyei1sXkgAhAWFgYjh8/3uxrDz/8ME6fPo2dO3fqzq1evRrjxo2DTCZDeXk58vPzsX79egwZMgQ9evTA9OnTcdtttyE/P193T319Pd58800MGjQIffr0QUND45aq9913H3r06IEbb7wRycnJzXbrd+zYEW5ubrqeD39/fygUCjz00EMAGr/ZX1VQUICJEydet1BMZGQkQkNDsWHDBkiShIKCAkyaNKnJdc8//zwyMjKQnJyM0NBQ3HXXXViwYAHefvttAECXLl0AAF5eXvD399f9DDQOJaxatQqRkZG4+eabm7T9888/Y926dcjLy8ODDz6I0NBQ3HnnnRgzZozB2ImodTE5IELz65Cv6tKlC+6++268//77AIBjx46huLgYiYmJAIADBw5Ao9Ggd+/e6Nixo+7Yvn07fvnlF107CoVC7y9Ib29vTJw4EfHx8bj//vvx3//+F2fOnDEpbldXV0yYMEE3PLB3714cPHgQEydONOr+SZMmIT8/H9u3b0d1dTVGjBjR5Jr9+/dj/vz5ep8tNTUVZ86c0fU6tKR79+56ycLf7du3D05OTrj99tuNipeI2gb3OSACcPjwYYSEhLT4emJiIp588kksXboUq1evRv/+/dG/f38AwOXLl+Hk5ITS0lI4OTnp3Xdt97ibm1uTBCQ/Px9PPvkkCgsLsXbtWsyePRubN2/GrbfeanTsjz76KCIiInDy5Enk5+dj2LBh6N69u1H3JiYm4tlnn8W8efMwYcIEODs3/SPh8uXLeP7553W9FNe63vpvd3d3g6+7ubkZFScRtS32HJDD27JlCw4cOIBRo0a1eM3IkSNRU1ODwsJCrF69WtdrADR2z2s0Gpw9exY9e/bUO4zZ1SwyMhIzZ87Et99+i379+mH16tXNXqdQKKDRaJqc79+/P6Kjo7F8+XKsXr262aGBlnh7e+OBBx7A9u3bW7xvwIABKCsra/LZevbsCbm88Y8QFxeXZmO7nv79+0Or1WL79u0m30tErYc9B+RQamtroVKpoNFoUFlZicLCQmRlZeG+++5DUlJSi/e5u7sjISEBc+bMweHDhzFu3Djda71790ZiYiKSkpLw6quvIjIyEufOnUNRURFuvvlm3Hvvvc22eezYMSxbtgwPPPAAAgICUFZWhiNHjrQYR3BwML766iuUlZXhhhtugKenJ1xcXAA09h6kpaXB3d0dDz74oEnPpKCgAG+++WaLyyYzMzNx3333oVu3bvjnP/8JuVyO/fv34+DBg1i4cKEutqKiIgwePBhKpRKdO3c26r2Dg4ORnJyMSZMm4fXXX0d4eDhOnDiBs2fP4uGHH272ngsXLqC8vBynT58GAJSVlQGAbi4GEVmOPQfkUAoLC9G1a1cEBwdj+PDh2Lp1K15//XV88sknTYYE/i4xMRH79+/HkCFDmkwazM/PR1JSEp5++mn06dMHCQkJ2LNnj8E9Azp06ICffvoJo0aNQu/evTF58mRMmTIF//73v5u9PjU1FX369EF0dDS6dOmCXbt26V4bN24cnJ2dMW7cOJO3enVzczO4n0J8fDw+++wz/O9//8Mtt9yCW2+9Fa+99pre0MWrr76KzZs3IygoCJGRkSa9/1tvvYV//vOfePzxxxEWFobU1FRUV1e3eP2mTZsQGRmpS7rGjh2LyMhI5ObmmvS+RNQymXTt+iwiskvHjx9Hjx49sGfPHgwYMMDa4RCRnWNyQGTH6uvr8dtvv2H69Ok4duyYXm8CEZG5OKxAZMd27dqFrl27Ys+ePexWJyJh2HNAREREethzQERERHqYHBAREZEeJgdERESkh8kBERER6WFyQERERHqYHBAREZEeJgdERESkh8kBERER6WFyQERERHr+P14uHaEop7aOAAAAAElFTkSuQmCC\n",
"text/plain": [
"