Copying ImageNet to EFS.
January 10, 2019 ยท View on GitHub
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copies ImageNet from Edinburgh to AWS\n",
"==================================\n",
"\n",
"The [original notebook][original] downloads imagenet from Kaggle. I'm paranoid, and I downloaded ImageNet from image-net.org directly. I want to absolutely sure that the version of ImageNet I'm running on AWS is the same as the one I'm running locally. So, I'm going to copy it from our machines to AWS.\n",
"\n",
"[original]: https://github.com/fastai/imagenet-fast/blob/master/aws/download_imagenet.ipynb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Create spot instance\n",
"2. Mount EFS\n",
"3. Copy from Edinburgh to AWS"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [],
"source": [
"# not goint to edit code so don't need this\n",
"#%reload_ext autoreload\n",
"#%autoreload 2\n",
"#%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from aws_setup import "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Define parameters"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"vpc_name='degenerate'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Get Existing VPC by tag name"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"vpc = get_vpc(vpc_name); vpc"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n"
]
}
],
"source": [
"print(vpc)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Why did that fail?\n",
"\n",
"Because this list is empty:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(ec2.vpcs.filter(Filters=[{'Name': 'tag-value', 'Values': [vpc_name]}]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Why is that list empty?"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ec2.vpcsCollection(ec2.ServiceResource(), ec2.Vpc)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ec2.vpcs.filter()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks like it's trying to connect to a fast-ai virtual cloud that exists in a different region. I guess I'll make my own, working from this boto4 tutorial."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"vpc-0db9a09055c68d289\n"
]
}
],
"source": [
"# create VPC\n",
"vpc = ec2.create_vpc(CidrBlock='192.168.0.0/16')\n",
"# we can assign a name to vpc, or any resource, by using tag\n",
"vpc.create_tags(Tags=[{"Key": "Name", "Value": f"{vpc_name}"}])\n",
"vpc.wait_until_available()\n",
"print(vpc.id)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ec2.Vpc(id='vpc-0db9a09055c68d289')"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vpc = get_vpc(vpc_name); vpc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Create EFS (if you haven't already)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"efs_tag = f'{vpc_name}-efs'"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Could not get VPC info: list index out of range\n"
]
},
{
"ename": "UnboundLocalError",
"evalue": "local variable 'sg' referenced before assignment",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mUnboundLocalError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32mm4.large seems like a safe choice."
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"instance_name = f'{vpc_name}-copy-imagenet'\n",
"# Recommend a high compute instance as we need to do multi-threaded resizing later on\n",
"instance_type = 'm4.large'"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Spot price: 0.032900, Bid price: 0.0987\n"
]
}
],
"source": [
"spot_price = get_spot_prices()[instance_type]\n",
"bid_price = "%.4f" % (float(spot_price)3)\n",
"print(f'Spot price: {spot_price}, Bid price: {bid_price}')"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [],
"source": [
"launch_specs = LaunchSpecs(vpc, instance_type=instance_type).build()"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [],
"source": [
"launch_specs['BlockDeviceMappings'][0]['Ebs']['VolumeSize'] = 1000"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'ImageId': 'ami-c6ac1cbc',\n",
" 'InstanceType': 'm4.large',\n",
" 'KeyName': 'aws-key-degenerate',\n",
" 'NetworkInterfaces': [{'DeviceIndex': 0,\n",
" 'SubnetId': 'subnet-04f6f32213886fb81',\n",
" 'Groups': ['sg-0279bcfad97a5dbc7'],\n",
" 'AssociatePublicIpAddress': True}],\n",
" 'BlockDeviceMappings': [{'DeviceName': '/dev/sda1',\n",
" 'Ebs': {'VolumeSize': 1000,\n",
" 'DeleteOnTermination': True,\n",
" 'VolumeType': 'gp2'}}]}"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"launch_specs"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"ename": "ClientError",
"evalue": "An error occurred (InvalidKeyPair.NotFound) when calling the RequestSpotInstances operation: The key pair 'aws-key-degenerate' does not exist",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mClientError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m\n", "rsync -avz -e \"ssh -i ~/.ssh/aws-key-degenerate.pem\" $PWD ubuntu@54.172.175.127:/home/ubuntu/efs_mount\n", ""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Checking progress. Full size to transfer is:"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"stty: standard input: Inappropriate ioctl for device\n",
"146G\t/disk/scratch/datasets/imagenet\n"
]
}
],
"source": [
"!ssh apollo1 "du -hs /disk/scratch/datasets/imagenet""
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [],
"source": [
"full_size = 1461e9"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [],
"source": [
"efs_size = lambda : 1e9~/efs."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('', '')"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"run_command(client, 'cp -r ~/efs_mount ~/efs') # no reformatting"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get rid of that instance:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'TerminatingInstances': [{'CurrentState': {'Code': 32,\n",
" 'Name': 'shutting-down'},\n",
" 'InstanceId': 'i-02d21e221f464e4c6',\n",
" 'PreviousState': {'Code': 16, 'Name': 'running'}}],\n",
" 'ResponseMetadata': {'RequestId': '94d722d5-32a2-45ef-b769-52556e2d2a83',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'text/xml;charset=UTF-8',\n",
" 'transfer-encoding': 'chunked',\n",
" 'vary': 'Accept-Encoding',\n",
" 'date': 'Thu, 10 Jan 2019 21:28:44 GMT',\n",
" 'server': 'AmazonEC2'},\n",
" 'RetryAttempts': 0}}"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"instance.terminate()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}