User Guide

User Guide

Introduction to the Collaboratory

Overview

The Cancer Genome Collaboratory (Collaboratory) is an academic compute cloud resource that allows researchers to run complex analysis operations across large ICGC cancer genome data sets.

Collaboratory Resource Allocations

A new project account creation on the Collaboratory can be requested through the application process. Upon submitting the signed application form(s), we will send you an email with further instructions.

Collaboratory users are expected to:

  • Make appropriate use of Collaboratory resources and use good social behavior.

  • Do not share private ssh keys or login information - one user per resource allocation.

  • Cite the Collaboratory in any papers describing research.

  • Regularly respond to quarterly Collaboratory allocation surveys.

  • Submit tickets to the Collaboratory support ticketing system when encountering technical issues not covered by the Collaboratory support documentation.

Protected Data Resources

The Collaboratory currently contains a growing repository of ICGC alignment and variant data sourced from the ICGG PanCancer Analysis of Whole Genomes project (PCAWG). The content is maintained in collaboration with the ICGC Data Coordination Center (DCC). It is the responsibility of the users to protect sensitive data they receive access to.

How can I use an OpenStack cloud?

As an OpenStack cloud end user, you can provision your own resources within the limits set by cloud administrators.

The examples in this guide show you how to perform tasks by using the following methods:

  • OpenStack dashboard: Use this web-based graphical interface, code named horizon, to view, create, and manage resources.

  • OpenStack command-line clients: Each core OpenStack project has a command-line client that you can use to run simple commands to view, create, and manage resources in a cloud and automate tasks by using scripts.

You can modify these examples for your specific use cases.

In addition to these ways of interacting with a cloud, you can access the OpenStack APIs directly or indirectly through cURL commands or open SDKs. You can automate access or build tools to manage resources and services by using the native OpenStack APIs or the EC2 compatibility API.

To use the OpenStack APIs, it helps to be familiar with HTTP/1.1, RESTful web services, the OpenStack services, and JSON or XML data serialization formats.

OpenStack dashboard

As a cloud end user, you can use the OpenStack dashboard to provision your own resources within the limits set by administrators. You can modify the examples provided in this section to create other types and sizes of server instances.

Log in to the dashboard

OpenStack dashboard — Project tab

Projects are organizational units in the cloud and are also known as tenants or accounts. Each user is a member of one or more projects. Within a project, a user creates and manages instances.

From the Project tab, you can view and manage the resources in a selected project, including instances and images. You can select the project from the drop-down menu at the top left. If the cloud supports multi-domain model, you can also select the domain from this menu.

OpenStack dashboard project tab

From the Project tab, you can access the following categories:

Compute tab

  • Overview: View reports for the project.

  • Instances: View, launch, create a snapshot from, stop, pause, or reboot instances, or connect to them through VNC.

    OpenStack dashboard compute tab instances window
  • Volumes: Use the following tabs to complete these tasks:

    • Volumes: View, create, edit, and delete volumes.

    • Volume Snapshots: View, create, edit, and delete volume snapshots.

    OpenStack dashboard compute tab volumes window
  • Images: View images and instance snapshots created by project users, plus any images that are publicly available. Create, edit, and delete images, and launch instances from images and snapshots.

    OpenStack dashboard compute tab images window
  • Access & Security: Use the following tabs to complete these tasks:

    • Security Groups: View, create, edit, and delete security groups and security group rules.

    • Key Pairs: View, create, edit, import, and delete key pairs.

    • Floating IPs: Allocate an IP address to or release it from a project.

    • API Access: View API endpoints.

    OpenStack dashboard compute tab accessibility and security window

Network tab

  • Network Topology: View the network topology.

  • Networks: Create and manage public and private networks.

    OpenStack dashboard network tab networks window
  • Routers: Create and manage routers.

Object Store tab

  • Containers: Create and manage containers and objects.

    OpenStack dashboard object store tab containers window

Upload and manage images

A virtual machine image, referred to in this document simply as an image, is a single file that contains a virtual disk that has a bootable operating system installed on it. Images are used to create virtual machine instances within the cloud. For information about creating image files, see the OpenStack Virtual Machine Image Guide.

Depending on your role, you may have permission to upload and manage virtual machine images. Operators might restrict the upload and management of images to cloud administrators or operators only. If you have the appropriate privileges, you can use the dashboard to upload and manage images in the admin project.

Note

You can also use the openstack, glance and nova command-line clients or the Image service and Compute APIs to manage images. For more information see Manage images.

Upload an image

Follow this procedure to upload an image to a project:

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Images category.

  4. Click Create Image.

  5. The Create An Image dialog box appears.

    A screenshot of the create an image dialog box
  6. Dashboard — Create Image

  7. Enter the following values:

    Name

    Enter a name for the image.

    Description

    Enter a brief description of the image.

    Image Source

    Choose the image source from the dropdown list. Your choices are Image Location and Image File.

    Image File or Image Location

    Based on your selection for Image Source, you either enter the location URL of the image in the Image Location field, or browse for the image file on your file system and add it.

    Format

    Select the image format (for example, QCOW2) for the image.

    Architecture

    Specify the architecture. For example, i386 for a 32-bit architecture or x86_64 for a 64-bit architecture.

    Minimum Disk (GB)

    Leave this field empty.

    Minimum RAM (MB)

    Leave this field empty.

    Copy Data

    Specify this option to copy image data to the Image service.

    Public

    Select this check box to make the image public to all users with access to the current project.

    Protected

    Select this check box to ensure that only users with permissions can delete the image.

  8. Click Create Image.

  9. The image is queued to be uploaded. It might take some time before the status changes from Queued to Active.

Update an image

Follow this procedure to update an existing image.

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. Select the image that you want to edit.

  4. In the Actions column, click the menu button and then select Edit Image from the list.

  5. In the Edit Image dialog box, you can perform various actions. For example:

    • Change the name of the image.

    • Select the Public check box to make the image public.

    • Clear the Public check box to make the image private.

  6. Click Edit Image.

Delete an image

Deletion of images is permanent and cannot be reversed. Only users with the appropriate permissions can delete images.

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Images category.

  4. Select the images that you want to delete.

  5. Click Delete Images.

  6. In the Confirm Delete Images dialog box, click Delete Images to confirm the deletion.

Configure access and security for instances

Before you launch an instance, you should add security group rules to enable users to ping and use SSH to connect to the instance. Security groups are sets of IP filter rules that define networking access and are applied to all instances within a project. To do so, you either add rules to the default security group or add a new security group with rules.

Key pairs are SSH credentials that are injected into an instance when it is launched. To use key pair injection, the image that the instance is based on must contain the cloud-init package. Each project should have at least one key pair. For more information, see the section “Add a key pair”.

If you have generated a key pair with an external tool, you can import it into OpenStack. The key pair can be used for multiple instances that belong to a project. For more information, see the section “Import a key pair”.

Note

A key pair belongs to an individual user, not to a project. To share a key pair across multiple users, each user needs to import that key pair.

When an instance is created in OpenStack, it is automatically assigned a fixed IP address in the network to which the instance is assigned. This IP address is permanently associated with the instance until the instance is terminated. However, in addition to the fixed IP address, a floating IP address can also be attached to an instance. Unlike fixed IP addresses, floating IP addresses are able to have their associations modified at any time, regardless of the state of the instances involved.

Add a rule to the default security group

This procedure enables SSH and ICMP (ping) access to instances. The rules apply to all instances within a given project, and should be set for every project unless there is a reason to prohibit SSH or ICMP access to the instances.

This procedure can be adjusted as necessary to add additional security group rules to a project, if your cloud requires them.

Note

When adding a rule, you must specify the protocol used with the destination port or source port.

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Access & Security category. The Security Groups tab shows the security groups that are available for this project.

  4. Select the default security group and click Manage Rules.

  5. To allow SSH access, click Add Rule.

  6. In the Add Rule dialog box, enter the following values:

    • Rule: SSH

    • Remote: CIDR

    • CIDR: 0.0.0.0/0

    Note

    0.0.0.0/0 represents the entire Internet anf it is not recommended to allow such a wide access. To accept requests from a particular range of IP addresses, specify the IP address block in the CIDR box.

  7. Click Add.

  8. Instances will now have SSH port 22 open for requests from any IP address.

    A screenshot of the Add Rule dialog box
  9. To add an ICMP rule, click Add Rule.

  10. In the Add Rule dialog box, enter the following values:

    • Rule: All ICMP

    • Direction: Ingress

    • Remote: CIDR

    • CIDR: 0.0.0.0/0

  11. Click Add.

  12. Instances will now accept all incoming ICMP packets.

Add a key pair

Create at least one key pair for each project.

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Access & Security category.

  4. Click the Key Pairs tab, which shows the key pairs that are available for this project.

  5. Click Create Key Pair.

  6. In the Create Key Pair dialog box, enter a name for your key pair, and click Create Key Pair.

  7. Respond to the prompt to download the key pair.

Import a key pair

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Access & Security category.

  4. Click the Key Pairs tab, which shows the key pairs that are available for this project.

  5. Click Import Key Pair.

  6. In the Import Key Pair dialog box, enter the name of your key pair, copy the public key into the Public Key box, and then click Import Key Pair.

  7. Save the *.pem file locally.

  8. To change its permissions so that only you can read and write to the file, run the following command:

    $ chmod 0600 yourPrivateKey.pem

    Note

    If you are using the Dashboard from a Windows computer, use PuTTYgen to load the *.pem file and convert and save it as *.ppk. For more information see the WinSCP web page for PuTTYgen.

  9. To make the key pair known to SSH, run the ssh-add command.

    $ ssh-add yourPrivateKey.pem

The Compute database registers the public key of the key pair. The Dashboard lists the key pair on the Access & Security tab.

Allocate a floating IP address to an instance

When an instance is created in OpenStack, it is automatically assigned a fixed IP address in the network to which the instance is assigned. This IP address is permanently associated with the instance until the instance is terminated.

However, in addition to the fixed IP address, a floating IP address can also be attached to an instance. Unlike fixed IP addresses, floating IP addresses can have their associations modified at any time, regardless of the state of the instances involved. This procedure details the reservation of a floating IP address from an existing pool of addresses and the association of that address with a specific instance.

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Access & Security category.

  4. Click the Floating IPs tab, which shows the floating IP addresses allocated to instances.

  5. Click Allocate IP To Project.

  6. Choose the pool from which to pick the IP address.

  7. Click Allocate IP.

  8. In the Floating IPs list, click Associate.

  9. In the Manage Floating IP Associations dialog box, choose the following options:

    • The IP Address field is filled automatically, but you can add a new IP address by clicking the + button.

    • In the Port to be associated field, select a port from the list.

    • The list shows all the instances with their fixed IP addresses.

  10. Click Associate.

Launch and manage instances

Instances are virtual machines that run inside the cloud. You can launch an instance from the following sources:

  • Images uploaded to the Image service.

  • Image that you have copied to a persistent volume. The instance launches from the volume, which is provided by the cinder-volume API.

  • Instance snapshot that you took.

Launch an instance

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Instances category.

    The dashboard shows the instances with its name, its private and floating IP addresses, size, status, task, power state, and so on.

  4. Click Launch Instance.

  5. In the Launch Instance dialog box, specify the following values:

    Details tab

    Instance Name

    Assign a name to the virtual machine.

    Availability Zone

    By default, this value is set to nova.

    Note

    The name you assign here becomes the initial host name of the server. If the name is longer than 63 characters, the Compute service truncates it automatically to ensure dnsmasq works correctly.

    After the server is built, if you change the server name in the API or change the host name directly, the names are not updated in the dashboard.

    Server names are not guaranteed to be unique when created so you could have two instances with the same host name.

    Flavor

    Specify the size of the instance to launch.

    Note

    The flavor is selected based on the size of the image selected for launching an instance. For example, while creating an image, if you have entered the value in the Minimum RAM (MB) field as 2048, then on selecting the image, the default flavor is m1.small.

    Count

    To launch multiple instances, enter a value greater than 1. The default is 1.

    Instance Boot Source

    Your options are:

    Boot from image
    If you choose this option, a new field for Image Name displays. You can select the image from the list.
    Boot from snapshot
    If you choose this option, a new field for Instance Snapshot displays. You can select the snapshot from the list.
    Boot from volume
    If you choose this option, a new field for Volume displays. You can select the volume from the list.
    Boot from image (creates a new volume)
    With this option, you can boot from an image and create a volume by entering the Device Size and Device Name for your volume. Click the Delete Volume on Instance Delete option to delete the volume on deleting the instance.
    Boot from volume snapshot (creates a new volume)
    Using this option, you can boot from a volume snapshot and create a new volume by choosing Volume Snapshot from a list and adding a Device Name for your volume. Click the Delete Volume on Instance Delete option to delete the volume on deleting the instance.

    Image Name

    This field changes based on your previous selection. If you have chosen to launch an instance using an image, the Image Name field displays. Select the image name from the dropdown list.

    Instance Snapshot

    This field changes based on your previous selection. If you have chosen to launch an instance using a snapshot, the Instance Snapshot field displays. Select the snapshot name from the dropdown list.

    Volume

    This field changes based on your previous selection. If you have chosen to launch an instance using a volume, the Volume field displays. Select the volume name from the dropdown list. If you want to delete the volume on instance delete, check the Delete Volume on Instance Delete option.

    A screenshot of the luanch instance disalogue's details tab

    Access & Security tab

    Key Pair

    Specify a key pair. If the image uses a static root password or a static key set (neither is recommended), you do not need to provide a key pair to launch the instance.

    Security Groups

    Activate the security groups that you want to assign to the instance that define which incoming network traffic is forwarded to instances. If you have not created any security groups, you can assign only the default security group to the instance.

    A screenshot of the luanch instance disalogue's access and security tab

    Networking tab

    Selected Networks

    To add a network to the instance, click the + in the Available field.
    A screenshot of the luanch instance disalogue's networking tab

    Post-creation tab

    Customization Script Source

    Specify a customization script that runs after your instance launches. Examples of customization scripts are available at here.

  6. Click Launch Instance.

The instance starts on a compute node in the cloud.

Note

If you did not provide a key pair, security groups, or rules, users can access the instance only from inside the cloud through VNC. Even pinging the instance is not possible without an ICMP rule configured.

When you launch an instance from an image, OpenStack creates a local copy of the image on the compute node where the instance starts.

Connect to your instance by using SSH

To use SSH to connect to your instance, use the downloaded keypair file.

Note

The user name is ubuntu for the Ubuntu cloud images. For all other images, the user can be found by clicking on the image name in the Images tab, and consulting the description attached to the image.

A screenshot of an example showing the description attached to an image.

Copy the IP address for your instance.

  1. Use the ssh command to make a secure connection to the instance. For example:

    $ ssh -i MyKey.pem ubuntu@142.1.177.X
  2. At the prompt, type yes.

Create an instance snapshot

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click the Instances category.

  4. Select the instance from which to create a snapshot.

  5. In the actions column, click Create Snapshot.

  6. In the Create Snapshot dialog box, enter a name for the snapshot, and click Create Snapshot.

  7. The Images category shows the instance snapshot.

A screenshot of the create snapshot dialog box

To launch an instance from the snapshot, select the snapshot and click Launch. Proceed with launching an instance.

Manage an instance

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Compute tab and click Instances category.

  4. Select an instance.

  5. In the menu list in the actions column, select the state.

  6. You can resize or rebuild an instance. You can also choose to view the instance console log, edit instance or the security groups. Depending on the current state of the instance, you can pause, resume, suspend, soft or hard reboot, or terminate it.

A screenshot of the instances category's actions column.

Volumes

Scope

In this section you will learn how to:

Introduction

Openstack volumes are logical block devices that can be attached to a single instance to provide a persistent location for data storage. Unlike a virtual machine's local disk which is destroyed along with the VM, volumes are decoupled from instances so that they may be attached and reattached to different instances. This allows you to create & destroy instances as you see fit while maintaining a persistent store for data.

Create a volume

From the Openstack dashboard, click Compute > Volumes > Create Volume, fill out the form by providing a meaningful name, description and size. Once this is done, click ‘Create Volume’. Your volume will be created and you can move on to the next step, ‘Attach a volume to an instance’.

A screenshot of the Create Volume dialog box

Attach a volume to an Instance

From the Openstack dashboard, click Compute > Volumes and then on the Actions drop down for the volume you want to attach to an Instance, click ‘Manage Attachments’.

A screenshot of the Actions drop down for the volumes.

From the Manage Volume Attachments form, select the Instance you wish to attach to and click ‘Attach Volume’. This will dynamically attach the volume as a new logical block device to your instance. Move on to the next section of identifying the block device from within your Instance.

A screenshot of the Manage Volume Attachments form.

Identifying volume within your Instance

SSH into your instance, su to root and run ‘lsblk’

A screenshot of result of running lsblk.

‘vda’ is the first disk of your Instance and contains one partition ‘vda1’ which contains the operating system. Any volumes attached to your Instance will be named ‘vdb, vdc, vdd’ & etc. Identify the new disk by checking the size and comparing that to the size you created in the Openstack dashboard. Now that you have identified the logical block device from within the Instance you can move on to Formatting.

Formatting the Volume

Prerequisite: Install xfsprogs

As root:

apt-get install xfsprogs -y

Format using xfs

We are using xfs in this example because it is widely supported and supports extremely large volume sizes (9 exabytes)

As root:

mkfs.xfs -f /dev/vdb
A screenshot of result of running the above code.

Mounting the Volume

Create a directory to mount the volume (example ‘db1’), mount the volume and then verify the mount.

As root:

"mkdir db1
mount -t xfs /dev/vdb /db1
df -hT /db1
"
A screenshot of result of running the above code.

To make the mount persist on reboots, add the following to /etc/fstab

/dev/vdb	/db1	xfs	defaults	0	0

At this point the volume is ready to read & write data.

Re-attaching the volume to a different Instance

Unmount the Volume

As root:

umount /db1

Remove or comment out the line added in /etc/fstab if applicable.

Detach Volume

From the Openstack dashboard, click Compute > Volumes and then on the Actions drop down for the volume you want to attach to an Instance, click ‘Manage Attachments’.

From the Manage Volume Attachments form, Click ‘Detach Volume’ for the instance you wish to detach the volume from.

A screenshot of the Manage Volume Attachments form.

Verify the device is no longer seen by the Instance by looking at the output of ‘lsblk’.

Re-attach Volume

Follow the steps at the beginning of this guide to attach the volume to a desired instance. DO NOT Format the volume from the new instance. Skip that step and go directly to mounting. The volume is now mounted on a different Instance and the data is preserved.

Extend Volume (Grow)

Volumes can also be extended in size. In order to Extend the volume it must be unmounted and detached from an Instance.

Note

Volumes cannot have their size reduced.

From the Openstack dashboard, click Compute > Volumes and then on the Actions drop down select ‘Extend Volume’.

A screenshot of the Extended Volume dialog box.

Enter the new size of the volume and click Extend Volume.

Re-attach the volume to your Instance, and remount the volume from within the instance. You will need to grow the xfs filesystem before it will make the new space available.

As root:

xfs_growfs /dev/vdb -d

Create and manage object containers

OpenStack Object Storage (swift) is used for redundant, scalable data storage using clusters of standardized servers to store petabytes of accessible data. It is a long-term storage system for large amounts of static data which can be retrieved and updated.

OpenStack Object Storage provides a distributed, API-accessible storage platform that can be integrated directly into an application or used to store any type of file, including VM images, backups, archives, or media files. In the OpenStack dashboard, you can only manage containers and objects.

In OpenStack Object Storage, containers provide storage for objects in a manner similar to a Windows folder or Linux file directory, though they cannot be nested. An object in OpenStack consists of the file to be stored in the container and any accompanying metadata.

Create a container

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Object Store tab and click Containers category.

  4. Click Create Container.

  5. In the Create Container dialog box, enter a name for the container, and then click Create.

You have successfully created a container.

Note

To delete a container, click the More button and select Delete Container.

Upload an object

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Object Store tab and click Containers category.

  4. Select the container in which you want to store your object.

  5. Click the Upload Object icon.

    A screenshot of the Project tab showing where the Upload Object icon is at the right upper corner.
  6. The Upload File To Container: <name> dialog box appears. <name> is the name of the container to which you are uploading the object.

  7. Enter a name for the object.

  8. Browse to and select the file that you want to upload.

  9. Click Upload File.

A screenshot of the Upload Object to container dialog box.

You have successfully uploaded an object to the container.

Note

To delete an object, click the More button and select Delete Object.

Manage an object

To edit an object
  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Object Store tab and click Containers category.

  4. Select the container in which you want to store your object.

  5. Click the menu button and choose Edit from the dropdown list.

  6. The Edit Object dialog box is displayed.

  7. Browse to and select the file that you want to upload.

  8. Click Update Object

Note

To delete an object, click the menu button and select Delete Object.

To create a metadata-only object without a file

You can create a new object in container without a file available and can upload the file later when it is ready. This temporary object acts a place-holder for a new object, and enables the user to share object metadata and URL info in advance.

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Object Store tab and click Containers category.

  4. Select the container in which you want to store your object.

  5. Click Upload Object.

  6. The Upload Object To Container: <name> dialog box is displayed.

  7. <name> is the name of the container to which you are uploading the object.

  8. Enter a name for the object.

  9. Click Update Object.

To create a pseudo-folder

Pseudo-folders are similar to folders in your desktop operating system. They are virtual collections defined by a common prefix on the object’s name.

  1. Log in to the dashboard.

  2. Select the appropriate project from the drop down menu at the top left.

  3. On the Project tab, open the Object Store tab and click Containers category.

  4. Select the container in which you want to store your object.

  5. Click Create Pseudo-folder.

    A screenshot of the Containers category showing where the create pseudo-folder button is located.
  6. The Create Pseudo-Folder in Container <name> dialog box is displayed. <name> is the name of the container to which you are uploading the object.

  7. Enter a name for the pseudo-folder.

  8. A slash (/) character is used as the delimiter for pseudo-folders in Object Storage.

  9. Click Create.

    A screenshot of the Containers category after creating a pseudo-folder.
International Cancer Genome Consortium
Dockstore
Ontario Institute for Cancer Research

© 2016 Cancer Genome Collaboratory. All rights reserved.