EPAM Cloud Orchestrator
EPAM Cloud Orchestrator AWS Utilization User Guide January 2015 CIUG-10 Version 2.2 EPAM Cloud Orchestrator - AWS Utilization Guide Contents Preface .................................................................................................................................... 4 About this Guide .............................................................................................................. 4 Audience ......................................................................................................................... 4 The Structure of the Guide .............................................................................................. 4 Documentation References ............................................................................................. 5 1 Overview ......................................................................................................................... 6 2 AWS Global Infrastructure and Regions ......................................................................... 7 3 AWS Images ................................................................................................................... 9 3.1 A Notion of an Image .............................................................................................. 9 3.2 Available AWS Images............................................................................................ 9 4 Working with AWS with EO ........................................................................................... 12 4.1 Activating an AWS Region in Cloud ...................................................................... 12 4.2 Managing Instances in AWS ................................................................................. 12 4.2.1 Running an Instance with Management Console ......................................... 13 4.2.2 Running an Instance with CLI ....................................................................... 14 4.3 Connecting to an Image ........................................................................................ 15 4.3.1 Connecting to a Linux VM ............................................................................. 15 4.3.2 Connecting to a Windows VM ....................................................................... 15 4.4 AWS Management Tools ...................................................................................... 16 4.5 Maestro CLI Specifics for AWS............................................................................. 17 5 EPAM Orchestration Services in AWS ......................................................................... 19 5.1 Mobile Management Console ............................................................................... 19 5.2 Cloud Monitoring and Audit for AWS .................................................................... 20 5.3 Auto Configuration Service ................................................................................... 21 5.3.1 Configuring your VM with Chef ..................................................................... 22 5.3.2 Viewing Chef Client Information .................................................................... 22 6 Environment Orchestration ........................................................................................... 23 6.1 AWS CloudFormation Template Introduction ....................................................... 23 6.2 Working with Stacks via Maestro-CLI ................................................................... 24 2 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 6.3 Working with Stacks via Create Wizard ................................................................ 25 7 Billing and Reporting ..................................................................................................... 26 7.1 Pricing ................................................................................................................... 26 7.2 Reporting ............................................................................................................... 26 7.3 Reporting Customization ....................................................................................... 27 8 Security ......................................................................................................................... 29 9 Self-Education ............................................................................................................... 30 9.1 AWS Partner Network ........................................................................................... 30 9.2 AWS Trainings and Webinars ............................................................................... 30 10 Support and Consulting................................................................................................. 31 Annex A. The Supported CLI Commands ............................................................................ 32 Basic Commands .......................................................................................................... 32 Instances ....................................................................................................................... 32 Storage Volumes ........................................................................................................... 33 Audit and Billing ............................................................................................................. 33 Security and Connection ............................................................................................... 34 Stacks ............................................................................................................................ 34 Images ........................................................................................................................... 34 Instance Properties ....................................................................................................... 35 Table of Figures ............................................................................................................ 36 Version History .............................................................................................................. 37 EPAM CONFIDENTIAL 3 EPAM Cloud Orchestrator - AWS Utilization Guide Preface About this Guide The guide is intended to provide the user with all the information that would allow them start working with AWS by means of EPAM Orchestration tools from scratch. Here, you can find the details on the integration of Amazon Web Services with EPAM Private Cloud. The guide contains the information on Amazon Global Infrastructure and resources specifics, the way an AWS-based infrastructure can be manipulated via EPAM Orchestration, as well as all the necessary technical information on the resources and controls. Audience This guide is designed for EPAM Private Cloud users who want to use Amazon Web Services in following cases: there is a need to place the resources in a public region; there is a need to use specific Amazon features; the customer wants AWS to be used; there is a need to locate the production servers in AWS; it is recommended due to the geographic position. The Structure of the Guide The guide includes ten sections: 1. The Overview section gives the general idea of AWS integration with EO. 2. The AWS Global Infrastructure and Regions section describes the general infrastructure of AWS, introduces the notions of a region and availability zone. 3. The AWS Images section provides the information on the machine images that can be used in AWS-type regions. 4. The Working with AWS via EO section gives the useful information on AWS regions activation in EPAM Orchestration and the ways AWS-based infrastructure can be manipulated with maestro CLI. 5. The EPAM Orchestration Services in AWS section describes how EPC services are used with AWS-based infrastructures. 6. The Environment Orchestration section gives the details on Maestro and CloudFormation stacks usage. 7. The Billing and Reporting section provides the information in AWS costs and the reports on AWS-based infrastructures. 8. The Security section describes the security arrangements introduced for AWS-based infrastructures. 9. In the Self-Education section, you can find the resources that you can youse to improve your AWS expertise. 10. The Support and Consulting section gives the contacts of the groups responsible for providing support to EPC users. 4 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide Documentation References The answers to most of your questions can be found in our Knowledge Base You might also want to check the following EPAM Cloud Orchestrator documents: Document Title Contains Information on Maestro CLI Setup Guide Installation of Maestro CLI Client Auto Configuration: Box Solutions Auto Configuration Service concept and usage Maestro CLI Reference Guide EPAM Private Cloud Billing Guide Graphical User Interface Guide Maestro Stacks Guide EPAM Cloud Orchestrator Command Line Interface and the list of CLI commands, their parameters and response examples Current billing model implemented for EPAM Private Cloud EPAM Cloud Orchestration graphical user interface Maestro Stacks creation and utilization Please email your comments and feedback to EPAM Cloud Consulting at [email protected] to help us provide you with documentation that is as clear, correct and readable as possible. EPAM CONFIDENTIAL 5 EPAM Cloud Orchestrator - AWS Utilization Guide 1 Overview It often happens that a project needs placing the development and production environments in different clouds. The development process is performed in EPAM Private Cloud (EPC) that is not accessible for external connections, and the production is deployed to AWS Cloud. One of the main inconveniences of such solution was the difference of the API of the two cloud providers. In order to solve this issue, Maestro CLI API was extended so that it now allows using the same set of commands for manipulating resources in both EPAM and Amazon clouds. This means that for the developer who uses EPAM Orchestration tools to work with EPC and AWS, the only thing that distinguishes between them is the region specification at the command call: EPC: or2run -p DEMO-PRO -r EPAM-MSQ -i W2012Std AWS: or2run -p DEMO-PRO -r AWS-USEAST -i W2012Std In spite EPC and AWS usage was unified, there is still a set of differences you should take into account in order to prevent unexpected issues. These differences are caused by the following factors: Internal Amazon specifics. AWS is based on several virtualization regions, each having its own specifics. That’s why some Amazon services are not available in this or that region. For more details, see the AWS Global Infrastructure and Regions section. Differences in Cloud providers facilities. Due to some functional dissimilarities of EPAM and Amazon Clouds, some of EPAM Orchestration features and services are not available in AWS. For more details, please, see the EPAM Orchestration Services in AWS section and Annex A. Security. The security of your data stored in AWS is provided by a set of additional measures introduced on two levels: o VM-layer security: Additional restrictions on authentication were introduced. Meanwhile, the software protection on AWS instances is weaker than that on o EPC. Networking: The VMs in EPAM private Cloud are placed inside the EPAM Network and are more protected than the VMs in AWS. That’s why we would recommend to take additional protection measures, such as placing your infrastructure in VPC (Amazon Virtual Cloud). For more details on the data security policy for AWS-based infrastructures, please, see the Security section. These and the other specifics and details of creating and manipulating a virtual infrastructure in AWS via EPAM Orchestration are given further in this document. For more details on AWS architecture in examples and infographics, as well as for useful tips and recommendations on building a reliable and highly scalable infrastructure in AWS, please, visit the Amazon Web Services Architecture Center page. 6 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 2 AWS Global Infrastructure and Regions To provide EPC users with the ability to work with Amazon Web Services, EPAM Orchestration has to access AWS infrastructure. AWS is hosted in multiple locations. Each location, or a region, is a separate geographic area and is independent of the other areas. At the moment when this document is being created, AWS users can deploy their applications to ten regions, and eight of them are available for EPAM Orchestrator. Each region consists of a number of availability zones. An availability zone is an area isolated from other zones (having a low latency connection with the other zones in the same region). In case one zone fails, the others are still up. Below, you can see the map of existing Amazon regions and availability zones. Figure 1 - AWS Regions Map EPAM Private Cloud Users have access to the following Amazon regions: Region Code ap-northeast-1 ap-southeast-1 ap-southeast-2 eu-west-1 eu-central-1 sa-east-1 us-east-1 us-west-1 us-west-2 EPAM CONFIDENTIAL Region Name Asia Pacific (Tokyo) Region Asia Pacific (Singapore) Region Asia Pacific (Sydney) Region EU (Ireland) Region EU (Frankfurt) Region South America (Sao Paulo) Region US East (Northern Virginia) Region US West (Northern California) Region US West (Oregon) Region EPC Reference Name AWS-AP-NORTHEAST AWS-AP-SOUTHEAST AWS-AP-SOUTHEAST-2 AWS-EUWEST AWS-EUCENTRAL AWS-SAEAST AWSUSEAST AWS-USWEST AWS-USWEST-2 7 EPAM Cloud Orchestrator - AWS Utilization Guide Each of the regions has its own specifics and AWS Services availability set. The region that covers all available AWS services is US-EAST-1 (Northern Virginia). The regions with the most restricted possibilities are SA-EAST-1 (Sao Paolo) and AP-SOUTHEAST-2 (Sydney). When you create resources in AWS using EPAM tools, you can specify the region, but the availability zone is selected automatically. The complete information on AWS services coverage by regions is given on AWS Products and Services by Region page. The more detailed information on the global structure of Amazon Web Services is given on Amazon Global Infrastructure page. 8 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 3 AWS Images 3.1 A Notion of an Image A special virtual appliance used as a basis for creating a virtual machine is called a machine image. In Internet, you can also come across the terms ‘template’ or just ‘image’. A machine image used for running instances in AWS is called an Amazon Machine Image (AMI). An AMI provides the information that is necessary to run a new instance in Cloud and includes the following elements: A template for the root volume for the instance (for example, an operating system, an application server, and applications) Launch permissions that control which AWS accounts can use the AMI A block device mapping that specifies the volumes that are to be attached to the instance when it’s launched (see more at Amazon Machine Images page). The owner of an AMI determines its availability by specifying launch permissions, which can be of one of the categories below: Public: all AWS accounts have the permissions to use the AMI. Explicit : only the accounts specified by the owner have the permissions to use the AMI. Implicit: the owner has implicit launch permissions for the AMI. 3.2 Available AWS Images The image format is typically bound to the virtualizer this image is aimed to be run with. Therefore, the images that can be used by one software, are inapplicable for another. That’s why EPAM Orchestration uses different types of images to run instances in EPC and in AWS. However, all the images available for AWS are functional ‘twins’ of EPC public images. This means that the VMs run from a pair of ‘twin’ images in different clouds will have the same Operating System. To provide better usability, Maestro CLI supports referencing images with a set of aliases, with one alias referencing both ‘twin’ images. Maestro CLI detects which image is exactly to be used basing on the region specification in the command where the alias is referenced: EPAM CONFIDENTIAL 9 EPAM Cloud Orchestrator - AWS Utilization Guide Figure 2 - Image Alias Processing Scheme The images run in EPC have a bigger set of pre-installed software than their AWS ‘twins’. This especially applies to Windows images. EPAM Private Cloud team prepared a set of images based on the AMI supplied by Amazon. These images were carefully configured and tested and comprise a set of default images available for AWS usage. The table below gives the full list of the EO images and their availability in AWS and EPAM MSQ regions: OS Name AWS MSQ Windows Server 2012 R2 Standard Edition - + Windows Server 2012 Standard Edition + + Windows Server 2003 R2 Standard Edition 32-bit + + Windows Server 2003 R2 Standard Edition 64-bit + + Windows Server 2008 R2 Enterprise Edition - - Windows Server 2008 R2 Standard Edition + + Windows 7 32-bit Enterprise - + Windows 7 64-bit Enterprise - + Windows 8 32-bit Enterprise - + Windows 8 64-bit Enterprise - + Windows 8.1 32-bit Enterprise - - Windows 8.1 64-bit Enterprise - + Windows XP Professional 32-bit - + Linux CentOS 5.9 64-bit + + Linux CentOS 6.5 64-bit + + Linux CentOS 6.4 32-bit + + Debian GNU/Linux 7 64-bit + + Oracle Linux 6 64-bit + + + Linux Ubuntu 12.04 64-bit + + (except for Sydney Region + (except for Sydney Region Linux Ubuntu 14.04 64-bit - + Linux Ubuntu 10.04 32-bit Linux Ubuntu 10.04 64-bit 10 EPAM CONFIDENTIAL + + EPAM Cloud Orchestrator - AWS Utilization Guide Please note that in AWS, Windows instances are more expensive than Linux ones, because the license fee is included to the price. Apart from the default public images, you can also create you custom ones. To do it, modify an instance based on a public image, and use the or2-create-image (or2cim) command to create your own custom image. The custom images are available only for the project-region combination they were created in. EPAM Orchestration does not support transparent image migration between regions. To get your image migrated between the regions, please, submit a support request on EPAM Service Portal. To see the list of the images available for your project and region, use the or2-describeimages (or2dim) command: Figure 3 - AWS Images List The command output provides the following information: ID: The machine image alias. Description: The description of the machine image. Typically provides its operation system details. Group: The image security group. The default public images belong to the Enterprise group, and the custom project images comprise the Project group. State: Current state of the image. When running an instance, use the value of the ID column as the image identifier: or2run -p DEMO-PRO -r AWS-USEAST -i Debian7_64-bit -k my_key EPAM CONFIDENTIAL 11 EPAM Cloud Orchestrator - AWS Utilization Guide 4 Working with AWS with EO 4.1 Activating an AWS Region in Cloud To activate an AWS region in EPAM Private Cloud, simply leave a respective request at EPAM Service Portal: Figure 4 - Region Activation Request in Service Catalog The request is free of charge and will be fulfilled during three working days. Please remember that it should be approved by the Project Manager/Project Coordinator. When the region is activated, all the project members get a corresponding email notification. They can see the new region in the list of available regions (run or2dreg command): >or2dreg -p DEMOPRO The command gives a similar response: Figure 5 - Describing Available Regions If you can see the requested region (AWS-USEAST in the picture above) in the response, you can start utilizing AWS resources via Maestro CLI Tools. 4.2 Managing Instances in AWS The AWS instances are managed by the same commands as EPC ones. Before you run an instance, you have to decide not only on its image, but also on its shape. A Shape is a combination of CPU and RAM that your new instance will have. The table below lists the existing shapes: 12 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide Instance Type Micro Mini Small Medium Large XL 2XL 3XL #vCPU 1 1 1 1 2 8 2 4 Memory 512MB 1GB 1.7GB 3.75GB 7.5GB 7.5GB 15GB 15GB AWS shape mapping t1.micro t2.micro m1.small m1.medium m1.large c1.xlarge m2.xlarge m1.xlarge To see the list of shapes available for your project and region, use the or2-describe-shapes (or2dshape) command: or2dshape -p DEMOPRO -r AWS-USEAST Please remember that the selected shape influences the instance price. There are two ways to run an instance: using CLI; using the graphical wizard on Orchestration Management Console. 4.2.1 Running an Instance with Management Console You can run an instance using a Create Wizard available on Orchestration Management Console. To call the Wizard, click on the Create button on the Orchestration Management Console: Figure 6 - Create Wizard call EPAM CONFIDENTIAL 13 EPAM Cloud Orchestrator - AWS Utilization Guide Then, follow the step by step instruction by selecting the target project, region and other parameters for your new VM. Please remember, that to create a Linux instance, you will have to create and specify an SSH key. When an instance is launched, a corresponding message will appear on the Audit page and you will get an automatic email notification. 4.2.2 Running an Instance with CLI To run an instance in AWS, use the or2run command. To run a Windows instance, specify the target project, region and the alias of the image to be used: or2run -p DEMOPRO -r AWS-USEAST -i W2012Std A Linux image can be run only with an SSH key specified. Use the or2-describe-keypairs (or2dkey) command to see the list of the available keys. If you don’t have a key, create one with the or2-create-keypair (or2addkey) command: or2addkey -p DEMOPRO -k my_key When a key is created, use it to run a new Linux instance: or2run -p DEMOPRO -r AWS-USEAST -i CentOS5-template -k my_key As soon as an image is run, you will get a letter giving you the instance details and providing the list of the most common commands ready for copy-pasting to your CLI: Figure 7 - Instance Run Notification 14 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 4.3 Connecting to an Image 4.3.1 Connecting to a Linux VM To connect to a Linux VM, run the following Linux command: ssh user@<hostname> -i <path> Where <hostname> stands for your VM DNS Name and <path> stands for the full path to your key file. 4.3.2 Connecting to a Windows VM To connect to a Windows VM, perform the following steps: 1. Run the Remote Desktop Connection tool and specify the VM DNS name and ‘localhost\user’ user name: Figure 8 - Connecting to a Windows VM, Step 1 2. Click the Connect button. 3. Input the password in the security dialog: 4fsD321fDFf35g: Figure 9 - Connecting to a Windows VM, Step 2 EPAM CONFIDENTIAL 15 EPAM Cloud Orchestrator - AWS Utilization Guide For security purpose, we strongly recommend to change the default passwords after the initial login. NB: In order to access Windows instances via SSH, you first have to access them via RDP and configure SSH connection. In case any issues with connecting to your VMs arise, especially if you connect through EPAM VPN, please, contact the WFT IT Services Network group. 4.4 AWS Management Tools As with EPAM Private Cloud, Amazon provides users with both CLI and Web Management Console controls over their resources and data. EPAM Private Cloud supports both these options the following way: Cloud users can now get AWS Management Console Access with a single CLI command that returns a URL to connect. By following this URL, you will get access to all AWS services, except for IAM. To get the URL, run the or2-aws-management-console command: or2-aws-management-console -p project Follow the link to login to Amazon Management Console directly, no credentials input will be needed, your EPAM credentials will be used automatically. Figure 10 - AWS Console 16 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide All the bills for the AWS services usage will be included to your project’s bill at the end of the billing month. All the AWS instances run in Cloud, have a specified IAM role, which supports the following manipulations: o Volume management o Tags management o Elastic IP management o S3 Management. All these operations are performed via Amazon CLI that is to be set up on an AWS instance, run with Orchestrator. To set up the AWS CLI on a VM, perform the following steps: o o Login to the VM via SSH Run the following commands in the VM’s console: apt-get update apt-get install python-pip pip install awscli o Check the AWS CLI work with Amazon S3: aws s3 ls s3: This command displays the folders in the S3 root. The output can be similar to this one: 2014-11-21 16:17:05 storage-eu-west 2014-11-21 16:10:02 storage-us-east To see the content of the subfolders, use the command: aws s3 l3://storage-eu-west 4.5 Maestro CLI Specifics for AWS As it has been mentioned before, the commands to control EPC and AWS infrastructure were unified. You can use the Maestro CLI to manipulate AWS resources without any additional preparations. However, some EPC services and facilities are not available in AWS. This is caused by engineering specifics of cloud providers and by security reasons. The following EPAM Orchestration facilities are not available in AWS: Working with checkpoints; Files manipulation (do not confuse with stack templates manipulation); Auto Configuration based services (do not confuse with the Auto Configuration service itself). EPAM CONFIDENTIAL 17 EPAM Cloud Orchestrator - AWS Utilization Guide The full list of the EPC commands and the information of their availability in Amazon is given in Annex A. For more details on the CLI commands usage, please, see the Maestro CLI Reference Guide. The AWS operations that are not covered by Maestro CLI are available for EPC users through AWS CloudFormation stacks that are described in details in the following section. 18 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 5 EPAM Orchestration Services in AWS AWS provides its users with a big set of services that are constantly updated and enhanced. As it was already mentioned before, the services availability varies in different AWS regions. The full list of AWS services and the information on their availability in different regions are given on the Products and Services by Region page. In addition to the core set of Amazon services, the users who create a virtual infrastructure in AWS via EPAM Orchestration have access to a set of EPC services and abilities, not included to AWS. These are the following: Mobile Management Console; Advanced Cloud Monitoring and Audit; Advanced Auto Configuration Service; Load-balancing Service; Ambari Service. The details on these services are given further in this section. 5.1 Mobile Management Console EPAM Orchestration Mobile Management Console allows you to access your virtual infrastructure at any time and from any place. Use https://cloud.epam.com to connect to EPAM Private Cloud Management Console from any place where internet connection is possible. The handy mobile interface provides you with an easy access to Orchestration UI, where you can quickly collect all the EPC and AWS resources information without need to connect to EPAM network. You can make the access to the mobile console even faster by adding the link to the desktop of your mobile device. For example, on Apple iOS, you can use the “Add” button ( ) and select “Add to your home screen”. If you use Android, you can find this option in the mobile browser menu. This all puts your AWS-based infrastructure only a couple of clicks away from you and provides you with the constant access to your resources. EPAM CONFIDENTIAL Figure 11 - Mobile Management Console 19 EPAM Cloud Orchestrator - AWS Utilization Guide 5.2 Cloud Monitoring and Audit for AWS You can easily monitor your AWS resources performance, utilization and changes using EO Management Console (both desktop and mobile). The process and facilities are similar to those designed for monitoring EPAM Orchestration instances. So, you can use the Console to get the following information: The detailed infrastructure actions audit (Audit page): Figure 12 - Audit on an AWS region Here, all the events on your EPC and AWS based infrastructures are reflected in realtime mode. Any action related to an instance state change (run, start, stop, kill) is described here. For EPC-based resources, an audit message includes the name of the user who initiated the action. For AWS, the user name is unknown. The detailed information on each of the VMs (Management page). Here, on selecting an instance, you can see the basic instance information: related project and zone, instance ID, shape, state, IP’s and DNS’s, instance Owner: Figure 13 - Amazon VM details The detailed information on VM performance, retrieved due to integration with Amazon Cloud Watch (Monitoring page). By default, you can see the data on the default AWS-supported metrics: CPU utilization, Disk Read/Write information, Network Traffic and Status Check. The data on each of the metrics is represented as a graph, the same way as with EO VMs metrics: 20 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide Figure 14 - AWS Cloud Watch In EO Management Console You can use the Zoom tool to change the period to be viewed on the graph. Please, remember that the maximum number of points that can be represented on the graph is 1400, which means that the graph detailing can change on zooming. If you need to get more information on your AWS VM, you can customize the metrics, but it will bring additional costs. For more details on Amazon Cloud Watch, please, see the Amazon Cloud Watch page. To find more about Monitoring in EPAM Private Cloud, please, see the Cloud Analytics guide. 5.3 Auto Configuration Service Auto Configuration service allows Cloud users running instances with pre-installed sets of software, effectively eliminating the need to install and configure software manually. It was designed to work in same manner in all regions, including AWS. Auto Configuration Service is activated by default as soon as the project gets activated in Cloud. The service is based on the Chef tool and performs auto-configuration using a set of roles, each containing a description of the configuration and settings to be installed on the target VM. EPAM CONFIDENTIAL 21 EPAM Cloud Orchestrator - AWS Utilization Guide 5.3.1 Configuring your VM with Chef In order to set a chef role or several roles to an instance, you can use the or2-set-instanceproperties (or2setp) command with the following flags: -c/--chefattribute to specify the desired chef attribute to be used -h/--chefrole - the role to be set to the instance: or2setp -i i-48cced22 -h role1 -h role2 -c value1 -c "recipename1.attribute1=value2" -p epmc-2chef You can find the detailed information on the service on our Auto Configuration page. 5.3.2 Viewing Chef Client Information As soon the Auto Configuration Service successfully performs at least one operation on a VM, the Chef Server starts collecting the information on the VM’s Chef Client. This information can be found on Management page of Orchestration UI: Figure 15 - Chef tab on the Monitoring Page When the tab is unfolded, you can see the main details on the Chef client installed on the machine and the actions that are performed there: Figure 16 - Chef details view The toolbar at the top of the Chef tab allows group actions on Chef details groups: you can Expand or Collapse all the groups and subgroups or leave them expanded up to a selected level (2, 3, 4 or 5). 22 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 6 Environment Orchestration EPAM Orchestrator supports two engines for automation of infrastructure setup. These are EPAM Private Cloud Maestro Stacks and AWS CloudFormation. Both engines provide the users with the ability to automatically perform a set of pre-defined actions with a few CLI commands or UI Wizard actions. The sets are called stack templates and are stored in JSON or XML files. Despite having similar purpose, Maestro Stacks and AWS CloudFormation stacks have a number of differences. The choice of the engine to be used should base on your needs and the engine facilities: Maestro Stacks: Maestro Stacks can be used in both EPC and AWS Clouds, and can be also used in Open Stack. This means, that you can create a template that would fit any Cloud you chose and you won’t have to adjust it when you have to migrate or create a new infrastructure in a new Cloud. However, when it comes to AWS, Maestro Stacks support only those Amazon actions that are supported by Orchestration, the rest of the Amazon facilities is not accessible. To find the detailed guidelines on working with Maestro Stacks, please, see our Maestro Stacks guide. CloudFormation: CloudFormation stacks can be used only for AWS-based infrastructures and are not applicable to EPAM Private Cloud. On the other hand, CloudFormation stacks support all the existing related AWS functionality, irrespectively of whether it is covered by EPAM Orchestration or not. This provides a big range of additional AWS opportunities for EPC users. The AWS CloudFormation page gives full information on the subject. 6.1 AWS CloudFormation Template Introduction A CloudFormation Template is a simple JSON file that describes the AWS infrastructure to be created. It allows creating and using resources which are not available through Maestro CLI commands. A template contains several sections. The Resources section is the only obligatory one. The template text should start with an open brace symbol ({) and end with a closed brace (}). Below, you can see the basic anatomy of a template with all the possible sections: EPAM CONFIDENTIAL 23 EPAM Cloud Orchestrator - AWS Utilization Guide { "AWSTemplateFormatVersion" : "version date", "Description" : "JSON string", "Parameters" : { "Mappings" : { set of parameters set of mappings }, }, "Conditions" : { set of conditions }, "Resources" : { set of resources }, "Outputs" : { set of outputs } } Format Version: Specifies the AWS CloudFormation template version that the template conforms to. The template format version is not the same as the API or WSDL version. The template format version can change independently of the API and WSDL versions. Description: A text string that describes the template. This section must always follow the template format version section. Parameters: Specifies values that you can pass in to your template at runtime (when you create or update a stack). You can refer to parameters in the Resources and Outputs sections of the template. Mappings: A mapping of keys and associated values that you can use to specify conditional parameter values, similar to a lookup table. You can match a key to a corresponding value by using the Fn::FindInMap intrinsic function in the Resources and Outputs section. Conditions: Defines conditions that control whether certain resources are created or whether certain resource properties are assigned a value during stack creation or update. For example, you could conditionally create a resource that depends on whether the stack is for a production or test environment. Resources: Specifies the stack resources and their properties, such as an Amazon EC2 instance or an Amazon S3 bucket. You can refer to resources in the Resources and Outputs sections of the template. Outputs: Describes the values that are returned whenever you view your stack's properties. The order of some of the sections in a JSON file can be changed. However, as the values from one section can be referenced in another, it is recommended to keep to the logical ordering in the scheme above. For more information about templates and snippets please visit the CloudFormation Template Reference page. 6.2 Working with Stacks via Maestro-CLI Maestro CLI tool has a set of commands designed to manipulate stacks. The AWS-stack related commands are similar to those used for Maestro Stacks, but there are some differences in the set of the commands and their syntax: 24 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide EPC CLI for Maestro Stacks or2-upload-file (or2uf) or2-describe-files (or2df) EPC CLI for AWS Stacks Result or2-upload-template (or2upt) Uploads a template file to the server Describes the templates uploaded to the server Describes a specified template stored in the Orchestrator storage and saves it to a local file Runs a new stack based on the specified template Describes the existing stacks for the specified project and region Returns the events related to the specified stack Describes the resources created during the stack execution Deletes the specified stack - or2-describe-templates (or2dt) or2-read-template (or2rt) or2-run-maestro-stack (or2rmstack) or2-describe-maestro-stacks (or2dmstack) or2-run-aws-stack (or2rawss) or2-describe-aws-stacks (or2dawss) or2-describe-maestro-stackresources (or2dmsr) or2-describe-aws-stackevents (or2dawsse) or2-describe-aws-stackresources (or2dawssr) or2-delete-maestro-stack (or2delmstack) or2-delete-aws-stack (or2delawss) The AWS-related commands can be used only for AWS-type regions. Maestro Stack commands can be run for both EPC and AWS. 6.3 Working with Stacks via Create Wizard EPAM Orchestration provides EPAM Private Cloud users with the ability to run Maestro and AWS stacks using a graphical wizard, same used for running instances. The wizard is reached from the Main Page of the EO Management Console. After you run the wizard, follow the step by step instruction by selecting the target project, region, stack and stack parameters: Figure 17 - Cloud Formation stack parameters selection EPAM CONFIDENTIAL 25 EPAM Cloud Orchestrator - AWS Utilization Guide 7 Billing and Reporting 7.1 Pricing The pricing for AWS machines run in Cloud differs from that of EO machines. You can find the actual prices and AWS billing models on the Amazon EC2 Pricing page on AWS website. To estimate the price of the AWS-based infrastructure you plan to create, you can use the AWS Simple Monthly Calculator. The table below gives the approximate costs for full-time monthly utilization of similar resources in Amazon and EPAM clouds. The data are retrieved according to the following pre-conditions: AWS Prices are based on US-EAST-1 (Virginia) region price list EPC Prices are based on EPAM-MSQ (Minsk) region price list The prices are true for 09/18/2014 Both AWS and EPC prices include storage price estimations. For AWS, they are default storage volumes supposed by AWS for each of the shapes. For EPC, they are default storage volumes provisioned for the OS type SHAPE SMALL MEDIUM LARGE 3XL Ondemand 31,680 62,640 126,000 252,000 Linux 1-year 3-year LU LU 24,480 19,440 48,960 38,880 97,920 77,760 195,120 154,800 EPC Monthly 40,26 57,06 73,86 150,56 Ondemand 54,000 107,280 215,280 430,560 Windows 1-year 3-year LU LU 42,480 36,720 84,960 74,160 169,200 146,880 338,400 293,760 EPC Monthly 53,23 70,03 86,84 163,54 Please note that storage billing principles in AWS and EPC differ. In AWS, the user pays for the provisioned space irrespectively of the actual storage usage. In EPAM Private Cloud, only the used storage is billed. Each Linux VM has a default 20 GB storage and Windows has 60, and at the VM start, only the storage taken by the system is considered used and is charged. 7.2 Reporting EPAM Private Cloud provides billing capabilities and features, such as reporting on hourly basis, total reporting etc. You can use these features to get reports for your AWS-based projects. Use the or2report command to get the necessary report: or2-report -r AWS-USEAST -p DEMOPRO -m 8 -y 2014 -t total The response of such command will contain total costs for the specified month. As with any of the EO regions, you can also use the Reporting page of Orchestration Management Console to see the reports on the AWS resources utilization: 26 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide Figure 18 - An AWS report on Orchestration Management Console Please, note that EPAM Orchestrator retrieves AWS-regions costs from Amazon, where the data is updated four times a day. Therefore, the hourly reports for current day may contain a bit outdated data. NB: In case an Amazon billing item is not bound to any of the regions, its cost will be added to one of the AWS regions costs (typically, AWS-USEAST). EPAM Orchestration uses linked account technologies in working with AWS. Each project gets its own account and all the costs for each month are billed to this root account by the 5th day of the next month. EPAM Orchestration processes this data and passes to the accounting department, and the monthly billing report for a project contains costs for both EPAM Cloud and AWS-based resources. For more information on reporting, please read our EPAM Private Cloud Billing Guide. 7.3 Reporting Customization To sort your AWS costs and optimize the reports you get, you can set cost allocation tags to AWS items. An AWS cost allocation tag consists of two parts - a key and a value that you define on the tag creation. The diagram below illustrates the concept of tags in AWS. Figure 19 - AWS Tags EPAM CONFIDENTIAL 27 EPAM Cloud Orchestrator - AWS Utilization Guide There are two Amazon instances, each having two tags, called Cost Center and Stack. The tags have an associated value. Both tagged and untagged resources will be included to the monthly reports, but you can use tags to aggregate the reporting data by logical, functional or any other criteria. For more details on AWS cost allocation tags, please, see the original AWS Billing and Cost Management Guide. 28 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 8 Security As AWS-regions based resources are placed in a public cloud, there is a set of security limitations introduced to ensure your data safety: All the VMs created in EPC contain a pre-installed set of security check software. The AWS instances don’t have this software installed, that’s why their security cannot be controlled by EPAM Orchestration, and the owners of the AWS-based VMs are responsible for their safety. In case it is necessary to create infrastructure in AWS, it is recommended to do it in VPC (Amazon Virtual Private Cloud) in order to ensure higher network safety. Authorization to Linux instances can be performed only via SSH keys. Only the images prepared and verified by EPAM Private Cloud team are allowed for using in AWS. To see all the images available for your project in the specified region, please use the or2-describe-images CLI command. You can also create your custom images based on the VMs run from the allowed public images. By default, EPAM employees have no access to AWS Management Console. However, in case of a project need, it is possible to provide temporary access to the Console that would allow to perform basic AWS-related configuration. To get such an access, please, submit a respective request on EPAM Service Portal or contact EPAM Private Cloud Consulting Team. In case you need to create infrastructure in a protected network, specify this requirement in the request for the AWS region activation. The infrastructure will be created in Amazon Virtual Private Cloud (VPC), an isolated section of AWS where you have the complete control over the virtual networking environment. This includes the ability to specify your own IP address range, to configure network gateways and route tables and to create subnets if needed. For more details on this feature, please, see the Amazon VPC official page. EPAM CONFIDENTIAL 29 EPAM Cloud Orchestrator - AWS Utilization Guide 9 Self-Education 9.1 AWS Partner Network EPAM Systems has become a member of AWS Partner Network (APN). The partnership program opens a wide range of possibilities for its members who have a good opportunity to improve their AWS skills, to get new experience and the evidence of their professionalism. Amazon Partners have access to Amazon Partners Training program which provides a variety of Partner Training resources in the APN Portal. Online APN Partner Accreditation courses provided at no cost help you effectively articulate AWS solutions and leverage AWS best practices with your customers. AWS provides partner discounts on hands-on instructor-led classes and self-paced labs that help you deepen technical your skills for working with AWS products and solutions. To receive the trainings and other training resources, simply complete the AWS Partner Network registration (http://www.apn-portal.com/SelfRegisterPartner) by registering with EPAM domain. If any questions or issues related to APN Portal Registration occur, feel free to contact the APN Support team ([email protected] ). The program gives you possibility to earn AWS Certifications to gain credibility with your customers for your expertise with AWS. We are glad to announce the list of EPAMers who have already passed the certification: Apart from getting an AWS certificate, all the people who pass the training automatically get an AWS Certification badge on EPAM Heroes portal. For more details, please, see the Amazon Partner Network page. 9.2 AWS Trainings and Webinars EPAM Private Cloud team provides a set of webinars and self-education programs aimed to help our users to get acquainted with AWS and to develop the existing skills and expertise: 30 Amazon Web Services self-education training will introduce the basic information on Amazon Web Services, auto configuration and stacks in Amazon cloud EPAM experts regularly provide trainings and webinars to help all the interested EPAMers to upgrade their AWS skills. Please, visit the EPAM Training Portal to search for the upcoming trainings and webinars. EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide 10 Support and Consulting In case you have any issues with using AWS with EPAM Orchestration, please, feel free to leave a support request on EPAM Service Portal or contact EPAM Private Cloud Consulting Team. You can also find the answers on the most frequently asked questions on our FAQ page. EPAM CONFIDENTIAL 31 EPAM Cloud Orchestrator - AWS Utilization Guide Annex A. The Supported CLI Commands All the CLI Commands can be logically divided into groups by purpose. In this Annex, you can see the list of the Command groups, the main purpose of each group and the extent to which the commands of this group are supported for AWS-type regions. Please note that some of EPAM Private Cloud facilities and services are not supported for AWS. The commands dealing with these facilities are not listed below: Checkpoints Files manipulation (do not confuse with stack templates manipulation) Auto Configuration based services (do not confuse with the Auto Configuration service itself) Basic Commands The basic commands group covers the commands needed to start working with Cloud and to get all the information needed for further infrastructure creation. CLI command or2-check-version or2-describe-projects or2-describe-regions or2-describe-shapes or2-describe-vlans or2-get-access or2-get-info or2-update-cli or2-view-pool-state AWS + + + + + + + - Instances The commands in this group deal with different instance-related manipulations and instance operations scheduling. CLI command 32 AWS or2-describe-instances + or2-create-schedule + or2-delete-schedule + or2-describe-schedules + or2-move-instance-to-vlan - or2-move-to-project - or2-reboot-instances + or2-run-instances + or2-schedule-add-instances + EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide CLI command AWS or2-schedule-remove-instances + or2-start-instances + or2-stop-instances + or2-suspend-instances - or2-terminate-instances + Storage Volumes These commands deal with storage volumes that can be used to increase HDD capacity of the VMs running in cloud. CLI command AWS or2-attach-volume + or2-create-attach-volume + or2-delete-volume - or2-describe-volumes + or2-detach-volume + or2-resize-volume - Audit and Billing Audit and Billing commands deal with project resources costs reporting, prices and events audit. CLI command AWS or2-add-eo-account + or2-audit + or2-delete-eo-account + or2-delete-tag + or2-describe-eo-account + or2-describe-tag + or2-price - or2-report + or2-set-tag + or2-update-eo-account + EPAM CONFIDENTIAL 33 EPAM Cloud Orchestrator - AWS Utilization Guide Security and Connection This Section lists a set of commands used to connect to your VMs and to provide their security. CLI command AWS or2-console - or2-create-keypair + or2-delete-keypair + or2-describe-keypairs + Stacks The stack-related commands deal with AWS and Maestro Stacks that can be run in both EPAM and Amazon Clouds. CLI command AWS or2-delete-aws-stack + or2-describe-aws-stack-events + or2-describe-aws-stack-resources + or2-describe-aws-stacks + or2-describe-templates + or2-read-template + or2-run-aws-stack + or2-upload-template + or2-delete-maestro-stack + or2-describe-maestro-stack-resources + or2-describe-maestro-stacks + or2-run-maestro-stack + Images These commands are aimed to create and delete custom images as well as to get the information about the images available for the specified project in the specified region. CLI command 34 AWS or2-create-image + or2-delete-image + or2-describe-images + EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide Instance Properties The commands in this section are used to create and manipulate user-defined metadata, known as properties CLI command AWS or2-delete-instance-properties + or2-describe-instance-properties + or2-set-instance-properties + EPAM CONFIDENTIAL 35 EPAM Cloud Orchestrator - AWS Utilization Guide Table of Figures Figure 1 - AWS Regions Map ................................................................................................. 7 Figure 2 - Image Alias Processing Scheme .......................................................................... 10 Figure 3 - AWS Images List .................................................................................................. 11 Figure 4 - Region Activation Request in Service Catalog ..................................................... 12 Figure 5 - Describing Available Regions ............................................................................... 12 Figure 6 - Create Wizard call ................................................................................................ 13 Figure 7 - Instance Run Notification...................................................................................... 14 Figure 8 - Connecting to a Windows VM, Step 1 .................................................................. 15 Figure 9 - Connecting to a Windows VM, Step 2 .................................................................. 15 Figure 10 - Mobile Management Console ............................................................................. 19 Figure 11 - Audit on an AWS region ..................................................................................... 20 Figure 12 - Amazon VM details ............................................................................................. 20 Figure 13 - AWS Cloud Watch In EO Management Console ............................................... 21 Figure 14 - Chef tab on the Monitoring Page ........................................................................ 22 Figure 15 - Chef details view ................................................................................................ 22 Figure 16 - Cloud Formation stack parameters selection ..................................................... 25 Figure 17 - An AWS report on Orchestration Management Console ................................... 27 Figure 18 - AWS Tags........................................................................................................... 27 36 EPAM CONFIDENTIAL EPAM Cloud Orchestrator - AWS Utilization Guide Version History Version Date Summary 1.0 September 12, 2013 - First published 1.01 November 28, 2013 - Added Preface Documentation links are updated 1.1 March 22, 2014 - Added new AWS regions, updated cost and billing sections 1.2 May 22, 2014 - Added the APN Training info to the Overview 2.0 September 20, 2014 - Renamed to “AWS Utilization” - Totally restructured and updated 2.1 November 1, 2014 - Documentation reference updated 2.2 January 31, 2015 - Added Frankfurt region to the list and EPC - Reference names for AWS regions Removed reattach volume info - Added AWS Management Tools section EPAM CONFIDENTIAL 37 Global 41 University Drive Suite 202, Newtown (PA), 18940, USA Phone: Fax: +1-267-759-9000 +1-267-759-8989 EU Corvin Offices I. Futó st 4753 Budapest, H-1082, Hungary Phone: +36-1-327-7400 Fax: +36-1-577-2384 CIS 9th Radialnaya Street, Building 2 Moscow, 115404, Russia Phone: +7-495-730-6360 Fax: +7-495-730-6361 © 1993-2014 EPAM Systems. All Rights Reserved.
© Copyright 2024