fog / fog-azure-rm

Fog for Azure Resource Manager

Fog Azure Resource Manager

This document describes how to get started with Fog using Microsoft Azure as a cloud resource management services provider.

Pre-requisites

• Fog-azure-rm supports Ruby version 2.0.0 or later

Installation

Add this line to your application's Gemfile:

gem 'fog-azure-rm'

And then execute:

$ bundle

Or install it yourself as:

$ gem install fog-azure-rm

Usage

You can use this gem against the Microsoft Azure Resource Manager Services in the cloud. Of course, to use the Microsoft Azure Resource Manager Services in the cloud, you need to first create a Microsoft Azure account.

• Set 'LOCATION' constant based on the Azure cloud you are using in [config.rb] (https://github.com/fog/fog-azure-rm/blob/master/lib/fog/azurerm/config.rb) file. By default it will be 'eastus'.

Authentication

The next step to use this gem, is authentication and permissioning. It is important to get familiar with this concept. For a reference on setting up a service principal from the command line see Authenticating a service principal with Azure Resource Manager or Unattended Authentication. For a more robust explanation of authentication in Azure, see Developer’s guide to auth with Azure Resource Manager API.

After creating the service principal, you should have three pieces of information, a client id (GUID), client secret (string) and tenant id (GUID).

Compute

Fog-AzureRM for compute includes implementation of Virtual Machines and Availability Sets. Readme for the usage of Compute module.

Resources

Fog-AzureRM for resources includes implementation of Resource Groups. Readme for the usage of Resources module.

DNS

Fog-AzureRM for dns includes implementation of Record sets and Zones. Readme for the usage of DNS module.

Network

Fog-AzureRM for network includes implementation of Network Interfaces, Public IPs, Subnets and Virtual Networks. Readme for the usage of Network module.

Storage

Fog-AzureRM for storage includes implementation of Storage Accounts. Readme for the usage of Storage module.

Traffic Manager

Fog-AzureRM for Traffic Manager includes implementation of Traffic Manager Profile and Traffic Manager End-points. Readme for the usage of Traffic Manager module.

SQL

Fog-AzureRM for SQL includes implementation of SQL Server, SQL Databases and Data Warehouse. Readme for the usage of Azure SQL module.

Supported Services

Use following command for the complete list of services, Fog provides for Azure Resource Manager.

Fog::AzureRM.services

Documentation

https://www.rubydoc.info/github/fog/fog-azure-rm/master

Contributing

See CONTRIBUTING.md in this repository.

License

The gem is available as open source under the terms of the MIT License.

fog-azure-rm's Issues

Check Storage Account Name Availability

Check storage account name availability needs to be implemented correctly. Consumer of fog should not pass an SDK object to the request. Instead only name and type of storage account should be passed.

Fix NIC for updating attributes of load balancer

NIC isn't handling update in load balancer attributes.

Add requests for Exists functionality

Fog ARM currently does not support checking for existence of a resource, however this feature will be used in future. Implement "exists" request for all the Fog ARM resource.

create_or_update_async option for creating VM

I looked over the source code for creating the VM. It looks like the method create_or_update was used instead of create_or_update_async. I think if this method (create_virtual_machine) is used on an application, it would create a delay after hitting the submit button (correct me if I'm wrong) before the web application can be redirected to another page.

If there is an option to create_or_update_async, it would create the VM and then allow us to use the promise feature detailed at azure computer sdk doc and also allow us to handle any errors. Is there an option to create or update the VM asynchronously (e.g. create_virtual_machine_async)? if not, can that feature be added in the next release?

Fix check exists name issue

Currently we have implemented requests to add 'exists' functionality in Fog-ARM. These request files contains '?' as part of their names. Windows OS doesn't allow us to create a file name having special character in the file name. So Windows users will be unable to pull this latest change.

Please update request file names having special characters in their names.

Make all other Azure methods (deallocate, stop, delete, etc.) asyncronous

A while back I requested to allow creating a VM asynchronously. Can we add the sync feature to all the other methods such as deallocate, stop, delete, etc. as shown here https://github.com/fog/fog-azure-rm/blob/master/lib/fog/azurerm/models/compute/server.rb? It seems to me that the async methods are available on the Azure SDK.

Add Support for Resource Tagging During Resource Creation

Currently, resource tagging is available after a resource has been created. We need to add support for resource tagging while a resource is being created.

Check Storage Account Exists

Check exists of storage account should be changed to check the provisioning state of the object.

fog-azure gem vs. fog-azure-rm gem vs. azure-sdk-for-ruby gem

I'm trying to determine the right gem to help our team incorporate Azure into our devops app but I see several Azure gems out there and not entirely sure of the right choice.

After going over several gems, it looks like the fog-azure gem is deprecated. Is Fog-azure-rm gem the replacement for the fog-azure gem?

Microsoft seems to have their own Azure sdk gem. Issues 229 and 232 reference the Azure SDK so I'm not sure what the relationship of the Azure SDK and this gem. Is this gem still actively being developed to support Azure?

Fog ARM Unit Tests

Verify if all the unit tests are working by running the tests

• Independently
• Rake task

The unit tests were failing when running them via Rake task. Identify and fix the issue.

Fog Models for Azure SDK Enums

Create Fog Models for Azure SDK Enums to be used in the entire code-base instead of using strings

One question about 'compare_blob'

@zeeshan-arshad-confiz, which is using 'compare_blob'?
I am refining storage part so I am considering to remove it if it is not used.
Who wants to use this function can use directories.get('c1').files and directories.get('c2').files to compare blobs easily.

The API to Storage doesn't match other providers

We're trying to make it possible to use azure storage the same way we use AWS or GCP, which is documented at http://fog.io/storage/. Unfortunately the methods and objects in the azure storage models don't match the other Fog providers. Would it be possible to implement the standard directories and files concept? A proposed set of API calls would be like so:

azure_storage_service = Fog::Storage.new(
  :provider => 'AzureRM',
  :azure_storage_account_name => '<StorageAccountName>'           # Name of an Azure Storage Account
  :azure_storage_access_key => '<StorageAccountKey>'              # Key of an Azure Storage Account
)

azure_storage_service.directories  # should list a bunch of blob "containers"
azure_storage_service.directories.create('newcontainer') # should make a container
azure_storage_service.directories.get('mycontainer').files  # should list blobs
azure_storage_service.directories.get('mycontainer').files.create(
  key: 'some/path/file.txt',
  body: File.open('localfile.txt'),
  public: true|false
)  # should upload a blob
azure_storage_service.directories.get('mycontainer').files.all # returns all blobs
azure_storage_service.directories.get('mycontainer', prefix: 'some/path').files  # returns all blobs under some/path
azure_storage_service.directories.get('mycontainer').files.head('some/path/file.txt') # returns a File object but doesn't download it
azure_storage_service.directories.get('mycontainer').files.get('some/path/file.txt') # returns a File object and downloads its body
azure_storage_service.directories.get('mycontainer').files.head('some/path/file.txt').public_url(Time.now + 3600) # returns a signed URL good for anonymous download for the next hour
azure_storage_service.directories.get('mycontainer').files.get('some/path/file.txt').destroy # delete a blob

I know it's a lot of work to change the API, but this would allow you to be used with codebases that are used to using Rackspace, Openstack, GCP, AWS, etc. What you all think?

public ip is not assigned to nic

following this example to create a network interface with a public ip:

nic = azure_network_service.network_interfaces.create(
    name: '<Network Interface name>',
    resource_group: '<Resource Group name>',
    location: 'eastus',
    subnet_id: '/subscriptions/<Subscriptionid>/resourceGroups/<Resource Group name>/providers/Microsoft.Network/virtualNetworks/<Virtual Network name>/subnets/<Subnet name>',
    ip_configuration_name: '<Ip Configuration Name>',
    private_ip_allocation_method: 'Dynamic'

)

Even though we are passing in a per-configured "ip_configuration_name" the Network Interface in the Azure console shows that Public IP addresses are disabled, we have to manually enable and assign the Public IP resource.

Check the virtual machine status

Hello,
is it possible to check the status of virtual machine and determine if it is Powered on or Powered off using this gem?

Add Support to Attach Managed Data Disk to VM

Presently, Fog ARM does not allow to attach managed data disk to a VM.
Add support to attach managed DD to VM.

Move sql module to azure sdk

As sql server and sql databases are using rest client calls for api calling, and some of the responses have been changed for sql databases, due to which parsing is breaking for sql databses so i suggest we should move sql server and sql databases to azure sdk 0.8.0, as they have added the support of sql in SDK.

Create deployment with template body

Hello. AzureRM fog currently does not support for creating a deployment with template body. It is only possible with the template link. Is there any way to create a deployment with template body by using ARM template's relative file path? Or Are you planning to add this feature?
Thank you.

Add support for uploading/downloading block blobs

I saw you added the support for getting/setting blob metadata. I am going to work on the functions of uploading/downloading block blobs. Does it conflicts with the plan in your team?

Move Recovery Vault to Azure SDK

Currently, Recovery Vault is using REST client calls for API calls, because Azure SDK did not provide support for Recovery Vault. Now, support for Recovery Vault has been added as a separate gem azure_mgmt_recovery_services. Recovery Vault needs to be moved to use the SDK calls provided in the gem instead of REST client calls.

Fix AccessPolicyEntry Model

In the service key_vault, the model AccessPolicyEntry should not have a save method. Update the model and remove the said method.

Region based Fault Domain and Update Domain handling in Availability Set for Azure Global

Currently, Fault domain and update domain handling is hard-coded in availability set. We need to add support to FD and UD based on region in availability set.

Unable to assign Network Security Group for Network Interface

I am unable to assign a network security group to a network interface card.
Need to fix it so I can assign any security group to a network interface.

Impact analysis one exisiting vs newly provisioned VMs

Add missing requests in virtual network

Implement missing request in Load Balancer:

1. list_all

Should not upload_block_blob_from_file use to upload a disk.

@samawia-moin-confiz Should not upload_block_blob_from_file use to upload a disk. Please create a new request to upload the disk, for example, upload_page_blob_from_file.

https://github.com/fog/fog-azure-rm/blob/master/lib/fog/azurerm/requests/storage/create_disk.rb#L12

disk = upload_block_blob_from_file('vhds', "#{blob_name}.vhd", path.to_s, options)

Uanble to get storage_account_name by getting virtual machine

Currently get virtual machine in fog is not returning storage account name in vm object. Modify the code to return storage account name.

Public IP is mandatory

The Server creation requires a public IP configuration, there are a number of use cases where VM's should not have a public IP address.

Can this be made optional

Add support for getting/setting blob properties

I saw you added the support for getting/setting blob metadata. I am going to work on the functions of getting/setting blob properties. Does it conflicts with the plan in your team?

fog-core dependency

I am using this library in Foreman Azure RM
to add ARM as a compute resource for The Foreman. Foreman's current fog-core version is pegged at 1.42.0, and this library's fog-core version is pegged to ~> 1.43.0, which is somewhat problematic for development.
The changelog for fog-core to 1.43 does not seem to mention anything which would affect this library, and I would like to know if you would consider downgrading your fog-core dependency to ~> 1.42.0 or something similar to allow for compatibility with The Foreman

Add New Fog Members

Hi @geemus ,

Please include following github users as repo members.
bilal-naeem-confiz, waqar-haider-confiz

Regards,

Add Support for Managed Availability Set

In Fog ARM, while creating availability set we are ONLY creating un-managed availability set(default). When we add a managed OS disk VM to this availability set, we are getting following error

Addition of a VM with managed disks to non-managed Availability Set or addition of a VM with blob based disks to managed Availability Set is not supported. Please create an Availability Set with 'Aligned' SKU in order to add a VM with managed disks to it. Type: MsRestAzure::AzureOperationError

Add option to create managed availability set as well so that managed OS disk VMs can be added to it. Here is the relevant API link

Create Models for SDK Enums

Currently we are using SDK models for string values such as Azure::ARM::Network::Models::IPAllocationMethod::Dynamic. We should create Fog models for each of these SDK models.

Enabling blob features of Azure storage

Hey, since there is support for managing storage accounts, I am going to add support for the Azure storage blob features in this gem.

The storage client will be created by:

azure_storage_client = Fog::Storage.new({
  :provider => 'AzureRM',
  :azure_storage_account_name => YOUR_AZURE_STORAGE_ACCOUNT_NAME,
  :azure_storage_access_key => YOUR_AZURE_STORAGE_ACCESS_KEY
})

The APIs I am going to work on are:

azure_storage_client.create_container
azure_storage_client.delete_container
azure_storage_client.get_container_properties
azure_storage_client.create_block_blob
azure_storage_client.get_blob
azure_storage_client.delete_blob
azure_storage_client.get_blob_properties

If you don't have concern, I will start coding and submit PR later. Thanks.

+@devigned for awareness

Add New Repo Members

Hi @geemus,

I hope you are doing great, please add following github users as repo members.
dandagopi, rameshjanjyam, sleepyorion, azeem-sajid-confiz, muhammad-asad-confiz, maham-nazir-confiz

Thanks,

Managed OS Disk

Unless I'm missing something, I don't think it is currently possible to use this library to create a VM with a managed disk for the OS disk, is this a planned feature?

Add support for Azure Managed Disk

Add support for Azure Managed Disk in Fog-ARM.
You can find Azure Managed Disk SDK here
You can find Azure Managed Disk API here
We should do following in this ticket:

• All CRUD operations support for Managed Disk in Fog-ARM
• Write unit tests. Please make sure your code coverage is 100%.
• Write documentation for it
• Run static code analysis i.e Rubocop
• Update changelog file
• Write integration test for it

lib/fog/azurerm/models/compute/server.rb cannot list windows machines

line 41 of lib/fog/azurerm/models/compute/server.rb assumes that the machine is a linux box and breaks if retrieving details of a windows host.

Current code:
hash['disable_password_authentication'] = vm['properties']['osProfile']['linuxConfiguration']['disablePasswordAuthentication']

Proposed "Quick Fix"
if vm['properties']['osProfile']['linuxConfiguration']
hash['disable_password_authentication'] = vm['properties']['osProfile']['linuxConfiguration']['disablePasswordAuthentication']
else
hash['disable_password_authentication'] = false
end

Fix Documentation Issue

Documentation for compute has some typos.

Remove unnecessary required attributes from Application Gateway

Remove unnecessary required attributes from Application Gateway and add some necessary attributes in it.

• Add subnet_id attribute in AG frontend_ip_configuration
• Add id attribute in all objects(listener, ssl_certificate) under application gateway.
• Remove some of unnecessary required statements from fronend_ip_config validation in model/application_gateway/gateway.rb file.
• Remove some of unnecessary required statements from backend_http_settings validation in model/application_gateway/gateway.rb file.

Add support for generic security groups

Different Azure cloud environment support

@abbas-sheikh-confiz storage in v0.1.1 only supports AzureCloud. Later we will add support for AzureChinaCloud, AzureUSGovernment and AzureGermanCloud.
To keep consistent with azure-cli and azure-powershell, I think we should use same names for these environments.

1. By set environment in options to use different Azure cloud environment
2. The optional values of environment should be one of AzureCloud, AzureChinaCloud, AzureUSGovernment orAzureGermanCloud.

So below code in this link should be updated. How do you think?

# Set a CLOUD value from 'Azure', 'AzureChina', 'AzureGermanCloud' and 'AzureUSGovernment'
CLOUD = 'Azure'.freeze

azurerm has no storage service (Fog::Service::NotFound)

Currently if we use fog (full addons) or fog-aws together with fog-azure will cause this error.

Because fog (full version) and others fog addons use fog-json to standardize the JSON encode/decode. So Fog::JSON.parse will use fog-json and this module doesn't have parse method. It raises NameError: undefined method and will be rescued and re-raise with Fog::Service::NotFound, "#{provider} has no #{service_name.downcase} service".

The quick fix is replacing allJSON.parse to ::JSON.parse.

IMHO we should use fog-json with the method Fog::JSON.decode instead of ::JSON.parse since it's more standard in fog services (fog-aws, fog-azure, fog-google...)

Multiple NICs support for service.servers.create

If I did not understand the implementation correctly I apologize in advance. It seems that only one NIC is allowed to be configured when creating a VM. And a custom one in particular. Perhaps this ticket offers an opportunity further the this great implementation.

VHDs management in storage account

Are you planning to add support for VHDs management. Would be very handy to be able to list, delete vhds.

General Code pruning

Make Azure Fog as rest calls to reduce dependency on azure sdk

Add missing requests in load balancer

Implement missing request in Load Balancer:

1. list_all

Create Virtual Machine Additional Parameters

The request to create virtual machine should be updated to include caching type of os_disk as an optional parameter.

Update Traffic Manager Profile to receive Endpoints

Update Traffic Manager Profile create method to receive and create Endpoints with it.