XC3 is a cloud agnostic and risk free package offering powered by Cloud Custodian that provides resource inventory, tagging compliance, unused or invalid resources cleanup, account maintenance, cost control, backups, monitoring and alerting.
User stories
Develop a comprehensive R&D document outlining the Unit Testing V2 Plan, including a detailed analysis of various approaches that can be utilized and effectively implemented.
As a software developer, I want to resolve PR conflicts from a feature branch to the main branch, so that I can merge the feature branch into the main branch.
Acceptance Criteria
Identify the conflicting files and lines of code between the feature branch and the main branch.
Manually resolve the conflicts by modifying the conflicting files.
Ensure that the resolved code is reviewed, approved, and merged into the main branch.
Test the merged code to ensure it functions as expected.
As a contributor to the XC3 repository, I want to resolve AWS billing and IAM permission issues, so that I can effectively collaborate and contribute to the project without any obstacles related to AWS services.
Acceptance Criteria
Check with Xmops to resolve the permission issues
Identify and apply related permission set on AWS billing and IAM resources.
Apply the permission set on each individual Organization OUs
Test to check if Billing information and IAM roles, policies is accessible for the XC3 contributors
Load balancer shows 502 error when deploying production environment. this issue is caused by two different environment(prod and dev) and networking issued.
This issue is fixed by adding condition on both of the environment ()
As a Grafana administrator, I want to update the region information with the region name in Grafana, so that I can accurately represent the geographical data in my dashboards.
Functional Requirements:
Use the Grafana administrative interface to access the region information settings.
Locate the region field in the Grafana settings.
Update the region field with the appropriate region name.
Save the changes to ensure the updated region information is persisted.
Verify that the updated region name is displayed correctly in the Grafana dashboards.
As a DevOps Engineer, I want to implement the alerting workflow in Grafana Dashboard so the user can be notified whenever AWS Cost Metrics meets a predefined threshold.
Acceptance Criteria
What must be achieved to consider this story complete?
To consider this story complete, the following objectives must be achieved:
Implement an end-to-end alerting mechanism: Develop a comprehensive system that covers all stages of the alerting process, from data collection to generating alerts.
Define the threshold criteria: Determine the specific conditions or values that need to be met to trigger an alert. This could involve setting thresholds based on certain metrics or variables.
Generate alerts: Once the threshold is met, the system should promptly generate alerts. This could involve generating notifications, messages, or any other form of communication that informs relevant parties about the alert.
Send alerts through communication channels: Establish the necessary integration with relevant communication channels, such as email, Slack, or any other relevant channel. Ensure that the alert information is effectively transmitted to the appropriate recipients.
Test and validate the alerting mechanism: Conduct thorough testing to ensure the system functions as intended. Validate that alerts are generated correctly and delivered to the correct channels, and address any potential issues or bugs that may arise during the testing phase.
Document the implementation: Create detailed documentation outlining the steps taken to implement the alerting mechanism. This will serve as a reference for future maintenance, troubleshooting, and enhancements.
Which open-source libraries are you using to implement this user story? (Answer cannot be empty or N/A)
Update the libraries you will be using
Functional requirements
Implementing an end-to-end alerting mechanism that covers all stages of the alerting process.
Defining the specific threshold criteria or conditions that trigger the alerts.
Generating alerts promptly when the threshold is met.
Sending the alert information to relevant communication channels, such as email, SMS, or instant messaging services.
Ensuring the proper integration and functionality of the communication channels to transmit the alerts effectively.
Providing a means to configure and customize the alerting system to accommodate different requirements.
Handling and addressing potential errors or exceptions that may occur during the alerting process.
Ensuring the scalability and performance of the alerting mechanism to handle large volumes of data and alerts.
Inputs
Make a parameterized variable for the threshold
Outputs
Alerts
Testing
Perform unit testing of the alerting mechanism and underlying infrastructure
Documentation
A detailed README should be created in the folder of the feature created outlining the steps taken to implement the alerting mechanism. This will serve as a reference for future maintenance, troubleshooting, and enhancements.
As a DevOps engineer, I will work on a feature to break down cost on the project level where users can see all resources cost under a single project.
Acceptance Criteria
Develop and deploy the cost breakdown feature:
Implement the necessary functionality to break down the cost of resources within a project. This may involve creating new code, modifying existing code, or integrating with relevant APIs or services.
Ensure accurate cost calculations:
Verify that the cost breakdown accurately reflects the expenses incurred by each resource within the project. Conduct thorough testing and validation to ensure the calculations are correct and reliable.
User interface integration:
Integrate the cost breakdown feature seamlessly into the user interface, making it accessible and intuitive for users. Consider usability, design consistency, and any necessary user feedback or notifications.
Test and debug:
Conduct comprehensive testing to identify and resolve any issues or bugs that may arise during the implementation process. Ensure the feature functions as expected across different environments and scenarios.
Documentation and knowledge sharing:
Create clear and concise documentation that explains the functionality and usage of the cost breakdown feature. This documentation should serve as a resource for both users and fellow team members. Share knowledge about the feature with relevant stakeholders to facilitate its adoption and understanding.
Review and acceptance:
Collaborate with relevant stakeholders, such as product owners, managers, or other team members, to review the implemented feature. Seek their feedback, make any necessary adjustments, and obtain their acceptance before considering the story complete.
Which open-source libraries are you using to implement this user story? (Answer cannot be empty or N/A)
Functional requirements
User Interface:
Develop a user interface component that allows users to select a project and view the cost breakdown of all resources associated with that project.
Cost Calculation:
Implement the logic to accurately calculate the cost of each resource within the project based on relevant factors such as usage, pricing models, and any applicable discounts.
Data Retrieval:
Integrate with the appropriate data sources (e.g., cloud service provider APIs, billing systems) to retrieve the necessary resource and cost information for the selected project.
Filtering and Sorting:
Provide options to filter and sort the resource cost breakdown based on different criteria such as resource type, cost, or usage.
Inputs
Project Selection:
The user should be able to select a specific project for which they want to view the cost breakdown.
Outputs
Cost Breakdown Visualization:
The system should present the cost breakdown of resources associated with the selected project in a clear and easily understandable manner, such as through tables, charts, or graphs.
Detailed Resource Information:
Users should be able to access detailed information about each resource, including its name, type, usage, and associated cost.
Testing
Unit Testing:
Test individual components and functions responsible for cost calculation, data retrieval, and filtering/sorting to ensure they function as expected.
Integration Testing:
Verify the integration between the cost breakdown feature and other relevant systems or APIs, ensuring data is retrieved accurately and calculations are correct.
User Interface Testing:
Validate the user interface's responsiveness, accessibility, and usability, making sure it provides the desired functionality and delivers a satisfactory user experience.
End-to-End Testing:
Conduct comprehensive testing to ensure all components of the cost breakdown feature work together seamlessly and produce the expected results.
Metrics
Resource Cost Accuracy:
Measure the accuracy of the cost breakdown calculations by comparing them against the actual billing information.
Documentation
Feature Overview: Provide an overview of the cost breakdown feature, explaining its purpose, benefits, and how it aligns with user needs.
Installation and Setup: Provide step-by-step instructions for installing and configuring the necessary dependencies or components required to use the feature.
User Guide: Create a comprehensive user guide that explains how to access and utilize the cost breakdown feature, including instructions on selecting projects, interpreting the cost breakdown, and leveraging filtering/sorting options.
API Documentation (if applicable): If the cost breakdown feature interacts with external APIs, document the API endpoints, parameters
As an XCBG Cloud Engineer, I want to add a condition to remove a private instance from the XC3 project in Terraform and use public instance for Development purpose
Acceptance Criteria
The condition for removing a private instance should be configurable through variables or input parameters in the Terraform code.
Add condition in a way such that for dev environment, Load balancer and private subnet is removed for dev flag
The Terraform code should handle the removal of the private instance gracefully, ensuring any dependencies or associated resources are properly managed.
Test the code to verify if it working properly or not.
The code changes should be properly reviewed and approved by a peer or team lead.
As a devops engineer, I want to address an issue, that we were creating the keypair using the project variable, and we are getting its value using the namespace variable, which can cause conflicts if namespace and project variables have different values. so i update the congfig.sh and init.sh file, and also updated the readme file.
Acceptance Criteria
Which open-source libraries are you using to implement this user story? (Answer cannot be empty or N/A)
Functional requirements
Keypair should be created successfully and Ec2 should be accessible using the keypair
Inputs
Outputs
Testing
Keypair should be created successfully and Ec2 should be accessible using the keypair
As a developer, I want to resolve backend issue related to terraform. Since it is not managing the state file of terraform when a deployment is done using pipeline.
Add more as required
Acceptance Criteria
Determine the appropriate solution or fix for the identified issue.
Implement the necessary changes to resolve the issue in the backend.tf file.
Ensure that the S3 bucket backend configuration allows Terraform to store the state file securely and efficiently.
Test the updated configuration by initializing Terraform and validating that it connects to the S3 bucket backend without errors.
As a cloud engineer, I want to implement EC2 Instance Connect Endpoint (EIC Endpoint) in our internal AWS account so that private instances can be accessed without bastion hosts.
Acceptance Criteria
Ensure that you have the necessary IAM permissions
Identify the private EC2 instances that you want to implement EIC Endpoiint
Do the necessary configurations and test the system
What resources need to be removed
What resources need to be added
New architecture diagram of XC3
Functional requirements
Use AWS CLI to configure EC2 Instance Connect Endpoint for the AWS account.
Ensure that SSH access to EC2 instances is restricted to IAM users with the necessary permissions.
Validate that EC2 instances can be accessed securely using EC2 Instance Connect through CLI
Specifies the Region where the bucket will be created. If you are creating a bucket on the US East (N. Virginia) Region (us-east-1), you do not need to specify the location.
Steps to reproduce
Change region in the config.sh script to us-east-1
I want to create a project timeline Excel sheet that outlines the implementation plan for the XGrid Cloud Control solution for each quarter of 2023. This will include defining the workflows to be implemented in each quarter, gathering feedback during the last quarter, and conducting reviews and testing.
Acceptance Criteria
The project timeline Excel sheet should be created and include the implementation plan for each quarter of 2023.
The implementation plan should outline the workflows to be implemented in each quarter.
Feedback gathering should be scheduled for the last quarter.
Reviews and testing should be conducted during the last quarter.
As a DevOps Engineer, I want to automate integration testing through CodePipeline, so that I can ensure the reliability and compatibility of our application across different stages of the deployment pipeline.
Acceptance Criteria
The integration tests should cover critical paths and verify the interactions between different components or services.
First run the lambda integration test through codepipeline to check if it is successful.
Configure CodePipeline to trigger the integration tests after each deployment to test environment.
After successful test, the pipeline should destroy the test environment through pipeline
#1 As a cloud administrator, I want to create AWS Budgets in all regions (if not created), so that I can monitor the costs and usage across the AWS accounts.
#2 As a cloud administrator, I want to create SNS Topics in all regions (if not created), so that I can receive notifications and alerts related to AWS Budgets.
Acceptance Criteria
Verify if AWS Budgets have been created in all regions.
Create AWS Budgets in any regions where they are missing.
Check for existing SNS Topics in all regions.
Verify subscriptions associated with the provided email address.
Add the provided email addresses (including "Saman" and "Usman Zahid") to the necessary SNS Topics.
As a DevOps Engineer, I'm creating codepipeline to automate the deployment process of XC3 infrastructure. There are different stages in codepipeline to build and automate the whole process.
As a DevOps engineer, I want to test the infrastructure in the Team01 AWS account using an IAM user with restricted access, so that I can ensure that the infrastructure is secure and working as intended.
Acceptance Criteria
The IAM user with restricted access should be created and configured to only have the necessary permissions to test the infrastructure.
The infrastructure should be tested using the IAM user to ensure that it is working as intended.
Check if the deployment is done without any problem
As a DevOps Engineer, I want to link the develop branch of the new public repo of XC3 to the pipeline source stage and add a test stage before the production stage. I also want to include a manual approval stage before the production stage. Additionally, I need to create a terraform.test.tfvars file in the root directory of the repository, which will be used in the pipeline test stage as a -var-file when deploying Terraform to the test environment.
Acceptance Criteria
The develop branch of the new public repo of XC3 should be configured as the source for the pipeline.
The test stage should be added to the pipeline configuration before the production stage.
A manual approval stage should be included after the test stage and before the production stage.
The terraform.test.tfvars file should be created in the root directory of the repository.
The terraform.test.tfvars file should be used as a -var-file during Terraform deployment to the test environment.
As a DevOps Engineer, I want to perform unit testing of AWS resources created through using pytest, so that I can ensure the correctness and functionality of the infrastructure code.
Acceptance Criteria
Use pytest as the testing framework.
Use the AWS SDK for Python (Boto3) to interact with AWS services and retrieve information about the created resources.
The unit tests should cover all essential aspects of the AWS resources, such as verifying the existence, attributes, and configurations of the created resources.
Capture and log test results and any failures or errors encountered during testing.
Currently, there is a need to implement an auto-tagging feature for both EC2 (Elastic Compute Cloud) and EKS (Elastic Kubernetes Service) instances. This feature will utilize the email addresses of all stakeholders through SES (Simple Email Service) for effective and efficient tagging.
Acceptance Criteria
Which open-source libraries are you using to implement this user story? (Answer cannot be empty or N/A)
Functional requirements
Develop a solution to automatically tag EC2 and EKS instances.
Utilize the email addresses of all stakeholders for tagging purposes.
Integrate with SES to retrieve and manage the required email addresses.
Ensure that the auto-tagging feature functions correctly and accurately assigns appropriate tags to instances.
Test the implementation thoroughly to validate its effectiveness and reliability.
Provide clear documentation on how to configure and utilize the auto-tagging feature.
When deploying production environment and try to login with cognito, it shows an error "user sync failed" and "auth error". We were unable to login to grafana dashboard.
As a contributor to the XC3 project, bachelor students want to have access to a single AWS account specifically dedicated to XC3 development, so that they can collaborate effectively and utilize AWS services for project-related tasks.
Acceptance Criteria
Consists of architecture diagram
Should contain previous expense through cost calculator and expected cost
Steps of provisioning of accounts through aws control tower
Details of budget alerting
Permission set, where admin access will be for master account and contributors would have admin access to the AWS resources
Distribution of team based upon region and finally SOPs
As a DevOps engineer, I will work on a feature to break down the cost of the expensive service where users can see all resources cost under a single expensive service.
Acceptance Criteria
What must be achieved to consider this story complete?
Develop a feature that enables the breakdown of costs for the expensive service:
I will implement functionality that allows users to view a detailed breakdown of the expenses associated with the expensive service. This breakdown should provide information on individual resource costs within the service.
Enable users to access the cost breakdown under a single interface:
I will ensure that users can easily access the cost breakdown information through a unified interface. This interface should provide a clear and intuitive view of the expenses associated with the expensive service.
Ensure accuracy and reliability of the cost breakdown:
I will thoroughly test the feature to ensure accurate and reliable cost calculations. It is essential to validate that the breakdown accurately reflects the costs of individual resources within the expensive service.
Address performance considerations:
I will optimize the feature to handle large datasets efficiently. The cost breakdown should be able to handle a significant number of resources within the expensive service without any performance issues.
Document the feature and provide necessary instructions:
I will create a comprehensive document that outlines the functionality of the cost breakdown feature. This documentation should include clear instructions on how users can access and interpret the cost breakdown information.
Which open-source libraries are you using to implement this user story? (Answer cannot be empty or N/A)
Functional requirements
Cost breakdown feature:
Develop a feature that breaks down the cost of the expensive service, allowing users to see the individual resource costs within the service.
Single interface:
Provide a unified interface where users can access the cost breakdown information easily.
Accuracy and reliability:
Ensure that the cost breakdown accurately reflects the costs of individual resources within the expensive service.
Performance optimization:
Optimize the feature to handle large datasets efficiently without any performance issues.
Documentation:
Create comprehensive documentation explaining the functionality and instructions for using the cost breakdown feature.
Inputs
User requests for cost breakdown:
User input to request the cost breakdown for the expensive service.
Outputs
Cost breakdown information:
A detailed breakdown of the costs associated with individual resources within the expensive service.
Unified interface:
A user-friendly interface where users can access and visualize the cost breakdown information.
Testing
Unit testing:
Test individual components of the feature to verify their functionality.
Integration testing:
Validate the integration of the cost breakdown feature with the existing system and verify the accuracy of the cost calculations.
Performance testing:
Evaluate the feature's performance by simulating a significant number of resources and verifying that it can handle large datasets efficiently.
Metrics
Cost breakdown accuracy:
Measure the accuracy of the cost calculations by comparing the calculated costs against the actual costs of individual resources.
Documentation
Feature overview:
Explain the purpose and functionality of the cost breakdown feature.
User instructions:
Provide step-by-step instructions on how to access and utilize the cost breakdown information.
Configuration guide:
Detail any necessary configurations or setup required to enable the cost breakdown feature.
Troubleshooting guide:
Include troubleshooting tips and solutions for common issues that users may encounter.
API documentation (if applicable):
If the cost breakdown feature provides an API for programmatic access, document the API endpoints
As a DevOps engineer, my goal is to streamline the deployment process of the XC3 infrastructure by automating it through a script. This script will gather all the necessary inputs from an input.sh file, allowing the user to easily update and customize the deployment settings. By executing the automation script, the entire deployment process will be handled automatically, relieving the user from manually performing each step.
Acceptance Criteria
All commands should run properly and Infrastructure should be deployed.
Functional requirements
Infrastructure should be deployed properly
Inputs
The user will give input in the input.sh file in the xc3/ directory
Outputs
Dns of the load-balancer
Testing
Infrastructure should be deployed properly
Documentation
Update the ReadMe file according to automation script
As a cloud engineer, I want to implement EC2 Instance Connect Endpoint (EIC Endpoint) in our internal AWS account so that private instances can be accessed without bastion hosts.
Acceptance Criteria
Ensure that you have the necessary IAM permissions
Identify the private EC2 instances that you want to implement EIC Endpoiint
Do the necessary configurations and test the system
Update the steps in documentation form
Functional requirements
Use AWS CLI to configure EC2 Instance Connect Endpoint for the AWS account.
Ensure that SSH access to EC2 instances is restricted to IAM users with the necessary permissions.
Validate that EC2 instances can be accessed securely using EC2 Instance Connect through CLI
As a DevOps Engineer, I want to perform integration testing of AWS resources created using pytest, so that I can ensure the proper functionality and interaction of the resources in the AWS environment.
Acceptance Criteria
Open-source libraries: pytest, Boto3 (Python SDK for AWS)
Functional requirements
Where this test will run and how?
Ensure that the system under test is seamless and successful.
Ensure that the interactions and dependencies between AWS resources are tested thoroughly.
Verify that the cleanup and removal of AWS resources are executed correctly after integration tests are completed.
As a developer, I want to implement unit testing for my serverless application, so that I can ensure the correctness and reliability of the application's individual components.
Acceptance Criteria
Write unit tests for the individual components (functions, handlers, etc.) of the serverless application.
Ensure that the unit tests cover the critical functionality and edge cases of the components.
Execute the unit tests and verify that they pass successfully.
Include negative unit tests to validate error handling and edge cases.
As a cloud cost optimization analyst, I want to create a proposal document comparing Grafana Alert and AWS billing alerts, so that I can recommend the most suitable solution to notify users when costs exceed a certain threshold.
Acceptance Criteria
Identify and compare the features and capabilities of Grafana Alert and AWS billing alerts.
Provide a detailed comparison table highlighting the strengths and weaknesses of each solution.
Include recommendations based on the comparison and specify the preferred solution.
Document the steps required to set up the recommended solution.
Review and proofread the document for accuracy and clarity.