aws-samples / aws-data-foundations-framework

The Data Foundations Framework accelerates deployment of a modern data platform using AWS native services and best practices.

Data Foundation Framework Accelerator

The Data Foundation Framework Accelerator uses AWS CDK to deploy the architecture. It consists of a secure data lake built on S3, augmented with security through KMS and IAM, and fine-grained data governance through Glue and Lake Formation. Once deployed, this data foundation allows a functional data system to be built on top of it, with the flexibility to customize the system to the customer’s needs.

Description

The Data Foundation asset builds the foundational components of a data system, following published AWS best practices.

• A customer-managed key (CMK) is created in KMS to encrypt the data lake buckets.
• In S3, three data lake buckets are created, with server access logs written to a fourth bucket. Furthermore, a fifth bucket is created for use with Athena.
• IAM Groups, Users, and Roles are created with least privilege access to the S3 data lake buckets, CMK, and other essential services such as Lake Formation and Glue.
• In Glue, three Databases are created in the Glue Data Catalog, each pointing to their respective data lake buckets in S3. Crawlers are also created to crawl each of the data lake buckets and update the corresponding Database in Glue Data Catalog.
• Lastly, Lake Formation is pre-configured to register the S3 data lake locations, create and associate tags to each Database, grant least privilege access permissions to each IAM User and Role, and also assign Lake Formation Admins.

Deployment Guide

1. Pre-configure Lake Formation

   1. From the AWS Console, go to Lake Formation. If this is your first time using Lake Formation, select "Add myself" > Get Started, in order to add your current admin user as a Lake Formation admin.
   2. From the left sidebar, go to Administration > Data Catalog Settings > Uncheck the two boxes under "Default permissions for newly created databases and tables" > Save
   3. From the left sidebar, go to Administration > Administrative Roles and Tasks > Under "Database creators", select "IAMAllowedPrincipals" > Click "Revoke" on the top right > Revoke

2. Update app.py

   1. On line 8, update the account number and region parameters

3. Deploy the CDK app

Contributing

See CONTRIBUTING for more information.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the MIT-0 License.

aws-data-foundations-framework's Issues

Deployment Error: Circular dependency between resources

Hi there,

during the cdk deploy step, I get an error about Circular dependency between resources. Here is the log:

❯ cdk deploy

✨  Synthesis time: 2.89s

This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).

Do you wish to deploy these changes (y/n)? y
WorkloadNameComponentNameDataFoundationsStack: deploying... [1/1]
WorkloadNameComponentNameDataFoundationsStack: creating CloudFormation changeset...

 ❌  WorkloadNameComponentNameDataFoundationsStack failed: Error [ValidationError]: Circular dependency between resources: [MdaDataFoundationManagementStacklakeformationBucketTags29666430, MdaDataFoundationManagementStacklfDataAdminCreateDbPerms101D1CA7, MdaDataFoundationManagementStacklfStageBucketTagA7E79F86, MdaDataFoundationManagementStackdataEngineerTbacTablePermsB840947B, MdaDataFoundationManagementStackdataEngineerTbacAnalyticsTablePerms5E969924, MdaDataFoundationManagementStackglueCrawlerRawPermissions1C1EA114, MdaDataFoundationStorageStackdatalakeStageBucket0DAE9A68, MdaDataFoundationManagementStackdataAdminStagePermissions813325D7, MdaDataFoundationManagementStackdataEngineerStagePermissions562B888C, MdaDataFoundationManagementStacklakeformationRaw47D4A111, MdaDataFoundationManagementStackdataAdminAnalyticsPermissions2F4A0DA0, MdaDataFoundationManagementStacklakeformationStage30323466, MdaDataFoundationCatalogStackcatalogAnalyticsDBBC7C4EAE, MdaDataFoundationManagementStackdataAnalystBucketTagPerms3217A296, MdaDataFoundationStorageStackdatalakeRawBucketPolicyE762664A, MdaDataFoundationCatalogStackstageBucketCrawler380D0FDE, MdaDataFoundationPermissionsStacklfDatalakeBucketsDataAnalystPolicy87BE6EE3, MdaDataFoundationManagementStackcrawlerRawDbPermsC180B7CE, MdaDataFoundationManagementStackcrawlerStageDbPermsD59EFBDD, MdaDataFoundationManagementStackdataEngineerTbacRawDbPerms374D1439, MdaDataFoundationManagementStacklfRawBucketTag2592BD22, MdaDataFoundationManagementStackdataAnalystTbacDbPermsF07D00DB, MdaDataFoundationPermissionsStacklfDatalakeBucketsReadWritePolicy31711E82, MdaDataFoundationStorageStackdatalakeAnalyticsBucketPolicy29315F35, MdaDataFoundationManagementStackglueCrawlerAnalyticsPermissions8A2148C5, MdaDataFoundationPermissionsStacklfDatalakeBucketsDataEngineerPolicy564CC9CB, MdaDataFoundationManagementStackdataAnalystTbacAnalyticsDbPermsA6EEB786, MdaDataFoundationManagementStackdataEngineerAnalyticsPermissionsBD8BDFA5, MdaDataFoundationManagementStackdataAdminRawPermissions9633C4C3, MdaDataFoundationStorageStackdatalakeLogsBucketPolicy0595C582, MdaDataFoundationManagementStackdataAdminTbacTablePerms987899EB, MdaDataFoundationManagementStackcrawlerAnalyticsDbPerms3BE47673, MdaDataFoundationCatalogStackDataCatalogEncryption9B700825, MdaDataFoundationManagementStackdataAnalystTbacTablePerms2F0485A2, MdaDataFoundationManagementStackdataAdminBucketTagPerms010386D7, MdaDataFoundationStorageStackdatalakeAnalyticsBucketDC3F41D7, MdaDataFoundationCatalogStackrawBucketCrawlerC8F9545F, MdaDataFoundationManagementStacklakeformationAnalyticsC7BF6412, MdaDataFoundationCatalogStackcatalogStageDB4641CD90, MdaDataFoundationManagementStackdataAdminTbacDbPerms275A4792, MdaDataFoundationCatalogStackcatalogRawDBA77C6A23, MdaDataFoundationStorageStackdatalakeRawBucket63CBCFDF, MdaDataFoundationStorageStackdatalakeStageBucketPolicy6062C4C9, MdaDataFoundationManagementStackdataEngineerTbacRawTablePerms67BB4A2B, MdaDataFoundationManagementStackdataAnalystAnalyticsPermissions4F8977BB, MdaDataFoundationCatalogStackanalyticsBucketCrawlerC6D8FD0F, MdaDataFoundationManagementStackglueCrawlerStagePermissions76A4A9CA, MdaDataFoundationManagementStackdataEngineerTbacDbPerms892833AE, MdaDataFoundationManagementStacklfAnalyticsBucketTag80C5D18F, MdaDataFoundationManagementStackdataEngineerBucketTagPermsB91FD49C, MdaDataFoundationCatalogStackdatalakeGlueSecurityConfiguration14663673]
    at Request.extractError (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:46692)
    at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:91452)
    at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:90900)
    at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:199296)
    at Request.transition (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:192848)
    at AcceptorStateMachine.runTo (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:157720)
    at /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:158050
    at Request.<anonymous> (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:193140)
    at Request.<anonymous> (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:199371)
    at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:91620) {
  code: 'ValidationError',
  time: 2024-05-27T12:31:36.220Z,
  requestId: '84d5fcaa-646f-42eb-9f88-5813464784e7',
  statusCode: 400,
  retryable: false,
  retryDelay: 281.94328857552466
}

 ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [MdaDataFoundationManagementStacklakeformationBucketTags29666430, MdaDataFoundationManagementStacklfDataAdminCreateDbPerms101D1CA7, MdaDataFoundationManagementStacklfStageBucketTagA7E79F86, MdaDataFoundationManagementStackdataEngineerTbacTablePermsB840947B, MdaDataFoundationManagementStackdataEngineerTbacAnalyticsTablePerms5E969924, MdaDataFoundationManagementStackglueCrawlerRawPermissions1C1EA114, MdaDataFoundationStorageStackdatalakeStageBucket0DAE9A68, MdaDataFoundationManagementStackdataAdminStagePermissions813325D7, MdaDataFoundationManagementStackdataEngineerStagePermissions562B888C, MdaDataFoundationManagementStacklakeformationRaw47D4A111, MdaDataFoundationManagementStackdataAdminAnalyticsPermissions2F4A0DA0, MdaDataFoundationManagementStacklakeformationStage30323466, MdaDataFoundationCatalogStackcatalogAnalyticsDBBC7C4EAE, MdaDataFoundationManagementStackdataAnalystBucketTagPerms3217A296, MdaDataFoundationStorageStackdatalakeRawBucketPolicyE762664A, MdaDataFoundationCatalogStackstageBucketCrawler380D0FDE, MdaDataFoundationPermissionsStacklfDatalakeBucketsDataAnalystPolicy87BE6EE3, MdaDataFoundationManagementStackcrawlerRawDbPermsC180B7CE, MdaDataFoundationManagementStackcrawlerStageDbPermsD59EFBDD, MdaDataFoundationManagementStackdataEngineerTbacRawDbPerms374D1439, MdaDataFoundationManagementStacklfRawBucketTag2592BD22, MdaDataFoundationManagementStackdataAnalystTbacDbPermsF07D00DB, MdaDataFoundationPermissionsStacklfDatalakeBucketsReadWritePolicy31711E82, MdaDataFoundationStorageStackdatalakeAnalyticsBucketPolicy29315F35, MdaDataFoundationManagementStackglueCrawlerAnalyticsPermissions8A2148C5, MdaDataFoundationPermissionsStacklfDatalakeBucketsDataEngineerPolicy564CC9CB, MdaDataFoundationManagementStackdataAnalystTbacAnalyticsDbPermsA6EEB786, MdaDataFoundationManagementStackdataEngineerAnalyticsPermissionsBD8BDFA5, MdaDataFoundationManagementStackdataAdminRawPermissions9633C4C3, MdaDataFoundationStorageStackdatalakeLogsBucketPolicy0595C582, MdaDataFoundationManagementStackdataAdminTbacTablePerms987899EB, MdaDataFoundationManagementStackcrawlerAnalyticsDbPerms3BE47673, MdaDataFoundationCatalogStackDataCatalogEncryption9B700825, MdaDataFoundationManagementStackdataAnalystTbacTablePerms2F0485A2, MdaDataFoundationManagementStackdataAdminBucketTagPerms010386D7, MdaDataFoundationStorageStackdatalakeAnalyticsBucketDC3F41D7, MdaDataFoundationCatalogStackrawBucketCrawlerC8F9545F, MdaDataFoundationManagementStacklakeformationAnalyticsC7BF6412, MdaDataFoundationCatalogStackcatalogStageDB4641CD90, MdaDataFoundationManagementStackdataAdminTbacDbPerms275A4792, MdaDataFoundationCatalogStackcatalogRawDBA77C6A23, MdaDataFoundationStorageStackdatalakeRawBucket63CBCFDF, MdaDataFoundationStorageStackdatalakeStageBucketPolicy6062C4C9, MdaDataFoundationManagementStackdataEngineerTbacRawTablePerms67BB4A2B, MdaDataFoundationManagementStackdataAnalystAnalyticsPermissions4F8977BB, MdaDataFoundationCatalogStackanalyticsBucketCrawlerC6D8FD0F, MdaDataFoundationManagementStackglueCrawlerStagePermissions76A4A9CA, MdaDataFoundationManagementStackdataEngineerTbacDbPerms892833AE, MdaDataFoundationManagementStacklfAnalyticsBucketTag80C5D18F, MdaDataFoundationManagementStackdataEngineerBucketTagPermsB91FD49C, MdaDataFoundationCatalogStackdatalakeGlueSecurityConfiguration14663673]
    at Request.extractError (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:46692)
    at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:91452)
    at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:90900)
    at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:199296)
    at Request.transition (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:192848)
    at AcceptorStateMachine.runTo (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:157720)
    at /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:158050
    at Request.<anonymous> (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:193140)
    at Request.<anonymous> (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:199371)
    at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:382:91620) {
  code: 'ValidationError',
  time: 2024-05-27T12:31:36.220Z,
  requestId: '84d5fcaa-646f-42eb-9f88-5813464784e7',
  statusCode: 400,
  retryable: false,
  retryDelay: 281.94328857552466
}

Circular dependency between resources: [MdaDataFoundationManagementStacklakeformationBucketTags29666430, MdaDataFoundationManagementStacklfDataAdminCreateDbPerms101D1CA7, MdaDataFoundationManagementStacklfStageBucketTagA7E79F86, MdaDataFoundationManagementStackdataEngineerTbacTablePermsB840947B, MdaDataFoundationManagementStackdataEngineerTbacAnalyticsTablePerms5E969924, MdaDataFoundationManagementStackglueCrawlerRawPermissions1C1EA114, MdaDataFoundationStorageStackdatalakeStageBucket0DAE9A68, MdaDataFoundationManagementStackdataAdminStagePermissions813325D7, MdaDataFoundationManagementStackdataEngineerStagePermissions562B888C, MdaDataFoundationManagementStacklakeformationRaw47D4A111, MdaDataFoundationManagementStackdataAdminAnalyticsPermissions2F4A0DA0, MdaDataFoundationManagementStacklakeformationStage30323466, MdaDataFoundationCatalogStackcatalogAnalyticsDBBC7C4EAE, MdaDataFoundationManagementStackdataAnalystBucketTagPerms3217A296, MdaDataFoundationStorageStackdatalakeRawBucketPolicyE762664A, MdaDataFoundationCatalogStackstageBucketCrawler380D0FDE, MdaDataFoundationPermissionsStacklfDatalakeBucketsDataAnalystPolicy87BE6EE3, MdaDataFoundationManagementStackcrawlerRawDbPermsC180B7CE, MdaDataFoundationManagementStackcrawlerStageDbPermsD59EFBDD, MdaDataFoundationManagementStackdataEngineerTbacRawDbPerms374D1439, MdaDataFoundationManagementStacklfRawBucketTag2592BD22, MdaDataFoundationManagementStackdataAnalystTbacDbPermsF07D00DB, MdaDataFoundationPermissionsStacklfDatalakeBucketsReadWritePolicy31711E82, MdaDataFoundationStorageStackdatalakeAnalyticsBucketPolicy29315F35, MdaDataFoundationManagementStackglueCrawlerAnalyticsPermissions8A2148C5, MdaDataFoundationPermissionsStacklfDatalakeBucketsDataEngineerPolicy564CC9CB, MdaDataFoundationManagementStackdataAnalystTbacAnalyticsDbPermsA6EEB786, MdaDataFoundationManagementStackdataEngineerAnalyticsPermissionsBD8BDFA5, MdaDataFoundationManagementStackdataAdminRawPermissions9633C4C3, MdaDataFoundationStorageStackdatalakeLogsBucketPolicy0595C582, MdaDataFoundationManagementStackdataAdminTbacTablePerms987899EB, MdaDataFoundationManagementStackcrawlerAnalyticsDbPerms3BE47673, MdaDataFoundationCatalogStackDataCatalogEncryption9B700825, MdaDataFoundationManagementStackdataAnalystTbacTablePerms2F0485A2, MdaDataFoundationManagementStackdataAdminBucketTagPerms010386D7, MdaDataFoundationStorageStackdatalakeAnalyticsBucketDC3F41D7, MdaDataFoundationCatalogStackrawBucketCrawlerC8F9545F, MdaDataFoundationManagementStacklakeformationAnalyticsC7BF6412, MdaDataFoundationCatalogStackcatalogStageDB4641CD90, MdaDataFoundationManagementStackdataAdminTbacDbPerms275A4792, MdaDataFoundationCatalogStackcatalogRawDBA77C6A23, MdaDataFoundationStorageStackdatalakeRawBucket63CBCFDF, MdaDataFoundationStorageStackdatalakeStageBucketPolicy6062C4C9, MdaDataFoundationManagementStackdataEngineerTbacRawTablePerms67BB4A2B, MdaDataFoundationManagementStackdataAnalystAnalyticsPermissions4F8977BB, MdaDataFoundationCatalogStackanalyticsBucketCrawlerC6D8FD0F, MdaDataFoundationManagementStackglueCrawlerStagePermissions76A4A9CA, MdaDataFoundationManagementStackdataEngineerTbacDbPerms892833AE, MdaDataFoundationManagementStacklfAnalyticsBucketTag80C5D18F, MdaDataFoundationManagementStackdataEngineerBucketTagPermsB91FD49C, MdaDataFoundationCatalogStackdatalakeGlueSecurityConfiguration14663673]
❯ cdk --version
2.137.0 (build bb90b4c)

Thanks for fixing this!