Comments (27)
I too am trying to write a terraform provider and I'm not able to get the information out of my code. I've got the log level set to DEBUG, and I've got plenty of log.Println()
lines, but don't see any of them.
from terraform-plugin-sdk.
So no standard way to inspect stdout/stderr
from plugins?
from terraform-plugin-sdk.
Any update on this? Not having a straight forward way to collect provider logs makes debugging very difficult. Even a blog post on how to set up the proxy mentioned above for debugging would help.
from terraform-plugin-sdk.
@tombuildsstuff thanks for shedding some light on this. I am desperately trying to fix something in the AWS provider but haven't even found my logs in stdout yet :)
IMHO this should be documented in either the main "Debugging Terraform" page here https://www.terraform.io/docs/internals/debugging.html or at some new page specific to plugin development.
from terraform-plugin-sdk.
@tamsky
I'm aware of some folks using VSCode's debugger to do this - by placing a breakpoint in the Resource's Create/Read method - and then debugging an Acceptance Test, which will end up calling into the Create/Read methods of the Resource. I've not tried this personally - but since other folks have mentioned this approach works for them I felt it's worth noting.
Hi @tombuildsstuff Would you know any repositories or documentation on how to use vscode to debug terraform?
from terraform-plugin-sdk.
Hey @PSanetra! I took a look at the PR in question, and I think that's a question for the Core team to answer. Right now, the logging infrastructure in Terraform is just fiddly, and they want to overhaul it entirely, because it can be hard to reason about after years of accreting layers of patches. So even though your change is small in terms of code, it still poses risk in terms of unknown interactions. How big a risk and how willing they are to take it is a question that I can't answer for them, unfortunately. I imagine after 0.13 gets wrapped up and delivered, you'll get some more engagement there.
from terraform-plugin-sdk.
hey @d-apurva
Thanks for opening this issue
Unfortunately debugging a Terraform Provider is complicated due to the fact that Plugins are run in separate processes - that said there's two approaches to debugging Providers:
-
Using the environment variables mentioned above to set the log level and parsing them. This generally contains the HTTP logs - but also contains the output from Terraform Core
-
Using an HTTP Proxy such as Charles.app or Fiddler (or any other Proxy) to intercept the HTTP(S) traffic via
http_proxy=http://localhost:8888 https_proxy=http://localhost:8888 terraform plan
- which will show the HTTP/HTTPS traffic from the Provider to the API.
From there you should be able to determine the necessary changes to the Providers (sometimes outputting logs via log.Printf()
and the first approach can come in handy) and re-compile as outlined in the BUILDING.md
file which should allow you to debug further :)
Thanks!
from terraform-plugin-sdk.
Is there any way around this?
Not at this time that I'm aware of unfortunately, the TF_LOG
environment variable is passed between Terraform Core and the Providers at the moment.
To give an update on the original "how to debug a Terraform Provider" question: I'm aware of some folks using VSCode's debugger to do this - by placing a breakpoint in the Resource's Create/Read method - and then debugging an Acceptance Test, which will end up calling into the Create/Read methods of the Resource. I've not tried this personally - but since other folks have mentioned this approach works for them I felt it's worth noting.
from terraform-plugin-sdk.
Not at this time that I'm aware of unfortunately, the TF_LOG environment variable is passed between Terraform Core and the Providers at the moment.
Which approach might be preferred toward improving the current situation?
Create:
- a
TF_LOG_PROVIDER
env var, or - a
TF_LOG_ATOMIC
env var flag/feature?
TF_LOG_PROVIDER
could set the TF_LOG env var seen by all Providers.
Perhaps even as granular as: TF_LOG_PROVIDER_aws
.
TF_LOG_ATOMIC
could alter the logging library's behavior to buffer all log messages until a newline -- after which it would perform a synchronous write()
+ fflush()
.
from terraform-plugin-sdk.
@paddycarver did you have a look at the PR hashicorp/terraform#25086? I think it would greatly improve the logging situation without any changes in the terraform-plugin-sdk and with minimal changes in terraform-core.
from terraform-plugin-sdk.
Is there a way to get access to some resource type name inside terraform provider or something like "request ID"?
I do write logs inside provider, but the problem is that if I provision many things it is becoming almost impossible to read the log as there is too much going on in parallel. Log messages are missing something like correlation_id or whatever to filter it out and see trace for one operation rather than trying to understand this manully.
I could add them to logs if I had access to them.
from terraform-plugin-sdk.
I'm bit by this right now trying to debug the AWS provider when doing iterative IAM permissions development.
It's very common that you need to keep running terraform over and over again to figure out what permissions are needed, so getting detailed information from the provider (aws) is highly valuable.
from terraform-plugin-sdk.
Any update on this? I'm debugging my custom plugin and just realized the first meaningful line of output from my plugin started on line 17,277 of the LF_LOG... Scrolling through 17k+ output lines to debug is a horrible experience.
from terraform-plugin-sdk.
@tombuildsstuff I have a few questions about what you described:
Unfortunately debugging a Terraform Provider is complicated due to the fact that Plugins are run in separate processes - that said there's two approaches to debugging Providers:
- Using the environment variables mentioned above to set the log level and parsing them. This generally contains the HTTP logs - but also contains the output from Terraform Core
My TF_LOGs contain Terraform Core log output interspersed with Provider log output...
2019/07/24 03:09:09 [TRACE] [walkPlan] Entering eva2019-07-24T03:09:09.896-0700 [DEBUG] plugin.terraform-provider-aws_v2.20.0_x4: 2019/07/24 03:09:09 [DEBUG] Canonical definitions are not equal.
Notice the truncated TRACE output.
Is there any way around this?
It'd be really nice if one could simply disable the Terrafrom Core TF_LOG, but leave it enabled for all Providers.
from terraform-plugin-sdk.
I have submitted a PR, which would make it easier to filter for log lines of certain terraform providers: hashicorp/terraform#25086
from terraform-plugin-sdk.
👋 hey y'all, so I took a look into some of the concerns behind this.
First, good news: improved support for debugging providers using tools like delve is planned for the v2 release of the SDK, which we're working on pushing out the door now. You'll need Terraform 0.12.26 or higher to take advantage of it, and your provider will need to be using v2 of the SDK, but ti should allow debugging with delve in tests and with production binaries. We will publish documentation on how to do this when it lands.
Second, bad news: this doesn't fix the logging situation. As a println-based debugger myself, I recognise the current logging situation isn't ideal. I've had success in my own provider development with logging to a file, using unique strings in log messages, and searching for those strings, but that's obviously a... rough experience that we could improve on. Unfortunately, Terraform's logging situation could stand some reworking, and that work isn't planned in the immediate future. I had a fruitful conversation about a design direction for logging just a provider's output to a specific file, however, and that may be an incremental improvement we can deliver before Terraform's logging situation gets updated. Given that we're busy with getting v2 of the SDK out the door, however, and that the core team is busy getting 0.13 out the door, this incremental improvement is likely going to need to wait until the dust clears on those releases before it can be undertaken. And even then, it would require coordination between the core team and the SDK team, and we both have lengthy backlogs, so I can't promise a timeline, just that someone has looked at it and found a potential path forward.
from terraform-plugin-sdk.
First, good news: improved support for debugging providers using tools like delve is planned for the v2 release of the SDK, which we're working on pushing out the door now. You'll need Terraform 0.12.26 or higher to take advantage of it, and your provider will need to be using v2 of the SDK, but ti should allow debugging with delve in tests and with production binaries. We will publish documentation on how to do this when it lands.
Hey @paddycarver, was wondering if there was any update on possibly getting some documentation for working with Delve and Terraform.
from terraform-plugin-sdk.
To anyone reading this thread, please use TF_LOG=TRACE and not DEBUG, TRACE has additional internal information. Also to avoid issues of multithreaded log messages, you can set -parallelism=1 on the command line and TFE_PARALLELISM=1 in cloud/enterprise to single thread the run.
from terraform-plugin-sdk.
I've opened hashicorp/terraform-website#1584 to add some docs on the state of debugging as of 0.15.0. I'll make sure these documents stay updated as we continue to build out support for more advanced and powerful debugging capabilities.
from terraform-plugin-sdk.
Our docs are now live on the website: https://www.terraform.io/docs/extend/debugging.html
There's also #695 available now for comment. It's a proposal for the solution I think will close out this issue. While making Terraform providers more debuggable is something we'll always be interested in and working towards, I think that represents the end of a focused project on the subject. We're very interested in hearing your thoughts and feedback. :)
from terraform-plugin-sdk.
Is there any better way to debug terraform provider other than using TF_LOG=DEBUG.
Please list the steps required to reproduce the log which i did.set TF_LOG=DEBUG
set TF_TF_LOG_PATH=/tmp/log
terraform apply
observe TRACE level logs in the file /tmp/logPlease let me know if there is better way than this.
is that we use logs set path log.sh file and call it from main.tf using provisioners. is this the right approach or any different kind of approach for logs
from terraform-plugin-sdk.
First, good news: improved support for debugging providers using tools like delve is planned for the v2 release of the SDK, which we're working on pushing out the door now. You'll need Terraform 0.12.26 or higher to take advantage of it, and your provider will need to be using v2 of the SDK, but ti should allow debugging with delve in tests and with production binaries. We will publish documentation on how to do this when it lands.
Looks like it's not entirely true - debugging doesn't work with any version between 0.12.26 and 0.13.0, seems like it's broken in the 0.12.x branch
from terraform-plugin-sdk.
I'd love a separate issue open, with information on how to reproduce and what "doesn't work" means in this context/what provider code is set up. :)
from terraform-plugin-sdk.
I don't know if it is really worth a separate issue, since v0.12.x branch is long deprecated, and probably only weirdos like me still tinker with it :) But basically, if you follow the manual and set up the TF_REATTACH_PROVIDERS environment variable, Terraform seems to ignore it and still looks for the provider binary in ~/.terraform.d/ Doing the very same with v0.13.0 seems to do the thing just right
from terraform-plugin-sdk.
Hi folks 👋
There is information about how to setup recent versions of the SDK with additional logging and debuggers such as delve
available at https://www.terraform.io/plugin/sdkv2/debugging. If you have a question about this functionality, please start a new topic in the HashiCorp Discuss forums. If you have specific bug report or feature request, please open a new issue and we can take a fresh look. Thanks!
from terraform-plugin-sdk.
To debug old Terraform providers (pre Terraform Plugin SDK versions 2.0.0 if I'm not wrong), you can use the technique described in https://groups.google.com/g/delve-dev/c/RTtmwkJe5Jw/m/sD1RylCWGwAJ.
from terraform-plugin-sdk.
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
from terraform-plugin-sdk.
Related Issues (20)
- `diff.GetOk` return `nil` for the value of `TypeList` defined in `TypeSet` HOT 1
- Not Helpful Err When Running Tests For Provider HOT 4
- Unsupported Field Type for Resource
- TypeSet resource wanting to destroy and recreate everything when diff is found HOT 2
- losing precision when using maximum value for a 64-bit signed integer
- helper/schema: Provide Functionality to Reduce Resident Memory Usage from Schemas HOT 1
- Consider Method for Disabling Legacy SDK Data Consistency Warning Log Demotion HOT 7
- Updates for Protocol Version 5.4 HOT 1
- Value out of range on 32-bit architecture HOT 3
- Update Go Module to Go 1.20 Minimum HOT 1
- `Schema.Set` adds an empty value when attempting to Patch HOT 1
- `meta.SDKVersion` reporting incorrect version HOT 6
- Terraform CloudFormation Multi Region Stacksets HOT 3
- Is it possible to have a dependency between 2 attributes? HOT 2
- `schema.ResourceData.GetRaw<whatever>()` not populated during provider configuration HOT 3
- Consider Migrating from mitchellh/mapstructure to go-viper/mapstructure
- Low performance in deduplicate strings function HOT 4
- ForceNew doesn't trigger resource recreation when defined on TypeSet, which leads to Update error HOT 1
- helper/resource: Add Go Deprecation Messages That Point To terraform-plugin-testing
- Don't print `UnexpectedStateError.LastError` when it's nil
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from terraform-plugin-sdk.