Comments (6)
Hello,
This illegalInstruction
error is most likely due to the fact you compiled for Arm64
and execute on x64
(or the other way around)
If you compile on Apple Silicon machines (M1 or newer), be sure to create a Lamdda function that runs on on Arm64
architecture.
If you use SAM to deploy, there is a a one-line code change in your SAM template :
Architectures:
- arm64
If you created your function in the AWS console, there is a similar parameter you can set at function creation time.
from swift-aws-lambda-runtime.
Hi Sébastien,
Thanks for replying.
I don’t think that is it, the architecture do match, and the lambda is working fine most of the time.
However, I do have a race condition or logic error that makes it crash from time to time, and I can’t get any information or stack trace on the AWS console (other than the illegal instruction message).
Since I can’t reproduce locally, it is a pain to debug, and I’m trying to figure out if there is any way to get the stack trace of the crash.
Thanks for your time!
Max.
from swift-aws-lambda-runtime.
@DCVortexxx You're saying that the error is intermitent, and most of the time, it works. That rules out an Architecture mismatch.
You can try to set the environemnt variable LOG_LEVEL=trace in the Lambda environment. The runtime will produce more tracing, maybe the cause will be visible there.
Otherwise, we will need to modify the error handling to produce more verbose output in case of a runtime crash
from swift-aws-lambda-runtime.
Hello @sebsto, and thanks for your reply.
Sorry about the delay, I set the log level and to be honest, it slipped out of my mind for a couple of weeks.
Unfortunately, setting LOG_LEVEL=trace
in the environment does indeed increase logs from the lambda runtime SDK, however, it does not include the stack trace of why a function exited with an error.
It is mentioned in the lifecycle management section of the README that:
By default, the library also registers a Signal handler that traps INT and TERM, which are typical Signals used in modern deployment platforms to communicate shutdown request.
What I would expect (or like) is when such a signal is captured, the SDK would provide the developer with sufficient informations about what happened, to fix his own issue.
However, I'm not fully sure how signal trapping works and maybe what I'm asking is impossible.
If so, maybe we could have an environment variable to disable signal trapping, letting the program crash and access the stack trace as we would in any other program that crashes?
Once again, I'm not very experienced in that area, so feel free to correct me if I'm misunderstanding something or if what I'm asking is impossible.
On a side note, I've added (a lot of) logs in my own function as well to help me debug, and I managed to pinpoint the location of the crash.
Still unsure about what happens and how to fix it, but at least there's progress 🙃
from swift-aws-lambda-runtime.
I'm not sure it's possible to print a stacktrace when the binary is compiled in release mode. Binaries typically crash with EXC_BAD_ACCESS error and nothing more. Can you reproduce the crash when executing locally in DEBUG mode ?
Another debug strategy I often use is to capture the raw event (as string) passed to the runtime. Setting LOG_LEVEL=trace should allow you to capture the raw JSON. Then I verify if the JSON can be decoded by the corresponding Lambda Event struct.
Anyway, we're on the verge to rewrite the Lambda runtime to accommodate for Swift 6 strict concurrent and Service lifecycle. I suggest to not change anything related to signal handling in this version but rather take this feedback into consideration for v2. @fabianfett wdyt ?
from swift-aws-lambda-runtime.
I'm not sure it's possible to print a stacktrace when the binary is compiled in release mode. Binaries typically crash with EXC_BAD_ACCESS error and nothing more.
That makes sense indeed.
However, I don't think the final, uploaded binary actually crashes, since as the documentation states, the termination signal is trapped.
So I was thinking that maybe, in that case, the stack trace of where the signal happened would be available somewhere.
Once again, that's only an assumption, I'm definitely not an expert in that field.
Can you reproduce the crash when executing locally in DEBUG mode ?
No I did not manage to reproduce it in debug, but given my logs in production, it happens ~0.05% of the time.
And since my project does not have a lot of users currently, the data is not that easy to get.
Another debug strategy I often use is to capture the raw event (as string) passed to the runtime. Setting LOG_LEVEL=trace should allow you to capture the raw JSON. Then I verify if the JSON can be decoded by the corresponding Lambda Event struct.
Yeah, all good on that side, there's nothing distinctive about the event that could explain it.
With the same event content, 99.95% of the time, the lambda executes and terminates as expected, but 0.05% of the time, the lambda logs Runtime exited with error: signal: illegal instruction
.
I managed to narrow it down to a call to URLSession.dataTask(with:completionHandler:)
.
I have some logs on the line just before that call, and some logs just after it, and the second ones are not shown.
The completion is not called either, of course.
This call is wrapped in a withCheckedContinuation
in order to make use of the Swift concurrency, because it is not available on Linux.
I'm currently trying to simply to get rid of this async-await wrapper, and see if it improves things.
Anyway, we're on the verge to rewrite the Lambda runtime to accommodate for Swift 6 strict concurrent and Service lifecycle. I suggest to not change anything related to signal handling in this version but rather take this feedback into consideration for v2. @fabianfett wdyt ?
That definitely makes sense.
Thanks again for your time!
from swift-aws-lambda-runtime.
Related Issues (20)
- Replacement for Lambda.run? HOT 5
- How to fix newline in input causing Swift AWS lambda to fail in localhost testing HOT 2
- README.MD example about SimpleLambdaHandler HOT 3
- Archive plugin execution error with AWS SDK HOT 14
- Discussions repository Feature HOT 5
- Archive plugin to generate libraries (*.so) HOT 2
- [Request] Support Serverless Framework with a plugin? HOT 6
- [Idea] Could Lambda Layers Custom Runtime improve cold start performance? HOT 3
- [Feature Request] Somehow don't touch Package.resolved during lambda package archive to work better with VSCode. HOT 1
- [BUG] Can't compile code with NSFileCoordinator (cannot find in scope) HOT 3
- CI builds fails with Swift 5.9 and Swift nightly HOT 4
- [Question] Can DocC remove license headers from source code file HOT 4
- Add support for SPM Resources HOT 1
- print statements do not work despite context.logger.info(stringLiteral:) working HOT 1
- Debugging locally on a different invocation endpoint HOT 3
- Event JSON HOT 4
- Compile error on Windows HOT 1
- Doesn't compile with the Linux Static SDK (aka musl)
- [plugin] add option to compile with the Static Linux SDK instead of docker HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swift-aws-lambda-runtime.