Comments (4)
For purposes of a spot instance, the solution to this would be a little more complicated.
When a spot instance is terminated, it's the same as terminating a typical instance. This means that all the setup (mainly via ansible) won't be done. So it won't be as simple as just starting the service again.
However, I think this would still be valuable for just being able to restart an instance normally. We'll likely want to create a oneshot
systemd
service that will run the conjob container when the VM is starting. This should work fine as long as the container was not removed.
It should also work well for spot instances as long as they aren't terminated. I'm doing work now to have spot instances stopped rather than terminated. They should start back up again automatically when there is spot instance availability again.
from conjob.
I provisioned the alt server with a spot instance specifying to stop the instance on interruption. The instance started back up and the container was still there.
Creating a oneshot
service to start the container should work fine.
However, there's a problem that needs to be resolved first. When the instance starts back up, it seems like it's possible for it to have a new IP address. This happened, and DNS was no longer mapping to the correct IP. I see two ways this could be fixed:
- Allocate a static IP on creation
Have the DNS record use an alias to point to the instance instead of an IP
For the former, I think I'd want to use an Elastic IP. It seems that there's no charge for having a single IP associated with a EC2 instance. There's a small charge for when it's associated and the machine is down. This applies to us since the alt server will be stopped occasionally since it's a spot instance. However, I don't believe it's stopped for very long and I doubt this will be a problem. Still something to potentially keep an eye on.
For the latter, I'm not entire sure what this would look like, but I'm just aware of alias records and how they can point to a specific AWS resource.
I'm not sure which solution I would prefer. I may need to look into each to see what the implementation would look like. The elastic IP solution seems somewhat clean, although it's another piece to deal with and it could come with other complications or limits and I feel like I shouldn't have to rely on a specific IP and the DNS is really what should be handling this automatically. The alias solution has a hint of hackyness to it since aliases are an AWS specific thing.
After a little research it appears that you can't create an alias directly to an EC2 instance, so the Elastic IP will hopefully be a clean implementation.
from conjob.
So far the EIP solution was implemented and it seems like the instance restarts just fine.
Next up is to implement a service that runs to start conjob on boot. I think a "oneshot" will work.
NOTE: most of the work done so far has just been for the alt-env and needs to be carried over to the main service.
from conjob.
Fixed by 47c48f6.
from conjob.
Related Issues (20)
- Look into code linting/formatting and advanced code analysis tools
- /var/lib/sysbox doesn't seem to be cleaned up HOT 3
- Move DockerAdapter out of core.job into new core.adapter package
- Upgrade to Gradle 7 HOT 2
- Should we write dropwizard integration tests even though it doesn't seem to integrate with jqwik?
- Introduce jqwik property tests into acceptance tests
- Performance tests are inconsistent
- Investigate flakiness with JobRunnerTest's HOT 1
- Fix unit test JRE crash HOT 2
- Use PostBodyTask for admin config endpoint
- Consider disabling unattended upgrades HOT 2
- Report more informative error to user when same job is run concurrently with docker cache mounted
- Look for tools to look for out of date dependencies (and possibly update them)
- Is it problematic that docker-build-push docker cache is shared between projects?
- Remove docker references from alt-env HOT 2
- Look into testspace.com for publishing test reports
- Reduce error output in unit tests HOT 1
- Use of mutation testing in conjob - Help needed
- Fix integration test failure HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conjob.