Comments (9)
@bourque, do you mean anything outside of the conda env stuff we might need?
from jwql.
@SaraOgaz I'm referring to another VM machine with the specs that we would want in a production server. As you elude to, we will also need to consider having a jwqlprod
conda
environment, akin to our jwqldev
environment.
from jwql.
Here is what I'm thinking:
Memory
With some JWST files being on the order of 10 GB in size, and with the possibility of several automated scripts reading in files simultaneously, we should have at least ~100GB of RAM. ITSD gave us 128GB for our dev machine so we can get away with having at least that much. Most of the available INS servers have 378GB of RAM, so I think we should ask for that.
Cores
Conceivably we could have several automated scripts using multiprocessing running at once. The more cores the better. Some of the INS servers have 40 CPUs, others have 32, one has 24. I think we should ask for 40.
Accounts
We will need to have our admin account be a member of these servers.
UNIX groups
We will need a prod
and test
UNIX group, similar to our existing dev
UNIX group
from jwql.
A request of Test and Production servers has been sent out!
from jwql.
I've talked with Thomas Walker on this. He indicated that our requests are asking for too many resources from a single VM. He advised me that it is much easier on ITSD's end if we ask for many smaller VMs instead of one large one. We will have to design a better system.
from jwql.
Here is what I'm thinking:
Machines
We can't have one large machine, so we need to split up our needs into several VMs. One school of thought is to have 6 identical machines, one for each instrument plus one for cross-instrument or jwql
specific work. Another school of thought is to have a few larger VMs and a few smaller VMs and to do some bookkeeping on our end to figure out which processes should go to which VM.
Memory
There may be some automated scripts that will have to operate on several large JWST files at once, so we should have at least one VM with 64 GB RAM.
Cores
We should ask for at least one VM with 20 cores for larger processes that use multiprocessing
Accounts
Our admin account will need to exist on every VM.
UNIX groups
We will need a prod
and test
UNIX group, similar to our existing dev
UNIX group
Conclusion
Here is what I think we should request:
prod1
: 64 GB RAM, 20 Cores
prod2
: 64 GB RAM, 20 Cores
prod3
: 32 GB RAM, 16 Cores
prod4
: 32 GB RAM, 16 Cores
prod5
: 24 GB RAM, 8 Cores
prod6
: 24 GB RAM, 8 Cores
Totals:
6 VMs, 240 GB RAM, 88 Cores
from jwql.
I have sent in a request for the test server. I was advised by Thomas Walker to wait until this machine is configured to our liking before submitting requests for the production servers.
from jwql.
The test server has been built, and myself and other members are able to log in successfully. I will send in the request for the production servers.
from jwql.
The production servers have been built and seem to work as expected.
from jwql.
Related Issues (20)
- Update dashboard to look at new anomaly db tables HOT 2
- Pipeline tasks are not aware of working/outputs distinction HOT 1
- Dark monitor attempts to use nonexistent "data_dir" attribute
- Improve performance of query page HOT 2
- Display information on missing exposures/observations
- Create Django models for the tables in JWQLDB HOT 1
- Migrate EDB monitors to django
- Remove existing SQL Alchemy code from JWQL
- Edit a sample monitor to query and update via the django models.
- Update Bad Pixel Monitor to Django
- Update claw monitor to use django tables
- Migrate dark monitor to django
- Migrate grating monitors to django
- Migrate bias monitor to django
- Migrate TA monitors to django
- Update filesystem monitors to use django
- Update monitor_utils to use django models
- Update data_containers to use django
- Move github/readthedocs fence code to a single file, and use it by import
- Add historical data to the NIRCam Claw Monitor
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jwql.