Comments (2)
Hi there.
We have a feature-preview support for AWS EMR dynamic cluster creation and documentation for this is located here: https://hydrosphere.io/mist-docs/aws-emr.html
Be careful, this version is not released yet and experimental. I wouldn't recommend using it in production.
We neither have official documentation or guide about Mist and AWS EMR integration, but I have some practical experience about it. You can try to follow it, but I can't guarantee it will work in your case.
In practice mist instance is always deployed alongside with your EMR cluster so it has a default access to EMR's yarn master service and has a spark EMR distribution. To achieve the default behaviour you either need to modify spark-defaults or add it in your Context
configuration of a function. There are several spark configurations that you need to override:
spark.executor.instances = <max num instances for a function execution>
spark.master = yarn
spark.submit.deployMode = "cluster"
spark.executor.memory = <depends on your workload and instance types>
spark.executor.cores = <depends on your workload and instance types>
spark.yarn.executor.memoryOverhead = <depends on your workload and instance types>
It should assign any available elasticIP when deployed so you can access it directly by this IP.
By setting above options you will be able to:
- start driver application (mist worker) in EMR in
cluster
mode - function context configuration will account for yarn's specific configuration
- you don't need explicitly set spark.master address it will be derived from spark on emr distribution.
After mist deployment and Context
configuration is set properly you should be able to invoke function through mist ui or through any other source (Http, Mqtt, Kafka ).
from mist.
Hi Blvp,
Does mist have to be installed on the EMR master node? if yes, then Mist would be tightly coupled with a single spark cluster? what if I want to run multiple jobs using the same mist.
and I am unable to create a Mist setup by using a cloud formation template. It's throwing an error image that does not exist (ami-027583e616ca104df).
So, If we are going with the other AMI, How to start the mist service without spark home directory ?
and How to apply cloned hello. mist conf file through Mist-CLI without Nginx.
ex:- mist-cli --host $MistLogin:$MistPassword@$public-dns-name --port 80 apply -f conf
If you run the above command with port 80 (without Nginx) throwing some error. Please find the below error log.
Error: 404 Client Error: Not Found for url: http://$MistLogin:$MistPassword@$public-dns-name:80/v2/api/contexts:
<title>404 Not Found</title>Not Found
The requested URL was not found on this server.
Request body: {"maxJobs": 1, "streamingDuration": "1s", "downtime": "1200s", "runOptions": "", "maxConnFailures": 1, "precreated": false, "sparkConf": {"spark.submit.deployMode": "cluster", "spark.master": "spark://holy-excalibur:7077"}, "workerMode": "exclusive", "name": "root_standalone"}
updating Function root_hello-mist-scala
Error: Function root_hello-mist-scala is not valid. Please check: Context root_10_emr_ctx.conf should exists remotely.
Thanks in advance.
Best Regards,
Miyandada.P
from mist.
Related Issues (20)
- HTTP API - Validate artifact file extension HOT 3
- mistlibpy - Row from SqlContext doesn't have method `asDict`
- Python - mistpy: BadParameterException HOT 3
- Facing error Couldn't find JsEncoder instance for Map[String,Any] HOT 2
- Required support for database other than H2 HOT 1
- Job cancellation returns 400 Bad Request in async mode
- Delete context by Id is not working HOT 4
- ContextFrontend: Ask worker connection for context failed HOT 1
- ERROR FunctionInfoProvider failed HOT 3
- 2.12 support - docs
- Strange debug-like code at Json4sConversion HOT 1
- PySpark - starting from Spark 2.4.1 python jobs don't work
- Mist : Unsupported major.minor version 52.0 HOT 3
- Starting child for FunctionInfoProvider failed HOT 6
- Unable to download mist-cli
- Run parallel jobs on-prem dynamic spark clusters
- Support k8s helm HOT 1
- Is Mist deprecated/abandoned HOT 3
- Long running spark jobs when cancelled from mist ui continue to stay in cancelling state and eventually fail with something went wrong error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mist.