Okay And this is your spark shell So now from here you can start creating our release and then writing transformations and everything that you want Okay Now I will just see if I can show you this So if you type this comment by spark simply type I spark Okay Bash Dash Help Sorry Sorry Bye Spark too right Bye spot Do hyphen Hyphen Help Yeah So I want to show you these options, Okay Just have a look at this So I’m just saying that I want to start a spot Shelly Okay on I need some help in starting that by So what it say’s what are the options you can have So oneness Master you are so this We don’t have to worry as off now Okay If I scroll down I can show you Can you see this driver memory.
So a new start spark Shelly Okay Our driver will be creative, okay And then you can ask for how many executors you want right How much memory you want for the driver by one question on then Here you see What is this executor memory I told you right You can ask for how many executors you want and how much memory So by default I think it is one g ve district in here The fart is one g b, Okay But you can ask for executing her memory right on if I scroll down Ah see here Executor core How many processor cores you weren’t for an executor right And see here.
Driver court How many processor Cory want for the driver Number off executed default So if you’re starting the spa action like this interactive shell You will get two executors one driver each execute her has one GB ram before sitting But you think these arguments I can say how many executives I want and what is their memory and so on Now another important point Let me try this, Okay Just give me one moment I was just doing a pie spark too Okay And so what you’re looking at here is the history server You are And this is not really useful to us Because once you start a spark-shell on then you accept it It really plays here You can see here by sparks.
Shell It was launched by me 1.8 minutes I exit from you on Then you can see here But this is not really useful to us because this will come early in the history server That means want to exit from the spot, Okay so there is Ah really why That port number is not working Currently When you’re running the spark it’ll show there But I can show you something If so this session we I stopped okay I didn’t do anything I start decision And if I goto this application This is the spark you white So here you can see the jobs and there are no jobs on If I expand this can you see executed one addict that is executed to add it Because of the fault you are having to execute Er’s on here You can see executors Andi There was one driver on to execute her.
So there was one driver active on two executors Regard as for this right up And one more important point is that when you simply safe I spark too If you simply say pie spark to spark will start in the local movie That means you’re not talking to young or anything If you want to talk to young you have to say hyphen Hyphen Master Yeah this is the option you have to use So whenever you’re starting the spark-shell other ways If you start in the local more what will happen Driver Executor Everything will be in a single jar beom and also the resources will be located by your operating system nor by young in the local more yarn has no role in your local more on that is the word for learning.
If you simply want to learn say Spice Part two and then you can just start writing your court Whatever I really when you’re launching the spark-shell you’re to say Vice Part two on the New York was a hyphen master young meaning I want to resist it with the yarn on the lawn spot So what This request will go to yon and Young will locate containers to you Now let’s try to do one thing Let’s try to launch it with our own parameters right So what were they about Where the options we have by the spark to Nash Nash help So let’s look at what are the options Let’s save even more executed I warned up Three executed Probably.
Right So what is option num executed Hyphen hyphen numb Execute terrorist I will say three on hyphen Execute her course Course Let it be there I want memory more so that his memory hyphen executor memory Right before this one G v I want probably g b Something like that right So what I can do I can simply go to my shell by a spot Do I can say master is yon and I can’t say number off executors Probably four I don’t know And then what is that Executor Memory Why This is okay executor memory I need to G That was easy Tender uh by this fire does not support any obligation option White is saying that probably via asking for more topsy Let’s see whether we can get three executors OK so first we will try that Then we will see whether we can get more memory or something So right now I’m saying that a launch a spark-shell the master this young Sorry Um not ma’am Right Memory right Yeah the Wrong command Correct.
So now I ask for three executors and it started on We can see it here unless we exit So let me exit from here So it’ll go to the history server Okay a little fresh here This is what we started right Can you see Executed One order to other three added Can you see So three executors were given to me and you can actually go here Executors Can you see This is one driver These are executed Iris right You get executors on here You see the memory All right so How much memory did you ask huh I think one of the four beasts Selector right ones Eby be selected actually.
So for our dearly stories right now you’re getting worked 3 84 840.1 megabytes So how this rule actually works Like I told around roughly 50% off the memory will be taken on remaining it will give it to you will actually get around 600 being this but it display only this much because you are not loading any date as off Now it is simply running executor running You don’t have any anything inside this night No for they are dirty It’s That’s what You don’t have any date Isis Right.
When we load something we can actually see that memory will occupy how much amount off are Really You have a locket Oh so 3 84 Right now it had a locator but there is no daytime Said that zero by releasing that But when you actually low let’s say 500 every data show 500 be actually occupied in this because up to maximum 600 envy anything you will get because 40% anyway will be gone for your system cars and J bm like so up to 600 500 something 50 or something You will see here So right now you can see only less memory because we’re not actually doing anything with you So that is why it is simply displaying Ah you know this much on The I exited from here So what if we try to change this number of execute us four different hyphen executor memory right My Marie, Oh gee there is a threshold value I think if it is more than one GB let’s try this.
There’s a threshold value or does this Ah it stays An error occurred while calling on Java spark context required Executor memory is about maximum pra sure for this cluster So the cluster administrator has said that you can ask only maximum this much So what They say 2048 plus 3 84 m b is about So it is saying that Max to Max you can ask only 2000 for the TV off memory But here what will happen when you launch and execute er is that initially led Locator 3 84 and B.
Okay then we keep our GBS You know what is the access to be in located So they’re saying that the total memories about the cluster administrator what he has said you cannot launch so we can try 1.5 g or something like that I think 1.5 gigabytes should work I really let me check if that will work God exit I just wanted to check whether the cluster is running Fine Okay So how do you mention So see here Size must be specified as bite kilobytes Megabytes gigabytes There are bites like so I think you cannot say 1.5 g That will not work Right So how do you mention, Ah megabytes Mm Right on There Event timeline How many executors regard 1234 Right And if I go here it could have located less memory Yeah so now it is saying some 6 50 beyond Leeches not giving complete That’s fine because we don’t have any date as off now But you can clearly see that you’re getting four executors here right So there are four executioners.