Acing Apache Spark DataFrames Interview Questions Series using PySpark with Window functions
In this blog we will see a scenario based Dataframe question.
A window function performs a calculation across a set of table rows that are somehow related to the current row
Using these functions we will perform pysparkic way of Dataframe Transformations.
Scenario Question:
Write a pyspark code to identify the Highest & Lowest Salaried Employee in each Department
>>> With window function we will obtain the rank and of each minimum and maximun salaries of employees in each department and combine required dataframes into single dataframe for final output.
I may confuse you guys…to be simple and clear please checkout the requirement in the below pic
Let’s check the code
Note:- If anyone has a better approach to generalizing this code happy to embed it in my script.
That’s all for now…Happy Learning….
Please clap and Subscribe/follow my profile…Don’t forget to Comment…