Acing Apache Spark DataFrames Interview Questions Series using PySpark with Window functions

In this blog we will see a scenario based Dataframe question.

A window function performs a calculation across a set of table rows that are somehow related to the current row

Using these functions we will perform pysparkic way of Dataframe Transformations.

Scenario Question:

Write a pyspark code to identify the Highest & Lowest Salaried Employee in each Department

>>> With window function we will obtain the rank and of each minimum and maximun salaries of employees in each department and combine required dataframes into single dataframe for final output.

I may confuse you guys…to be simple and clear please checkout the requirement in the below pic

Let’s check the code

Note:- If anyone has a better approach to generalizing this code happy to embed it in my script.

That’s all for now…Happy Learning….

Please clap and Subscribe/follow my profile…Don’t forget to Comment…