The native progress bar is enabled by default when creating a SparkSession. The following is an example of the native progress bar, which displays information such as the current stage execution progress, number of tasks, and amount of scanned data:

The following is an example of the enhanced progress bar, which, in addition to the native progress bar features, also displays operation ID, stage details, task execution progress, input data volume, and execution duration:

Enhanced progress bar installation and configuration:

1. Install dependencies

   python -m pip install ipython ipywidgets
   

2. Enable the advanced progress bar by using .useNotebookProgress()

   spark = (
       EMRSparkConnectSession.builder
       .region(REGION)
       .credentials(StaticCredentials(access_key, secret_key))
       .serverId(SERVER_ID)
       .useNotebookProgress()
       .create()
   )