Running connected components leaves dataframes cached in the spark runtime which can take up a substantial amount of memory when it is run many times. Specifically, the dfs in lastRoundPersistedDFs do not get cleaned up after convergence == true
Since the cached dataframe is not the one returned by the run method, the caller is unable to unpersist them without clearing all persisted dataframes.
Running connected components leaves dataframes cached in the spark runtime which can take up a substantial amount of memory when it is run many times. Specifically, the dfs in
lastRoundPersistedDFsdo not get cleaned up afterconvergence == trueSince the cached dataframe is not the one returned by the run method, the caller is unable to unpersist them without clearing all persisted dataframes.