Feature step7 integration #53
Conversation
kmilo9999
commented
Jan 3, 2024
kmilo9999
commented
- Add analysis_db/B06_correct_separate_var.py for step 7
- Cleaned code
- Formatted file
| import os, sys | ||
|
|
||
|
|
There was a problem hiding this comment.
Looks like this bit is in the wrong place – all imports should be under the docstring
We checked this with John in the dev meeting. The files should be compared against the |
There was a problem hiding this comment.
I say merge for now and refactor later.
If you like, consider moving all function definitions that happen in the middle of the script to the top just after the import statements. I also saw an import statement in the middle that might have slipped through the cracks. But I can do all of this later when implementing the cli!
| """ | ||
| This file open a ICEsat2 track applied filters and corections and returns smoothed photon heights on a regular grid in an .nc file. | ||
| This is python 3 | ||
| """ |
There was a problem hiding this comment.
It seems like the docstrings are not changing. @mochell, you might want to consider updating these.
| m slope of the fitted line | ||
| b intersect of the fitted line | ||
| """ | ||
| from scipy.ndimage.measurements import label |
There was a problem hiding this comment.
Maybe move import to the top? In this case it's repeated so it can be deleted.
| N_photons = np.zeros(GSUM.N_per_stancil.size) | ||
|
|
||
| counter = 0 | ||
| for k, I in Gdict.items(): |
There was a problem hiding this comment.
k may be unused in this loop.
| for k, I in Gdict.items(): | |
| for _, I in Gdict.items(): |
| try: | ||
| io.save_pandas_table( | ||
| B2_v2, "B06_" + ID_name + "_B06_corrected_resid", save_path | ||
| ) # all photos but heights adjusted and with distance coordinate | ||
| except: | ||
| os.remove(save_path + "B06_" + ID_name + "_B06_corrected_resid.h5") | ||
| io.save_pandas_table( | ||
| B2_v2, "B06_" + ID_name + "_B06_corrected_resid", save_path | ||
| ) # all photos but heights adjusted and with distance coordinate | ||
|
|
||
| try: | ||
| io.save_pandas_table( | ||
| B3_v2, "B06_" + ID_name + "_binned_resid", save_path | ||
| ) # regridding heights | ||
| except: | ||
| os.remove(save_path + "B06_" + ID_name + "_binned_resid.h5") | ||
| io.save_pandas_table( | ||
| B3_v2, "B06_" + ID_name + "_binned_resid", save_path | ||
| ) # regridding heights |
There was a problem hiding this comment.
This can be refactored to something like this but it's tricky to tell it works as it should. To be safe better wait for test suite.
| try: | |
| io.save_pandas_table( | |
| B2_v2, "B06_" + ID_name + "_B06_corrected_resid", save_path | |
| ) # all photos but heights adjusted and with distance coordinate | |
| except: | |
| os.remove(save_path + "B06_" + ID_name + "_B06_corrected_resid.h5") | |
| io.save_pandas_table( | |
| B2_v2, "B06_" + ID_name + "_B06_corrected_resid", save_path | |
| ) # all photos but heights adjusted and with distance coordinate | |
| try: | |
| io.save_pandas_table( | |
| B3_v2, "B06_" + ID_name + "_binned_resid", save_path | |
| ) # regridding heights | |
| except: | |
| os.remove(save_path + "B06_" + ID_name + "_binned_resid.h5") | |
| io.save_pandas_table( | |
| B3_v2, "B06_" + ID_name + "_binned_resid", save_path | |
| ) # regridding heights | |
| def save_table(data, tablename, save_path): | |
| try: | |
| io.save_pandas_table(data, tablename, save_path) | |
| except Exception as e: | |
| tabletoremove = save_path + tablename + ".h5" | |
| print(e, f"Failed to save table. Removing {tabletoremove} and re-trying..") | |
| os.remove(tabletoremove) | |
| io.save_pandas_table(data, tablename, save_path) | |
| B06_ID_name = "B06_" + ID_name | |
| table_names = [B06_ID_name + suffix for suffix in ["_B06_corrected_resid", "_binned_resid"]] | |
| data = [B2_v2, B3_v2] | |
| for tablename, data in zip(table_names, data): | |
| save_table(data, tablename, save_path) |
There was a problem hiding this comment.
I like this suggestion, pro-coding style. I dont see a reason why not to include it. I just did and it works as expected.
| def reconstruct_displacement(Gx_1, Gk_1, T3, k_thresh): | ||
| """ | ||
| reconstructs photon displacement heights for each stancil given the model parameters in Gk_1 | ||
| A low-pass frequeny filter can be applied using k-thresh | ||
|
|
||
| inputs: | ||
| Gk_1 model data per stencil from _gFT_k file with sin and cos coefficients | ||
| Gx_1 real data per stencil from _gFT_x file with mean photon heights and coordindate systems | ||
| T3 | ||
| k_thresh (None) threshold for low-pass filter | ||
|
|
||
| returns: | ||
| height_model reconstucted displements heights of the stancil | ||
| poly_offset fitted staight line to the residual between observations and model to account for low-pass variability | ||
| nan_mask mask where is observed data in | ||
| """ | ||
|
|
||
| dist_stencil = Gx_1.eta + Gx_1.x | ||
| dist_stencil_lims = dist_stencil[0].data, dist_stencil[-1].data | ||
|
|
||
| gFT_cos_coeff_sel = np.copy(Gk_1.gFT_cos_coeff) | ||
| gFT_sin_coeff_sel = np.copy(Gk_1.gFT_sin_coeff) | ||
|
|
||
| gFT_cos_coeff_sel = gFT_cos_coeff_sel * tanh_fitler(Gk_1.k, k_thresh, sigma_g=0.003) | ||
| gFT_sin_coeff_sel = gFT_sin_coeff_sel * tanh_fitler(Gk_1.k, k_thresh, sigma_g=0.003) | ||
|
|
||
| FT_int = gFT.generalized_Fourier(Gx_1.eta + Gx_1.x, None, Gk_1.k) | ||
| _ = FT_int.get_H() | ||
| FT_int.p_hat = np.concatenate( | ||
| [-gFT_sin_coeff_sel / Gk_1.k, gFT_cos_coeff_sel / Gk_1.k] | ||
| ) | ||
|
|
||
| dx = Gx.eta.diff("eta").mean().data | ||
| height_model = FT_int.model() / dx | ||
| dist_nanmask = np.isnan(Gx_1.y_data) | ||
| height_data = np.interp( | ||
| dist_stencil, T3_sel["dist"], T3_sel["heights_c_weighted_mean"] | ||
| ) | ||
| return height_model, np.nan, dist_nanmask |
There was a problem hiding this comment.
Maybe move function definitions to the top after the imports? To me it feels like they're in the way.
There was a problem hiding this comment.
We can do a refactor by moving all this functions to a separate file, so B06_correct_separate_var only contains the logic applied to step 7. This can also be done to all the other steps.
