Skip to content

monkey patch services#2193

Merged
bsipocz merged 16 commits into
astropy:mainfrom
jaymedina:monkey-patch-services
Jan 26, 2022
Merged

monkey patch services#2193
bsipocz merged 16 commits into
astropy:mainfrom
jaymedina:monkey-patch-services

Conversation

@jaymedina

@jaymedina jaymedina commented Nov 2, 2021

Copy link
Copy Markdown
Contributor

To support our back-end unit testing that will involve patching the service variables currently hard-coded in several mast scripts, I'm requesting to have service call static class attributes that live outside of the methods, which allows the STScI developers to patch these as necessary for our unit tests.

Please note: This pull request is time-sensitive, and should take precedent over the other pull requests I have open for astroquery. I aim to have this ready and merged before the end of November.

Update: After a shift of the scope for this year, we've changed the timeline to have it ready by early December or beginning of January.

  • Observations
  • Catalogs
  • Cutouts

@keflavich keflavich added the mast label Nov 2, 2021
@pep8speaks

pep8speaks commented Nov 2, 2021

Copy link
Copy Markdown

Hello @jaymedina! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-01-26 17:23:50 UTC

@bsipocz bsipocz marked this pull request as draft November 2, 2021 16:56
@bsipocz

bsipocz commented Nov 2, 2021

Copy link
Copy Markdown
Member

@jaymedina - I've converted this to a draft PR, please change it back once it's out of the WIP state and ready for review.

Comment thread astroquery/mast/observations.py Outdated
Comment thread astroquery/mast/observations.py Outdated
@codecov

codecov Bot commented Nov 10, 2021

Copy link
Copy Markdown

Codecov Report

Merging #2193 (0f93106) into main (c933c3d) will increase coverage by 0.01%.
The diff coverage is 93.10%.

❗ Current head 0f93106 differs from pull request most recent head 45d7ab8. Consider uploading reports for the commit 45d7ab8 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2193      +/-   ##
==========================================
+ Coverage   62.69%   62.70%   +0.01%     
==========================================
  Files         131      131              
  Lines       16822    16828       +6     
==========================================
+ Hits        10546    10552       +6     
  Misses       6276     6276              
Impacted Files Coverage Δ
astroquery/mast/observations.py 76.41% <88.23%> (+0.52%) ⬆️
astroquery/mast/discovery_portal.py 67.57% <100.00%> (ø)
astroquery/mast/services.py 78.35% <100.00%> (+0.16%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c933c3d...45d7ab8. Read the comment docs.

@tomdonaldson tomdonaldson left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. I left a couple minor comments.

Another very minor suggestion is to update the comments on observations.py lines 176 and 330 to mention ObservationsClass.caom_filtered_position instead of Mast.Caom.Filtered.Position

Comment thread astroquery/mast/observations.py Outdated
Comment thread astroquery/mast/observations.py Outdated
@falkben

falkben commented Dec 7, 2021

Copy link
Copy Markdown
Contributor

Is there any thought to having this this accept environment variables as configuration overrides? If we allowed for environment variables it would make it much easier to reconfigure without the need to make code changes to people's scripts and potentially easier to test, as well.

@jaymedina

Copy link
Copy Markdown
Contributor Author

Is there any thought to having this this accept environment variables as configuration overrides? If we allowed for environment variables it would make it much easier to reconfigure without the need to make code changes to people's scripts and potentially easier to test, as well.

Some of these attributes are "built" off attributes that are inherited from other imports, so I'm not sure if that would make things cleaner. For example MAST_REQUEST_URL = conf.server + "/api/v0/invoke" under PortalAPI, as an example, uses conf.server taken from the conf import (config file). I also worry people might overwrite the environment variables by mistake, or there might be a typo somewhere that breaks the scripts. But maybe Tom or Brigitta have another take.

@jaymedina

Copy link
Copy Markdown
Contributor Author

Will need to patch the services.py URLs as well

@jaymedina

jaymedina commented Jan 6, 2022

Copy link
Copy Markdown
Contributor Author

I've addressed all previous threads and installed the final "hooks" that the STScI devs need for unit testing patchwork. I've been running my unit tests successfully off this branch, so I'm going to mark this as Ready for Review. There's some new code that's been modified under services.py so I'd appreciate it if that is looked over to make sure there are no typos/skips in logic.

Thanks for taking some time out to review this already @bsipocz @tomdonaldson @dr-rodriguez and @falkben , feel free to send back any further comments.

@jaymedina jaymedina marked this pull request as ready for review January 6, 2022 22:30
@jaymedina jaymedina changed the title [WIP] monkey patch services monkey patch services Jan 6, 2022
@jaymedina

Copy link
Copy Markdown
Contributor Author

With the OK from you @bsipocz I'd like to do a rebase and rerun the CI. I already created a backup branch just in case.

@bsipocz

bsipocz commented Jan 24, 2022

Copy link
Copy Markdown
Member

@jaymedina - please go ahead. I'll do a quick review afterwards.

@jaymedina jaymedina force-pushed the monkey-patch-services branch from 974a7db to 0f30b74 Compare January 24, 2022 20:59
@jaymedina

jaymedina commented Jan 24, 2022

Copy link
Copy Markdown
Contributor Author

The latest readthedocs build is failing with an error from another module called esa:

astroquery/docs/api/astroquery.esa.jwst.JwstClass.rst:23: WARNING: autosummary: failed to import JwstClass.login_gui
WARNING: autodoc: failed to import method 'JwstClass.login_gui' from module 'astroquery.esa.jwst'; the following exception was raised:

[...]

AttributeError: type object 'JwstClass' has no attribute 'login_gui'

I'm looking at the most recently merged PR and think it might be the source of the bug: #2260

I don't see any CI tests, just the code coverage report.


Side note: astroquery.mast unit tests are passing on this branch

test_mast.py .........................................                   [100%]

============================== 41 passed in 0.93s ==============================
test_mast_remote.py ..................s....................              [100%]

================== 38 passed, 1 skipped in 238.54s (0:03:58) ===================

@bsipocz

bsipocz commented Jan 25, 2022

Copy link
Copy Markdown
Member

Hmm, thanks for catching this. That PR you refer has a fully passing CI prior the merge, I'm not sure what went wrong. Nevertheless, it's indeed unrelated to this one.

@eerovaher

Copy link
Copy Markdown
Member

The latest readthedocs build is failing with an error from another module called esa:

astroquery/docs/api/astroquery.esa.jwst.JwstClass.rst:23: WARNING: autosummary: failed to import JwstClass.login_gui
WARNING: autodoc: failed to import method 'JwstClass.login_gui' from module 'astroquery.esa.jwst'; the following exception was raised:

[...]

AttributeError: type object 'JwstClass' has no attribute 'login_gui'

There is no such error anywhere in the documentation build log. See #2262 for the successful fix.

@jaymedina

Copy link
Copy Markdown
Contributor Author

I've received that error when building the docs locally in this branch, and a freshly checked out branch from main. Regarding #2262, how is that PR related to this?

@eerovaher

Copy link
Copy Markdown
Member

Some of the cached files in your docs/api/ or docs/_build/ have become outdated. Delete both directories to force Sphinx to rebuild them from scratch.

Regarding #2262, how is that PR related to this?

It fixes the documentation build.

@jaymedina jaymedina force-pushed the monkey-patch-services branch 2 times, most recently from ba567be to 0bd695b Compare January 25, 2022 16:00
@jaymedina

jaymedina commented Jan 25, 2022

Copy link
Copy Markdown
Contributor Author

Hmm unfortunately that didn't work @eerovaher . I removed api and _build . Does it work on your end?


Edit: Now receiving this warning:

WARNING: Astrometry.net API key not found in configuration file [astroquery.astrometry_net.core]
WARNING: You need to manually edit the configuration file and add it [astroquery.astrometry_net.core]
WARNING: You may also register it for this session with AstrometryNet.key = 'XXXXXXXX' [astroquery.astrometry_net.core]
/home/docs/checkouts/readthedocs.org/user_builds/astroquery/envs/2193/lib/python3.8/site-packages/astroquery/fermi/__init__.py:37: UserWarning: Experimental: Fermi-LAT has not yet been refactored to have its API match the rest of astroquery.

@jaymedina jaymedina force-pushed the monkey-patch-services branch from 0bd695b to 8042ab0 Compare January 25, 2022 18:14
@eerovaher

eerovaher commented Jan 25, 2022

Copy link
Copy Markdown
Member

Deleting the docs/api/ and docs/_build/ directories was meant to ensure that the error regarding JwstClass not having attribute login_gui stops being an issue. It wasn't supposed address the other warnings.

@bsipocz

bsipocz commented Jan 25, 2022

Copy link
Copy Markdown
Member

(unrelated to the PR, but I think good rule of thumbs: If you run into weird issues with docs build/tests, running a git clean is a useful command to clean up everything not in the repo (be careful, and exclude any draft or notes or ide directory that you may have, e.g. this is the one I usually run git clean -dfx -e .idea -e notes -e benchmarks. Alternatively usually make clean in the docs directory also works).

@jaymedina jaymedina force-pushed the monkey-patch-services branch from 8042ab0 to dc26a3e Compare January 25, 2022 20:23
@jaymedina

Copy link
Copy Markdown
Contributor Author

Just rebased on the new merged changes and readthedocs build is working again, thanks @eerovaher & @bsipocz ! CI checks are passing and unit tests are passing (remote included). I think this PR is ready for a final review and merge. Thanks in advance.

@bsipocz bsipocz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me. There is one nitpick (no need to address), and a big picture item, that may point beyond this PR anyway.

I also wonder whether it would be possible to squash a few back and forth commits together to form the logical blocks (e.g. squash formatting/cleanup ones into one, etc.).

Comment thread astroquery/mast/observations.py Outdated
Comment thread astroquery/mast/discovery_portal.py
@jaymedina jaymedina force-pushed the monkey-patch-services branch from dc26a3e to 903cb97 Compare January 26, 2022 16:41
@jaymedina jaymedina force-pushed the monkey-patch-services branch from 903cb97 to 2a18971 Compare January 26, 2022 16:47
@jaymedina jaymedina force-pushed the monkey-patch-services branch from 2a18971 to 64abfa6 Compare January 26, 2022 17:00
@jaymedina jaymedina force-pushed the monkey-patch-services branch from 64abfa6 to b6fee48 Compare January 26, 2022 17:09
…caom_filtered_position and fixing indentation
@jaymedina jaymedina force-pushed the monkey-patch-services branch from 0f93106 to 45d7ab8 Compare January 26, 2022 17:23
@jaymedina jaymedina self-assigned this Jan 26, 2022
@bsipocz bsipocz merged commit 4a3f214 into astropy:main Jan 26, 2022
@bsipocz

bsipocz commented Jan 26, 2022

Copy link
Copy Markdown
Member

Thanks @jaymedina!

@jaymedina jaymedina deleted the monkey-patch-services branch January 26, 2022 19:00
@bsipocz bsipocz added this to the v0.4.6 milestone Mar 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants