[1.10] Backport always merge DVs on commit path (#15006) to 1.10.x#15654
Conversation
|
OK core tests in 1.10 with this because: that's a new test I added as part of the original change but the test relies on making sure DV fields are properly copied over in the FileMetadata builder If not, then we just update this test in this 1.10 branch to just explicitly pass through the value in the builder |
|
I would probably just modify/ remove the failing test to avoid the behavior change unless you are 100% confident we would really be always fixing a bug and not break anyone with slightly different behavior. We can always have folks upgrade to 11 if they need the check to work. |
|
Yeah I agree @RussellSpitzer , I took another look and while we could cherry pick the specific parts of #14351 to address this bug, it's probably not worth it for a patch release where it's unclear if someone relies on the fields being nulled out or something after a copy. It's better to have that change be in 1.11. I updated the tests to work around the issue. |
singhpk234
left a comment
There was a problem hiding this comment.
+1, thanks @amogh-jahagirdar !
|
Thanks all, merging! |
Backports always merge DVs on commit path (#15006) to 1.10.x
Note:
This was not a completely clean backport because 1.10 doesn't have spark 4.1 and in the original PR some 4.1 tests had to be updated. This does include the test updates for the spark versions that were part of the 1.10 release.
Fixes #15599