Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support date_format via Gpu for non-UTC time zone [databricks] #9721

Merged
merged 11 commits into from
Dec 14, 2023
Merged

Support date_format via Gpu for non-UTC time zone [databricks] #9721

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7625e16
Support from_unixtime via Gpu for non-UTC time zone
Dec 4, 2023
6c961fb
Add is_dst_tz; Fix
Dec 4, 2023
4c6133e
Update comments; Refactor; Use Gpu path
Dec 5, 2023
92741c6
Add licence info; re-order imports
Dec 5, 2023
877130a
xfail case instead of skipif
Dec 5, 2023
e3941c2
Support date_format via GPU for non-UTC time zone
Dec 5, 2023
1538d82
Fix; Update license; Update allow onon gpu list
Dec 5, 2023
4eaa4dd
Merge branch-24.02
Dec 8, 2023
7e61915
Remove useless code
Dec 8, 2023
96924eb
Add date_format test cases
Dec 8, 2023
f1314e0
Merge branch-24.02
Dec 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions integration_tests/src/main/python/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
import random
import warnings

from time_zone_utils import *

# TODO redo _spark stuff using fixtures
#
# Don't import pyspark / _spark directly in conftest globally
Expand Down Expand Up @@ -86,6 +88,14 @@ def is_utc():
def is_not_utc():
return not is_utc()

def is_dst_time_zone():
"""
Is current TZ is Daylight Saving Time zone
"""
tz = get_test_tz()
assert tz in all_valid_time_zones
return tz in dst_time_zones

_is_nightly_run = False
_is_precommit_run = False

Expand Down
25 changes: 22 additions & 3 deletions integration_tests/src/main/python/date_time_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

import pytest
from asserts import assert_gpu_and_cpu_are_equal_collect, assert_gpu_fallback_collect, assert_gpu_and_cpu_error
from conftest import is_not_utc
from conftest import is_dst_time_zone
from data_gen import *
from datetime import date, datetime, timezone
from marks import ignore_order, incompat, allow_non_gpu
Expand Down Expand Up @@ -467,13 +467,32 @@ def test_gettimestamp_ansi_exception():

supported_date_formats = ['yyyy-MM-dd', 'yyyy-MM', 'yyyy/MM/dd', 'yyyy/MM', 'dd/MM/yyyy',
'MM-dd', 'MM/dd', 'dd-MM', 'dd/MM']
@pytest.mark.parametrize('date_format', supported_date_formats, ids=idfn)
@pytest.mark.parametrize('date_format', [date_gen], ids=idfn)
@pytest.mark.parametrize('data_gen', date_n_time_gens, ids=idfn)
@allow_non_gpu(*non_utc_allow)
winningsix marked this conversation as resolved.
Show resolved Hide resolved
def test_date_format(data_gen, date_format):
def test_date_format_for_date(data_gen, date_format):
assert_gpu_and_cpu_are_equal_collect(
lambda spark : unary_op_df(spark, data_gen).selectExpr("date_format(a, '{}')".format(date_format)))

@pytest.mark.parametrize('date_format', supported_date_formats, ids=idfn)
@pytest.mark.parametrize('data_gen', [timestamp_gen], ids=idfn)
@pytest.mark.xfail(is_dst_time_zone(), reason="only support non-DST time zone, refer to https://github.com/NVIDIA/spark-rapids/issues/6839")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid xfail here?

If is_dst_time_zone = false, test result equals without fallback.
If is_dst_time_zone = true, test result equals with CPU operator fallback. This one can be combined with test_date_format_for_date.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using xfail is to record the xfail/xpass cases.
I'm OK to use skipif, but we may forget it in the future.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use fallback check instead? I am nervous on xfail which hides errors.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, will do it

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

def test_date_format_for_time(data_gen, date_format):
conf = {'spark.rapids.sql.nonUTC.enabled': True}
assert_gpu_and_cpu_are_equal_collect(
lambda spark : unary_op_df(spark, data_gen).selectExpr("date_format(a, '{}')".format(date_format)),
conf)

@pytest.mark.parametrize('date_format', supported_date_formats, ids=idfn)
# from 0001-02-01 to 9999-12-30
@pytest.mark.parametrize('data_gen', [LongGen(min_val=int(datetime(1, 2, 1).timestamp()), max_val=int(datetime(9999, 12, 30).timestamp()))], ids=idfn)
@pytest.mark.xfail(is_dst_time_zone(), reason="only support non-DST time zone, refer to https://github.com/NVIDIA/spark-rapids/issues/6839")
def test_from_unixtime(data_gen, date_format):
conf = {'spark.rapids.sql.nonUTC.enabled': True}
assert_gpu_and_cpu_are_equal_collect(
lambda spark : unary_op_df(spark, data_gen, length=5).selectExpr("from_unixtime(a, '{}')".format(date_format)),
conf)

unsupported_date_formats = ['F']
@pytest.mark.parametrize('date_format', unsupported_date_formats, ids=idfn)
@pytest.mark.parametrize('data_gen', date_n_time_gens, ids=idfn)
Expand Down
Loading