Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

json-valid PPL function #3230

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

json-valid PPL function #3230

wants to merge 17 commits into from

Conversation

kenrickyap
Copy link

Description

Based on this feature request: #3207

Added json_valid request.

### `JSON_VALID`

**Description**

`json_valid(jsonStr)` Evaluates whether a json-encoded string contains valid JSON syntax and returns TRUE or FALSE.

**Argument type:** STRING

**Return type:** BOOLEAN

Example:

    os> source=people | eval `valid_json` = json_valid('[1,2,3,4]'), `invalid_json` = json_valid('{"invalid": "json"') | fields `valid_json`, `invalid_json`
    fetched rows / total rows = 1/1
    +--------------+----------------+
    | valid_json   | invalid_json   |
    +--------------+----------------+
    | True         | False          |
    +--------------+----------------+

    os> source=accounts | where json_valid('[1,2,3,4]') and isnull(email) | fields account_number, email
    fetched rows / total rows = 1/1
    +------------------+---------+
    | account_number   | email   |
    |------------------+---------|
    | 13               | null    |
    +------------------+---------+

Related Issues

Resolves #3207

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@kenrickyap kenrickyap marked this pull request as draft January 3, 2025 18:53
@14yapkc1
added implementation
70152eb 
Signed-off-by: Kenrick Yap <[email protected]>
@YANG-DB
Copy link
Member

YANG-DB commented Jan 6, 2025

@kenrickyap can you please add the relevant documentation for this new function?

@kenrickyap
addressed PR comments
ce2c551 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
Copy link
Author

@kenrickyap can you please add the relevant documentation for this new function?

Added doctest, integ-test, and unit tests

@kenrickyap kenrickyap marked this pull request as ready for review January 6, 2025 23:28
acarbonetto
acarbonetto reviewed Jan 6, 2025
View reviewed changes
core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java Outdated Show resolved Hide resolved
core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java Outdated Show resolved Hide resolved
core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java Outdated Show resolved Hide resolved
core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java Outdated Show resolved Hide resolved
core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java Outdated Show resolved Hide resolved
core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java Outdated Show resolved Hide resolved
docs/user/ppl/functions/json.rst Outdated Show resolved Hide resolved
docs/user/ppl/functions/json.rst Outdated Show resolved Hide resolved
integ-test/src/test/resources/indexDefinitions/json_test_index_mappping.json Outdated Show resolved Hide resolved
@kenrickyap kenrickyap marked this pull request as draft January 7, 2025 00:26
@kenrickyap
fixed unit tests
ad1bde3 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
addressed pr comments
ccf47a2 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
Copy link
Author

3 unit test are failing due to 2024 -> 2025 year change

DateTimeFunctionTest > testWeekOfYearWithTimeType()
ExtractTest > testExtractDatePartWithTimeType()
YearweekTest > testYearweekWithTimeType()

There is existing issues for said failed test cases: #2477

I have disabled test until above issue is addressed

@kenrickyap
addressed PR comments
acc76a0 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
removed unused dependencies
519c6f2 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap kenrickyap marked this pull request as ready for review January 7, 2025 21:45
@kenrickyap
linting
2e319fe 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
Copy link
Author

pipeline is failing as disabled test are affecting code coverage. will first resolve #2477, then merge this ticket

kenrickyap and others added 4 commits January 9, 2025 13:01
@kenrickyap @acarbonetto
Update docs/user/ppl/functions/json.rst
7ef6cc9 
Co-authored-by: Andrew Carbonetto <[email protected]>
Signed-off-by: kenrickyap <[email protected]>
@kenrickyap @acarbonetto
Update integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionIT…
e5e90ac 
….java

Co-authored-by: Andrew Carbonetto <[email protected]>
Signed-off-by: kenrickyap <[email protected]>
@kenrickyap
nit
2187a5a 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
Merge branch 'feature/json-valid' of https://github.com/Bit-Quill/ope…
5e1e488 
…nsearch-project-sql into feature/json-valid

Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
fixed integ test
3512b33 
Signed-off-by: Kenrick Yap <[email protected]>
@kenrickyap
change text type to keyword
9fea606 
Signed-off-by: Kenrick Yap <[email protected]>
YANG-DB
YANG-DB reviewed Jan 10, 2025
View reviewed changes
docs/user/ppl/functions/json.rst
@@ -0,0 +1,35 @@
====================
IP Address Functions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the title IP Address Functions ?

YANG-DB
YANG-DB reviewed Jan 10, 2025
View reviewed changes
doctest/test_data/json_test.json
{"test_name":"json array", "json_string":"[1, 2, 3, 4]"}
{"test_name":"json scalar string", "json_string":"\"abc\""}
{"test_name":"json empty string","json_string":""}
{"test_name":"json invalid object", "json_string":"{\"invalid\":\"json\", \"string\"}"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also add complex nested json object sample ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YANG-DB YANG-DB YANG-DB left review comments

@acarbonetto acarbonetto acarbonetto approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@LantaoJin LantaoJin Awaiting requested review from LantaoJin LantaoJin is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE] Add json_valid as a PPL function
4 participants
@kenrickyap @YANG-DB @acarbonetto @14yapkc1