Benoit/eng-362-update-ndc-postgres-to-ndc_models-020 #666

BenoitRanque · 2025-01-01T20:22:54Z

What

This PR updates ndc-postgres to ndc spec v0.2.0
This includes a lot of changes to tests. These have been justified in individual commits.

How

BenoitRanque

Note: Failing tests: we expect failing tests related to the deprecation of the root column comparison.
These will be fixed in a separate PR, to be merged on this one before merging to main.

This has now been merged.

BenoitRanque · 2025-01-14T08:42:47Z

crates/configuration/src/version4/to_runtime_configuration.rs

                    },
                )
            })
            .collect(),
    )
 }

+/// Infer scalar type representation from scalar type name, if necessary. Defaults to JSON representation
+fn convert_or_infer_type_representation(


V0.2.0 requires type representation.

Type representation comes from introspection configuration, and may be absent.

So, if type representation is missing, we infer the type based on the name and fetch the corresponding type representation from the default introspection configuration.

Note we are pointing to a specific sdk revision We should tag a release and point to that

…n does not include a type representation, we infer one based on the scalar type name. We default to JSON representation if we don't recognize the scalar type. The mapping is pulled from the default introspection configuration. This should enable a smooth upgrade, but we may need to publish a new version of the configuration with a mechanism to guarantee type representations, later.

Note! This is a regression with regards to named scopes, which replace the previously supported RootTableColumn. There was technically no way to consume this api from the engine, so this is not a major issue, and will be addressed in an upcoming PR.

Type representations are no longer optional Schema Response now includes a reference to the scalar type to be used for count results. AggregateFunctionDefinition is now an enum, so we map based on function name. Note! We are currently lying by omission about the return types. Postgres aggregates will return NULL if aggregating over no rows, except COUNT. We should have a discussion about wether we want to change aggregate function definitions to reflect this behavior, whether all these scalars will be implicitly nullable, or whether we want to change the SQL using COALESCE to default to some value when no rows are present. Arguably, there's no proper MAX, MIN, or AVG default values. As for SUM, ndc-test expects all SUM return values to be either represented as 64 bit integers or 64 bit floats. Postgres has types like INTERVAL, which is represented as a string, and can be aggregated with SUM. We need to discuss whether any of the above needs to be revisited. We cannot represent intervals as float64 or int64.

…, so that we may count nested properties using field_path

…eign key may be on a nested field. for now, we do not suport relationships.nested, so erroring out in that case

Add reference to configuration.schema.json Add missing type representations Add missing scalar types (v4 did not require all referenced scalar types to be defined)

note thise feature is still not implemented so the test still fails

…only non-null rows, instead of COUNT(*) which would count all rows

…alidation

ndc spec expects sum aggregates return a scalar represented as either return f64 or i64 Because ndc-postgres represents i64 as a string, we only mark sum aggregates returning a f64 any other sum aggregate will function as a custom aggregate and have no special meaning additionally, we wrap SUM with `COALESCE(SUM(col), 0)` to ensure we return 0 when aggregating over no rows. similarly, we only mark avg functions returning a f64, and treat any other avg as a custom aggregate

…y tables in scope for an exists, instead of only root and current. (#674)  ### What  `ComparisonTarget::RootCollectionColumn` was removed, to be replaced by [named scopes](https://github.com/hasura/ndc-spec/blob/36855ff20dcbd7d129427794aee9746b895390af/rfcs/0015-named-scopes.md). This PR implements the replacement functionality.  ### How  This PR replaces RootAndCurrentTables, with TableScope, a struct that keeps track of the current table and any tables in scope for exists expression. See the accompanying review for details on the code itself.

…(e) operators

BenoitRanque · 2025-01-16T22:27:14Z

crates/configuration/src/version3/mod.rs

                    },
                )
            })
            .collect(),
    )
 }

+/// Infer scalar type representation from scalar type name, if necessary. Defaults to JSON representation
+fn convert_or_infer_type_representation(


V0.2.0 requires type representation, but configuration did not require configuration to be present.

To maximize compatibility with older configuration versions, we infer missing type representations based on scalar type name. If missing, we default to JSON representation.

BenoitRanque · 2025-01-16T22:30:28Z

crates/configuration/src/version4/comparison.rs

@@ -29,22 +29,22 @@ impl ComparisonOperatorMapping {
            ComparisonOperatorMapping {
                operator_name: "<=".to_string(),
                exposed_name: "_lte".to_string(),
-                operator_kind: OperatorKind::Custom,
+                operator_kind: OperatorKind::LessThanOrEqual,


Default introspection configuration changed to tag lt(e),gt(e) operators.

This will only affect new configurations, so any deployments with existing configuration will see no change in behavior.

BenoitRanque · 2025-01-16T22:36:27Z

crates/connectors/ndc-postgres/src/schema/mod.rs

+                                function_name.as_str(),
+                                function_definition.return_type.as_str(),
+                            ) {
+                                ("sum", "float8" | "int8") => {


v0.2.0 adds standard aggregate functions. These have specific expectations, such as sum needing to return a scalar represented as either Float64 or Int64.

We check for specific aggregate functions returning matching data types, and mark applicable functions as such.

Non-compliant functions (eg. sum on interval types which are represented as strings) will be tagged as custom aggregate functions

BenoitRanque · 2025-01-16T22:37:23Z

crates/connectors/ndc-postgres/src/schema/mod.rs

    Ok(models::SchemaResponse {
        collections,
        procedures,
        functions: vec![],
        object_types,
        scalar_types,
+        capabilities: Some(models::CapabilitySchemaInfo {


Adding this is required, but also means we will see a change in returned schemas, even if configuration has not been changed.

BenoitRanque · 2025-01-16T23:45:37Z

crates/query-engine/translation/src/translation/query/filtering.rs

+                        field_path,
+                        scope,
+                    } => {
+                        let scoped_table = current_table_scope.scoped_table(scope)?;


Apply scope, if any, before traversing path

BenoitRanque · 2025-01-16T23:54:40Z

crates/query-engine/translation/src/translation/query/sorting.rs

+                                                args: vec![column],
+                                            }
+                                        }
+                                        OrderByAggregate::CountStar | OrderByAggregate::Count => {


Count Star and Count actually behave the same, where we only count left-hand rows that actually exists.

This is important, as left joins + count(*) will actually count all rows, even if there were no matching left-hand rows.

I believe those semantics are correct, but something to double check.

BenoitRanque · 2025-01-16T23:58:22Z

crates/query-engine/translation/src/translation/query/sorting.rs

 }
+
+enum OrderByAggregate {


We created a new enum for the various ordering aggregates

BenoitRanque · 2025-01-16T23:58:55Z

crates/query-engine/translation/src/translation/query/sorting.rs

@@ -703,10 +740,10 @@ fn translate_targets(
                                // Aggregates do not have a field path.
                                field_path: (&None).into(),
                                expression: sql::ast::Expression::Value(sql::ast::Value::Int4(1)),
-                                aggregate: Some(sql::ast::Function::Unknown("COUNT".to_string())),
+                                aggregate: Some(OrderByAggregate::CountStar),


We used our new ordering aggregate enum instead of a direction SQL AST function.

BenoitRanque · 2025-01-17T00:09:20Z

crates/tests/tests-common/src/request.rs

+/// this test should be ignored unless explicitly invoked
+#[ignore]
+#[test]
+fn generate_query_request_schema() {


Added these utilities to generate query and mutation request schemas, to validate tests files. These tests are not invoked unless explicitly called upon. There's probably better ways to do this, and we can remove them if there's any feelings against keeping them around.

BenoitRanque · 2025-01-17T00:09:43Z

crates/tests/tests-common/src/router.rs

@@ -21,9 +21,9 @@ pub async fn create_router(
    )]);
    let setup = PostgresSetup::new(environment);

-    let state = ndc_sdk::default_main::init_server_state(setup, &absolute_configuration_directory)
+    let state = ndc_sdk::state::init_server_state(setup, &absolute_configuration_directory)


No idea what I did here or why, just made the SDK/compiler happy

BenoitRanque commented Jan 1, 2025

View reviewed changes

danieljharvey requested a review from a team January 7, 2025 18:47

BenoitRanque commented Jan 14, 2025

View reviewed changes

BenoitRanque force-pushed the benoit/eng-362-update-ndc-postgres-to-ndc_models-020 branch from ed42b7e to 6fa57df Compare January 16, 2025 16:48

BenoitRanque added 26 commits January 16, 2025 18:00

Update to ndc-models 0.2.0

c1a0017

Note we are pointing to a specific sdk revision We should tag a release and point to that

Count aggregates now take an expression instead of a column reference…

25b96a7

…, so that we may count nested properties using field_path

remove todo

96985b5

remove todo, jandle a couple additional cases as errors/unsupported

8f5f678

add additional error types

0be15c4

add test for aggregate of nested field

47245b8

relationship mapping now have an array on the right-hand side, so for…

3131257

…eign key may be on a nested field. for now, we do not suport relationships.nested, so erroring out in that case

scalar type no longer optional

7277f34

nested field collection not supported, add error handling

8dc0211

update filtering, aggregates, sorting, to 0.2.0

16e32fe

add schema files for configurations, mutations, queries

a87ffc9

Update translation test configurations to v5

1a1c0ce

Add reference to configuration.schema.json Add missing type representations Add missing scalar types (v4 did not require all referenced scalar types to be defined)

add reference to query/mutation request schema json

08612c1

right hand side of relationship column mapping is now an array

3740a0e

update aggregate expresion

4b2d373

use exists instead of left-hand side path

3694e96

update root_collection_column test to use scopes instead

9c4c662

note thise feature is still not implemented so the test still fails

Correct sorting behavior. We use CCUNT("col") for CountStar to count …

6ea5b3a

…only non-null rows, instead of COUNT(*) which would count all rows

use exists instead of left-hand side path

e947d61

v0.2.0 schema update

bea5ab4

use exists instead of left-handed path

bc4138b

add reference to query/mutation request schema in test request files

67595f0

right-hand side of relationship column mapping is now an array

0598a2f

BenoitRanque added 14 commits January 16, 2025 18:00

use exists instead of left-hand side path

14cad36

aggregates have changed

cf47f9c

comparison target columns have been simplified

a874165

root column comparison now replaced with scopes

0bfd425

update tests to new sdk

46f4ab9

add manually triggered tests to generate schema files for test file v…

a77fbf6

…alidation

remove debug print statement

d56fb32

fix: sums that return int8 should be marked with the meaning field

80f187e

We'll need to implement filtering by aggregates later

dc70526

add standard comparison operators, make no changes to defaults

cb3cce7

schema snapshots updated to reflect changes in sum aggregates definition

3102706

change default introspection settings to include meaning for gt(e),lt…

829886f

…(e) operators

BenoitRanque force-pushed the benoit/eng-362-update-ndc-postgres-to-ndc_models-020 branch from 4234509 to 829886f Compare January 16, 2025 22:11

update Cargo.lock

15fa6df

BenoitRanque commented Jan 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benoit/eng-362-update-ndc-postgres-to-ndc_models-020 #666

Benoit/eng-362-update-ndc-postgres-to-ndc_models-020 #666

BenoitRanque commented Jan 1, 2025

BenoitRanque left a comment •

edited

Loading

BenoitRanque Jan 14, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 16, 2025

BenoitRanque Jan 17, 2025

BenoitRanque Jan 17, 2025

Benoit/eng-362-update-ndc-postgres-to-ndc_models-020 #666

Are you sure you want to change the base?

Benoit/eng-362-update-ndc-postgres-to-ndc_models-020 #666

Conversation

BenoitRanque commented Jan 1, 2025

What

How

BenoitRanque left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenoitRanque left a comment •

edited

Loading