[DRAFT] fix: Use CTE in Enrollment analytics queries [DHIS-16705]

[luciano-fiandesio]

WIP

Changes

This PR addresses the generation of SQL queries for Enrollment analytics

Problem

Currently, the Enrollment queries are structured so that sub-queries are used to fetch events values from the analytics_event_* tables.
For instance:

select
    enrollment,
    ...
    ax."ou",
    (select
         "fyjPqlHE7Dn"
     from
	 analytics_event_M3xtLkYBlKI
     where
	 analytics_event_M3xtLkYBlKI.eventstatus != 'SCHEDULE'
	 and analytics_event_M3xtLkYBlKI.enrollment = ax.enrollment
	 and ps = 'CWaAcQYKVpq'
limit 1 ) as "CWaAcQYKVpq[-1].fyjPqlHE7Dn",
from
     analytics_enrollment_m3xtlkyblki as ax
...

The above query works in Postgres but does not work in Doris, because correlation with outer layers of the parent query is not supported.

Mitigation

The current approach is trying to refactor the Enrollment query so that the subqueries (both as select and as where conditions) are "moved" into CTE (Common Table Expressions).
Common Table Expressions are temporary result sets in SQL, defined within a WITH clause, that simplify complex queries by improving readability and modularizing logic. They are particularly useful for recursive queries and can be referenced multiple times within the main query.

The above query can be rewritten like so:

with ps_cwaacqykvpq_fyjpqlhe7dn as (
select
	distinct on
	(enrollment) enrollment,
	"fyjPqlHE7Dn" as value
from
	analytics_event_M3xtLkYBlKI
where
	eventstatus != 'SCHEDULE'
	and ps = 'CWaAcQYKVpq'
	
order by
	enrollment,
	occurreddate desc,
	created desc)
select
	ax.enrollment,
	...
	ps_cwaacqykvpq_fyjpqlhe7dn.value as "CWaAcQYKVpq.fyjPqlHE7Dn"
from
	analytics_enrollment_m3xtlkyblki as ax
left join ps_cwaacqykvpq_fyjpqlhe7dn on
	ax.enrollment = ps_cwaacqykvpq_fyjpqlhe7dn.enrollment
where
	(((enrollmentdate >= '2021-01-01'
	and enrollmentdate < '2022-01-01')))
	and (ax."uidlevel1" = 'ImspTQPwCqd' )
...

The above query structure is compatible with Doris and it makes the execution of the query faster.
As a comparison:

Original query with sub-select

Refactored query with CTEs

Testing strategy

I am using the e2e project and aim at having 100% green tests on both Postgres and Doris.

To fix the problem, we need to validate the offset parameter before performing any arithmetic operations. Specifically, we should ensure that the offset value is within a safe range to prevent overflow or underflow. We can achieve this by adding a guard clause that checks if the offset value is within the range of Integer.MIN_VALUE to Integer.MAX_VALUE.

1. Add a validation check for the offset parameter in the computeRowNumberOffset method.
2. If the offset value is outside the safe range, we can either throw an exception or clamp the value to the nearest safe limit.

[larshelge]

Let's use lower-case for SQL keywords in Java, i.e. with.

[larshelge]

Remove println.

[larshelge]

Use lower-case keywords for SQL in Java.

[larshelge]

Remove outer @.

[larshelge]

Rename to contributeCte.

[sonarqubecloud]

Quality Gate failed

Failed conditions
53 New issues
3 New Critical Issues (required ≤ 0)
53 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE