Duplicate Logs Issue Due to next_start_date Configuration in Google Workspace Admin Module

[ninasjfrog]

While using the Google Workspace Admin module for Filebeat, I've encountered an issue where duplicate logs are being received.

The problem seems to originate from the url.params.startTime configuration, particularly with the following logic:

value: >-
        [[- if eq .cursor.pagination_finished "true" -]]
          [[- .cursor.next_start_date -]]
        [[- else -]]
          [[- .cursor.last_response_date -]]
        [[- end -]]
      default: '[[formatDate (now (parseDuration "-{{.initial_interval}}"))]]'

Here, next_start_date is being set to [[ .first_event.id.time ]] (the event time). However, this causes the module to set the startTime parameter to the time of the event, resulting in the same event being pulled repeatedly.

Since startTime sets the beginning of the range for the logs, this behavior causes multiple logs to be received for the same event, leading to duplication.

Steps to Reproduce:

1. Configure the Google Workspace Admin module as per the documentation.
2. Enable the module and start Filebeat.
3. Observe the logs being ingested with duplication cased on the uniqueQualifier.

Environment:
Filebeat version: 8.16.1
Module: Google Workspace Admin

Discuss Forum URL: https://discuss.elastic.co/t/duplicate-logs-issue-due-to-next-start-date-configuration-in-google-workspace-admin-module/372082

[botelastic]

This issue doesn't have a Team:<team> label.