How to fetch Pinot data from pinpoint web

[HanChanYoung]

Hello, I am using Pinpoint version 3.0.0 and am viewing various information (e.g., URL statistics) on the Pinpoint web using Pinot data.

I have a question.

1. When I look at the Pinot tables, there are realtime and offline tables. Currently, it seems like data is being fetched from the realtime table. Under what conditions is data fetched from the offline table?

2. If data is only loaded into the realtime table and data in the offline table shows as NaN, what could be the problem? Thank you.

[HanChanYoung]

I think I've found a partial solution to number 2.

For example, with uristat, there was no minion instance and no task, so it couldn't execute. But another problem has arisen, and I'm getting continuous errors. Can you tell me what the error is?

org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 5 attempts at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) at org.apache.pinot.plugin.minion.tasks.SegmentConversionUtils.uploadSegment(SegmentConversionUtils.java:140) at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.pushSegment(BaseMultipleSegmentsConversionExecutor.java:352) at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:308) at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:77) at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:157) at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:118) at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75) at java.base/java.util.co

[donghun-cho]

RealtimeToOfflineSegmentsTask is used to ensure speed when table usage is high.
If you have an acceptable response time, you can disable it to save disk space.

1. condition is time-boundary
   Please refer to the following document regarding the time-boundary: https://docs.pinot.apache.org/basics/concepts/components/table/time-boundary

2. There is no segment in offline table
   RealtimeToOfflineSegmentsTask requires a Pinot Minion, as you mentioned.

I once experienced that RealtimeToOfflineSegmentsTask will not run if all of the Kafka partitions are filled with data (in Pinot 1.0.0)
It might be helpful to manually trigger minon task from the Pinot controller's Minion Task Manager to find the reason

[HanChanYoung]

@donghun-cho
Thank you very much for your kind response.

As I mentioned for number 2, I had add Pinot Minion.

However, I continued to get errors, and upon checking, I found that while the Realtime Table had "replicasPerPartition": "1", and Offline Table had "replication": "3".

It seems like these conflicting settings were causing the continuous errors. Ultimately, when I set both the Realtime Table and the Offline Table to "1", the RealtimeToOfflineSegmentsTask worked correctly.

And I have a one more Questions.
Unlike other tables,(e.g.,uristat)
ExceptionTrace Realtime table, there is no defined task(https://github.com/pinpoint-apm/pinpoint/blob/master/exceptiontrace/exceptiontrace-common/src/main/pinot/pinot-exceptionTrace-realtime-table.json) to transfer segments to the offline table. Is there a reason for this?

[donghun-cho]

So replication value was the cause of the error.
The replication value and replicasPerPartition value should be less than or equal to the number of Pinot servers.

@intr3p1d Could you check the question above?

[intr3p1d]

@HanChanYoung
I have reviewed the settings you mentioned and identified that the settings were not appropriately updated during the development process.

To address this, the settings have been updated in #11455.

Thank you for bringing this to our attention and for reviewing the settings.