Documentation and other papercuts for datasource api launch

datasources/src/main/java/org/opensearch/sql/datasources/settings/DataSourceSettings.java

@@ -11,9 +11,12 @@
 
 public class DataSourceSettings {
 
+  // we are keeping this to not break upgrades if the config is already present.
+  // This will be completely removed in 3.0.
   public static final Setting<InputStream> DATASOURCE_CONFIG = SecureSetting.secureFile(
       "plugins.query.federation.datasources.config",
-      null);
+      null,
+      Setting.Property.Deprecated);
 
   public static final Setting<String> DATASOURCE_MASTER_SECRET_KEY = Setting.simpleString(
       "plugins.query.datasources.encryption.masterkey",

datasources/src/main/resources/datasources-index-mapping.yml

@@ -4,9 +4,7 @@
 # SPDX-License-Identifier: Apache-2.0
 ##
 
-# Schema file for the observability index
-# Since we only search based on "access", sort on lastUpdatedTimeMs & createdTimeMs,
-# other fields are not used in mapping to avoid index on those fields.
+# Schema file for the .ql-datasources index
 # Also "dynamic" is set to "false" so that other fields can be added.
 dynamic: false
 properties:
@@ -15,5 +13,5 @@ properties:
     fields:
       keyword:
         type: keyword
-  connectorType:
+  connector:
     type: keyword

datasources/src/main/resources/datasources-index-settings.yml

@@ -4,7 +4,7 @@
 # SPDX-License-Identifier: Apache-2.0
 ##
 
-# Settings file for the observability index
+# Settings file for the .ql-datasources index
 index:
   number_of_shards: "1"
   auto_expand_replicas: "0-2"

docs/user/ppl/admin/datasources.rst

@@ -17,6 +17,11 @@ The concept of ``datasource`` is introduced to support the federation of SQL/PPL
 This helps PPL users to leverage data from multiple data stores and derive correlation and insights.
 Datasource definition provides the information to connect to a data store and also gives a name to them to refer in PPL commands.
 
+Refer below sections for quick setup.
+
+* `Datasource configuration APIs`_
+* `Master Key config for encrypting credential information`_
+
 
 Definitions of datasource and connector
 ====================================
@@ -47,7 +52,7 @@ Datasource configuration Restrictions.
     * ``prometheus`` [More details: `Prometheus Connector <prometheus_connector.rst>`_]
 * All the allowed config parameters in ``properties`` are defined in individual connector pages mentioned above.
 
-Datasource configuration Management.
+Datasource configuration APIs
 ======================================
 Datasource configuration can be managed using below REST APIs. All the examples below are for OpenSearch domains enabled with secure domain.
 we can remove authorization and other details in case of security disabled domains.
@@ -113,6 +118,24 @@ Each of the datasource configuration management apis are controlled by following
 
 Only users mapped with roles having above actions are authorized to execute datasource management apis.
 
+Master Key config for encrypting credential information
+========================================================
+* When users provide credentials for a data source, the system encrypts and securely stores them in the metadata index. System uses "AES/GCM/NoPadding" symmetric encryption algorithm.
+* Users can set up a master key to use with this encryption method by configuring the plugins.query.datasources.encryption.masterkey setting in the opensearch.yml file.
+* The master key must be 16, 24, or 32 characters long.
+* It's highly recommended that users configure a master key for better security.
+* If users don't provide a master key, the system will default to "0000000000000000".
+* Sample python script to generate a 24 character master key ::
+
+    import random
+    import string
+
+    # Generate a 24-character random master key
+    master_key = ''.join(random.choices(string.ascii_letters + string.digits, k=24))
+
+    # Print the master key
+    print("Generated master key:", master_key)
+
 Using a datasource in PPL command
 ====================================
 Datasource is referred in source command as show in the code block below.
@@ -127,7 +150,7 @@ Example source command with prometheus datasource ::
 
 
 Authorization of PPL commands on datasources
-==============================================
+============================================
 In case of secure opensearch domains, only admins and users with roles mentioned in datasource configuration are allowed to make queries.
 For example: with below datasource configuration, only admins and users with prometheus_access role can run queries on my_prometheus datasource. ::
 
@@ -144,7 +167,7 @@ For example: with below datasource configuration, only admins and users with pro
     }
 
 
-Limitations of datasource
-====================================
-Datasource settings are global and users with PPL access are allowed to fetch data from all the defined datasources.
-PPL access can be controlled using roles.(More details: `Security Settings <security.rst>`_)
+Moving from keystore datasource configuration
+=============================================
+* In versions prior to 2.7, the plugins.query.federation.datasources.config key store setting was used to configure datasources, but it has been deprecated and will be removed in version 3.0.
+* To port previously configured datasources from the keystore, users can use the `create datasource` REST API mentioned in the above section.

doctest/build.gradle

@@ -121,7 +121,6 @@ String mlCommonsPlugin = 'opensearch-ml'
 
 testClusters {
     docTestCluster {
-        keystore 'plugins.query.federation.datasources.config', new File("$projectDir/datasource", 'datasources.json')
         // Disable loading of `ML-commons` plugin, because it might be unavailable (not released yet).
         /*
         plugin(provider(new Callable<RegularFile>(){

integ-test/src/test/java/org/opensearch/sql/ppl/InformationSchemaCommandIT.java

@@ -16,6 +16,7 @@
 import org.json.JSONObject;
 import org.junit.After;
 import org.junit.AfterClass;
+import org.junit.BeforeClass;
 import org.junit.jupiter.api.AfterAll;
 import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.Test;
@@ -28,6 +29,18 @@ protected void init() throws Exception {
     loadIndex(Index.DATASOURCES);
   }
 
+  /**
+   * Integ tests are dependent on self generated metrics in prometheus instance.
+   * When running individual integ tests there
+   * is no time for generation of metrics in the test prometheus instance.
+   * This method gives prometheus time to generate metrics on itself.
+   * @throws InterruptedException
+   */
+  @BeforeClass
+  protected static void metricGenerationWait() throws InterruptedException {
+    Thread.sleep(10000);
+  }
+
   @Test
   public void testSearchTablesFromPrometheusCatalog() throws IOException {
     JSONObject result =

integ-test/src/test/java/org/opensearch/sql/ppl/PrometheusDataSourceCommandsIT.java

@@ -18,6 +18,7 @@
 import org.apache.commons.lang3.StringUtils;
 import org.json.JSONArray;
 import org.json.JSONObject;
+import org.junit.BeforeClass;
 import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.Test;
 
@@ -29,6 +30,18 @@ protected void init() throws Exception {
     loadIndex(Index.DATASOURCES);
   }
 
+  /**
+   * Integ tests are dependent on self generated metrics in prometheus instance.
+   * When running individual integ tests there
+   * is no time for generation of metrics in the test prometheus instance.
+   * This method gives prometheus time to generate metrics on itself.
+   * @throws InterruptedException
+   */
+  @BeforeClass
+  protected static void metricGenerationWait() throws InterruptedException {
+    Thread.sleep(10000);
+  }
+
   @Test
   @SneakyThrows
   public void testSourceMetricCommand() {

integ-test/src/test/java/org/opensearch/sql/ppl/ShowDataSourcesCommandIT.java

@@ -14,6 +14,7 @@
 
 import java.io.IOException;
 import org.json.JSONObject;
+import org.junit.BeforeClass;
 import org.junit.jupiter.api.Test;
 
 public class ShowDataSourcesCommandIT extends PPLIntegTestCase {
@@ -22,6 +23,18 @@ protected void init() throws Exception {
     loadIndex(Index.DATASOURCES);
   }
 
+  /**
+   * Integ tests are dependent on self generated metrics in prometheus instance.
+   * When running individual integ tests there
+   * is no time for generation of metrics in the test prometheus instance.
+   * This method gives prometheus time to generate metrics on itself.
+   * @throws InterruptedException
+   */
+  @BeforeClass
+  protected static void metricGenerationWait() throws InterruptedException {
+    Thread.sleep(10000);
+  }
+
   @Test
   public void testShowDataSourcesCommands() throws IOException {
     JSONObject result = executeQuery("show datasources");