Feature: CVE Remediations - Twistlock scan results have identified Critical and High vulnerabilities to be remediated

[carter-cundiff]

Description

The aiSSEMBLE base images were scanned using Twistlock to identify any potential CVEs that may prevent an aiSSEMBLE-based project from being authorized to deploy in a secure computing environment.

We want to focus on Critical CVEs first, and resolve Highs second. Will be worthwhile to verify which versions are cited to ensure we can retain core aiSSEMBLE capabilities.

DOD

• Update the following images to resolve all critical vulnerabilities:
  • aissemble-hive-service:1.10.0
    • Resolved 42 critical vulnerabilities pertaining to com.fasterxml.jackson.core_jackson-databind
  • aissemble-pipeline-invocation:1.10.0
  • aissemble-configuration-store:1.10.0

----------------Follow On Ticket------------------------

• Update the following images to resolve all critical vulnerabilities:
  • aissemble-spark:1.10.0
  • aissemble-spark-operator:1.10.0
• Update the following images to resolve all high vulnerabilities:
  • aissemble-spark:1.10.0
  • aissemble-spark-operator:1.10.0
  • aissemble-hive-service:1.10.0
  • aissemble-pipeline-invocation:1.10.0
  • aissemble-configuration-store:1.10.0

Test Strategy/Script

• OTS Only:

  • Within the aiSSEMBLE repo, run the following and verify it builds successfully:

     mvn clean install -pl :aissemble-hive-service -Dmaven.build.cache.skipCache
    

• Create a downstream project:

mvn archetype:generate -U -DarchetypeGroupId=com.boozallen.aissemble \
  -DarchetypeArtifactId=foundation-archetype \
  -DarchetypeVersion=1.11.0-SNAPSHOT \
  -DgroupId=com.test \
  -DartifactId=test-518 \
  -DprojectGitUrl=test.url \
  -DprojectName=test-518 \
  && cd test-518

• Add the attached SparkPipeline.json to the test-518-pipeline-models/src/main/resources/pipelines/ directory
• Add the attached PersonDictionary.json to the test-518-pipeline-models/src/main/resources/dictionaries/ directory
• Add the attached Person.json to the test-518-pipeline-models/src/main/resources/records/ directory
• Run mvn clean install until all the manual actions are complete
• Add the following execution to the test-518-deploy/pom.xml:

<execution>
    <id>trino</id>
    <phase>generate-sources</phase>
    <goals>
        <goal>generate-sources</goal>
    </goals>
    <configuration>
        <basePackage>com.test</basePackage>
        <profile>data-access-trino-deploy-v2</profile>
        <!-- The property variables below are passed to the Generation Context and utilized
                to customize the deployment artifacts. -->
        <propertyVariables>
            <appName>trino</appName>
        </propertyVariables>
    </configuration>
</execution>

• Add the following to the test-518-pipelines/spark-pipeline/src/main/java/com/test/TestSyncStep.java:

+import java.util.List;
+import java.util.stream.Stream;
+import simple.test.record.Person;
+import simple.test.record.PersonSchema;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;

...

    @Override
    protected void executeStepImpl() {
-         // TODO: Add your business logic here for this step!
-         logger.error("Implement executeStepImpl(..) or remove this pipeline step!");
+        logger.info("Saving Person to table People");
+        Person person = new Person();
+        person.setName("John Smith");
+        person.setAge(50);
+        PersonSchema personSchema = new PersonSchema();
+        List<Row> rows = Stream.of(person).map(PersonSchema::asRow).toList();
+        Dataset<Row> dataset = sparkSession.createDataFrame(rows, personSchema.getStructType());
+        saveDataset(dataset, "People");
+        logger.info("Completed saving to table People");
    }

• Run mvn clean install -Dmaven.build.cache.skipCache to get any remaining manual actions
• Complete the manual actions and run tilt up
• Once all the resources are ready on the tilt ui, start the spark-pipeline resource
• Verify you see the following log ouput in the pipeline:

INFO TestSyncStep: Completed saving to table People

• Connect to Trino using the cli: ./trino --server http://localhost:8084
  • See Trino CLI documentation for details on installation
• Run the following command to query the data:

select * from hive.default.people;

• Verify you get the following output:

    name    | age
------------+-----
 John Smith |  50
(1 row)

Query 20241122_143943_00000_c3nss, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
2.65 [1 rows, 14B] [0 rows/s, 5B/s]

• tilt down

References/Additional Context

A clear and concise description of any alternative solutions or features you've considered.
Add any other context, links, or screenshots about the feature request here.

[carter-cundiff]

DOD with @ganoshkin

[nartieri]

Testing Status: Passed! ✅

• Successfully saved the table to hive
• 
• Successfully accessed saved data
•